Repository navigation
[SPARK-62004][CONNECT][PYTHON] Pickle DataFrames of failing plans and table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark - #849
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
HyukjinKwon
force-pushed
the
SPARK-62004-dataframe-pickles-copies
branch
from
November 18, 2026 17:00
58b29e7 to
493ecbf
Compare
dongjoon-hyun
approved these changes
Nov 18, 2026
… table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark - A DataFrame whose plan fails with an AttributeError or another PySpark error class pickles, and its unpickle raises the error (NOT_ITERABLE of its schema request), as PySpark's. - A copy, deep copy or unpickle keeps the schema that it requests, which schema, printSchema, the set operations and numeric aggregates read, as PySpark's _cached_schema_serialized. - A TableArg pickles of its relation, levels, partitioning, ordering and single partition, and loads through _of_relation, or as PySpark Connect's TableArg on a worker. - A deep copy into the same session rebuilds its relation at each build, so its warnings and origins are PySpark's. - Window frame bounds that the core holds raise VALUE_NOT_BETWEEN when the plan is built. - Column._expr and TableArg._subquery_expr report their pickler objects to Python's GC, and copy as PySpark's. - The docs describe these and what remains. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62004-dataframe-pickles-copies
branch
from
November 18, 2026 17:56
493ecbf to
24d4302
Compare
HyukjinKwon
added a commit
that referenced
this pull request
Nov 18, 2026
…park does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString() ### What changes were proposed in this pull request? Four changes, each in its own commit, that make the drop-in treat user-defined type (UDT) objects as pyspark-client 4.3.0 does. A last commit renames a helper that `scripts/audit_no_stubs.sh` refused. With SPARK-61999 (#848), the toNullable of an array, a map or a struct is `as_nullable` of the objects, and a type of metadata or user attributes pickles of them, which keeps the UDT objects of part 2 too. With SPARK-62004 (#849), the schema that a copy or an unpickle of a DataFrame requested is converted as a schema of the server (part 3). **1. The simple strings of the mllib vector and matrix types** (core `DataType::simple_string`). The `VectorUDT` and `MatrixUDT` of `pyspark.mllib.linalg` are named `vector` and `matrix`, as those of `pyspark.ml.linalg` already were (SPARK-61690). A cast to them shows `VECTOR` and `MATRIX`, as PySpark's `CastExpression.__repr__` does. Other UDTs keep `udt`, which is PySpark's `UserDefinedType.simpleString()`. **2. A UDT object is kept in the types that hold it until a use needs its JSON** (`types.rs`). PySpark's `ArrayType`, `MapType`, `StructField` and `StructType` keep the type objects that they are given as they are, and read one only for a use that needs its JSON or its conversion. The drop-in lowered the child into a core type when it made the type, which calls `jsonValue()`. So the type of a UDT whose `module()` or `sqlType()` is not implemented, such as `UserDefinedType()` itself, raised `NOT_IMPLEMENTED` when it was made, by its constructor, `add` or a setter. - Now the child is lowered leniently when it is made (`lenient_type`). A type that does not lower is kept as its object, with the null type standing for it in the snapshot. - A use that needs the JSON reads the object again through `KeptType::live`, so it raises then, as PySpark's does: `json()`, the plans of a cast, `DataFrame.to`, `from_json`, a table, a Java UDF, a UDTF, a reader's schema, and data of no rows. - The uses that PySpark answers from the objects fall back to them when the strict lowering fails: - `simpleString` and `needConversion`; - `==`, by the attributes, as PySpark's `__dict__` comparison; - the hash; - pickles, copies and deep copies; - `toNullable`; - the tree string, which names a stand-in by its object's `typeName()`; - `names` and `add`. - `DataFrame.to` of such a schema raises the type's error at each request, where it raised `UNSUPPORTED_OPERATION` of the schema. **3. The schemas of the server make new UDT objects** (`types::schema_to_py`). PySpark converts the types of the server with `UserDefinedType.fromJson`, which makes a new object of the class, `UDT()` (`python/pyspark/sql/types.py:1986-2005`). The drop-in made them from a process-wide registry of the objects that the client lowered, by their JSON. So: - `df.schema["p"].dataType` was the user's own object, with its attributes; - lowering an unrelated UDT of the same JSON later changed the schema of an untouched DataFrame; - a class whose `__init__` takes arguments did not raise, where PySpark's `df.schema`, `df.dtypes` and `df.collect()` raise `TypeError`. Now the schema of a DataFrame and its `dtypes`, the schemas of the rows that `collect`, `toLocalIterator`, `head` and `take` convert, those of `toPandas` and `_to_table`, and `_data_type_of_proto` make each UDT a new object of its class. The array, map, field or struct that holds it keeps that object. The registry stays for the types that the client makes of core types (SPARK-61917). `DataFrame.to` keeps the struct that it is given as its cached schema, as SPARK-61962 made it, so its `schema` holds copies of the user's UDT objects, as PySpark's. **4. A cast renders the `simpleString()` of its type object** (core `CastTarget::Shown`, `Column::cast_shown`; pyspark-rs `cast_column`). PySpark's `CastExpression.__repr__` shows `simpleString().upper()` of the type object. The core rendered the simple string of its core type, so: - a UDT of its own `simpleString()` showed `UDT`, where PySpark shows `CAST(x AS SCALE2)` and `STRUCT<A:SCALE2,B:SCALE10>`; - a type of a Python subclass showed its class's (`CAST(id AS BOOLEAN)` where PySpark shows `CAST(id AS SUB)`), a known difference that this removes. The core has a hidden cast target that sends the type and renders a given string. The drop-in uses it with the object's `simpleString()` when the type holds a UDT or a type of a Python subclass. A cast to a type that does not lower now is made, renders its `simpleString()`, and raises at each request, as PySpark's. ### Why are the changes needed? So that UDT objects behave as in pyspark-client 4.3.0: - the types of a UDT of no JSON are made; - the schemas of DataFrames do not share the user's UDT objects, nor change with other types; - a UDT of arguments raises where PySpark's does; - casts show PySpark's strings. ### Does this PR introduce _any_ user-facing change? Yes. With `u = PtUDT(); u.marker = 1`, a `df` of `createDataFrame` of a schema of `u`, a `ScaleUDT(2)` whose `simpleString()` is `scale2`, and an `ArgUDT` whose `__init__` takes a scale: | | PySpark 4.3.0, and now | Before | |---|---|---| | `ArrayType(UserDefinedType())` | `ArrayType(UserDefinedType(), True)`; its `json()` raises `[NOT_IMPLEMENTED] module() is not implemented.` | raises that error | | `df.schema["p"].dataType is u` | `False`, a new `PtUDT()` without the attribute | `True` | | `df.schema` of a schema of `ArgUDT(2)` | `TypeError: ... missing 1 required positional argument: 'scale'` | the user's `ArgUDT(2)` | | `F.col("x").cast(ScaleUDT(2))` | `Column<'CAST(x AS SCALE2)'>` | `Column<'CAST(x AS UDT)'>` | | `F.col("x").cast(pyspark.mllib.linalg.VectorUDT())` | `Column<'CAST(x AS VECTOR)'>` | `Column<'CAST(x AS UDT)'>` | ### How was this patch tested? - Offline tests, of official PySpark 4.3.0's strings and errors. Their bodies pass on pyspark-client 4.3.0, but for the requests to the mock server, and they fail on master: - `test_casts_to_the_linalg_types_show_their_simple_strings_as_pyspark` (`test_dropin_offline.py`); - `test_types_of_a_user_defined_type_of_no_json_keep_it_as_pysparks` (`test_dropin_types_columns_offline.py`), of three such UDTs and seven containers: reprs, `simpleString`, `needConversion`, `==`, hash, copies, pickles, `json()`, `toInternal`, the tree string, `add`, `toNullable` and the setters; - `test_casts_show_the_simple_strings_of_their_type_objects_as_pysparks` (`test_dropin_types_columns_offline.py`). - E2E tests against a Spark 4.3.0 server. They pass on pyspark-client 4.3.0 and fail on master: - `test_uses_of_the_json_of_a_type_of_a_user_defined_type_of_no_json_raise_its_error`; - `test_schemas_of_the_server_make_new_user_defined_type_objects_as_pyspark`. `test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pyspark` defines its UDT at module level, which `fromJson` imports, so that it passes on pyspark-client 4.3.0 too. - Rust tests: - `simple_string_of_the_ml_udts_matches_pyspark` (`types_behavior.rs`); - `cast_shown_renders_its_shown_type_and_sends_its_type` (core `column.rs`). - Probes against the same server, of both clients, compared line by line. All match, but for `createDataFrame` of local rows or a pandas DataFrame of a UDT of no JSON, which is left for later: - 417 container operations of three UDTs of no JSON; - the uses of their JSON; - the schemas of the server; - the cast reprs of 10 types. - The gates below ran on the branch merged with master at 7ddb02d, with the commit of the stacked GC PR on top of this one's. - The Rust gates: - `cargo fmt --all --check`; - `cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings`; - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core`; - `cargo test -p pyspark-rs --lib` as `rust.yml` runs it: 15 passed; - `cargo test -p apache-spark-connect -p apache-spark-connect-core --release` as `rust.yml` runs it: 1706 passed, 18 ignored. - The Python gates: - ruff 0.14.0 `check` and `format --check`, `test_ruff_excludes.py` and `scripts/audit_no_stubs.sh`; - `gen_type_stubs.py --pyspark-src <v4.3.0>/python --check` and `gen_fn_dispatch.py --check`; - `test_type_stubs_offline.py` with mypy 2.4.1: 12 passed; - the guards of SPARK-61851. - The offline suite of `coverage.yml`: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that need `SPARK_REMOTE`, and `test_type_stubs_offline.py`, which needs mypy and ran above. - `test_dropin_e2e.py` against a Spark 4.3.0 server: 328 passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. Closes #865 from HyukjinKwon/SPARK-62016-udt-objects-as-pyspark. Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the documented differences of DataFrames against PySpark 4.3.0, and the other session's review of SPARK-61971's GC helpers. Each item was probed on both clients against a Spark 4.3.0 server, through a recording proxy.
1. DataFrames whose plan fails with an AttributeError pickle, as PySpark's do: a join on a tuple, or on conditions of a Row. Their unpickle raises the error,
NOT_ITERABLEof the schema request, as PySpark's does. The errors that a plan carries now cover the builtinAttributeErrorand the PySpark error classes. Columns of such errors pickle too, as in PySpark.2. A copy, deep copy or unpickle keeps the schema that it requests, as PySpark's
_cached_schema_serialized(connect/dataframe.py:2024-2050).schema,printSchema, the set operations' merge and the numeric aggregates ofgroupBy(_numeric_agg) from that schema.columns,dtypes,repranddirfrom the cached one.toArrowraised.filter,alias,limit,groupBy().sum(),toArrow,to()and the maps) gives the same results and the same requests in each phase on both clients.3. A table argument pickles.
df.asTable()pickles and cloudpickles its relation, its partitioning and ordering Columns, and its single partition, and loads through_of_relationand the same calls.TableArg. The official client loads the drop-in's cloudpickled table arguments, and a UDTF of them returns the same rows._subquery_exprstays a property.4. A deep copy into the same session rebuilds its relation at each build, through its picklers, so its requests give the no-partition window warning and the origins of a join's
andat each build, as PySpark's do. A pickle, and a copy into another session, carry the built relation as before.5. Window frame bounds that the core holds, any int of a long and the sentinels, raise
VALUE_NOT_BETWEENwhen the plan is built, as PySpark does, not atrowsBetweenorrangeBetween. A bound that the core cannot hold,Trueor an int beyond a long, still raises PySpark's first error at the call. This is documented.6.
Column._exprandTableArg._subquery_exprare reported to Python's garbage collector, as SPARK-61971 did for the other holders, so a reference cycle through a UDF of them is collected, as in PySpark. Theircopy.copyandcopy.deepcopycopy the core Column, as a Column's do, where they raisedTypeError: cannot pickle '_ColumnExpression' object.docs/compatibility.mdanddev/parity/known-differences.mddescribe these and the remaining differences:copy.pycall site of the schema requests of a copy.Why are the changes needed?
So that copies, pickles and table arguments behave as PySpark's, warnings and origins are PySpark's, and reference cycles through Column expressions are collected.
Does this PR introduce any user-facing change?
Yes, as above. In the Rust API: a doc-hidden
TableArg::relation_with_levels.How was this patch tested?
test_dropin_offline.py:test_copies_keep_the_schema_that_they_request_apart_from_the_cached_one_as_pyspark;_exprand_subquery_exprin the GC tests and the creation-traversal test.test_table_arg_pickles_as_pysparkintest_dropin_window_tablearg_offline.py.raise_errorcases intest_dropin_types_columns_offline.py.dataframe.rsandtable_arg.rs.cargo test -p apache-spark-connect -p apache-spark-connect-core --release, and--features wasm-udf, and in a debug build;cargo test -p pyspark-rs --lib, linked asrust.ymllinks it.coverage.ymlon pandas 2 and 3. Livetest_dropin_e2e.py.cargo fmt --all --check,cargo +1.100.0 clippy -p pyspark-rs -p apache-spark-connect --all-targets -- -D warnings, ruff 0.14.0,audit_no_stubs.shandscripts/tests.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.