Skip to content

[SPARK-62004][CONNECT][PYTHON] Pickle DataFrames of failing plans and table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark - #849

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62004-dataframe-pickles-copies

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the documented differences of DataFrames against PySpark 4.3.0, and the other session's review of SPARK-61971's GC helpers. Each item was probed on both clients against a Spark 4.3.0 server, through a recording proxy.

1. DataFrames whose plan fails with an AttributeError pickle, as PySpark's do: a join on a tuple, or on conditions of a Row. Their unpickle raises the error, NOT_ITERABLE of the schema request, as PySpark's does. The errors that a plan carries now cover the builtin AttributeError and the PySpark error classes. Columns of such errors pickle too, as in PySpark.

2. A copy, deep copy or unpickle keeps the schema that it requests, as PySpark's _cached_schema_serialized (connect/dataframe.py:2024-2050).

  • PySpark answers schema, printSchema, the set operations' merge and the numeric aggregates of groupBy (_numeric_agg) from that schema.
  • It answers columns, dtypes, repr and dir from the cached one.
  • The drop-in answered all from the cached schema. So a view replaced between the DataFrame and its copy gave other results, and the copy's toArrow raised.
  • A probe of 225 cases (copies, deep copies, unpickles, copies of copies, every set operation, filter, alias, limit, groupBy().sum(), toArrow, to() and the maps) gives the same results and the same requests in each phase on both clients.

3. A table argument pickles.

  • df.asTable() pickles and cloudpickles its relation, its partitioning and ordering Columns, and its single partition, and loads through _of_relation and the same calls.
  • On a Python worker of the server, it loads as PySpark Connect's own TableArg. The official client loads the drop-in's cloudpickled table arguments, and a UDTF of them returns the same rows.
  • _subquery_expr stays a property.

4. A deep copy into the same session rebuilds its relation at each build, through its picklers, so its requests give the no-partition window warning and the origins of a join's and at each build, as PySpark's do. A pickle, and a copy into another session, carry the built relation as before.

5. Window frame bounds that the core holds, any int of a long and the sentinels, raise VALUE_NOT_BETWEEN when the plan is built, as PySpark does, not at rowsBetween or rangeBetween. A bound that the core cannot hold, True or an int beyond a long, still raises PySpark's first error at the call. This is documented.

6. Column._expr and TableArg._subquery_expr are reported to Python's garbage collector, as SPARK-61971 did for the other holders, so a reference cycle through a UDF of them is collected, as in PySpark. Their copy.copy and copy.deepcopy copy the core Column, as a Column's do, where they raised TypeError: cannot pickle '_ColumnExpression' object.

docs/compatibility.md and dev/parity/known-differences.md describe these and the remaining differences:

  • a table argument unpickles where the session of its ID is active;
  • the copy.py call site of the schema requests of a copy.

Why are the changes needed?

So that copies, pickles and table arguments behave as PySpark's, warnings and origins are PySpark's, and reference cycles through Column expressions are collected.

Does this PR introduce any user-facing change?

Yes, as above. In the Rust API: a doc-hidden TableArg::relation_with_levels.

How was this patch tested?

  • New and extended tests.
    • test_dropin_offline.py:
      • test_copies_keep_the_schema_that_they_request_apart_from_the_cached_one_as_pyspark;
      • AttributeError joins in the unpickle test;
      • deep copies in the window-warning and origin tests;
      • the frame-bound tests at the build;
      • _expr and _subquery_expr in the GC tests and the creation-traversal test.
    • test_table_arg_pickles_as_pyspark in test_dropin_window_tablearg_offline.py.
    • raise_error cases in test_dropin_types_columns_offline.py.
    • Core unit tests in dataframe.rs and table_arg.rs.
  • Probes of both clients: the pickles of failing plans (33 and 30 cases), the copy schemas (225 cases), the table-argument pickles at protocols 0-5 and their requests, the deep-copy warnings and origins, the frame bounds (37 cases), and the GC of the expressions.
  • Rust. cargo test -p apache-spark-connect -p apache-spark-connect-core --release, and --features wasm-udf, and in a debug build; cargo test -p pyspark-rs --lib, linked as rust.yml links it.
  • Python. The offline suite of coverage.yml on pandas 2 and 3. Live test_dropin_e2e.py.
  • Lint. cargo fmt --all --check, cargo +1.100.0 clippy -p pyspark-rs -p apache-spark-connect --all-targets -- -D warnings, ruff 0.14.0, audit_no_stubs.sh and scripts/tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62004-dataframe-pickles-copies branch from 58b29e7 to 493ecbf Compare November 18, 2026 17:00
… table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark

- A DataFrame whose plan fails with an AttributeError or another PySpark error class pickles,
  and its unpickle raises the error (NOT_ITERABLE of its schema request), as PySpark's.
- A copy, deep copy or unpickle keeps the schema that it requests, which schema, printSchema,
  the set operations and numeric aggregates read, as PySpark's _cached_schema_serialized.
- A TableArg pickles of its relation, levels, partitioning, ordering and single partition, and
  loads through _of_relation, or as PySpark Connect's TableArg on a worker.
- A deep copy into the same session rebuilds its relation at each build, so its warnings and
  origins are PySpark's.
- Window frame bounds that the core holds raise VALUE_NOT_BETWEEN when the plan is built.
- Column._expr and TableArg._subquery_expr report their pickler objects to Python's GC, and copy
  as PySpark's.
- The docs describe these and what remains.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62004-dataframe-pickles-copies branch from 493ecbf to 24d4302 Compare November 18, 2026 17:56
HyukjinKwon added a commit that referenced this pull request Nov 18, 2026
…park does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()

### What changes were proposed in this pull request?

Four changes, each in its own commit, that make the drop-in treat user-defined type (UDT) objects as pyspark-client 4.3.0 does. A last commit renames a helper that `scripts/audit_no_stubs.sh` refused. With SPARK-61999 (#848), the toNullable of an array, a map or a struct is `as_nullable` of the objects, and a type of metadata or user attributes pickles of them, which keeps the UDT objects of part 2 too. With SPARK-62004 (#849), the schema that a copy or an unpickle of a DataFrame requested is converted as a schema of the server (part 3).

**1. The simple strings of the mllib vector and matrix types** (core `DataType::simple_string`). The `VectorUDT` and `MatrixUDT` of `pyspark.mllib.linalg` are named `vector` and `matrix`, as those of `pyspark.ml.linalg` already were (SPARK-61690). A cast to them shows `VECTOR` and `MATRIX`, as PySpark's `CastExpression.__repr__` does. Other UDTs keep `udt`, which is PySpark's `UserDefinedType.simpleString()`.

**2. A UDT object is kept in the types that hold it until a use needs its JSON** (`types.rs`). PySpark's `ArrayType`, `MapType`, `StructField` and `StructType` keep the type objects that they are given as they are, and read one only for a use that needs its JSON or its conversion. The drop-in lowered the child into a core type when it made the type, which calls `jsonValue()`. So the type of a UDT whose `module()` or `sqlType()` is not implemented, such as `UserDefinedType()` itself, raised `NOT_IMPLEMENTED` when it was made, by its constructor, `add` or a setter.
- Now the child is lowered leniently when it is made (`lenient_type`). A type that does not lower is kept as its object, with the null type standing for it in the snapshot.
- A use that needs the JSON reads the object again through `KeptType::live`, so it raises then, as PySpark's does: `json()`, the plans of a cast, `DataFrame.to`, `from_json`, a table, a Java UDF, a UDTF, a reader's schema, and data of no rows.
- The uses that PySpark answers from the objects fall back to them when the strict lowering fails:
  - `simpleString` and `needConversion`;
  - `==`, by the attributes, as PySpark's `__dict__` comparison;
  - the hash;
  - pickles, copies and deep copies;
  - `toNullable`;
  - the tree string, which names a stand-in by its object's `typeName()`;
  - `names` and `add`.
- `DataFrame.to` of such a schema raises the type's error at each request, where it raised `UNSUPPORTED_OPERATION` of the schema.

**3. The schemas of the server make new UDT objects** (`types::schema_to_py`). PySpark converts the types of the server with `UserDefinedType.fromJson`, which makes a new object of the class, `UDT()` (`python/pyspark/sql/types.py:1986-2005`). The drop-in made them from a process-wide registry of the objects that the client lowered, by their JSON. So:
- `df.schema["p"].dataType` was the user's own object, with its attributes;
- lowering an unrelated UDT of the same JSON later changed the schema of an untouched DataFrame;
- a class whose `__init__` takes arguments did not raise, where PySpark's `df.schema`, `df.dtypes` and `df.collect()` raise `TypeError`.

Now the schema of a DataFrame and its `dtypes`, the schemas of the rows that `collect`, `toLocalIterator`, `head` and `take` convert, those of `toPandas` and `_to_table`, and `_data_type_of_proto` make each UDT a new object of its class. The array, map, field or struct that holds it keeps that object. The registry stays for the types that the client makes of core types (SPARK-61917). `DataFrame.to` keeps the struct that it is given as its cached schema, as SPARK-61962 made it, so its `schema` holds copies of the user's UDT objects, as PySpark's.

**4. A cast renders the `simpleString()` of its type object** (core `CastTarget::Shown`, `Column::cast_shown`; pyspark-rs `cast_column`). PySpark's `CastExpression.__repr__` shows `simpleString().upper()` of the type object. The core rendered the simple string of its core type, so:
- a UDT of its own `simpleString()` showed `UDT`, where PySpark shows `CAST(x AS SCALE2)` and `STRUCT<A:SCALE2,B:SCALE10>`;
- a type of a Python subclass showed its class's (`CAST(id AS BOOLEAN)` where PySpark shows `CAST(id AS SUB)`), a known difference that this removes.

The core has a hidden cast target that sends the type and renders a given string. The drop-in uses it with the object's `simpleString()` when the type holds a UDT or a type of a Python subclass. A cast to a type that does not lower now is made, renders its `simpleString()`, and raises at each request, as PySpark's.

### Why are the changes needed?

So that UDT objects behave as in pyspark-client 4.3.0:
- the types of a UDT of no JSON are made;
- the schemas of DataFrames do not share the user's UDT objects, nor change with other types;
- a UDT of arguments raises where PySpark's does;
- casts show PySpark's strings.

### Does this PR introduce _any_ user-facing change?

Yes. With `u = PtUDT(); u.marker = 1`, a `df` of `createDataFrame` of a schema of `u`, a `ScaleUDT(2)` whose `simpleString()` is `scale2`, and an `ArgUDT` whose `__init__` takes a scale:

| | PySpark 4.3.0, and now | Before |
|---|---|---|
| `ArrayType(UserDefinedType())` | `ArrayType(UserDefinedType(), True)`; its `json()` raises `[NOT_IMPLEMENTED] module() is not implemented.` | raises that error |
| `df.schema["p"].dataType is u` | `False`, a new `PtUDT()` without the attribute | `True` |
| `df.schema` of a schema of `ArgUDT(2)` | `TypeError: ... missing 1 required positional argument: 'scale'` | the user's `ArgUDT(2)` |
| `F.col("x").cast(ScaleUDT(2))` | `Column<'CAST(x AS SCALE2)'>` | `Column<'CAST(x AS UDT)'>` |
| `F.col("x").cast(pyspark.mllib.linalg.VectorUDT())` | `Column<'CAST(x AS VECTOR)'>` | `Column<'CAST(x AS UDT)'>` |

### How was this patch tested?

- Offline tests, of official PySpark 4.3.0's strings and errors. Their bodies pass on pyspark-client 4.3.0, but for the requests to the mock server, and they fail on master:
  - `test_casts_to_the_linalg_types_show_their_simple_strings_as_pyspark` (`test_dropin_offline.py`);
  - `test_types_of_a_user_defined_type_of_no_json_keep_it_as_pysparks` (`test_dropin_types_columns_offline.py`), of three such UDTs and seven containers: reprs, `simpleString`, `needConversion`, `==`, hash, copies, pickles, `json()`, `toInternal`, the tree string, `add`, `toNullable` and the setters;
  - `test_casts_show_the_simple_strings_of_their_type_objects_as_pysparks` (`test_dropin_types_columns_offline.py`).
- E2E tests against a Spark 4.3.0 server. They pass on pyspark-client 4.3.0 and fail on master:
  - `test_uses_of_the_json_of_a_type_of_a_user_defined_type_of_no_json_raise_its_error`;
  - `test_schemas_of_the_server_make_new_user_defined_type_objects_as_pyspark`.

  `test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pyspark` defines its UDT at module level, which `fromJson` imports, so that it passes on pyspark-client 4.3.0 too.
- Rust tests:
  - `simple_string_of_the_ml_udts_matches_pyspark` (`types_behavior.rs`);
  - `cast_shown_renders_its_shown_type_and_sends_its_type` (core `column.rs`).
- Probes against the same server, of both clients, compared line by line. All match, but for `createDataFrame` of local rows or a pandas DataFrame of a UDT of no JSON, which is left for later:
  - 417 container operations of three UDTs of no JSON;
  - the uses of their JSON;
  - the schemas of the server;
  - the cast reprs of 10 types.
- The gates below ran on the branch merged with master at 7ddb02d, with the commit of the stacked GC PR on top of this one's.
- The Rust gates:
  - `cargo fmt --all --check`;
  - `cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings`;
  - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core`;
  - `cargo test -p pyspark-rs --lib` as `rust.yml` runs it: 15 passed;
  - `cargo test -p apache-spark-connect -p apache-spark-connect-core --release` as `rust.yml` runs it: 1706 passed, 18 ignored.
- The Python gates:
  - ruff 0.14.0 `check` and `format --check`, `test_ruff_excludes.py` and `scripts/audit_no_stubs.sh`;
  - `gen_type_stubs.py --pyspark-src <v4.3.0>/python --check` and `gen_fn_dispatch.py --check`;
  - `test_type_stubs_offline.py` with mypy 2.4.1: 12 passed;
  - the guards of SPARK-61851.
- The offline suite of `coverage.yml`: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that need `SPARK_REMOTE`, and `test_type_stubs_offline.py`, which needs mypy and ran above.
- `test_dropin_e2e.py` against a Spark 4.3.0 server: 328 passed.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

Closes #865 from HyukjinKwon/SPARK-62016-udt-objects-as-pyspark.

Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants