Skip to content

[SPARK-62007][CONNECT][PYTHON] Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request - #851

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62007-dataframe-patches-pandas-paths

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the documented differences of DataFrames (dev/parity/known-differences.md, docs/compatibility.md) against PySpark 4.3.0, and the other session's review of the last 100 commits. Each item was probed on both clients against a Spark 4.3.0 server.

1. Patches of pyspark.sql.DataFrame reach native frames as PySpark's frames see them.

  • PySpark 4.3.0's Connect DataFrame defines every public name of the base itself. So its frames ignore a patch of an existing name of the base, and see it only through DataFrame.m(df) (utils.py:363-387 dispatch_df_method).
  • The drop-in forwarded every patch to its native frames:
    • DataFrame.count = wrapper made df.count() return the wrapper's result, where PySpark returns the count;
    • mock.patch.object(DataFrame, "count", return_value=42) gave 42 where PySpark gives the count;
    • a patch of __getattr__ reached the frames;
    • a wrapper of the base and one of pyspark.sql.connect.dataframe.DataFrame that call their saved originals made df.count() raise RecursionError, where PySpark returns the count.
  • Now:
    • a patch of a name of PySpark's Connect class (its vars(), 141 names), or of __getattr__, is seen through DataFrame.m(df) only;
    • a new name, and __arrow_c_stream__, which PySpark's Connect class inherits, are forwarded, as PySpark's frames see them;
    • wrappers of inherited names end at the implementation;
    • the base's empty methods and properties are PySpark's: DataFrame.toPandas(df) and DataFrame.columns.fget(df) give None, as in PySpark, and the base has PySpark's annotations.

2. pandas createDataFrame takes PySpark's Arrow path for a StructType or StructField subclass that overrides none of the reads of the conversion, and for column names that hold "\x00".

  • PySpark tests isinstance (connect/session.py:600). Before, such data took the row path even with PyArrow, so 2**53+1 under a DoubleType field converted instead of raising, nanoseconds were truncated silently, and errors were of other classes.
  • A probe of 24 cases is now identical; 19 of them differed.
  • Names that hold NUL are read through Arrow IPC, as the C data interface truncates C strings at the NUL. This also fixes NUL names in nested inferred types.

3. createDataFrame names that UTF-8 cannot encode fail at the first request, through toDF, as PySpark's do, not at the call (session.rs data_name, label_name, pyarrow_import.rs create_dataframe).

4. Docs. In dev/parity/known-differences.md and docs/compatibility.md:

  • the reasons of the entries of the row paths without PyArrow (PySpark Connect cannot create a session without it), of a join on a tuple, of & of join conditions and of the native class's MRO;
  • the invalid UTF-8 of toArrow and toPandas (both raise UnicodeDecodeError);
  • the time zones;
  • the depth of plans that the server takes (1,024 nested messages, about 510 expressions);
  • StorageLevel pickles at protocols 0 and 1;
  • the Connect builder;
  • __copy__;
  • the surrogate names, of which PySpark fails only some at the call;
  • the batch reader's option of None, which PySpark's table() and changes() fail.
    docs/architecture.md describes the IPC read of NUL names.

Why are the changes needed?

So that code which patches or mocks pyspark.sql.DataFrame, as tests do, behaves as with PySpark and does not recurse; so that pandas data converts with PySpark's checks; and so that the docs are accurate.

Does this PR introduce any user-facing change?

Yes, as above:

  • a patch of an existing name of pyspark.sql.DataFrame no longer changes the drop-in's frames, as it does not change PySpark's; patch pyspark.sql.connect.dataframe.DataFrame, as with PySpark;
  • pandas data of struct subclasses and NUL names takes PySpark's Arrow path;
  • unencodable names fail at the request.

How was this patch tested?

  • New and changed tests.
    • test_ml_offline.py: the tests that pinned the forwarding of every patch now pin PySpark's behaviour; test_public_dataframe_wrappers_of_inherited_attributes_end_at_the_implementation; an annotations check.
    • test_arrow_handoff.py:
      • test_struct_subclasses_take_pysparks_conversion, and a test that subclasses which override the reads keep the row path;
      • test_nul_names_take_the_integer_path_whole and test_nul_names_take_the_arrow_paths_whole;
      • test_names_that_utf8_cannot_encode_fail_the_requests_as_pyspark.
  • Probes of both clients:
    • the patches: 10 of 10, 4 of 4 (was a RecursionError) and 12 of 12;
    • the pandas paths: 10 of 10, 7 of 7, and the other session's 72 of 72;
    • the names: 24 of 24;
    • the surrogate and reader-option cases.
  • Python. The offline suite of coverage.yml on pandas 2 and 3. Live test_arrow_e2e.py and test_dropin_e2e.py. The type-stub drift check and test, with mypy and pyright.
  • Rust. cargo test -p pyspark-rs --lib, linked as rust.yml links it.
  • Lint. cargo fmt --all --check, cargo +1.100.0 clippy -p pyspark-rs --all-targets -- -D warnings, ruff 0.14.0, audit_no_stubs.sh and scripts/tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62007-dataframe-patches-pandas-paths branch 4 times, most recently from 5943507 to a16095d Compare November 18, 2026 17:00
…reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request

- A patch of the public DataFrame reaches the native frames only where PySpark's Connect frames
  see it: a name that PySpark's Connect DataFrame defines (and __getattr__) is seen through
  DataFrame.m(df) only, a new name is forwarded, and wrappers of inherited names end at the
  implementation, so the RecursionError of two wrappers is gone. The empty methods and
  properties of the base are PySpark's.
- pandas createDataFrame takes PySpark's Arrow path for StructType and StructField subclasses
  that override none of its reads, and for names that hold NUL, which are read through Arrow IPC.
- createDataFrame names that UTF-8 cannot encode fail at the first request, through toDF.
- The known differences and the docs are updated (reasons, stale entries, the batch reader's
  option of None, the surrogate names).

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62007-dataframe-patches-pandas-paths branch from a16095d to 18be3ea Compare November 18, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants