Repository navigation
[SPARK-62007][CONNECT][PYTHON] Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request - #851
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
HyukjinKwon
force-pushed
the
SPARK-62007-dataframe-patches-pandas-paths
branch
4 times, most recently
from
November 18, 2026 17:00
5943507 to
a16095d
Compare
dongjoon-hyun
approved these changes
Nov 18, 2026
…reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request - A patch of the public DataFrame reaches the native frames only where PySpark's Connect frames see it: a name that PySpark's Connect DataFrame defines (and __getattr__) is seen through DataFrame.m(df) only, a new name is forwarded, and wrappers of inherited names end at the implementation, so the RecursionError of two wrappers is gone. The empty methods and properties of the base are PySpark's. - pandas createDataFrame takes PySpark's Arrow path for StructType and StructField subclasses that override none of its reads, and for names that hold NUL, which are read through Arrow IPC. - createDataFrame names that UTF-8 cannot encode fail at the first request, through toDF. - The known differences and the docs are updated (reasons, stale entries, the batch reader's option of None, the surrogate names). Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62007-dataframe-patches-pandas-paths
branch
from
November 18, 2026 17:56
a16095d to
18be3ea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the documented differences of DataFrames (
dev/parity/known-differences.md,docs/compatibility.md) against PySpark 4.3.0, and the other session's review of the last 100 commits. Each item was probed on both clients against a Spark 4.3.0 server.1. Patches of
pyspark.sql.DataFramereach native frames as PySpark's frames see them.DataFramedefines every public name of the base itself. So its frames ignore a patch of an existing name of the base, and see it only throughDataFrame.m(df)(utils.py:363-387dispatch_df_method).DataFrame.count = wrappermadedf.count()return the wrapper's result, where PySpark returns the count;mock.patch.object(DataFrame, "count", return_value=42)gave 42 where PySpark gives the count;__getattr__reached the frames;pyspark.sql.connect.dataframe.DataFramethat call their saved originals madedf.count()raiseRecursionError, where PySpark returns the count.vars(), 141 names), or of__getattr__, is seen throughDataFrame.m(df)only;__arrow_c_stream__, which PySpark's Connect class inherits, are forwarded, as PySpark's frames see them;DataFrame.toPandas(df)andDataFrame.columns.fget(df)giveNone, as in PySpark, and the base has PySpark's annotations.2. pandas
createDataFrametakes PySpark's Arrow path for aStructTypeorStructFieldsubclass that overrides none of the reads of the conversion, and for column names that hold"\x00".isinstance(connect/session.py:600). Before, such data took the row path even with PyArrow, so2**53+1under aDoubleTypefield converted instead of raising, nanoseconds were truncated silently, and errors were of other classes.3.
createDataFramenames that UTF-8 cannot encode fail at the first request, throughtoDF, as PySpark's do, not at the call (session.rsdata_name,label_name,pyarrow_import.rscreate_dataframe).4. Docs. In
dev/parity/known-differences.mdanddocs/compatibility.md:&of join conditions and of the native class's MRO;toArrowandtoPandas(both raiseUnicodeDecodeError);StorageLevelpickles at protocols 0 and 1;__copy__;None, which PySpark'stable()andchanges()fail.docs/architecture.mddescribes the IPC read of NUL names.Why are the changes needed?
So that code which patches or mocks
pyspark.sql.DataFrame, as tests do, behaves as with PySpark and does not recurse; so that pandas data converts with PySpark's checks; and so that the docs are accurate.Does this PR introduce any user-facing change?
Yes, as above:
pyspark.sql.DataFrameno longer changes the drop-in's frames, as it does not change PySpark's; patchpyspark.sql.connect.dataframe.DataFrame, as with PySpark;How was this patch tested?
test_ml_offline.py: the tests that pinned the forwarding of every patch now pin PySpark's behaviour;test_public_dataframe_wrappers_of_inherited_attributes_end_at_the_implementation; an annotations check.test_arrow_handoff.py:test_struct_subclasses_take_pysparks_conversion, and a test that subclasses which override the reads keep the row path;test_nul_names_take_the_integer_path_wholeandtest_nul_names_take_the_arrow_paths_whole;test_names_that_utf8_cannot_encode_fail_the_requests_as_pyspark.RecursionError) and 12 of 12;coverage.ymlon pandas 2 and 3. Livetest_arrow_e2e.pyandtest_dropin_e2e.py. The type-stub drift check and test, with mypy and pyright.cargo test -p pyspark-rs --lib, linked asrust.ymllinks it.cargo fmt --all --check,cargo +1.100.0 clippy -p pyspark-rs --all-targets -- -D warnings, ruff 0.14.0,audit_no_stubs.shandscripts/tests.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.