Repository navigation
[SPARK-62041][CONNECT][PYTHON] Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark - #869
Open
HyukjinKwon wants to merge 1 commit into
Open
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…lings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark - A window of no partitions after a pickler in the first build is recorded as its position, and pickle_in_turn gives each warning in its turn among the picklings, stopping at the first error, as PySpark's build warns. - merge_clause keeps the Column::sent of the proto built at the clause, of the condition and the assignments, so a merge sends the clause-time pickles, as PySpark's _build_merge_action. - toPandas keeps the execution info only once its conversion returns. - A pickled Column of a subquery carries the plans of its subqueries (SentExpression's subqueries), which its requests send, also on a Python worker; _ColumnExpression pickles as (getattr, (Column, "_expr")). - A type that does not convert fails the request of a pickled UDF or UDTF, not the pickle. - The docs: the window order, the merge, the subquery and _expr pickles, the reasons of the GC and of pickled Columns, and the Rust API's new SentExpression field. Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the documented differences of UDFs and pickles against PySpark 4.3.0, and of items found by other reviews. Each item was probed on both clients against a Spark 4.3.0 server, the merge pickles through a decoding proxy.
1. A window of no partitions warns in its turn with the picklings of UDFs.
W P:a P:bad, where PySpark givesP:a W P:bad).pickle_in_turngives each warning in its turn among the picklings, stopping at the first error, so an "error" filter raises at the same point as in PySpark.2. A merge sends its clauses as they were built.
_build_merge_actionbuilds the protos of a clause's condition and assignments when the clause is added, so the merge carries those pickles. The drop-in's merge carried pickles made when the merge ran (a decoding proxy: K=3 against PySpark's K=1), and a window in a clause warned twice.merge_clausenow keeps theColumn::sentof the proto built at the clause. Copies of the builder keep them, as PySpark's do. A clause that does not pickle still raises at the clause.3. A
toPandasthat fails after its execution leavesexecutionInfoas PySpark's does. PySpark sets it only once its conversion and copy return, so a failed conversion leavesNonethe first time, or the previous one. The drop-in kept the execution's at once. It now keeps the info only after_to_pandasreturns (DataFrame::keep_execution_info).4. Pickled subquery Columns keep the plans of their subqueries.
df.scalar(),df.exists()and the like lost the plan of its subquery: its requests failed on the server withINTERNAL_ERROR, also of a function on a Python worker of the server. PySpark's unpickled Column works.SentExpression::subqueries), each with its plan ID, which the request that holds it sends in itsWithRelations.SubqueryExpressions over the sent relations, which PySpark's_collect_referencescollects.5.
pickle.dumps(c._expr)pickles, as PySpark's expression does:_ColumnExpression.__reduce__gives(getattr, (Column, "_expr")). It raisedTypeError: cannot pickle '_ColumnExpression' object.6. A type that does not convert fails the request, not the pickle. The pickle of a Column of a UDF, or of a UDTF, of a return type that PySpark does not convert raised
UNSUPPORTED_OPERATIONat the pickle. Such types areAtomicType()withuseArrow=False, a struct changed after the call, or a UDTF return type 5. PySpark's pickle succeeds, and the Column or the table argument raises at its request. The pickle now keeps the error as that of its plan.7. Docs.
_exprpickles, the window order, and the Rust API's newSentExpressionfield and renamed hidden method.Why are the changes needed?
So that warnings, merges, execution info and pickles behave as PySpark's, and pickled subquery Columns work, in the client and on the server's workers.
Does this PR introduce any user-facing change?
Yes, as above. In the Rust API:
SentExpressionhas a new public field, so a struct expression that names every field must add it;PreparedExecutionis renamed.How was this patch tested?
test_dropin_offline.py, each failing on master:test_windows_of_no_partitions_warn_in_turn_with_the_picklings_of_udfs_as_pyspark;test_merges_send_the_clauses_as_their_actions_built_them_as_pyspark;test_a_to_pandas_that_fails_to_convert_leaves_the_execution_info_as_pyspark;test_pickles_of_types_that_do_not_convert_fail_where_the_plan_is_built_as_pyspark;test_pickled_columns_of_subqueries_send_the_plans_of_their_subqueries_as_pyspark;_exprat every protocol in the test of Column pickles.foreachBatchfunction on a worker._exprpickles, and the pickles of types that do not convert. All are identical.cargo test -p apache-spark-connect -p apache-spark-connect-core --release, and--features wasm-udf, and in a debug build;cargo test -p pyspark-rs --lib, linked asrust.ymllinks it.coverage.ymlin its order, on pandas 3, on a UTC host withoutTZ=UTC. Livetest_dropin_e2e.py. The type-stub drift check and test.cargo fmt --all --check,cargo +1.100.0 clippy --all-targets -- -D warnings, ruff 0.14.0,audit_no_stubs.sh,scripts/testsandRUSTDOCFLAGS="-D warnings" cargo doc --no-deps.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.