Skip to content

[SPARK-62041][CONNECT][PYTHON] Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark - #869

Open
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62041-window-warnings-merge-pickles-subqueries

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the documented differences of UDFs and pickles against PySpark 4.3.0, and of items found by other reviews. Each item was probed on both clients against a Spark 4.3.0 server, the merge pickles through a decoding proxy.

1. A window of no partitions warns in its turn with the picklings of UDFs.

  • PySpark's build of a plan pickles each UDF as it builds the UDF's expression, and warns of a window of no partitions as it builds the window (connect/expressions.py:1126). So a UDF that does not pickle before the window raises its error, with no warning.
  • The drop-in warned first, then pickled (W P:a P:bad, where PySpark gives P:a W P:bad).
  • During the first build of a plan with picklers, a window after a pickler is now recorded as its position. pickle_in_turn gives each warning in its turn among the picklings, stopping at the first error, so an "error" filter raises at the same point as in PySpark.
  • A window before any pickler warns at once, as before.

2. A merge sends its clauses as they were built.

  • PySpark's _build_merge_action builds the protos of a clause's condition and assignments when the clause is added, so the merge carries those pickles. The drop-in's merge carried pickles made when the merge ran (a decoding proxy: K=3 against PySpark's K=1), and a window in a clause warned twice.
  • merge_clause now keeps the Column::sent of the proto built at the clause. Copies of the builder keep them, as PySpark's do. A clause that does not pickle still raises at the clause.

3. A toPandas that fails after its execution leaves executionInfo as PySpark's does. PySpark sets it only once its conversion and copy return, so a failed conversion leaves None the first time, or the previous one. The drop-in kept the execution's at once. It now keeps the info only after _to_pandas returns (DataFrame::keep_execution_info).

4. Pickled subquery Columns keep the plans of their subqueries.

  • A pickled Column of df.scalar(), df.exists() and the like lost the plan of its subquery: its requests failed on the server with INTERNAL_ERROR, also of a function on a Python worker of the server. PySpark's unpickled Column works.
  • The pickle of a Column now carries its subqueries' relations (SentExpression::subqueries), each with its plan ID, which the request that holds it sends in its WithRelations.
  • On a worker, the expression's children are SubqueryExpressions over the sent relations, which PySpark's _collect_references collects.
  • All 7 subquery cases give PySpark's rows, in the client and on a worker.

5. pickle.dumps(c._expr) pickles, as PySpark's expression does: _ColumnExpression.__reduce__ gives (getattr, (Column, "_expr")). It raised TypeError: cannot pickle '_ColumnExpression' object.

6. A type that does not convert fails the request, not the pickle. The pickle of a Column of a UDF, or of a UDTF, of a return type that PySpark does not convert raised UNSUPPORTED_OPERATION at the pickle. Such types are AtomicType() with useArrow=False, a struct changed after the call, or a UDTF return type 5. PySpark's pickle succeeds, and the Column or the table argument raises at its request. The pickle now keeps the error as that of its plan.

7. Docs.

  • The GC entry gives its real constraint, where it said "so that no reference is counted twice": the holders share one reference to a function, so reporting it from one could let the collector clear it while another live holder reaches it.
  • The pickled-Column entry gives its trade-off: carrying the functions would make a plain pickle of a lambda-UDF Column fail as in PySpark, and would change every plan holder's pickle.
  • The merge entry is removed.
  • New: the subquery and _expr pickles, the window order, and the Rust API's new SentExpression field and renamed hidden method.

Why are the changes needed?

So that warnings, merges, execution info and pickles behave as PySpark's, and pickled subquery Columns work, in the client and on the server's workers.

Does this PR introduce any user-facing change?

Yes, as above. In the Rust API:

  • SentExpression has a new public field, so a struct expression that names every field must add it;
  • a hidden method of PreparedExecution is renamed.

How was this patch tested?

  • New tests in test_dropin_offline.py, each failing on master:
    • test_windows_of_no_partitions_warn_in_turn_with_the_picklings_of_udfs_as_pyspark;
    • test_merges_send_the_clauses_as_their_actions_built_them_as_pyspark;
    • test_a_to_pandas_that_fails_to_convert_leaves_the_execution_info_as_pyspark;
    • test_pickles_of_types_that_do_not_convert_fail_where_the_plan_is_built_as_pyspark;
    • test_pickled_columns_of_subqueries_send_the_plans_of_their_subqueries_as_pyspark;
    • _expr at every protocol in the test of Column pickles.
  • A live test of pickled subquery Columns, also in a foreachBatch function on a worker.
  • Core tests of the window's turn and of sent subqueries.
  • Probes of both clients: the pickle and warning traces (7 cases × "always"/"error"), the merge pickles, the execution info, 15 subquery lines, the _expr pickles, and the pickles of types that do not convert. All are identical.
  • Rust. cargo test -p apache-spark-connect -p apache-spark-connect-core --release, and --features wasm-udf, and in a debug build; cargo test -p pyspark-rs --lib, linked as rust.yml links it.
  • Python. The offline suite of coverage.yml in its order, on pandas 3, on a UTC host without TZ=UTC. Live test_dropin_e2e.py. The type-stub drift check and test.
  • Lint. cargo fmt --all --check, cargo +1.100.0 clippy --all-targets -- -D warnings, ruff 0.14.0, audit_no_stubs.sh, scripts/tests and RUSTDOCFLAGS="-D warnings" cargo doc --no-deps.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…lings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark

- A window of no partitions after a pickler in the first build is recorded as its position, and
  pickle_in_turn gives each warning in its turn among the picklings, stopping at the first error,
  as PySpark's build warns.
- merge_clause keeps the Column::sent of the proto built at the clause, of the condition and the
  assignments, so a merge sends the clause-time pickles, as PySpark's _build_merge_action.
- toPandas keeps the execution info only once its conversion returns.
- A pickled Column of a subquery carries the plans of its subqueries (SentExpression's
  subqueries), which its requests send, also on a Python worker; _ColumnExpression pickles as
  (getattr, (Column, "_expr")).
- A type that does not convert fails the request of a pickled UDF or UDTF, not the pickle.
- The docs: the window order, the merge, the subquery and _expr pickles, the reasons of the GC
  and of pickled Columns, and the Rust API's new SentExpression field.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants