Skip to content

[SPARK-62021][CONNECT][PYTHON] Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's - #862

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62021-placeholder-keeps-plan
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62021-placeholder-keeps-plan

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #816 (SPARK-61939), from a review of the documented differences of SparkSession.client against pyspark-client 4.3.0.

The relation of a DataFrame's plan, df._plan.plan(client), was a placeholder: an extension whose value is a marker. When a plan, a command or an analysis that holds it was sent, the core replaced it with the relation of the DataFrame of the marker. It looked the DataFrame up in a registry of weak references, so a placeholder whose DataFrame was gone failed with a placeholder of a DataFrame that no longer exists.

PySpark's LogicalPlan.plan(session) builds the relation at the call, so the message stands on its own. Checked on both clients against a Spark 4.3.0 server:

pyspark-client 4.3.0 drop-in before
client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS a")._plan.plan(client)), {}) {'a': [1]} a placeholder of a DataFrame that no longer exists
a relation kept after its DataFrame is freed: to_table, twice, schema, under a Limit, both sides of a join, a CreateDataFrameView the results the same error
pb2.Plan.FromString(plan.SerializeToString()) of such a plan {'id': [1, 2]} the same error
plan(client) of a DataFrame of a UDF that does not pickle PicklingError, and no request returns; the request raises it
plan(client) of a window of no partitions warns at the call warns at the request
a UDF whose state changes after plan(client) sent with the state of the call sent with the state of the request

The placeholder holds the relation.

  • _plan.plan(client) gives DataFrame::standalone_placeholder_relation. The core builds the relation at the call, as PySpark's plan does: it pickles the functions of its UDFs, warns of a window of no partitions, and raises the errors of the build.
  • It encodes the relation in the value of the extension, after the marker and the levels of the relation. The placeholder is still one message: Python's protobuf never parses the relation of a deep DataFrame, which the extension holds as bytes.
  • The core replaces the placeholder with the relation that it holds, decoded on a stack grown for its levels. This works after the DataFrame is gone, and of the bytes of a plan parsed again.
  • The registry by marker keeps its weak entries only to find the Observations of the DataFrame of a placeholder (child_observations). It keeps nothing alive.

A placeholder of a marker alone stays for the plans that the core builds over a DataFrame and resolves while the DataFrame lives (DataFrame::placeholder_relation, unchanged). These are the binding's ML transforms of a DataFrame of another session.

The messages of extensions. The core recognized an unchanged placeholder by the exact length of the encoding of a marker. It now recognizes one by the key of the extension, which starts its encoding, and still by its decoding and re-encoding.

The core's tests decode the placeholders of deep DataFrames, as the drop-in does. The pyspark-rs crate lifts prost's limit of 100 nested messages, so crates/spark-connect's dev-dependencies lift it for the tests too.

The docs. docs/compatibility.md, dev/parity/known-differences.md, AGENTS.md and the docstring of _DataFramePlan describe the placeholder. They also give the reasons that a review found missing or wrong for what stays different:

  • the core does not decode the expressions of a plan built in Python, and takes a placeholder in an extension's message by its encoding, which a plugin's change alters;
  • DataFrame(plan, session) takes the client's plan ID, as the core numbers its relation with its own counter, which the plans of pyspark.sql.connect.plan start past (1 << 48);
  • the transport does not call a retry policy's own can_retry or to_state, as it decides each retry in Rust without the GIL;
  • get_tags and SparkSession.getTags copy the tags of the core session.

Why are the changes needed?

So that the relations of the plans of DataFrames, which SparkSession.client and the plugins take as PySpark's, are sent whenever PySpark sends them. Their pickles, warnings and errors then come at the call, as in PySpark. Before, a plan built in one expression, the common way to write it, failed.

Does this PR introduce any user-facing change?

Yes:

  • a plan that holds the relation of df._plan.plan(client) is sent after the DataFrame is gone, also of its bytes;
  • plan(client) builds the relation, so it raises the errors of the build, such as the PicklingError of a UDF, and warns of a window of no partitions, as PySpark's does;
  • the relation sent is that of the call.

The placeholder is larger, as it holds the relation; it is still of one level.

How was this patch tested?

  • Rust unit tests in plan.rs, in a debug build:
    • a_placeholder_is_of_one_level_and_stands_for_the_relation_of_its_node (200 levels) also resolves the placeholder, and its bytes decoded again, after the node and the session's plan IDs are dropped;
    • a_placeholder_of_a_dropped_node_stands_for_its_relation_and_keeps_nothing_alive;
    • a_placeholder_of_a_marker_alone_stands_for_the_relation_of_its_node_while_it_lives;
    • the extension test also resolves a placeholder of a dropped DataFrame in a plugin's message.
  • Offline tests in test_dropin_offline.py, with the mock server. Both fail on master and pass here:
    • new: test_the_relations_of_the_plans_of_dataframes_are_built_when_they_are_given_as_pyspark covers the inline plan, a relation kept after a collection, the bytes parsed again, the pickles of the call, and the PicklingError of the call with no request;
    • updated: test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark. The placeholder of a deep DataFrame is an extension that holds its relation, and a plan over the placeholder of a freed DataFrame sends its relation.
  • e2e test test_the_relations_of_the_plans_of_dataframes_are_sent_after_the_dataframes_are_gone in test_dropin_e2e.py, against a Spark 4.3.0 server. It fails on master, passes here, and its body passes on pyspark-client 4.3.0.
  • Probes of both clients: the 9 cases of the table give pyspark-client 4.3.0's results. DataFrame(SubqueryAlias(df._plan, "s"), spark) of a dropped DataFrame works on both.
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps of the published crates with wasm-udf, as the lint job runs it;
    • cargo test -p pyspark-rs --lib, linked as rust.yml links it: 13 passed;
    • cargo test -p apache-spark-connect --lib in a debug build: 792 passed.
  • The Python gates:
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --check and gen_fn_dispatch.py --check;
    • the test-list guards;
    • test_type_stubs_offline.py with mypy: 7 passed. The 2 pyright cases are skipped, as pyright is not installed here.
  • The offline suite of coverage.yml: 6336 passed and 77 skipped. The skips: 68 "not a public constructor", 8 that need SPARK_REMOTE, and test_type_stubs_offline.py, which needs mypy and ran above.
  • test_dropin_e2e.py against a Spark 4.3.0 server: 320 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…ent) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's

df._plan.plan(client) gave a placeholder of a marker, which the core replaced at the send with the
relation of the DataFrame of the marker, looked up in a registry of weak references. A placeholder
whose DataFrame was gone failed with "a placeholder of a DataFrame that no longer exists": the
inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)), {}), a relation kept after
its DataFrame, and a plan parsed again of its bytes. PySpark's plan(session) builds the relation at
the call, with its pickles, warnings and errors.

- The placeholder of _plan.plan(client) holds the relation, built at the call and encoded in the
  value of its one extension, with the marker and the levels (standalone_placeholder_relation).
  The core replaces it with that relation; the registry keeps weak entries for the Observations.
- A placeholder of a marker alone stays for the core's own plans that resolve it at once.
- The core's tests lift prost's recursion limit, as the drop-in does.
- The docs state the reasons of the differences that stay: expression extensions, placeholders
  that a plugin changes, the plan ID of DataFrame(plan, session), the retry policies of can_retry
  and to_state, and get_tags.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62021-placeholder-keeps-plan branch from 7bbf21a to 00f2684 Compare November 18, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants