Repository navigation
[SPARK-62021][CONNECT][PYTHON] Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's - #862
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
dongjoon-hyun
approved these changes
Nov 18, 2026
…ent) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
df._plan.plan(client) gave a placeholder of a marker, which the core replaced at the send with the
relation of the DataFrame of the marker, looked up in a registry of weak references. A placeholder
whose DataFrame was gone failed with "a placeholder of a DataFrame that no longer exists": the
inline client.to_table(pb2.Plan(root=spark.sql(q)._plan.plan(client)), {}), a relation kept after
its DataFrame, and a plan parsed again of its bytes. PySpark's plan(session) builds the relation at
the call, with its pickles, warnings and errors.
- The placeholder of _plan.plan(client) holds the relation, built at the call and encoded in the
value of its one extension, with the marker and the levels (standalone_placeholder_relation).
The core replaces it with that relation; the registry keeps weak entries for the Observations.
- A placeholder of a marker alone stays for the core's own plans that resolve it at once.
- The core's tests lift prost's recursion limit, as the drop-in does.
- The docs state the reasons of the differences that stay: expression extensions, placeholders
that a plugin changes, the plan ID of DataFrame(plan, session), the retry policies of can_retry
and to_state, and get_tags.
Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62021-placeholder-keeps-plan
branch
from
November 18, 2026 19:00
7bbf21a to
00f2684
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #816 (SPARK-61939), from a review of the documented differences of
SparkSession.clientagainst pyspark-client 4.3.0.The relation of a DataFrame's plan,
df._plan.plan(client), was a placeholder: an extension whose value is a marker. When a plan, a command or an analysis that holds it was sent, the core replaced it with the relation of the DataFrame of the marker. It looked the DataFrame up in a registry of weak references, so a placeholder whose DataFrame was gone failed witha placeholder of a DataFrame that no longer exists.PySpark's
LogicalPlan.plan(session)builds the relation at the call, so the message stands on its own. Checked on both clients against a Spark 4.3.0 server:client.to_table(pb2.Plan(root=spark.sql("SELECT 1 AS a")._plan.plan(client)), {}){'a': [1]}a placeholder of a DataFrame that no longer existsto_table, twice,schema, under aLimit, both sides of a join, aCreateDataFrameViewpb2.Plan.FromString(plan.SerializeToString())of such a plan{'id': [1, 2]}plan(client)of a DataFrame of a UDF that does not picklePicklingError, and no requestplan(client)of a window of no partitionsplan(client)The placeholder holds the relation.
_plan.plan(client)givesDataFrame::standalone_placeholder_relation. The core builds the relation at the call, as PySpark'splandoes: it pickles the functions of its UDFs, warns of a window of no partitions, and raises the errors of the build.child_observations). It keeps nothing alive.A placeholder of a marker alone stays for the plans that the core builds over a DataFrame and resolves while the DataFrame lives (
DataFrame::placeholder_relation, unchanged). These are the binding's ML transforms of a DataFrame of another session.The messages of extensions. The core recognized an unchanged placeholder by the exact length of the encoding of a marker. It now recognizes one by the key of the extension, which starts its encoding, and still by its decoding and re-encoding.
The core's tests decode the placeholders of deep DataFrames, as the drop-in does. The
pyspark-rscrate lifts prost's limit of 100 nested messages, socrates/spark-connect's dev-dependencies lift it for the tests too.The docs.
docs/compatibility.md,dev/parity/known-differences.md,AGENTS.mdand the docstring of_DataFramePlandescribe the placeholder. They also give the reasons that a review found missing or wrong for what stays different:DataFrame(plan, session)takes the client's plan ID, as the core numbers its relation with its own counter, which the plans ofpyspark.sql.connect.planstart past (1 << 48);can_retryorto_state, as it decides each retry in Rust without the GIL;get_tagsandSparkSession.getTagscopy the tags of the core session.Why are the changes needed?
So that the relations of the plans of DataFrames, which
SparkSession.clientand the plugins take as PySpark's, are sent whenever PySpark sends them. Their pickles, warnings and errors then come at the call, as in PySpark. Before, a plan built in one expression, the common way to write it, failed.Does this PR introduce any user-facing change?
Yes:
df._plan.plan(client)is sent after the DataFrame is gone, also of its bytes;plan(client)builds the relation, so it raises the errors of the build, such as thePicklingErrorof a UDF, and warns of a window of no partitions, as PySpark's does;The placeholder is larger, as it holds the relation; it is still of one level.
How was this patch tested?
plan.rs, in a debug build:a_placeholder_is_of_one_level_and_stands_for_the_relation_of_its_node(200 levels) also resolves the placeholder, and its bytes decoded again, after the node and the session's plan IDs are dropped;a_placeholder_of_a_dropped_node_stands_for_its_relation_and_keeps_nothing_alive;a_placeholder_of_a_marker_alone_stands_for_the_relation_of_its_node_while_it_lives;test_dropin_offline.py, with the mock server. Both fail on master and pass here:test_the_relations_of_the_plans_of_dataframes_are_built_when_they_are_given_as_pysparkcovers the inline plan, a relation kept after a collection, the bytes parsed again, the pickles of the call, and thePicklingErrorof the call with no request;test_ml_plans_send_a_deep_dataframe_without_parsing_it_as_pyspark. The placeholder of a deep DataFrame is an extension that holds its relation, and a plan over the placeholder of a freed DataFrame sends its relation.test_the_relations_of_the_plans_of_dataframes_are_sent_after_the_dataframes_are_goneintest_dropin_e2e.py, against a Spark 4.3.0 server. It fails on master, passes here, and its body passes on pyspark-client 4.3.0.DataFrame(SubqueryAlias(df._plan, "s"), spark)of a dropped DataFrame works on both.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-depsof the published crates withwasm-udf, as the lint job runs it;cargo test -p pyspark-rs --lib, linked asrust.ymllinks it: 13 passed;cargo test -p apache-spark-connect --libin a debug build: 792 passed.checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --checkandgen_fn_dispatch.py --check;test_type_stubs_offline.pywith mypy: 7 passed. The 2 pyright cases are skipped, as pyright is not installed here.coverage.yml: 6336 passed and 77 skipped. The skips: 68 "not a public constructor", 8 that needSPARK_REMOTE, andtest_type_stubs_offline.py, which needs mypy and ran above.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 320 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.