Skip to content

[SPARK-60898][CONNECT] Keep plan IDs stable across re-executions to reuse the server plan cache - #147

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-60898-stable-plan-ids
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-60898-stable-plan-ids

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

SPARK-60898: the Rust client now gives each plan node its plan ID once, the first time it is serialized, and reuses that ID afterwards, as PySpark assigns plan IDs once per LogicalPlan. Before, every request assigned fresh IDs to every relation, so the same DataFrame never produced the same relation twice.

  • PlanIds, one per session, maps each shared Arc<LogicalPlan> node to its ID. Each entry holds a Weak reference to the node, so its address cannot be reused while the entry exists, and entries of dropped nodes are pruned as the map grows.
  • Serialization assigns the IDs in the same pass (relation_with_ids), replacing the post-order proto walker (assign_plan_ids). Checkpoint, writeStream, and ML transform relations, which had no plan IDs, now carry them too, as in PySpark.
  • A node that occurs more than once in a plan, such as both sides of a self-join, has one ID wherever it occurs, as in PySpark. SPARK-60549 had kept one ID per occurrence to leave the wire format unchanged; its test now asserts the PySpark behavior.

Why are the changes needed?

Spark Connect 4.2.0 and 4.3.0 servers cache resolved plans per session (SessionHolder.usePlanCache), keyed by the whole relation and only for relations with a plan ID. ExecutePlan and AnalyzePlan resolve their root with cachePlan = true, and every relation in a plan is looked up. With fresh IDs on every request, the Rust client never hit this cache, so each re-execution resolved its plan again. For cached Arrow input, resolving includes reading and decoding the cached chunks.

Server plan-cache hits ("Using cached plan for relation" at DEBUG) for the same steps on a Spark 4.3.0 server:

Step Official PySpark 4.3.0 master This PR
query.collect() 1 0 0
query.collect() again (same DataFrame) 1 0 1
df.columns 0 0 0
df.filter(...).collect() (a new query over df) 1 0 1

PySpark's first collect() also hits because it analyzes the plan before executing it.

Does this PR introduce any user-facing change?

Yes. Re-executing a DataFrame, or running a new query over a DataFrame the server already resolved (for example after df.columns), can now reuse the server's resolved plan, and results match PySpark's. A reused resolution can differ from a fresh one. A DataFrame from spark.read.parquet(path) keeps its file listing, so running it again after the files under path are overwritten fails with FAILED_READ_FILE.FILE_NOT_EXIST, as with PySpark; on master it read the new files. Read.DataSource relations are cached even when a request does not cache its plan (spark.connect.session.planCache.alwaysCacheDataSourceReadsEnabled). A temporary view is resolved again on each run, so a spark.table("v") DataFrame run again after CREATE OR REPLACE TEMP VIEW v reads the new view, here as with PySpark. Setting spark.connect.session.planCache.enabled to false, on the server or for a session with spark.conf.set, turns the cache off. Requests now repeat plan IDs across requests, and within a plan for shared nodes, as PySpark's do.

How was this patch tested?

  • New unit tests: re-serializing a plan gives an identical relation; a derived DataFrame keeps its input's relation and IDs; separately built nodes get distinct IDs, even for equal plans; a self-join repeats the shared side's IDs; while entries of dropped nodes are pruned, a new node never gets a dropped node's ID and live nodes keep theirs; sessions number their plans independently.
  • Against a local Spark 4.3.0 server, after merging master with [SPARK-60675][CONNECT] Support PyArrow Table input through the C Stream interface #118: the lib and core unit tests and every integration target with the server-gated e2e suites (631 passed, 12 ignored as on master); the offline Python suites (420 passed, 8 live cases skipped) and the live drop-in, Arrow, and live-case suites (21 and 9 passed). All integration targets also compile with wasm-udf.
  • The cache-hit table above, from the server's DEBUG log.
  • Re-running a temporary-view DataFrame and a parquet DataFrame after their sources change, against official PySpark 4.3.0 and master: this PR matches PySpark in both, and spark.connect.session.planCache.enabled=false on the session restores master's re-read of the parquet files.

Warmed aggregation benchmark. The sums of eight int64 columns over a PyArrow table, against one local Spark 4.3.0 server (8 GiB driver heap; the 3 GiB heap of the first attempt ran out during the cached-input runs) on an Apple M3 Max, with x86_64 Python 3.11 and PyArrow 22 for every client. Two patterns, each in a fresh session with a freshly created DataFrame: same re-executes one df.agg(...) DataFrame; derived calls df.columns, then builds a new df.agg(...) query each time. Each round runs one untimed and three timed queries per pattern and keeps their median. Cells are the median [minimum–maximum] in ms of five rounds, each a fresh client process, in alternating client order, after one discarded warm-up round; every query checked all eight sums.

64 MB, sent inline (master's Arrow input path):

Client same derived
Official PySpark 4.3.0 1533 [1099–1648] 1435 [1101–1626]
master 3049 [2479–3278] 2684 [2476–2836]
This PR 2195 [1922–2424] 2227 [2031–2593]

136 MB, uploaded as cache artifacts (#118's path; this PR merged with it):

Client same derived
Official PySpark 4.3.0 2370 [2308–2424] 2425 [2377–3078]
#118 alone 3495 [3177–3848] 3525 [3135–4306]
#118 with this PR 2644 [2292–3054] 2421 [2365–3122]

With this PR, the 136 MB cached case that #118 measured as about 49% slower than PySpark is at parity for derived queries and about 12% slower for same (overlapping ranges). The inline 64 MB case improves by about 17–28% but stays slower than PySpark. Both clients resend the 64 MB relation with every execution; where the remaining time goes was not isolated here.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

@HyukjinKwon
HyukjinKwon requested a review from viirya November 5, 2026 01:02

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assigning each node its ID once, keyed by the Arc allocation and kept alive by the Weak, looks correct, and it lets the server's plan cache actually hit. It also replaces the hand-maintained assign_plan_ids walker, so new relation types can't silently miss their IDs anymore.

One correction to the description: "Results are unchanged" is too strong. Once the cache hits, the server reuses the earlier resolution. Read.DataSource is also cached when cachePlan = false (alwaysCacheDataSourceReadsEnabled). So re-running spark.read.parquet(path) after the files under path are overwritten, or spark.table("v") after CREATE OR REPLACE TEMP VIEW v, can now return different results than master. PySpark behaves the same way, so this is expected; I'd just say "results match PySpark" instead.

Comment thread crates/spark-connect/src/plan.rs Outdated
/// Entries of dropped nodes are pruned as the map grows.
#[derive(Debug)]
pub(crate) struct PlanIds {
next: AtomicI64,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

next is only incremented while nodes is locked, so it could be a plain i64 inside PlanIdNodes. Then one lock clearly guards all the state.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3c286dd: next is a plain i64 in PlanIdNodes, so the one lock guards all the state.

let spark = session();
let kept = spark.range(1).unwrap();
let kept_ids = plan_ids_of(&kept.relation());
for end in 0..5000 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop is where addresses actually get reused. The nodes are dropped right away, and after a prune their allocations are freed. That makes it the natural place to test the key invariant: a new node never inherits a dropped node's ID. Could we assert that each relation() here gets an ID we haven't seen before, e.g. assert!(seen.insert(id))? Today only len() is checked. The <= 2048 bound could also be <= 1024, which is what the prune policy guarantees.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3c286dd. The loop asserts seen.insert(id) for every serialized node, and checks <= 1024 on every iteration. It also runs in a session of its own, so plans from other tests cannot change the entries.

Comment thread crates/spark-connect/src/dataframe.rs Outdated
assert!(spark.plan_ids().len() <= 2048, "{}", spark.plan_ids().len());
assert_eq!(plan_ids_of(&kept.relation()), kept_ids);
// A new session numbers its own plans.
assert_eq!(spark.new_session().plan_ids().len(), 0);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only shows that a freshly built PlanIds is empty. If the intent is "a new session numbers its own plans", it would be better to serialize a plan in each session and compare the IDs. Otherwise I'd drop this assertion.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced in 3c286dd by a_new_session_numbers_its_own_plans: two new sessions each give their first plan the ID 1, and plans in one do not advance the numbering of the other.

Comment thread crates/spark-connect/src/dataframe.rs Outdated
@@ -993,8 +996,7 @@ impl DataFrame {
/// Build this DataFrame's plan as a proto `Relation` with plan ids assigned
/// (the shape the `Persist`/`Unpersist`/`GetStorageLevel` analyze ops take).
fn analyze_relation(&self) -> Result<proto::Relation> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that serialization can't fail, analyze_relation is the same as self.relation() and could be removed. build_input_relation (L1856) could return proto::Relation directly. ml.rs:307 could then use df.relation(), like build_ml_relation does a few lines above.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3c286dd: analyze_relation is gone, build_input_relation returns the relation directly, and evaluate_via_command uses df.relation().

Comment thread docs/architecture.md
Each plan node gets its plan ID once, the first time it is sent, and keeps it, as
in the reference client. Re-executing a DataFrame therefore sends the same
relation, and a DataFrame built on another keeps that input's plan IDs, so the
server's per-session plan cache (`spark.connect.session.planCache.*`) can reuse

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add one sentence on what changes for users: a re-executed DataFrame can reuse an earlier resolution (as in PySpark), and spark.connect.session.planCache.enabled=false turns this off on the server.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3c286dd. I checked the examples against PySpark 4.3.0 first. A spark.read.parquet(path) DataFrame run again after the files are overwritten fails with FAILED_READ_FILE.FILE_NOT_EXIST, with PySpark and with this PR, while master read the new files. A spark.table("v") DataFrame, though, reads the new view after CREATE OR REPLACE TEMP VIEW v in all three, because the view is resolved again. So the docs use the parquet example. They also say the cache can be turned off on the server, or for a session with spark.conf.set, which restores master's re-read.

@HyukjinKwon

Copy link
Copy Markdown
Member Author

Thanks for the review. I changed "Results are unchanged" in the description to "results match PySpark's", and spelled out the parquet case: a re-run spark.read.parquet(path) DataFrame keeps its file listing, as with PySpark. The temporary-view example does not reproduce, in PySpark 4.3.0 or here: the view is resolved again on each run, so spark.table("v") reads the new view after CREATE OR REPLACE TEMP VIEW v. The inline comments are addressed in 3c286dd.

…euse the server plan cache

As PySpark assigns a plan ID once per plan, give each plan node its ID
the first time it is sent, from a registry of the session keyed by the
node's allocation, and keep it, so that re-executing a DataFrame sends
the same relation and a DataFrame built on another keeps that input's
plan IDs. The server's per-session plan cache can then reuse the plans
it already resolved. The registry holds weak references and prunes the
nodes that were dropped as it grows.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-60898-stable-plan-ids branch from 3c286dd to 51fedc3 Compare November 5, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants