Repository navigation
[SPARK-60898][CONNECT] Keep plan IDs stable across re-executions to reuse the server plan cache - #147
HyukjinKwon wants to merge 1 commit into
Conversation
viirya
left a comment
There was a problem hiding this comment.
Assigning each node its ID once, keyed by the Arc allocation and kept alive by the Weak, looks correct, and it lets the server's plan cache actually hit. It also replaces the hand-maintained assign_plan_ids walker, so new relation types can't silently miss their IDs anymore.
One correction to the description: "Results are unchanged" is too strong. Once the cache hits, the server reuses the earlier resolution. Read.DataSource is also cached when cachePlan = false (alwaysCacheDataSourceReadsEnabled). So re-running spark.read.parquet(path) after the files under path are overwritten, or spark.table("v") after CREATE OR REPLACE TEMP VIEW v, can now return different results than master. PySpark behaves the same way, so this is expected; I'd just say "results match PySpark" instead.
| /// Entries of dropped nodes are pruned as the map grows. | ||
| #[derive(Debug)] | ||
| pub(crate) struct PlanIds { | ||
| next: AtomicI64, |
There was a problem hiding this comment.
next is only incremented while nodes is locked, so it could be a plain i64 inside PlanIdNodes. Then one lock clearly guards all the state.
There was a problem hiding this comment.
Done in 3c286dd: next is a plain i64 in PlanIdNodes, so the one lock guards all the state.
| let spark = session(); | ||
| let kept = spark.range(1).unwrap(); | ||
| let kept_ids = plan_ids_of(&kept.relation()); | ||
| for end in 0..5000 { |
There was a problem hiding this comment.
This loop is where addresses actually get reused. The nodes are dropped right away, and after a prune their allocations are freed. That makes it the natural place to test the key invariant: a new node never inherits a dropped node's ID. Could we assert that each relation() here gets an ID we haven't seen before, e.g. assert!(seen.insert(id))? Today only len() is checked. The <= 2048 bound could also be <= 1024, which is what the prune policy guarantees.
There was a problem hiding this comment.
Done in 3c286dd. The loop asserts seen.insert(id) for every serialized node, and checks <= 1024 on every iteration. It also runs in a session of its own, so plans from other tests cannot change the entries.
| assert!(spark.plan_ids().len() <= 2048, "{}", spark.plan_ids().len()); | ||
| assert_eq!(plan_ids_of(&kept.relation()), kept_ids); | ||
| // A new session numbers its own plans. | ||
| assert_eq!(spark.new_session().plan_ids().len(), 0); |
There was a problem hiding this comment.
This only shows that a freshly built PlanIds is empty. If the intent is "a new session numbers its own plans", it would be better to serialize a plan in each session and compare the IDs. Otherwise I'd drop this assertion.
There was a problem hiding this comment.
Replaced in 3c286dd by a_new_session_numbers_its_own_plans: two new sessions each give their first plan the ID 1, and plans in one do not advance the numbering of the other.
| @@ -993,8 +996,7 @@ impl DataFrame { | |||
| /// Build this DataFrame's plan as a proto `Relation` with plan ids assigned | |||
| /// (the shape the `Persist`/`Unpersist`/`GetStorageLevel` analyze ops take). | |||
| fn analyze_relation(&self) -> Result<proto::Relation> { | |||
There was a problem hiding this comment.
Now that serialization can't fail, analyze_relation is the same as self.relation() and could be removed. build_input_relation (L1856) could return proto::Relation directly. ml.rs:307 could then use df.relation(), like build_ml_relation does a few lines above.
There was a problem hiding this comment.
Done in 3c286dd: analyze_relation is gone, build_input_relation returns the relation directly, and evaluate_via_command uses df.relation().
| Each plan node gets its plan ID once, the first time it is sent, and keeps it, as | ||
| in the reference client. Re-executing a DataFrame therefore sends the same | ||
| relation, and a DataFrame built on another keeps that input's plan IDs, so the | ||
| server's per-session plan cache (`spark.connect.session.planCache.*`) can reuse |
There was a problem hiding this comment.
Maybe add one sentence on what changes for users: a re-executed DataFrame can reuse an earlier resolution (as in PySpark), and spark.connect.session.planCache.enabled=false turns this off on the server.
There was a problem hiding this comment.
Done in 3c286dd. I checked the examples against PySpark 4.3.0 first. A spark.read.parquet(path) DataFrame run again after the files are overwritten fails with FAILED_READ_FILE.FILE_NOT_EXIST, with PySpark and with this PR, while master read the new files. A spark.table("v") DataFrame, though, reads the new view after CREATE OR REPLACE TEMP VIEW v in all three, because the view is resolved again. So the docs use the parquet example. They also say the cache can be turned off on the server, or for a session with spark.conf.set, which restores master's re-read.
|
Thanks for the review. I changed "Results are unchanged" in the description to "results match PySpark's", and spelled out the parquet case: a re-run |
…euse the server plan cache As PySpark assigns a plan ID once per plan, give each plan node its ID the first time it is sent, from a registry of the session keyed by the node's allocation, and keep it, so that re-executing a DataFrame sends the same relation and a DataFrame built on another keeps that input's plan IDs. The server's per-session plan cache can then reuse the plans it already resolved. The registry holds weak references and prunes the nodes that were dropped as it grows. Co-authored-by: Isaac <no-reply@databricks.com>
3c286dd to
51fedc3
Compare
What changes were proposed in this pull request?
SPARK-60898: the Rust client now gives each plan node its plan ID once, the first time it is serialized, and reuses that ID afterwards, as PySpark assigns plan IDs once per
LogicalPlan. Before, every request assigned fresh IDs to every relation, so the same DataFrame never produced the same relation twice.PlanIds, one per session, maps each sharedArc<LogicalPlan>node to its ID. Each entry holds aWeakreference to the node, so its address cannot be reused while the entry exists, and entries of dropped nodes are pruned as the map grows.relation_with_ids), replacing the post-order proto walker (assign_plan_ids). Checkpoint,writeStream, and ML transform relations, which had no plan IDs, now carry them too, as in PySpark.Why are the changes needed?
Spark Connect 4.2.0 and 4.3.0 servers cache resolved plans per session (
SessionHolder.usePlanCache), keyed by the whole relation and only for relations with a plan ID. ExecutePlan and AnalyzePlan resolve their root withcachePlan = true, and every relation in a plan is looked up. With fresh IDs on every request, the Rust client never hit this cache, so each re-execution resolved its plan again. For cached Arrow input, resolving includes reading and decoding the cached chunks.Server plan-cache hits ("Using cached plan for relation" at DEBUG) for the same steps on a Spark 4.3.0 server:
query.collect()query.collect()again (same DataFrame)df.columnsdf.filter(...).collect()(a new query overdf)PySpark's first
collect()also hits because it analyzes the plan before executing it.Does this PR introduce any user-facing change?
Yes. Re-executing a DataFrame, or running a new query over a DataFrame the server already resolved (for example after
df.columns), can now reuse the server's resolved plan, and results match PySpark's. A reused resolution can differ from a fresh one. A DataFrame fromspark.read.parquet(path)keeps its file listing, so running it again after the files underpathare overwritten fails withFAILED_READ_FILE.FILE_NOT_EXIST, as with PySpark; on master it read the new files.Read.DataSourcerelations are cached even when a request does not cache its plan (spark.connect.session.planCache.alwaysCacheDataSourceReadsEnabled). A temporary view is resolved again on each run, so aspark.table("v")DataFrame run again afterCREATE OR REPLACE TEMP VIEW vreads the new view, here as with PySpark. Settingspark.connect.session.planCache.enabledto false, on the server or for a session withspark.conf.set, turns the cache off. Requests now repeat plan IDs across requests, and within a plan for shared nodes, as PySpark's do.How was this patch tested?
wasm-udf.spark.connect.session.planCache.enabled=falseon the session restores master's re-read of the parquet files.Warmed aggregation benchmark. The sums of eight int64 columns over a PyArrow table, against one local Spark 4.3.0 server (8 GiB driver heap; the 3 GiB heap of the first attempt ran out during the cached-input runs) on an Apple M3 Max, with x86_64 Python 3.11 and PyArrow 22 for every client. Two patterns, each in a fresh session with a freshly created DataFrame: same re-executes one
df.agg(...)DataFrame; derived callsdf.columns, then builds a newdf.agg(...)query each time. Each round runs one untimed and three timed queries per pattern and keeps their median. Cells are the median [minimum–maximum] in ms of five rounds, each a fresh client process, in alternating client order, after one discarded warm-up round; every query checked all eight sums.64 MB, sent inline (master's Arrow input path):
136 MB, uploaded as cache artifacts (#118's path; this PR merged with it):
With this PR, the 136 MB cached case that #118 measured as about 49% slower than PySpark is at parity for derived queries and about 12% slower for same (overlapping ranges). The inline 64 MB case improves by about 17–28% but stays slower than PySpark. Both clients resend the 64 MB relation with every execution; where the remaining time goes was not isolated here.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.