Repository navigation
feat: add zero-wait distributed runtime filtering to AQE - #2749
Conversation
ff642c2 to
957f55a
Compare
|
@andygrove The TPC-H benchmark shows ~10% improvement for the Q19. |
andygrove
left a comment
There was a problem hiding this comment.
Disclaimer: this review was generated by an LLM (Claude Code) at my request, against 957f55ae6. File/line references were checked against the code and the planner behaviour below was reproduced with scratch planner tests, but please treat the conclusions as suggestions and push back where it's wrong.
Thanks for this, and for gating it behind a default-off flag and posting TPC-H numbers. I couldn't find a wrong-result path:
- Filtering either input is safe for inner and semi joins.
- NULL-safe equality keeps NULL keys only when the build saw one.
- Filters never go below a fetch or a non-value-preserving operator.
- A payload is bound only when every successful producer task reported.
- The fixed-seed foldhash state hashes the same on every executor.
It also merges cleanly with current main (after #2682 and #2671), and the scheduler lib tests pass on the merged tree.
My main question is where the cross-stage path fires today.
1. The partitioned fallback never gets the filter. With stage_build_side on (the default), a build side that measures too large leaves a pending probe exchange that is created after the build has finished. That is the one case where a complete filter can cut the probe's shuffle with no waiting. But discover_pair returns as soon as the build exchange is shuffle_created() (runtime_filter_planner.rs:113), so the probe stage is planned without an apply and the build tap's work is wasted. Reproduced with MemTable scans, a filter on both sides and broadcast_join_threshold_bytes = 500:
pass 1, stage 0:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
RuntimeFilterBuildExec: id=0, key=id@0
FilterExec: val@1 > 0
DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]
build measured too large, pass 2, stage 1:
SortShuffleWriterExec: partitioning=Hash([id@0], 4)
FilterExec: val@1 > 0, projection=[id@0]
DataSourceExec: partitions=4, partition_sizes=[1, 1, 1, 1]
When the build already has a registered filter, binding a not-yet-planned probe exchange to it looks safe, and prepare would find the summary complete at first dispatch.
2. Sibling stages mostly bypass. When both sides are shuffled at once, the apply is planned in the probe stage. It only binds if the build finishes before the probe's first dispatch, which is unusual when slots are free. prepare also freezes the plan when bind_tasks_with merely fetches the stage (cluster/mod.rs:754), even if no task is bound in that round. Until the waiting follow-up lands, would it be worth not tapping builds whose only consumer is a sibling probe?
So today the path that reliably fires is a broadcast join whose probe scan sits in the same task as the build reader (Q19). It would help to say that in the docs section.
3. Bloom sizing. With 8 KiB and k=4, the false-positive rate is about:
- 0.4% at Q19's 4,754 keys
- 4% at 10k
- 50% at ~30k
- 99% at 100k
RuntimeFilterApplyExec only skips at full saturation (~191k distinct keys), so in between every probe row is hashed for little pruning. prepare already holds the merged Bloom, so a cheap guard is to bypass when its fill ratio implies a high false-positive rate. The MinMax bounds aren't used for row filtering either; a range check is cheap and stays selective after the Bloom saturates.
4. Tests. runtime_filters_preserve_adaptive_planning (stage_build_side.rs:171) can't fail. Its StatsExec tables aren't DataSourceExec, so no filter is ever placed; I checked, and there is no tap or apply in either pass. assert_plan! snapshots of the instrumented stage plans over MemTable scans would pin the plan shape the way the other AQE rules' tests do. Cover the broadcast, fallback and sibling flows. That would also cover the EXPLAIN diff I asked for on #2701. Plain EXPLAIN renders the static planner's stages, so these operators only appear under EXPLAIN ANALYZE; an excerpt for Q19 in the PR body would help.
5. Benchmark details. Could you add the cluster setup, the settings used for A and B (AQE, prefer_hash_join, stage_build_side), and the run count? With three runs per side the ±3% rows are hard to read; Q19 looks real. The numbers predate #2671, whose distinct counts feed the join estimates, so a re-run on main would be good. TPC-DS star joins would exercise this more than TPC-H.
Smaller things:
- Each successful build task keeps an 8–9 KB payload per filter in
TaskInfofor the life of the job, so 10k tasks is about 80 MB per filter. Worth a line in the docs, as #2671 did for HLL. instrumentunwraps the boundary's stage id (runtime_filter_planner.rs:182-188). An internal error would be safer in the scheduler.- A consumer stage that re-runs after a rollback keeps the filter it first bound, because
preparedis never cleared. That's fine for deterministic builds; worth a comment, or rebinding on a new attempt. - configs.md asks users to set
prefer_hash_join = true, butDynamicJoinSelectionExec::to_actual_joinsays AQE deliberately ignores it. Is it needed?
|
Thanks for the review. To clarify, each query has 3 measured runs per arm, thus the total measure number is 3*3 = 9. I’ve updated the PR body to make this explicit. Will evaluate in this PR
I previously removed a broader speculative multi-join producer guard after SF10 experiments did not establish a clear benefit. The narrower sibling-only case still needs evaluation. Already deferred to personal PRs |
TPC-DS SF10 candidate evaluation
D1–D4 queries and query optionsD1 — selective build, partitioned fallback -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 4096;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;D2 — selective build, sibling stages -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month = 1 AND c.c_birth_day = 1
AND s.ss_quantity BETWEEN 10 AND 90;D3 — near-saturated Bloom -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.broadcast_join_threshold_bytes = 1500000;
SET ballista.optimizer.stage_build_side_max_estimate_multiple = 1024;
SET datafusion.optimizer.default_filter_selectivity = 100;
SET datafusion.optimizer.hash_join_single_partition_threshold = 0;
SET datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE CAST(c.c_birth_month AS VARCHAR) IN ('1', '2')
AND s.ss_quantity BETWEEN 10 AND 90;D4 — broad build, sibling stages -- Same query options in A and B.
SET ballista.planner.runtime_filter.enabled = true;
SET datafusion.optimizer.prefer_hash_join = true;
SET ballista.optimizer.stage_build_side = false;
SET ballista.optimizer.broadcast_join_threshold_bytes = 0;
SET ballista.optimizer.broadcast_join_threshold_rows = 0;
SELECT SUM(s.ss_ext_sales_price * c.c_birth_day) AS revenue
FROM customer c JOIN store_sales s ON c.c_customer_sk = s.ss_customer_sk
WHERE c.c_birth_month <= 11
AND s.ss_quantity BETWEEN 10 AND 90;Per-visit results — all timed standard and directed queries
|
957f55a to
65162ce
Compare
|
@andygrove, could you take another look? The A/B experiments showed benefits from reusing filters for partitioned fallback and bypassing Bloom checks at ≥99% occupancy in their targeted cases, and I've added them in two separate commits. I've also rerun the benchmarks against the upstream base and updated the PR description with the results. |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @wirybeaver, this is a big step forward. Reusing the completed build for the partitioned fallback is exactly the case I hoped would light up, and the D1 result shows it. Thanks also for the much more thorough benchmark write-up. The paired AB/BA rounds and the correctness checks against DataFusion make the numbers easy to trust. Dropping the sibling-only skip after measuring it makes sense to me.
A few things from the last round still look open:
runtime_filters_preserve_adaptive_planningstill can't fail, see inline.instrumentstill unwraps the boundary's stage id, see inline.- Could the docs section mention that each successful build task keeps an ~8–9 KB payload per filter in
TaskInfofor the life of the job? - Could it also say which joins benefit today? That's broadcast joins like Q19 plus the partitioned fallback.
- The docs ask users to set
prefer_hash_join = trueandenable_dynamic_filter_pushdown = false, but I couldn't find anything that checks either one. Are they required? If so, what happens with the defaults? If not, could we drop them so the setup is just the one flag? - A short comment on
preparednoting that a re-run consumer keeps the filter it first bound would help the next reader.
On the Bloom bypass, 99% occupancy is roughly 96% false positives, so there's still a band where we hash every probe row for little pruning. I'm happy for that to be part of the MinMax and sizing work in Task 3.
|
Thanks, @andygrove. The docs now show a single enable flag and describe the supported join paths and per-task memory cost. I also added a comment above |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @wirybeaver, the earlier points all look addressed. The new end-to-end test is a nice addition and it already caught something, see inline. CI is red because of it, but the fix looks small. One smaller question on pop_next_task is inline too.
| let ExecutionStage::Running(stage) = self.stages.get_mut(&id).unwrap() else { | ||
| unreachable!() | ||
| }; | ||
| stage.plan = prepared; |
There was a problem hiding this comment.
Thanks for the new end-to-end test, it caught a real interaction with #2875 that landed on main. That change caches RunningStage::plan_metrics_len when the stage is created. Here we swap in the prepared plan afterwards, and when a filter is bypassed that plan has one fewer operator. Every task's metrics then get rejected as a shape mismatch, the stage succeeds with no metrics, and EXPLAIN ANALYZE fails with Missing metrics for operator SortShuffleWriterExec. It passes on the branch alone and fails once merged with main.
Could you rebase and either refresh plan_metrics_len when the plan is replaced, ideally through a small RunningStage::replace_plan so the two can't drift again, or leave the unbound RuntimeFilterApplyExec in place as a pass-through so the stage keeps its shape? I tried the first option locally and the test passes.
There was a problem hiding this comment.
Done. RunningStage::replace_plan now refreshes the cached metrics count when the plan changes.
| false | ||
| } | ||
| }).map(|(stage_id, stage)| { | ||
| let ready_id = self.get_running_stage_id(&[]); |
There was a problem hiding this comment.
Small question here. Switching pop_next_task to get_running_stage_id means it now also calls revive(), for every AQE job whether or not runtime filters are on. Is that intended? If it's only there so prepare runs, could it be limited to when runtime_filters is Some, or called out in the description?
There was a problem hiding this comment.
Done. pop_next_task keeps its original stage revival behavior.
andygrove
left a comment
There was a problem hiding this comment.
Thanks @wirybeaver, this is looking close. The two inline comments from my last pass are still open. The plan_metrics_len one is what's turning CI red, since distributed_runtime_filter_matches_unfiltered_joins fails on all three test jobs with Missing metrics for operator SortShuffleWriterExec. The pop_next_task question is still open too.
Could you also switch the description over to the PR template sections? In particular, "Are there any user-facing changes?" should mention the new ballista.planner.runtime_filter.enabled setting.
Once CI is green I think this is ready.
| let Some(filter) = self.filter.clone() else { | ||
| return Ok(input); | ||
| }; | ||
| // Preserve the payload's bounds for scan pruning, but avoid hashing |
There was a problem hiding this comment.
Small one. The comment here says the bounds are kept for scan pruning, but I don't think anything reads them yet. Could we reword it until Task 3 lands? Also, since prepare already has the merged filter, would it be simpler to make the near-saturation decision there once? Then a bypassed filter doesn't ship its payload to every task or recount the bits per partition.
There was a problem hiding this comment.
Done. The scheduler now decides whether to bypass the merged Bloom before dispatch. I removed the outdated scan-pruning comment.
Attach taps to executable stage copies without changing AQE planning or stage selection. At first dispatch, freeze only summaries with complete current-attempt coverage and bypass unavailable producers. Keep saturated Bloom hashing bounded while continuing NULL and MinMax collection.
8b4ca49 to
01760d8
Compare
|
Thanks, @andygrove. I rebased on |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @wirybeaver, everything from the last round looks addressed. replace_plan keeping the plan and the metrics count together is a nice way to stop them drifting. Moving the saturation check into completed_filter means a skipped filter never ships to tasks, which is great. CI is green on current main.
Looks good to me. Thanks for sticking with this through all the rounds.
|
I ran some independent benchmarks on a 32-core / 124 GB Linux box at SF100, which is a step up from the SF10 numbers in the description. Setup
Summary
D1 is a great result. It went from 2.9s to 0.64s, and it was consistent in every round. With the flag off, the PR is neutral against Where it fires at SF100 I ran For those 15 queries, pruning lots of rows mostly didn't turn into wall-clock wins.
Q37 and Q82 are the clearest case. Both on runs were slower than all four other runs. In Q82's stage 2, the apply sits on the Meanwhile, the big win in those queries is left on the table. The A couple of thoughts, all fine as follow-ups since this is default off.
TPC-H SF100, all queries
TPC-DS SF100, all queries
|
|
Thanks for the thorough benchmarking on SF100 size. I admired that you have a 32-core / 124 GB Linux box :) I will add a Task 5 for the optimization point of "In the same-stage broadcast case, would it make sense to only place the apply when something costly sits between the scan and the join" |
Which issue does this PR close?
Part of #2700 (Task 2), building on the runtime filter operators and task reports.
Rationale for this change
Use completed build-side summaries to reduce probe-side row processing without adding a runtime-filter wait. Existing broadcast dependencies, as in Q19, can make summaries available before probe execution. Optional nonzero waiting is scoped to Task 4 (personal PR #13).
What changes are included in this PR?
Attach build/apply operators only to executable stage copies after AQE chooses stage boundaries. A job-scoped hub merges reports from complete successful producer coverage through the existing task-status lifecycle.
At stage preparation, bind complete filters already available or bypass them, then freeze the stage plan. Probe stages proceed immediately, so independently runnable sibling probes bypass filtering when the build report is still pending.
The feature is disabled by default and covers the first supported direct-column equality key of inner and semi hash joins, including NULL-safe equality. Fully saturated Blooms bypass hashing while NULL and MinMax collection continue.
The distributed runtime filter test compares filtered and unfiltered join results and checks
EXPLAIN ANALYZEfor broadcast and independently runnable partitioned stages.Are there any user-facing changes?
Yes. Set
ballista.planner.runtime_filter.enabled = trueto enable zero-wait runtime filtering for eligible AQE hash joins. The setting defaults tofalse.Benchmarking Result
Before the latest rebase and review fixes, A (upstream base) and B (PR with runtime filtering enabled) were compared on 21 TPC-H SF10 shapes (excluding Q16), 13 TPC-DS SF10 shapes, and two custom shapes using TPC-DS data. On one host with two 4-core/4-GiB executors per arm, each query ran five paired rounds (AB/BA/AB/BA/AB) after preflight and one suite warmup; each timed visit followed one short warmup, with servers kept running. All 72 correctness checks matched DataFusion and all 360 timed visits completed. Most latencies stayed close to base, with pronounced reductions for TPC-H Q19 (−8.93%) and custom D1 (−56.80%); Q3/Q6 measured +3.30%, whose cause remains unconfirmed.
Per-visit results — upstream base vs PR HEAD
Custom D1 — partitioned fallback
Custom D3 — near-saturated Bloom
Non-default query options (main A/B benchmark)
datafusion.optimizer.prefer_hash_jointruetrueballista.planner.runtime_filter.enabledtrueballista.optimizer.broadcast_join_threshold_bytes40964096ballista.optimizer.broadcast_join_threshold_bytes15000001500000ballista.optimizer.stage_build_side_max_estimate_multiple10241024datafusion.optimizer.default_filter_selectivity100100datafusion.optimizer.hash_join_single_partition_threshold00datafusion.optimizer.hash_join_single_partition_threshold_rows00Bloom bypass threshold: 100% vs 99%