perf(sql): prune total derived grouping keys by XuPeng-SH · Pull Request #31040 · matrixorigin/matrixone

XuPeng-SH · 2026-10-24T10:12:07Z

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

GROUP BY x,x-1,x-2,x-3 carries redundant exact integer keys through aggregation. Remove derived keys only when the existing range proof establishes a total function of explicitly retained, identically typed grouping columns, and the proposed retained inputs and complete aggregate list are within the shared fixed-input/grouped-only/ordinary COUNT cost domain. Reconstruct consumers from original bound trees, preserving resolved overloads and complete result types.

Run at the existing all-consumers-bound boundary, before aggregate layout is fixed. Reuse source substitution, registry compaction and the atomic bound-slot remapper shared with affine SUM rewriting. Preflight all consumers; unsupported arithmetic, diagnostics, provenance, query references, stale types/slots, grouping extensions and excessive work retain the existing plan. Generic Group/MergeGroup, memory admission and spill remain the runtime owners.

Keep aggregate distribution at its existing owner. Its bounded pressure estimate recognizes known fixed group/COUNT state that is expected to spill per partial owner but fit with complete partition ownership. Reuse runtime hash capacity estimates, DOP and spill-threshold resolution, with topology, selected-NDV and raw-transport guards. Share this same eligibility/payload calculation with pruning admission; do not copy executor state layouts or infer state from result width.

The two review regressions are addressed together: COUNT retains the common-owner pressure correction; SUM, AVG, MIN/MAX, DISTINCT, variable-state/mixed aggregates, unsupported retained keys and wide/computed aggregate inputs now skip pruning atomically. Their original keys and distribution decisions remain. The handwritten single-key SUM retains its pre-existing policy and can still spill. This deliberately narrows optimization coverage instead of adding aggregate-specific estimators or forcing shuffle.

The versioned design, including the cost-domain amendments, was reviewed by gpt-6.1-sol / xhigh before implementation. The same selected model completed the final overall review with PASS / no unresolved material findings on the exact delivered source fingerprint. Delivered head: 4be001a3fac34a80bdc8b3ed454feee7e3025796.

Validation on rebased main 501a569224d2586effd7460df5bf94790128b8cf, Go 1.27.1 / macOS arm64, matching verified native libraries:

Full planner normal suite: 4,157 root tests pass. New/direct pruning, affine and shuffle tests: 14 roots / 138 nodes pass once with race. Formatting, vet and configured incremental golangci pass (0 issues). Molint retains five advisories at unchanged sites. New atomic tests check consumers, aggregate lists, keys, flags and all registries; an exact old-pruner overlay fails the intended SUM plan and atomic assertions.
Existing metadata-comparing BVT: 23/23 statements pass twice on one instance, including NULL/boundaries, diagnostics, prepared reuse and teardown. Fresh base/fix/fix/base public comparisons match 22 records of rows, metadata, errors/warnings and prepared execution. Invalid integer CAST follows rebased main's zero + warning 1292 contract; invalid DECIMAL input still errors.
Moderate-cardinality regression: one million INT rows / 100,000 groups, DOP4, 4 MiB. In A/B/B/A blocks, each with two warmups and four measured samples, the derived SUM query has base/fixed medians 18.10/17.55 ms. Both retain four keys, range shuffle and zero spill, with about 13.4 MiB reported aggregate memory. The prior head independently reproduced 10.90 MiB spill locally; its older physical dataset is not used to claim a matched speedup ratio.
Derived COUNT at 4 MiB has base/fixed medians 14.66/6.17 ms; fixed derived COUNT, single-key COUNT and grouped-only use complete ownership without spill. Each block also checks all 100,000 SUM result groups against an independent formula.
Nineteen public performance/control shapes cover auto/16 MiB budgets, DOP1/2/8, single SUM/COUNT, mixed aggregates, wide and computed-wide inputs, low NDV, selective/skewed data and actual forced spill. Raw samples are retained; overlapping control ranges do not support attributing small timing differences. Single-key SUM still spills at 4 MiB, as on base. At moderate cardinality with auto/16 MiB resident budgets, COUNT still uses partial/merge ownership: aggregate memory can rise from about 13.45 MiB to 26.16 MiB while output shrinks from 3.43 MiB to 1.14 MiB and timing ranges overlap. That existing resident-policy tradeoff is not fixed here. No universal latency or memory reduction is claimed.
Historical 10M/50M INT+COUNT evidence below remains applicable only to its unchanged high-NDV pruning/ordinary-shuffle path; those timings were not rerun. Aggregate memory figures are operator diagnostics, not process RSS.

Controlled synthetic Q36 input, unchanged SQL, four workers, same persisted data and build settings; A/B/B/A blocks each use two warmups and four measured runs (eight measured samples per variant):

Input	Baseline median	Candidate median	Wall reduction
10 million rows	0.250s	0.142s	43.3%
50 million rows	1.542s	0.904s	41.4%

Every timed result is checked. The 50M plan reduces four grouping keys to one; reported aggregate memory falls from 1.84 GiB to 799.34 MiB and aggregate output from 858.21 MiB to 286.07 MiB, with no spill. These are operator measurements, not process RSS. The original single-key, retained composite-key and low-cardinality resident controls have overlapping sample ranges; the additional moderate-cardinality budget controls above address distribution changes. This uses synthetic data, not the official ClickBench distribution, and does not establish closure of the remaining single-key gap to DuckDB.

Remote CI for the updated head is not yet verified. The prior head's latest UT job failed during Java TLS download, and coverage failed during the five-minute source checkout; neither reached test execution. The earlier, separate embedded-cluster admission incident is tracked in matrixorigin/CI#468; full owning-runner validation remains separate.

qodo-code-review · 2026-10-24T10:12:13Z

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

XuPeng-SH · 2026-10-24T10:14:27Z

mo-bot review · mo-bot-local · done

Commit: d27daa42a50c8fca50d348454b8f744f245d6cc6

Review completed. Read the assessment.

XuPeng-SH

mo-bot review · mo-bot-local · review

Commit: d27daa42a50c8fca50d348454b8f744f245d6cc6

Assessment: request_changes

The logical pruning proof and shared remapper are reasonable, but downstream aggregate planning introduces a confirmed memory and spilling regression.

Model: gpt-6-astra / xhigh

Session: 01a13dcf-1517-7f90-b3f9-c5f9f9e7031b · slot 0 · generation 1

P2 pkg/sql/plan/derived_group_by.go:181: Populate t(x INT) with SELECT result % 100000 FROM generate_series(1,1000000) g. Set max_dop=4 and agg_spill_mem=4194304, then execute SELECT x,x-1,x-2,x-3,count(*) FROM t GROUP BY x,x-1,x-2,x-3 ORDER BY x LIMIT 2.
Impact: Compacting the keys changes cardinality estimates and disables range shuffle. The resulting partial/merge plan duplicates group state across workers and spills where the base remains resident, increasing measured median latency approximately 6.9×. Update the existing distribution-cost owner to account for replicated state and spill costs, and retain this moderate-cardinality regression case.
Evidence: Base median: 19.1 ms, no spill. Candidate median: 131.2 ms, 930,512 spilled rows and 11.12 MiB written; control-query timings remained comparable. Retained memory-base.json, memory-candidate.json and corresponding logs document results and plans. Matching receipts: checks/1792837861743258849-4.json and checks/1792837913416343328-3.json.

Validation:

Reviewed the complete eight-file delta, design history, proof boundary, consumer remapping, aggregate layout, statistics and shuffle ownership.
All 4,155 planner test roots passed; two unrelated subcases skipped. Changed-file formatting and planner go vet passed.
Thirty base/candidate public records matched, including rows, metadata, errors, warnings and prepared reuse. Twelve configurations passed independent row oracles with actual DOP 1/4/8, resident/spilling execution, and COUNT/group-only shapes.
Resident A/B/B/A measurements showed aggregate memory increasing from 13.45–13.46 MiB to 26.16 MiB despite output shrinking from 3.43 MiB to 1.14 MiB. Resident timing ranges overlapped.
Eleven successful checks were reconciled against commands, source fingerprints, exit receipts and retained logs in /home/xupeng/.local/state/mo-bot/jobs/149878b7ec17ed71c9677298/attempt-1/evidence-audit.json. Initial qualification and reviewer-probe failures are separately documented. Source remained unchanged.

Limitations:

Execution used one Linux/amd64 CN. No independent multi-CN, race, configured-linter or full distributed BVT run; author 10M/50M artifacts were unavailable.
Memory figures are operator diagnostics, not process RSS. Base controls used exact merge-base production-file overlays, documented in the artifact audit.

XuPeng-SH · 2026-10-24T11:25:35Z

mo-bot review · mo-bot-local · done

Commit: 81d49e695790712d5960467edd15852280f221a1

Review completed. Read the assessment.

XuPeng-SH

mo-bot review · mo-bot-local · review

Commit: 81d49e695790712d5960467edd15852280f221a1

Assessment: request_changes

Reviewed the complete ten-file delta and affected owners. The bounded rewrite and shared remapper are reasonable, and the previous COUNT regression is fixed. However, pruning still introduces a confirmed SUM performance regression.

Model: gpt-6-astra / xhigh

Session: 01a13dcf-1586-7a01-9000-d45e20995cf8 · slot 1 · generation 1

P2 pkg/sql/plan/shuffle.go:1592: Populate t(x INT) with SELECT result % 100000 FROM generate_series(1,1000000) g. Set max_dop=4 and agg_spill_mem=4194304, then execute SELECT x,x-1,x-2,x-3,SUM(x) FROM t GROUP BY x,x-1,x-2,x-3 ORDER BY x LIMIT 2.
Impact: Pruning lowers the group estimate and removes the base plan's range shuffle. This COUNT-only fallback rejects SUM, leaving replicated partial aggregation and MergeGroup spilling instead of resident distributed aggregation. Median latency increases approximately 6.3×. Preserve an appropriate distribution decision or decline pruning when the supported cost policy cannot prevent this regression; add a SUM plan regression test.
Evidence: Matched public SQL runs show base median 23.90 ms versus head 151.36 ms: four base samples and eight head samples across two runs. EXPLAIN ANALYZE confirms DOP4, no base spill, and 606360–922744 head spill rows / 7.26–11.03 MiB. Results match. perf-A2.json, perf-B1.json and perf-B2.json correspond to successful receipts 1792841926724666134-2, 1792841856960656443-2 and 1792841895435064848-2. Exact base overlay bytes were independently verified.

Validation:

Preflight and native provenance passed; source remains unchanged. Full planner suite passed 4156 root tests; focused normal and race tests passed.
Thirty base/head public SQL records match, including results, metadata, prepared parameters, errors and warnings. Forced-spill behavior passed. Audited earlier same-PR evidence covers unchanged paths at actual DOP1/4/8 with complete result oracles.
Derived COUNT improves from 22.22 to 8.44 ms without spill. Handwritten grouped COUNT and grouped-only also improve. Wide COUNT controls retain their existing topology; small timing differences are not attributed to this change.
go vet completed successfully and changed Go files pass gofmt. The composite static check subsequently exited 2 for an extra blank line at the design document's EOF; it is not counted as a passing composite check.
Receipts, logs, fingerprints, overlays and hashes are reconciled in /home/xupeng/.local/state/mo-bot/jobs/6ad49edd8728003218c43803/attempt-1/evidence-audit.json. Detailed analysis and the saved assessment are in review-notes.md and assessment.json.

Limitations:

Frozen local_validation is empty and substantive CI was pending. Author summaries were not treated as verified execution evidence; golangci-lint was unavailable locally.
Fresh performance measurements use one embedded CN and one million rows. Broader 10M/50M and multi-CN measurements were not repeated. These limits are not additional findings; the demonstrated SUM regression warrants changes.

XuPeng-SH · 2026-10-24T12:16:33Z

mo-bot review · mo-bot-local · done

Commit: 4be001a3fac34a80bdc8b3ed454feee7e3025796

Review completed. Read the assessment.

XuPeng-SH

mo-bot review · mo-bot-local · review

Commit: 4be001a3fac34a80bdc8b3ed454feee7e3025796

Assessment: approve

Reviewed the complete ten-file delta and affected callers. The bounded expression proof, shared atomic remapper, and existing distribution/spill ownership form a coherent design. Current evidence closes the earlier COUNT and SUM regressions; no actionable findings remain.

Model: gpt-6-astra / xhigh

Session: 01a13dcf-1517-7f90-b3f9-c5f9f9e7031b · slot 0 · generation 1

Validation:

Passed 4,157 planner test roots and 19 focused roots under normal and race modes. Two unrelated subcases skipped. Vet, formatting, diff checks, and incremental golangci-lint passed.
Matched 31 base/head public SQL records covering rows, metadata, errors, warnings, and prepared reuse. Same-instance reruns passed; 12 additional scenarios verified actual DOP 1/4/8, complete 100,001-group results, and forced spill.
Balanced measurements: derived COUNT at 4 MiB improved from 19.27 to 9.15 ms without spill. Derived SUM retained four keys and resident range shuffle; timing ranges overlapped. Lower-budget probes also improved.
Verified native provenance, exact merge-base overlays, unchanged source, commands, exit receipts, and logs. Evidence index: /home/xupeng/.local/state/mo-bot/jobs/2f7d10ded67a68358e9ba8b7/attempt-1/evidence-audit.json.

Limitations:

Resident auto-budget aggregation retains the documented tradeoff: operator memory increased approximately 13.47 to 26.16 MiB while output decreased; timing ranges overlapped.
Validation used one Linux/amd64 CN. Full distributed BVT, multi-CN execution, and historical 10M/50M measurements were not independently reproduced. Frozen substantive CI remained pending.
An initial reviewer topology probe misparsed summarized eight-worker diagnostics; its failed receipt is retained separately from the corrected passing check.

Third base update, following the resolved-blocker contract for tracked CI blocker matrixorigin#31044 (open -> closed transition while tracked). Verified closure: NOT_PLANNED, closed 2026-10-24T21:37:23Z by XuPeng-SH, whose review of the complete retained coverage artifact confirmed the complete CHECK namespace and all four cross-target recovery scenarios pass on current main dae02bb (linux/amd64 race + matrixone_test, package 120.959s), explicitly covered this PR's run-40105576983 attempt-4 reproduction record, and re-classified it as package-budget boundary behavior; no pkg/embed change was merged - the six upstream commits touch no path under pkg/embed. The PR head f79bbb0 is terminally red only on the coverage lanes whose failure that issue tracked (UT Coverage producer + Coverage/CI Required projections), and run 40105576983's failed-jobs rerun budget is consumed, so the required checks need a fresh evaluation on a new head containing current main; no rerun of the stale run is issued. Clean auto-merge, zero conflicts (merge-tree dry run: no CONFLICT entries): - The only shared file pkg/sql/plan/query_builder.go changes at disjoint hunks: this feature's NameQuantileExactWeighted constant stays around line 5591 while upstream (matrixorigin#31040 derived-group-by pruning, matrixorigin#31048 composite secondary-index filters) reworks createQuery around line 4456 and the bindSelect aggregate-consumer region around line 6663. - Upstream allocated no SQL function ids, so QUANTILE_EXACT_WEIGHTED keeps catalog id 616 and FUNCTION_END_NUMBER 617 (first-unassigned-id rule as delivered in the second merge) and the D10 wire gate MORPCVersion131 is untouched; no manual conflict resolution and no design revision involved. - No upstream parser-grammar change: the generated mysql_sql.go delivered by the second merge is not rebuilt (its goyacc-verified bytes still carry both this feature's QUANTILE_EXACT_WEIGHTED rules and upstream's grammar). - The upstream range contains no .github/workflows or .github/ci content. Merge, not rebase: every commit hash this PR's evidence cites is preserved byte-for-byte - the approved design revision-5 content byte-preserved at introducing commit 60d88ff, review-remediation commits 0afd975/5b60194404db, prior merge f79bbb0, and the recorded runs 40062137563/40099808332/40105576983 - so those verifications remain traceable from the new head.

@XuPeng-SH

`quantileExactWeighted` has no entry anywhere in the SQL stack: the parser grammar carries no token or aggregate rule (so `quantileExactWeighted(...) OVER (...)` and `DISTINCT` forms fail to parse), the function registries carry no `quantileexactweighted` key (flat calls return (20105, "not supported: function or operator")), and the exact-percentile executors (median / approx_percentile / percentile_cont / percentile_disc) retain only unweighted value rows. Unweighted `PERCENTILE_CONT` computes a different statistic: on the issue's fixture it interpolates 3 where the weighted lower-CDF threshold selects 2. This PR adds the native exact weighted percentile `quantileExactWeighted(value, weight, level)` computing `min { v present : F(v) >= k }` with `k = ceil(level * W)` in exact big-integer arithmetic: - Grammar: a `QUANTILE_EXACT_WEIGHTED` aggregate rule with `DISTINCT` and window support, the keyword map entry, and `mysql_sql.go` regenerated through the repository's in-tree goyacc make target under its zero-conflict gate; a parser UT covers the plain, DISTINCT, window-partition and window-ROWS-frame forms, and the full parser corpus passes. - Registration/binding: function id 616 in both registries (re-issued from the pre-merge draft 606 when upstream #31017's indicator kernels allocated 605-615; the design's first-unassigned-id rule, recorded in the document's second post-merge annotation); typed weight admission (int widths, BIT, UINT64, FLOAT32/64, DECIMAL64/128) with DECIMAL256 rejected at bind; flat three-argument layout whose level parameter is constant-or-prepared configuration, re-evaluated per EXECUTE through the shared validated percentile codec; sliding time-window contexts rejected like the ordered-set percentiles. - Executor (new `pkg/sql/colexec/aggexec/quantile_exact_weighted.go`): weights validated in their own type before exact uint64 conversion with a total-overflow abort; the paired-NULL rule; `DISTINCT` deduplicates exact (value, weight) typed pairs; typed saved state with interim wire and spill round trips; finalization restores one group's typed value rows once while converting every retained weight exactly once in the same arena scan (per-group temporary footprint is the restored value rows plus the 8-byte-per-row sort selectors and converted uint64 collapse weights — (valueWidth + 16) × N plus the result/bitmap tail — with no second pair-payload copy), sorts through an interruptible in-package heap order whose comparator reproduces SortByVectors' SQL-order relation exactly, checks context at chunk entry, per-row entry, the restore scan, inside the sort at a bounded swap cadence and the cumulative scan at a bounded row cadence, frees partially built results only on the error path, transfers successful output ownership to the caller, and constructs retained-pair errors only inside failing branches so a valid finalization reports no errors. - Window: admitted beside the ordered-set percentiles for full-partition frames (ordered-set fast path) and sliding ROWS frames (generic per-frame path); the window binder carries `DISTINCT` through for this aggregate exactly like the group-aggregate binder records it, while every other DISTINCT window aggregate stays NYI. - Distributed: destination-verified remote dispatch gated on MORPC protocol 131 at encode time (requiredProtocol floor over pipelines containing the aggregate), send time (per-destination stream confirmation) and compile time (AllCNsSupportProtocol fast-fail plus the coordinator self-gate). ### Issue-to-test proof (#29944) - Red/green on the assigned base `e3a6ffe27c`: the identical self-contained BVT file `test/distributed/cases/function/func_aggr_weighted_quantile.test` (76 statements) fails on the clean base - the issue's own probe is rejected 3/3 with (20105, "not supported: function or operator 'quantileexactweighted'") and the compared mo-tester run passes only 47/76 with every quantileExactWeighted statement failing (20105 bind rejections for flat calls, SQL parser errors for window/DISTINCT forms) - and passes 76/76 at a 100% success rate on this branch, where the issue fixture returns 2 beside the 3.0 unweighted control. - Every generated BVT value was hand-verified against exact `k = ceil(level * W)` mathematics before adoption: GROUP BY partitions, full-partition and cumulative-ROWS windows, DISTINCT windows (distinct 2.0 vs plain 6.0 with the dedup row flipping inside both a partition and a bounded frame), ties, zero-weight endpoints (trailing zero weight at level 1, massless leading value at level 0, all-zero group), empty and all-skipped groups returning NULL, typed UINT64 weights above 2^53 accepted exactly (W=18014398509481985, k=9007199254740993 = F(1)), DECIMAL(20,0) and literal weights, prepared levels re-evaluated per EXECUTE (0.25 -> -1, 0.75 -> 4), and the documented rejections (level out of range, fractional and negative weights, DECIMAL(65,0) weight rejected at bind). ### Validation Executed locally on the final tree (macOS/arm64, native cgo artifacts built by `make cgo`), using `.agents/skills/mo-dev/scripts/mo-cgo-test` as the repository CGo wrapper: - `--count=1 -run 'TestQuantile|TestAccounted...AllocationFailure' ./pkg/sql/colexec/aggexec/` -> ok: the executor matrix (issue fixture, endpoints, 128-bit threshold arithmetic, typed-weight witnesses, overflow with allocated-memory accounting, merge equivalence, wire round trips, DISTINCT + spill, dispatch/config guards, the Flush -> executor-free -> repeated-read output-ownership witness), the rewritten large-group cancellation witness with deterministic flip points mid-restore, at sort entry, deep inside the sort's element work, mid cumulative scan and between groups (leak-free on every abort), the sort-interruption unit witnesses (pre-cancelled context stops the sort with zero element moves; mid-work abort after the bounded swap checkpoint), the ordering-parity witness against SortByVectors for every supported value type (NaN/±Inf/−0/+0/extremes/duplicates), the 8192-row peak-scratch witness (measured within (valueWidth + 16) × N and admitted under an enforced-budget account) and the two allocation-failure sweep cases green. - `--count=1 -run 'TestWindowQuantileExactWeighted' ./pkg/sql/colexec/window/` -> ok: full-partition broadcast, retained-partition-result survival after executor destruction across output chunks, and DISTINCT windows over a full partition and a bounded frame, plus the 40000-row large-partition cancellation regression (a context cancelled after Prepare makes vm.Exec return the standard cancellation error before any partition result is broadcast; the finalize's own mid-work abort points are pinned by the executor-level regression). - `--count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> ok: prepared-level preflight and reuse, config extraction keeping original argument types, the protocol-131 floor, the 129/130/131 dispatch matrix (including a worker advertising only upstream's temporal 130), the coordinator self-gate and the stream-confirmation seam. - `--count=1 -run 'TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters' ./pkg/sql/plan/` -> ok. - Full normal-mode owning packages: `./pkg/sql/colexec/aggexec/`, `./pkg/sql/colexec/window/`, `./pkg/sql/compile/`, `./pkg/sql/plan/`, `./pkg/defines/` all ok; the full parser corpus and `./pkg/sql/plan/function/` also pass. - BVT via the shared mo-tester against an isolated local service: genrs regenerated the checkpoint, every value inspected, compared run 76/76 with run exit 0 on this branch and 47/76 with run exit 1 on the clean base build. - `gofmt -l` empty over all changed Go files; `go vet` clean on all owning packages. ### Residual risks / follow-ups - Only `quantileExactWeighted` carries DISTINCT through windows; generalizing DISTINCT windows to other aggregates is out of scope here. - Sliding ROWS frames recompute per frame through the generic aggregate path; the full-partition fast path covers the common case. - The MORPC-131 mixed-version gates are proven at seam/UT level and by single-process SQL; a real multi-CN mixed-version cluster remains CI territory. - The BVT checkpoint embeds current diagnostic wording; unrelated wording drift would need one checkpoint regeneration, as with any BVT result. ### Design gate: recorded approval This feature triggers the mandatory design-first gate (new public SQL API, cross-package aggregate state, mixed-version MORPC dispatch; contract at `.agents/skills/mo-dev/references/feature-design-review.md`). Review order was honored end to end: a versioned design document was written first and reviewed in five design-only rounds while implementation stayed blocked — four REQUEST_CHANGES decisions (revisions 1-4; each returned blocker and its fix is recorded in the document's own revision log: typed weight admission, exact big.Int threshold domain, destination-verified MORPC protocol gating, typed saved-state schema, finalization resource and output ownership) — and then: - **Reviewed design**: [`docs/design/20261023-exact-weighted-percentiles.md`](https://github.com/gouhongshen/matrixone/blob/60d88ff6adb4ae6ea9e1ba0946ee82bc61f26ea7/docs/design/20261023-exact-weighted-percentiles.md), revision 5. The approved revision-5 content is SHA256 `b2705c13ab4d3f9965ab3d83a6f3e4a83d3426e4ffefdcdc8b6713864c69e083`, byte-preserved at commit `60d88ff6adb4ae6ea9e1ba0946ee82bc61f26ea7` (the commit that introduced the document), so the reviewed bytes remain verifiable against that hash. - **Reviewer**: independent design-review session, separate from the implementing author (issue-fix loop review session `01a16996-b2e8-76a1-aa31-f47fd5a0737d`), applying the repository's feature-design-review contract; design-only checkpoint, no repository state modified during the review. - **Decision**: PASS, 2026-10-24 — "Numerical semantics, typed saved state, destination protocol gates and corrected output ownership are coherent with the inspected code. No blocking design findings remain." One non-blocking P3 (exercise executor destruction before retained-result reads); the window UT covers it (flush -> executor teardown -> retained-result reads -> caller release). - **Document/PR reconciliation**: commit `0afd975232` records this decision inside the design document itself — its Status section now states the approval (decision, date, reviewer, reviewed revision and content hash) and the header links this PR, replacing the pre-review "pending" wording. Documentation-only annotation; no reviewed decision, invariant, or validation plan changed. - The separate implementation review reused this recorded design approval, verified the implementation against the design, and returned its own PASS. ### CI follow-up: race-mode UT failure fixed The `UT Test on Ubuntu/x86` job runs the unit tests with `-race`, where `pkg/container/vector`'s race build (`compile_config_race.go` -> `checkType`) enforces generic access typing. `TestQuantileExactWeightedEmptyAndSkippedGroups` appended the untyped literals `0`/`1` through `vector.AppendFixed` into `T_float64` vectors, binding the generic parameter to `int`: under `-race` the weight append panicked with `type mismatch: casting float64 vector to []int` (surfaced by the test's deferred allocation check, 0 expected vs 557056 actual), while a non-race build silently wrote int bit patterns into the float64 weights. The two literals are now the float64 literals `0.0`/`1.0`, so the generic binds to the vector's element type. A sweep of the other generic vector call sites this PR adds found no similar untyped-literal misuse. - Red/green at the pushed head `3fb5563e` with the repository wrapper: `.agents/skills/mo-dev/scripts/mo-cgo-test -race -count=1 -run '^TestQuantileExactWeightedEmptyAndSkippedGroups$' ./pkg/sql/colexec/aggexec` failed with the exact CI signature (`checkType` panic reached through `AppendFixed` at `quantile_exact_weighted_test.go:135`), and passes after the change. - Package-level race parity for the failed CI job on the fixed tree: `.agents/skills/mo-dev/scripts/mo-cgo-test -race -count=1 ./pkg/sql/colexec/aggexec` -> ok (31.8s), zero failing tests. - Regression and static checks: focused normal-mode `-run 'TestQuantileExactWeighted'` on the same package passes; `gofmt` and `go vet` on the package are clean. ### Review-fix round: finalization cancellation and memory conformance (D12) The requested-changes review (`mo-bot-local`, commit `0afd975`) returned two P2 findings on finalization — cancellation inside the sort/scan of a large group or window partition, and a temporary footprint beyond the approved resource model — and the delivered code has been conformed to the approved design revision 5 (no design decision changed, so no re-review was required). - **In-sort and in-scan cancellation (was `quantile_exact_weighted.go:371`)**: `SortByVectors` carried no context and the final cumulative scan had no checkpoint, so cancellation waited for the entire sort. Finalization now sorts through an executor-owned interruptible heap order (`quantileSortSelectors`) that checks the context at sort entry and after every 1024 elementary swaps, mirroring pkg/sort's own heapsort shape; the cumulative scan checks the same 1024-row cadence. The comparator reproduces `SortByVectors`' SQL-order relation per value type (integer/BIT width compares, `Float32/64OrderAscCompare`'s deterministic NaN/−0 total order, `Decimal64/128.Lt`), pinned by `TestQuantileSortSelectorsMatchesSortByVectors` over identity selectors for every supported value type including NaN/±Inf payloads, −0.0/+0.0, width extremes and duplicates. - **Single-scan restore (was `:475`)**: the finalize path no longer materializes a typed weight column at all. `restorePairs` walks the retained arena once per group, restoring only the typed value rows (the sort key and `UnionOne` output source — the same in-flight restore `orderedPercentileDiscreteExec` performs) while converting every retained weight exactly once into the converted-weight scratch and accumulating the total W with the overflow abort in the same pass. Per-group temporary footprint is now (valueWidth + 16) × N plus the result/bitmap tail — 24N for INT64/UINT64 (was 32N) and 32N for two DECIMAL128 columns (was 48N) — with no second pair-payload copy. - **Regressions**: `TestQuantileExactWeightedCancellation` was rewritten from three single-pair groups (the reviewer's stale-fixture concern) to a 4096-row group whose deterministic check sequence is learned from a counting-context warmup, pinning flips mid-restore, at the sort's entry, deep inside the sort's element work, mid cumulative scan, and between groups after partial output — each abort surfaces the cause at exactly the first cancelled check and returns the pool to its baseline. `TestQuantileSortSelectorsInterruptsInsideElementWork` proves a pre-cancelled context stops the sort before its first element move and a bounded-checkpoint abort stops mid-work (selectors neither identity nor fully sorted). `TestQuantileExactWeightedFinalizeScratchHonorsDesignBudget` arms an allocation-account controller over a real 8192-row group for INT64/UINT64, INT64-with-DECIMAL128-weight and DECIMAL128/DECIMAL128: the measured peak temporary capacity stays within — and above — the (valueWidth + 16) × N budget (the lower bound proves the restore, selectors and converted weights are genuinely simultaneous), and an admission run that denies any byte beyond the budget still finalizes the group correctly. `TestWindowQuantileExactWeightedLargePartitionHonorsCancellation` drives a 40000-row full-partition window with a context cancelled after Prepare and gets the standard cancellation error back through `vm.Exec` before any partition result is broadcast; the finalize's own mid-work abort points are pinned deterministically by the executor-level witnesses, and window partitions reuse the same `FlushWithContext` finalize and therefore the same cadence and budget. - Overflow semantics are unchanged: the uint64 wrap is detected during the restore scan instead of after the sort (any accumulation order detects the same wrap), with the same invalid-input abort and error-only result cleanup. DISTINCT/merge/wire/spill state is untouched. Local evidence (`mo-cgo-test`): the focused aggexec executor suite plus both allocation-failure sweeps, the `TestQuantileSortSelectors*` interruption and ordering-parity witnesses, and the window quantile suite including the large-partition cancellation all exit 0; `gofmt` is clean and `go vet` and `golangci-lint --new-from-rev` report zero findings on the changed packages. ### Update: semantic merge of `main` — MORPC capability slot re-minted 130 -> 131 Upstream `main` merged #30311 (`5a813a324d`, incomplete temporal string overloads), which independently minted `MORPCVersion130` and moved the `MORPCLatestVersion` sentinel to 130 — the same slot this branch had drafted for `quantileExactWeighted`'s remote-dispatch gate, leaving the PR DIRTY. Merge commit `2eecad8753` resolves it semantically; per the approved design's D10, no design decision changed, so no design re-review is required. - `pkg/defines/const.go`: upstream's temporal `MORPCVersion130` is kept, and this aggregate's capability version is re-minted at the next free slot — `MORPCVersion131 int64 = 131 // exact weighted percentile aggregate remote dispatch` — with `MORPCLatestVersion` moved to 131. All `MORPCLatestVersion` consumers use the symbol, so they follow the bump unchanged. - The aggregate's three verified gate layers (the `encodeRemoteScopeWithProtocol` `requiredProtocol` floor, the `AllCNsSupportProtocol` compile-time fast-fail, the coordinator self-gate message) and the protocol UT now reference `MORPCVersion131`; every upstream temporal gate keeps 130. The self-gate message now reads `quantileExactWeighted remote execution requires MORPC protocol version 131`. - The protocol UT gains the mixed-version counterexample the collision creates: a worker advertising only the temporal `MORPCVersion130` is below the aggregate's gate and fails closed both at the inventory fast-fail (coordinator at 131, worker at 130 — `supportsQuantileExactWeightedDispatch` false, shuffle-group compilation refused, remote group wire unsupported) and at the pre-dispatch stream confirmation (confirming 131 against a 130 worker errors before any pipeline payload). - The approved design document records this in its revision log as a post-merge annotation: D10's reviewed contract is that dispatch is gated by the capability version minted for THIS aggregate at whichever slot is free when it merges — only the concrete slot number moved (130 drafted pre-collision, 131 delivered), with the approved revision-5 bytes still byte-preserved at introducing commit `60d88ff6ad`. No decision, invariant, layering or validation plan changed. Validation of the merged tree (the tree that became `2eecad8753`, local macOS/arm64, `mo-cgo-test` wrapper): - `mo-cgo-test --count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> exit 0: the re-minted gate floor (`requiredProtocol` = 131 when an encoded pipeline carries the aggregate), the coordinator self-gate message at version 131, the compile-time inventory fast-fail across 129/130/131 — including the new counterexample: coordinator at 131 with a worker advertising only the temporal 130 fails closed (`supportsQuantileExactWeightedDispatch` false, shuffle-group compilation refused, remote group wire unsupported) — and the pre-dispatch stream confirmation (131 accepted; the temporal-130 worker, 129 and unknown capability rejected before any payload). A `-list` enumeration confirms all three expected test names matched, and a `-v` rerun on the same tree records every PASS line, including the `worker_at_temporal_gate,_below_aggregate_gate` subtest. - `mo-cgo-test --count=1 -run 'TestQuantile|TestAccounted' ./pkg/sql/colexec/aggexec/` -> exit 0: the full quantile executor matrix (issue fixture, endpoints, exact threshold arithmetic, typed weights, overflow, merge, wire, DISTINCT + spill, interruptible finalize) plus every allocation accounted/sweep case on the merged tree — upstream's Decimal128 short-coefficient rework (`0abc1d317a`) is inside this package's dependency closure. - `mo-cgo-test --count=1 -run 'TestWindowQuantileExactWeighted' ./pkg/sql/colexec/window/` -> exit 0: full-partition broadcast, retained-result survival after executor destruction, DISTINCT windows and the 40000-row large-partition cancellation witness on the merged tree (all four expected test names enumerated via `-list`). - `mo-cgo-test --count=1 -run 'TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters' ./pkg/sql/plan/` -> exit 0: binder/prepared-parameter behavior on the merged tree (upstream's base_binder/query_builder/utils changes auto-merged here; all three test names enumerated). - `mo-cgo-test --count=1 -run 'TestAggregateExecutorIDs|TestQuantileExactWeightedRegistration|TestQuantileExactWeightedCheckFnVerifiesTypedAdmission' ./pkg/sql/plan/function/` -> exit 0: the executor/function-id registry test (function id equality across both registries; 606 at that time, 616 after the second merge renumber) and typed-admission/registration checks on the merged tree. - `mo-cgo-test --count=1 ./pkg/sql/parsers/...` -> exit 0: the full parser corpus (`pkg/sql/parsers`, `dialect/mysql`, `dialect/postgresql`, `tree` all `ok`; `pkg/sql/parsers/util` changed upstream) — includes this PR's quantile grammar parser test. - Compile/link closure of the merged tree before the focused runs: defines, aggexec, compile and parsers packages all built and linked; static checks on the cons closure: `gofmt -l` clean over every changed file and `go test -vet=all` (full vet) exit 0 on `pkg/defines` and `pkg/sql/compile`. Local logs are preserved under `issue-29944/merge-conflict-resolution/` in the loop's state artifacts directory. ### Update: semantic merge of `main` (second) — ID-space collision 606 -> 616 and window-binder coexistence Upstream `main` advanced past the first merge's base (`ac13078b98`) with #31017 (`6fb6531432`, technical-indicator window kernels, whose registry allocated SQL function ids 605-615 beside `MO_VIEW`'s internal 603/604 helpers), #31021 (`a1888d62f8`) and #31035 (`f4343be0ae`), leaving the PR DIRTY (content conflicts in `pkg/sql/plan/function/function_id.go`, `function_id_test.go`, `pkg/sql/plan/window_binder.go` and the generated parser). The resolution keeps every upstream change and re-issues this aggregate's colliding bits: - `QUANTILE_EXACT_WEIGHTED = 616`, the sentinel `FUNCTION_END_NUMBER = 617`: upstream's indicator registry owns 605-615, so this aggregate takes the first unassigned catalog id past the current sentinel — the approved design's own allocation rule, with only the concrete number moved (606 drafted, 616 delivered; recorded in the design document's second post-merge annotation, a documentation change, not a design revision). `AggIdOfQuantileExactWeighted` tracks it as `616 << 32`; `Test_funids`'s predefined-Funids mapping, the composite-id assertion and the stale "cluster protocol 130" comment on the constant (now 131) follow. - `window_binder.go` keeps both independent hooks: this feature's DISTINCT window-aggregate recording and upstream's indicator-window `validateIndicatorWindowSpec` gate — disjoint paths, both exercised by their respective suites on the merged tree. `mysql_sql.go` is regenerated through the repository's in-tree goyacc make target under its zero-shift/reduce-conflict gate from the cleanly merged grammar (both this feature's `QUANTILE_EXACT_WEIGHTED` rules and upstream's grammar changes retained). - The dispatch gate stays on `MORPCVersion131`; `pkg/defines/const.go` auto-merged cleanly. Validation of the merged tree (local macOS/arm64, `mo-cgo-test` wrapper, ladder v3): - `mo-cgo-test --count=1 -run '^$' ./pkg/defines/ ./pkg/sql/colexec/aggexec/ ./pkg/sql/colexec/window/ ./pkg/sql/compile/ ./pkg/sql/plan/ ./pkg/sql/parsers/...` -> exit 0: compile/link of the whole owning closure on the merged tree (the working tree that became the merge commit). - `mo-cgo-test --count=1 -run 'TestQuantile|TestAccounted' ./pkg/sql/colexec/aggexec/` -> exit 0: the full quantile executor matrix — 67 matched tests (`-list`), including the 8192-row peak-scratch budget, cancellation/witness and `SortByVectors`-parity suites — green with `AggIdOfQuantileExactWeighted = 616 << 32`. - `mo-cgo-test --count=1 -run 'Test_funids' ./pkg/sql/plan/function/` and `-run 'TestAggregateExecutorIDs'` -> exit 0 both: the predefined-Funids consistency audit (616/617 entries, upstream's 605-615 kept) and the cross-registry id-equality circuit. - `mo-cgo-test --count=1 -run 'Quantile|FunctionID|Funids' ./pkg/sql/plan/function/` -> exit 0: registration and typed-admission checks on the merged registry (`TestQuantileExactWeightedRegistration`, `TestQuantileExactWeightedCheckFnVerifiesTypedAdmission` enumerated via `-list`). - `mo-cgo-test --count=1 -run 'TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters|TestIndicatorBinderSurface|TestIndicatorRegistryBits|TestIndicatorDeduceNotNullablePins' ./pkg/sql/plan/` -> exit 0: the manually resolved window-binder seam holds both sides — this feature's DISTINCT binding and upstream's indicator binder surface (all six names enumerated via `-list`). - `mo-cgo-test --count=1 -run 'TestWindowQuantileExactWeighted|TestIndicatorOperator|TestIndicatorChunkedEmit|TestIndicatorScanCancellationLatency' ./pkg/sql/colexec/window/` -> exit 0: this feature's four window-quantile witnesses and upstream's indicator operator emit/reset/cancel/lifecycle seams coexist on the auto-merged `window.go` (eleven names enumerated via `-list`). - `mo-cgo-test --count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> exit 0: the 129/130/131 dispatch matrix (including the temporal-130-only counterexample) and prepared-level preflight re-proven on the merged tree. - `mo-cgo-test --count=1 ./pkg/sql/parsers/...` -> exit 0: full parser corpus (`pkg/sql/parsers`, `dialect/mysql`, `dialect/postgresql`, `tree`) against the regenerated `mysql_sql.go`, which is goyacc-verified (zero shift/reduce conflicts) to carry both this feature's `QUANTILE_EXACT_WEIGHTED` grammar and upstream's changes. - `gofmt -l` clean over every changed file; `git diff --check` clean. Ladder v3 preserved per-step exit-status files: every step 0. Local logs are preserved under `issue-29944/merge2-conflict-resolution/` in the loop's state artifacts directory. ### Update: third merge of `main` — tracked CI blocker #31044 resolved (closed NOT_PLANNED); fresh-head re-evaluation With the second merge's head `f79bbb0dbe` terminally red only on the coverage lanes that tracked issue #31044 (run [40105576983](https://github.com/matrixorigin/matrixone/actions/runs/40105576983) attempt 4: `UT Coverage` producer flake with `Coverage`/`CI Required` as its projections) and that run's failed-jobs rerun budget consumed (attempts 3–4 were re-queued externally), the open→closed transition of #31044 was verified and the resolved-blocker contract was followed: current `main` was incorporated so the required checks get a fresh evaluation on a new head, and the stale run is not rerun. - **Verified blocker resolution**: #31044 was closed at 2026-10-24T21:37:23Z as NOT_PLANNED by XuPeng-SH — the closing review of the complete retained coverage artifact confirmed the complete `CHECK` namespace and all four cross-target recovery scenarios pass on current main `dae02bbaaa` (linux/amd64 race + matrixone_test, package 120.959s, manual-ready 16.71s), explicitly reviewed this PR's attempt-4 reproduction record, and re-classified it as package-budget boundary behavior rather than a recovery defect. No `pkg/embed` change was merged upstream (the six upstream commits touch no path under `pkg/embed`), so current main is the upstream state under which the blocker was declared resolved. - **Base update**: merge of `main` `dae02bbaaa` (upstream #31040 derived-group-by pruning, #31048 composite secondary-index filters, #30886 logservice mutation budgets, #30925 FK `ON DELETE/UPDATE SET NULL` guard, #31001 partition Top-1, #31041 `COM_QUIT` connection close) into `f79bbb0dbe` → `bbd761648c`. A clean auto-merge with zero content conflicts (`git merge-tree --write-tree` dry run: none): the only shared file `pkg/sql/plan/query_builder.go` merged at disjoint hunks (this feature's `NameQuantileExactWeighted` constant vs upstream's `createQuery`/`bindSelect` aggregate-consumer regions). `QUANTILE_EXACT_WEIGHTED` keeps catalog id 616 and `FUNCTION_END_NUMBER` 617 (upstream allocated no SQL function ids), the D10 wire gate `MORPCVersion131` is untouched, and the goyacc-verified generated parser stands byte-unchanged (no upstream grammar change); the upstream range contains no `.github` content. No reviewed design decision changed, so no design re-review was required. The update is a merge, not a rebase: every cited evidence hash stays reachable from the new head (the approved revision-5 bytes byte-preserved at `60d88ff6adb`, the remediation commits `0afd975232`/`5b60194404db`, prior merge `f79bbb0dbe`, and the recorded run identities). Validation of the merged tree (local macOS/arm64, `mo-cgo-test` wrapper, ladder v4 — the second merge's proven ladder set, plus the upstream test families that landed in packages this feature owns or at its binding seam: `sum_local_table_test.go` in aggexec and `derived_group_by_test.go`'s aggregate-consumer pruning tests in plan): - `mo-cgo-test --count=1 -run '^$' ./pkg/defines/ ./pkg/frontend/ ./pkg/logservice/ ./pkg/util/executor/ ./pkg/sql/colexec/aggexec/ ./pkg/sql/colexec/window/ ./pkg/sql/colexec/partition/ ./pkg/sql/compile/ ./pkg/sql/plan/ ./pkg/sql/plan/function/ ./pkg/sql/parsers/...` -> exit 0: compile/link proof of the whole owning closure extended with the upstream-touched runtime packages (`frontend`, `logservice`, `util/executor`, `colexec/partition`) — every test binary builds against the merged tree. - `mo-cgo-test --count=1 -run 'TestQuantile|TestAccounted|TestSumFullLocalTable' ./pkg/sql/colexec/aggexec/` -> exit 0 (`ok ... 4.391s`): 67 matched tests via `-list` (45 `TestQuantile*` + 20 `TestAccounted*` + upstream's 2 new `TestSumFullLocalTable*`) — the full quantile executor matrix including the 8192-row peak-scratch budget, cancellation/witness and `SortByVectors`-parity suites green with `AggIdOfQuantileExactWeighted = 616 << 32`, plus upstream's grouped-SUM local-table regressions coexisting in the same package. - `mo-cgo-test --count=1 -run 'Quantile|FunctionID|Funids' ./pkg/sql/plan/function/` -> exit 0 (`ok ... 2.015s`): registration, typed-admission and Funids registry-consistency checks (616/617 kept; upstream's 605-615 intact). - `mo-cgo-test --count=1 -run 'TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters|TestIndicatorBinderSurface|TestIndicatorRegistryBits|TestIndicatorDeduceNotNullablePins|TestPruneDerivedGroupBy|TestBuildPlanPrunesDerivedGroupKeys' ./pkg/sql/plan/` -> exit 0 (`ok ... 2.533s`): 10 matched tests via `-list` — this feature's DISTINCT window binding, helper-validation and prepared-parameter cases, upstream's indicator binder surface, and upstream's new derived-group-by pruning tests at the aggregate-consumer seam, all exercising the merged `query_builder.go`. - `mo-cgo-test --count=1 -run 'TestWindowQuantileExactWeighted|TestIndicatorOperator|TestIndicatorChunkedEmit|TestIndicatorScanCancellationLatency' ./pkg/sql/colexec/window/` -> exit 0 (`ok ... 2.674s`): 11 matched tests via `-list` — the window-quantile witnesses (including the 40000-row full-partition cancellation) and upstream's indicator operator emit/reset/lifecycle seams. - `mo-cgo-test --count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> exit 0 (`ok ... 4.440s`): the 129/130/131 dispatch matrix (including the temporal-130-only counterexample) and prepared-level preflight re-proven on the merged tree. - `mo-cgo-test --count=1 ./pkg/sql/parsers/...` -> exit 0: full parser corpus — `pkg/sql/parsers`, `dialect/mysql`, `dialect/postgresql` and `tree` all `ok` against the byte-unchanged goyacc-verified `mysql_sql.go`. - Static: `gofmt -l` over the feature-owned files and the auto-merged `query_builder.go` -> clean; `git diff --check` -> clean; `git status --porcelain` -> empty. Every step wrote an individual exit-status file; the ladder's overall status is 0. Local logs and per-step exit-status files are preserved under `issue-29944/merge3-conflict-resolution/` in the loop's state artifacts directory. ### Update: fourth merge of `main` — function-id re-issue 616 -> 619 (#31020 claimed the catalog tail) GitHub reported this PR DIRTY/CONFLICTING against head `bbd761648c` because upstream `main` advanced to `ac4ec31587` (#31020, local sliding-window ranks `mrank`/`cumrank`/`tmrank` — a feature that itself had been renumbered around #31017's 605-615 indicator block when it merged, and allocated SQL function ids `MRANK = 616`, `CUMRANK = 617`, `TMRANK = 618` with `FUNCTION_END_NUMBER = 619`). That is exactly the tail this aggregate occupied after its own second merge (`QUANTILE_EXACT_WEIGHTED = 616`, sentinel 617), so three content conflicts followed: `pkg/sql/plan/function/function_id.go`, `function_id_test.go`, and the generated parser `pkg/sql/parsers/dialect/mysql/mysql_sql.go`; the grammar (`mysql_sql.y`), keywords, window operator and window-binder seams auto-merged with disjoint paths. The upstream range contains no `.github` content. The resolution applies the approved design's recorded first-unassigned-id allocation rule unchanged — the same rule as the second merge, applied to new inputs; no design decision, invariant, admission set or validation plan changed, and the D10 wire gate `MORPCVersion131` is untouched: - Upstream's `MRANK = 616`, `CUMRANK = 617`, `TMRANK = 618` are kept; `QUANTILE_EXACT_WEIGHTED` is re-issued to the first unassigned catalog id `619`, with `FUNCTION_END_NUMBER = 620` (one past the highest allocated id). Every reference to this aggregate's id is by name, so renumbering is safe. - `AggIdOfQuantileExactWeighted` tracks the catalog id as `619 << 32` (in `pkg/sql/colexec/aggexec/function_id.go`); the executor-id tracking assertion in `pkg/sql/plan/function/quantile_exact_weighted_test.go` and the `Test_funids` predefined-pin map follow (the merged map pins MRANK 616, CUMRANK 617, TMRANK 618, QUANTILE_EXACT_WEIGHTED 619, FUNCTION_END_NUMBER 620). - The generated parser was regenerated through the repository's in-tree goyacc make target under its zero shift/reduce-conflict gate (0 conflicts, exit 0) from the cleanly merged grammar, which carries both this feature's `QUANTILE_EXACT_WEIGHTED` productions and upstream's `MRANK`/`CUMRANK`/`TMRANK` grammar (confirmed in the merged `mysql_sql.y` and `keywords.go`: `"quantileexactweighted"` beside `"mrank"`/`"cumrank"`/`"tmrank"`). - The auto-merged seams keep both sides' coexisting paths: upstream's rank-width binder/operator additions beside this aggregate's DISTINCT window hook, percentile-config normalization and executor wiring (`base_binder.go`, `window_binder.go`, `pkg/sql/colexec/window/window.go`). - `docs/design/20261023-exact-weighted-percentiles.md` records the re-issue as its third post-merge annotation — documentation change only, not a design revision; the approved revision-5 bytes remain byte-preserved at introducing commit `60d88ff6adb`. Merge (not rebase), preserving every cited evidence hash — revision-5 design bytes at `60d88ff6adb`, remediation commits `0afd975232`/`5b60194404db`, prior merges `2eecad8753`/`f79bbb0dbe`/`bbd761648c`, and the recorded workflow-run identities. Validation of the merged tree (local macOS/arm64, `mo-cgo-test` wrapper, ladder v5, `ddc46c416b`; every step exit 0, per-step status files preserved under `issue-29944/merge4-conflict-resolution/` in the loop's state artifacts): - `mo-cgo-test --count=1 -run '^$' ./pkg/defines/ ./pkg/frontend/ ./pkg/logservice/ ./pkg/util/executor/ ./pkg/sql/colexec/aggexec/ ./pkg/sql/colexec/window/ ./pkg/sql/colexec/partition/ ./pkg/sql/compile/ ./pkg/sql/plan/ ./pkg/sql/plan/function/ ./pkg/sql/parsers/...` -> exit 0: compile/link of the owning closure on the merged tree (including upstream-touched window/plan/parser/special-types files and upstream's `bindFuncExprImplByPlanExpr` signature change, whose merged call sites all use the new signature). - `mo-cgo-test --count=1 -run 'TestQuantile|TestAccounted|TestSumFullLocalTable' ./pkg/sql/colexec/aggexec/` -> exit 0 (`ok 1.999s`): the full quantile executor matrix — the same 67-test set as ladder v4 (the merge's only aggexec delta vs `bbd761648c` is the tracking constant `619 << 32`) — green. - `mo-cgo-test --count=1 -run 'Quantile|FunctionID|Funids' ./pkg/sql/plan/function/` -> exit 0 (`ok 1.522s`): `Test_funids` with the merged pin map and `TestAggregateExecutorIDs` cross-registry equality (catalog 619 == `619 << 32` executor id), registration and typed-admission checks. - `mo-cgo-test --count=1 -run '...8-name set...|TestLocalRank' ./pkg/sql/plan/` (full pattern: `TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters|TestIndicatorBinderSurface|TestIndicatorRegistryBits|TestIndicatorDeduceNotNullablePins|TestPruneDerivedGroupBy|TestBuildPlanPrunesDerivedGroupKeys|TestLocalRank`) -> exit 0 (`ok 2.098s`): 22 matched tests (`-list`) — this feature's binder/prepared-parameter/indicator surfaces beside upstream's 12 `TestLocalRank` binder and prepared-width families on the auto-merged `window_binder.go`/`base_binder.go`. - `mo-cgo-test --count=1 -run 'TestWindowQuantileExactWeighted|TestIndicatorOperator|TestIndicatorChunkedEmit|TestIndicatorScanCancellationLatency|TestLocalRank' ./pkg/sql/colexec/window/` -> exit 0 (`ok 12.374s`): 37 matched tests (`-list`) — the 4 window-quantile witnesses (including large-partition cancellation and executor-destruction retention) and upstream's 7 indicator seams and 26 `TestLocalRank` operator suites on the auto-merged `window.go`/`interval.go`/`local_rank.go`. - `mo-cgo-test --count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> exit 0 (`ok 1.690s`): the 129/130/131 dispatch matrix and prepared-level preflight re-proven on the merged tree. - `mo-cgo-test --count=1 ./pkg/sql/parsers/...` -> exit 0: full parser corpus (`pkg/sql/parsers`, `dialect/mysql` incl. upstream's `window_local_rank_parser_test.go` and this feature's `mysql_sql_test.go`, `dialect/postgresql`, `tree`) against the regenerated goyacc-verified parser. - Static: `gofmt -l` over the resolved and feature-owned files -> clean; `git diff --check` and `git diff --cached --check` -> clean. ### Update: review remediation — equal-key sort-work cancellation cadence (P2 · `quantile_exact_weighted.go:584`) The `mo-bot-local` review of head `ddc46c416b` requested changes for one P2: the interruptible heap sort charged its cancellation-checkpoint counter on sift-down element swaps only, and not even root-extraction swaps received a charge. With an equal-valued group every sift-down returns at the comparison without a swap, so the counter never advanced again: cancelling immediately after the sort's entry check let the whole sort run to completion (the review's 65,536-equal-values probe recorded `checks=1`, `selector_moves=65536`, `err=nil`), with the cancelled cause surfacing only at the later cumulative-scan checkpoints — a departure from approved D12's "bounded per-row cadence inside each group's finalization". Fix (commit `a4a7ef92bd`, `pkg/sql/colexec/aggexec/quantile_exact_weighted.go`): the checkpoint cadence now charges every elementary heap step — each sift-down iteration, including the comparing iterations that return without swapping, plus each root-extraction swap (the review's requested "count comparisons or iterations, including no-swap work"). `context.Cause` is re-checked at the sort's entry (unchanged) and after every `quantileSortCheckSteps = 1024` charged steps, before the next element move, so the work between checks stays bounded for every value distribution. The comparator, heap shape and selection order are untouched (the `SortByVectors` parity suite re-proves it); no design decision, invariant, admission set or validation plan changed, and DISTINCT/merge/wire/spill state is untouched. Regressions added per the finding (all in `quantile_exact_weighted_test.go`): - `TestQuantileSortSelectorsInterruptsEqualValuedWork` — the sort-level probe over 65,536 identical values: the counting warmup gates on real in-sort checkpoints; flipping at the first in-sort checkpoint surfaces the cancelled cause with zero element moves; a learned mid-extraction flip aborts with selectors neither identity nor the completed permutation. - `TestQuantileExactWeightedEqualValuedCancellation` — the executor-level regression ("cancel the finalization of a large equal-valued aggregate or window partition"; window partitions finalize through the same `FlushWithContext` path): a 4096-row equal-valued group's finalization cancelled at the sort's first in-work checkpoint and deep inside its element work surfaces the cause at exactly the first flipped check, frees every scratch and partial result, and a fresh executor over the same group still finalizes the correct value. - `TestQuantileSortSelectorsInterruptsInsideElementWork` retuned to the iteration-charged cadence (its old `n=2` flip pinned "after exactly 1024 swaps", which under descending input now fires during the no-swap heap build): the flip point is learned from a counting warmup, deep inside the extraction phase, keeping the same three assertions; it is an invariance witness on both cadences, green before and after this change. Validation of the remediation (local macOS/arm64, `mo-cgo-test` wrapper; per-step logs and exit-status files preserved under `issue-29944/review-equal-key-cancellation/logs/` in the loop state artifacts): - Pre-fix failure of the same new tests, implementation hunk stashed (`base-prefixed-code-fails.log`, exit 1): the sort-level equal-key warmup records exactly 1 checkpoint over 65,536 equal values and the executor-level equal-key group learns 11 total checks — zero in-sort checkpoints — both the review probe's signature. - Post-fix new regressions plus retuned descending witness exit 0 (`postfix-new-regressions-pass.log`, `ok 2.367s`). - Full aggexec matrix `TestQuantile|TestAccounted|TestSumFullLocalTable` exit 0 (`aggexec-full-quantile-matrix.log`, `ok 4.847s`; 69 matched tests via `-list`: 22 `TestQuantile*` + 45 `TestAccounted*` + 2 `TestSumFullLocalTable*` — the ladder-v5 67-test set plus the two new regressions), including the descending large-group cancellation matrix, the `SortByVectors` ordering-parity suite, the 8192-row peak-scratch budget admission and the allocation-failure sweeps. - `TestWindowQuantileExactWeighted` exit 0 (`window-quantile-witnesses.log`, `ok 1.419s`), including the 40000-row full-partition window cancellation through the shared finalization route. - Static: `gofmt -l` clean; `go vet` and `golangci-lint --new-from-rev ac4ec31` report zero findings on the changed closure under the repository CGo native environment; `git diff --check` clean. The fresh MatrixOne ALL CI run on the new head `a4a7ef92bd` re-evaluates every lane; the previous head `ddc46c416b` holds the recorded green verdicts (shared build, Ubuntu/x86 UT, arm64 SCA, UT Coverage, Coverage, `CI Required`, both multi-CN BVT compose legs, preflight, Iceberg Connector E2E). ### Update: fifth merge of `main` — conflict resolved against #30973's widened RANGE frame bounds GitHub reported this PR DIRTY/CONFLICTING against head `a4a7ef92bd` because upstream `main` advanced to `80ea60aaeb`, three commits past the merge-#4 base `ac4ec31587`: fix(window) #30973 (widen narrow integer RANGE frame bounds), fix(plan) #31015 (overlapping ON DELETE CASCADE/SET NULL order-independence) and fix(frontend) #31052 (preserve microseconds in session timestamp overrides). `merge-tree` pinned exactly **one** content conflict: `pkg/sql/plan/window_binder_test.go`, where #30973 appends `TestRangeFrameBoundTypeWidening` (plus its `T_int32`→`T_int64` prepared-marker assertion updates) at the same file tail this feature's `TestQuantileExactWeightedDistinctWindowBinding` occupies. Resolution (semantic union; no design change): the conflicted test file now carries **both** appended functions with upstream's widened marker assertions. `pkg/sql/plan/window_binder.go` and `pkg/sql/colexec/window/window.go` auto-merged because the seams are disjoint — this feature's DISTINCT window hook lives in `bindWindowFuncExpr` while #30973's `rangeFrameBoundType` call site lives in `bindWindowSpec`, and this feature's `orderedSetWindowAggregate` entry is a different function from #30973's `signedNarrowRangeBound`/`signedRangeOffsetValue` additions. This feature's tests use only ROWS frames, so the bound-widening change cannot alter any of this PR's expected outputs. Upstream allocated no SQL function IDs in this range: `QUANTILE_EXACT_WEIGHTED = 619`, `FUNCTION_END_NUMBER = 620` and every recorded id evidence stand, so no design-document annotation is required (no design decision, invariant, admission set or validation plan changed). The upstream range touches no `.github` and no mo-tester content. Merge (not rebase), preserving every cited evidence hash — revision-5 design bytes at `60d88ff6adb`, remediation commits `0afd975232`/`5b60194404db`/`a4a7ef92bd`, prior merges `2eecad8753`/`f79bbb0dbe`/`bbd761648c`/`ddc46c416b` — plus the recorded workflow-run identities. Validation of the merged tree (local macOS/arm64, `mo-cgo-test` wrapper, ladder v6, `a56f0b0313910268d8500260ec5572dfad514df7`; every step exit 0; console log, per-step status files and `-list` proofs preserved under `issue-29944/merge5-conflict-resolution/logs/` in the loop's state artifacts): - `mo-cgo-test --count=1 -run '^$' ./pkg/defines/ ./pkg/frontend/ ./pkg/logservice/ ./pkg/tests/issues/ ./pkg/util/executor/ ./pkg/sql/colexec/aggexec/ ./pkg/sql/colexec/window/ ./pkg/sql/compile/ ./pkg/sql/plan/ ./pkg/sql/plan/function/ ./pkg/sql/parsers/...` -> exit 0: compile/link of the owning closure on the merged tree (upstream's frontend, window, plan-DML and tests-issues files beside this feature's packages). - `mo-cgo-test --count=1 -run 'TestQuantile|TestAccounted|TestSumFullLocalTable' ./pkg/sql/colexec/aggexec/` -> exit 0 (`ok 1.512s`): the full quantile executor matrix — including the equal-key cancellation regressions, SortByVectors parity and budget/sweep witnesses — unchanged and green through the merge. - `mo-cgo-test --count=1 -run 'Quantile|FunctionID|Funids' ./pkg/sql/plan/function/` -> exit 0 (`ok 1.074s`): `Test_funids` with the unchanged pin map (MRANK 616, CUMRANK 617, TMRANK 618, QUANTILE_EXACT_WEIGHTED 619, FUNCTION_END_NUMBER 620) and `TestAggregateExecutorIDs` cross-registry equality (catalog 619 == `619 << 32`). - `mo-cgo-test --count=1 -run 'TestQuantileExactWeightedDistinctWindowBinding|TestBindWindowFuncExprValidationAndHelpers|TestPreparedQuantileExactWeightedParameters|TestIndicatorBinderSurface|TestIndicatorRegistryBits|TestIndicatorDeduceNotNullablePins|TestPruneDerivedGroupBy|TestBuildPlanPrunesDerivedGroupKeys|TestLocalRank|TestRangeFrameBoundTypeWidening|TestPreparedWindowRangeFrameMarkers' ./pkg/sql/plan/` -> exit 0 (`ok 1.623s`): 25 matched tests (`-list`) — this feature's binder/prepared-parameter surfaces and upstream's 12 `TestLocalRank` binder families (the merge-#4 set) beside the three new #30973 binder regressions (`TestRangeFrameBoundTypeWidening`, `TestPreparedWindowRangeFrameMarkers`, `TestPreparedWindowRangeFrameMarkersInBothBounds`), all on the auto-merged `window_binder.go`. - `mo-cgo-test --count=1 -run 'TestWindowQuantileExactWeighted|TestIndicatorOperator|TestIndicatorChunkedEmit|TestIndicatorScanCancellationLatency|TestLocalRank|TestBuildRangeIntervalWideSignedBounds' ./pkg/sql/colexec/window/` -> exit 0 (`ok 3.888s`): 38 matched tests (`-list`) — the four window-quantile witnesses, upstream's 7 indicator seams and 26 `TestLocalRank` operator suites (the merge-#4 37-test set) beside `TestBuildRangeIntervalWideSignedBounds`, on the auto-merged `window.go`. - `mo-cgo-test --count=1 -run 'TestQuantileExactWeighted|TestPreflightQuantileExactWeighted' ./pkg/sql/compile/` -> exit 0 (`ok 1.405s`): the compile dispatch and preflight sets re-proven on the merged tree. - `mo-cgo-test --count=1 ./pkg/sql/parsers/...` -> exit 0 (`pkg/sql/parsers`, `dialect/mysql`, `dialect/postgresql`, `tree` all `ok`): full parser corpus against the byte-unchanged goyacc-verified parser (this upstream range made no parser change; no regeneration was needed). - Static: `gofmt -l` over the resolved and auto-merged files -> clean; `git diff --check` and `git diff --cached --check` -> clean. The fresh MatrixOne ALL CI run on the new head re-evaluates every lane; the previous head `a4a7ef92bd` holds the fully green exact-head CI record (shared build, Ubuntu/x86 UT, arm64 SCA, UT Coverage, Coverage, `CI Required`, both multi-CN BVT compose legs, preflight, pr-size-label, coverage eligibility, Iceberg Connector E2E), and the standing review state — CHANGES_REQUESTED from the mo-bot review of the superseded `ddc46c416b`, with the equal-key remediation answered on `a4a7ef92bd` and every affected thread resolved — carries to the new head as CI and review proceed. Approved by: @XuPeng-SH, @iamlinjunhong

XuPeng-SH requested review from LeftHandCold, aptend, aunjgr and fengttt as code owners October 24, 2026 10:12

XuPeng-SH had a problem deploying to ci October 24, 2026 10:12 — with GitHub Actions Error

XuPeng-SH temporarily deployed to ci October 24, 2026 10:12 — with GitHub Actions Inactive

XuPeng-SH had a problem deploying to ci October 24, 2026 10:12 — with GitHub Actions Error

matrix-meow added the size/XL Denotes a PR that changes [1000, 1999] lines label Oct 24, 2026

mergify Bot added the kind/enhancement label Oct 24, 2026

mergify Bot temporarily deployed to ci October 24, 2026 10:14 Inactive

mergify Bot had a problem deploying to ci October 24, 2026 10:14 Failure

mergify Bot temporarily deployed to ci October 24, 2026 10:24 Inactive

XuPeng-SH commented Oct 24, 2026

View reviewed changes

mergify Bot had a problem deploying to ci October 24, 2026 11:04 Failure

XuPeng-SH had a problem deploying to ci October 24, 2026 11:10 — with GitHub Actions Failure

XuPeng-SH temporarily deployed to ci October 24, 2026 11:10 — with GitHub Actions Inactive

XuPeng-SH had a problem deploying to ci October 24, 2026 11:10 — with GitHub Actions Error

XuPeng-SH had a problem deploying to ci October 24, 2026 11:10 — with GitHub Actions Failure

XuPeng-SH mentioned this pull request Oct 24, 2026

ci: use bounded cluster admission for coverage UT matrixorigin/CI#468

Merged

XuPeng-SH temporarily deployed to ci October 24, 2026 11:20 — with GitHub Actions Inactive

XuPeng-SH commented Oct 24, 2026

View reviewed changes

XuPeng-SH had a problem deploying to ci October 24, 2026 12:20 — with GitHub Actions Failure

XuPeng-SH temporarily deployed to ci October 24, 2026 12:20 — with GitHub Actions Inactive

XuPeng-SH had a problem deploying to ci October 24, 2026 12:20 — with GitHub Actions Failure

XuPeng-SH temporarily deployed to ci October 24, 2026 12:30 — with GitHub Actions Inactive

XuPeng-SH commented Oct 24, 2026

View reviewed changes

XuPeng-SH had a problem deploying to ci October 24, 2026 13:14 — with GitHub Actions Failure

XuPeng-SH had a problem deploying to ci October 24, 2026 15:29 — with GitHub Actions Failure

XuPeng-SH had a problem deploying to ci October 24, 2026 15:35 — with GitHub Actions Failure

XuPeng-SH temporarily deployed to ci October 24, 2026 18:45 — with GitHub Actions Inactive

XuPeng-SH temporarily deployed to ci October 24, 2026 19:41 — with GitHub Actions Inactive

Merge branch 'main' into codex/issue31037-derived-group-keys

02b99ab

XuPeng-SH temporarily deployed to ci October 24, 2026 20:35 — with GitHub Actions Inactive

XuPeng-SH merged commit dae02bb into main Oct 24, 2026
27 checks passed

XuPeng-SH deleted the codex/issue31037-derived-group-keys branch October 24, 2026 20:35

XuPeng-SH temporarily deployed to ci October 24, 2026 20:45 — with GitHub Actions Inactive

XuPeng-SH temporarily deployed to ci October 24, 2026 21:26 — with GitHub Actions Inactive

gouhongshen mentioned this pull request Oct 24, 2026

feat(sql): add quantileExactWeighted exact weighted percentile #31000

Merged

1 of 7 tasks

Navigation Menu

Repository navigation

perf(sql): prune total derived grouping keys - #31040

perf(sql): prune total derived grouping keys#31040
XuPeng-SH merged 8 commits into
mainfrom
codex/issue31037-derived-group-keys

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Oct 24, 2026

Uh oh!

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

Uh oh!

XuPeng-SH left a comment

Uh oh!

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

Uh oh!

XuPeng-SH left a comment

Uh oh!

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

Uh oh!

XuPeng-SH left a comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

XuPeng-SH commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Uh oh!

qodo-code-review Bot commented Oct 24, 2026

Qodo reviews are paused for this user.

Uh oh!

XuPeng-SH commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuPeng-SH left a comment

Choose a reason for hiding this comment

Uh oh!

XuPeng-SH commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuPeng-SH left a comment

Choose a reason for hiding this comment

Uh oh!

XuPeng-SH commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuPeng-SH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

XuPeng-SH commented Oct 24, 2026 •

edited

Loading

XuPeng-SH commented Oct 24, 2026 •

edited

Loading