Repository navigation
fix(ci): locate native libraries in coverage test executables - #465
Merged
XuPeng-SH merged 2 commits intoOct 20, 2026
Merged
Conversation
2 tasks done
XuPeng-SH
added a commit
to matrixorigin/matrixone
that referenced
this pull request
Oct 21, 2026
…ism (#30686) ## What type of PR is this? - [x] Improvement - [x] Test and CI ## Which issue(s) this PR fixes: Refs #29249. ## What this PR does / why we need it: Recent UT race jobs grew from roughly 55 minutes to 68–70 minutes. Two slow jobs spent 1082–1091 seconds in three table-dump roots. Avoid cloning already-canonical bound expressions, lazily reuse visited timezone rules within one verification, and run independent table-dump cases at bounded in-process parallelism 3. Process admission and the heavy-package budget remain unchanged. Keep every original scenario and comparison rule, cancellation behavior and candidate budget. Named rules and failed loads expire after each verification; UTC/Local matches require no named loads, and an early named witness never loads later candidates. Fixed numeric offsets have a finite shared cache. Differential tests retain the old expression comparator as an oracle. The same change also enforces nonempty, unique and complete raw test/package lifecycles, restores Iceberg fixture state according to ownership, fixes legacy UT cleanup under nounset, and places native object inputs before static-library dependencies during shared linking. Controlled race measurements: identical 91 original scenarios, precompiled binaries, GOMAXPROCS=8, baseline parallelism 1 vs current parallelism 3. The baseline is main `180608ea8c`; relevant baseline inputs are unchanged by the subsequent frontend-only main merge. Current treatment is `b691e23ef9`. | Metric | Baseline | Current | |---|---:|---:| | Execution wall | 373.7s | 117.2s | | User + system CPU | 391.7s | 284.8s | | Peak RSS | 264.6 MiB | 297.4 MiB | | Cumulative Go allocation (sampled pprof) | 40.76 GiB | 28.01 GiB | Focused wall decreased 68.6%, CPU 27.3%, cumulative Go allocation 31.3%. Peak RSS increased 12.4% (~33 MiB). Allocation traffic is not peak residency and excludes C/race-shadow allocation. All 91 named scenarios match exactly, and both current timing/allocation reports pass strict lifecycle validation. Applying the wall ratio only to the three expensive historical CI roots estimates ~12.4 minutes saved if other stages are unchanged. This is an estimate, not measured end-to-end CI improvement. Real-loader early-witness benchmarks also verify that laziness is preserved: ~22us and 13.4KB per operation, versus ~4.8ms and 6.24MB with eager cache population. Validation: - Full local UT race at `7572bb36a7`: passed in 25m56.7s under 8 CPU / 16 GiB, 511 packages, 82,156 explicit test starts, zero failures. Strict report validation: zero malformed/incomplete lifecycles, five known terminal-only events. Observed scope peak 12.24 GiB, swap/OOM/OOM-kill zero. Warm local caches prevent direct comparison to CI total duration. - After main `774d0eda08` was merged, both new frontend BIT export tests passed focused race (1.154s). - Current lazy iterator: standalone race passed; actual Analyze/comparator focused race passed (96.814s). Complete plan-package race passed in eight fresh processes with test parallelism 3 (209.343s): all 3,962 inventoried roots executed, 17,839 starts/terminals, zero malformed/incomplete lifecycles. - Full optools race and affected-package incremental SCA (optools/embed/plan) passed; current iterator follow-up passed incremental plan SCA. - Requested 6.1-sol-xhigh design/stage review covered the complete change; the lazy follow-up passed incremental review with no unresolved code blockers. Current-head (`b7d2208f83`) ordinary UT CI [job 118880250801](https://github.com/matrixorigin/matrixone/actions/runs/39495715240/job/118880250801) passed on the same `amd64-mo-shanghai-8c16g` runner class as both slow baseline jobs: | CI measurement | Slow baseline A | Slow baseline B | Current | |---|---:|---:|---:| | Entire UT job | 70m04s | 68m39s | 49m22s | | Unit Testing step | 68m10s | 66m52s | 47m46s | | Plan shard 1 to test-phase completion | 1363.7s | 1368.1s | 519.7s | | Final cumulative runner CPU | 13590.9s | 13747.9s | 12843.1s | | Sampled anonymous occupancy | 7754.4 GiB-seconds | 8034.5 GiB-seconds | 7025.3 GiB-seconds | | Sampled anonymous peak | 7.259 GiB | 6.810 GiB | 6.088 GiB | Observed whole-job saving is 19m17s–20m42s; CPU decreases 5.5%–6.6%, anonymous occupancy 9.4%–12.6%, and sampled anonymous peak 10.6%–16.1%. These historical jobs have different source SHAs and hosts; this is an observational same-runner-class comparison supported by controlled focused measurements, not a randomized whole-CI A/B. The incremental 8-batch policy is not credited with the entire difference between CI runs. The current strict raw-report gate passed: 974,310 events, 82,195 starts, 82,200 terminals, five known terminal-only events, zero malformed/incomplete lifecycles. No OOM/OOM-kill. Five additional starts match the five new scheduler matrix rows; package events increase by eight as expected from four additional fresh embed processes. Memory limits: anonymous peaks are minute-heartbeat samples, and occupancy uses trapezoids only between the first and last emitted heartbeat, without extrapolation. Mean sampled anonymous occupancy rises to 2.64 GiB from 1.98/2.11 GiB despite lower total occupancy and sampled peaks. Exact total cgroup peak remains approximately 16 GiB, including file cache; the current memory-limit event counter is six but OOM/OOM-kill remain zero. This does not establish lower average residency, exact RSS peak or lower total cgroup peak. Current-head full CI SCA also passed ([job 118880250719](https://github.com/matrixorigin/matrixone/actions/runs/39495715240/job/118880250719)). Coverage failure is triggered by this PR's native link-order change interacting with the reusable workflow's missing cgo runtime path. A successful coverage run on another PR uses the same workflow revision and linker flags. Under controlled `--as-needed` linking, old-order libmo is dropped from the test executable and passes, while new-order libmo is retained and fails to load; adding only the cgo rpath restores success. The failure was reproduced with actual MatrixOne coverage compiler flags and closed locally by adding only that rpath. Companion [matrixorigin/CI#465](matrixorigin/CI#465) fixes the shared workflow and adds an automated real-loader regression. It must be merged and consumed by a fresh coverage run before coverage CI can be called green. Ordinary UT and SCA in run 39481820248 both succeeded. Companion CI#465 workflow validation is green, including the new loader regression. ### Bounded embedded process generations On the measured four-worker path, `pkg/embed` now uses eight fresh race-test batches while retaining at most four active children, GOMAXPROCS=2, the existing shared deadline, inventory/report validation and failed-drain handling. Other packages and worker counts retain four batches; the whole-package rollback and prebuild paths are unchanged. Matched original-inventory-order full-package comparison using the same compiled race binary (8 CPU/16 GiB, no swap): | Metric | 4 batches | 8 batches | |---|---:|---:| | Complete-wave wall | 231.64s | 208.84s | | CPU | 361.43s | 379.27s | | Sampled cgroup anonymous peak | 7.73 GiB | 7.27 GiB | | Anonymous occupancy | 1120.25 GiB-seconds | 1020.81 GiB-seconds | This reduces sampled anonymous peak by 5.9%, anonymous occupancy by 8.9% and wall by 9.8%, at a cost of 17.85 CPU seconds for this package. These are observed whole-wave samples including the observer, not exact memory peaks or a claim of a production leak. All 138 roots ran exactly once; all 497 named terminal/action pairs, including the same two existing skips, matched. All twelve JSON streams passed strict lifecycle validation, all process groups drained, and neither arm swapped or OOMed. Exploratory sorted partitions also improved memory but are not used as the production baseline. The incremental scheduler matrix covers four-worker embed selection, lower-worker rollback, mixed packages and stop-before-next-package on undrained children. Complete optools race passed in 23.439s and incremental optools SCA reported zero issues. The current-head CI measurements above now cover the complete UT workload; these local controls explain the scheduler tradeoff without substituting for that whole-run evidence. Current head `b7d2208f83` ordinary UT passed in [run 39495715240](https://github.com/matrixorigin/matrixone/actions/runs/39495715240). SCA job 118880250719 passed. Coverage job 118880524894 failed with the confirmed native-link-order/runtime-path interaction; CI#465 remains open, so coverage success remains unverified. Recovery after CI#465 merges: the caller references the shared coverage workflow by `main`. GitHub reuses the first-attempt reusable-workflow SHA when re-running only failed/specific jobs. Use a fresh workflow run or re-run all jobs to consume the updated shared workflow; confirm the resolved workflow revision in the job log. Preserve the existing UT/SCA run first rather than cancelling its resource measurement. Reference: https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations#behavior-of-reusable-workflows-when-re-running-jobs Final check audit: no running jobs remain. The three failed checks are the coverage producer loader failure and its dependent coverage-merge/CI-Required gates; the merge log explicitly reports no complete producer profile and no coverage verdict. CI#465 has green workflow validation, is mergeable, and awaits GitHub-required review (`REVIEW_REQUIRED` / `BLOCKED`). Its normal approval and merge are the external prerequisite for fresh consumer verification. No default-branch mutation or review bypass has been performed. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage test executables link against
cgo/libmo.so, but their linker flags only embed a runtime path forthirdparties/install/lib. With no ambientLD_LIBRARY_PATH, Go starts the binaries from its temporary build tree and the loader fails before any tests run. MatrixOne PR matrixorigin/matrixone#30686 coverage job 118833087280 exhibits this across native-transitive packages.Add the workspace cgo directory to coverage executable rpath, retaining the existing third-party rpath and coverage selection. A real-compiler regression reads the workflow flags, builds libraries in both locations, and runs the executable from another directory with
LD_LIBRARY_PATHremoved. Removing just the cgo rpath reproduces the loader failure. Wire this test into the workflow-validation gate.Validation: the original coverage flags reproduce
libmo.so: cannot open shared object filein MatrixOnepkg/common/bitmapTestC; adding only the cgo rpath makes the same actual Go coverage test pass (0.014s). Six related helper tests passed; the new standard-library unittest passed directly; actionlint passed for both changed workflows with the repository's shellcheck/pyflakes settings. Requested 6.1-sol-xhigh incremental design/review found no blockers. Full coverage job recovery remains unverified until this reusable-workflow fix is merged and consumed by a fresh run.Trigger clarification: MatrixOne#30686 changes the native library link order. Other PRs can pass with the same coverage flags when
--as-neededdrops the unused libmo dependency. Controlled actual bitmap coverage tests: old order passes without libmo DT_NEEDED; new order retains libmo and exits127 without a cgo runtime path; adding only the cgo rpath passes. This fixes the workflow contract needed by that build change; it is not a claim that all coverage PRs currently fail.Recovery after CI#465 merges: the caller references the shared coverage workflow by
main. GitHub reuses the first-attempt reusable-workflow SHA when re-running only failed/specific jobs. Use a fresh workflow run or re-run all jobs to consume the updated shared workflow; confirm the resolved workflow revision in the job log. Preserve the existing UT/SCA run first rather than cancelling its resource measurement. Reference: https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations#behavior-of-reusable-workflows-when-re-running-jobs