fix: identify hyperd by executable path, and report decimal MB in the bench harness - #256
Merged
StefanSteiner merged 3 commits intoSep 6, 2026
Conversation
StefanSteiner
force-pushed
the
fix/watchdog-flake-and-mb-units
branch
from
September 6, 2026 01:53
77834dc to
9b14bf8
Compare
`slow_health_watchdog_reaps_hyperd_after_child_timeout` failed once on
`test (ubuntu-latest)` and passed on a plain re-run. The CI log names the
cause exactly:
refusing to terminate reported PID 20613:
process is "hyperdMain\n", not hyperd
`stop_reported_hyperd` refuses to signal a PID whose identity it cannot
confirm, and `validate_hyperd_process` confirmed identity with
`ps -p <pid> -o comm=`. On Linux that reads `/proc/<pid>/comm`, which is
the **main thread's name** — not the process image — and `hyperd` renames
its main thread to `hyperdMain` via `pthread_setname_np` during startup
(the string and the `pthread_setname_np` import are both in the shipped
binary). So on Linux the guard rejected the very process it existed to
reap, every single time it ran.
It looked healthy only because it was almost always *skipped*: the engine
is normally already dead by the time the guard is reached, and the
`Err(_) if !process_is_alive(pid)` arm forgives an identity failure on an
exited process. The flake is therefore not a race in the watchdog's 250 ms
poll at all — it is the probability that `hyperd` is still alive when the
guard runs, multiplied by a guard that was deterministically wrong.
Measured, rather than assumed:
- On Linux (procps-ng 4.0.2, an executable named `hyperd` that renames its
main thread as the real one does), over 200 iterations against a live
process: the old `comm` probe rejected a genuine `hyperd` **200/200**;
`readlink /proc/<pid>/exe` rejected it **0/200**.
- On macOS, 700 iterations of the failing test: **0** failures, because
macOS `ps -o comm=` prints the executable *path*, so the platform cannot
observe this bug. The guard was nonetheless reached with the engine still
alive in ~26% of runs, and hit the full Linux-fatal condition in 5/700
(~0.7%) — the same order as a "failed once" CI flake.
- The old `0.0.26359` pin shows the same rate as `0.0.26479` (3/200 vs
2/500), so the engine bump did not introduce this; it is pre-existing.
Resolve identity from the kernel's own record of the mapped image —
`/proc/<pid>/exe` on Linux, which no `prctl`/`pthread_setname_np` can
rewrite — and keep `ps -o comm=` on macOS/BSD, where it is a path and
therefore already sound. This is a deterministic construction rather than a
longer timeout: nothing about it depends on who wins the shutdown race.
Add `hyperd_identity_guard_accepts_a_live_engine`, which validates identity
against a running engine unconditionally. All three `slow_health_*` tests
shared the defect through `stop_reported_hyperd`, and all three only
exercised the guard by accident; this pins it on every run.
`BenchRecord::mb_per_sec`, `fmt_mb` and the `memory_*_mb` helpers in `benches/common.rs` all divided byte counts by 1024² while labelling the result `MB`, so the harness emitted MiB. Their sibling formatters in the same module — `fmt_count`, `fmt_rate`, `fmt_size` — were already decimal, which made the MiB ones the outliers rather than the convention. The consequence reached the published numbers: in `BENCHMARK_GUIDE.md` the macOS tables are decimal MB while the native Windows tables are raw MiB, under one shared `MB/sec` header. Verified arithmetically against the recorded times and the 24 B/row schema — macOS full-scan at 3.218 s reads 745.8 (decimal; MiB would be 711.3), Windows sync `Inserter` at 22.716 s reads 100.8 (MiB; decimal would be 105.7), and Windows `query.filtered` at 1.263 s reads 90.6 on its 12 B/row projection (MiB; decimal 95.0). The gap is 4.86%. Make the harness decimal (`/1e6`) rather than relabelling to MiB. Relabelling would make the freshly re-measured macOS tables overstate by 4.86%; going decimal makes the harness self-consistent with its own sibling formatters, matches the conventional unit for I/O throughput, and validates the already-published macOS numbers. Also converts the same 1024²-under-an-`MB`-label divisions in `benchmark` (whose own `mb_per_sec` helper was binary while its `QueryBenchmarkResult::new` was already decimal — inconsistent within one file), `arrow_batching_benchmark` and `async_parallel_benchmark`. Installed RAM stays binary, with a comment saying why: a 36 GiB machine must report "36.0 GB", not "38.7". Windows tables are left exactly as measured and footnoted as MiB rather than multiplied by 1.048576. They are stale on three further axes — `hyperdb-api` 0.1.0-rc.1, rustc 1.92.0, and an unrecorded `hyperd` predating `0.0.26479` — so they need re-measuring regardless, and an arithmetic conversion would dress stale data up as a fresh sample. Their column header now reads `MiB/sec` so the table is not self-contradictory in the meantime. Windows benchmarks cannot be run from this host. Adds `hyperdb-api/tests/bench_common_tests.rs`, which pins the unit. The benches are registered as *examples* (`autobenches = false`), so `cargo test --benches` runs nothing and a `#[test]` inside a bench file would never execute; this target pulls in the same `common.rs` through the same `#[path]` include the benches use, so the assertions run under `make test`. Per AGENTS.md, converts the narrowing `i64 as i32` in `gen_id` to `TryFrom`. It is the canonical shared row generator, and past 2^31 rows the cast silently wrapped to duplicate and negative IDs — corrupting the very throughput numbers the suite publishes.
StefanSteiner
force-pushed
the
fix/watchdog-flake-and-mb-units
branch
from
September 6, 2026 02:13
08cc685 to
82d832b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent fixes, one commit each. They share no code and can be reviewed separately.
6f4e739—fix(mcp): identifyhyperdby executable, not thread name77834dc—fix(bench): report decimal MB, not MiB under anMBlabel1 — The flaky watchdog test was a real Linux bug, not timing
What was wrong.
slow_health_watchdog_reaps_hyperd_after_child_timeoutfailedintermittently in CI. The cause was not timing. The reaping guard
validate_hyperd_processconfirmed a process's identity withps -p <pid> -o comm=. On Linux that reads/proc/<pid>/comm— the mainthread's name, not the process image — and
hyperdrenames its main thread tohyperdMainviapthread_setname_npat startup. The guard therefore concludedthe PID was not
hyperd, andstop_reported_hyperdrefused to signal the veryprocess it exists to reap.
Why it stayed invisible for so long. Two things hid it:
usually already dead, and an
Err(_) if !process_is_alive(pid)arm forgives anidentity failure on an exited process. The test was passing for the wrong
reason — it only failed when the engine was still alive at that moment.
ps -o comm=prints the executablepath rather than a thread name, so every local run is green regardless.
Measured on real Linux. The old guard rejected a genuine, live
hyperd200 out of 200 times. The new
/proc/<pid>/exeguard rejected it 0 out of200 times.
The fix. Read identity from the kernel's record of the mapped executable
(
/proc/<pid>/exe) on Linux, which noprctl/pthread_setname_npcan rewrite,and keep
pson macOS/BSD where it is sound.This is engine-independent: the previous
0.0.26359pin flakes at the same rateas the current
0.0.26479, so it is not a regression from the pin bump.All three
slow_health_*tests shared this fragility and the single fix coversthem. A new test,
hyperd_identity_guard_accepts_a_live_engine, exercises theidentity path unconditionally, so it now runs on every build instead of only when
the timing happens to expose it.
2 —
MB/secsilently meant two different unitsWhat was wrong.
BenchRecord::mb_per_sec(),fmt_mb()and thememory_*_mbhelpers divided by 1024² while labelling the result
MB— so they emitted MiB.The sibling formatters in the same module (
fmt_count,fmt_rate,fmt_size)were already decimal, so the harness was internally inconsistent.
Why it stayed invisible. The two conventions are only 4.86% apart, and both
landed in the same column of
docs/BENCHMARK_GUIDE.md: the macOS tables weredecimal MB while the Windows tables were raw MiB, under one shared
MB/secheader. Nothing in the output distinguished them.
The arithmetic that pins it down.
MiB would give 711.3. So that row is decimal.
Inserter, 22.716 s → the table reads 100.8. That is MiB;decimal would give 105.7. So that row is binary.
The fix. Make the harness decimal behind a documented
BYTES_PER_MBconstant, rather than relabelling the column to MiB. Relabelling would have been
the wrong direction: the macOS figures were freshly re-measured and are genuinely
decimal, so calling them MiB would overstate them by 4.86%. Installed RAM stays
binary — a 36 GiB machine must read
36.0 GB, not38.7— with a comment sayingwhy it is deliberately the exception.
The Windows tables are not converted. Their header is relabelled to an
explicit
MiB/secand footnoted instead, because that run is stale on threeother axes as well; multiplying by 1.048576 would dress stale data up as a fresh
sample.
Test placement is load-bearing here. Coverage went into a new
hyperdb-api/tests/bench_common_tests.rstarget. The benches are registered asexamples with
autobenches = false, socargo test --benchesruns nothing anda
#[test]inside a bench file would never execute at all.Also fixed in passing: a narrowing
i64 as i32ingen_id, which silentlywrapped past 2^31 rows and emitted duplicate and negative IDs — corrupting the
very throughput numbers being measured. It is now
i32::try_from(..).expect(..).Deliberately left for follow-ups
i64 as i32pattern remains in the per-row generators ofbenchmark.rs,async_parallel_benchmark.rsandbenchmark_suite.rs. Keptout of a units commit on purpose.
hyperdb-api-salesforce/examples/salesforce_auth_example.rs:425has theidentical MiB-labelled-as-MB defect.
headers can be unified to
MB/sec (10⁶ B/s).Verification
No rebase was needed. This branch's base (
chore/post-rc-cleanup) landed inmainas #253, andmain's only change since is the npm workflow's mirroredhyperdpin — a file this branch does not touch.git log upstream/main..HEADis exactly the two commits above.
Re-run against
mainat9dabbef:cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningsmake testnpx markdownlint-cli2Test count derivation:
mainis 1586, and this branch adds 11 — 10 in the newbench_common_tests.rsplushyperd_identity_guard_accepts_a_live_engine— for1597. The markdownlint findings for both changed Markdown files
(
docs/BENCHMARK_GUIDE.md,hyperdb-api/CHANGELOG.md) were diffed againstupstream/main's copies and are identical rule-for-rule, so this branch adds nonew findings to the existing backlog.