Skip to content

fix: identify hyperd by executable path, and report decimal MB in the bench harness - #256

Merged
StefanSteiner merged 3 commits into
tableau:mainfrom
StefanSteiner:fix/watchdog-flake-and-mb-units
Sep 6, 2026
Merged

fix: identify hyperd by executable path, and report decimal MB in the bench harness#256
StefanSteiner merged 3 commits into
tableau:mainfrom
StefanSteiner:fix/watchdog-flake-and-mb-units

Conversation

@StefanSteiner

Copy link
Copy Markdown
Contributor

Two independent fixes, one commit each. They share no code and can be reviewed separately.

  • 6f4e739fix(mcp): identify hyperd by executable, not thread name
  • 77834dcfix(bench): report decimal MB, not MiB under an MB label

1 — The flaky watchdog test was a real Linux bug, not timing

What was wrong. slow_health_watchdog_reaps_hyperd_after_child_timeout failed
intermittently in CI. The cause was not timing. The reaping guard
validate_hyperd_process confirmed a process's identity with
ps -p <pid> -o comm=. On Linux that reads /proc/<pid>/comm — the main
thread's name
, not the process image — and hyperd renames its main thread to
hyperdMain via pthread_setname_np at startup. The guard therefore concluded
the PID was not hyperd, and stop_reported_hyperd refused to signal the very
process it exists to reap.

Why it stayed invisible for so long. Two things hid it:

  • The guard is normally skipped. By the time the watchdog fires the engine is
    usually already dead, and an Err(_) if !process_is_alive(pid) arm forgives an
    identity failure on an exited process. The test was passing for the wrong
    reason — it only failed when the engine was still alive at that moment.
  • macOS cannot observe the bug at all. There, ps -o comm= prints the executable
    path rather than a thread name, so every local run is green regardless.

Measured on real Linux. The old guard rejected a genuine, live hyperd
200 out of 200 times. The new /proc/<pid>/exe guard rejected it 0 out of
200 times
.

The fix. Read identity from the kernel's record of the mapped executable
(/proc/<pid>/exe) on Linux, which no prctl/pthread_setname_np can rewrite,
and keep ps on macOS/BSD where it is sound.

This is engine-independent: the previous 0.0.26359 pin flakes at the same rate
as the current 0.0.26479, so it is not a regression from the pin bump.

All three slow_health_* tests shared this fragility and the single fix covers
them. A new test, hyperd_identity_guard_accepts_a_live_engine, exercises the
identity path unconditionally, so it now runs on every build instead of only when
the timing happens to expose it.


2 — MB/sec silently meant two different units

What was wrong. BenchRecord::mb_per_sec(), fmt_mb() and the memory_*_mb
helpers divided by 1024² while labelling the result MB — so they emitted MiB.
The sibling formatters in the same module (fmt_count, fmt_rate, fmt_size)
were already decimal, so the harness was internally inconsistent.

Why it stayed invisible. The two conventions are only 4.86% apart, and both
landed in the same column of docs/BENCHMARK_GUIDE.md: the macOS tables were
decimal MB while the Windows tables were raw MiB, under one shared MB/sec
header. Nothing in the output distinguished them.

The arithmetic that pins it down.

  • macOS full scan, 3.218 s → the table reads 745.8. Decimal MB gives 745.8;
    MiB would give 711.3. So that row is decimal.
  • Windows sync Inserter, 22.716 s → the table reads 100.8. That is MiB;
    decimal would give 105.7. So that row is binary.

The fix. Make the harness decimal behind a documented BYTES_PER_MB
constant, rather than relabelling the column to MiB. Relabelling would have been
the wrong direction: the macOS figures were freshly re-measured and are genuinely
decimal, so calling them MiB would overstate them by 4.86%. Installed RAM stays
binary — a 36 GiB machine must read 36.0 GB, not 38.7 — with a comment saying
why it is deliberately the exception.

The Windows tables are not converted. Their header is relabelled to an
explicit MiB/sec and footnoted instead, because that run is stale on three
other axes as well; multiplying by 1.048576 would dress stale data up as a fresh
sample.

Test placement is load-bearing here. Coverage went into a new
hyperdb-api/tests/bench_common_tests.rs target. The benches are registered as
examples with autobenches = false, so cargo test --benches runs nothing and
a #[test] inside a bench file would never execute at all.

Also fixed in passing: a narrowing i64 as i32 in gen_id, which silently
wrapped past 2^31 rows and emitted duplicate and negative IDs — corrupting the
very throughput numbers being measured. It is now
i32::try_from(..).expect(..).


Deliberately left for follow-ups

  • The same narrowing i64 as i32 pattern remains in the per-row generators of
    benchmark.rs, async_parallel_benchmark.rs and benchmark_suite.rs. Kept
    out of a units commit on purpose.
  • hyperdb-api-salesforce/examples/salesforce_auth_example.rs:425 has the
    identical MiB-labelled-as-MB defect.
  • The Windows tables need re-measuring on a Windows host before both table
    headers can be unified to MB/sec (10⁶ B/s).

Verification

No rebase was needed. This branch's base (chore/post-rc-cleanup) landed in
main as #253, and main's only change since is the npm workflow's mirrored
hyperd pin — a file this branch does not touch. git log upstream/main..HEAD
is exactly the two commits above.

Re-run against main at 9dabbef:

Gate Result
cargo fmt --all -- --check clean, exit 0
cargo clippy --workspace --all-targets --all-features -- -D warnings exit 0, 0 diagnostics (483 artifacts, 75 test targets)
make test 1597 passed, 0 failed, 50 ignored, exit 0
npx markdownlint-cli2 126 issues in 31 files — identical to baseline

Test count derivation: main is 1586, and this branch adds 11 — 10 in the new
bench_common_tests.rs plus hyperd_identity_guard_accepts_a_live_engine — for
1597. The markdownlint findings for both changed Markdown files
(docs/BENCHMARK_GUIDE.md, hyperdb-api/CHANGELOG.md) were diffed against
upstream/main's copies and are identical rule-for-rule, so this branch adds no
new findings to the existing backlog.

@StefanSteiner
StefanSteiner force-pushed the fix/watchdog-flake-and-mb-units branch from 77834dc to 9b14bf8 Compare September 6, 2026 01:53
StefanSteiner and others added 3 commits September 5, 2026 19:13
`slow_health_watchdog_reaps_hyperd_after_child_timeout` failed once on
`test (ubuntu-latest)` and passed on a plain re-run. The CI log names the
cause exactly:

    refusing to terminate reported PID 20613:
    process is "hyperdMain\n", not hyperd

`stop_reported_hyperd` refuses to signal a PID whose identity it cannot
confirm, and `validate_hyperd_process` confirmed identity with
`ps -p <pid> -o comm=`. On Linux that reads `/proc/<pid>/comm`, which is
the **main thread's name** — not the process image — and `hyperd` renames
its main thread to `hyperdMain` via `pthread_setname_np` during startup
(the string and the `pthread_setname_np` import are both in the shipped
binary). So on Linux the guard rejected the very process it existed to
reap, every single time it ran.

It looked healthy only because it was almost always *skipped*: the engine
is normally already dead by the time the guard is reached, and the
`Err(_) if !process_is_alive(pid)` arm forgives an identity failure on an
exited process. The flake is therefore not a race in the watchdog's 250 ms
poll at all — it is the probability that `hyperd` is still alive when the
guard runs, multiplied by a guard that was deterministically wrong.

Measured, rather than assumed:

- On Linux (procps-ng 4.0.2, an executable named `hyperd` that renames its
  main thread as the real one does), over 200 iterations against a live
  process: the old `comm` probe rejected a genuine `hyperd` **200/200**;
  `readlink /proc/<pid>/exe` rejected it **0/200**.
- On macOS, 700 iterations of the failing test: **0** failures, because
  macOS `ps -o comm=` prints the executable *path*, so the platform cannot
  observe this bug. The guard was nonetheless reached with the engine still
  alive in ~26% of runs, and hit the full Linux-fatal condition in 5/700
  (~0.7%) — the same order as a "failed once" CI flake.
- The old `0.0.26359` pin shows the same rate as `0.0.26479` (3/200 vs
  2/500), so the engine bump did not introduce this; it is pre-existing.

Resolve identity from the kernel's own record of the mapped image —
`/proc/<pid>/exe` on Linux, which no `prctl`/`pthread_setname_np` can
rewrite — and keep `ps -o comm=` on macOS/BSD, where it is a path and
therefore already sound. This is a deterministic construction rather than a
longer timeout: nothing about it depends on who wins the shutdown race.

Add `hyperd_identity_guard_accepts_a_live_engine`, which validates identity
against a running engine unconditionally. All three `slow_health_*` tests
shared the defect through `stop_reported_hyperd`, and all three only
exercised the guard by accident; this pins it on every run.
`BenchRecord::mb_per_sec`, `fmt_mb` and the `memory_*_mb` helpers in
`benches/common.rs` all divided byte counts by 1024² while labelling the
result `MB`, so the harness emitted MiB. Their sibling formatters in the
same module — `fmt_count`, `fmt_rate`, `fmt_size` — were already decimal,
which made the MiB ones the outliers rather than the convention.

The consequence reached the published numbers: in `BENCHMARK_GUIDE.md` the
macOS tables are decimal MB while the native Windows tables are raw MiB,
under one shared `MB/sec` header. Verified arithmetically against the
recorded times and the 24 B/row schema — macOS full-scan at 3.218 s reads
745.8 (decimal; MiB would be 711.3), Windows sync `Inserter` at 22.716 s
reads 100.8 (MiB; decimal would be 105.7), and Windows `query.filtered`
at 1.263 s reads 90.6 on its 12 B/row projection (MiB; decimal 95.0). The
gap is 4.86%.

Make the harness decimal (`/1e6`) rather than relabelling to MiB.
Relabelling would make the freshly re-measured macOS tables overstate by
4.86%; going decimal makes the harness self-consistent with its own sibling
formatters, matches the conventional unit for I/O throughput, and validates
the already-published macOS numbers.

Also converts the same 1024²-under-an-`MB`-label divisions in `benchmark`
(whose own `mb_per_sec` helper was binary while its
`QueryBenchmarkResult::new` was already decimal — inconsistent within one
file), `arrow_batching_benchmark` and `async_parallel_benchmark`. Installed
RAM stays binary, with a comment saying why: a 36 GiB machine must report
"36.0 GB", not "38.7".

Windows tables are left exactly as measured and footnoted as MiB rather
than multiplied by 1.048576. They are stale on three further axes —
`hyperdb-api` 0.1.0-rc.1, rustc 1.92.0, and an unrecorded `hyperd`
predating `0.0.26479` — so they need re-measuring regardless, and an
arithmetic conversion would dress stale data up as a fresh sample. Their
column header now reads `MiB/sec` so the table is not self-contradictory in
the meantime. Windows benchmarks cannot be run from this host.

Adds `hyperdb-api/tests/bench_common_tests.rs`, which pins the unit. The
benches are registered as *examples* (`autobenches = false`), so
`cargo test --benches` runs nothing and a `#[test]` inside a bench file
would never execute; this target pulls in the same `common.rs` through the
same `#[path]` include the benches use, so the assertions run under
`make test`.

Per AGENTS.md, converts the narrowing `i64 as i32` in `gen_id` to
`TryFrom`. It is the canonical shared row generator, and past 2^31 rows the
cast silently wrapped to duplicate and negative IDs — corrupting the very
throughput numbers the suite publishes.
@StefanSteiner
StefanSteiner force-pushed the fix/watchdog-flake-and-mb-units branch from 08cc685 to 82d832b Compare September 6, 2026 02:13
@StefanSteiner
StefanSteiner merged commit 816ecbb into tableau:main Sep 6, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant