Skip to content

test(ts): seed each differential case's inputs by its name - #522

Merged
kixelated merged 6 commits into
mainfrom
quest/m0/checks/case-inputs
Nov 3, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m0/checks/case-inputs

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

differential.ts drew every random input from one seeded stream, and a harness picked the case by leading bytes modulo its case count: stock's and codec's first byte, lang's first two, and the first byte of each of task's steps. So adding a case shifted which case every random input reached, and the full-width boundary inputs reached only whatever sat in the first eight ops. A PR adding a case could then fail on another case's inputs, as #436 did on buf::write_all.

Approach

  • ts/tests/inputs.ts now does the input generation that used to be inline in differential.ts. A harness that exports ops, a Map from each case's name to the bytes that pick it, gets its inputs per case: the op alone, the op with every short tail, the u64 boundaries if it exports wide, and count random tails from a xorshift stream seeded by the FNV-1a hash of the case's name. A harness with no cases (share, bind, crates) keeps the single stream it had.
  • lang.ts and shapes' caller export ops as a Map. shapes names its 0xff Fault entry fmt::Fault.
  • stock's caller exports ops and wide. codec's harness names its cases, and a new caller.ts exports ops and wide.
  • task.ts exports ops and an input(name, tail) hook. A step's random scripts run that step plus only a fixed list of drivers (open, send, close, clone, run, step, poll, drop). A step added later never appears in another step's scripts.
  • Counts are now per case, each set to the largest share any case had under the old scheme: codec 2,300 (was 20,000 total over 9 cases), stock 320 (was 5,000 over 17), task 800 per step, lang unchanged at 151.
  • Fixes the first bug the new inputs found. capacity::vecs hit it with its own seeded input 0000670350e11df08b00993f: reserve, then Vec::from(VecDeque::from(v)), then clear. TS dropped below the capacity the reserve had held. rt.dequeToVec left a capacity that pushes alone would reach implicit, and capOf recomputes an implicit capacity from the length, so it shrank when the length did. It now holds the deque's buffer with capHold, as iterToArray already does for the allocation it reuses. This is the only conversion that builds a fresh Vec from another allocation's capacity (a heap reuses its own array).
  • Deletes quest/m0/checks/fixture-layout.md: #474 delivered its whole goal (lang split per module, a shared wasm target dir, reject keyed by item, targeted runs).

Proof

  • fixtures::case_inputs runs tests/stable.ts:
    • Appending a case leaves every other case's inputs byte-identical.
    • Inserting a case first, removing one, or reversing the order leaves every other case's inputs identical after its op bytes.
    • A new case's random tails differ from every existing case's.
    • Every script of every real task step runs only that step and the drivers.
    • Mutation-checked: a single shared stream fails the append check, a name-independent seed fails the "new case draws its own" check, and mixing in all steps fails the task check.
  • A demonstration on the real stock fixture: I inserted ("added", made) as the third case, translated, and hashed each case's inputs past its op. All 17 original cases' digests stayed identical, and added got its own:
$ diff before.txt after.txt
2a3
> added 1702 5e67fbfa364b9c82

Coverage, per case, against origin/main

  • lang: 151 random inputs per case, exactly. Before, it was 151 on average, depending on where random first bytes landed. Total input counts are unchanged.
  • stock and codec: every case now gets the boundary inputs. Before, only the first eight did (stock's buf::reserves onward and codec's errors never got them). Random inputs per case are at least the old expected share.
  • task: each step appears in 5,252 to 7,302 of its own random scripts' steps. The old uniform share was at most 5,208. Under the old scheme, steps that interesting()'s fixed bytes landed on got more (open and released about 34k, via 0x00 and 0xff), and that was an artifact of position.

CPU

Node-side CPU for every affected differential fixture (all 69 lang modules, stock, codec, task), same translated outputs, old vs new: 129 s vs 148 s user. Re-running ints and hints, where the gap showed, gave equal times both ways (12.1 vs 12.3 s, 2.9 vs 3.0 s), so the difference is host noise. task went from 11.6 s to 8.0 s and stock from 3.2 s to 4.5 s (boundaries for all 17 cases). Net: about neutral.

Impact

  • Public API: none. The changes are test-harness internals: ops is a Map, and the new wide/input exports are read by differential.ts.
  • codec's harness::cases now returns (name, case) pairs, and its snapshot changes to match.

Alternatives

  • Seed by op bytes instead of by name: an op's bytes are its position, so inserting or removing a case would still move every case after it.
  • Mix all steps into task's scripts: any script that mixes in the whole step list changes when a step is added. Getting every step back to the old ~4.4k scripts containing it, each with its own stream, would cost about 8x the CPU.
  • Give lang boundary inputs too: 242 cases x 1,365 inputs is about 9x lang's inputs. Its callers already name the full-width inputs their cases read.

Choices for review

  • task drops the boundary scripts its first eight steps got. Steps take one-byte arguments, and those scripts were mostly open and released with arguments 0x00 and 0xff, which the random scripts' interesting() bytes already produce for every step. Opting task into wide would cost about 45k more scripts.
  • In task, a step's scripts no longer mix in other non-driver steps (for example sum beside held). That cross-step mixing is what made the inputs unstable.
  • Named inputs (cases) now run after the generated ones instead of before the random ones. Both sides still run the same order.

Follow-ups

  • Any Vec whose capacity stays implicit (grown by pushes, or made exact by vecExact) still reports it shrinking when its length does. For example, Vec::new(), six 1280-byte pushes, then clear gives capacity 0 in TS where native keeps 8. capacity's Known only checks capacity against a reserved floor, so the fixture does not see this. Suggested as a quest: pin the implicit capacity before any step that drops the length.
  • rust2-go's go/tests/fixtures.rs still picks unnamed cases (match pick % n) from one stream. Its spike fixtures have fixed case lists, so it is left as is here. Its doc comment no longer claims to mirror rust2-ts. Suggested as a quest if the Go spike continues.

Review

🤖 Generated with Claude Code

kixelated and others added 2 commits November 3, 2026 05:45
A harness that names its cases (`lang`'s fixtures, `stock`, `codec`, `task`'s steps)
now draws each case's random inputs from a stream seeded by its name, so adding,
removing, or reordering a case changes no other case's inputs. `stable.ts` checks it.

Also deletes quest/m0/checks/fixture-layout.md, which #474 delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A case's inputs stay as cases are added test(ts): seed each differential case's inputs by its name Nov 3, 2026
kixelated and others added 4 commits November 3, 2026 06:21
…h drops

`dequeToVec` left the capacity implicit when pushes alone would reach it, so a
`clear` after `Vec::from(VecDeque::from(v))` dropped the capacity a `reserve` had
held. `capacity::vecs`' own seeded inputs found it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 3, 2026 14:34
@kixelated
kixelated added this pull request to the merge queue Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 726f967b-82d0-49c3-93ff-9097ab334a08
Commits

Reviewing files that changed from the base of the PR and between b3c1195 and 4018307.

Files selected for processing: 17
  • go/tests/fixtures.rs
  • quest/m0/checks/README.md
  • quest/m0/checks/case-inputs.md
  • quest/m0/checks/fixture-layout.md
  • ts/README.md
  • ts/fixtures/codec/caller.ts
  • ts/fixtures/codec/expected/harness.ts
  • ts/fixtures/codec/src/harness.rs
  • ts/fixtures/lang/lang.ts
  • ts/fixtures/lang/shapes/caller.ts
  • ts/fixtures/stock/caller.ts
  • ts/runtime/rt/collections.ts
  • ts/tests/differential.ts
  • ts/tests/fixtures.rs
  • ts/tests/inputs.ts
  • ts/tests/stable.ts
  • ts/tests/task.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

Merged via the queue into main with commit 95d9a90 Nov 3, 2026
5 checks passed
@kixelated
kixelated deleted the quest/m0/checks/case-inputs branch November 3, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant