Skip to content

test(e2e): add headless bootstrap negative controls - #342

Merged
roodboi merged 2 commits into
nextfrom
wf/f15-headless-negatives-b16
Oct 20, 2026
Merged

roodboi merged 2 commits into
nextfrom
wf/f15-headless-negatives-b16

Conversation

@roodboi

@roodboi roodboi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Final behavior

Adds the hosted Docker scenario portable-bootstrap-negatives for HACK-1143 (WU30). It extends the portable bootstrap fixture from #146/#147 with negative controls that do not depend on WU12. No product code changes.

The scenario runs the selected CLI in slim mode with cached alpine:3.20 pinned by immutable ID, network_mode: none, no published ports and no pulls. A deps producer fills a locked-v1 dependency cache with a frozen install; the app consumer mounts it read-only and loads the cache only after an app-scoped encrypted synthetic token matches its digest. Every stage runs hack up --detach app:

Stage Required result
refuse-missing-key, refuse-wrong-key, refuse-wrong-overlay refused; no owned container, network or volume; no worktree diff (the first refusal may only add the managed .hack/.gitignore)
bootstrap-cold consumer created; one initialization; one ready cache; no diff
bootstrap-repeat same consumer ID and start time, same cache, still one initialization, same index, no diff
refuse-wrong-key-running refused; running consumer unchanged
refuse-stale-image refused; consumer unchanged; stale ID not pulled; cached tag keeps its ID
refuse-frozen-lockfile refused; consumer unchanged; lockfile bytes kept; new cache quarantined without a ready marker
bootstrap-changed-lockfile consumer replaced on a new cache initialized once; previous ready cache kept
refuse-corrupted-cache refused; consumer unchanged; no reinstall; corrupted marker not rewritten
bootstrap-new-generation consumer replaced on a new cache; corrupted cache kept

Output and tracked files must never contain the keys or token values; the scenario reads only fixed container fields because the consumer's environment holds the token. Cleanup checks exact owner and Compose project labels; an adjacent canary volume must survive.

The scenario ends with one value-free portable-bootstrap receipt {...} line: per stage the outcome, consumer transition, duration, stdout/stderr bytes, tracked fixture bytes, mounted-cache bytes and owned cache-volume count, plus the pinned image size. tests/e2e/portable-bootstrap-controls.ts refuses any other shape (unknown keys, open-ended strings, missing or reordered stages, non-integer measurements).

Also:

  • tests/portable-bootstrap-env-refusal.test.ts runs the source CLI on the same fixture against a recording Docker stub (via privateCliEnv): missing, wrong and wrong-overlay keys fail before any Compose create/run/start/up; the correct key reaches the deps bootstrap first.
  • tests/e2e-portable-bootstrap-controls.test.ts covers stage expectations, consumer classification, worktree diff, receipt validation and the synthetic v1 envelope.
  • encryptSynthetic is now exported from scripts/portable-bootstrap-smoke.ts (no behavior change).
  • Registered in tests/e2e/run.ts with a 45 s shard estimate; docs in tests/e2e/README.md and docs/guides/portable-bootstrap-qualification.md.

Validation (local, macOS, Bun 1.4.3; pinned 1.4.2)

  • Focused: tests/portable-bootstrap-env-refusal.test.ts, tests/e2e-portable-bootstrap-controls.test.ts, tests/cli-spawn-env-isolation.test.ts, tests/integration-ownership.test.ts, tests/e2e-shard.test.ts, tests/e2e-scenario-selection.test.ts: 54 pass, 0 fail.
  • bun run --cwd packages/cli typecheck: pass. bun run --cwd packages/cli check: 0 errors (existing warnings only). bunx ultracite check on all changed TS files: clean. bun run privacy:check: ok.
  • bun tests/e2e/run.ts --list shows the scenario in the docker tier.
  • Mutations (each restored afterwards):
    • Remove both missing-key guards in project-env-config.ts: the missing-key refusal test fails (Compose run reached).
    • Swallow decryption failures: the wrong-key and wrong-overlay tests fail.
    • Ignore start time in consumer classification, drop the liveness check, accept negative counts, drop rename handling, accept unknown transitions, or change the stale-image or repeat expectation: each fails the controls test.
  • tests/portable-bootstrap-smoke.test.ts is flaky on this shared host under load (average about 35); it fails intermittently on unchanged origin/next code as well.
  • No Docker was run locally.

Hosted validation (exact head 4babce5b)

  • docker-e2e shard 4/4 passed the scenario in two runs: run 39281111519 (37.2 s, 1 pass) and run 39288207970. Receipt from the first run (value-free):

    Stage Outcome Consumer ms stdout B stderr B cache B cache volumes
    refuse-missing-key refused absent 228 962 0 - 0
    refuse-wrong-key refused absent 170 778 0 - 0
    refuse-wrong-overlay refused absent 172 778 0 - 0
    bootstrap-cold bootstrapped created 551 71 506 104 1
    bootstrap-repeat bootstrapped unchanged 553 71 179 104 1
    refuse-wrong-key-running refused unchanged 196 778 0 104 1
    refuse-stale-image refused unchanged 245 147 371 104 2
    refuse-frozen-lockfile refused unchanged 309 147 485 104 3
    bootstrap-changed-lockfile bootstrapped replaced 10494 71 508 104 4
    refuse-corrupted-cache refused unchanged 307 147 293 104 4
    bootstrap-new-generation bootstrapped replaced 10565 71 508 104 5

    Tracked fixture bytes were 4,872 before the managed .hack/.gitignore and 5,068 after (5,098 while the manifest was deliberately edited). The image is 7,807,582 bytes. The ~10.5 s replacements are mostly Compose stopping the old sleep consumer. The stale-image stage leaves one empty cache volume for its new fingerprint.

  • Docker mutation control: a temporary commit (633a1820, since force-pushed away) made the locked-v1 installer accept a mismatched ready marker. Shard 4 failed at exactly refuse-corrupted-cache: expected refused, observed bootstrapped after all earlier stages passed, and cleanup completed. The mutated run was then cancelled.

  • In run 39281111519, every job passed except docker-e2e shard 1/4: native-config-networks failed with "Fixture command must exit 0" (unrelated; that shard runs only that scenario). In run 39288207970, triggered by the force-push back to the same SHA, shard 1/4, the docker-e2e fan-in, test and all Linux jobs passed. Its macOS jobs were still queued when this was written; they passed in run 39281111519.

Remaining gaps

  • WU12 storage is not covered: dedicated mounts, free-space/inode admission, active-generation holds, interrupted cleanup, recoverable reclamation and archives. Old cache volumes are kept, never reclaimed.
  • "Stale image" means a pinned ID absent locally, not a baked-tool manifest mismatch (no such product mechanism exists yet).
  • Only the hosted Linux Docker runner is qualified; not native VM, Codex Cloud or an authorized disposable QA runner.

Linear: HACK-1143

🤖 Generated with Claude Code

Extend the portable bootstrap fixture (#146/#147) with a hosted
docker-e2e scenario, `portable-bootstrap-negatives`, for HACK-1143. On
the slim path it drives `hack up --detach app` through eleven ordered
stages against a locked-v1 dependency cache with a frozen installer and
an app-scoped encrypted synthetic token:

- missing key, wrong key and an overlay sealed under another key refuse
  before any owned container, network or volume exists;
- two identical bootstraps keep the same consumer, cache and index,
  initialize once and leave no worktree diff;
- a wrong key, a stale pinned image, a lockfile that no longer matches
  its manifest and a corrupted ready marker each refuse without touching
  the running consumer, pulling, rewriting the lockfile or reinstalling;
- a changed lockfile and a new cache generation select new caches and
  keep the previous ones.

Each stage records a value-free receipt (duration, output bytes, tracked
bytes, mounted-cache bytes, cache-volume count, consumer transition).
Pure stage expectations and receipt validation live in
tests/e2e/portable-bootstrap-controls.ts with unit tests. A local test
runs the source CLI on the same fixture against a recording Docker stub
to show the env refusals precede any Compose mutation.

WU12 storage admission, retention and archives remain out of scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi
roodboi force-pushed the wf/f15-headless-negatives-b16 branch from 633a182 to 4babce5 Compare October 19, 2026 16:45
Resolve tests/e2e/run.ts and tests/e2e/README.md by keeping both sides:
the portable-bootstrap-negatives scenario from this branch and the
P01/P02/P07 pattern scenarios merged into next in #336.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

@roodboi
roodboi marked this pull request as ready for review October 20, 2026 17:59
@roodboi
roodboi merged commit b992dd1 into next Oct 20, 2026
31 of 33 checks passed
@roodboi
roodboi deleted the wf/f15-headless-negatives-b16 branch October 20, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant