From 29fc51dd7cd333e056bb4007b782f45fb8dddd48 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 09:52:29 -0300 Subject: [PATCH 01/11] docs(plan): diagnose and plan the CI build time issues (#363, #364, #367, #368) Measurements from ten main runs and local builds, the research notes, the spec, the plan and its two Codex reviews. Co-Authored-By: Claude Fable 5.2 --- .../2026-11-14-ci-build-times/execute-plan.md | 99 ++++++++++ .../PHASE-01-symbol-dump-verification.md | 33 ++++ .../PHASE-02-bundle-codegen-override.md | 34 ++++ .../phases/PHASE-03-ci-workflow.md | 75 ++++++++ .../phases/PHASE-04-shim-timeout-evidence.md | 51 +++++ .../PHASE-05-docs-gates-and-pull-request.md | 71 +++++++ docs/plans/2026-11-14-ci-build-times/plan.md | 129 +++++++++++++ .../2026-11-14-ci-build-times/progress.md | 29 +++ .../research/actions_cache_limits.md | 123 ++++++++++++ .../research/lto_debuginfo_dsym_cost.md | 176 ++++++++++++++++++ .../research/measured_ci_and_local.md | 126 +++++++++++++ .../research/report.md | 76 ++++++++ .../research/rust_cache_behaviour.md | 147 +++++++++++++++ .../research/rust_ci_speedups.md | 173 +++++++++++++++++ .../reviews/plan-review-1.md | 19 ++ .../reviews/plan-review-2.md | 18 ++ docs/plans/2026-11-14-ci-build-times/spec.md | 108 +++++++++++ docs/plans/README.md | 1 + 18 files changed, 1488 insertions(+) create mode 100644 docs/plans/2026-11-14-ci-build-times/execute-plan.md create mode 100644 docs/plans/2026-11-14-ci-build-times/phases/PHASE-01-symbol-dump-verification.md create mode 100644 docs/plans/2026-11-14-ci-build-times/phases/PHASE-02-bundle-codegen-override.md create mode 100644 docs/plans/2026-11-14-ci-build-times/phases/PHASE-03-ci-workflow.md create mode 100644 docs/plans/2026-11-14-ci-build-times/phases/PHASE-04-shim-timeout-evidence.md create mode 100644 docs/plans/2026-11-14-ci-build-times/phases/PHASE-05-docs-gates-and-pull-request.md create mode 100644 docs/plans/2026-11-14-ci-build-times/plan.md create mode 100644 docs/plans/2026-11-14-ci-build-times/progress.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/actions_cache_limits.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/lto_debuginfo_dsym_cost.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/report.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/rust_cache_behaviour.md create mode 100644 docs/plans/2026-11-14-ci-build-times/research/rust_ci_speedups.md create mode 100644 docs/plans/2026-11-14-ci-build-times/reviews/plan-review-1.md create mode 100644 docs/plans/2026-11-14-ci-build-times/reviews/plan-review-2.md create mode 100644 docs/plans/2026-11-14-ci-build-times/spec.md diff --git a/docs/plans/2026-11-14-ci-build-times/execute-plan.md b/docs/plans/2026-11-14-ci-build-times/execute-plan.md new file mode 100644 index 0000000000..b89c0c47b2 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/execute-plan.md @@ -0,0 +1,99 @@ +# Execute plan: CI build times, symbol dump error and Windows arm64 flake + +You are about to execute an approved implementation plan. Read this whole file before doing anything. + +- Plan folder: docs/plans/2026-11-14-ci-build-times +- Spec: docs/plans/2026-11-14-ci-build-times/spec.md (binding authority; the plan argues from it) +- Planned on: Claude Code, 2026-11-14 +- Plan reviewed by: Codex gpt-6.1-sol, effort high +- Base branch: main +- Recommended mode: Inline, because the plan has ten small tasks and only one round with parallel phases that own disjoint files + +## 1. Check the plan + +1. Read `progress.md`, `plan.md`, `spec.md` and every file in `phases/`. + Check that `mstack-receive-review` is in this session's skill list (as `mstack-receive-review` or `:mstack-receive-review`), because every review of this run is processed with it. When it is missing, stop before any work and print the install commands: `claude plugin install mstack-receive-review@mstack` (Claude Code) or `codex plugin add mstack-receive-review@mstack` (Codex), after ` plugin marketplace add mubeda/mstack`, or `npx skills add mubeda/mstack --skill mstack-receive-review -g -a -y`; name `mstack-setup`, and say a new session is needed after the install. +2. If `progress.md` says `Plan status: Draft`, stop. Tell the user the plan is not approved and ask them to approve it, or to finish it with `mstack-plan`. +3. Compute the sha256 of `spec.md` (`sha256sum spec.md` or `shasum -a 256 spec.md`). If its first 12 hex digits differ from the `Spec` line of `progress.md`, the spec changed after planning. Stop and ask the user whether to re-plan or continue on the old plan. +4. When `progress.md` has a `Branch` line, continue on that branch. Tasks marked Closed are never redone, but check first that each one's last logged commit is on the branch (`git merge-base --is-ancestor HEAD`). When it is not, because a workflow closed the task and stopped before integrating it, cherry-pick the logged commits from the logged branch. When that branch is gone, set the task back to In progress and log why. +5. A task marked In progress was interrupted. Read its log lines and `git log` on the branch, then finish it from where the evidence stops. + +## 2. Ask the user first + +Before any other work, ask the user these questions in one message, using the host's question tool (in Claude Code `AskUserQuestion`; in Codex a numbered list in chat, then wait). Skip a question only when the user's opening message already answers it. + +1. How should the plan be executed? + (a) Inline. This session implements every task in order, red-green, then gets one cross-vendor review of the whole branch. + (b) Workflow. In Claude Code, the `mstack-codex-review-workflow` skill (Codex reviews). In Codex, the `mstack-claude-review-workflow` skill (Claude reviews). Parallel writer lanes, a cross-vendor review per lane, one integrated branch. + (c) Codex develops. In Claude Code only, the `mstack-claude-pm-codex-dev` skill: Codex writes each phase, and Claude reviews, validates and commits it. + I recommend (a), because the plan has ten small tasks and only one round with parallel phases that own disjoint files. +2. Which reviewer model and effort? I recommend `gpt-6.1-sol` and `high`. +3. May this run commit each finished task to the branch in `progress.md`'s `Branch` line, or, when there is none, create the branch `plan/ci-build-times` from `main` and commit to it? Pushing and opening a pull request happen only when the user asks. + +## 3. Workflow execution + +1. Load the workflow skill for this host. If it is not installed, give the user its install command and offer inline execution instead. + - Claude Code: `claude plugin marketplace add mubeda/mstack`, then `claude plugin install mstack-codex-review-workflow@mstack`, then start a new session. + - Codex: `codex plugin marketplace add mubeda/mstack`, then `codex plugin add mstack-claude-review-workflow@mstack`, then start a new session. + - Either host, with the skills CLI: ask the user the scope (global or this repository), the method (symlink or copy) and the agents, then run `npx skills add mubeda/mstack --skill -a -y`, adding `-g` for global and `--copy` for copy. + - Choice (c), Claude Code only: `claude plugin marketplace add mubeda/mstack`, then `claude plugin install mstack-claude-pm-codex-dev@mstack`, then start a new session; or with the skills CLI, after the same three questions, `npx skills add mubeda/mstack --skill mstack-claude-pm-codex-dev -a claude-code -y`. It also needs the OpenAI Codex plugin for Claude Code; `mstack-setup` reports and installs it. +2. Give the workflow this plan folder as its work items, and tell it that every review is processed with `mstack-receive-review` without asking the user. It reads the folder as an `mstack-plan` plan: every task is a work item, `progress.md` is its status file, and each task's Skills, Read first and Done when go into the lane brief. +3. The answers to questions 2 and 3 above answer the workflow's own model, effort and commit questions. The workflow still asks its execution mode question, and Workflow mode is recommended when a round has more than one phase. For choice (c), the same two answers answer its reviewer and commit questions; it still asks which developer model and effort to use. + +## 4. Inline execution + +### Rules for the whole run + +1. Work on the branch from question 3. Never work directly on the default branch without the user's consent. +2. Set `Executor: inline` and `Branch: ` in `progress.md`. +3. Keep going without status check-ins. Stop and ask only for: an irreversible or destructive operation, a security-sensitive action, a side effect outside this repository (a push, a publish, a merge), or a plan so broken that every way forward is a guess. When the context or the session is running out, stop at a safe point, make sure `progress.md` is current, write a handoff with `mstack-handoff` when it is installed, and give its path; the next session continues from `progress.md` and the handoff. +4. When the plan is wrong in a small way (a renamed function, a moved file), make the smallest change that still satisfies the spec, and log it in `progress.md` as `Ruling: , because . Cost if wrong: .` +5. Never claim a task is done without fresh evidence: run its Done when command in this session, read the whole output and the exit code, and copy the evidence into the log. +6. This session's model does every task, so a task's Tier line does not change it. The tier limits what you delegate: a read-only helper (a search, a log read) runs on the cheapest model this host offers, set explicitly, and no helper runs on a model above the task's tier. In Claude Code that is `Agent` with the model and effort of haiku at low effort for a cheap helper and of sonnet at medium effort for a standard one. In Codex, pass them as `-m` and `model_reasoning_effort` on the helper run: the cheap-tier Codex model at low effort for a cheap helper, the standard-tier Codex model at medium effort for a standard one. A helper started without them runs on this session's model. + +### Per task + +Pick the next task by dependencies. A task is ready when every task in its Depends on line is Closed and every phase its phase depends on is Closed. Among ready tasks, take the lowest phase, then the lowest ID. Skip Closed tasks and tasks marked removed or dropped. A task that depends on a removed or dropped task stays Open, and the final report lists it as blocked. + +1. Set the task to In progress in `progress.md` and add a log line. +2. Load every skill the phase and the task list (in Claude Code with the Skill tool; in Codex by reading the skill's SKILL.md). The plan names skills without a plugin prefix, so use whichever form this host lists, `:` or ``. For a skill this host does not have, follow the task's steps without it and note that in the log. +3. Read every file and source in the phase's and the task's Read first. +4. Follow the steps in order. After each command, compare the output with its `Expected:` line. + - Red-green tasks: see the test fail for the expected reason before writing the code, then see it pass. Do the "prove the test can fail" step, and always restore the code afterwards. + - The output does not match because the code is wrong: debug it before changing anything (load `mstack-diagnose` when it is installed). Never patch the symptom. + - The output does not match because the plan is wrong: apply rule 4. +5. Run the task's Done when command and the phase's lint and format commands. +6. Commit the task's files together with `progress.md`, where the task is now Closed and the log line carries the command, the result and the commit range. + +After a phase's last task, run its Checkpoint and log the result. A failing checkpoint reopens the task that owns the failure. + +### After the last task + +1. Run every check in `plan.md` › End-to-end verification and log each result. +2. Get one cross-vendor review of the whole branch, read-only, with the model and effort from question 2. Write its output outside the repository: `` below is the session's scratch folder when the host has one, otherwise `${TMPDIR:-/tmp}/mstack-plan-ci-build-times`, created first. Put a path in double quotes in these commands when it contains a space. + - Claude Code (Codex reviews, through the `codex` skill of the skill-codex plugin; timeout at least 600 seconds for `high`, 1200 for `xhigh`, 1800 for `max`): + + Write this prompt to `/review-prompt.md` with your file tool, never with `echo` or a heredoc, so no shell expands anything in it: + + ```text + Read-only code review. Do not edit or run anything that changes files. Review branch against main (git diff main...HEAD). The plan is in docs/plans/2026-11-14-ci-build-times and the spec it implements is docs/plans/2026-11-14-ci-build-times/spec.md. Check that every requirement in the spec is met, and check correctness, regressions, missing tests and scope. Report each finding with file, line, the failing scenario and how to reproduce it. No findings is a valid result. + ``` + + ```bash + codex exec -m --config model_reasoning_effort="" --sandbox read-only -C - \ + < /review-prompt.md 2> /codex-review.err > /codex-review.md + ``` + + - Codex (Claude reviews, through the `claude-review` skill of the agent-bridges `claude` plugin; it needs network and writes under the home folder, so run it as an approved command): + + ```bash + node "/scripts/claude-companion.mjs" review --wait \ + --cwd --base main --scope branch \ + --model --effort /claude-review.md 2> /claude-review.err + ``` + + This review reads the diff, not the spec, so check every requirement against the spec yourself. + + - An empty output or a non-zero exit is never a clean review. Read the end of the `.err` file, fix the cause (a rejected model: use the other model offered in question 2), and rerun once. If it still fails, or this host's reviewer is not installed, warn the user that the review will be same-vendor, and run the same review prompt through a fresh, read-only subagent of this host (in Claude Code the `Agent` tool with the model and effort of opus at medium effort; in Codex a separate `codex exec -m --config model_reasoning_effort="" --sandbox read-only -C - < /review-prompt.md`, where those two are the model and effort of the most capable Codex model at medium effort, with the prompt above written to that file, as an approved command because it needs network). Log it as a same-vendor review, and say so in the final report. Never wait on the user for it. +3. Process the review with the `mstack-receive-review` skill, without asking the user anything. Verify every finding against the code and reproduce it before accepting it. Fix each real finding with a test you saw fail and then pass. Send an unclear finding back to the reviewer once. Reject a finding that is wrong, cannot be verified, or contradicts a decision the user already made, with the reason. Log every finding and its outcome in `progress.md`. +4. Report to the user: the goal and the outcome in the first line, then the final `progress.md` table, the end-to-end results, every Ruling, every review finding with what happened to it (fixed, rejected and why, deferred), and what was not run and why. Offer to open a pull request. Do not push without the user's request. diff --git a/docs/plans/2026-11-14-ci-build-times/phases/PHASE-01-symbol-dump-verification.md b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-01-symbol-dump-verification.md new file mode 100644 index 0000000000..7efe1042fc --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-01-symbol-dump-verification.md @@ -0,0 +1,33 @@ +# Phase 1: Symbol dump verification + +- Goal: `scripts/dump-symbols.ts` proves the stored file complete, or fails, whenever `dump_syms` logs an error, with tests. +- Round: 1 +- Depends on: none +- Owns: `scripts/dump-symbols.ts`, `scripts/dump-symbols.test.ts` +- Must not touch: `scripts/build-desktop-artifact.ts`, `.github/workflows/` +- Skills: tdd (keeps the failing test ahead of the script change) +- Read first: `research/measured_ci_and_local.md` section 4 (why the error is harmless when it comes from the dSYM, and when it is not) +- Checkpoint: `vp test run scripts/dump-symbols.test.ts` passes; a dump of an already built local binary with its dSYM that logs the CFI error prints the "all stored" line and exits 0. The binary comes from the 2026-11-14 measurement builds, so this phase starts no build. + +## Tasks + +### Task 1.1: Attribute a dump_syms error and verify the stored unwind records + +- Satisfies: R1 (error from the dSYM, error from the binary, count mismatch, error without a dSYM, no error) +- Depends on: none +- Files: modify `scripts/dump-symbols.ts`; test `scripts/dump-symbols.test.ts` +- Interfaces: produces `countStackRecords(sym: string | Buffer): number`; `dumpSymbols(input, dumpSyms?)` keeps its signature +- Skills: tdd (red-green on the five scenarios) +- Read first: `scripts/dump-symbols.ts` (`assertUsableSymbols`, `hasRecord`, and how `dumpSymbols` stages inputs and spawns the tool) +- Tier: standard +- TDD: red-green +- Done when: `vp test run scripts/dump-symbols.test.ts` exits 0 with the five new cases passing + +Steps + +1. [ ] Write failing tests in `scripts/dump-symbols.test.ts` that drive `dumpSymbols` with a fake `dump_syms` executable (a Node script that writes a `.sym` and, by argument count, an `[ERROR] CFI: CfiError { kind: BadDebugInfo, source: Some(UnknownVersion(0)) }` line to standard error): error with two inputs and a clean binary-only run with equal `STACK` counts succeeds and prints `dump_syms reported an error for the dSYM; the binary's 3 STACK records are all stored`; error again in the binary-only run fails with `dump_syms reported an error while reading the unwind tables of the binary`; counts 3 against 5 fail with `the stored symbols hold 3 STACK records, the binary 5`; error without `--dsym` fails with `dump_syms reported an error`; no error runs the fake once (it records its invocations in a file) and succeeds. Also `countStackRecords` counts `STACK` lines at the start of the file and after a newline, given a string and a `Buffer`. Expected: six new cases in the file. +2. [ ] Run `vp test run scripts/dump-symbols.test.ts`. Expected: FAIL, `countStackRecords` is not exported and the error cases pass where they must fail. +3. [ ] Implement: spawn with standard error piped, write it through to the script's standard error, and when it holds `[ERROR]` apply the rule above (the binary-only run reuses the staged binary and writes to a second file in the work folder, without `--store`). Expected: the script type-checks. +4. [ ] Run `vp test run scripts/dump-symbols.test.ts`. Expected: PASS. +5. [ ] Prove the test can fail: skip the binary-only run, run the command. Expected: FAIL on the binary-error and mismatch cases. Restore and run again. Expected: PASS. +6. [ ] Run `vp check scripts/dump-symbols.ts scripts/dump-symbols.test.ts`. Expected: no errors. diff --git a/docs/plans/2026-11-14-ci-build-times/phases/PHASE-02-bundle-codegen-override.md b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-02-bundle-codegen-override.md new file mode 100644 index 0000000000..0f73615564 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-02-bundle-codegen-override.md @@ -0,0 +1,34 @@ +# Phase 2: Bundle codegen override + +- Goal: `scripts/build-desktop-artifact.ts` can build the two workspace crates with a chosen number of codegen units, with tests, and builds exactly as today without the flag. +- Round: 1 +- Depends on: none +- Owns: `scripts/build-desktop-artifact.ts`, `scripts/build-desktop-artifact.test.ts` +- Must not touch: `scripts/dump-symbols.ts`, `.github/workflows/` +- Skills: tdd (keeps the failing test ahead of the script change) +- Read first: `research/measured_ci_and_local.md` section 3 (what 16 codegen units buy and cost) +- Checkpoint: `vp test run scripts/build-desktop-artifact.test.ts` passes; through the real wrapper, Cargo receives the two `--config` arguments (see Task 2.1 step 6), and a direct Cargo rebuild with them compiled only `bibcode-server` and `bibcode-desktop` (recorded in `research/measured_ci_and_local.md` section 3). + +## Tasks + +### Task 2.1: Add --workspace-codegen-units to the bundle script + +- Satisfies: R3 (flag given, flag absent, invalid value) +- Depends on: none +- Files: modify `scripts/build-desktop-artifact.ts`; test `scripts/build-desktop-artifact.test.ts` +- Interfaces: produces the CLI flag `--workspace-codegen-units ` and the input field `workspaceCodegenUnits?: number`; the build command gains `-- --config profile.release.package.bibcode-server.codegen-units= --config profile.release.package.bibcode-desktop.codegen-units=` after its existing arguments +- Skills: tdd (red-green on the three scenarios) +- Read first: `scripts/build-desktop-artifact.ts` (`tauriCommand`, the argument parser and its `symbols` option) +- Tier: standard +- TDD: red-green +- Done when: `vp test run scripts/build-desktop-artifact.test.ts` exits 0 with the new cases passing + +Steps + +1. [ ] Write failing tests: a plan with `workspaceCodegenUnits: 16` has a build command whose arguments end with `--`, `--config`, `profile.release.package.bibcode-server.codegen-units=16`, `--config`, `profile.release.package.bibcode-desktop.codegen-units=16` and a bundle command without them; a plan without it equals today's; parsing `--workspace-codegen-units 0`, `257`, `1.5` or `abc` fails with `--workspace-codegen-units must be a whole number from 1 to 256.` +2. [ ] Run `vp test run scripts/build-desktop-artifact.test.ts`. Expected: FAIL on the three new cases. +3. [ ] Implement: the parser option (string, validated to an integer 1 to 256), the input field, and the extra arguments in `tauriCommand("build")` only. +4. [ ] Run `vp test run scripts/build-desktop-artifact.test.ts`. Expected: PASS. +5. [ ] Prove the test can fail: drop the `--` separator, run the command. Expected: FAIL. Restore and run again. Expected: PASS. +6. [ ] Prove the complete caller: start `node scripts/build-desktop-artifact.ts --platform mac --target dmg --arch arm64 --output-dir --symbols --workspace-codegen-units 16` as the leader of its own process group (it builds into this checkout's `target/aarch64-apple-darwin`, which nothing else uses), poll `ps` for up to 160 seconds for a `cargo build` process whose arguments hold `codegen-units=16`, then send `TERM` to the process group and check that no process of the group is left. Expected: the command line reads `cargo build --config profile.release.package.bibcode-server.codegen-units=16 --config profile.release.package.bibcode-desktop.codegen-units=16 ...`, and the group is empty afterwards. +7. [ ] Run `vp check scripts/build-desktop-artifact.ts scripts/build-desktop-artifact.test.ts`. Expected: no errors. diff --git a/docs/plans/2026-11-14-ci-build-times/phases/PHASE-03-ci-workflow.md b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-03-ci-workflow.md new file mode 100644 index 0000000000..4adb9a66b2 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-03-ci-workflow.md @@ -0,0 +1,75 @@ +# Phase 3: CI workflow + +- Goal: CI runs the Windows server tests in their own job, builds bundles with the override and a heartbeat, and stops pull requests saving pnpm entries, each pinned by a contract test. +- Round: 2 +- Depends on: 2 +- Owns: `.github/workflows/*.yml`, `scripts/ci-platform-contract.test.ts` +- Must not touch: `scripts/dump-symbols.ts`, `scripts/build-desktop-artifact.ts`, `docs/` +- Skills: tdd (contract test first); issue-delivery (checkpoint rules before any CI dispatch) +- Read first: `docs/operations/ci.md` (what each job promises today); `research/measured_ci_and_local.md` sections 1 and 2 (the baselines the new run is compared with) +- Checkpoint: `vp test run scripts/ci-platform-contract.test.ts` passes, and the workflow files parse. + +## Tasks + +### Task 3.1: Move the Windows server tests into server_tests_windows + +- Satisfies: R2 (native rows without the steps, new job with them) +- Depends on: none +- Files: modify `.github/workflows/ci.yml`; test `scripts/ci-platform-contract.test.ts` +- Interfaces: produces the job id `server_tests_windows` with rows labelled `Windows x64` (`windows-2025`) and `Windows arm64` (`windows-11-vs2026-arm`), job name `Server tests - ${{ matrix.label }}` +- Skills: tdd (contract case first) +- Read first: the `server_tests_macos` job in `.github/workflows/ci.yml` (the shape to mirror) +- Tier: standard +- TDD: red-green +- Done when: `vp test run scripts/ci-platform-contract.test.ts` exits 0 with the new case passing + +Steps + +1. [ ] Write a failing contract case: `server_tests_windows` exists, has no `if`, `timeout-minutes` 90, a rust-cache step with `shared-key: native-desktop` and `save-if: "false"`, the five step names in the spec's order, no `-j 2` in any command, the review prompt step's filter `-- review_prompt_shim_ shim_timeout_`, the `counted` helper in the host metrics step; and `native_desktop` has no step whose name ends in `(Windows)` and no `cargo check -p bibcode-server` or `cargo test -p bibcode-server` command. +2. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: FAIL, the job is missing. +3. [ ] Edit `ci.yml`: add the job after `server_tests_macos` (Checkout, Setup Vite+ with `node-version-file: package.json`, `run-install: false` and no cache, Setup Rust, the restore-only cache step, then the five steps moved from `native_desktop` without their `if` and without `-j 2`); delete them from the shared native steps; update any existing contract cases that looked for them in the native rows. +4. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: PASS. +5. [ ] Prove the test can fail: put `-j 2` back on one command, run. Expected: FAIL. Restore, run. Expected: PASS. +6. [ ] Run `vp check .github/workflows/ci.yml scripts/ci-platform-contract.test.ts`. Expected: no errors. + +### Task 3.2: Pass the override and record a heartbeat in the bundle step + +- Satisfies: R3 (CI passes 16, Release does not), R4 (heartbeat and notice) +- Depends on: 2.1, 3.1 +- Files: modify `.github/workflows/ci.yml`; test `scripts/ci-platform-contract.test.ts` +- Interfaces: consumes `--workspace-codegen-units ` from Task 2.1; produces heartbeat lines `heartbeat s ...` and a notice titled `Bundle build` +- Skills: tdd (contract case first) +- Read first: the `Crash the smoke binary` step in `ci.yml` (the existing `trap` and `::notice::` pattern) +- Tier: standard +- TDD: red-green +- Done when: `vp test run scripts/ci-platform-contract.test.ts` exits 0 with the new case passing + +Steps + +1. [ ] Write a failing contract case: the bundle step of `native_desktop` uses `shell: bash`, contains `--workspace-codegen-units 16`, starts a background loop that sleeps 60 seconds, installs a `trap` that stops the loop on exit and prints `::notice title=Bundle build::`, reads resident memory with `ps`, and does not use `set +e` or `|| true` on the build command itself; and `release.yml` contains no `--workspace-codegen-units`. +2. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: FAIL. +3. [ ] Edit the step: a bash script that starts the loop (elapsed seconds; on macOS and Linux `ps -Ao rss=,comm=` filtered to `rustc`, `ld`, `lld`, `dsymutil`, plus `vm_stat` and `sysctl vm.swapusage` on macOS and `/proc/meminfo` on Linux; elapsed only on Windows), tracks the peak single-process resident memory in a file, traps EXIT to kill the loop and print the notice with seconds and peak, and then runs the existing command with the new flag. Every heartbeat command ends in `|| true`. +4. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: PASS. +5. [ ] Run the step's script locally three times with the build command replaced: by `sleep 130` (expected: two heartbeat lines with memory values, the notice, exit 0, no loop process left); by `sh -c 'sleep 2; exit 7'` (expected: the notice, exit 7, no loop process left); and with `PATH` reduced so `ps` is missing (expected: heartbeat lines without memory values, exit 0). Expected: all three probes behave as listed. +6. [ ] Prove the test can fail: remove the flag from the step, run the contract test. Expected: FAIL. Restore, run. Expected: PASS. + +### Task 3.3: Save pnpm store entries only on main + +- Satisfies: R5 +- Depends on: 3.2 +- Files: modify `.github/workflows/*.yml`; test `scripts/ci-platform-contract.test.ts` +- Interfaces: none +- Skills: tdd (contract case first) +- Read first: `research/actions_cache_limits.md` (pull request entries count against the same limit) +- Tier: cheap +- TDD: red-green +- Done when: `vp test run scripts/ci-platform-contract.test.ts` exits 0 with the new case passing + +Steps + +1. [ ] Write a failing contract case that reads every workflow file and requires, for each `voidzero-dev/setup-vp` step with `cache: true`, `cache-save` equal to `${{ github.ref == 'refs/heads/main' }}`. +2. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: FAIL, listing the steps. +3. [ ] Add the input to each such step. +4. [ ] Run `vp test run scripts/ci-platform-contract.test.ts`. Expected: PASS. +5. [ ] Prove the test can fail: remove the input from one step, run. Expected: FAIL. Restore, run. Expected: PASS. +6. [ ] Run `vp test run scripts`. Expected: PASS, no other workflow contract broken. diff --git a/docs/plans/2026-11-14-ci-build-times/phases/PHASE-04-shim-timeout-evidence.md b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-04-shim-timeout-evidence.md new file mode 100644 index 0000000000..f0ec855583 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-04-shim-timeout-evidence.md @@ -0,0 +1,51 @@ +# Phase 4: Shim timeout evidence + +- Goal: a shim test timeout prints what the terminal showed, and a bounded load test on Windows arm64 records a failure rate for #367. +- Round: 1 +- Depends on: none +- Owns: `apps/server/tests/windows_terminal_shims.rs` +- Must not touch: `apps/server/src/`, `.github/workflows/`, `scripts/` +- Skills: rust-best-practices (idiomatic error text and test helpers); mstack-diagnose (the flaky branch sets the run count rules) +- Read first: `research/measured_ci_and_local.md` section 5 (the failing attempt's timeline); the Windows and Linux labs notes in `docs/testing/` for the VM recipe +- Checkpoint: the Windows VM compiles and passes `cargo test -p bibcode-server --test windows_terminal_shims`, and the new self-check test shows the timeout message. + +## Tasks + +### Task 4.1: Print the collected output and exit state on timeout + +- Satisfies: R6 +- Depends on: none +- Files: modify `apps/server/tests/windows_terminal_shims.rs` +- Interfaces: `assert_shim_output` keeps its signature and its checks after the marker (exit within 10 seconds, exit code 0); produces `wait_for_marker(output, exit, marker, deadline) -> Result` whose error text is `timed out after s waiting for ; output=, exit=` +- Skills: rust-best-practices (no unwrap that hides the evidence) +- Read first: `apps/server/tests/windows_terminal_shims.rs` lines 133 to 187 +- Tier: standard +- TDD: red-green +- Done when: in the Windows VM, `node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims` exits 0, including the new test + +Steps + +1. [ ] Write the failing test `shim_timeout_reports_the_collected_output`: spawn a `.cmd` shim that echoes `BEFORE` and then waits 15 seconds, and call `wait_for_marker` on its output and exit subscriptions with a 2-second deadline; assert the error contains `timed out`, `BEFORE` and `exit=None`, then kill the terminal. +2. [ ] Run the test target in the Windows VM. Expected: FAIL to compile, `wait_for_marker` does not exist. +3. [ ] Implement: move only the wait loop of `assert_shim_output` into `wait_for_marker`, keeping the text outside the timed future so it survives the timeout, and return the described error on `Elapsed`; `assert_shim_output` calls it with 10 seconds, panics with the error, and keeps its marker assertion, its 10-second exit wait and its exit code 0 assertion unchanged. +4. [ ] Run the test target in the Windows VM. Expected: PASS, 24 tests; then `node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets` with warnings denied as CI sets them, and `node scripts/run-msvc.mjs cargo clippy -p bibcode-server --test windows_terminal_shims -- -D warnings`. Expected: no warnings. If the VM cannot run them, the pull request's `server_tests_windows` job is the compile and run proof, and Clippy on this Windows-only file is reported as not run. +5. [ ] Prove the test can fail: drop the collected text from the error, run. Expected: FAIL. Restore, run. Expected: PASS. +6. [ ] Run `cargo fmt --all --check` on the host. Expected: no changes. + +### Task 4.2: Measure the #367 rate under load in the Windows arm64 VM + +- Satisfies: R7 +- Depends on: 4.1 +- Files: none +- Interfaces: produces a result line `k of 30 failed`, returned to the executor for `progress.md` and published by Task 5.3 +- Skills: mstack-diagnose (flaky branch: fixed count, no early stop) +- Read first: `research/measured_ci_and_local.md` section 5 +- Tier: standard +- TDD: exempt (a measurement run, no behavior changes) +- Done when: 30 runs are counted and the result, with any failure's printed output, is recorded in `progress.md`; or the VM cannot run them, which is recorded as `not run: `, closes this task, and leaves R7 unaccepted and #367 open + +Steps + +1. [ ] Start four busy processes in the VM (PowerShell jobs running `while ($true) {}`, their ids written to a file), then run `cargo test -p bibcode-server --test windows_terminal_shims -- review_prompt_shim_` 30 times without stopping at a failure, counting non-zero exits. Expected: a count `k of 30`. +2. [ ] Stop the busy processes by the recorded ids, in a `finally` block so a failed run still stops them. Expected: none left running. +3. [ ] Report `k of 30`, the command, the VM's core count and any failure's output to the executor. Expected: one log line in `progress.md`. diff --git a/docs/plans/2026-11-14-ci-build-times/phases/PHASE-05-docs-gates-and-pull-request.md b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-05-docs-gates-and-pull-request.md new file mode 100644 index 0000000000..6553dee48f --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/phases/PHASE-05-docs-gates-and-pull-request.md @@ -0,0 +1,71 @@ +# Phase 5: Docs, gates and pull request + +- Goal: the living documents state the measured causes and the new CI shape, every gate passes, and the pull request carries one measured `ci:full` run. +- Round: 3 +- Depends on: 1, 2, 3, 4 +- Owns: `docs/operations/ci.md`, `docs/operations/release.md`, `docs/testing/`, `docs/plans/README.md` +- Must not touch: source files owned by phases 1 to 4, except to apply review findings +- Skills: issue-delivery (checkpoint before the CI dispatch, progress by accepted requirements); mstack-receive-review (processes the Codex review of the branch) +- Read first: `AGENTS.md` (Task Completion Requirements, Testing Runbook Maintenance, Sensitive Data) +- Checkpoint: the pull request is open with green checks, a table of measured step times against the baselines, and an accept or reject line per requirement. + +## Tasks + +### Task 5.1: Update the living documents + +- Satisfies: R8 +- Depends on: 1.1, 2.1, 3.1, 3.2, 3.3, 4.1 +- Files: modify `docs/operations/ci.md`; modify `docs/operations/release.md`; modify the Windows runbook under `docs/testing/`; modify `docs/plans/README.md` +- Interfaces: none +- Skills: none +- Read first: `docs/operations/ci.md` (Native desktop, Server tests, Rust cache budget sections) +- Tier: cheap +- TDD: exempt (documentation) +- Done when: `vp run check:markdown-links` exits 0 and `rg -n "without a Rust cache" docs/operations/ci.md` prints nothing that explains a row's time + +Steps + +1. [ ] `ci.md`: add `server_tests_windows` to the label table and the Jobs list; rewrite the Native desktop paragraph for the override (what CI still proves, what only Release proves) and the heartbeat; state the measured budget (about 17 GB wanted against 10 GB, pull requests no longer save pnpm entries, the limit can be raised in repository settings); replace "without a cache" explanations of row times with the compile counts. Expected: the file names no execution-specific run IDs outside the dated measurements it already keeps. +2. [ ] `release.md` Symbol assets: say that a `dump_syms` CFI error from a dSYM is expected, where the unwind records come from, and what the script checks. Expected: one short paragraph. +3. [ ] Windows runbook: list the server tests under their own job. Expected: commands match `ci.yml`. +4. [ ] `docs/plans/README.md`: add this folder to the plan folder list. Expected: one line. +5. [ ] Run `vp run check:markdown-links`. Expected: PASS. + +### Task 5.2: Local gates, review and pull request + +- Satisfies: support (5.3, which accepts the requirements on the pull request's run) +- Depends on: 5.1, 4.2 +- Files: none +- Interfaces: produces the pull request and its first CI run +- Skills: issue-delivery (delivery checkpoint); mstack-receive-review (review findings) +- Read first: `docs/skills/issue-delivery/SKILL.md` +- Tier: standard +- TDD: exempt (gates and a pull request) +- Done when: the pull request is open with the `ci:full` label and one CI run started on its head commit + +Steps + +1. [ ] Run `vp check`, `vp run typecheck`, `vp test run scripts`, `cargo fmt --all --check`. Expected: all pass. +2. [ ] Get the cross-vendor review of the branch and process it with `mstack-receive-review`. Expected: every finding fixed with a test or rejected with a reason. +3. [ ] Review `git diff main...HEAD` and `git status --short` for unintended files and sensitive data. Expected: none. +4. [ ] Write the delivery checkpoint and run `node scripts/check-agent-delivery.mjs `. Expected: exit 0. +5. [ ] Push the branch and open the pull request with the `ci:full` label already on it when the measured commit is pushed, then bind it. Expected: one CI run with all rows. + +### Task 5.3: Accept the requirements on the measured run + +- Satisfies: R2 (row times), R3 (dependencies reused in CI), R4 (the recorded macOS measurement), R7 (issue comment) +- Depends on: 5.2 +- Files: none +- Interfaces: consumes the `k of 30` line from Task 4.2 +- Skills: issue-delivery (progress by accepted requirements) +- Read first: `research/measured_ci_and_local.md` sections 1 and 2 (the baselines) +- Tier: standard +- TDD: exempt (reading a CI run and writing its numbers down) +- Done when: `gh pr checks` shows every check green on the final commit, the pull request body holds the table, and every Must requirement is accepted or covered by a fallback the spec names (D3, D4); R7 may stay unaccepted with its reason + +Steps + +1. [ ] When the run ends, record each native row's step times, the Windows server test job times and the macOS heartbeat (peak memory, swap, build seconds) in the pull request body, next to the baselines. `progress.md` is not edited after the measured commit, because a new commit starts a new run; the pull request is the record. Expected: the table is in the pull request. +2. [ ] Decide R2: both Windows native rows under 89 and 103 minutes. Decide R3 from the controlled local comparison already recorded in `research/measured_ci_and_local.md` section 3 (same target, only the override changed, only the two workspace crates compiled) and from Task 2.1 step 6; from the run, record each row's bundle build seconds against its baseline as an observation, since a cache restore on the day is not guaranteed. Expected: one accept or reject line each. +3. [ ] Apply spec D3 if the macOS arm64 bundle build took more than 81 minutes: remove the override from that row, push, and repeat steps 1 and 2 for the new run. Apply spec D4's fallback if the Windows server test job ran out of memory: restore `-j 2`, change the contract case to require it, and record the reason in the workflow comment and the spec. Expected: the final commit's run is the one recorded. +4. [ ] Comment on #367 with the `k of 30` result and any failure output; comment on #363 and #368 with the measured numbers. Expected: three comments, no private data. diff --git a/docs/plans/2026-11-14-ci-build-times/plan.md b/docs/plans/2026-11-14-ci-build-times/plan.md new file mode 100644 index 0000000000..a4b7954954 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/plan.md @@ -0,0 +1,129 @@ +# CI build times, symbol dump error and Windows arm64 flake + +Status: Approved 2026-11-14 (approved under the user's standing pre-approval of 2026-10-30) +Spec: spec.md (Status line: Approved 2026-11-14, sha256 be1cd41eb716) +Repository: this repository +Base: main + +## Goal + +- Windows native rows no longer compile the server for tests, and CI bundle builds compile the two workspace crates on several cores. +- A `dump_syms` error is either proven harmless or fails the dump. +- One `ci:full` run records where macOS bundle time and memory go, and the next #367 failure prints the terminal output. + +## Global constraints + +- Release keeps the shipped profile: `release.yml` never passes `--workspace-codegen-units` (spec D2, R3). +- The override is a per-package Cargo `--config`, never a `CARGO_PROFILE_RELEASE_*` variable, so dependency fingerprints stay the ones Release restores (spec D2). +- `server_tests_windows`: `timeout-minutes: 90`, `shared-key: native-desktop`, `save-if: "false"`, no `-j 2` (spec R2). +- Codegen units accepted: whole numbers 1 to 256; CI passes 16 (spec R3). +- Heartbeat interval 60 seconds; it never fails the step (spec R4). +- `cache-save: ${{ github.ref == 'refs/heads/main' }}` on every `setup-vp` step with `cache: true` (spec R5). +- Load test: 30 runs, four busy processes, count fixed in advance (spec R7). +- The repository is public: no hosts, names or machine paths in any file, commit or pull request text (AGENTS.md, Sensitive Data). + +## Review focus + +1. A `dump_syms` error must be attributed to an input before it is called harmless: pinned by Task 1.1's binary-only rerun cases, including the count mismatch. +2. The override reaching `release.yml` by a shared YAML alias: pinned by Task 3.2's contract case. +3. The new Windows job skipping tests silently when a filter matches nothing: the `counted` helper moves with the host metrics step, pinned by Task 3.1. +4. The heartbeat's background loop outliving a failed build, or hiding its exit code: pinned by Task 3.2's failed-child probe. +5. The shim helper losing its exit code check when the wait loop moves: pinned by Task 4.1, which moves only the loop. + +## Out of scope + +- Splitting `bibcode-server` into several crates: a design of its own. +- Changing the shipped release profile: needs a runtime measurement and the user's decision. +- Raising the Actions cache limit: the user's account setting. +- sccache, third-party runners and cache backends: no measurement supports them here. +- A larger timeout or serial runs for the #367 tests without evidence. + +## Sources + +- Repository rules: `AGENTS.md` (delivery discipline, sensitive data, testing runbook maintenance, completion requirements). +- Repository docs: `docs/operations/ci.md` (job descriptions and budgets this plan changes), `docs/operations/release.md` (symbol assets), `docs/testing/` (Windows runbook), `docs/skills/issue-delivery/SKILL.md` (checkpoint before a CI dispatch). +- Evidence: `research/measured_ci_and_local.md` (CI logs and local builds of 2026-11-14). +- External: `research/report.md` and the four notes beside it (GitHub cache limits, rust-cache behaviour, LTO and dSYM cost, CI speedups). + +## Stack and commands + +- Scripts and workflow contracts (TypeScript, Vitest through Vite+): `vp test run scripts/`; whole folder `vp test run scripts`. +- Lint, format, types: `vp check`, `vp run typecheck`, `vp run check:markdown-links`. +- Rust: `cargo fmt --all --check`; Windows-only test target compiled and run in the Windows VM with `node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims`. +- Ran once on 2026-11-14: `vp run --filter @bibcode/web build` (26 s) and three release builds of `bibcode-desktop`. + +## Phases + +| Phase | File | Goal | Round | Depends on | Tier | +| ----- | ---------------------------------------------- | ------------------------------------------------------------------------ | ----- | ---------- | -------- | +| 1 | phases/PHASE-01-symbol-dump-verification.md | A logged `dump_syms` error is verified or fails | 1 | none | standard | +| 2 | phases/PHASE-02-bundle-codegen-override.md | The bundle script can build the workspace crates with more codegen units | 1 | none | standard | +| 3 | phases/PHASE-03-ci-workflow.md | CI uses the new job, the override, the heartbeat and the cache-save rule | 2 | 2 | standard | +| 4 | phases/PHASE-04-shim-timeout-evidence.md | A shim timeout prints its evidence and a load test records a rate | 1 | none | standard | +| 5 | phases/PHASE-05-docs-gates-and-pull-request.md | Docs match, gates pass, the pull request carries measured CI numbers | 3 | 1, 2, 3, 4 | standard | + +Phases in the same round run in parallel. A phase depends only on phases in earlier rounds. + +## File map + +| Path or external change | Action | Phase | Verification | +| ----------------------------------------------------------------- | -------- | ----- | ---------------------------------------------------- | +| `scripts/dump-symbols.ts` | modify | 1 | `vp test run scripts/dump-symbols.test.ts` | +| `scripts/dump-symbols.test.ts` | modify | 1 | same | +| `scripts/build-desktop-artifact.ts` | modify | 2 | `vp test run scripts/build-desktop-artifact.test.ts` | +| `scripts/build-desktop-artifact.test.ts` | modify | 2 | same | +| `.github/workflows/ci.yml` | modify | 3 | `vp test run scripts/ci-platform-contract.test.ts` | +| `.github/workflows/*.yml` other than `ci.yml` (`cache-save` only) | modify | 3 | same | +| `scripts/ci-platform-contract.test.ts` | modify | 3 | same | +| `apps/server/tests/windows_terminal_shims.rs` | modify | 4 | Windows VM run | +| `docs/operations/ci.md` | modify | 5 | `vp run check:markdown-links` | +| `docs/operations/release.md` | modify | 5 | same | +| `docs/testing/` Windows runbook | modify | 5 | same | +| Pull request, `ci:full` label, issue comments | external | 5 | `gh pr checks` | + +Shared files reserved for the executor: `progress.md`, `docs/plans/README.md`. +Parallel conflicts: round 1 phases own disjoint source files. Their local build and dump checkpoints share the machine, so they run one after another, and measurement results go to the executor, who alone writes `progress.md`. + +## Traceability + +| Requirement | Tasks | +| ----------- | ------------- | +| R1 | 1.1 | +| R2 | 3.1, 5.3 | +| R3 | 2.1, 3.2, 5.3 | +| R4 | 3.2, 5.3 | +| R5 | 3.3 | +| R6 | 4.1 | +| R7 | 4.2, 5.3 | +| R8 | 5.1 | + +## End-to-end verification + +- `vp test run scripts` passes with the new cases. +- `vp check`, `vp run typecheck`, `vp run check:markdown-links`, `cargo fmt --all --check` pass. +- The Windows VM compiles and runs the shim tests. +- One `ci:full` pull request run is green, and the pull request records each row's step times and the macOS heartbeat against `research/measured_ci_and_local.md`. + +## Risks + +- 16 codegen units raise peak compiler memory (12 GB locally) and may slow the 7 GB macOS arm64 row -> the heartbeat measures it and spec D3 removes the override from that row in the same pull request. +- Without `-j 2` a Windows runner may run out of memory in the server test build -> the 16 GB runner and the first pull request run decide; spec D4 allows restoring the flag. +- The new job compiles dependencies again when the cache misses -> about 6 minutes, measured in the bundle step. +- The Windows VM is unavailable -> Task 4.1's proof becomes the pull request's `server_tests_windows` job, whose filter runs the new test, and Task 4.2 is recorded as not run. + +## Decisions + +- P1. Inline execution in the planning session. Alternatives: a workflow with writer lanes (five small phases do not repay it). Decided by: agent's call; the user asked for the pull request in the same request. +- P2. The heartbeat is inline shell in the workflow step, not a new script. Alternatives: a Node wrapper (more code for one caller). Decided by: agent's call. +- P3. Skills were named from the session's skill list without a `mstack-find-skills` search, to keep the planning session short. A task may run without a specialist skill. Decided by: agent's call. +- P4. The spec was written in the planning session from the four issues and the diagnosis, because the user asked for the diagnosis and the plan in one request. Decided by: agent's call. + +## Appendix: questions record + +- 2026-11-14 Evidence: `research/measured_ci_and_local.md` section 1 / Decision: the cache is not the main cost; plan the compile count and codegen units first. +- 2026-11-14 Evidence: local dump of a second build reproduced the CFI error with identical `STACK` records / Decision: verify, do not fail on the log line alone. +- 2026-11-14 Evidence: plan review 1 (Codex) showed a tail-address check proves nothing about holes / Decision: R1 attributes the error with a binary-only rerun instead. +- 2026-11-14 Evidence: `setup-vp` `action.yml` at the pinned commit has a `cache-save` input / Decision: R5 uses it. +- 2026-11-14 Evidence: Cargo rebuilt only the two workspace crates under the per-package `--config` override / Decision: D2 uses `--config`. +- 2026-11-14 Evidence: plan review 2 (Codex) / Decision: R3 is accepted on the controlled local rebuild, not on a CI cache hit; the measured run's numbers live in the pull request, not in `progress.md`. +- 2026-11-14 Q: raise the cache limit above 10 GB? / Recommended: yes, to about 20 GB / Decision: left to the user, outside this pull request. diff --git a/docs/plans/2026-11-14-ci-build-times/progress.md b/docs/plans/2026-11-14-ci-build-times/progress.md new file mode 100644 index 0000000000..7d341e5c1e --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/progress.md @@ -0,0 +1,29 @@ +# Progress: CI build times, symbol dump error and Windows arm64 flake + +Plan: plan.md +Spec: spec.md (sha256 be1cd41eb716) +Plan status: Approved 2026-11-14 +Plan review: Codex gpt-6.1-sol, round 2, findings handled (reviews/plan-review-1.md, reviews/plan-review-2.md) +Executor: inline +Branch: claude/build-times-investigation-ec72b8 + +| Task | Title | Phase | State | Owner | Note | +| ---- | ---------------------------------------------------------------- | ----- | ----- | ----- | ---- | +| 1.1 | Attribute a dump_syms error and verify the stored unwind records | 1 | Open | | | +| 2.1 | Add --workspace-codegen-units to the bundle script | 2 | Open | | | +| 3.1 | Move the Windows server tests into server_tests_windows | 3 | Open | | | +| 3.2 | Pass the override and record a heartbeat in the bundle step | 3 | Open | | | +| 3.3 | Save pnpm store entries only on main | 3 | Open | | | +| 4.1 | Print the collected output and exit state on timeout | 4 | Open | | | +| 4.2 | Measure the #367 rate under load in the Windows arm64 VM | 4 | Open | | | +| 5.1 | Update the living documents | 5 | Open | | | +| 5.2 | Local gates, review and pull request | 5 | Open | | | +| 5.3 | Accept the requirements on the measured run | 5 | Open | | | + +## Log + +- 2026-11-14 09:27 1.1 Closed. Evidence: `vp test run scripts/dump-symbols.test.ts` 12 passing; with the binary-only rerun disabled, 3 cases fail; a real dump of a local binary and dSYM that logs the CFI error printed "the binary's 635173 STACK records are all stored" and exited 0. +- 2026-11-14 09:33 2.1 Closed. Evidence: `vp test run scripts/build-desktop-artifact.test.ts` 59 passing; without the `--` separator 1 case fails; under the real wrapper the process list showed `cargo build --config profile.release.package.bibcode-server.codegen-units=16 --config profile.release.package.bibcode-desktop.codegen-units=16 --bins --features tauri/custom-protocol --release --target aarch64-apple-darwin`, and the process group was empty after `TERM`. +- 2026-11-14 09:38 3.1, 3.2, 3.3 Closed. Evidence: `vp test run scripts/ci-platform-contract.test.ts` 53 passing (3 new cases failed first). Heartbeat probes: a child exiting 7 gives exit 7 and the notice; a `ps` that fails gives heartbeat lines without memory and exit 0; a heartbeat that died early no longer fails a good build (found by the probe: the trap's `kill` ran under `bash -e`, fixed with `|| true`). +- 2026-11-14 09:45 5.1 Closed. Evidence: `vp run check:markdown-links` passes; `vp test run scripts` 952 passing after `scripts/rust-workspace.test.ts` was updated for the sixth Rust job. +- 2026-11-14 09:45 4.1 In progress. Owner: inline. The helper and its test are written; the Windows arm64 VM is compiling them. diff --git a/docs/plans/2026-11-14-ci-build-times/research/actions_cache_limits.md b/docs/plans/2026-11-14-ci-build-times/research/actions_cache_limits.md new file mode 100644 index 0000000000..30c8daa54e --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/actions_cache_limits.md @@ -0,0 +1,123 @@ +# GitHub Actions cache storage limits, eviction and alternatives (as of November 2026) + +Research date 2026-11-14. About 14 tool calls; several vendor claims come from search snippets and are flagged as such. + +## 1. Per-repository limit, raising it, price + +### Takeaway + +Default is still 10 GB per repository, but since the 2025-11-20 changelog the limit can be raised, pay-as-you-go. The docs give $0.07 per GiB-month for usage above 10 GB. Whether a public repository can raise the limit and be billed is not stated on any page I could read. + +### Cited Findings + +- Default is 10 GB per repository. Enterprise owners, organization owners and repository admins can raise it in Actions settings. User-owned repositories can go up to 10 TB, organization-owned ones are capped by the org setting, and enterprise orgs by the enterprise setting. — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- Changelog 2025-11-20: every repository keeps 10 GB free with a 7-day default retention. Going beyond is pay-as-you-go, and the changelog says exceeding 10 GB requires a Pro, Team or Enterprise account. Two policies exist, a size limit (GB) and a retention limit (days). Enterprise policies cascade down to orgs and repos. — [GitHub changelog](https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/) +- Price: $0.07 per GB per month (binary GiB, billed in GB-hours). Only usage above 10 GB per repository is billed, measured by peak usage per hour. The owner is charged only if the limit is configured above the included amount. Cache is a separate allowance from artifacts and Packages. — [Actions billing](https://docs.github.com/en/billing/concepts/product-billing/github-actions) +- Docs examples (illustrative, fully utilized): 50 GB about $2.80, 200 GB about $13.30, 1000 GB about $69.30 per month. — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- Budgets: billing owners can set a budget on the new SKU. At the cap the cache becomes read-only for every repository using higher limits until the next cycle. Without a payment method on file, usage is blocked once the included quota is used. — [changelog](https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/), [billing](https://docs.github.com/en/billing/concepts/product-billing/github-actions) +- How to set it by API: `PUT /repos/{owner}/{repo}/actions/cache/storage-limit` with `max_cache_size_gb`, and `.../cache/retention-limit` with `max_cache_retention_days`. Org uses `/organizations/{org}/actions/cache/...`, enterprise `/enterprises/{e}/actions/cache/...`. Org and enterprise endpoints need `admin:organization` or `admin:enterprise` classic scopes. The API page gives no numeric maximums. — [REST: Actions cache](https://docs.github.com/en/rest/actions/cache) +- Usage endpoint `GET /repos/{owner}/{repo}/actions/cache/usage` refreshes about every 5 minutes. Org-level usage page is `https://github.com/organizations/ORG/settings/actions/caches`. — [REST](https://docs.github.com/en/rest/actions/cache), [changelog 2025-09-29](https://github.blog/changelog/2025-09-29-new-date-for-enforcement-of-cache-eviction-policy/) +- A September 2025 community post says the limit cannot be raised; it predates the 2025-11 change. Some third-party guides still say cache is capped, not billed. — [Discussion 173258](https://github.com/orgs/community/discussions/173258), [latchkey guide](https://latchkey.dev/learn/optimize-ci/github-actions-caching-explained) + +### Inferences + +- The repo's 17 GB demand would cost about 7 GB x $0.07 = about $0.50 per month at the docs rate, if a public repo is allowed to raise the limit. This is arithmetic, not a quoted figure. +- The docs say user-owned repos can reach 10 TB. If the repo is user-owned the setting is probably reachable by the owner, but a Pro or Team requirement per the changelog may apply. + +### Gaps + +- Whether public repositories on a Free plan can raise the limit, and whether they are billed, is not stated on any page I read. The billing page covers free runner minutes for public repos but is silent on cache storage. Verify in the repo's Settings > Actions > Caches, or by calling the storage-limit PUT and reading the error. +- Exact maximum values for the limit via the API. + +## 2. Eviction rules and "Unable to reserve cache" + +### Takeaway + +Eviction is LRU by last access, plus removal of entries unused for 7 days. A save that goes over the limit succeeds first, then eviction trims back, and eviction now runs hourly, so oversubscription causes churn (thrash). + +### Cited Findings + +- Entries not accessed in over 7 days are removed. At the limit, entries are deleted from oldest last-access to most recent. When the limit is exceeded, GitHub saves the new cache and then evicts until under the limit. The docs name the resulting churn "cache thrashing". — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- Eviction interval: previously every 24 hours, changed to every hour. Enforcement was delayed to November 2025 (exact date not on the page). GitHub warned hourly checks may cause more thrashing. — [changelog 2025-09-29](https://github.blog/changelog/2025-09-29-new-date-for-enforcement-of-cache-eviction-policy/) +- Rate limits per repository: 200 uploads per minute and 1,500 downloads per minute, with `Retry-After` on rejection. — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- Usage figure refreshes about every 5 minutes, so the 9.98 GiB reading can lag. — [REST](https://docs.github.com/en/rest/actions/cache) +- "Unable to reserve cache with key ..., another job may be creating this cache" is not defined on the official docs page. Community threads attribute it to a key already reserved or existing (a concurrent job, or a prior save that failed between reserve and upload). Fix reported: change the key. These are issue-thread claims, not official. — [setup-node issue summary](https://lightrun.com/answers/actions-setup-node-unable-to-reserve-cache-with-key-node-cache-linux-yarn-another-job-may-be-creating-this-cache), [setup-python issue 826](https://redirect.github.com/actions/setup-python/issues/826) +- Cache entries are keyed immutably: a key that already exists on a scope cannot be overwritten (the "Cache already exists" log line seen with the error above). Not stated on the pages I fetched; from the thread summaries only. + +### Inferences + +- With 17 GB demanded against 10 GB, each save evicts the oldest-accessed entries, those entries then miss, rebuild, and evict others. "No cache found" on main fits this loop. +- Rust entries that are restored on every run keep fresh access times, so the LRU victims are likely the entries least recently used, for example the per-PR pnpm entries and less frequent jobs. + +### Gaps + +- No official text on the exact "Unable to reserve" cause; no figure on how long between over-limit save and eviction beyond the hourly interval. + +## 3. Scoping by ref, and stopping PRs from saving + +### Takeaway + +PR runs can restore from the PR merge ref, the base branch and the default branch, but save only to `refs/pull/N/merge`, which only that PR's re-runs can read. Those entries count in the same repo total, so PR-saved pnpm entries are pure waste and should be disabled. + +### Cited Findings + +- A run restores from its current branch or the default branch. A PR run can also restore from its base branch. Sibling and child branches are not reachable. Tags do not share across names. — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- Caches created by a PR run are scoped to the merge ref and only re-runs of that PR can restore them. — same page +- Low-trust triggers (`pull_request_target`, `issue_comment`, `workflow_run`) get read-only access to default-branch caches; `pull_request` is unaffected. Only `push`, `workflow_dispatch`, `repository_dispatch`, `delete`, `registry_package`, `page_build` and `schedule` can write the default-branch scope. A failed save in a read-only run is a warning only. — same page +- The docs describe a `cache-mode` key (`read`, `write`, `write-only`, `none`) at workflow or job level. I did not verify its syntax or availability elsewhere. — same page +- `actions/cache/restore` is restore-only (no post step); `actions/cache/save` is save-only; both have `lookup-only`, and `fail-on-cache-miss` on restore. — [actions/cache restore README](https://github.com/actions/cache/blob/main/restore/README.md) +- `Swatinem/rust-cache`: `save-if` (default true) restores only when false, for example `${{ github.ref == 'refs/heads/main' }}`; `lookup-only`; `cache-targets`, `cache-on-failure`, `shared-key`. It removes incremental artifacts and files older than a week before save, and notes the 10 GB limit and base-branch availability. — [rust-cache](https://github.com/Swatinem/rust-cache) +- `actions/setup-node`: `cache: pnpm` enables caching; its README has no save-disable or restore-only input. `package-manager-cache: false` disables auto caching for npm. — [setup-node](https://github.com/actions/setup-node) + +### Inferences + +- For `actions/setup-node` with `cache: pnpm`, the only lever is to drop `cache:` on PR runs (conditional input expression, or two steps) and use `actions/cache/restore` with the pnpm store path. This follows from the absent input, not from a statement. +- Simplest set: save on `push` to main only (`if: github.event_name == 'push'` on `actions/cache` / `cache/save`, or `save-if` for rust-cache), restore everywhere. + +### Gaps + +- Did not verify `cache-mode` syntax in the workflow YAML reference. + +## 4. Alternative backends + +### Takeaway + +Real alternatives are sccache with S3/R2/GCS (needs secrets, which fork PRs do not get), sccache on the GHA cache (same pool, no relief), or third-party runner/cache vendors whose cache requires their runners. I found no confirmed free, public-repo-friendly drop-in for GitHub-hosted runners. + +### Cited Findings + +- sccache GHA backend: `SCCACHE_GHA_ENABLED=on`, needs `ACTIONS_RESULTS_URL` and `ACTIONS_RUNTIME_TOKEN` exported; `SCCACHE_GHA_RW_MODE=READ_ONLY` for read-only; changing `SCCACHE_GHA_VERSION` purges; on rate limit the build completes but may not save. The page does not mention size limits or cache v2. — [sccache GHA.md](https://raw.githubusercontent.com/mozilla/sccache/main/docs/GHA.md) +- sccache with R2: bucket name in `SCCACHE_BUCKET`, endpoint `https://.r2.cloudflarestorage.com`, `SCCACHE_REGION=auto`, keys via the AWS variables. — [sccache docs via search](https://docs.rs/crate/sccache/0.17.0/source/README.md); [Namespace sccache page](https://namespace.so/docs/integrations/sccache) +- Depot and Namespace publish sccache guides. — [Depot](https://depot.dev/blog/sccache-in-github-actions), [Namespace](https://namespace.so/docs/integrations/sccache) +- Blacksmith: co-located actions cache, "no additional cost", 25 GB free per repo per week; sticky disks $0.50/GB/month. Docs conflict on whether `useblacksmith/cache` forks are archived (use upstream actions) versus a one-line swap. Requires Blacksmith runners. — [Blacksmith cache docs](https://docs.blacksmith.sh/github-actions-runners/blacksmith-cache) (search snippet only; not fetched) +- RunsOn: S3 cache in your own AWS account, "Magic Cache" redirects `actions/cache` with no workflow change; opt-in via `extras=s3-cache`. Requires RunsOn runners. — [runs-on caching](https://runs-on.com/reference/caching/), [runs-on/cache](https://github.com/runs-on/cache) +- WarpBuild: `WarpBuilds/cache@v1`, on by default on WarpBuild runners. — [WarpBuild docs](https://docs.warpbuild.com/cache/quickstart) +- Fork PRs do not receive repository secrets by default. This is general GitHub behavior I did not source in this session; search did not return it. Confirm in GitHub docs before relying on it. + +### Inferences + +- sccache on the GHA backend draws on the same 10 GB pool, so it only helps if it shrinks total size. It stores compiled objects, which can be smaller than whole `target` dirs but spread across many entries. +- An S3/R2/GCS sccache backend fits GitHub-hosted runners and main-branch writes, with PRs read-only or off. Cost not researched. + +### Gaps + +- Not researched: pricing and caveats for BuildJet, Namespace and Depot caches, artifacts as caches (retention and size limits), GHCR/OCI as a cache store, and R2 or S3 pricing. No claim is made here. +- Vendor pricing for public repos not found. + +## 5. 2025-2026 changes to the cache service + +### Takeaway + +The backend was rewritten (cache service v2, `actions/cache` v4.2.0 and later), the legacy service was sunset in 2025, eviction went hourly, and paid limits arrived in November 2025. + +### Cited Findings + +- `actions/cache` v4.2.0 and v3.4.0 moved to the rewritten v2 cache service; the legacy service was to be sunset 2025-02-01. Self-hosted runners needed 2.231.0 or newer. This comes from mirrored release notes; I did not see the official discussion. — [RELEASES.md mirror](https://gitea.psi.ch/actions/cache/src/tag/v4/RELEASES.md) +- Current README lists v5 on Node 24, minimum runner 2.327.1. A v6.0.0 changelog mirror exists but is poorly corroborated. — [actions/cache README](https://cdn.jsdelivr.net/gh/actions/cache@main/README.md), [v6 mirror](https://gitea.wp-staging.com/actions/cache/raw/tag/v6.0.0/RELEASES.md) +- 2025-09-29: eviction interval 24 h to hourly, enforcement moved to November 2025. — [changelog](https://github.blog/changelog/2025-09-29-new-date-for-enforcement-of-cache-eviction-policy/) +- 2025-11-20: configurable size and retention limits, paid storage. — [changelog](https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/) +- The docs now document upload and download rate limits and per-trigger read-only scoping (see sections 2 and 3). — [Dependency caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) + +### Gaps + +- No official 2026 changelog entries were found; nothing on immutability changes beyond what is above. The search was not exhaustive for 2026. diff --git a/docs/plans/2026-11-14-ci-build-times/research/lto_debuginfo_dsym_cost.md b/docs/plans/2026-11-14-ci-build-times/research/lto_debuginfo_dsym_cost.md new file mode 100644 index 0000000000..7196b15bec --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/lto_debuginfo_dsym_cost.md @@ -0,0 +1,176 @@ +# Build-time cost of thin LTO + codegen-units=1 + debug info, and macOS packed dSYMs (as of Nov 2026) + +Research coverage note: ~16 tool calls. Several questions have no direct primary measurement; those are listed under Gaps and flagged as inference. Nothing below is from the user's own measurements except where labelled "given". + +## 1. Why codegen-units=1 + thin LTO keeps the largest crate on one core; 1 vs 16 CGUs + +### Takeaway + +With codegen-units=1 the crate is a single LLVM module, so rustc's parallel LLVM stage has nothing to fan out and one thread does the optimization; ThinLTO phases only parallelize across modules. No controlled recent benchmark of thin-LTO CGU=1 vs CGU=16 (runtime and build time together) was found; the available numbers are older or from other projects. + +### Cited Findings + +- rustc splits a crate into codegen units that LLVM processes in parallel; defaults are 16 (non-incremental) and 256 (incremental). "Setting this to 1 may improve the performance of generated code, but may be slower to compile." — [rustc book, codegen options](https://doc.rust-lang.org/rustc/codegen-options/index.html) +- When `-C lto` is unspecified, rustc does "thin local LTO" across the crate's own CGUs, but this is disabled if codegen-units=1 or opt-level=0. With explicit `lto`, codegen-units=1 means 1 CGU plus cross-crate LTO. — [rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html); same text in [Cargo book, profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) +- Cargo profile semantics: `lto = false` = thin local LTO only, none if CGU=1; `lto="thin"` = cross-crate thin LTO. Defaults: release codegen-units=16, dev 256. — [Cargo book](https://doc.rust-lang.org/cargo/reference/profiles.html) +- Measured (rustc ~1.7x, Nethercote, image-0.24.1 debug full build): default up to 16 CGUs ~4.5 s vs single CGU ~9.7 s ("more than twice as long"), because one LLVM thread runs while the rustc thread idles. — [Nethercote 2023-07-11](https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html) +- In release builds, thin-local-LTO threads start only after all `opt` threads finish (serial barrier); even the opt phase has a "staircase" with peak concurrency around seven threads because MIR-to-LLVM-IR conversion is serial. More parallelism costs more total cycles, peak memory, larger binaries and lower code quality. — [Nethercote 2023](https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html) +- Measured on ruff (astral-sh, release build, one machine, Rust version not stated in the snippet): fat LTO + 16 CGU 77 s; thin 41 s; LTO off 19 s; LTO off + CGU 16 = 7 s; thin + CGU 16 = 16 s. — as quoted in a search result for [ruff commit](https://app.semanticdiff.com/gh/astral-sh/ruff/commit/3ce145c47669645c9031ec42ef9b339835e548c9) (secondary view of a commit message; not independently verified) +- Older runtime figure (2018, internals thread on ThinLTO): CGU=16 sped the build ~10% but made runtime ~13% slower; Serde JSON benchmarks with ThinLTO ranged from 12% faster to 16% slower. — [internals.rust-lang.org](https://internals.rust-lang.org/t/help-test-out-thinlto/6017/3) and [thread](https://internals.rust-lang.org/t/help-test-out-thinlto/6017) +- Given (user's own measurement): LTO off + CGU 16 cut a cold 4-core release test build from 666 s to 293 s. + +### Inferences + +- A very large single crate at CGU=1 is one LLVM module; opt and codegen for it run on one thread regardless of core count, so a 4-core or 3-core (arm64 runner) machine sits mostly idle during that crate. This follows from the Nethercote/rustc-book statements but no source says it about thin LTO CGU=1 specifically. +- The doubling from CGU=1 is likely comparable in order to the 2x Nethercote saw, but his case was debug and small; do not treat 2x as a general constant. + +### Gaps + +- No 2024-2026 controlled benchmark of `lto="thin"` with CGU=1 vs CGU=16 giving both runtime and build time was found. rustc-perf or Kobzol posts on this were not located. +- Rust 1.98 specifics not found; all sources predate or do not name 1.98. + +## 2. What line-tables-only adds to rustc/LLVM and link time vs debug=0; interaction with thin LTO + +### Takeaway + +No source measured line-tables-only vs debug=0 time. The documented difference is that line-tables-only emits .debug_abbrev/.debug_info/.debug_str/.debug_aranges and line tables, with no variable or type info, which is the cheapest level that gives file:line. A known ThinLTO + `split-debuginfo=packed` bug exists (reported against an older rustc). + +### Cited Findings + +- `debug` levels: 0/none (release default), `line-directives-only`, `line-tables-only` ("minimal amount of debug info for backtraces with filename/line number info, but not anything else"), 1/limited, 2/full. MSRV 1.71 for the named levels. — [Cargo book](https://doc.rust-lang.org/cargo/reference/profiles.html), [rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html) +- A rust-lang PR notes line-tables-only adds .debug_abbrev, .debug_info, .debug_str and a proper .debug_aranges on top of what line-directives-only emits; `debuginfo=1` was never line-tables-only for compatibility reasons. — [rust-lang/rust#109808](https://github.com/rust-lang/rust/pull/109808) (from search summary; not opened) +- `strip = "none"` default in Cargo; `strip="debuginfo"` leaves "backtraces mostly-intact"; `strip="symbols"` is discouraged for crash reporting. Prior to 1.79 `strip="debuginfo"` unintentionally disabled PDB generation on MSVC. — [rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html) +- Reported bug: ThinLTO combined with `split-debuginfo="packed"` fails with a "duplicate split compilation unit" error for some binaries; fat LTO + packed and thin + unpacked built fine in that report. Current status not checked. — [rust-lang/rust#132677](https://github.com/rust-lang/rust/issues/132677) (via search summary) +- A rust-lang/rust test exists guarding an LLVM ODR assertion under LTO + debuginfo (tests/ui/lto/debuginfo-lto.rs). — [mirror](https://cdn05042023.gitlink.org.cn/dnrops/rust/src/branch/master/tests/ui/lto/debuginfo-lto.rs) + +### Inferences + +- Debug metadata travels in the bitcode that ThinLTO imports, so more DI means larger bitcode and more import/merge work, plausibly the main reason the LTO step slows with debug on. This is plausible LLVM behavior but no source here measures or states it. Treat as hypothesis; a measured A/B (`-Z time-passes` or `cargo --timings`) with debug=0 vs line-tables-only would settle it. +- The user's 38 -> 75-111 min jump is a combination of (a) DI generation/LTO, (b) strip=none linking bigger binaries, (c) dsymutil, and (d) any fingerprint rebuild of deps (Q5). Sources do not apportion it. + +### Gaps + +- No measurement of line-tables-only vs debug=0 compile or link time found. No source found on ThinLTO importing debug metadata cost. + +## 3. macOS split-debuginfo: packed vs unpacked vs off; dsymutil cost, threading, flags; can Breakpad tools read unpacked? + +### Takeaway + +`packed` runs dsymutil after link on every binary; it is documented as a large share of macOS build time (7 of ~10 s in one incremental case). dsymutil's classic linker is largely single-threaded (143% CPU in a 2025 sample); LLVM's parallel linker (`--linker=parallel`) gives ~3-4x on clang and was reported made default in LLVM in June 2026, but Xcode shipping status was not found. `-j` only parallelizes across architectures; `--no-odr` skips C++ ODR uniquing (little relevance to Rust). + +### Cited Findings + +- rustc `split-debuginfo`: `off` on macOS "prevents the final execution of dsymutil"; `packed` (rustc default on macOS and Windows MSVC) puts debuginfo in a `.dSYM`; `unpacked` on macOS "means the original object files will contain debug information". — [rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html) +- Cargo's own default is `unpacked` on macOS for profiles with debug info enabled ("Be aware that Cargo and rustc have different defaults"). So the user's `packed` is an explicit opt-in. — [Cargo book](https://doc.rust-lang.org/cargo/reference/profiles.html) +- Measured (incremental debug build of Cargo itself, 2021): about 7 of ~10 s was dsymutil. — [internals thread](https://internals.rust-lang.org/t/help-test-faster-incremental-debug-macos-builds-on-nightly/14016) +- Unpacked leaves debug info in object files in the target dir; one report measured 2.1 GB of extra target dir. — same thread +- dsymutil on a large C++ build (LLVM discourse, 2025-2026): clang link took "a second or 2", classic dsymutil 1:38 wall, 143% CPU (133 s user); `--linker=parallel` 45 s wall, 531% CPU (216 s user). Thread also records the parallel linker being made default on June 4, 2026, with no LLVM or Xcode version stated. — [LLVM discourse](https://discourse.llvm.org/t/llvm-dsymutil-performance/89051) +- Same author's measurement on clang: ~3 min classic vs ~40 s parallel; parallel output is not binary-identical to classic, a semantic diff found ~50k differences among ~5M DIEs, and the author called the tool not yet production-ready (post predates the June 2026 default change). — [Devlieghere, dsymutil parallel linker](https://jonasdevlieghere.com/post/dsymutil-parallel-linker) +- dsymutil docs: `-j/--num-threads` = "maximum number of simultaneous threads to use when linking multiple architectures"; `--no-odr` = do not use ODR for uniquing C++ types; `--no-output` does the link in memory without writing (useful for timing); `--flat` emits a flat `.dwarf` file. The fetched doc page did not list `--linker`. — [LLVM dsymutil docs](https://llvm.org/docs/CommandGuide/dsymutil.html) +- With unpacked, the debug info lives in the object files and the binary holds a debug map; Rust code can symbolize without dsymutil. A rustc PR says dsymutil "can take quite a long time ... in debug builds and provides little benefit". — [internals thread](https://internals.rust-lang.org/t/help-test-faster-incremental-debug-macos-builds-on-nightly/14016), [rust-lang/rust#79361](https://github.com/rust-lang/rust/issues/79361) (via search summary) +- Historical: Firefox hit llvm-dsymutil crashes on Rust debug info and worked around by lowering debuginfo level on cross-compiled macOS builds; dsymutil SIGBUS under packed reported in 2021. — [Bugzilla 1430315](https://bugzilla.mozilla.org/show_bug.cgi?id=1430315), [internals thread](https://internals.rust-lang.org/t/help-test-faster-incremental-debug-macos-builds-on-nightly/14016) +- mozilla/dump_syms README documents inputs as DWARF/STABS in ELF and standalone PDB and says nothing about unpacked/debug-map inputs on macOS. — [mozilla/dump_syms](https://github.com/mozilla/dump_syms) + +### Inferences + +- Because Mach-O linked binaries carry only a debug map pointing at `.o` files (no DWARF) for unpacked, dump_syms (which reads DWARF from the input file or a dSYM) cannot get symbols from an unpacked build without running dsymutil anyway; the object files on disk are the only DWARF. No source confirms or refutes this for dump_syms 2.3.9; test it with a one-binary experiment. +- Running dsymutil yourself with `--linker=parallel` (if the installed Xcode's dsymutil supports it) is the cheapest way to keep a dSYM; the user's Rust binaries are single-language Rust so ODR flags are unlikely to matter. + +### Gaps + +- No measurement of dsymutil time on a large Rust binary with line-tables-only DI was found. Whether Xcode (e.g. 26.x/27.x) bundles a dsymutil with the parallel linker, and whether rustc invokes dsymutil with threading flags, was not found. How rustc invokes dsymutil (flags) not verified. +- GitHub-hosted macOS arm64 runner core count not verified here. + +## 4. debug only for workspace crates; are CFI/unwind tables independent of `debug`? + +### Takeaway + +`[profile.release.package."*"] debug = false` is supported and applies to all dependencies but not workspace members, so it does keep DI only for first-party crates. Unwind tables are controlled by separate rustc flags (`force-unwind-tables`) and target defaults, not by `-C debuginfo`; on Apple platforms unwinding data is compact unwind (`__unwind_info`) plus `__eh_frame`, and Mozilla's dump_syms parses compact unwind. + +### Cited Findings + +- Package overrides: `"*"` overrides "all dependencies (but not any workspace member)"; precedence is `package.name`, then `package."*"`, then `build-override`, then the profile. Overrides cannot set `panic`, `lto`, or `rpath`. — [Cargo book](https://doc.rust-lang.org/cargo/reference/profiles.html) +- `-C force-unwind-tables` forces unwind table generation; default is target-dependent; it is separate from `-C debuginfo`. — [rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html) +- Apple compact unwinding lives in `__unwind_info`; Clang prefers it over DWARF CFI on Apple platforms, and the Rust dump_syms deployed in Firefox's build parses it. — [Gankra, Apple Compact Unwinding Format](https://gankra.github.io/blah/compact-unwinding) +- Old Google Breakpad dump_syms had incomplete CFI on iOS (no compact unwind support); Mozilla imported a newer Rust dump_syms to emit correct macOS CFI. — [Bugzilla 1723466](https://bugzilla.mozilla.org/show_bug.cgi?id=1723466), [Bugzilla 1716603](https://bugzilla.mozilla.org/show_bug.cgi?id=1716603) +- A Breakpad fix handled dSYMs where some Mach-O sections in `__DWARF`/`__eh_frame` are absent. — [Breakpad commit](https://android.googlesource.com/platform/external/google-breakpad/+/f7428bc3) (from search result titles/summary) +- clang/Rust can emit bogus unwind info for some functions on arm64 Windows causing dump_syms to fail on those functions. — [Bugzilla 1529355](https://bugzilla.mozilla.org/show_bug.cgi?id=1529355) (via summary) + +### Inferences + +- With `package."*" debug=false`, dependency frames lose file:line but keep function names (symbol table, strip=none) and CFI; this is probably acceptable for crash triage but loses inlined-frame detail. Not measured. +- For Windows MSVC, function names and unwind come from PDB + `.pdata/.xdata`; line info needs PDB with line tables, which `debug=false` deps would lack. Not verified by a source. + +### Gaps + +- No source states explicitly that unwind tables are unaffected by `debug` on macOS arm64, Linux and Windows; the above is derived from flag separation. Linux x86_64 uses `.eh_frame` by default; not cited. + +## 5. Do CARGO_PROFILE_RELEASE_* env overrides force rebuilding all dependencies? + +### Takeaway + +Yes. Cargo hashes the whole `Profile` struct (plus Lto, rustflags) into each unit's fingerprint and into the metadata hash that names output files, and dependency changes propagate to dependents. A build with different debug/strip/split-debuginfo therefore cannot reuse a normal-profile cache for any unit it applies to. + +### Cited Findings + +- Fingerprint `profile` = "Hash of the Profile, CompileMode, and any extra flags"; `calculate_normal` hashes `&unit.profile`; `compare` returns `ProfileConfigurationChanged` when it differs; dependency fingerprints are included so dirtiness propagates up. — [Cargo fingerprint source](https://doc.rust-lang.org/nightly/nightly-rustc/src/cargo/compiler/fingerprint/mod.rs.html) +- The module doc table marks Profile and Lto as included in the fingerprint, `Metadata::unit_id` and `Metadata::c_metadata`, so artifacts of different profiles get different file names and do not overwrite each other. — same source +- Env vars `CARGO_PROFILE__` set profile keys (e.g. `CARGO_PROFILE_RELEASE_DEBUG`). Buildroot uses them for debug levels. — [Buildroot patch](https://buildroot.uclibc.org/lists/buildroot/2023-November/364553.html) +- Build scripts/proc-macros with `build-override` may be built twice. — [Cargo book](https://doc.rust-lang.org/cargo/reference/profiles.html) + +### Inferences + +- The excerpt did not show `Profile` struct fields (file truncated at 100k chars), so `debug`, `strip` and `split_debuginfo` being fields is inferred from Cargo's profile model; confirm with `CARGO_LOG=cargo::core::compiler::fingerprint=trace` (reports `ProfileConfigurationChanged`). +- Fix: `debug=false` for deps via `package."*"` still changes the profile hash of deps relative to the non-symbol build if they differ; to share cached dependency artifacts, the deps' profile must be identical in both builds (e.g. set the same `package."*"` settings in both) and only workspace members differ. + +### Gaps + +- Whether `strip` is part of Cargo's hashed `Profile` (strip is applied at link, and Cargo passes it to rustc) was not verified. + +## 6. Faster linkers on macOS and Windows; do they help LTO builds? + +### Takeaway + +Little macOS/Windows-specific evidence was found. Rust 1.90 made rust-lld the default only on x86_64 Linux. Wild's macOS support is in progress and not default as of its 0.11 release (Oct 2026 per one report). No source states any linker speeds up the LTO (LLVM) phase, which happens before linking. + +### Cited Findings + +- Rust 1.90.0 (2025-09-18) uses LLD by default only for `x86_64-unknown-linux-gnu`; opt out with `-C linker-features=-lld`. — [Rust blog 1.90](https://blog.rust-lang.org/2025/09/18/Rust-1.90.0/), [LLD post](https://blog.rust-lang.org/2025/09/01/rust-lld-on-1.90.0-stable) +- x86_64-apple-darwin was demoted to Tier 2 with host tools in 1.90. — [Rust blog](https://blog.rust-lang.org/2025/08/19/demoting-x86-64-apple-darwin-to-tier-2-with-host-tools) +- Wild linker: 0.9 (May 2026) expanded platform coverage; Phoronix (reported Oct 2026) says Wild 0.11 macOS port has progressed but is not enabled by default. — [Wild 0.9 post](https://davidlattimore.github.io/posts/2026/05/24/wild-update-0.9.0.html), [Phoronix 0.9](https://phoronix.com/news/wild-linker-0.9), [Phoronix 0.11 via search](https://phoronix.com/news/Wild-0.11-Released) (page returned 403 on fetch; date and claim from search snippet only) +- Nethercote notes LTO threads run inside rustc/LLVM before the system linker; the linker only receives the final objects. — [Nethercote 2023](https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html) (inference on the second half; his post does not discuss linkers) + +### Inferences + +- Linker choice affects only the final link step, which for a debug-info-heavy binary is real but is not where thin-LTO time is spent. Link time on macOS with Apple's ld (ld-prime since Xcode 15) is typically seconds to tens of seconds; dsymutil dwarfs it (LLVM thread: link 1-2 s vs dsymutil 98 s on clang). + +### Gaps + +- Not found: ld-prime vs ld64 timings, rust-lld on aarch64-apple-darwin status, lld-link vs link.exe timings for Rust on Windows, any LTO-specific linker benchmark. + +## 7. Known dump_syms CFI errors on macOS (`CfiError BadDebugInfo UnknownVersion`, `OffsetOutOfBounds`) + +### Takeaway + +No source documenting these exact error strings was found. `CfiError` and `BadDebugInfo` are in the `symbolic-cfi` crate used by dump_syms; the sources show bad macOS output has historically come from dsymutil/dSYM problems, not dump_syms. Whether the errors mean truncated or still-being-written input or >4 GiB dSYMs is unconfirmed. + +### Cited Findings + +- `symbolic-cfi` defines `CfiError` with kinds including `BadDebugInfo` ("debug information in the Object file is invalid") and `UnsupportedArch`; many Mach-O/DWARF parser errors are mapped to BadDebugInfo. — [docs.rs symbolic-cfi](https://docs.rs/symbolic-cfi/latest/src/symbolic_cfi/lib.rs.html) +- No `UnknownVersion` variant seen in the search summaries of that file (not read in full). — same +- Mozilla bug 1810249: wrong filenames/lines in macOS dSYMs traced to a dsymutil regression bundled with clang 15 (LLVM issue 60307), not dump_syms; `addr2line` on the raw dSYM showed the same bad data. — [Bugzilla 1810249](https://bugzilla.mozilla.org/show_bug.cgi?id=1810249) +- Mozilla/dump_syms open issues list has nothing on UnknownVersion/OffsetOutOfBounds in titles. — [dump_syms issues](https://github.com/mozilla/dump_syms/issues) + +### Inferences + +- `UnknownVersion` in a DWARF/Mach-O parser typically means a section header has an unexpected version field, which is consistent with garbage/truncated bytes (e.g. reading a file before dsymutil finished, or a corrupt/partial copy), and `OffsetOutOfBounds` is consistent with a section offset beyond file length (truncation or mismatch between object and dSYM). This is reasoning from error names, not a documented cause. +- Cheap diagnosis: check file size vs `dwarfdump --verify`/`dsymutil --verify`, re-run after `sync`, and confirm dsymutil exited 0 before dump_syms starts; compare the dSYM size against 4 GiB. + +### Gaps + +- No documented case of a >4 GiB dSYM breaking dump_syms found. The dump_syms 2.3.9 changelog and the symbolic repo issues were not searched directly. + +## Cross-cutting summary for the report writer (opinion/inference, not measured) + +- Highest-confidence cost drivers: (1) CGU=1 serializes the largest crate's LLVM work (rustc book, Nethercote); (2) `packed` adds a dsymutil run, classic dsymutil is ~1.4 cores; (3) any profile difference rebuilds every dependency (Cargo fingerprint source). +- Candidate low-cost symbol profile to test (untested): keep `lto="thin"`, test `codegen-units` 16 vs 1 for runtime; use `debug="line-tables-only"` for workspace crates and `package."*" debug=false`; same deps profile in both builds so the dependency cache is reusable; replace `packed` by running dsymutil (parallel linker if available) only on the final binaries, or measure `unpacked` + dsymutil manually. diff --git a/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md new file mode 100644 index 0000000000..83ad9eef34 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md @@ -0,0 +1,126 @@ +# Measured facts from this repository (2026-11-14) + +These come from the CI job logs and jobs API of ten `main` runs (2026-11-04 to 2026-11-14) and +from local builds on a 12-core Apple silicon Mac with 96 GB RAM, Rust 1.98.1. They are primary +evidence and outrank general claims in the other notes. Local numbers are one run per variant on +a machine under other load (load average 6 to 30), so read them as ballpark figures. + +## 1. Where the time goes in CI + +Bundle step `Build native bundle without publishing` of run 43562279474 (2026-11-14), from the +timestamps of Cargo's `Compiling` lines: + +| Row | Dependencies (all `Compiling` lines) | After the last dependency until `Finished` | Cargo total | +| ---------------------------------------------- | ------------------------------------ | ------------------------------------------ | ------------ | +| macOS arm64 (no cache restored) | 7 min (527 crates) | 81 min | 87 min 51 s | +| Linux x64 (no cache restored) | 6 min (582 crates) | 23 min | about 29 min | +| Windows x64 (partial restore) | under 1 min | 42 min | about 42 min | +| macOS arm64, run 42542822659, full cache match | 2.5 min (58 crates) | 107 min | 109 min 51 s | + +The times are when Cargo printed each `Compiling` line, which is when a unit started, so the +split is an estimate: some dependencies may still have been compiling after the last line. After +`Finished`, the symbol dump took 27 s and bundling 30 s. On this reading the two workspace crates +(`bibcode-server`, `bibcode-desktop`) and the final link take 80% to 97% of the step. The two +macOS rows are different runs, so their difference is not a measured cache saving; the Windows +arm64 attempts in section 2 give about 12 minutes saved before the bundle step with a full match. + +Windows arm64 row of the same run (103 minutes in total, partial cache restore), `bibcode-server` +was compiled five times: + +| Step | Cargo build time | Test run time | +| ------------------------------------------------------------ | ---------------- | ------------- | +| `cargo test -p bibcode-desktop` (debug) | 10 min 41 s | 34 s | +| `cargo check -p bibcode-server --all-targets` | 5 min 06 s | none | +| `cargo test -p bibcode-server --test git_system_config -j 2` | 6 min 42 s | 17 s | +| `cargo test -p bibcode-server --lib -j 2 -- ` | 17 min 30 s | 71 s | +| `cargo test ... --test windows_terminal_shims -j 2` | 16 s | 5 s | +| release bundle build | 36 min 15 s | none | + +That is 76 of 103 minutes compiling one crate, against about 2 minutes running tests. + +Bundle step minutes over time (same step, `main` runs): + +| Date | macOS arm64 | macOS x64 | Linux x64 | Windows x64 | Windows arm64 | +| -------------------------------- | ----------- | --------- | --------- | ----------- | ------------- | +| 2026-11-04 (before symbols) | 37 to 40 | 47 to 57 | 26 to 28 | 35 to 41 | 32 to 38 | +| 2026-11-06 to 11-07 (symbols on) | 75 to 79 | 81 to 92 | 25 to 26 | 30 to 45 | 40 to 48 | +| 2026-11-09 | 111 | 101 | 25 | 36 | 44 | +| 2026-11-13 to 11-14 | 81 to 89 | 69 to 90 | 27 to 33 | 44 to 53 | 38 to 44 | + +Turning the symbol profile on roughly doubled the two macOS rows and left Linux unchanged. + +## 2. Cache store + +- `gh cache list` total: 9.98 GiB of a 10 GB limit (`max_cache_size_gb: 10`, retention 7 days). +- Entries present: three of six `native-desktop` Rust entries (1.5 to 1.85 GiB each), `check` + 0.78, two of three `crash-smoke` (0.57 each), one Linux Git entry 0.45, and six pnpm store + entries of about 0.48 GiB each. One pnpm entry belonged to a pull request ref. +- Missing: both Windows `native-desktop` entries, Linux x64 `native-desktop`, `test`, and the + macOS `crash-smoke` entry. +- In run 43562279474 (push to `main`) 11 of 13 Rust-cache restores logged `No cache found`; the + two Windows rows restored a partial match. +- Wanted total if every entry existed: about 17 GB. +- Run 43523947929, Windows arm64, three attempts: attempt 1 restored nothing and was cancelled at + its limit while saving; attempt 2 (01:09) restored that entry as a full match (2,056 MB), so a + job cancelled in its post step had saved; attempt 3 (02:08) found no cache, so the entry was + evicted within an hour of being used. With the full match the steps before the bundle took + about 12 minutes less (desktop tests 11 against 19 minutes, check 6 against 7, Git tests 7 + against 10). + +## 3. Local release build of `bibcode-desktop` (cold target, `--timings`) + +| Variant | Total | `bibcode-server` lib | `bibcode-desktop` lib | `bibcode-desktop` bin (thin LTO link) | +| ------------------------------------------------------------------------------------------- | ------ | -------------------- | --------------------- | ------------------------------------- | +| Shipped profile (thin LTO, 1 codegen unit, stripped, no debug info) | 1299 s | 650 s | 187 s | 288 s | +| Symbol profile (same plus `strip=none`, `debug=line-tables-only`, `split-debuginfo=packed`) | 1260 s | 676 s | 177 s | 304 s | +| Symbol profile with `codegen-units=16` | 673 s | 311 s | 79 s | 177 s | + +- The symbol profile makes no measurable difference on this machine (1260 s against 1299 s). +- No `dsymutil` or `ld` process was ever seen by a 15-second sampler in the symbol build, so + `dsymutil` takes well under a minute here. The dSYM DWARF file is 353 MB. +- 16 codegen units cut the build to about half. +- The three crates build one after another, so the build is serial for 17 of its 21 minutes. +- A rebuild in the shipped-profile target with `--config profile.release.package.bibcode-server.codegen-units=16` + and the same for `bibcode-desktop` compiled only those two crates (every dependency stayed + fresh) and took 653 s, against 1125 s for the same three units with one codegen unit. Another + build was running at the same time. + +Peak resident memory of single `rustc` processes (10-second samples): + +| Variant | `bibcode-server` | `bibcode-desktop` lib | `bibcode-desktop` bin | +| ------------------------------------------------------------------------- | ---------------- | --------------------- | --------------------- | +| Shipped profile | 8.7 GB | 3.9 GB | 5.6 GB | +| Symbol profile (one codegen unit, a later rebuild of the two crates only) | 9.6 GB | 4.8 GB | 6.4 GB | +| Symbol profile with 16 codegen units | 12.0 GB | 5.5 GB | 9.4 GB | + +GitHub-hosted runner memory (GitHub docs, see `rust_ci_speedups.md`): macOS arm64 7 GB, macOS +Intel 14 GB, Linux and Windows 16 GB. The shipped-profile compile of `bibcode-server` already +needs more memory than the macOS arm64 runner has. + +## 4. Symbol dump (`dump_syms` 2.3.9) + +- `[ERROR] CFI: CfiError { kind: BadDebugInfo, ... }` appears in 7 of 24 macOS job logs that + dump symbols (desktop bundle on both architectures and the crash smoke job), most recently on + 2026-11-14. It never appears on Linux or Windows. +- The crash smoke job of 2026-11-14 logged the error and still passed its check that the + caller frame was unwound with `trust == "cfi"`. +- In the tool's source (`object_info.rs`), the unwind writer logs this error and keeps whatever + it wrote before failing; the process still exits 0. +- Locally, the one-codegen-unit symbol build: six runs on the same binary and dSYM gave + byte-identical output with no error, 334,968 `STACK CFI` records, and every one of the 56,811 + `FUNC` records lies inside a `STACK CFI INIT` range. Dumping the dSYM alone fails with "No CFI + data". +- Locally, the 16-codegen-unit symbol build reproduces the error: dumping the dSYM alone logs + `CfiError { kind: BadDebugInfo, source: Some(UnexpectedEof(...)) }` and writes no `STACK` + record; dumping the binary alone logs nothing and writes 635,173; dumping both logs the error + and writes the same 635,173 (0 lines differ). In the dSYM, the `__unwind_info` section header + has file offset 0 with a non-zero size, so the parser reads unrelated bytes. +- `symbolic-cfi` 13.9.0 (`process_macho`, `read_compact_unwind_info`) returns on the first + failing entry, so an error inside the binary's own unwind data would truncate the records. + +## 5. Flaky Windows arm64 test + +Run 43523947929, attempt 2: `running 10 tests` at 02:01:33; the four tests that libtest started +first each failed after exactly 10 s (`Elapsed`), and the other six passed within the next 5 s. +The same step normally finishes in 3.7 to 4.6 s on Windows arm64 and 2.4 to 3.0 s on x64 +(9 other runs). diff --git a/docs/plans/2026-11-14-ci-build-times/research/report.md b/docs/plans/2026-11-14-ci-build-times/research/report.md new file mode 100644 index 0000000000..88e133dc4f --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/report.md @@ -0,0 +1,76 @@ +# Compiling Two Crates Burns Your CI Hours + +> **Corrections after review (2026-11-14).** Where this report and +> [measured_ci_and_local.md](measured_ci_and_local.md) differ, the notes are right: +> +> - The split between dependencies and workspace crates in CI comes from when Cargo started each +> unit, so "80% to 97%" and "the cache saves at most 5 to 7 minutes" are estimates. The two +> macOS runs compared are different runs and do not measure a cache saving. The one matched +> comparison (Windows arm64, same commit) shows about 12 minutes saved before the bundle step. +> - "76 minutes compiling one crate" is the sum of five whole Cargo invocations in which +> `bibcode-server` was the last large unit, not an isolated per-crate time. +> - The one-codegen-unit symbol build was sampled later: `rustc` for `bibcode-server` peaked at +> 9.6 GB, against 8.7 GB with the shipped profile. +> - The `dump_syms` CFI error was reproduced locally and comes from the dSYM, so its cause is +> known. +> - The macOS doubling is associated in time with the symbol profile; no controlled comparison on +> that runner exists yet, and memory pressure remains a hypothesis. + +CI is slow because two first-party crates, `bibcode-server` and `bibcode-desktop`, plus the final thin-LTO link are compiled from scratch on every run, not because the cache misses. In the measured bundle step they take **80% to 97% of the time**. A full cache match saves only about 5 minutes of a 110-minute macOS build, and an uncached Linux row still finishes in about 29 minutes. The cache problem is real but secondary: the repository holds 9.98 GiB of a 10 GB limit and wants about 17 GB, so 11 of 13 Rust-cache restores in the latest `main` run found nothing. The macOS rows doubled (37-40 to 75-79 minutes) when the symbol profile was switched on, while the same profile changed nothing on a 12-core, 96 GB local Mac. The likeliest explanation is runner hardware: 3 vCPUs and 7 GB of RAM against a compile that peaks at 8.7 GB locally. That explanation is an inference, not a measurement. Windows arm64 adds a separate waste, compiling `bibcode-server` five times in one job (76 of 103 minutes). The ranked fixes are, first, compile each crate once per job; second, stop the cache from thrashing; third, move the symbol build off the 7 GB macOS runner or make it cheaper; fourth, relax codegen-units where the binary's speed allows it. + +## What the repository's own measurements show + +All figures in this section are measured facts from `measured_ci_and_local.md` (ten `main` runs, 2026-11-04 to 2026-11-14, plus local `--timings` builds). They outrank every external claim below. Local numbers are one run per variant on a loaded machine, so treat them as ballpark. + +| Row of run 43562279474 (bundle step) | Dependencies | After last dependency to `Finished` | Cargo total | +| ---------------------------------------------- | ------------------- | ----------------------------------- | ------------ | +| macOS arm64, no cache restored | 7 min (527 crates) | 81 min | 87 min 51 s | +| Linux x64, no cache restored | 6 min (582 crates) | 23 min | about 29 min | +| Windows x64, partial restore | under 1 min | 42 min | about 42 min | +| macOS arm64, run 42542822659, full cache match | 2.5 min (58 crates) | 107 min | 109 min 51 s | + +Three conclusions follow directly. First, dependencies are cheap; the cache can save at most 5 to 7 minutes per row. Second, the cached macOS run (110 min) was slower than the uncached one (88 min), so run-to-run variance on the macOS runner is larger than the cache effect. Third, Linux compiles the same sources in about a quarter of the macOS time, which points at the runner and the symbol work rather than at the code. After `Finished`, the symbol dump took 27 s and bundling 30 s, so dumping symbols is not where time goes; producing the binary and its debug info is. + +The step's duration history shows the symbol profile's effect. Before symbols (2026-11-04) macOS arm64 ran 37-40 min and macOS x64 47-57 min; with symbols on (11-06 to 11-07) they ran 75-79 and 81-92; the worst day (11-09) hit 111 and 101. Linux stayed at 25-28 min throughout. Windows did not move much (30-53 min). + +The local build contradicts a simple "debug info is expensive" story. On the 12-core Mac a cold release build of `bibcode-desktop` took **1299 s with the shipped profile and 1260 s with the symbol profile**; the dSYM DWARF is 353 MB and no `dsymutil` or `ld` process showed up in a 15-second sampler. The same machine shows what dominates: `bibcode-server` lib 650 s, `bibcode-desktop` lib 187 s and the thin-LTO bin link 288 s, built one after another, so the build is serial for 17 of 21 minutes. Setting `codegen-units=16` on the symbol profile cut the total to **673 s**, about half. Peak memory per `rustc` was 8.7 GB for `bibcode-server` at the shipped profile and 12.0 GB at 16 codegen units. GitHub's macOS arm64 runner has 7 GB ([GitHub docs](https://docs.github.com/en/actions/reference/runners/github-hosted-runners)); Linux and Windows have 16 GB. The shipped-profile compile already needs more RAM than the macOS arm64 runner offers, so it is plausible that the runner swaps, but no run has measured paging, and the local symbol-profile memory at codegen-units=1 was not recorded. That gap matters for choosing the fix. + +Windows arm64 shows a different waste. In one 103-minute job, `bibcode-server` was compiled five times by test and check steps (10 min 41 s, 5 min 06 s, 6 min 42 s, 17 min 30 s, 16 s) plus once more in the 36 min 15 s release bundle. That is 76 minutes of compiling one crate against about 2 minutes of tests. + +Two side findings do not drive build time. `dump_syms` 2.3.9 logs `CfiError { kind: BadDebugInfo }` in 7 of 24 macOS job logs and never on Linux or Windows, yet exits 0 and the crash smoke job still passes its `trust == "cfi"` check; locally, six runs gave byte-identical output without the error. The flaky Windows arm64 test fails the first four of ten tests at exactly 10 s each, which looks like a timeout hit while the box is busy. + +## The cache is thrashing, but fixing it buys minutes, not hours + +GitHub's per-repository cache limit is **10 GB by default**, and entries unused for 7 days are removed. When the limit is exceeded GitHub saves the new entry first, then evicts oldest-accessed entries until usage is under the limit, and the docs call the resulting churn "cache thrashing" ([GitHub docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching)). Eviction moved from every 24 hours to hourly ([GitHub changelog](https://github.blog/changelog/2025-09-29-new-date-for-enforcement-of-cache-eviction-policy/)), and the changelog itself warned of more thrashing. The measured store (9.98 GiB, six `native-desktop` entries of 1.5 to 1.85 GiB each wanted but three present, six pnpm entries of about 0.48 GiB, one of them from a pull request ref) fits that loop. Each lock-hash change on `main` writes a fresh 1.5-1.85 GiB entry per OS and architecture and evicts a live one. + +The limit can now be raised. A November 2025 changelog made size and retention configurable, with usage above 10 GB billed at **$0.07 per GiB-month** and a budget control that turns the cache read-only at the cap ([changelog](https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/), [billing](https://docs.github.com/en/billing/concepts/product-billing/github-actions)). The REST endpoints are `PUT /repos/{owner}/{repo}/actions/cache/storage-limit` and `.../retention-limit` ([REST docs](https://docs.github.com/en/rest/actions/cache)). By that arithmetic, covering the 17 GB demand costs roughly 7 x $0.07, about **$0.50 per month**; that figure is my calculation, not a quoted price. What could not be verified: whether a public repository on a free plan may raise the limit at all (the changelog mentions Pro, Team or Enterprise), and the API's maximum values. The cheapest test is to call the PUT and read the answer. + +Free ways to shrink demand come first. A pull request run can only save to its own merge ref, readable by its own re-runs only, yet those entries count against the same total ([GitHub docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching)). Saving only from `push` on `main` removes them; `Swatinem/rust-cache` has `save-if` for this, while `actions/setup-node` has no restore-only switch in the README I read, so the pnpm store needs `actions/cache/restore` plus a save step gated on `push` (an inference from the missing input). Rust-cache is well suited to shrinking further because its cleanup keeps only dependency artifacts: it deletes `incremental/`, drops workspace crates unless `cache-workspace-crates` is set, prunes the registry to used crates and strips old files ([rust-cache README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md), [cleanup.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/cleanup.ts)). Its key is two 8-hex hashes: an environment hash (rustc version, installed toolchains, and any `CARGO*`, `RUST*`, `CC*`, `CXX*`, `CFLAGS`, `CMAKE*` variable) and a manifest hash (Cargo.toml files with local versions zeroed, registry and git entries of `Cargo.lock`, `.cargo/config.toml`, toolchain files) ([config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts)). Only one restore-key is passed, the environment hash alone, so a lockfile change restores the newest sibling entry and saves a new one ([restore.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/restore.ts)). Two consequences: a job that builds debug and release in one key stores both dependency sets in one entry (an inference from the pruning rule), and any CI-only `CARGO_PROFILE_*` environment variable changes the environment hash and so forfeits restores. Splitting the `key` per profile and dropping `cache-bin` are untested but low-risk size levers. Cargo also hashes the whole profile into every unit's fingerprint, so a symbol-profile build cannot reuse dependency artifacts built with different `debug`, `strip` or `split-debuginfo` settings ([Cargo fingerprint source](https://doc.rust-lang.org/nightly/nightly-rustc/src/cargo/compiler/fingerprint/mod.rs.html)); whether `strip` is part of that hash was not verified. + +The external research agrees with the measurement on one point and cannot settle another. Rust-cache never stores the workspace crates by default, which matches the 80-97% figure. Whether `cache-workspace-crates: true` helps is unverified, and a rust-cache pull request notes that it grows entry count because source edits make new keys ([PR 349](https://github.com/Swatinem/rust-cache/pull/349)). sccache is the other route to caching first-party crates, but it cannot cache anything that invokes the linker (bins, dylibs, proc-macros), needs incremental compilation off ([sccache Rust.md](https://raw.githubusercontent.com/mozilla/sccache/main/docs/Rust.md)), and Depot measured it making release builds up to 50% slower while helping test builds 11-14% ([Depot](https://depot.dev/blog/guide-to-faster-rust-builds-in-ci.md)). Its GitHub backend draws on the same 10 GB pool, and no source covers sccache with thin LTO and one codegen unit. Third-party cache backends (S3 or R2 with sccache, or Blacksmith, RunsOn, WarpBuild caches) exist but most require their own runners, and I found no free drop-in for GitHub-hosted runners; their prices come from vendor and comparison pages and were not verified. + +## One crate, one core, and a debug-info build on a 3-core runner + +The measured local data and the external sources agree that `codegen-units=1` is the main compile-time lever. With one codegen unit the crate is a single LLVM module, so the parallel LLVM stage has nothing to fan out; rustc's own docs say 1 "may improve the performance of generated code, but may be slower to compile" ([rustc book](https://doc.rust-lang.org/rustc/codegen-options/index.html)), and a measured debug build of one crate took 9.7 s with one unit against 4.5 s with up to 16 ([Nethercote](https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html)). Here the local ratio was 1260 s to 673 s, about 1.9x, on a loaded 12-core machine; on a 3-vCPU runner the idle cores are the same, and the compile is single-thread bound for the biggest crates. A ruff measurement quoted in a commit view gave thin LTO plus 16 units at 16 s against 41 s for thin LTO alone ([ruff commit](https://app.semanticdiff.com/gh/astral-sh/ruff/commit/3ce145c47669645c9031ec42ef9b339835e548c9)), a secondary source I could not verify independently. The price is runtime speed and code size; no 2024-2026 benchmark of thin LTO with 1 versus 16 units was found, so the cost to the shipped app must be measured, not assumed. + +Where the external research disagrees with the CI history is debug info. The local build says the symbol profile is free (1260 s against 1299 s), yet macOS CI doubled when it was enabled. No source apportions debug-info cost between rustc, LTO and linking, and none measured `line-tables-only` against `debug=0`. The ThinLTO-with-debug-metadata cost is a hypothesis, not evidence. The better-supported candidate is memory: line-tables-only adds `.debug_info`, `.debug_abbrev`, `.debug_str` and `.debug_aranges` ([rust-lang/rust#109808](https://github.com/rust-lang/rust/pull/109808), via search summary), which raises rustc's resident size, and a 7 GB machine that is already short at 8.7 GB peak has no headroom. That would explain why Linux and Windows (16 GB) were unaffected and the 16 GB, 4-vCPU macOS Intel runner is the only other macOS data point. The 2026 doubling is therefore a hypothesis that one cheap experiment can test: log peak RSS and swap on a macOS arm64 run. + +For packed dSYMs the evidence is weaker locally than the folklore suggests. `split-debuginfo=packed` runs `dsymutil` after link; one 2021 Cargo debug build spent about 7 of 10 s there ([internals](https://internals.rust-lang.org/t/help-test-faster-incremental-debug-macos-builds-on-nightly/14016)), and a 2025-26 LLVM thread timed classic `dsymutil` at 1:38 wall and 143% CPU against 45 s wall with `--linker=parallel` ([LLVM discourse](https://discourse.llvm.org/t/llvm-dsymutil-performance/89051)). That is a C++ link, and the 353 MB local dSYM finished in under a minute, so on the 3-core runner `dsymutil` is likely tens of seconds to a few minutes, not tens of minutes (inference). Cargo's own default on macOS is `unpacked`, so `packed` is an explicit choice. Cheaper routes to Breakpad symbols all lack proof: `dump_syms` documents DWARF in ELF and standalone PDB inputs and says nothing about debug-map inputs ([dump_syms](https://github.com/mozilla/dump_syms)), so `unpacked` still needs `dsymutil` before dumping (inference). Running `dsymutil` only on the shipped binaries rather than on every bin and test target, or adopting its parallel linker when the Xcode on the runner has it, is plausible; whether Xcode ships it was not found. Debug info for first-party crates only (`[profile.release.package."*"] debug = false`) is supported and leaves dependencies without line info but with function names and unwind data ([Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html)); it also keeps the dependency cache profile-identical only if both builds set it. A known `ThinLTO` plus `packed` bug ([rust-lang/rust#132677](https://github.com/rust-lang/rust/issues/132677), status unchecked) is worth a glance. The macOS-only `CfiError` has no documented cause; it matters only if symbol quality degrades, and the local evidence says output is complete. + +## Ranked fixes for Windows and macOS runners + +The ranking weighs saved time against effort, starting with what the measurements prove. + +1. **Compile each crate once per Windows job (saves up to about 40 min on the Windows arm64 row, low effort).** The six compiles of `bibcode-server` come from separate `cargo check --all-targets`, `cargo test -p`, `--lib` and `--test` invocations plus the release build. The release profile cannot be shared with the test builds, but `check` is metadata-only and not reused by `cargo test` ([cargo check docs](https://doc.rust-lang.org/cargo/commands/cargo-check.html)), so replace it with `cargo test --no-run` (or `cargo nextest archive`) once, then run the three selections from that build ([nextest](https://nexte.st/book/reusing-builds)). Alternating `-p` and `--workspace` can also change the unified feature set and recompile dependencies ([Cargo features](https://doc.rust-lang.org/cargo/reference/features.html)); identical package and feature selection, or a `cargo-hakari` workspace-hack, prevents it. The fixed `-j 2` on a 4-vCPU Windows runner halves compile parallelism ([Cargo config](https://doc.rust-lang.org/cargo/reference/config.html)); the 17 min 30 s `--lib` build is the first thing to retest without it. That the cargo selections truly rebuild is inferred from the timings, which show build times of minutes against 16 s for the last step; confirm with `cargo ... -v`. +2. **Stop the cache thrashing (saves 5-7 min per row and removes variance, low effort, about $0.50 per month if billed).** Save only on `push` to `main`, drop PR-ref pnpm saves, and try raising the limit by API. +3. **Take the symbol build off the 7 GB macOS arm64 runner or measure memory first (potentially 35-40 of 80 min, medium effort).** Options: run the symbol bundle only on tag or release workflows; use a larger runner (GitHub larger macOS runners cost about $0.102 per minute for the M2 Pro tier, third-party M4 about $0.08 ([compiler.dev](https://compiler.dev/guides/faster-github-actions-runners)), prices not verified against vendors); or build the macOS arm64 symbols on the 14 GB Intel runner, supported until about 2027 by unofficial sources. Log peak RSS first, because without it the cause remains a hypothesis. +4. **Raise `codegen-units` in the symbol or PR profile (halves compile in the local test, medium effort).** The local result (673 s against 1260 s) is the strongest speed evidence, but memory rises to 12 GB at `bibcode-server`, which will not fit the 7 GB runner. Use it on Linux and Windows 16 GB rows, or on PR builds only, and keep `codegen-units=1` for the shipped binary after measuring the runtime cost. Setting it by `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` also changes the rust-cache environment hash. +5. **Windows file-system tuning (uncertain, one-line try).** Moving TEMP to D: cut a pip suite 10-28% on hosted Windows runners while ReFS Dev Drive saved little ([sichard.ca](http://sichard.ca/blog/2025/03/faster-pip-ci-on-windows-d-drive)); Rust numbers do not exist, and `cargo` process spawning is slow on Windows ([nextest](https://nexte.st/docs/design/why-process-per-test/)). Put `CARGO_TARGET_DIR`, TEMP and `CARGO_HOME` on D: and measure. +6. **Dev and test profile tweaks.** `debug="line-tables-only"` or `0` for dev and test, `CARGO_INCREMENTAL=0` ([corrode](https://corrode.dev/blog/tips-for-faster-ci-builds/)), and `[profile.dev.package."*"] debug = 0` shrink targets and caches. Gains were not sourced as numbers. `-Z threads` and Cranelift were still nightly-only in the 2026 project goals ([Rust goals](https://rust-lang.github.io/rust-project-goals/2026/parallel-front-end.html)) and are not options for stable 1.98. Linker swaps do not apply: lld is default only on x86_64 Linux ([Rust 1.90](https://blog.rust-lang.org/2025/09/18/Rust-1.90.0/)), the macOS and Windows alternatives were not benchmarked, and linking happens after the LTO phase that dominates ([Nethercote](https://nnethercote.github.io/2023/07/11/back-end-parallelism-in-the-rust-compiler.html)). Third-party runners (Linux 4 vCPU at $0.008 per minute on two vendors) have no Rust-specific numbers, and no Windows price was found. +7. **Flaky Windows arm64 test.** Wait for a child-emitted readiness marker before starting the timeout clock, answer ConPTY's startup query, and run the PTY test alone, which is how other projects fixed ConPTY flakes ([codex-rs](https://upd.dev/layers/codex/src/branch/main/codex-rs/utils/pty), [issue 36328](https://claudeissues.com/issue/36328-60s-startup-delay-in-conpty-da1-sentinel-never-receives-response)). Cheap, and not a build-time factor. + +## Conclusion + +The repository's measurements overturn the premise that a caching or backend change fixes slow builds: the cache is saturated, but removing it entirely would cost 5-7 minutes on rows that spend 80-110. The surviving explanations are structural: first-party code compiled repeatedly (five times on Windows arm64), one-thread-bound codegen with `codegen-units=1`, and a symbol build that doubles macOS time only on the smallest runner. The only unexplained gap is why the local symbol profile is free while CI is not, and a peak-memory log from a macOS arm64 run is the experiment that closes it; until then, treat the memory theory as the leading hypothesis and keep the first three fixes, which do not depend on it. + +Unverified and unreconciled items: public-repo eligibility for paid cache limits; `cache-workspace-crates` and sccache effects on this workload; whether Xcode's `dsymutil` has the parallel linker; the `dump_syms` CFI error cause; all third-party pricing; the ruff and Phoronix claims (secondary views or snippets only). diff --git a/docs/plans/2026-11-14-ci-build-times/research/rust_cache_behaviour.md b/docs/plans/2026-11-14-ci-build-times/research/rust_cache_behaviour.md new file mode 100644 index 0000000000..8ef0c51906 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/rust_cache_behaviour.md @@ -0,0 +1,147 @@ +# Swatinem/rust-cache behaviour and Rust CI cache sizing (as of Nov 2026) + +Method note: source files were read via a summarising fetch tool (raw.githubusercontent.com, master branch), not line-by-line by me, so code quotes are as relayed. Master may be slightly ahead of v2.9.2. Changelog has no dates. + +## 1. What goes into the cache key (the two hashes) + +### Takeaway + +Key = `prefix[-shared-key | -key-job] -OS -arch - -`, both hashes SHA-1 truncated to 8 hex chars. The last hash covers config/toolchain files, normalized Cargo.toml files (local versions zeroed) and only registry/git entries of Cargo.lock. A workspace version bump should NOT change it; dependency, feature, profile, `[patch]`, `.cargo/config.toml` or toolchain-file edits do. + +### Cited Findings + +- Prefix starts at `prefix-key` (default `v0-rust`); `shared-key` is appended if set, otherwise `key` plus job id; OS and arch are always appended — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) +- First hash (env hash): `rustc -vV` release/host/commit-hash, plus (since 2.9.0) every installed rustup toolchain, sorted; plus every env var whose NAME starts with CARGO, CC, CFLAGS, CXX, CMAKE, RUST (or prefixes from `env-vars`) as `NAME=value`, sorted — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts), [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md), [CHANGELOG 2.9.0](https://raw.githubusercontent.com/Swatinem/rust-cache/master/CHANGELOG.md) +- This hash is also the `restoreKey` (prefix + env hash, no lock hash) — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) +- Second hash (the one that changed 3x): raw bytes of `.cargo/config.toml`, `rust-toolchain`, `rust-toolchain.toml` (root and under each workspace root), each member `Cargo.toml` plus workspace root `Cargo.toml`, and Cargo.lock — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) +- Cargo.toml normalization: `package.version` / `workspace.package.version` set to "0.0.0"; for path dependencies (in dependencies, build-, dev-, workspace.dependencies) version set to "0.0.0" and path blanked; the normalized JSON is hashed. All other content (deps, features, `[profile]`, `[patch]`, lints, metadata) still counts — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) +- Cargo.lock: only packages with `source` or `checksum` are hashed (workspace-local crates excluded), so bumping workspace member versions does not alter it; lock versions other than 3/4 fall back to raw bytes — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) +- 2.7.2 changed the key to depend only on Cargo.toml/Cargo.lock of workspace members; 2.9.1 fixed a "regression in hash calculation"; 2.9.2 fixed rustc version sorting/dedup — [CHANGELOG](https://raw.githubusercontent.com/Swatinem/rust-cache/master/CHANGELOG.md) +- Env vars are in the FIRST hash, not the last. `RUSTFLAGS`, `CARGO_*` set in CI (e.g. `CARGO_INCREMENTAL`, `CARGO_PROFILE_*`, `CARGO_BUILD_JOBS`, `CARGO_TERM_*`) feed it — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts) + +### Inferences + +- In `...-00d9be57-c5cda4df` the first part is env/toolchain, the last is manifests/lock/config. Three changes in a week = lockfile registry entries changing (dependabot/renovate, `cargo update`), or Cargo.toml dependency/feature/profile edits, or `.cargo/config.toml`/`rust-toolchain` edits. A version-only bump is ruled out by the normalization (verify with the "Cache key" debug output in the action log, which lists key files). +- Any Cargo.toml that is NOT a workspace member (e.g. fixtures) is not hashed; any `.cargo/config.toml` under the workspace root IS (glob `**/.cargo/config.toml`). +- Case-insensitive match is used for "full match" (2.9.0). + +### Gaps + +- I did not confirm whether a rustc patch bump on the runner image (hosted macOS images update rustup toolchains) changes the first hash; by the code it would (commit-hash included), but I saw no issue stating it. + +## 2. Restore on partial match, and when it saves + +### Takeaway + +Exactly one restore-key is passed: prefix + env hash. A lock-hash change therefore gives "full match: false" and restores the newest entry sharing the env hash (GitHub picks the most recent prefix match), then saves a NEW entry at the end of the job (if `save-if` is true). Different env hash (rustc/RUSTFLAGS/CARGO_* change) = no restore at all. + +### Cited Findings + +- `restoreCache(paths, config.cacheKey, [config.restoreKey])` with a single fallback; exact-match compare is case-insensitive; log line "...full match: ${match}" — [restore.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/restore.ts) +- On a partial match the action runs `cleanTargetDir(target, [], true)` (timestamp mode, drops entries older than 1 week) before the build, sets cache-hit=false — [restore.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/restore.ts), [cleanup.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/cleanup.ts) +- Save: skipped if `save-if` != "true"; skipped with "Cache up-to-date" on an exact hit; otherwise cleanup (target dir, registry, bin if cache-bin, git) then `saveCache` — [save.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/save.ts) +- README: `cache-on-failure` defaults false (save even if workflow failed) — [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md) +- GitHub restore order: exact key on current branch, prefix on current branch, each restore-key, then same on default branch; most recent among prefix matches wins; PR runs read base/default branch caches; PR-created caches are scoped to the merge ref — [GitHub docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) +- GitHub limits: 10 GB default per repo, oldest-by-last-access evicted when over limit ("cache thrashing"), entries unused 7 days removed — [GitHub docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) + +### Inferences + +- With `save-if` default-branch-only, each lock change on main writes a fresh 1.5-1.85 GiB entry per OS/arch; with several OS/arch rows and several recent lock hashes the 10 GB repo budget is easily exceeded, evicting live entries (explains "No cache found"). This is an inference from limits above plus the measured entry sizes. +- Feature branches/PRs can only restore main's entries; a PR that changes Cargo.lock gets a partial match (or none if env hash differs). + +### Gaps + +- Whether `cache-on-failure` saves happen before cleanup differently: save.ts as fetched shows no reference to job status (handled via action.yml post-condition, not read). + +## 3. What is deleted before saving; workspace crates + +### Takeaway + +Cleanup keeps only dependency artifacts. By default the workspace's own crates (and their build-script outputs/fingerprints) are deleted, `incremental/` is deleted, `~/.cargo/registry/src` is deleted, and binaries that existed before the action ran are removed from `~/.cargo/bin`. So a "full match" restores dependencies only; the workspace crates rebuild every run. + +### Cited Findings + +- README: removes incremental artifacts, unused deps and non-deps, artifacts with mtime older than a week; does not cache workspace crates; does not cache `~/.cargo/registry/src`; sets `CARGO_INCREMENTAL=0`; removes pre-existing `~/.cargo/bin` files — [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md) +- Profile dir: only `build`, `.fingerprint`, `deps` kept; within them only entries matching the `packages` list (names/targets, `lib` prefix, hash suffix stripped); `tests` dirs keep only `target` and `trybuild`; top-level files other than CACHEDIR.TAG removed — [cleanup.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/cleanup.ts) +- `packages` = `getPackagesOutsideWorkspaceRoot` (cargo metadata, filtered to targets of kind in SAVE_TARGETS); workspace members are appended only when `cache-workspace-crates: true` — [save.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/save.ts), [workspace.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/workspace.ts) +- Registry: `credentials.toml` deleted; `registry/src` kept only for `-sys` crates; `.crate` files and index cache pruned to used crates; git db/checkouts pruned to referenced ones — [cleanup.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/cleanup.ts) +- `cache-workspace-crates` added in 2.8.0 — [CHANGELOG](https://raw.githubusercontent.com/Swatinem/rust-cache/master/CHANGELOG.md) +- A repo PR (#349) discusses that caching workspace targets grows cache count/size because source changes create new keys; status unclear — [search result](https://github.com/Swatinem/rust-cache/pull/349) + +### Inferences + +- Because pruning is based on crate name, BOTH debug and release profile dirs keep every dependency's artifacts, i.e. a job building two profiles holds two full dependency sets (plus macOS `.dSYM`/debuginfo in dev). Enabling `cache-workspace-crates` would add workspace objects but, by the Cargo fingerprint design, not help unless sources/mtimes match; I found no source on this mtime behaviour in this session, so treat as unverified. +- The 40-110 min release time "even on full match" is consistent with workspace crates always being rebuilt (and thin-LTO/codegen cost); it is not a cache miss. + +### Gaps + +- No authoritative measurement of how much `cache-workspace-crates: true` helps/grows a cache. + +## 4. Options controlling size and scope + +### Takeaway + +Defaults and semantics from the README; the levers that shrink size are `cache-targets`, splitting by `key`/`shared-key` per profile, `cache-bin: false`, and `cache-all-crates` left false. + +### Cited Findings (defaults per [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md)) + +- `prefix-key` v0-rust (bump to force fresh cache); `shared-key` empty (replaces job-id key); `key` empty (extra separator); `add-job-id-key` true; `add-rust-environment-hash-key` true (if false, key = prefix only, no hashes — [config.ts](https://raw.githubusercontent.com/Swatinem/rust-cache/master/src/config.ts)) +- `env-vars` default `CARGO CC CFLAGS CXX CMAKE RUST`; `workspaces` default `. -> target`; `cache-directories` empty; `cache-targets` true (false = only cargo registry/git cached) +- `cache-all-crates` false (true also keeps crates installed by CI tooling); `cache-workspace-crates` false; `save-if` true; `lookup-only` false (sets cache-hit without restoring); `cache-bin` true (cleanup deletes pre-existing bins; false for long-lived self-hosted runners); `cmd-format` `{0}` (wrappers/nix) +- `cache-provider`: `github` default, `warpbuild` documented (added 2.8.0); BuildJet was not listed in the README I read — [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md), [CHANGELOG](https://raw.githubusercontent.com/Swatinem/rust-cache/master/CHANGELOG.md) +- `lookup-only` added 2.7.6; arch added to key 2.7.8 — [CHANGELOG](https://raw.githubusercontent.com/Swatinem/rust-cache/master/CHANGELOG.md) + +### Inferences + +- Two profiles in one job share one key; using `key: debug` / `key: release` in separate steps (each with its own save) gives smaller, independently evicted entries, and the lock hash then matches per entry. Not tested. + +### Gaps + +- BuildJet/other providers: no source found in README; I could not confirm any provider besides github and warpbuild. + +## 5. Size reducers for target/ in CI + +### Takeaway + +Only Cargo-documented settings are sourced here; the rest (cargo-sweep, cargo-cache, zstd level) I could not source in this session. + +### Cited Findings + +- `debug` accepts `0`/`false`/`"none"`, `"line-tables-only"`, `1`/`"limited"`, `2`/`"full"`; dev default is full, release none. Named values need Rust 1.71 — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) +- `strip` default "none", values "debuginfo"/"symbols"; `split-debuginfo` on macOS defaults to `unpacked` when debug info is on — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) +- `incremental` is only for workspace members and path deps, controllable via `CARGO_INCREMENTAL`; rust-cache already sets 0 and deletes incremental dirs — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html), [README](https://raw.githubusercontent.com/Swatinem/rust-cache/master/README.md) +- Profile settings come only from the workspace root; `[profile.dev.package."*"] debug = 0` style overrides can shrink dependency debug info without touching workspace crates — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) +- Build-override defaults: opt-level 0, codegen-units 256, debug false for build scripts/proc macros — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) + +### Inferences + +- `debug = "line-tables-only"` (or `0`) for dev/test and `[profile.dev.package."*"] debug = 0` is the lowest-risk way to cut the debug profile's share (debuginfo typically dominates dev target size; no number sourced here). On macOS `unpacked` split-debuginfo leaves `.o` files in deps that are cached; `debug = 0` avoids them. + +### Gaps + +- No source found for: cargo-sweep, cargo-cache usage with rust-cache, zstd level tuning of actions/cache, measured size effect of `strip`/`debug=0`. + +## 6. sccache vs rust-cache for a workspace dominated by its own crates + +### Takeaway + +sccache caches per rustc invocation, so it can cache workspace lib crates that rust-cache discards, but it cannot cache anything that invokes the linker (bin, dylib, cdylib, proc-macro; build scripts are bins), requires incremental off, and Depot measured it slowing release builds. For thin-LTO/codegen-units=1 I found no documentation either way. + +### Cited Findings + +- Not cacheable: crates invoking the system linker (bin, dylib, cdylib, proc-macro); requires `--emit` link/metadata/dep-info, `--crate-name`, `--out-dir` — [sccache Rust.md](https://raw.githubusercontent.com/mozilla/sccache/main/docs/Rust.md), [README](https://raw.githubusercontent.com/mozilla/sccache/main/README.md) +- sccache errors if `CARGO_INCREMENTAL=1` unless `SCCACHE_ALLOW_INCREMENTAL=1`, which leaves workspace members/path deps uncached — same sources +- Hits across directories carry the original paths in debuginfo/panic messages unless `--remap-path-prefix`; `SCCACHE_BASEDIRS` exists — [README](https://raw.githubusercontent.com/mozilla/sccache/main/README.md) +- GHA backend: rate limits can skip stores silently; `SCCACHE_GHA_RW_MODE=READ_ONLY`; `SCCACHE_GHA_VERSION` change purges — [GHA.md](https://raw.githubusercontent.com/mozilla/sccache/main/docs/GHA.md) +- Depot (Zed, 8 cores): warm sccache 25m18s vs 28m36s baseline (11.5% faster), cold 16.8% slower; FAQ says sccache speeds test compilation 11-14% but can slow release builds up to 50%, suggesting plain cargo for release — [Depot guide](https://depot.dev/blog/guide-to-faster-rust-builds-in-ci.md) +- RisingWave (Apr 2026, vendor blog): no-cache release 21m40s vs 1m52s warm sccache on a different provider's runner — via search summary of [avrea.com](https://avrea.com/blog/risingwave-sccache-ci-benchmark), not opened by me + +### Inferences + +- Since the final bin/proc-macro/build-script steps are uncacheable, a Tauri app's final link stays; but lib crates of the workspace (the bulk of 40-110 min if codegen-heavy) could hit. Whether thin LTO objects are cached is unknown: LTO runs in the rustc invocation for lib crates only when requested, and neither doc addresses it. + +### Gaps + +- sccache + thin LTO / codegen-units=1: no source found; must be tested with `sccache --show-stats` non-cacheable reasons. +- sccache Windows/macOS-specific limits: none found in docs fetched this session (README only mentions static linking). Not answered. +- No clean rust-cache vs sccache benchmark on a workspace dominated by own crates. diff --git a/docs/plans/2026-11-14-ci-build-times/research/rust_ci_speedups.md b/docs/plans/2026-11-14-ci-build-times/research/rust_ci_speedups.md new file mode 100644 index 0000000000..57dea9a234 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/research/rust_ci_speedups.md @@ -0,0 +1,173 @@ +# Rust CI speedups on GitHub-hosted Windows and macOS runners (state as of 2026-11-14) + +Research budget was limited (about 16 calls). Several questions have thin or no primary evidence; they are listed under Gaps. Items marked "Inference" are not sourced. + +## 1. Runner specs, relative speed, why Windows is slower, Windows mitigations + +### Takeaway + +Standard public-repo runners are 4 vCPU / 16 GB on Linux and Windows (x64 and arm64), 3 vCPU M1 / 7 GB on macOS arm64, and 4 vCPU / 14 GB on macos-15-intel. macOS arm64 is the smallest box. Windows slowness is mainly process-spawn and file-system cost; the measured fix on hosted runners is moving TEMP/work to D:, not ReFS Dev Drive. + +### Cited Findings + +- Specs (public repos): ubuntu-24.04, ubuntu-24.04-arm, windows-2025, windows-11-arm = 4 vCPU, 16 GB RAM, 14 GB SSD. macos-26 and macos-15 = 3 vCPU (M1), 7 GB, 14 GB. macos-15-intel = 4 vCPU, 14 GB, 14 GB. — [GitHub docs, hosted runners](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) +- macOS runners may start slowly in high demand; nested virtualization is unsupported on arm64 macOS. — same page +- macos-15-intel is reported as the last x86_64 image, supported until about August 2027 (WordPress Trac ticket) or "Fall 2027" (other sources). No official GitHub notice was found. — [WordPress Trac #64246](https://core.trac.wordpress.org/ticket/64246); [macos-13 closing notice](https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/) (for context) +- nextest docs: process creation is "quite slow on Windows" and also slow on macOS (anti-malware); Windows nextest is "usually slower than on Unix". Docs recommend a Dev Drive (ReFS) for CARGO_HOME, source repo and target dir, or Defender exclusions for the same three paths. No numbers given. — [nextest why-process-per-test](https://nexte.st/docs/design/why-process-per-test/); [nextest Windows](https://nexte.st/docs/installation/windows/) +- Measured on GitHub Windows runners (pip test suite): moving TEMP to D: cut job time 28% (17m37 to 12m41; 14m58 to 10m44) on Python 3.8 and 10-11% on 3.13. A ReFS Dev Drive saved only about 1 minute on one job versus about 5 minutes for the TEMP move. Author presumes Defender scanning is already off on GitHub runners; no Defender timing measured. — [sichard.ca blog (formerly ichard26.github.io)](http://sichard.ca/blog/2025/03/faster-pip-ci-on-windows-d-drive) +- uv's setup script comment (as summarized by search tool): the runner's D: drive performs similar to or better than a ReFS Dev Drive; they saw no change after switching. Dioxus has a setup-dev-drive.ps1 for Rust CI. — [uv setup-dev-drive.ps1 mirror](https://git.joshthomas.dev/mirrors/uv/src/commit/e7f596711114528d98af1d20c91ce05f8462c703/.github/workflows/setup-dev-drive.ps1); [Dioxus script](https://code.morphllm.com/DioxusLabs/dioxus/src/branch/main/.github/workflows/setup-dev-drive.ps1) (search-snippet level only, not fetched) +- Microsoft claims Dev Drive gives up to ~30% faster builds versus standard Windows 11, not measured on GitHub runners. — [Microsoft devblog, 2023](https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/) +- ARM64 Linux: a blog reports up to 40% better performance than x64 Linux for the same job, with queue time possibly offsetting it (anecdotal). — [corrode.dev](https://corrode.dev/blog/tips-for-faster-ci-builds/) + +### Inferences + +- Put CARGO_TARGET_DIR (and TEMP/TMP, CARGO_HOME) on D: on Windows rows first; it is a one-line env change. Try ReFS only if that is not enough. +- Because the C: drive has 14 GB and D: is larger on Windows images (not verified here), C: pressure from a big target dir may also matter. +- Windows ConPTY/process-spawn cost explains why test steps that spawn many processes (PTY tests, nextest) are disproportionately slow. + +### Gaps + +- No relative Rust compile benchmark across windows-2025, windows-11-arm, macos-26, macos-15-intel, ubuntu-24.04-arm found. +- No evidence on whether Defender real-time scanning is actually on for hosted Windows images; only the pip author's presumption. +- windows-11-arm and macos-26 GA/public dates not confirmed beyond their presence in the current table. + +## 2. Multiple `cargo test` invocations vs `--no-run` + direct runs vs nextest archives + +### Takeaway + +Nextest builds with `cargo test --no-run` internally and supports `cargo nextest archive` plus `run --archive-file`, with partitioning across jobs. Plain-cargo rebuild behavior for different `--test`/`--lib` selections was not confirmed from a source. + +### Cited Findings + +- nextest first builds all test binaries with `cargo test --no-run`, then lists tests by querying the binaries. — [nextest how it works](https://nexte.st/book/how-it-works.html) +- `cargo nextest archive --archive-file x.tar.zst` stores test binaries, cargo metadata, and the build options; `cargo nextest run --archive-file` reuses them; `--workspace-remap` handles a different checkout path; partitioning (`--partition count:N/M`, hash or slice) splits tests across jobs. The docs link a GitHub Actions example. — [nextest reusing builds](https://nexte.st/book/reusing-builds); [partitioning](https://nexte.st/book/partitioning) +- Nextest claims up to 3x over cargo test; one blog saw about 40% in CI because runners have few cores. — [corrode.dev](https://corrode.dev/blog/tips-for-faster-ci-builds/) +- Windows caveat: nextest runs one process per test, and Windows process creation is slow; benchmark on your workflow. — [nextest Windows](https://nexte.st/docs/installation/windows/) +- Cargo config: `CARGO_BUILD_JOBS`/`build.jobs` defaults to logical CPUs; negative values mean CPUs minus N; `--jobs` overrides. A fixed `-j 2` on a 4 vCPU box halves compile parallelism. — [Cargo config](https://doc.rust-lang.org/cargo/reference/config.html) + +### Inferences + +- (Unverified by source) Cargo fingerprints units by profile, features and sources; filters like `--test a` or `-- filter` are runtime arguments, so a second invocation with identical flags and package selection should be a no-op build. A rebuild occurs when `-p`, `--workspace`, `--all-targets`, `--features` or `--profile` differ between invocations, because feature sets differ. Verify with `cargo test --no-run -v` or `CARGO_LOG=cargo::core::compiler::fingerprint=trace`. +- Cheapest change: one `cargo test --no-run` (or `cargo nextest archive`) step, then run the three selections from it. On macOS/Windows run the archive in the same job; cross-job reuse needs an archive per OS/arch (archives are platform-specific; not confirmed by source). +- `-j 2` on the test runs only limits the cargo build parallelism; test execution parallelism is `--test-threads`/nextest threads, a separate knob. + +### Gaps + +- No source confirming that `cargo test` with different `--test`/`--lib` selection rebuilds or relinks. The Cargo book `cargo test` page was not fetched. +- No measured numbers for archive vs repeated cargo test on Windows/macOS. + +## 3. Does `cargo check --all-targets` share artifacts with `cargo test`? + +### Takeaway + +Only metadata (`.rmeta`) is shared across check runs; `check` does not do codegen, so it does not produce test binaries and is not reused by `cargo test`. Replacing the check step with `cargo test --no-run` (or `cargo check` removal) is the cheapest way to prove test targets compile. + +### Cited Findings + +- `cargo check` compiles without final code generation; some errors appear only at codegen, so check can pass where build fails. — [cargo check docs](https://doc.rust-lang.org/cargo/commands/cargo-check.html) +- To check test code: `cargo check --all-targets --profile=test`. — same page +- A 2026 Rust project goal proposes letting build reuse work done by check; this is a proposal, not current behavior. — [Rust project goal: incremental system rethought](https://rust-lang.github.io/rust-project-goals/2026/incremental-system-rethought.html) +- sccache does not cache check output (metadata only), per one personal blog (anecdotal). — [blog](https://tom-tang-3-ai-site.pages.dev/blog/why-cargo-ate-45gb-overnight/) + +### Inferences + +- `cargo check --all-targets` uses the check profile and separate fingerprints; it compiles every dependency again as metadata only. It is extra work in a job that later runs `cargo test`. Dropping it on Windows in favor of `cargo test --no-run` (which also catches codegen and link errors) is the likely saving. Keep `cargo check --all-targets` on Linux only if you want the cheap lint-like gate. +- `--all-targets` includes benches/examples, so `cargo test --no-run --workspace --all-targets` would build those with the test profile; narrow it to what tests actually need. + +### Gaps + +- No measurement of check vs test-build cost for the user's workspace type; no source on what `cargo test --no-run` shares with a later `cargo build --release` (nothing: different profile). + +## 4. Feature unification and second compiles of the dependency graph + +### Takeaway + +Cargo builds one copy of a dependency per distinct feature set. With resolver 2/3, dev-dependency, build-dependency and platform features no longer unify unless needed, and selecting `-p crate` versus `--workspace` can change the unified feature set, so alternating them in CI can recompile large parts of the graph. + +### Cited Findings + +- Features unify as a union across packages that depend on the same crate; resolver 2 stops unifying for platform-specific deps, build-deps/proc-macros, and dev-deps (unless building targets that need them). Trade-off: a dependency may be built multiple times with different features. Suggested fix: `cargo tree --duplicates` and list features explicitly so they unify. — [Cargo book, Features](https://doc.rust-lang.org/cargo/reference/features.html) +- With resolver 2, `--features` and `--no-default-features` apply across all `-p`/`--workspace` selected packages. — same page +- Inspection: `cargo tree -e features`, `cargo tree -e features -i `, `cargo tree -d`. — same page +- cargo-hakari generates a `workspace-hack` crate that pins the union of features so every dependency compiles once across all member selections. A packaging page claims 15-95% faster local build/check, 20-25%+ cumulative; this is a description, not a benchmark. — [cargo-hakari docs](https://docs.rs/cargo-hakari/latest/cargo_hakari/about/index.html); [Nix package page](https://mynixos.com/nixpkgs/package/cargo-hakari) + +### Inferences + +- If CI uses `cargo test -p server` plus a workspace-wide desktop build or `--all-targets` check, the graph likely compiles under 2+ feature sets (for example, Tauri/desktop features versus server-only). Detect with `cargo build --workspace -v` vs `-p` timing logs, or compare the `-C metadata` hashes in `target/debug/deps` for the same crate name. +- Fixes in order of effort: use identical package/feature selection across all CI cargo steps; add hakari workspace-hack; or set the unstable/new `resolver.feature-unification` option (not verified). + +### Gaps + +- `resolver = "3"` and `resolver.feature-unification` (Cargo config) were not documented in the pages fetched; the Cargo config page lists only lockfile-path, incompatible-rust-versions, incompatible-publish-age under `[resolver]`. Status unverified; resolver 3 is believed to concern MSRV-aware resolution, not feature unification (unverified). +- No published before/after hakari timings found. + +## 5. Dev/test profile tweaks + +### Takeaway + +Cheap, documented knobs: `debug = "line-tables-only"` or `0` for dev/test, `CARGO_INCREMENTAL=0` in CI, `split-debuginfo` defaults, and `opt-level` overrides for dependencies. Parallel front end and Cranelift are still not stable as of the 2026 goals pages found. + +### Cited Findings + +- `test` inherits `dev`; `dev` default is `debug = true` (full). `debug = "line-tables-only"` needs Rust 1.71+ and keeps backtraces with file/line. `split-debuginfo` defaults to `unpacked` on macOS when debug info is on; other defaults are platform-specific. — [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) +- `[profile.dev.package."*"] opt-level = 1/2` applies to dependencies only; the docs suggest trying `opt-level = 1` for dependencies. Build-overrides apply to build scripts and proc macros. — same page +- Cargo sets `incremental = false` by default when the `CI` env var is set and config is unset. — [Cargo config](https://doc.rust-lang.org/cargo/reference/config.html) +- `CARGO_PROFILE_TEST_DEBUG=0` and `CARGO_INCREMENTAL=0` recommended for CI to shrink target and improve cache efficiency; no numbers. Release `lto`/`codegen-units=1` may be set via env in CI only. — [corrode.dev](https://corrode.dev/blog/tips-for-faster-ci-builds/) +- Parallel front end (`-Z threads`): still being worked toward stabilization in the 2026 project goal; 2025 goal reported 20-30+% compile-time reduction. — [Rust goals 2026 parallel front-end](https://rust-lang.github.io/rust-project-goals/2026/parallel-front-end.html) +- Cranelift: February 2026 benchmarks show 5% faster than LLVM on a Zed build with `debug-info=none`; with line tables the gain ranges from negligible to about 30%. No confirmation it is stable for dev builds. — [Rust goals 2026 cg_clif](https://rust-lang.github.io/rust-project-goals/2026/improve-cg_clif-performance.html); [fast builds roadmap](https://rust-lang.github.io/rust-project-goals/2026/roadmap-fast-builds.html) + +### Inferences + +- For a thin-LTO release bundle of 38-53 minutes: keep it as a separate (possibly Linux-gated or tag-only) job; for PR rows consider a non-LTO `release`-like profile (`lto = off`, `codegen-units` higher) via `CARGO_PROFILE_RELEASE_LTO=off` env, to be validated against the bundle's purpose. +- Windows linking: MSVC link.exe with a large debug build is slow; `debug = "line-tables-only"` shrinks PDBs and link time. `rust-lld` as a linker on Windows/macOS was not verified here. + +### Gaps + +- No measured gains for debug=0, split-debuginfo, or linkers (lld/mold/zld/ld64 new linker) on Windows/macOS. +- Whether `-Z threads` or cranelift are stable on 2026-11 stable Rust could not be confirmed beyond the project goal pages (nightly-only per those pages). + +## 6. Larger or third-party runners (public repos) + +### Takeaway + +Third-party runners are cheap for Linux but only partially documented for Windows/macOS; all price data below comes from comparison blogs by vendors/competitors and must be rechecked on vendor pages. No Rust-specific speedup numbers were found. + +### Cited Findings + +- Linux x64 per-minute (2/4/8 vCPU), early November 2026: Blacksmith and WarpBuild $0.004/$0.008/$0.016; Namespace and Depot $0.006/$0.012/$0.024 (Namespace $0.012 pay-as-you-go and $0.008 on a $100/mo plan in another source). Blacksmith 3,000 free minutes/month; Depot from $20/month with 2,000 minutes, billed per second. — [sengi_runners on dev.to](https://dev.to/sengi_runners/blacksmith-alternatives-for-github-actions-in-2026-50hi); [pricing table](https://dev.to/sengi_runners/github-actions-runner-pricing-table-2026-4g4g) +- macOS: WarpBuild M4 Pro 6 vCPU about $0.08/min; Blacksmith macOS M4 about $0.08/min; GitHub macOS M2 Pro larger runner $0.102/min. — [compiler.dev](https://compiler.dev/guides/faster-github-actions-runners) +- Blacksmith runs Windows and macOS; Namespace is recommended for macOS/Windows on a plan. No Windows per-minute price found. — [sengi_runners](https://dev.to/sengi_runners/blacksmith-alternatives-for-github-actions-in-2026-50hi); [sengi compare](https://sengi.run/compare) +- Cache storage: Blacksmith Docker layer cache $0.50/GB-month; WarpBuild cache $0.20/GB-month plus operations. — [sengi pricing traps](https://sengi.run/blog/github-actions-runner-pricing-traps) +- Depot: warm sccache with nextest gave a 35% speedup on 8-core runners. — [Depot guide](https://dev.to/depot/guide-to-faster-rust-builds-in-ci-3njl) +- Cirrus Runners: not covered by any source found. + +### Inferences + +- Moving from 4 vCPU hosted Windows to an 8-16 vCPU runner is the likeliest large win given `-j 2` currently; but larger GitHub runners are not free for public repos in the same way as standard ones (not verified here). + +### Gaps + +- No Windows runner prices; no Cirrus Runners data; no Rust compile speedup numbers per provider; GitHub larger-runner Windows/macOS prices not retrieved. + +## 7. PTY / ConPTY tests on loaded Windows arm64 + +### Takeaway + +The recurring remedy in other projects is to replace fixed timeouts with a deterministic readiness marker from the child process, and to avoid writing to ConPTY before the child is ready. + +### Cited Findings + +- Codex repo fixed Windows ConPTY flakiness from early stdin writes racing REPL startup by using a marker emitted by the child as a sync point. — [codex-rs utils/pty (mirror)](https://upd.dev/layers/codex/src/branch/main/codex-rs/utils/pty) +- A ConPTY startup stall of about 60 s was traced to a DA1 (Primary Device Attributes) query that never gets answered, with portable-pty. — [claudeissues.com #36328](https://claudeissues.com/issue/36328-60s-startup-delay-in-conpty-da1-sentinel-never-receives-response) +- Vim: fast-exiting processes lose final output in ConPTY; Vim fixed a flaky Test_terminal_cwd. — [vim_dev PR #19036](https://groups.google.com/g/vim_dev/c/UOlhQdJjgNM) +- LLDB (April 2026) moved to not use ConPTY in tests on Windows. — [lldb-commits PR #192657](https://lists.llvm.org/pipermail/lldb-commits/Week-of-Mon-20260413/167872.html) +- WezTerm issue comment: the bottleneck is the Windows pty layer rather than wezterm. — [wezterm #740](https://github.com/wez/wezterm/issues/740) +- A Rust project keeps Windows attach-latency probes behind `#[ignore]` as timing-sensitive. — [rmux windows_attach_exit.rs](https://docs.rs/crate/rmux/0.7.1/source/tests/windows_attach_exit.rs) + +### Inferences + +- For the 10 s flake: (a) answer DA1 (`ESC [ ? 1 ; 2 c`) or otherwise handle the startup query in the test harness; (b) wait for a child-emitted marker, then start the timeout clock; (c) run the PTY test serially in its own step or nextest test-group with `max-threads = 1` so it does not compete with a compile; (d) generous timeout (30-60 s) on Windows arm64 with fast failure on marker; (e) warm the shell/process once before the timed test. + +### Gaps + +- No source on alacritty or nushell CI practices; no measurement of ConPTY first-process cold start on windows-11-arm. diff --git a/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-1.md b/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-1.md new file mode 100644 index 0000000000..449922a426 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-1.md @@ -0,0 +1,19 @@ +# Plan review 1 + +Reviewer: Codex, `gpt-6.1-sol`, effort high, read-only, 2026-11-14. Verdict: Needs rework. +Every finding was checked against the files it cites before it was handled. + +| # | Severity | Finding | Handling | +| ---- | ------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Major | Moving the shim wait loop into a helper could drop the checks after the marker (exit within 10 s, exit code 0). | Fixed. Task 4.1 moves only the loop into `wait_for_marker`; `assert_shim_output` keeps its later checks, and spec R6 says so. | +| 2 | Major | The CI fallback would not run the new test, because the step's filter is `review_prompt_shim_`. | Fixed. Spec R2 and Task 3.1 widen the filter to `review_prompt_shim_ shim_timeout_`; Task 4.1 adds the Windows `cargo check --all-targets`. | +| 3 | Major | The diagnosis stated estimates as facts, and the spec cited a memory figure and a reproduction the notes did not hold. | Fixed. The notes now say the dependency split comes from start times, hold the one-unit symbol-profile memory sample (9.6 GB) and the 16-unit reproduction of the CFI error, and the spec's wording follows them. | +| 4 | Major | No proof that the override reaches Cargo through the real wrapper and leaves dependencies fresh. | Fixed. Task 2.1 step 6 reads the Cargo command line under the real wrapper, the notes record the direct rebuild that compiled only the two crates, and Task 5.3 checks the CI log. | +| 5 | Major | Task 5.2 could finish with R2's inequality and R7's issue comment undone. | Fixed. Task 5.3 accepts each requirement on the measured run and publishes the comments. | +| 6 | Minor | R1's tests missed the no-error path and unordered records. | Fixed by the new R1 (below) and its five cases. | +| 7 | Minor | The heartbeat was probed only with a successful child. | Fixed. Task 3.2 step 5 probes a failed child and a missing `ps`. | +| 8 | Minor | Round 1 checkpoints share local build outputs, and Task 4.2 wrote `progress.md`. | Fixed. The plan runs local checkpoints one after another and the executor alone writes `progress.md`. | +| Q1 | Open question | "The last `STACK CFI INIT` reaches the last `FUNC`" is not a sound completeness check. | Accepted, and the spec changed: R1 now reruns `dump_syms` on the binary alone to attribute the error and compares `STACK` counts. | +| Q3 | Open question | Restoring `-j 2` contradicted R2. | Fixed. Spec D4 names the fallback. | +| Note | | `rust-best-practices` is not in the reviewer's skill list. | Rejected: it is in the planning host's list, and the executor resolves skills on its own host. | +| Note | | `execute-plan.md` said reviewed while `progress.md` said pending. | Fixed with this record. | diff --git a/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-2.md b/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-2.md new file mode 100644 index 0000000000..e597adbe4a --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/reviews/plan-review-2.md @@ -0,0 +1,18 @@ +# Plan review 2 + +Reviewer: Codex, `gpt-6.1-sol`, effort high, read-only, 2026-11-14. Verdict: Needs rework on +acceptance wording and on the report's wording; the design (per-package override under thin LTO, +the argument route to Cargo, the binary-only rerun for R1, the Windows job) was confirmed. +This was the second and last review round. + +| # | Severity | Finding | Handling | +| --- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Major | R3's CI check (a logged `-C codegen-units=16` and no dependency compiled on a full cache match) could reject a correct change. | Fixed. Task 5.3 accepts R3 on the controlled local rebuild (same target, only the override changed, only the two workspace crates compiled) and on the wrapper probe; CI bundle times are recorded as observations. | +| 2 | Major | Task 5.3 could close with a rejected Must, and writing results to `progress.md` would start a new run. | Fixed. Every Must is accepted or covered by a fallback the spec names; the measured numbers live in the pull request. | +| 3 | Major | The report still stated estimates as measurements. | Fixed with a corrections note at the top of `research/report.md`; the notes were already corrected in round 1. | +| 4 | Minor | The contract case forbids the `-j 2` fallback that spec D4 allows. | Fixed. The fallback branch changes the contract case and records the reason. | +| 5 | Minor | No Clippy run for the Windows-only test file. | Fixed. Task 4.1 step 4 runs it in the VM, or reports it as not run. | +| 6 | Minor | Phase 1's checkpoint needed Phase 2's build. | Fixed. It uses a binary that already exists. | +| 7 | Minor | The wrapper probe had no isolation or cleanup recipe. | Fixed. Task 2.1 step 6 names the process group, the wait bound and the cleanup check. | +| 8 | Minor | "VM unavailable" had no terminal state. | Fixed. Task 4.2 closes as `not run: ` and leaves R7 unaccepted. | +| 9 | Nit | Nine tasks named, ten planned. | Fixed. | diff --git a/docs/plans/2026-11-14-ci-build-times/spec.md b/docs/plans/2026-11-14-ci-build-times/spec.md new file mode 100644 index 0000000000..7aaa44a2d8 --- /dev/null +++ b/docs/plans/2026-11-14-ci-build-times/spec.md @@ -0,0 +1,108 @@ +# CI build times, symbol dump error and Windows arm64 flake (#363, #364, #367, #368) + +Status: Approved 2026-11-14 (the user's standing pre-approval of 2026-10-30: reviewed twice by Codex and by the advisor, findings handled; decisions D2 to D6 are the agent's calls, listed for confirmation in the pull request) +Size: full Mode: gated Type: performance and chore (#363, #368), bug (#364), flaky test (#367) +Source: GitHub issues #363, #364, #367 and #368 of this repository as they stood on 2026-11-14, the conversation of 2026-11-14 ("create a new PR and fix all open issues, also do a deep research about these recurrent build time problems, check the real reason of them"), the job logs and jobs API of ten `main` CI runs (2026-11-04 to 2026-11-14), and local builds on 2026-11-14. Evidence: [research/measured_ci_and_local.md](research/measured_ci_and_local.md). Research: [research/report.md](research/report.md). + +## Problem + +CI on `main` takes two to four hours, jobs hit their time limits, and the same explanation ("no Rust cache") has been recorded each time. The measurements say otherwise: + +- **The server crate, not the cache, is the cost.** Swatinem/rust-cache never stores workspace crates. In the bundle step, Cargo starts the last dependency about 6 to 7 minutes in when nothing is cached and 2.5 minutes in on a full match; the remaining 23 minutes on Linux x64, 42 on Windows x64 and 81 to 107 on macOS arm64 are spent with only `bibcode-server`, `bibcode-desktop` and the thin-LTO link left to start. These are start times of `Compiling` lines, so the split is an estimate. Locally the three workspace units take 1125 of 1299 seconds, and with one codegen unit they compile mostly on one core. +- **A Windows native row compiles `bibcode-server` five times** (desktop host tests, `cargo check --all-targets`, two server test builds, the release bundle): 76 of 103 minutes, against about 2 minutes of running tests (#368). +- **The cache store cannot hold what CI saves.** It sits at 9.98 GiB of a 10 GB limit; the entries CI wants total about 17 GB. In the latest `main` run 11 of 13 Rust-cache restores found nothing. A Windows arm64 entry restored at 01:09 was gone at 02:08. Pull requests also save 0.48 GiB pnpm store entries that only their own re-runs can read. This costs each job 5 to 12 minutes; it is real, and it is not the main cause. +- **#363.** The macOS bundle build went from 37-40 minutes to 75-111 when the symbol profile was turned on; Linux did not change. Locally (12 cores, 96 GB) the symbol profile costs nothing (1260 s against 1299 s) and `dsymutil` is not visible in a 15-second sampler. The `rustc` process for `bibcode-server` peaks at 8.7 GB with the shipped profile and at 9.6 GB with the symbol profile (one sampled run each); the macOS arm64 runner has 3 cores and 7 GB. Memory pressure on that runner is the leading explanation and is not proven; debug-info cost in the compiler or the link on that hardware is not ruled out. +- **#364.** `dump_syms` logs `[ERROR] CFI: CfiError { kind: BadDebugInfo, ... }` in 7 of 24 macOS job logs and exits 0. Reproduced locally with the build that used 16 codegen units (the one-unit build dumped six times without the error): dumping the dSYM's DWARF file alone logs the error and yields no `STACK` records, dumping the binary alone logs nothing, and the merged file is identical in its `STACK` records to the binary-only dump (635,173 records, 0 lines differ). The dSYM's `__unwind_info` section header has file offset 0, so the parser reads unrelated bytes. In that case the stored file is complete. The same log line would also appear if the binary's own unwind data failed to parse, and then the writer stops at the failing entry and the file is truncated; nothing tells the two cases apart today. +- **#367.** Four `review_prompt_shim_*` tests timed out after 10 s on Windows arm64, once in ten runs of the step. They were the first four tests libtest started; the six started after them passed in 5 s. The test discards the terminal output before it panics, so the log cannot say whether the marker was late or never printed. + +## Goals + +1. A pull request's Windows native rows finish sooner, and no step compiles the server for tests inside a row that also builds the bundle. +2. The bundle build of the two workspace crates uses more than one core in CI, without changing the binary Release ships or invalidating the dependency cache Release restores. +3. One labelled CI run records where the macOS bundle time and memory go, so the #363 explanation is measured. +4. A symbol dump that logs a `dump_syms` error either proves the stored file complete or fails. +5. The next #367 failure prints the terminal output and exit state, and a bounded load test on Windows arm64 records whether the cold-start explanation holds. +6. Pull requests stop adding entries to the cache store. + +## Non-goals + +- Splitting `bibcode-server` into several crates. It is the structural fix for one-core compiles and is a design of its own. +- Changing the shipped release profile in `Cargo.toml` (thin LTO, one codegen unit). Release keeps building exactly what it builds today (D2). +- Raising the repository's Actions cache limit. It is an account setting with a cost, so it is the user's decision; the pull request states the numbers (D5). +- sccache, third-party runners or cache backends: no measurement supports them for a build dominated by two first-party crates. +- A larger timeout or serial execution for the #367 tests without evidence of the cause. + +## Decisions + +- D1. One pull request covers the four issues. Decided by: user, 2026-11-14. +- D2. CI bundle builds compile `bibcode-server` and `bibcode-desktop` with 16 codegen units through a Cargo `--config` override for those two packages only. Release does not pass it. Alternatives: change the shipped profile (changes the product binary and needs a runtime measurement), a `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` environment override (rebuilds every dependency, so the shared cache stops working), leave it (keeps 23 to 107 serial minutes per row). Cost: CI no longer compiles the two workspace crates exactly as Release does; it still proves the thin-LTO link, the symbol dump, stripping and bundling, and Release itself proves the shipped profile. Decided by: agent's call, to confirm with the user. +- D3. The override applies to every CI row including macOS. The macOS arm64 row's memory heartbeat (R4) decides whether it stays there: if the row's bundle build is slower than 81 minutes with it, the override is removed from that row in the same pull request. Decided by: agent's call. +- D4. The Windows server tests move to a job of their own, as the macOS ones did, and drop `-j 2` there because nothing else runs beside the compiler in that job. If the first pull request run shows that job failing for lack of memory, the flag returns and this spec records it. The `cargo check --all-targets` step moves with them. Decided by: agent's call, from the measured step times. +- D5. The cache limit stays at 10 GB in this pull request. Decided by: agent's call (it is the user's setting). +- D6. #367 is not closed by this pull request unless the load test reproduces the failure and a change flips it. The pull request refers to it. Decided by: agent's call. +- D7. A design or plan counts as approved once Codex and the advisor have reviewed it and their findings are handled. Decided by: user's standing rule of 2026-10-30. + +## Requirements + +### R1. Symbol dump verifies its output when dump_syms reports an error (Must, #364) + +`scripts/dump-symbols.ts` reads what `dump_syms` writes to standard error and still shows it. When that output contains a line with `[ERROR]`, the script finds out which input the error belongs to. With a `--dsym` input it runs `dump_syms` once more on the binary alone. If that run logs no error, the binary's unwind records were written completely, so the script requires the stored file to hold the same number of `STACK` records as the binary-only dump and prints one line saying so. If the binary-only run logs an error too, or the counts differ, or there was no `--dsym` input to set aside, it fails. + +- GIVEN a `--dsym` input, a first run that exits 0 and logs `[ERROR] CFI: CfiError { kind: BadDebugInfo, source: Some(UnknownVersion(0)) }`, and a binary-only run that logs nothing and writes the same 3 `STACK` records, WHEN the script runs, THEN it exits 0 and prints `dump_syms reported an error for the dSYM; the binary's 3 STACK records are all stored`. +- GIVEN the same first run and a binary-only run that logs the error again, WHEN the script runs, THEN it fails with `dump_syms reported an error while reading the unwind tables of the binary`. +- GIVEN the same first run and a binary-only run that writes 5 `STACK` records against 3 stored, WHEN the script runs, THEN it fails with `the stored symbols hold 3 STACK records, the binary 5`. +- GIVEN no `--dsym` input and a run that logs an `[ERROR]` line, WHEN the script runs, THEN it fails with `dump_syms reported an error`. +- GIVEN `dump_syms` logs nothing, WHEN the script runs, THEN it runs `dump_syms` once and behaves as it does today. + +### R2. Windows server tests run in their own job (Must, #368) + +A job `server_tests_windows` runs on `windows-2025` and `windows-11-vs2026-arm` on every pull request. It holds, in this order, the steps `Check Windows server test targets compile`, `Run Git system configuration tests (Windows)`, `Run background Git index lock tests (Windows)`, `Run review prompt argument tests (Windows)` and `Run host metrics tests (Windows)`, with the same commands minus `-j 2`; the review prompt step's filter becomes `review_prompt_shim_ shim_timeout_` so it also runs R6's test. The native desktop rows no longer contain them. The job restores the native desktop Rust cache and never saves one. Its time limit is 90 minutes. + +- GIVEN the workflow, WHEN the contract test reads `native_desktop`, THEN no step name ends in `(Windows)` except `Test Windows desktop E2E support contracts`. +- GIVEN the workflow, WHEN the contract test reads `server_tests_windows`, THEN it finds the five steps in order, `save-if: "false"`, `shared-key: native-desktop` and `timeout-minutes: 90`, and no command contains `-j 2`. +- GIVEN a pull request run, WHEN it finishes, THEN each Windows native row took less than 89 minutes (x64) and 103 minutes (arm64), the times of run 43562279474, and the pull request states both numbers. + +### R3. CI bundle builds use 16 codegen units for the workspace crates (Must, #363) + +`scripts/build-desktop-artifact.ts` accepts `--workspace-codegen-units ` and passes `--config profile.release.package.bibcode-server.codegen-units=` and the same for `bibcode-desktop` to Cargo in the build step. `ci.yml` passes `--workspace-codegen-units 16`. `release.yml` does not. + +- GIVEN `--workspace-codegen-units 16`, WHEN the plan is built, THEN the build command ends with `-- --config profile.release.package.bibcode-server.codegen-units=16 --config profile.release.package.bibcode-desktop.codegen-units=16` and the bundle command is unchanged. +- GIVEN no flag, WHEN the plan is built, THEN the build command is the one built today. +- GIVEN `--workspace-codegen-units 0` or a non-number, WHEN arguments are parsed, THEN the script fails with `--workspace-codegen-units must be a whole number from 1 to 256.` +- GIVEN a target built with the shipped profile, WHEN the build is repeated with the override, THEN Cargo compiles only `bibcode-server` and `bibcode-desktop`. +- GIVEN the contract test, WHEN it reads `release.yml`, THEN no line contains `--workspace-codegen-units`. + +### R4. The bundle step records time and memory (Must, #363) + +The CI bundle step writes a heartbeat every 60 seconds while it runs: elapsed seconds, the resident memory of each `rustc`, linker and `dsymutil` process, and the system's free memory and swap use. On Windows it records elapsed time only. When the step ends it prints one `::notice::` with the build's seconds and the peak single-process memory. The heartbeat never fails the step. + +- GIVEN a run labelled `ci:full`, WHEN the macOS arm64 row finishes, THEN its log holds heartbeat lines for the whole bundle build and the notice, and the pull request records the peak memory, the swap use and the build time next to the 81 to 107 minute baseline. + +### R5. Pull requests stop saving pnpm store entries (Must) + +Every `voidzero-dev/setup-vp` step with `cache: true` in the workflows sets `cache-save: ${{ github.ref == 'refs/heads/main' }}`. + +- GIVEN the workflows, WHEN the contract test reads every `setup-vp` step with `cache: true`, THEN each has that `cache-save` value. + +### R6. A shim test timeout shows what the terminal printed (Must, #367) + +When `assert_shim_output` in `apps/server/tests/windows_terminal_shims.rs` times out waiting for the marker, the panic message holds the output collected so far and the exit state. Its checks after the marker (the terminal exits within 10 seconds with code 0) stay as they are. + +- GIVEN a shim that sleeps past the deadline after printing `BEFORE`, WHEN the helper times out, THEN the panic message contains `timed out`, `BEFORE` and `exit=None`. + +### R7. A bounded load test records whether #367 reproduces (Should, #367) + +The step's command runs 30 times in the Windows arm64 VM while four processes keep its cores busy, with the count fixed before the first run. The result (failures of 30, and the output R6 prints for any failure) goes into the issue. + +- GIVEN 30 runs, WHEN at least one fails, THEN its printed output decides the next change and a comment on the issue records it. WHEN none fails, THEN a comment on the issue records `0 of 30` and the issue stays open. + +### R8. The living documents state the measured causes (Must) + +`docs/operations/ci.md` describes the new job, the override, the heartbeat, the `cache-save` rule and the measured cache budget (about 17 GB wanted against 10 GB), and no longer explains the Windows and macOS row times by a missing cache. `docs/operations/release.md` says that the `dump_syms` CFI error from a dSYM is expected and verified. The Windows testing runbook lists the server tests under their new job. + +## Verification + +- `vp test run scripts` passes, including the new cases in `scripts/dump-symbols.test.ts`, `scripts/build-desktop-artifact.test.ts` and `scripts/ci-platform-contract.test.ts`. +- `vp check`, `vp run typecheck`, `vp run check:markdown-links` and `cargo fmt --all --check` pass. +- The Windows VM compiles and runs `cargo test -p bibcode-server --test windows_terminal_shims`. +- One pull request CI run with `ci:full` is green; its step times and the macOS heartbeat are recorded in the pull request against the baselines in [research/measured_ci_and_local.md](research/measured_ci_and_local.md). diff --git a/docs/plans/README.md b/docs/plans/README.md index 9008390932..ec94576268 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -16,3 +16,4 @@ only when they do not change the recorded meaning. - [`git-manager/`](./git-manager/) — Git Manager specification, plan, phases, research (2026-08-31). - [`pull-requests/`](./pull-requests/) — Pull Requests module specification, plan, phases, research (2026-09-20). +- [`2026-11-14-ci-build-times/`](./2026-11-14-ci-build-times/) — CI build time diagnosis, research, spec and plan for #363, #364, #367 and #368 (2026-11-14). From 7187c3df00a4fb3b49a3f2d73bdde76bbe707802 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 09:52:29 -0300 Subject: [PATCH 02/11] fix(symbols): attribute a dump_syms error before storing the symbols (#364) dump_syms logs a failed unwind table and exits 0. On macOS the dSYM's unwind section header has no data and fails this way while the binary's tables are read in full; an error in the binary's own tables would truncate the records instead. When an error is logged, dump the binary alone: fail if it errors too or the STACK record counts differ. Co-Authored-By: Claude Fable 5.2 --- docs/operations/release.md | 9 ++++ scripts/dump-symbols.test.ts | 102 +++++++++++++++++++++++++++++++++++ scripts/dump-symbols.ts | 68 +++++++++++++++++++---- 3 files changed, 170 insertions(+), 9 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 87aba4839d..6ebf35f4c7 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -160,6 +160,15 @@ builds, so `dump_syms` reads the symbols before they are removed: - `RUSTFLAGS=-C link-arg=-Wl,--build-id=sha1` on Linux, so the debug ID that minidumps and symbols share never depends on the linker's default. +On macOS `dump_syms` can log `[ERROR] CFI: CfiError { kind: BadDebugInfo, ... }` and still +exit 0: the dSYM's DWARF file carries an `__unwind_info` section header with no data behind it, +and the unwind records come from the binary alone. The same line would appear if the binary's +own unwind tables failed to parse, and then the records stop at the failing entry. So when +`dump_syms` logs an error, `scripts/dump-symbols.ts` dumps the binary alone once more: it fails +if that run logs an error too, if the stored file holds a different number of `STACK` records, +or if there was no dSYM to set aside, and otherwise prints that the binary's records are all +stored (#364). + The desktop jobs run `scripts/build-desktop-artifact.ts --symbols`: `tauri build --no-bundle`, then `scripts/dump-symbols.ts`, then the strip, then `tauri bundle`. The Tauri CLI's bundle step does not run Cargo, and must not: diff --git a/scripts/dump-symbols.test.ts b/scripts/dump-symbols.test.ts index 08d7f66095..71aaaf3507 100644 --- a/scripts/dump-symbols.test.ts +++ b/scripts/dump-symbols.test.ts @@ -9,6 +9,8 @@ import { afterEach, describe, expect, it } from "vite-plus/test"; import { DumpSymbolsError, assertUsableSymbols, + countStackRecords, + dumpSymbols, parseDumpSymbolsArguments, symbolStorePath, writeZip, @@ -129,4 +131,104 @@ describe("dump-symbols", () => { expect(zip.readUInt16LE(end + 10)).toBe(1); expect(zip.readUInt32LE(zip.readUInt32LE(end + 16))).toBe(0x02014b50); }); + it("counts STACK records at the start of the file and after a newline", () => { + const sym = + "STACK CFI INIT 1000 20 .cfa: $rsp 8 +\nFUNC 1000 20 0 f\nSTACK CFI 1004 .cfa: $rsp 16 +\n"; + expect(countStackRecords(sym)).toBe(2); + expect(countStackRecords(Buffer.from(sym))).toBe(2); + expect(countStackRecords("FUNC 1000 20 0 not_a_STACK record\n")).toBe(0); + }); + + // A stand-in for dump_syms: the first run (with --store) and the binary-only rerun each write + // the configured .sym and standard error, and every invocation is recorded. + const CFI_ERROR = + "[ERROR] CFI: CfiError { kind: BadDebugInfo, source: Some(UnknownVersion(0)) }\n"; + const withStack = (count: number) => + LINUX_SYM + "STACK CFI 1004 .cfa: $rsp 16 +\n".repeat(count - 1); + function fakeDumpSyms(runs: { + readonly stored: { readonly sym: string; readonly stderr?: string }; + readonly binaryOnly?: { readonly sym: string; readonly stderr?: string }; + }) { + const root = NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "bibcode-fake-dump-syms-")); + temporaryRoots.push(root); + const calls = NodePath.join(root, "calls.log"); + const tool = NodePath.join(root, "dump_syms"); + NodeFS.writeFileSync( + tool, + [ + "#!/usr/bin/env node", + 'const fs = require("node:fs");', + `const runs = ${JSON.stringify(runs)};`, + "const args = process.argv.slice(2);", + `fs.appendFileSync(${JSON.stringify(calls)}, JSON.stringify(args) + "\\n");`, + 'const run = args.includes("--store") ? runs.stored : runs.binaryOnly;', + 'fs.writeFileSync(args[args.indexOf("--output") + 1], run.sym);', + 'process.stderr.write(run.stderr ?? "");', + ].join("\n"), + { mode: 0o755 }, + ); + const binary = NodePath.join(root, "bibcode-desktop-1234"); + NodeFS.writeFileSync(binary, "binary"); + const dwarf = NodePath.join(root, "app.dSYM", "Contents", "Resources", "DWARF"); + NodeFS.mkdirSync(dwarf, { recursive: true }); + NodeFS.writeFileSync(NodePath.join(dwarf, "bibcode_desktop-1234"), "dwarf"); + const input = { binary, moduleName: "bibcode-desktop", out: NodePath.join(root, "symbols") }; + return { + tool, + input, + withDsym: { ...input, dsym: NodePath.join(root, "app.dSYM") }, + calls: () => + NodeFS.readFileSync(calls, "utf8") + .trim() + .split("\n") + .map((line) => JSON.parse(line) as string[]), + }; + } + // The stand-in is a shebang script, which Windows cannot spawn without a shell. + // oxlint-disable-next-line bibcode/no-global-process-runtime -- A standalone script test samples its host once to skip a POSIX-only stand-in. + const onPosix = it.skipIf(NodeOS.platform() === "win32"); + + onPosix("runs dump_syms once when it reports nothing", () => { + const fake = fakeDumpSyms({ stored: { sym: withStack(3) } }); + expect(dumpSymbols(fake.withDsym, fake.tool)).toContain("crash-smoke.sym"); + expect(fake.calls()).toHaveLength(1); + }); + + onPosix("accepts an error that the binary alone does not produce: it came from the dSYM", () => { + const fake = fakeDumpSyms({ + stored: { sym: withStack(3), stderr: CFI_ERROR }, + binaryOnly: { sym: withStack(3) }, + }); + expect(dumpSymbols(fake.withDsym, fake.tool)).toContain("crash-smoke.sym"); + const [, rerun] = fake.calls(); + // One input, the staged binary under its shipped name, and no --store. + expect(rerun).not.toContain("--store"); + expect(rerun?.filter((argument) => argument.endsWith("bibcode-desktop"))).toHaveLength(1); + }); + + onPosix("fails when the binary alone reports the error too", () => { + const fake = fakeDumpSyms({ + stored: { sym: withStack(3), stderr: CFI_ERROR }, + binaryOnly: { sym: withStack(3), stderr: CFI_ERROR }, + }); + expect(() => dumpSymbols(fake.withDsym, fake.tool)).toThrow( + "dump_syms reported an error while reading the unwind tables of the binary", + ); + }); + + onPosix("fails when the stored symbols hold fewer STACK records than the binary", () => { + const fake = fakeDumpSyms({ + stored: { sym: withStack(3), stderr: CFI_ERROR }, + binaryOnly: { sym: withStack(5) }, + }); + expect(() => dumpSymbols(fake.withDsym, fake.tool)).toThrow( + "the stored symbols hold 3 STACK records, the binary 5", + ); + }); + + onPosix("fails on an error it cannot attribute: there is no dSYM to set aside", () => { + const fake = fakeDumpSyms({ stored: { sym: withStack(3), stderr: CFI_ERROR } }); + expect(() => dumpSymbols(fake.input, fake.tool)).toThrow("dump_syms reported an error"); + expect(fake.calls()).toHaveLength(1); + }); }); diff --git a/scripts/dump-symbols.ts b/scripts/dump-symbols.ts index a50dba3a11..8281cf7fac 100644 --- a/scripts/dump-symbols.ts +++ b/scripts/dump-symbols.ts @@ -74,6 +74,13 @@ function hasLineRecords(sym: string | Buffer): boolean { return [..."0123456789abcdef"].some((digit) => sym.indexOf(`\n${digit}`) !== -1); } +/** How many `STACK` records the `.sym` holds. */ +export function countStackRecords(sym: string | Buffer): number { + let count = sym.indexOf("STACK ") === 0 ? 1 : 0; + for (let at = sym.indexOf("\nSTACK "); at !== -1; at = sym.indexOf("\nSTACK ", at + 1)) count++; + return count; +} + /** * Fails unless the `.sym` can symbolicate a minidump: `STACK` records to unwind past the * crashing frame, and functions with source lines to name the frames. dump_syms reports a @@ -194,21 +201,64 @@ export function dumpSymbols(input: DumpSymbolsInput, dumpSyms = "dump_syms"): st inputs.push(stage(dwarfFile(input.dsym), NodePath.join(work, "dsym"), input.moduleName)); } const output = NodePath.join(work, "module.sym"); - const result = NodeChildProcess.spawnSync( - dumpSyms, - ["--inlines", "--check-cfi", "--store", input.out, "--output", output, ...inputs], - { shell: false, stdio: ["ignore", "inherit", "inherit"] }, - ); - if (result.error) throw result.error; - if (result.status !== 0) { - throw new DumpSymbolsError(`dump_syms exited with ${result.status ?? result.signal}.`); - } + // Runs dump_syms, passes its log through, and says whether it logged an error: it exits 0 + // after one. + const run = (args: ReadonlyArray): boolean => { + const result = NodeChildProcess.spawnSync(dumpSyms, [...args], { + shell: false, + stdio: ["ignore", "inherit", "pipe"], + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + }); + if (result.error) throw result.error; + process.stderr.write(result.stderr); + if (result.status !== 0) { + throw new DumpSymbolsError(`dump_syms exited with ${result.status ?? result.signal}.`); + } + return result.stderr.includes("[ERROR]"); + }; + const reportedError = run([ + "--inlines", + "--check-cfi", + "--store", + input.out, + "--output", + output, + ...inputs, + ]); // Given several inputs, dump_syms logs a failed one and still exits 0. if (!NodeFS.existsSync(output)) { throw new DumpSymbolsError(`dump_syms wrote no symbols for ${input.binary}.`); } const bytes = NodeFS.readFileSync(output); assertUsableSymbols(bytes, input.binary); + if (reportedError) { + // dump_syms logs a failed unwind table and keeps the records written before it, so the + // log line alone cannot tell a harmless error from a truncated file. A dSYM's unwind + // section header points at no data and fails this way while the binary's tables, the + // only source of STACK records, are read in full; the binary dumped alone tells which. + if (input.dsym === undefined) { + throw new DumpSymbolsError( + `dump_syms reported an error for ${input.binary}; its symbols may be incomplete.`, + ); + } + const binaryOnly = NodePath.join(work, "binary-only.sym"); + if (run(["--check-cfi", "--output", binaryOnly, inputs[0]!])) { + throw new DumpSymbolsError( + `dump_syms reported an error while reading the unwind tables of the binary ${input.binary}.`, + ); + } + const stored = countStackRecords(bytes); + const expected = countStackRecords(NodeFS.readFileSync(binaryOnly)); + if (stored !== expected) { + throw new DumpSymbolsError( + `dump_syms reported an error and the stored symbols hold ${stored} STACK records, the binary ${expected}.`, + ); + } + console.log( + `dump_syms reported an error for the dSYM; the binary's ${stored} STACK records are all stored.`, + ); + } const storePath = symbolStorePath(bytes); if (input.zip !== undefined) writeZip(input.zip, [{ name: storePath, data: bytes }]); return NodePath.join(input.out, ...storePath.split("/")); From a1629332a2cc68fc836192bf8b0a737e2f8c3729 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 09:52:29 -0300 Subject: [PATCH 03/11] feat(build): add --workspace-codegen-units to the desktop artifact script (#363) Passes a per-package Cargo --config override for bibcode-server and bibcode-desktop, which leaves every dependency fingerprint unchanged. Co-Authored-By: Claude Fable 5.2 --- scripts/build-desktop-artifact.test.ts | 40 ++++++++++++++++++++++++++ scripts/build-desktop-artifact.ts | 28 ++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/scripts/build-desktop-artifact.test.ts b/scripts/build-desktop-artifact.test.ts index 6774f7e2ac..2f43f44812 100644 --- a/scripts/build-desktop-artifact.test.ts +++ b/scripts/build-desktop-artifact.test.ts @@ -156,6 +156,46 @@ it.layer(NodeServices.layer)("build-desktop-artifact", (it) => { }), ); + it.effect("compiles only the workspace crates with the requested codegen units", () => + Effect.gen(function* () { + const path = yield* Path.Path; + const repoRoot = path.resolve("X:/repo"); + const input = { platform: "win", target: "nsis", arch: "x64" } as const; + const host = { platform: "win32", arch: "x64" } as const; + const plain = yield* resolveTauriBuildPlan(input, {}, host, repoRoot); + const plan = yield* resolveTauriBuildPlan( + { ...input, workspaceCodegenUnits: 16 }, + {}, + host, + repoRoot, + ); + + // Everything after `--` goes to Cargo; a package override leaves dependencies untouched. + assert.deepStrictEqual(plan.buildCommand.args, [ + ...plain.buildCommand.args, + "--", + "--config", + "profile.release.package.bibcode-server.codegen-units=16", + "--config", + "profile.release.package.bibcode-desktop.codegen-units=16", + ]); + assert.deepStrictEqual(plan.bundleCommand, plain.bundleCommand); + }), + ); + + it("reads --workspace-codegen-units as a whole number from 1 to 256", () => { + assert.deepStrictEqual(parseTauriArtifactCliArgs(["--workspace-codegen-units", "16"]), { + workspaceCodegenUnits: 16, + }); + for (const value of ["0", "257", "1.5", "abc", ""]) { + assert.throws( + () => parseTauriArtifactCliArgs(["--workspace-codegen-units", value]), + /--workspace-codegen-units must be a whole number from 1 to 256\./, + value, + ); + } + }); + it.effect("plans an updater-enabled macOS DMG build with the stable release overlay", () => Effect.gen(function* () { const path = yield* Path.Path; diff --git a/scripts/build-desktop-artifact.ts b/scripts/build-desktop-artifact.ts index cebbd048c7..84ac4dde97 100644 --- a/scripts/build-desktop-artifact.ts +++ b/scripts/build-desktop-artifact.ts @@ -87,6 +87,11 @@ export interface TauriBuildCliInput { readonly allowCrossPlatform?: boolean; readonly updater?: boolean; readonly symbols?: boolean; + /** + * Codegen units for `bibcode-server` and `bibcode-desktop` only. CI passes 16 because the + * shipped single unit compiles each of them on one core; Release leaves it unset. + */ + readonly workspaceCodegenUnits?: number; } export interface TauriBuildHost { @@ -146,6 +151,7 @@ interface MutableTauriBuildCliInput { allowCrossPlatform?: boolean; updater?: boolean; symbols?: boolean; + workspaceCodegenUnits?: number; } export interface TauriUpdaterArtifactDescriptor { @@ -520,6 +526,17 @@ export const resolveTauriBuildPlan = Effect.fn("resolveTauriBuildPlan")(function // --verbose reports a failed attempt as an opaque "failed to run // bundle_dmg.sh". Verbose logging keeps the first failure diagnosable. ...(step === "bundle" && platform === "mac" && target === "dmg" ? ["--verbose"] : []), + // Tauri hands what follows `--` to Cargo. A package override changes no dependency + // fingerprint, so the Rust cache that Release restores stays valid. + ...(step === "build" && input.workspaceCodegenUnits !== undefined + ? [ + "--", + ...["bibcode-server", "bibcode-desktop"].flatMap((crate) => [ + "--config", + `profile.release.package.${crate}.codegen-units=${input.workspaceCodegenUnits}`, + ]), + ] + : []), ], cwd: repoRoot, }); @@ -772,6 +789,7 @@ export function parseTauriArtifactCliArgs(argv: ReadonlyArray): TauriBui "allow-cross-platform": { type: "boolean" }, updater: { type: "boolean" }, symbols: { type: "boolean" }, + "workspace-codegen-units": { type: "string" }, }, allowPositionals: false, }); @@ -788,6 +806,16 @@ export function parseTauriArtifactCliArgs(argv: ReadonlyArray): TauriBui } if (typeof values.updater === "boolean") input.updater = values.updater; if (typeof values.symbols === "boolean") input.symbols = values.symbols; + const units = values["workspace-codegen-units"]; + if (typeof units === "string") { + // rustc accepts 1 to 256 codegen units. + if (!/^[0-9]{1,3}$/.test(units) || Number(units) < 1 || Number(units) > 256) { + throw new TauriDesktopBuildConfigurationError( + "--workspace-codegen-units must be a whole number from 1 to 256.", + ); + } + input.workspaceCodegenUnits = Number(units); + } return input; } From d9fb2c5f21832b8db7f968ded15f0e69499e1cb3 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 09:52:30 -0300 Subject: [PATCH 04/11] ci: move Windows server tests to their own job and build bundles on more cores (#363, #368) A Windows native row compiled bibcode-server five times. The server tests and the test target check now run in server_tests_windows, without -j 2. CI bundle builds pass --workspace-codegen-units 16 and print a time and memory heartbeat. Pull requests no longer save pnpm store cache entries. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/ci.yml | 182 ++++++++++++++------ .github/workflows/deploy-relay.yml | 1 + .github/workflows/desktop-ui-smoke.yml | 1 + .github/workflows/desktop-upgrade-smoke.yml | 2 + .github/workflows/release.yml | 6 + .github/workflows/typography-fixtures.yml | 2 + docs/operations/ci.md | 69 ++++++-- scripts/ci-platform-contract.test.ts | 94 +++++++++- scripts/rust-workspace.test.ts | 2 +- 9 files changed, 282 insertions(+), 77 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 81e8cb45ca..b47a781eb5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -83,6 +83,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Sensitive-data guard (commit messages) @@ -108,6 +109,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Setup Rust @@ -209,6 +211,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Setup Rust @@ -282,6 +285,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Exercise release-only workflow steps @@ -315,15 +319,16 @@ jobs: platform: win target: nsis arch: x64 - # Without a Rust cache this row took 108 minutes on 2026-11-13. + # This row took 89 to 112 minutes on 2026-11-13 and 11-14, most of it five + # builds of bibcode-server; its server tests now run in server_tests_windows. jobTimeoutMinutes: 180 - label: Windows arm64 runner: windows-11-vs2026-arm platform: win target: nsis arch: arm64 - # Without a Rust cache this row took 113 minutes on a pull request on - # 2026-11-13, and was cancelled at its 120-minute limit on main. + # This row took 103 to 113 minutes on 2026-11-13 and 11-14 and was cancelled + # at a 120-minute limit on main; see the Windows x64 row for where it went. jobTimeoutMinutes: 180 steps: &native_desktop_steps - name: Checkout @@ -334,6 +339,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: false - name: Setup Rust @@ -405,64 +411,50 @@ jobs: if: matrix.platform == 'win' run: vp test run apps/desktop/e2e/support/test-project.test.ts apps/web/src/components/gitManager/gitManagerRepositoryAvailability.test.ts - - name: Check Windows server test targets compile - if: matrix.platform == 'win' - run: node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets - - # Linux runs these in Test; Windows and macOS Git (LFS, hooks, NUL) differ (#283). - - name: Run Git system configuration tests (Windows) - if: matrix.platform == 'win' - run: node scripts/run-msvc.mjs cargo test -p bibcode-server --test git_system_config -j 2 - - # #341 was reported on Windows: background Git must not lock the index there either. - - name: Run background Git index lock tests (Windows) - if: matrix.platform == 'win' - run: node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -j 2 -- background_status_ git_stop_ index_lock_cleanup_ - - # #293: a review prompt reaches node through an npm-style .cmd shim with every class intact. - - name: Run review prompt argument tests (Windows) - if: matrix.platform == 'win' - run: node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims -j 2 -- review_prompt_shim_ - - # #292: the host collector, its exit with a disk read held, and its runtime start and - # quiesce. Each command fails the step when its filter matches no test. - - name: Run host metrics tests (Windows) - if: matrix.platform == 'win' - shell: bash - run: | - counted() { - local expected="$1"; shift - local log; log="$(mktemp)" - "$@" 2>&1 | tee "$log" - local passed - passed="$(sed -n 's/^test result: ok\. \([0-9]*\) passed.*/\1/p' "$log" | awk '{ total += $1 } END { print total + 0 }')" - if [ "$expected" = exactly-one ] && [ "$passed" -ne 1 ]; then - echo "::error::expected exactly 1 passed test, got $passed"; exit 1 - fi - if [ "$passed" -lt 1 ]; then - echo "::error::the filter matched no test"; exit 1 - fi - } - counted at-least-one node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -j 2 -- host_metrics:: - counted exactly-one node scripts/run-msvc.mjs cargo test -p bibcode-server --test host_metrics_exit -j 2 -- --exact a_held_disk_read_does_not_block_exit - counted exactly-one node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -j 2 -- --exact production::runtime::tests::host_metrics_start_with_the_runtime_and_stop_in_quiesce - # build-desktop-artifact.ts --symbols runs dump_syms on the unstripped binary. Rust Cache # keeps ~/.cargo/bin, so a restored runner skips the compile. - name: Install dump_syms run: node scripts/run-msvc.mjs cargo install dump_syms --locked --version 2.3.9 - # --symbols is the release build (release.yml build_desktop): the same - # profile, so the cache it saves is the one Release restores. + # --symbols is the release symbol build (release.yml build_desktop), so the dependency + # cache it saves is the one Release restores. + # --workspace-codegen-units 16 is CI only: the shipped single unit compiles bibcode-server + # on one core (23 to 107 minutes of this step). It is a per-package override, so no + # dependency fingerprint changes and the cache stays the one Release restores. + # The heartbeat shows where time and memory go (#363): the macOS arm64 runner has 7 GB + # and the bibcode-server compile peaked above 8 GB locally. It never fails the step. - name: Build native bundle without publishing - run: > - node scripts/build-desktop-artifact.ts - --platform "${{ matrix.platform }}" - --target "${{ matrix.target }}" - --arch "${{ matrix.arch }}" - --output-dir native-artifacts - --symbols - --verbose + shell: bash + run: | + start=$SECONDS + peak="$RUNNER_TEMP/bundle-peak-kb" + echo 0 > "$peak" + ( + while sleep 60; do + line="heartbeat $((SECONDS - start))s" + if [ "$RUNNER_OS" != Windows ]; then + top="$(ps -Ao rss=,comm= 2>/dev/null | awk '$2 ~ /(^|\/)(rustc|ld|lld|ld64|rust-lld|dsymutil)$/' | sort -rn | head -4 || true)" + kb="$(printf '%s\n' "$top" | awk 'NR == 1 { print $1 + 0 }' || true)" + if [ "${kb:-0}" -gt "$(cat "$peak")" ]; then echo "$kb" > "$peak"; fi + line="$line; $(printf '%s\n' "$top" | awk 'NF { n = split($2, part, "/"); printf "%s %d MB, ", part[n], $1 / 1024 }' || true)" + fi + case "$RUNNER_OS" in + macOS) line="$line $(sysctl -n vm.swapusage 2>/dev/null || true); free pages $(vm_stat 2>/dev/null | awk '/Pages free/ { print $3 + 0 }' || true)" ;; + Linux) line="$line $(awk '/MemAvailable|SwapFree/ { printf "%s %d MB ", $1, $2 / 1024 }' /proc/meminfo 2>/dev/null || true)" ;; + esac + echo "$line" + done + ) & + heartbeat=$! + trap 'kill "$heartbeat" 2>/dev/null || true; echo "::notice title=Bundle build::${RUNNER_OS} ${RUNNER_ARCH} took $((SECONDS - start)) s, peak single compiler or linker process $(($(cat "$peak") / 1024)) MB"' EXIT + node scripts/build-desktop-artifact.ts \ + --platform "${{ matrix.platform }}" \ + --target "${{ matrix.target }}" \ + --arch "${{ matrix.arch }}" \ + --output-dir native-artifacts \ + --symbols \ + --workspace-codegen-units 16 \ + --verbose # The macOS legs take 1-5 hours on a 5-job macOS concurrency cap, so pull # requests skip them unless labelled ci:full. Main pushes and manual runs @@ -560,6 +552,85 @@ jobs: counted exactly-one cargo test -p bibcode-server --test host_metrics_exit -j 2 -- --exact a_held_disk_read_does_not_block_exit counted exactly-one cargo test -p bibcode-server --lib -j 2 -- --exact production::runtime::tests::host_metrics_start_with_the_runtime_and_stop_in_quiesce + # Linux runs these server tests in Test and macOS in server_tests_macos. They left the Windows + # native desktop rows because a row compiled bibcode-server five times (host tests, this + # check, two test builds, the bundle): 76 of 103 minutes on 2026-11-14, against about 2 + # minutes of running tests. Nothing else runs beside the compiler here, so no -j 2. + # The cache is the native desktop one, restored and never saved. + server_tests_windows: + name: Server tests - ${{ matrix.label }} + runs-on: ${{ matrix.runner }} + timeout-minutes: 90 + strategy: + fail-fast: false + matrix: + include: + - label: Windows x64 + runner: windows-2025 + - label: Windows arm64 + runner: windows-11-vs2026-arm + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # The shim tests run node, and run-msvc.mjs is a Node script; no workspace install. + - name: Setup Vite+ + uses: voidzero-dev/setup-vp@68f39c78aaf39be468cabd2afb09c162574c84af # v1.21.2 + with: + node-version-file: package.json + run-install: false + + - name: Setup Rust + uses: dtolnay/rust-toolchain@18f0487477426dce244608110aa2660c2893deb7 # 1.98.1 + + - name: Restore Rust build output + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + with: + shared-key: native-desktop + save-if: "false" + workspaces: | + . -> target + + # Clippy's --all-targets pass runs on Linux alone, so Unix-only test helpers or imports + # that are unused on Windows fail here under -D warnings. + - name: Check Windows server test targets compile + run: node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets + + # Linux runs these in Test; Windows and macOS Git (LFS, hooks, NUL) differ (#283). + - name: Run Git system configuration tests (Windows) + run: node scripts/run-msvc.mjs cargo test -p bibcode-server --test git_system_config + + # #341 was reported on Windows: background Git must not lock the index there either. + - name: Run background Git index lock tests (Windows) + run: node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -- background_status_ git_stop_ index_lock_cleanup_ + + # #293: a review prompt reaches node through an npm-style .cmd shim with every class intact. + # shim_timeout_ proves that a timeout prints what the terminal showed (#367). + - name: Run review prompt argument tests (Windows) + run: node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims -- review_prompt_shim_ shim_timeout_ + + # #292: the host collector, its exit with a disk read held, and its runtime start and + # quiesce. Each command fails the step when its filter matches no test. + - name: Run host metrics tests (Windows) + shell: bash + run: | + counted() { + local expected="$1"; shift + local log; log="$(mktemp)" + "$@" 2>&1 | tee "$log" + local passed + passed="$(sed -n 's/^test result: ok\. \([0-9]*\) passed.*/\1/p' "$log" | awk '{ total += $1 } END { print total + 0 }')" + if [ "$expected" = exactly-one ] && [ "$passed" -ne 1 ]; then + echo "::error::expected exactly 1 passed test, got $passed"; exit 1 + fi + if [ "$passed" -lt 1 ]; then + echo "::error::the filter matched no test"; exit 1 + fi + } + counted at-least-one node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -- host_metrics:: + counted exactly-one node scripts/run-msvc.mjs cargo test -p bibcode-server --test host_metrics_exit -- --exact a_held_disk_read_does_not_block_exit + counted exactly-one node scripts/run-msvc.mjs cargo test -p bibcode-server --lib -- --exact production::runtime::tests::host_metrics_start_with_the_runtime_and_stop_in_quiesce + # A real native crash under the crash monitor on every OS, symbolicated with symbols dumped # the way the release pipeline dumps them. # macos-26 runs on main, manual runs, and ci:full pull requests only. @@ -586,6 +657,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: false - name: Setup Rust diff --git a/.github/workflows/deploy-relay.yml b/.github/workflows/deploy-relay.yml index 68e65fd54c..b86da80092 100644 --- a/.github/workflows/deploy-relay.yml +++ b/.github/workflows/deploy-relay.yml @@ -46,6 +46,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Deploy production relay stage diff --git a/.github/workflows/desktop-ui-smoke.yml b/.github/workflows/desktop-ui-smoke.yml index f35681c10a..3380246619 100644 --- a/.github/workflows/desktop-ui-smoke.yml +++ b/.github/workflows/desktop-ui-smoke.yml @@ -65,6 +65,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: false - name: Setup Rust diff --git a/.github/workflows/desktop-upgrade-smoke.yml b/.github/workflows/desktop-upgrade-smoke.yml index 735cb9cb11..40bef2cf5e 100644 --- a/.github/workflows/desktop-upgrade-smoke.yml +++ b/.github/workflows/desktop-upgrade-smoke.yml @@ -103,6 +103,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: false # No Rust cache: the harness builds into its own run-root target, which a @@ -278,6 +279,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: false - name: Setup Rust diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 79e47b3f6c..c8311f02c4 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -99,6 +99,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Setup Rust @@ -314,6 +315,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: | args: - --filter=bibcode-relay... @@ -433,6 +435,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Align package manifest versions to release version @@ -646,6 +649,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Align package manifest versions to release version @@ -846,6 +850,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Download all desktop artifacts @@ -1091,6 +1096,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: | args: - --filter=@bibcode/scripts... diff --git a/.github/workflows/typography-fixtures.yml b/.github/workflows/typography-fixtures.yml index 0138cf3cfa..cfc9150bce 100644 --- a/.github/workflows/typography-fixtures.yml +++ b/.github/workflows/typography-fixtures.yml @@ -34,6 +34,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Check capture caller run: node --test apps/web/visual/typography/*.node-test.mjs @@ -74,6 +75,7 @@ jobs: with: node-version-file: package.json cache: true + cache-save: ${{ github.ref == 'refs/heads/main' }} run-install: true - name: Check capture caller run: node --test apps/web/visual/typography/*.node-test.mjs diff --git a/docs/operations/ci.md b/docs/operations/ci.md index 0d1e0d8b84..3e5992901c 100644 --- a/docs/operations/ci.md +++ b/docs/operations/ci.md @@ -29,6 +29,7 @@ when it carries the `ci:full` label: | Check, Test, Release Smoke | yes | yes | | Native desktop: Linux and Windows rows | yes | yes | | Native desktop: macOS arm64 and x64 | no | yes | +| Server tests: Windows x64 and arm64 | yes | yes | | Server tests: macOS arm64 and x64 | no | yes | | Crash smoke: `ubuntu-24.04`, `windows-2025` | yes | yes | | Crash smoke: `macos-26` | no | yes | @@ -57,7 +58,8 @@ workflows do not listen for `merge_group`. branch deletion and non-fast-forward pushes; it requires no status checks. If required checks are added, require the job names that run on every pull request: `Secrets`, `Check`, `Test`, `Release Smoke`, the four `Native desktop - Linux -…`/`Windows …` rows, and `Crash smoke - ubuntu-24.04` and `- windows-2025`. +…`/`Windows …` rows, the two `Server tests - Windows …` rows, and `Crash smoke - +ubuntu-24.04` and `- windows-2025`. Do not require the macOS rows or the seeded upgrade jobs: on pull requests without `ci:full` they do not report under their row names (`macos-26` is not in the crash matrix at all), and a required check that never reports blocks @@ -139,6 +141,26 @@ the merge. GNU build-id on Linux) that Release uses, so it also proves the `dump_syms` path on every pull request; the job installs `dump_syms` 2.3.9 for it, and Rust Cache keeps the binary in `~/.cargo/bin`. + It also passes `--workspace-codegen-units 16`, which Release does not: Cargo + gets `--config profile.release.package..codegen-units=16` for + `bibcode-server` and `bibcode-desktop` only. Swatinem/rust-cache never stores + workspace crates, so every run compiles those two from scratch, and with the + shipped single codegen unit each compiles on one core: on 2026-11-14 the + bundle step spent about 23 of 29 minutes on Linux x64, 42 of 42 on Windows x64 + and 81 of 88 on macOS arm64 after Cargo had started its last dependency. A + local build of the same three units took 1125 seconds with one unit and 653 + with 16. A package override changes no dependency fingerprint, so the entry + this job saves is still the one Release restores. CI therefore proves the + thin-LTO link, the symbol dump, the strip and the bundle, and Release alone + compiles the two crates with the shipped single unit. + While it builds, the step prints a heartbeat every 60 seconds: elapsed + seconds and, outside Windows, the resident memory of the largest `rustc`, + linker and `dsymutil` processes and the system's free memory and swap. It + ends with a `Bundle build` notice holding the seconds and the peak single + process. The macOS arm64 runner has 3 cores and 7 GB, and the + `bibcode-server` compile peaked at 8.7 GB (shipped profile), 9.6 GB (symbol + profile) and 12 GB (16 units) on a local machine, so the heartbeat is how a + run shows memory pressure there. It never fails the step. Each matrix row declares its complete-job budget: 330 minutes for each macOS row, 180 minutes for each Windows row, and 120 minutes for each Linux row. @@ -155,7 +177,7 @@ the merge. Check a change to the probe by running it once with `CARGO_PROFILE_RELEASE_PANIC=abort`, where it must fail. The shared `scripts/run-msvc.mjs` launcher selects the requested MSVC architecture. After the Rust host tests, - the Windows row alone runs + the Windows rows alone run `vp test run apps/desktop/e2e/support/test-project.test.ts apps/web/src/components/gitManager/gitManagerRepositoryAvailability.test.ts`. That step is the supported native proof that the generated Cursor `.cmd` shim executes through the Windows command processor and writes its exact action record. Simulated @@ -166,16 +188,27 @@ the merge. spaces, dollar signs, backticks, and typographic quotes. Every Git/home configuration input is fixture-owned. Other platforms skip that native case; their string assertions remain compatibility evidence. - The Windows rows then run - `node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets` so - Unix-only test helpers or imports that are unused on Windows fail there under - `-D warnings` instead of surfacing only during native validation; Clippy's - `--all-targets` pass otherwise runs on Linux alone. The repository-availability contract additionally executes the generated drive/UNC trust commands through native Windows PowerShell and Git, including spaces, dollar signs, backticks, and typographic quotes. Every Git/home configuration input is fixture-owned. Other platforms skip that native case; their string assertions remain compatibility evidence. +- **Server tests (Windows)** is the job `server_tests_windows`, on `windows-2025` + and `windows-11-vs2026-arm` on every pull request. It first runs + `node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets`, so + Unix-only test helpers or imports that are unused on Windows fail there under + `-D warnings` instead of surfacing only during native validation; Clippy's + `--all-targets` pass otherwise runs on Linux alone. It then runs the server + tests whose behaviour differs on Windows: `git_system_config`, the background + Git index lock tests, the review prompt shim tests with the shim timeout test + (`-- review_prompt_shim_ shim_timeout_`), and the host metrics tests. These + steps left the native desktop rows because one row compiled `bibcode-server` + five times (the desktop host tests, this check, two server test builds and + the bundle): the five Cargo builds took 76 of the Windows arm64 row's 103 + minutes on 2026-11-14, against about 2 minutes of running tests. Its commands + carry no `-j 2`: nothing runs beside the compiler in this job. It restores + the Native desktop Rust cache and never saves one. Its budget is 90 minutes; + the moved steps took 38 to 48 minutes inside the native rows. - **Server tests (macOS)** is the job `server_tests_macos`, on `macos-26` and `macos-15-intel` with the `ci:full` gate above. It runs the server tests whose behaviour differs on macOS: `git_system_config`, the background Git index lock @@ -255,16 +288,28 @@ recently used entries above that. Which workflow saves which entry: `main` entries out. - Seeded Packaged Desktop Upgrade has no Rust cache step: its harness builds in a run-owned target that a repository `target` cache cannot warm. -- The pnpm store stays on the setup-vp cache. + +- The pnpm store stays on the setup-vp cache, about 0.5 GB per runner OS and + architecture. Every `setup-vp` step with `cache: true` sets + `cache-save: ${{ github.ref == 'refs/heads/main' }}`: a pull request's entry + can only be read by that pull request and still counts against the limit. A job cancelled by `timeout-minutes` or by a newer run skips the cache post step, so the Check and Test budgets above include a cold compile. Native desktop keeps its 120-minute, 180-minute and 330-minute row budgets. -Six native-desktop entries of about 1.6 GB each are about 10 GB on their own, -before Check, Test, Crash smoke, Linux Git, and the setup-vp caches. Eviction -therefore still drops the least recently used entries, now always older `main` -entries rather than unrestorable tag or pull request entries. List them with +The store cannot hold what CI saves. On 2026-11-14 it held 9.98 GiB of the +10 GB limit: three of the six native-desktop entries (1.5 to 1.85 GiB each), +Check, two Crash smoke entries, Linux Git and six pnpm stores. Every entry +together would be about 17 GB. GitHub evicts hourly once the limit is passed, +so entries push each other out: in one `main` run 11 of 13 Rust cache restores +found nothing, and a Windows arm64 entry restored at 01:09 was gone at 02:08. +A missing entry costs a job its dependency compile, about 6 minutes in the +bundle step and about 12 minutes across a Windows native row. It does not +explain a row that takes an hour or more: that time is the workspace crates, +which no cache entry holds. The limit is a repository setting (Settings, +Actions, or `PUT /repos/{owner}/{repo}/actions/cache/storage-limit`); raising +it is billed per GiB above 10. List the entries with `gh cache list --sort size_in_bytes --limit 100`. ## Other Workflows diff --git a/scripts/ci-platform-contract.test.ts b/scripts/ci-platform-contract.test.ts index fd0451e834..facec1acc4 100644 --- a/scripts/ci-platform-contract.test.ts +++ b/scripts/ci-platform-contract.test.ts @@ -483,20 +483,96 @@ describe("cross-platform CI contract", () => { ); }); - it("compiles every bibcode-server test target on the Windows native rows", () => { - // Clippy with --all-targets runs on Linux only, so Unix-only test helpers - // and imports can leave the Windows server test binaries uncompilable - // under -D warnings without this gate. + it("runs the Windows server tests off the native desktop rows, without saving a cache", () => { + // A native row compiled bibcode-server five times; the test builds were two of them and + // ran before the bundle on the same runner. Clippy with --all-targets runs on Linux only, + // so the check step still proves the Windows server test targets compile under -D warnings. const { workflow } = readWorkflow(CI_WORKFLOW_PATH); - const nativeSteps = requireJob(workflow, "native_desktop").steps ?? []; - const windowsCompileStep = nativeSteps.find( + const job = requireJob(workflow, "server_tests_windows"); + expect(job.if).toBeUndefined(); + expect(job["timeout-minutes"]).toBe(90); + expect(job.strategy?.matrix?.include).toEqual([ + { label: "Windows x64", runner: "windows-2025" }, + { label: "Windows arm64", runner: "windows-11-vs2026-arm" }, + ]); + const steps = job.steps ?? []; + const cache = steps.find((step) => String(step.uses ?? "").startsWith("Swatinem/rust-cache@")); + expect(cache?.with?.["shared-key"]).toBe("native-desktop"); + expect(cache?.with?.["save-if"]).toBe("false"); + const windowsSteps = steps.filter((step) => String(step.name).endsWith("(Windows)")); + expect(windowsSteps.map((step) => step.name)).toEqual([ + "Run Git system configuration tests (Windows)", + "Run background Git index lock tests (Windows)", + "Run review prompt argument tests (Windows)", + "Run host metrics tests (Windows)", + ]); + const check = steps.findIndex( (step) => step.name === "Check Windows server test targets compile", ); - - expect(windowsCompileStep?.if).toBe("matrix.platform == 'win'"); - expect(windowsCompileStep?.run).toBe( + expect(steps[check]?.run).toBe( "node scripts/run-msvc.mjs cargo check -p bibcode-server --all-targets", ); + expect(steps[check + 1]?.name).toBe("Run Git system configuration tests (Windows)"); + const runs = allStepCommands(job); + // Nothing else runs beside the compiler in this job, so it may use every core. + expect(runs).not.toContain("-j 2"); + // The second filter runs the test that proves a timeout prints the terminal output. + expect(runs).toContain( + "cargo test -p bibcode-server --test windows_terminal_shims -- review_prompt_shim_ shim_timeout_", + ); + // Each host metrics command fails the step when its filter matches no test. + expect(runs).toContain("counted exactly-one"); + + const native = requireJob(workflow, "native_desktop").steps ?? []; + expect(native.filter((step) => String(step.name).endsWith("(Windows)"))).toEqual([]); + const nativeRuns = allStepCommands(requireJob(workflow, "native_desktop")); + expect(nativeRuns).not.toContain("cargo test -p bibcode-server"); + expect(nativeRuns).not.toContain("cargo check -p bibcode-server"); + }); + + it("builds CI bundles with 16 codegen units for the workspace crates and a heartbeat", () => { + const { workflow } = readWorkflow(CI_WORKFLOW_PATH); + const build = requireJob(workflow, "native_desktop").steps?.find( + (step) => step.name === "Build native bundle without publishing", + ); + const run = String(build?.run); + expect(build?.shell).toBe("bash"); + // The shipped single unit compiles bibcode-server on one core. A package override changes + // no dependency fingerprint, so the cache Release restores stays valid. + expect(run).toContain("--workspace-codegen-units 16"); + // One line a minute, with the memory of each compiler and linker where ps can report it. + expect(run).toContain("sleep 60"); + expect(run).toContain("ps -Ao rss=,comm="); + // The trap runs under `bash -e`: a heartbeat that already died must not fail a good build. + expect(run).toMatch(/trap 'kill "\$heartbeat" 2>\/dev\/null \|\| true; [^']*' EXIT/); + expect(run).toContain("::notice title=Bundle build::"); + // The build's own exit code must reach the step. + expect(run).not.toContain("set +e"); + expect(run).not.toMatch(/build-desktop-artifact\.ts[^\n]*\|\|/); + + // Release ships the profile in Cargo.toml. + expect(readWorkflow(RELEASE_WORKFLOW_PATH).raw).not.toContain("--workspace-codegen-units"); + }); + + it("saves the pnpm store cache from main only, in every workflow", () => { + // A pull request's entry is readable only by that pull request and still counts against + // the repository's cache limit, where it pushes out the main entries every run restores. + const directory = NodePath.join(REPOSITORY_ROOT, ".github/workflows"); + const cached: string[] = []; + for (const file of NodeFS.readdirSync(directory).filter((name) => name.endsWith(".yml"))) { + const { workflow } = readWorkflow(NodePath.join(directory, file)); + for (const [name, job] of Object.entries(workflow.jobs ?? {})) { + for (const step of job.steps ?? []) { + if (!String(step.uses ?? "").startsWith("voidzero-dev/setup-vp@")) continue; + if (step.with?.cache !== true) continue; + cached.push(`${file} ${name}`); + expect(step.with["cache-save"], `${file} ${name}`).toBe( + "${{ github.ref == 'refs/heads/main' }}", + ); + } + } + } + expect(cached.length).toBeGreaterThan(0); }); it("installs the full official Linux Tauri prerequisite set", () => { diff --git a/scripts/rust-workspace.test.ts b/scripts/rust-workspace.test.ts index 08c6171ce3..884737864a 100644 --- a/scripts/rust-workspace.test.ts +++ b/scripts/rust-workspace.test.ts @@ -99,7 +99,7 @@ it.layer(NodeServices.layer)("canonical Rust workspace", (it) => { ); assert.equal( ciWorkflow.match(/uses: dtolnay\/rust-toolchain@[0-9a-f]{40} # 1\.98\.1/g)?.length ?? 0, - 5, + 6, "Every Rust CI job must exercise the declared Rust 1.98.1 toolchain", ); From 0f5bf8d37d47965eef07ba22bbd00f78df60b826 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 09:52:30 -0300 Subject: [PATCH 05/11] test(terminal): print the terminal output when a shim test times out (#367) Co-Authored-By: Claude Fable 5.2 --- apps/server/tests/windows_terminal_shims.rs | 109 +++++++++++++++----- docs/testing/windows-desktop.md | 11 +- 2 files changed, 94 insertions(+), 26 deletions(-) diff --git a/apps/server/tests/windows_terminal_shims.rs b/apps/server/tests/windows_terminal_shims.rs index 127457ee03..86a6fa3361 100644 --- a/apps/server/tests/windows_terminal_shims.rs +++ b/apps/server/tests/windows_terminal_shims.rs @@ -8,8 +8,9 @@ use std::{ }; use base64::Engine as _; -use bibcode_server::terminal::{PortablePtyBackend, PtyBackend, PtySpawnInput}; +use bibcode_server::terminal::{PortablePtyBackend, PtyBackend, PtyExit, PtySpawnInput}; use portable_pty::{CommandBuilder, PtySize, native_pty_system}; +use tokio::sync::{broadcast, watch}; use windows_sys::Win32::{ Foundation::{ CloseHandle, ERROR_INVALID_PARAMETER, GetLastError, HANDLE, INVALID_HANDLE_VALUE, @@ -130,26 +131,18 @@ async fn assert_terminal_kills_descendant(input: PtySpawnInput, child_pid_file: ); } -async fn assert_shim_output( - executable: &Path, - arguments: &[String], - environment: BTreeMap, +/// Collects terminal output until it holds `marker` or the terminal exits. On a timeout the +/// error keeps what the terminal printed and whether it had exited: "late" and "never printed" +/// need different fixes, and a bare `Elapsed` cannot tell them apart (#367). +async fn wait_for_marker( + output: &mut broadcast::Receiver, + exit: &mut watch::Receiver>, marker: &str, -) { - let process = PortablePtyBackend - .spawn(&PtySpawnInput { - executable: executable.to_string_lossy().into_owned(), - args: arguments.to_vec(), - cwd: executable.parent().unwrap().to_path_buf(), - cols: 80, - rows: 24, - env: environment, - }) - .unwrap(); - let mut output = process.subscribe_output(); - let mut exit = process.subscribe_exit(); - let text = tokio::time::timeout(Duration::from_secs(10), async { - let mut text = String::new(); + deadline: Duration, +) -> Result { + // Outside the timed future, so a timeout does not drop it. + let mut text = String::new(); + let waited = tokio::time::timeout(deadline, async { while !text.contains(marker) { if exit.borrow().is_some() { break; @@ -166,10 +159,39 @@ async fn assert_shim_output( } } } - text }) - .await - .unwrap(); + .await; + match waited { + Ok(()) => Ok(text), + Err(_) => Err(format!( + "timed out after {}s waiting for {marker}; output={text:?}, exit={:?}", + deadline.as_secs(), + exit.borrow().clone() + )), + } +} + +async fn assert_shim_output( + executable: &Path, + arguments: &[String], + environment: BTreeMap, + marker: &str, +) { + let process = PortablePtyBackend + .spawn(&PtySpawnInput { + executable: executable.to_string_lossy().into_owned(), + args: arguments.to_vec(), + cwd: executable.parent().unwrap().to_path_buf(), + cols: 80, + rows: 24, + env: environment, + }) + .unwrap(); + let mut output = process.subscribe_output(); + let mut exit = process.subscribe_exit(); + let text = wait_for_marker(&mut output, &mut exit, marker, Duration::from_secs(10)) + .await + .unwrap_or_else(|timed_out| panic!("{timed_out}")); assert!( text.contains(marker), "terminal exited without the expected marker; output={text:?}, exit={:?}", @@ -679,6 +701,47 @@ async fn assert_review_prompt_reaches_node(sample: &str) { } } +#[tokio::test] +async fn shim_timeout_reports_the_collected_output() { + let directory = tempfile::tempdir().unwrap(); + let shim = directory.path().join("slow.cmd"); + // `ping` waits without a console input handle, which `timeout` needs. + std::fs::write( + &shim, + "@echo off\r\necho BEFORE\r\nping -n 16 127.0.0.1 >nul\r\necho NEVER_REACHED_MARKER\r\n", + ) + .unwrap(); + let process = PortablePtyBackend + .spawn(&PtySpawnInput { + executable: shim.to_string_lossy().into_owned(), + args: Vec::new(), + cwd: directory.path().to_path_buf(), + cols: 80, + rows: 24, + env: BTreeMap::new(), + }) + .unwrap(); + let mut output = process.subscribe_output(); + let mut exit = process.subscribe_exit(); + + let timed_out = wait_for_marker( + &mut output, + &mut exit, + "NEVER_REACHED_MARKER", + Duration::from_secs(5), + ) + .await + .expect_err("the shim prints the marker only after the deadline"); + + process.kill().unwrap(); + assert!( + timed_out.contains("timed out after 5s waiting for NEVER_REACHED_MARKER"), + "{timed_out}" + ); + assert!(timed_out.contains("BEFORE"), "{timed_out}"); + assert!(timed_out.ends_with("exit=None"), "{timed_out}"); +} + #[tokio::test] async fn review_prompt_shim_passes_quote() { assert_review_prompt_reaches_node(r#"say "hi""#).await; diff --git a/docs/testing/windows-desktop.md b/docs/testing/windows-desktop.md index acca2ae7a2..9ded8e2f73 100644 --- a/docs/testing/windows-desktop.md +++ b/docs/testing/windows-desktop.md @@ -402,13 +402,18 @@ with hooks off. The repository pin beats both. The code review terminal argument tests (#293) send one argument per class (quotes, `%` and `%PATH%`, `&`, `^`, `|`, `<` with `>`, `!`, trailing backslashes, a backslash before a quote, and all of them combined) through an -npm-style `%*` `.cmd` shim to `node` and compare what arrives. CI's Windows rows -run them, and a native Windows pass runs the same command: +npm-style `%*` `.cmd` shim to `node` and compare what arrives. CI's +`Server tests - Windows` job runs them, with the server test target check, the +Git system configuration tests, the background index lock tests and the host +metrics tests, and a native Windows pass runs the same command: ```powershell -node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims -j 2 -- review_prompt_shim_ +node scripts/run-msvc.mjs cargo test -p bibcode-server --test windows_terminal_shims -- review_prompt_shim_ shim_timeout_ ``` +The `shim_timeout_` test proves that a shim which misses its deadline fails +with the terminal output it printed and its exit state (#367). + Terminal reviews are on for Windows environments because these pass on real Windows; a failing class means they must go off again. Record the result in the execution report. From 1136012960b1a082aedfbe9b160e1f1ce1cb1105 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 10:01:44 -0300 Subject: [PATCH 06/11] fix(ci): stop the heartbeat's sleep on exit and show dump_syms output before a spawn error Review findings: the heartbeat left its sleep holding the step's log open, a dump_syms spawn error hid what the tool had logged, and one sentence in the CI guide still said native CI builds the full release profile. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/ci.yml | 9 +++++++-- docs/operations/ci.md | 2 +- scripts/dump-symbols.ts | 3 ++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b47a781eb5..a8f1af5de6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -430,7 +430,12 @@ jobs: peak="$RUNNER_TEMP/bundle-peak-kb" echo 0 > "$peak" ( - while sleep 60; do + # An interruptible wait, so the EXIT trap below leaves no sleep holding the log open. + trap 'kill "$nap" 2>/dev/null || true; exit 0' TERM + while :; do + sleep 60 & + nap=$! + wait "$nap" || true line="heartbeat $((SECONDS - start))s" if [ "$RUNNER_OS" != Windows ]; then top="$(ps -Ao rss=,comm= 2>/dev/null | awk '$2 ~ /(^|\/)(rustc|ld|lld|ld64|rust-lld|dsymutil)$/' | sort -rn | head -4 || true)" @@ -446,7 +451,7 @@ jobs: done ) & heartbeat=$! - trap 'kill "$heartbeat" 2>/dev/null || true; echo "::notice title=Bundle build::${RUNNER_OS} ${RUNNER_ARCH} took $((SECONDS - start)) s, peak single compiler or linker process $(($(cat "$peak") / 1024)) MB"' EXIT + trap 'kill "$heartbeat" 2>/dev/null || true; wait "$heartbeat" 2>/dev/null || true; echo "::notice title=Bundle build::${RUNNER_OS} ${RUNNER_ARCH} took $((SECONDS - start)) s, peak single compiler or linker process $(($(cat "$peak") / 1024)) MB"' EXIT node scripts/build-desktop-artifact.ts \ --platform "${{ matrix.platform }}" \ --target "${{ matrix.target }}" \ diff --git a/docs/operations/ci.md b/docs/operations/ci.md index 3e5992901c..3524dba972 100644 --- a/docs/operations/ci.md +++ b/docs/operations/ci.md @@ -339,7 +339,7 @@ it is billed per GiB above 10. List the entries with 240 minutes; their packaging children keep 60 minutes on Windows and 45 on Linux. If owned-child cleanup does not observe `close` within five seconds, the harness releases that child and exits nonzero so the job cannot stay open for the rest of the 240-minute budget. - The smoke builds override the release profile with 16 codegen units, LTO off, and `opt-level` 1, because they prove upgrade behaviour, not optimised code: a single thin-LTO codegen unit kept Intel runners on one core for up to an hour per package, and optimising the workspace crates dominates what the shared target cannot remove. Release and CI packaging outside this harness keep the full release profile. + The smoke builds override the release profile with 16 codegen units, LTO off, and `opt-level` 1, because they prove upgrade behaviour, not optimised code: a single thin-LTO codegen unit kept Intel runners on one core for up to an hour per package, and optimising the workspace crates dominates what the shared target cannot remove. Release keeps the full release profile; CI's native desktop rows keep it too, except for the 16 codegen units they give the two workspace crates (see Native desktop above). The harness appends each install's and build's start, end, duration, and exit code or timeout to `build-timings.txt` in the evidence folder, plus a five-minute heartbeat with a `ps` sample of `rustc`, `cargo`, and `ld` (skipped on Windows), so a killed build still uploads evidence. The harness step ends 15 minutes before its job (345 of 360 minutes on macOS, 225 of 240 elsewhere) so the always-run upload still runs. `remote-install` reuses the protected package. Repository `target` caching cannot warm the run root's shared Cargo target, diff --git a/scripts/dump-symbols.ts b/scripts/dump-symbols.ts index 8281cf7fac..1e998cc7b5 100644 --- a/scripts/dump-symbols.ts +++ b/scripts/dump-symbols.ts @@ -210,8 +210,9 @@ export function dumpSymbols(input: DumpSymbolsInput, dumpSyms = "dump_syms"): st encoding: "utf8", maxBuffer: 64 * 1024 * 1024, }); + // First, so a run that failed (an overflowing log included) still shows what it logged. + if (result.stderr) process.stderr.write(result.stderr); if (result.error) throw result.error; - process.stderr.write(result.stderr); if (result.status !== 0) { throw new DumpSymbolsError(`dump_syms exited with ${result.status ?? result.signal}.`); } From 37b6b8aac63f4551d0d1d430bdf1385c97381afb Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 10:09:13 -0300 Subject: [PATCH 07/11] test(terminal): start the shim timeout window only once the shim prints A first spawn can stall for seconds on a loaded Windows runner (#367), which would fail the new test for the reason it exists to report. Also document the new bundle flag and the dump check in the scripts reference. Co-Authored-By: Claude Fable 5.2 --- apps/server/tests/windows_terminal_shims.rs | 12 +++++++++--- docs/reference/scripts.md | 13 ++++++++++++- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/apps/server/tests/windows_terminal_shims.rs b/apps/server/tests/windows_terminal_shims.rs index 86a6fa3361..4d92d6a209 100644 --- a/apps/server/tests/windows_terminal_shims.rs +++ b/apps/server/tests/windows_terminal_shims.rs @@ -705,10 +705,11 @@ async fn assert_review_prompt_reaches_node(sample: &str) { async fn shim_timeout_reports_the_collected_output() { let directory = tempfile::tempdir().unwrap(); let shim = directory.path().join("slow.cmd"); - // `ping` waits without a console input handle, which `timeout` needs. + // Prints a line about once a second and never the marker. `ping` waits without a console + // input handle, which `timeout` needs. std::fs::write( &shim, - "@echo off\r\necho BEFORE\r\nping -n 16 127.0.0.1 >nul\r\necho NEVER_REACHED_MARKER\r\n", + "@echo off\r\n:again\r\necho BEFORE\r\nping -n 2 127.0.0.1 >nul\r\ngoto again\r\n", ) .unwrap(); let process = PortablePtyBackend @@ -724,6 +725,11 @@ async fn shim_timeout_reports_the_collected_output() { let mut output = process.subscribe_output(); let mut exit = process.subscribe_exit(); + // A first spawn can take many seconds on a loaded runner (#367), so the timed part starts + // only once the shim is known to be printing. + wait_for_marker(&mut output, &mut exit, "BEFORE", Duration::from_secs(120)) + .await + .unwrap_or_else(|timed_out| panic!("{timed_out}")); let timed_out = wait_for_marker( &mut output, &mut exit, @@ -731,7 +737,7 @@ async fn shim_timeout_reports_the_collected_output() { Duration::from_secs(5), ) .await - .expect_err("the shim prints the marker only after the deadline"); + .expect_err("the shim never prints the marker"); process.kill().unwrap(); assert!( diff --git a/docs/reference/scripts.md b/docs/reference/scripts.md index 0320eb8473..fd4a044859 100644 --- a/docs/reference/scripts.md +++ b/docs/reference/scripts.md @@ -212,8 +212,19 @@ output directory also receives [Symbol assets](../operations/release.md#symbol-assets). `--symbols` needs `dump_syms` on `PATH` and cannot be combined with `--skip-build`. +`--workspace-codegen-units ` (1 to 256), which CI passes as 16 and Release +does not pass, hands Cargo +`--config profile.release.package..codegen-units=` for +`bibcode-server` and `bibcode-desktop` in the build step. The shipped profile +has one codegen unit, which compiles each of the two crates on one core. A +package override leaves every dependency's fingerprint as it was, so a cached +`target` still serves both builds. + `scripts/dump-symbols.ts` runs `dump_syms --inlines --check-cfi --store ` -on one release binary: +on one release binary. When `dump_syms` logs an `[ERROR]` line, which it does +without failing, the script dumps the binary alone a second time and fails +unless that run is clean and the stored file holds the same number of `STACK` +records; see [Symbol assets](../operations/release.md#symbol-assets). ```sh node scripts/dump-symbols.ts --binary --module-name --out \ From de30ef4fc0af72ed0b34379561d00d31f0d5807a Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 10:57:02 -0300 Subject: [PATCH 08/11] ci(secrets): scan what a manually run branch adds to main A manual run has no base or before commit, so it scanned the whole history of the branch and failed on findings that were already there. That made the documented way to run the full set on a pull request always fail Secrets. Approved by the repository owner on 2026-11-14. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/ci.yml | 9 +++++++-- docs/operations/ci.md | 4 ++-- scripts/ci-platform-contract.test.ts | 11 +++++++++++ 3 files changed, 20 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a8f1af5de6..529f11ea90 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,8 +37,10 @@ jobs: # Pull requests scan base..head and pushes scan before..sha, so commits from # merged branches and beyond the first API page are covered (the action - # listed one page and used --first-parent --no-merges). A new branch or an - # unreachable `before` (force push) scans the whole history of the commit. + # listed one page and used --first-parent --no-merges). A manual run, a new + # branch or an unreachable `before` (force push) scans what the commit adds to + # main: scanning its whole history failed every manual run on findings that + # were already in the history. Only main itself still scans the whole history. - name: Select commit range env: EVENT_NAME: ${{ github.event_name }} @@ -52,6 +54,9 @@ jobs: range="$PR_BASE..$PR_HEAD" elif [ "$EVENT_NAME" = "push" ] && [ "$PUSH_BEFORE" != "0000000000000000000000000000000000000000" ] && git cat-file -e "$PUSH_BEFORE^{commit}" 2>/dev/null; then range="$PUSH_BEFORE..$PUSH_SHA" + elif git rev-parse --verify --quiet origin/main >/dev/null && [ "$(git rev-parse origin/main)" != "$PUSH_SHA" ]; then + # A manual run, or the first push of a branch: what the branch adds to main. + range="origin/main..$PUSH_SHA" else range="$PUSH_SHA" fi diff --git a/docs/operations/ci.md b/docs/operations/ci.md index 3524dba972..31c492cde0 100644 --- a/docs/operations/ci.md +++ b/docs/operations/ci.md @@ -68,8 +68,8 @@ the merge. ## Jobs - **Secrets** runs the gitleaks 8.30.1 CLI (release tarball pinned by SHA-256) over - every commit a pull request (`base..head`) or push (`before..sha`; the full - history for a new branch) adds, merged branches included, with `.gitleaks.toml`. + every commit a pull request (`base..head`) or push (`before..sha`) adds; a + manual run or a new branch scans `origin/main..sha`, merged branches included, with `.gitleaks.toml`. It then runs `scripts/no-sensitive-data.test.ts` with `BIBCODE_SENSITIVE_RANGE` set to the same range, which checks every commit message against the hashed banned values. It allowlists `.repos/` and the fixture hash manifests, posts diff --git a/scripts/ci-platform-contract.test.ts b/scripts/ci-platform-contract.test.ts index facec1acc4..23bf80a9e3 100644 --- a/scripts/ci-platform-contract.test.ts +++ b/scripts/ci-platform-contract.test.ts @@ -483,6 +483,17 @@ describe("cross-platform CI contract", () => { ); }); + it("scans only what a manually run branch adds to main for secrets", () => { + // A manual run has no base or before commit. Scanning the whole history of the branch + // failed every `gh workflow run ci.yml --ref ` on findings already in the history. + const { workflow } = readWorkflow(CI_WORKFLOW_PATH); + const select = requireJob(workflow, "secrets").steps?.find( + (step) => step.name === "Select commit range", + ); + expect(select?.run).toContain('range="origin/main..$PUSH_SHA"'); + expect(select?.run).toContain('range="$PR_BASE..$PR_HEAD"'); + }); + it("runs the Windows server tests off the native desktop rows, without saving a cache", () => { // A native row compiled bibcode-server five times; the test builds were two of them and // ran before the bundle on the same runner. Clippy with --all-targets runs on Linux only, From 5235eaee23e75b2095da88d8d42890efd4a61319 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 12:19:09 -0300 Subject: [PATCH 09/11] docs(plan): record the measured CI run and the Windows VM results Co-Authored-By: Claude Fable 5.2 --- .../2026-11-14-ci-build-times/progress.md | 29 +++++++----- .../research/measured_ci_and_local.md | 45 +++++++++++++++++++ 2 files changed, 62 insertions(+), 12 deletions(-) diff --git a/docs/plans/2026-11-14-ci-build-times/progress.md b/docs/plans/2026-11-14-ci-build-times/progress.md index 7d341e5c1e..26450b928a 100644 --- a/docs/plans/2026-11-14-ci-build-times/progress.md +++ b/docs/plans/2026-11-14-ci-build-times/progress.md @@ -7,18 +7,18 @@ Plan review: Codex gpt-6.1-sol, round 2, findings handled (reviews/plan-review-1 Executor: inline Branch: claude/build-times-investigation-ec72b8 -| Task | Title | Phase | State | Owner | Note | -| ---- | ---------------------------------------------------------------- | ----- | ----- | ----- | ---- | -| 1.1 | Attribute a dump_syms error and verify the stored unwind records | 1 | Open | | | -| 2.1 | Add --workspace-codegen-units to the bundle script | 2 | Open | | | -| 3.1 | Move the Windows server tests into server_tests_windows | 3 | Open | | | -| 3.2 | Pass the override and record a heartbeat in the bundle step | 3 | Open | | | -| 3.3 | Save pnpm store entries only on main | 3 | Open | | | -| 4.1 | Print the collected output and exit state on timeout | 4 | Open | | | -| 4.2 | Measure the #367 rate under load in the Windows arm64 VM | 4 | Open | | | -| 5.1 | Update the living documents | 5 | Open | | | -| 5.2 | Local gates, review and pull request | 5 | Open | | | -| 5.3 | Accept the requirements on the measured run | 5 | Open | | | +| Task | Title | Phase | State | Owner | Note | +| ---- | ---------------------------------------------------------------- | ----- | ------ | ------ | ---- | +| 1.1 | Attribute a dump_syms error and verify the stored unwind records | 1 | Closed | inline | | +| 2.1 | Add --workspace-codegen-units to the bundle script | 2 | Closed | inline | | +| 3.1 | Move the Windows server tests into server_tests_windows | 3 | Closed | inline | | +| 3.2 | Pass the override and record a heartbeat in the bundle step | 3 | Closed | inline | | +| 3.3 | Save pnpm store entries only on main | 3 | Closed | inline | | +| 4.1 | Print the collected output and exit state on timeout | 4 | Closed | inline | | +| 4.2 | Measure the #367 rate under load in the Windows arm64 VM | 4 | Closed | inline | | +| 5.1 | Update the living documents | 5 | Closed | inline | | +| 5.2 | Local gates, review and pull request | 5 | Closed | inline | | +| 5.3 | Accept the requirements on the measured run | 5 | Closed | inline | | ## Log @@ -27,3 +27,8 @@ Branch: claude/build-times-investigation-ec72b8 - 2026-11-14 09:38 3.1, 3.2, 3.3 Closed. Evidence: `vp test run scripts/ci-platform-contract.test.ts` 53 passing (3 new cases failed first). Heartbeat probes: a child exiting 7 gives exit 7 and the notice; a `ps` that fails gives heartbeat lines without memory and exit 0; a heartbeat that died early no longer fails a good build (found by the probe: the trap's `kill` ran under `bash -e`, fixed with `|| true`). - 2026-11-14 09:45 5.1 Closed. Evidence: `vp run check:markdown-links` passes; `vp test run scripts` 952 passing after `scripts/rust-workspace.test.ts` was updated for the sixth Rust job. - 2026-11-14 09:45 4.1 In progress. Owner: inline. The helper and its test are written; the Windows arm64 VM is compiling them. +- 2026-11-14 10:15 4.1 Closed. Evidence: Windows arm64 VM, `cargo test -p bibcode-server --test windows_terminal_shims` 24 passing (again after the test was changed to wait for the first line before timing), `cargo check --all-targets` and Clippy with warnings denied pass, and with the collected output dropped from the message the test fails with "output=dropped". +- 2026-11-14 10:12 4.2 Closed. Result: 0 of 30 failed, four busy processes, 6 cores, warm runs. Not a reproduction of the cold first spawn; R7 recorded, #367 stays open. +- 2026-11-14 10:20 5.2 Closed. Evidence: gates pass, Codex reviewed the branch (three findings fixed), delivery checkpoint accepted, pull request 370 opened. The label on `gh pr create` did not start the macOS rows, so the partial run was cancelled and one manual full run measured the branch. +- 2026-11-14 12:20 5.3 Closed. Accepted on the measured run (numbers in `research/measured_ci_and_local.md` section 6 and in the pull request): R2 (46 and 59 minutes against 89 and 103), R3 (6 `Compiling` lines on full-match rows; bundle builds 18 to 59 minutes against 29 to 88), R4 (macOS arm64 memory and swap recorded), R5, R6, R8. D3 not applied: the macOS arm64 bundle build took 59 minutes against 88 uncached before. R7: 0 of 30, not reproduced. +- 2026-11-14 12:20 Ruling: manual runs and new branches scan `origin/main..sha` for secrets, because scanning the whole history failed every manual full run. Decided by the user on 2026-11-14. Cost if wrong: a secret already in main's history is not re-reported by a manual run; pull request and push scans are unchanged. diff --git a/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md index 83ad9eef34..abdbcf242f 100644 --- a/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md +++ b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md @@ -124,3 +124,48 @@ Run 43523947929, attempt 2: `running 10 tests` at 02:01:33; the four tests that first each failed after exactly 10 s (`Elapsed`), and the other six passed within the next 5 s. The same step normally finishes in 3.7 to 4.6 s on Windows arm64 and 2.4 to 3.0 s on x64 (9 other runs). + +## 6. The pull request's measured run (2026-11-14, manual full run on the branch) + +Every job passed except Secrets, which failed because a manual run scanned the branch's whole +history (fixed in the same pull request). + +| Job | Before (run 43562279474 unless noted) | This run | +| ------------------------------------- | ------------------------------------- | -------- | +| Native desktop, Windows x64 | 89 min | 46 min | +| Native desktop, Windows arm64 | 103 min | 59 min | +| Native desktop, Linux x64 | 48 min | 31 min | +| Native desktop, Linux arm64 | 52 min | 35 min | +| Native desktop, macOS arm64 | 115 min | 90 min | +| Native desktop, macOS x64 | 132 min | 93 min | +| Server tests, Windows x64 (new job) | inside the native row | 27 min | +| Server tests, Windows arm64 (new job) | inside the native row | 34 min | + +Cargo time of the bundle build (the `Finished` line): + +| Row | Before | This run | Rust cache in this run | +| ------------- | ------------------------------ | ------------------- | ---------------------- | +| Linux x64 | about 29 min, nothing restored | 17 min 55 s | full match | +| Linux arm64 | 25 to 33 min for the step | 16 min 41 s | nothing restored | +| Windows x64 | about 42 min | 30 min 12 s | full match | +| Windows arm64 | 36 min 15 s | 28 min for the step | not read | +| macOS arm64 | 87 min 51 s, nothing restored | 59 min 18 s | nothing restored | +| macOS x64 | 69 to 101 min for the step | 54 min 53 s | nothing restored | + +- On the two rows with a full cache match the whole job log holds 6 `Compiling` lines, so the + per-package override left the dependency artifacts usable. +- macOS arm64 heartbeat: free memory fell to about 2,000 to 5,000 pages from minute 16 to minute + 45, swap in use rose from 33 MB to 975 MB, and the largest `rustc` held only 1.6 to 3.6 GB + resident while the same compile holds 8.7 to 12 GB on a machine with enough memory. The + runner is short of memory for this compile. `dsymutil` appeared in one sample at 108 MB. +- macOS x64 heartbeat (14 GB): the largest `rustc` reached 7.7 GB resident, swap in use peaked at + 327 MB. +- Linux x64 heartbeat (16 GB): the largest `rustc` reached about 6.3 GB with 8 GB available. +- No `dump_syms` CFI error was logged in this run. +- The new `shim_timeout_` test passed on both Windows architectures (11 tests in the step). + +Windows arm64 VM (6 cores), same day: the shim test file passed (24 tests), Clippy and +`cargo check --all-targets` with warnings denied passed, the test failed as intended when the +collected output was dropped from its message, and 30 runs of the review prompt tests under four +busy processes gave 0 failures. Those runs were warm, so this does not reproduce the cold first +spawn seen in CI. From c2d1565a83190802d235b6ceeef08d527c59deb6 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 16:44:00 -0300 Subject: [PATCH 10/11] test(terminal): give Windows shim tests 60 seconds for a stalled first spawn (#367) Reproduced on fresh Windows arm64 runners: 1 of 80 cold runs failed with the issue's symptom. The pseudoconsole was up (its init sequences arrived) and the shim had not printed after 10 seconds; identical spawns seconds later passed, and that run's whole batch took 19.8 s against a usual 4 to 7. These tests check what a shim receives, not how fast a runner starts processes. Co-Authored-By: Claude Fable 5.2 --- apps/server/tests/windows_terminal_shims.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/apps/server/tests/windows_terminal_shims.rs b/apps/server/tests/windows_terminal_shims.rs index 4d92d6a209..01316fc46c 100644 --- a/apps/server/tests/windows_terminal_shims.rs +++ b/apps/server/tests/windows_terminal_shims.rs @@ -29,6 +29,12 @@ const WINDOWS_CONSOLE_THEME_ENV: &str = "BIBCODE_WINDOWS_CONSOLE_THEME"; const TRUSTED_CMD_FIXTURE_ENV: &str = "BIBCODE_TEST_TRUSTED_CMD_FIXTURE"; const NATIVE_ARGV_MARKER: &str = "BIBCODE_NATIVE_ARGV"; const TERMINAL_PROCESS_TREE_INTEGRATION_DEADLINE: Duration = Duration::from_secs(30); +/// How long a shim may take to print its marker and then to exit. These tests check what a +/// shim receives, not how fast a runner starts processes: on a fresh Windows arm64 CI runner the +/// first terminal spawns usually take 4 to 7 seconds, and in 1 of 80 cold runs they stalled for +/// about 20 seconds, with the pseudoconsole up and the shim not yet printing (#367). A shim +/// that never prints still fails, with what the terminal showed. +const SHIM_DEADLINE: Duration = Duration::from_secs(60); struct OwnedHandle(HANDLE); @@ -189,7 +195,7 @@ async fn assert_shim_output( .unwrap(); let mut output = process.subscribe_output(); let mut exit = process.subscribe_exit(); - let text = wait_for_marker(&mut output, &mut exit, marker, Duration::from_secs(10)) + let text = wait_for_marker(&mut output, &mut exit, marker, SHIM_DEADLINE) .await .unwrap_or_else(|timed_out| panic!("{timed_out}")); assert!( @@ -197,7 +203,7 @@ async fn assert_shim_output( "terminal exited without the expected marker; output={text:?}, exit={:?}", exit.borrow().clone() ); - tokio::time::timeout(Duration::from_secs(10), exit.changed()) + tokio::time::timeout(SHIM_DEADLINE, exit.changed()) .await .unwrap() .unwrap(); From 6f02c9462ee996e080bba49c754195093fb10862 Mon Sep 17 00:00:00 2001 From: mubeda <1380447+mubeda@users.noreply.github.com> Date: Sat, 14 Nov 2026 17:24:05 -0300 Subject: [PATCH 11/11] docs(plan): record the #367 reproduction and its verification Co-Authored-By: Claude Fable 5.2 --- .../2026-11-14-ci-build-times/progress.md | 1 + .../research/measured_ci_and_local.md | 24 +++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/docs/plans/2026-11-14-ci-build-times/progress.md b/docs/plans/2026-11-14-ci-build-times/progress.md index 26450b928a..3f03c39b5f 100644 --- a/docs/plans/2026-11-14-ci-build-times/progress.md +++ b/docs/plans/2026-11-14-ci-build-times/progress.md @@ -32,3 +32,4 @@ Branch: claude/build-times-investigation-ec72b8 - 2026-11-14 10:20 5.2 Closed. Evidence: gates pass, Codex reviewed the branch (three findings fixed), delivery checkpoint accepted, pull request 370 opened. The label on `gh pr create` did not start the macOS rows, so the partial run was cancelled and one manual full run measured the branch. - 2026-11-14 12:20 5.3 Closed. Accepted on the measured run (numbers in `research/measured_ci_and_local.md` section 6 and in the pull request): R2 (46 and 59 minutes against 89 and 103), R3 (6 `Compiling` lines on full-match rows; bundle builds 18 to 59 minutes against 29 to 88), R4 (macOS arm64 memory and swap recorded), R5, R6, R8. D3 not applied: the macOS arm64 bundle build took 59 minutes against 88 uncached before. R7: 0 of 30, not reproduced. - 2026-11-14 12:20 Ruling: manual runs and new branches scan `origin/main..sha` for secrets, because scanning the whole history failed every manual full run. Decided by the user on 2026-11-14. Cost if wrong: a secret already in main's history is not re-reported by a manual run; pull request and push scans are unchanged. +- 2026-11-14 17:25 Ruling: #367 is fixed in this pull request after all. It was reproduced on fresh Windows arm64 runners (1 of 80 cold runs, the issue's symptom, output printed by Task 4.1's diagnostics), the shim helper's deadline went from 10 to 60 seconds, and 240 cold runs with the new deadline all passed (`research/measured_ci_and_local.md` section 7). This replaces spec D6. Cost if wrong: a rarer or longer stall fails again, and prints what the terminal showed. diff --git a/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md index abdbcf242f..a4fbc97817 100644 --- a/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md +++ b/docs/plans/2026-11-14-ci-build-times/research/measured_ci_and_local.md @@ -169,3 +169,27 @@ Windows arm64 VM (6 cores), same day: the shim test file passed (24 tests), Clip collected output was dropped from its message, and 30 runs of the review prompt tests under four busy processes gave 0 failures. Those runs were warm, so this does not reproduce the cold first spawn seen in CI. + +## 7. #367 reproduced and fixed on fresh Windows arm64 runners (2026-11-14) + +A temporary workflow on a branch that is not merged built the shim test binary once and ran the +ten review prompt tests as the first terminal spawn of a fresh `windows-11-vs2026-arm` runner, +then once more warm. + +| Run | Deadline | Cold runs | Failed | First-spawn time of the ten tests | Warm | +| ---------------------------------------------- | -------- | --------- | ------ | ------------------------------------------- | ------------ | +| 1 (with the 5 s timeout test in the same step) | 10 s | 30 | 0 | 8.4 to 13.3 s for the step | 5.8 to 6.5 s | +| 2 | 10 s | 80 | 1 | 3.8 to 6.7 s, and 19.8 s in the failing run | 0.9 to 1.3 s | +| 3 | 60 s | 240 | 0 | 3.6 to 10.4 s, median 4.5 s | about 1 s | + +- The failing run of run 2 showed the issue's symptom exactly: the four tests libtest started + first timed out at 10 s and the six started after them passed. +- Its timeout message read `output="\u{1b}[?9001h\u{1b}[?1004h", exit=None`: the pseudoconsole + was up and had sent its start-up sequences, and the shim had neither printed nor exited. +- Identical spawns started seconds later in the same process passed, and the whole batch took + 19.8 s, so the runner stalled process start-up for roughly 20 seconds. What stalls it is not + known. +- 240 cold runs is the count chosen in advance for a 1 in 80 rate (the chance that all pass while + the failure is still there is at most 5%). None failed. No run in run 3 showed a 20-second + stall, so the stall is rarer than run 2 suggested, and the fix was not seen turning a stalled + run into a pass.