Skip to content

ci: stop the macOS native rows building the app twice, and run macOS server tests in their own job - #359

Merged
mubeda merged 5 commits into
mainfrom
fix/macos-ci-index-lock-step-time
Nov 13, 2026
Merged

mubeda merged 5 commits into
mainfrom
fix/macos-ci-index-lock-step-time

Conversation

@mubeda

@mubeda mubeda commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What this changes

The two macOS native desktop rows ran out of time on main (240 minutes each) after pull request 349. Two causes, both fixed here.

  1. The exception probe compiled the whole app a second time. Verify macOS release exception recovery built an example of bibcode-desktop. An example enables the dev-dependency features, so Cargo compiled the desktop and server graph once for the probe and again for the bundle. On 2026-11-13 the probe's build took 84 of the 215 minutes of the arm64 row and 81 of 228 on x64. The probe only uses objc2 and objc2-foundation, so it now lives in its own crate, tools/objc-exception-probe, with the objc2 features the desktop graph enables, under the shipped release profile.
  2. Three macOS server test steps sat in the same job. They now run in their own job, Server tests - macOS arm64 / x64 (150 minutes, restores the Rust cache, saves none).

Both macOS native rows get 330 minutes. arm64 used 215 of its 240 with a Rust cache.

Why the slowdown happened

Pull request 301 (2026-11-05) turned on debug info for both optimized builds (strip none, debug line-tables-only, split-debuginfo packed). On the same runner image and with a Rust cache hit, the two builds went from 43m 47s and 38m 23s ([optimized]) to 75m 55s and 73m 41s ([optimized + debuginfo]). The runner image, a cold cache and the Breakpad dump (26 to 41 seconds) were ruled out from the job logs.

Evidence

Check Result
New probe, release profile, macOS arm64 (local) builds and passes in 16 seconds
Same probe with CARGO_PROFILE_RELEASE_PANIC=abort (local) aborts, exit 134, as it must
objc2 features, probe and desktop graph (cargo tree -e features -i objc2) identical: alloc, default, disable-encoding-assertions, exception, std
vp test run scripts 941 passed, 23 skipped
vp check, vp run typecheck, cargo fmt --all --check, Clippy on the new crate with warnings denied pass
Server tests on macOS in their own job (run on the previous commit) arm64 54 minutes, x64 40 minutes, both passed

To confirm in this pull request's CI run: the probe step takes minutes, and each macOS native job has one long optimized build instead of two.

Not fixed here

  • The bundle build still pays for debug info: 38 minutes before pull request 301, 75 to 111 since. How that splits between compiler, link and dsymutil is not measured.
  • The bundle build grew from about 75 to about 107 minutes after 2026-11-07; the cause is not known.
  • No run without a Rust cache has completed, so the 330-minute limit is not proven for a cold run.
  • Neither the old nor the new probe exercises a real WebKit cancellation.

Profiler coverage: none applies. The new binary is a CI-only tool, is not shipped, starts no process and registers no RPC.

Runbooks: docs/testing/macos-desktop.md updated (probe command); the others were reviewed and remain accurate.

🤖 Generated with Claude Code

Both macOS Native desktop rows hit their 240-minute limit on main after
the background Git index lock step was added: the server's test build
took 32 to 56 minutes there, and the run had no Rust cache.

Move the three macOS server test steps (Git system configuration, index
lock, host metrics) to a new job, server_tests_macos, which restores the
Native desktop cache and never saves one. Raise the macOS x64 row to 300
minutes: its remaining steps needed about 260 minutes cold.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@mubeda mubeda added the ci:full Run full macOS + seeded upgrade CI label Nov 13, 2026
The Intel figure is an estimate from a cancelled run, so leave more room
under the 360-minute hosted limit. The contract test now looks at every
step that runs on the macOS rows, and requires 330 minutes for x64.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T13:54:30.491266Z 4c696a9 PR opened
🔒 Security Review ✅ Completed 2026-11-13T13:55:59.543279Z 4c696a9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Nov 13, 2026
mubeda and others added 3 commits November 13, 2026 11:45
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The probe was an example of bibcode-desktop, so Cargo compiled the whole
desktop and server graph a second time with the dev-dependency features
before the bundle build compiled it again: 84 of the 215 minutes of the
macOS arm64 row on 2026-11-13, and 81 of 228 on x64. The probe uses only
objc2 and objc2-foundation, so it now lives in tools/objc-exception-probe
with the objc2 features the desktop graph enables and runs under the
shipped release profile. Locally it builds and passes in 16 seconds and
aborts with CARGO_PROFILE_RELEASE_PANIC=abort.

Both macOS rows get 330 minutes: arm64 used 215 of its 240 with a Rust
cache, and the bundle build alone has taken 75 to 111 minutes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@mubeda mubeda changed the title ci: run the macOS server tests in their own job ci: stop the macOS native rows building the app twice, and run macOS server tests in their own job Nov 13, 2026
@mubeda
mubeda merged commit 706d63b into main Nov 13, 2026
30 checks passed
@mubeda
mubeda deleted the fix/macos-ci-index-lock-step-time branch November 14, 2026 10:46
mubeda added a commit that referenced this pull request Nov 14, 2026
Bump every version source to 0.16.0 and turn the Unreleased CHANGELOG
section (PRs #349, #359, #365 and #369) into the v0.16.0 entry. Raise
the Claude Code additive-hook ceiling to 2.1.323 after the live
PreToolUse guard proof passed on that build, and move the above-ceiling
test example to 2.1.324. Give the CI Test lookup contract test a
30-second limit: it runs bash and jq eleven times and timed out against the
5-second default under load.

Verified locally: cargo check for the workspace, vp check, vp run
typecheck, the full vp run -r --concurrency-limit 1 test graph in all
nine packages, release:smoke, the updater regression sets, cargo test
-p bibcode-desktop, and the macOS arm64 DMG signature and Finder icon
check. CI: the Seeded Packaged Desktop Upgrade workflow from v0.8.0
passed on all seven lanes for this branch (macOS x64 on a rerun after
its hosted runner lost its connection). Reviewed by Codex with no
findings.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run full macOS + seeded upgrade CI size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant