Skip to content

fix(browser): background tab screenshots, wait_for, visibility reason, DMG manifest - #71

Merged
nickrroberts merged 5 commits into
mainfrom
fix/browser-background-capture
Oct 22, 2026
Merged

nickrroberts merged 5 commits into
mainfrom
fix/browser-background-capture

Conversation

@nickrroberts

@nickrroberts nickrroberts commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Why

After 0.0.66, agents still couldn't screenshot background browser tabs in the installed app, preview_open(open: true) came back visible: false with no explanation, and 0.0.66's update manifest listed a stale DMG checksum.

Root causes (reproduced in isolated dev runtimes)

  • Background screenshots: packaged builds hand the main window back to background throttling after first paint (DesktopWindow.ts reveal handler). With the window minimized or occluded, an offscreen agent tab paints about once a second, so every capturePage attempt (1s x3) timed out while evaluate and click kept working. Dev builds never throttle, which is why it looked fine in dev. Reproduced on an isolated 0.0.65 build with packaged throttling injected and the window minimized: every snapshot failed with capturePage timeouts.
  • preview_wait_for returned success immediately for any selector: Playwright's injected querySelector returns undefined, the check compared !== null. Found live (11ms "success" on a missing selector).
  • visible=false after open=true: by design visible means on screen right now, but nothing told the agent the tab is open in a thread the human isn't viewing.
  • Stale DMG digest: electron-builder hashes the DMG before codesign/notarize/staple rewrite it.

Changes

  • Still captures take a throttling lease: main window and guest are unthrottled for the capture's duration, then restored to their previous state. The lease is coordinated with recording/PiP sessions so neither re-throttles under the other.
  • wait_for uses != null.
  • PreviewAutomationStatus.presentation: on-screen | shown-in-thread | background (optional, backward compatible), documented on the tools.
  • build-desktop-artifact refreshes DMG sha512/size in *-mac.yml after stapling.

Verification

  • Unit: wait_for timeout/appear (fails on old code), still-capture lease incl. overlap with recording (fails without lease), dev window stays unthrottled, presentation states, manifest refresh incl. primary path.
  • Live (isolated dev desktop, real Codex agent through MCP, broker, host, desktop): see PR comment.

nickrroberts and others added 5 commits October 21, 2026 18:07
… yet

Playwright's injected querySelector returns undefined when nothing
matches, and the match check compared with null, so every selector wait
succeeded immediately. Found live: a wait on a missing selector returned
in 11ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
visible only reports whether the human can see the tab now, so an
open=true preview in a thread the human is not viewing came back
visible=false with no explanation. Status and open now include
presentation: on-screen, shown-in-thread, or background.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
electron-builder hashes the DMG before codesign and stapling rewrite it,
so 0.0.66 shipped a manifest whose DMG sha512 and size did not match the
published file. Recompute them after stapling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w is hidden

Packaged builds hand the main window back to background throttling after
first paint. When the window is minimized or occluded, an agent's offscreen
browser tab then paints about once a second, so every capturePage attempt
missed its 1s deadline: snapshots and screenshots failed while evaluate and
click kept working.

Still captures now lift main-window and guest throttling for their
duration. The lease is counted together with recording and picture-in-
picture sessions, so neither re-throttles under the other.

Reproduced in an isolated dev build with packaged throttling and a
minimized window: 0.0.65 failed every snapshot with 3x1s capturePage
timeouts, this change captures in 7-40ms and restores throttling after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Restore the main window's previous throttling after a still capture
  instead of forcing it on, so dev builds stay unthrottled.
- Replacing the main window or failing a frame-capture start no longer
  re-throttles while a still capture holds its lease.
- presentation counts a right-panel browser tab as shown in its thread,
  and only reports on-screen while the app window is visible.
- preview_status documents presentation.

Verified live: after a snapshot from a minimized window the guest goes
back to zero frames and the window is throttled again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Oct 22, 2026
@nickrroberts

Copy link
Copy Markdown
Collaborator Author

Live runtime evidence (isolated dev desktops, separate homes, real Codex agent driving preview_* through MCP, broker, web host and desktop)

Background screenshot defect, reproduced, then fixed under the same fault

  • Fault: packaged main-window throttling injected through an uncommitted, env-gated edit. The dev window was minimized from the Electron main inspector while the agent's background tab (open=false, thread off screen) was captured.
  • 0.0.65 (cc4d7b45ec): automationSnapshot.capturePage failed in 3254ms twice (3x1s timeouts). evaluate succeeded.
  • This branch: snapshots succeeded in 40ms and 7ms, and the saved PNG shows the rendered page. 8s after a capture from a minimized window, the guest was back to 0 frames and main-window throttling was true again.
  • Without the fault (dev default): background snapshots work on both versions, including 12 tabs at once and 6+ minutes idle.

wait_for

  • Before: preview_wait_for #never-appears returned success in 11ms.
  • After: the 60s waits time out at about 60.0s, and an 8-12s wait times out on its own budget.

presentation (raw tool output)

  • open=true while the thread was viewed: "visible":true,"presentation":"on-screen".
  • Same tab after navigating away: "visible":false,"presentation":"shown-in-thread".
  • open=false: "presentation":"background".

Timeout isolation (0.0.66 broker fix, now live)

  • Thread A held seven back-to-back 60s wait_for timeouts on a background tab (18:05 to 18:12).
  • Concurrently, thread B's open, snapshot, evaluate, click and screenshot all succeeded on the same host (18:05:57 to 18:06:07).
  • Afterwards, thread A's evaluate, status and both snapshots succeeded.
  • No "went silent", ClientDisconnected or NoAvailableHost occurred.

Not proven live

  • A packaged signed build with real macOS occlusion. That is covered by the injected throttling plus minimize.

@nickrroberts

Copy link
Copy Markdown
Collaborator Author

Local checks vs origin/main 2e2ad7a407 (no remote CI)

  • Tests, per package: no new failures. server 51 vs main 53 (2 main-only flakes), web 9 = 9, desktop 4 = 4 (main also had 3 file-load flakes), scripts 1 = 1. client-runtime, shared and contracts pass.
  • typecheck: no new errors.
  • lint: 740 findings, identical to main.
  • Independent review: nothing blocking. M1, L1, L2, L3 and L4 fixed in 63cb900. M2 (window staying awake) was refuted live.

@nickrroberts
nickrroberts merged commit 577b786 into main Oct 22, 2026
6 of 15 checks passed
@nickrroberts
nickrroberts deleted the fix/browser-background-capture branch October 22, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant