[maschwenk]

start on item 5 from the bake-off: one vocabulary for agents driving apps and browser tabs. design note only, docs/internals/computer-use.md, no code yet

recommendation

• keep computer_* and preview_* as two families, but both speak the same steps, refs and output. not one family with a target selector, since only snapshot and act overlap and everything else is per target anyway. and not a codex-style js repl: codex's models are trained on theirs and ours arent, and codex's first getApp on the bake-off page was ~23k chars, ~20k of it api docs
• steps copy upstream's preview_act flat shape ({action:"click", ref:12}). preview_* belongs to upstream and they already folded the per-action tools into preview_act, so computer_* (fork-only) is what moves. press always means a key now, click absorbs press/click_at, and hover absorbs move/move_at
• refs are integers carried per target on both. i spiked it: playwright 1.60 ariaSnapshot({mode:"ai"}) refs already follow our carryRefs rule (8 of 10 carried over, a renamed element gets a new ref, removed refs never come back). only catch is that navigation restarts at e1, so tabs keep an int → playwright ref map whose counter never resets
• every act on both returns the ~ / + / removed: diff that computer_action got in the diffs PR below
• don't wait on upstream's cua stack and don't adopt its vocabulary

first step
computer_action steps take the shared verbs. only fork-only files change: a flat mcp step schema maps onto the worker's existing actions, so the native worker stays as is. ~6 files, +180/−150. verified with planStep tests per verb, plus the 11-step bake-off on chrome ported from the worker protocol to mcp calls through computer-use-dev.ts (same dom events as codex, cursor and front app untouched, responses under ~600 chars)

tradeoff: two families means two tool names per verb forever, and tabs and apps still deliver input differently (tab typing is one insertText event, tab scroll is js, no wheel). the note says tabs should report via the way apps do instead of pretending they're the same

decided (11-08)

• step 1 is done: the bake-off is node scripts/computer-use-dev.ts bakeoff and computer_action takes the shared verbs, plus reflowed elements go on one moved: line instead of a ~ line each (hover/scroll responses ~4.1k → ~450 chars)
• preview_observe / preview_execute: port them as is in the next upstream sync, retire them once preview_act takes steps + returns diffs (step 4)
• upstream discussion offering the tab half: holding for now
• the doc's table and migration notes now describe computer_action as it is after step 1

plan (lives here, AGENTS.md keeps plans out of the repo)

Every step is accepted by the 11-step bake-off:

• Tasks: button, canvas point, hover by ref and by point, revealed button, typing, Return, list scroll, drag, select.
• Page: logs every DOM event with isTrusted.
• Harness: today it drives the native worker protocol directly. Each step ports it to MCP calls through scripts/computer-use-dev.ts, so it can run against Chrome as an app and against a preview tab.
• Pass:
  • each step logs the same events as Codex's run (kind, trusted, point);
  • the final state matches;
  • the user's cursor and front app don't move;
  • each response stays within Codex's 125–580 characters per action.

The steps:

1. computer_action steps take the shared verbs. Fork-only. A flat MCP step schema maps onto the worker's existing actions, so nativeWorker.ts and macNative.ts don't change.
   • Files: protocol.ts, ComputerUse.ts, computer/tools.ts, ComputerUse.test.ts, the runbook and computer-use-dev.ts. About +180/−150.
   • Verify: planStep tests for each verb (a click on a pressable element becomes the accessibility press; a click at x,y becomes an exact point), then the bake-off on Chrome.
2. The upstream sync takes preview_act. This is needed anyway. It ports preview_observe and preview_execute unchanged.
3. Tabs use the shared format. A new fork-only module turns ariaSnapshot({ mode: "ai", boxes: true }) into elements with integer refs, and the apps formatter moves into a shared module. preview_act takes ref or name + role.
   • Size: about 5 files, +350.
   • Verify: an integration test on a real headless page that makes the spike's changes, then the bake-off on a preview tab.
4. preview_act takes steps and returns the diff.
   • preview_observe and preview_execute go away (BrowserExecutor.ts alone is 302 lines).
   • Rewrite the injected browser block and add presentation rows.
   • Size: about 7 files, net negative.
   • Verify: the bake-off on both targets.
5. Tab coordinates use image pixels, and tabs report via.
   • Size: about 3 files, +60.
   • Verify: the canvas step lands at the requested point (101, 60).

First step: step 1. It is about 330 lines in fork-only files with no upstream conflict. After it, an agent using both families sees one set of verbs.

refs

• feat(server): cut T3's MCP tool schemas by a third and merge 18 tools into action tools pingdotgg/t3code#25423
• feat(cua): host Cua Driver on macOS, Linux, and Windows pingdotgg/t3code#16527
• feat(cua): let agents use your signed-in browser tabs on any host pingdotgg/t3code#16532
• perf(computer-use): computer_action returns what changed, not the whole tree #36
• perf(preview): add compact observations with explicit baselines #23
• feat(preview): execute browser batches with persistent tab handles #25
• test(computer-use): bake-off subcommand for background Chrome #39
• feat(computer-use): computer_action takes the shared step verbs #40