[maschwenk]

step 1 of the design note. computer_action steps and the browser tools spoke two languages: apps used {ref, action: {kind: "press"}}, key, click_at, move_at, move, dx/dy, replace, tabs use click, hover, type (clear), press (key), scroll (deltaX/deltaY), drag. and press meant opposite things, activate a control on apps vs press a key on tabs. an agent using both had to keep two vocabularies in its head

what changed

• steps are flat and use upstream preview_act's verbs + field names: {action: "click", ref: 17}, {action: "click", x: 101, y: 60}, {action: "hover", ref: 30}, {action: "type", name: "Name", text: "hi", clear: true}, {action: "press", key: "Return"}, {action: "scroll", ref: 4, deltaY: 100}, {action: "drag", ref: 4, target: {ref: 9}}. apps also keep perform (perform: "expand"), menu (path) and wait (ms)
• press is always a key now, and takes upstream's modifiers too ({key: "=", modifiers: ["Shift"]}). activating a control is click: a single click on an element listing press already went through the ax press in the worker, and a right click on one listing show_menu opens its menu. click/hover at x, y are the old click_at/move_at, exact point
• planStep maps each verb onto the worker's existing actions, so nativeWorker.ts and macNative.ts don't change. it also checks what each verb needs (type with no text, scroll with no delta, an element and a point on the same step → invalid_input)
• tool descriptions, error messages, the runbook, smoke and the bake-off all use the new verbs
• planStep tests per verb, the click one goes through the worker's real backgroundAction to show a click on a pressable element ends up as the ax press

+187/−122 outside tests, +313/−194 with them. the note guessed +180/−150

bake-off from the pr below, each side right after a fresh dev server start (ref numbers grow with uptime and change response length a bit, so this keeps sizes comparable)

task	page events, both	before chars, via	after chars, via
button	mousemove, mousedown, btn.click	438 pointer_event	438 pointer_event
canvas click at 101,60	mousemove, mousedown, canvas.click 101,60	460 pointer_event	460 pointer_event
hover by ref	hover.enter, mousemove	4145 pointer_event	4145 pointer_event
move away by point	mousemove	4083 pointer_event	4083 pointer_event
hover by point	hover.enter, mousemove	4145 pointer_event	4145 pointer_event
revealed button	mousedown, secret.click	457 accessibility	457 accessibility
typing	keydown ×11, input ×11	495 keyboard_event	495 keyboard_event
Return	keydown Enter	187 keyboard_event	187 keyboard_event
list scroll	wheel	4184 pointer_event	4184 pointer_event
drag	mousemove ×2, drag.down, mousedown, drag.up inside	553 pointer_event	553 pointer_event
select	mousedown, select chocolate	1051 + 823 accessibility	1051 + 823 accessibility

11/11 both sides, identical down to the char, my pointer and front app (helium) never moved. 2 more runs on this branch with a warm server were also 11/11

calculator 7 + 8 = through the new verbs, 15 every way, pointer and front app unchanged:

• smoke (clicks by ref): 5 steps via accessibility
• clicks by name, {action: "click", name: "Add"} etc: 5 steps via accessibility
• keys, {action: "press", key: "=", modifiers: ["Shift"]} for +: 5 steps via keyboard_event, last expression went 9+6 → 7+8
• clicks at x, y from an includeImage snapshot: 5 steps via pointer_event

also in here: elements that only reflowed go on one moved: line. revealing the hover button makes its row 3 px taller, so every element below it came back as a ~ line just for its new position, ~4.1k chars per hover/scroll. now an element whose only change is position or size under the same parent is listed in moved: 3-4, 9-40. a change of parent is still a ~ line (calculator's "7+8" moving into Last Expression), and the tool text says to take an includeImage snapshot before using coordinates near moved refs. codex's ax text has no coordinates at all, so it never had this

task	before chars	after chars
hover by ref	4145	440
move away by point	4083	378
hover by point	4145	440
list scroll	4184	456

same bake-off, 11/11, same page events, pointer and front app unchanged

tradeoffs

• no aliases. old {action: {kind: "press"}} steps now fail schema decode. agents read the schema every session, and the note wants no second name per tool. old threads only render, and computer calls still go through the generic renderer
• one flat struct with optional fields like upstream, not a union per verb, so the advertised schema doesn't say which fields go together, planStep does. unknown fields are ignored, same as before
• a click on an element that doesn't list press used to be a press that failed with unsupported_action. now it's a posted pointer click. more forgiving, but it is real pointer input. chrome right after launch is exactly this case, see the button row
• drag's destination is target (upstream's field name) but takes {ref}, {name, role} or {x, y}, not a locator string
• clickCount is 1-2 and button left/right, upstream allows 3 and middle. those get a decode error instead of a silent downgrade

not in here

• {action: "press", key: "+"} presses the = key, keys.ts maps by us key code (already like that). playwright's press("+") types a plus. use modifiers: ["Shift"] or type

• no presentation rows for computer_* yet, still the generic renderer (later step)

• docs(computer-use): one vocabulary for apps and browser tabs #38

• test(computer-use): bake-off subcommand for background Chrome #39

written by claude opus 5.5 in claude code

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.9 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	21.2 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	1	—	8	✅

Baseline: unavailable · PR result: 855cd3f · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 108.5 KiB
• Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.