From affd44c26ed208d5b1fa933a74df6f866b837ba0 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 19:51:54 +0000 Subject: [PATCH 01/13] Add cookbook: Playwright mode with a CUA fallback Documents three ways to hand a stuck playwright-mode agent off to computer use (trigger-tool budget, flat step-count budget, mixed toolset with a steering prompt) using @onkernel/browser-loop. --- browsers/cua-fallback.mdx | 351 ++++++++++++++++++++++++++++++++++++++ docs.json | 3 +- 2 files changed, 353 insertions(+), 1 deletion(-) create mode 100644 browsers/cua-fallback.mdx diff --git a/browsers/cua-fallback.mdx b/browsers/cua-fallback.mdx new file mode 100644 index 0000000..c79e106 --- /dev/null +++ b/browsers/cua-fallback.mdx @@ -0,0 +1,351 @@ +--- +title: "Playwright Mode with a CUA Fallback" +description: "Run an agent on fast DOM-ref tools, hand it off to computer use only when the DOM won't cooperate" +--- + +Some page interactions don't survive a trip through the DOM. Pointer-based drag-and-drop is the common one: a library like `dnd-kit` or `SortableJS` needs a real sequence of mousemove events to recognize a drag, and a DOM-ref tool that jumps straight from press to release never crosses that threshold. The tool call still reports success — the card just doesn't move. + +The fix isn't to run every task through the slower, pixel-based path. It's to run fast by default and fall back to computer use only for the step that actually needs it, without losing the agent's place in the task when it does. + +This cookbook covers three ways to wire that handoff with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), Kernel's tool package for browser agents, plus the code for each. + +## The example task + +Every snippet below runs the same task against [magnitasks.com](https://magnitasks.com), a public Kanban-style board: + +``` +Go to magnitasks.com, navigate to Tasks, filter for Alice's tasks, +then drag every item card to the Done column. +``` + +Filtering and navigating work fine through the DOM. Dragging a card onto the board doesn't — `magnitasks.com`'s board uses pointer-based drag-and-drop, so it's a small, honest example of the failure mode this cookbook exists for. + +## Playwright mode vs. CUA mode + +`browser-loop` compiles two tool catalogs for the same browser session: + +| | playwright mode | CUA mode | +| --- | --- | --- | +| tools | `browser_*` (`loop.toolsets.browser()`) | `computer_*` (`loop.toolsets.computer()`) | +| driven by | an accessibility snapshot, resolved by ref | a screenshot, read pixel by pixel | +| a click is | one DOM lookup + one CDP dispatch | a vision call, then an OS-level click at a coordinate — see [Computer Controls](/browsers/computer-controls) | +| cost | cheap, fast, deterministic when the DOM cooperates | a full model round trip per action, but works on anything the page renders regardless of DOM structure | + +Playwright mode should be your default: it's faster and cheaper per action. Reach for CUA mode — Kernel's [computer use](/integrations/computer-use/overview) surface — for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. + +## Picking a harness + +The snippets below use [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness` — currently the only agent harness `browser-loop` binds to. It's the right choice for this pattern specifically because of one method: `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript. The model that picks up in CUA mode still has the entire playwright-mode conversation as context — it knows what it already tried and why it stopped. + +If your agent already runs on a different harness — Claude Agent SDK, Codex, or your own loop — you won't get that live swap; those harnesses don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. + +## Setup + +```bash +npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core tsx +``` + + +`browser-loop` pins an exact `@onkernel/sdk` version as a peer dependency. Installing a newer `@onkernel/sdk` elsewhere in the same project can hoist a mismatched version, which produces two incompatible `Kernel` client types — `attach({ client, browser })` won't type-check. Run `npm ls @onkernel/sdk` if you hit a type error there. + + +Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): + +```bash +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts +``` + +## Three ways to hand off + +All three run playwright mode first and switch to CUA mode once it stops making progress. They differ in what you need to know about the task at integration time: + +| you know | approach | integration cost | +| --- | --- | --- | +| the specific playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | +| the general shape of the task, not the specific risky action | [flat step-count budget](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | +| little to nothing — tasks arrive from an end user at runtime | [mixed toolset + steering prompt](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | + +If you can name the risky tool, do — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. + +### Approach A: trigger-tool budget + +Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. + +```ts +import Kernel from "@onkernel/sdk"; +import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; +import { loop } from "@onkernel/browser-loop"; +import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; + +const TASK_PROMPT = + "Go to magnitasks.com, navigate to Tasks, filter for Alice's tasks, " + + "then drag every item card to the Done column."; + +const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; + +// The playwright-mode tool this task depends on and expects to be unreliable. +// Swap per task — a different task might key on browser_fill for a +// canvas-backed input, or nothing at all if no single tool is the known risk. +const FALLBACK_TRIGGER_TOOLS = new Set(["browser_drag"]); +const FALLBACK_TRIGGER_BUDGET = 2; + +// Backstop: total tool calls playwright mode gets regardless of whether it +// ever reaches the trigger tool, so a wrong selector or navigation failure +// can't run forever. +const ACTION_SAFETY_CAP = 20; + +async function main(): Promise { + const kernelApiKey = process.env.KERNEL_API_KEY; + if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); + requireLoopEnvApiKeyForModel(MODEL); + + const client = new Kernel({ apiKey: kernelApiKey }); + const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ client, browser }); + + try { + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua" }); + + const playwrightPair = kb.compile({ + model: MODEL, + tools: loop.toolsets.browser(), + }); + const harness = new AgentHarness({ + session, + model: playwrightPair.model, + models: playwrightPair.models, + tools: [...playwrightPair.tools], + activeToolNames: playwrightPair.tools.map((tool) => tool.name), + systemPrompt: "Use the supplied browser tools to complete the task efficiently.", + }); + playwrightPair.activate(harness); + + let actionTurns = 0; + let triggerAttempts = 0; + let budgetExhausted = false; + const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { + if (event.type !== "tool_execution_end") return; + actionTurns += 1; + if (FALLBACK_TRIGGER_TOOLS.has(event.toolName)) triggerAttempts += 1; + if (budgetExhausted) return; + if (triggerAttempts >= FALLBACK_TRIGGER_BUDGET) { + budgetExhausted = true; + void harness.abort(); + } else if (actionTurns >= ACTION_SAFETY_CAP) { + budgetExhausted = true; + void harness.abort(); + } + }); + + let final = await harness.prompt(TASK_PROMPT); + unsubscribe(); + + if (final.stopReason === "aborted") { + const cuaPair = kb.compile({ + model: MODEL, + tools: loop.toolsets.computer(), + }); + await cuaPair.apply(harness); + + final = await harness.prompt( + "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + + "and end so the board can see the pointer move and recognize a drag.", + ); + } + + console.log(`final stopReason: ${final.stopReason}`); + for (const block of final.content) { + if (block.type === "text") console.log(block.text); + } + } finally { + await kb.dispose(); + await client.browsers.deleteByID(browser.session_id); + } +} + +void main(); +``` + +### Approach B: flat step-count budget + +Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so playwright mode gets a fixed total before the handoff regardless of what those actions turn out to be. + +Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs CUA. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. + +```ts +import Kernel from "@onkernel/sdk"; +import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; +import { loop } from "@onkernel/browser-loop"; +import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; + +const TASK_PROMPT = + "Go to magnitasks.com, navigate to Tasks, filter for Alice's tasks, " + + "then drag every item card to the Done column."; + +const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; + +// How many tool calls of any kind playwright mode gets before handing off. +// Size it to comfortably cover the task's normal setup plus a couple of +// attempts at whatever turns out to be the hard part. +const PLAYWRIGHT_ACTION_BUDGET = 10; + +async function main(): Promise { + const kernelApiKey = process.env.KERNEL_API_KEY; + if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); + requireLoopEnvApiKeyForModel(MODEL); + + const client = new Kernel({ apiKey: kernelApiKey }); + const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ client, browser }); + + try { + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-stepcount" }); + + const playwrightPair = kb.compile({ + model: MODEL, + tools: loop.toolsets.browser(), + }); + const harness = new AgentHarness({ + session, + model: playwrightPair.model, + models: playwrightPair.models, + tools: [...playwrightPair.tools], + activeToolNames: playwrightPair.tools.map((tool) => tool.name), + systemPrompt: "Use the supplied browser tools to complete the task efficiently.", + }); + playwrightPair.activate(harness); + + let actionTurns = 0; + let budgetExhausted = false; + const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { + if (event.type !== "tool_execution_end") return; + actionTurns += 1; + if (actionTurns >= PLAYWRIGHT_ACTION_BUDGET && !budgetExhausted) { + budgetExhausted = true; + void harness.abort(); + } + }); + + let final = await harness.prompt(TASK_PROMPT); + unsubscribe(); + + if (final.stopReason === "aborted") { + const cuaPair = kb.compile({ + model: MODEL, + tools: loop.toolsets.computer(), + }); + await cuaPair.apply(harness); + + final = await harness.prompt( + "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + + "and end so the board can see the pointer move and recognize a drag.", + ); + } + + console.log(`final stopReason: ${final.stopReason}`); + for (const block of final.content) { + if (block.type === "text") console.log(block.text); + } + } finally { + await kb.dispose(); + await client.browsers.deleteByID(browser.session_id); + } +} + +void main(); +``` + +### Approach C: mixed toolset and steering prompt + +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both playwright-mode and CUA-mode tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. + +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job (`computer_drag`) over reassembling the effect from lower-level primitives (`computer_mouse_down` / `computer_move` / `computer_mouse_up`). + +```ts +import Kernel from "@onkernel/sdk"; +import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; +import { loop } from "@onkernel/browser-loop"; +import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; + +const TASK_PROMPT = + "Go to magnitasks.com, navigate to Tasks, filter for Alice's tasks, " + + "then drag every item card to the Done column."; + +const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; + +async function main(): Promise { + const kernelApiKey = process.env.KERNEL_API_KEY; + if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); + requireLoopEnvApiKeyForModel(MODEL); + + const client = new Kernel({ apiKey: kernelApiKey }); + const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ client, browser }); + + try { + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-mixed" }); + + const compiled = kb.compile({ model: MODEL, tools: loop.toolsets.mixed() }); + const harness = new AgentHarness({ + session, + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), + systemPrompt: + "Use the browser_* tools (ref-based, DOM-driven) as your default for navigation, clicking, " + + "filling forms, and reading page state -- they're faster and cheaper. Only reach for the " + + "computer_* tools (screenshot-driven, pixel-coordinate based) when a browser_* action, " + + "especially browser_drag, doesn't produce the effect you expected after a snapshot or " + + "screenshot check. Don't retry a failing browser_* action more than once before switching " + + "to the computer_* equivalent for that specific step.", + }); + compiled.activate(harness); + + harness.subscribe((event: AgentHarnessEvent) => { + if (event.type !== "tool_execution_end") return; + console.log(`${event.toolName} error=${event.isError}`); + }); + + const final = await harness.prompt(TASK_PROMPT); + console.log(`final stopReason: ${final.stopReason}`); + for (const block of final.content) { + if (block.type === "text") console.log(block.text); + } + } finally { + await kb.dispose(); + await client.browsers.deleteByID(browser.session_id); + } +} + +void main(); +``` + +## Choosing between them + +| approach | needs knowing | integration effort | switching behavior | +| --- | --- | --- | --- | +| A: trigger-tool budget | the exact risky tool | one config line, no tuning | fires precisely once the named tool has had its shot | +| B: flat step-count budget | enough about the task to size a number | a number to find and re-check per task | fires on total action count, blind to which tool it was | +| C: mixed toolset + prompt | nothing in advance | no handoff logic at all | left entirely to the model's own judgment, with no hard bound | + +As a starting rule: reach for **A** first, even if it costs you one exploratory run to find the tool name. Fall back to **B** only when the task genuinely varies enough that naming a trigger tool isn't realistic. Reach for **C** only when you can't write task-specific logic at all — and if you do, consider pairing it with a hard action cap as a backstop against its unbounded soft constraint. + +## Notes + +- **None of the three hand control back.** Once a run switches to CUA mode, playwright mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA mode re-establishes progress. +- **Swap models freely.** All three default to `anthropic:claude-sonnet-5` via `LOOP_MODEL`. Use `listLoopModels()` from `@onkernel/browser-loop/pi` to see everything else the catalog supports for both toolsets. +- **Record the whole handoff on one video.** Wrap the run in [`client.browsers.replays.start()` / `.stop()`](/browsers/replays) so the playwright-mode attempt and the CUA-mode recovery land on the same replay. + +## Next steps + +- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA mode drives +- [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on Kernel more generally +- [Replays](/browsers/replays) — record a session end to end +- [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place diff --git a/docs.json b/docs.json index e62171e..522be28 100644 --- a/docs.json +++ b/docs.json @@ -131,7 +131,8 @@ "browsers/curl", "browsers/ssh", "browsers/computer-controls", - "browsers/playwright-execution" + "browsers/playwright-execution", + "browsers/cua-fallback" ] }, { From ae2154a8d33cc19cd736038deba456e187057beb Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 19:58:54 +0000 Subject: [PATCH 02/13] Use generic language for non-Kernel agent harnesses Avoid naming specific third-party AI products in prose outside a dedicated integration page. --- browsers/cua-fallback.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/browsers/cua-fallback.mdx b/browsers/cua-fallback.mdx index c79e106..d2c3f52 100644 --- a/browsers/cua-fallback.mdx +++ b/browsers/cua-fallback.mdx @@ -37,7 +37,7 @@ Playwright mode should be your default: it's faster and cheaper per action. Reac The snippets below use [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness` — currently the only agent harness `browser-loop` binds to. It's the right choice for this pattern specifically because of one method: `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript. The model that picks up in CUA mode still has the entire playwright-mode conversation as context — it knows what it already tried and why it stopped. -If your agent already runs on a different harness — Claude Agent SDK, Codex, or your own loop — you won't get that live swap; those harnesses don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +If your agent already runs on a different harness, you likely won't get that live swap; most agent SDKs don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. ## Setup From fb68cc0be40b201ba2b332ae90b8e9ea571f2864 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:25:27 +0000 Subject: [PATCH 03/13] Rename cookbook to Code Mode with Computer Use Fallback Renames the file and aligns terminology throughout: "playwright mode" becomes "code mode" and "CUA mode" becomes "computer use", matching the new title. --- ...ua-fallback.mdx => code-mode-fallback.mdx} | 88 +++++++++---------- docs.json | 2 +- 2 files changed, 45 insertions(+), 45 deletions(-) rename browsers/{cua-fallback.mdx => code-mode-fallback.mdx} (75%) diff --git a/browsers/cua-fallback.mdx b/browsers/code-mode-fallback.mdx similarity index 75% rename from browsers/cua-fallback.mdx rename to browsers/code-mode-fallback.mdx index d2c3f52..40d28e5 100644 --- a/browsers/cua-fallback.mdx +++ b/browsers/code-mode-fallback.mdx @@ -1,6 +1,6 @@ --- -title: "Playwright Mode with a CUA Fallback" -description: "Run an agent on fast DOM-ref tools, hand it off to computer use only when the DOM won't cooperate" +title: "Using Code Mode with Computer Use Fallback" +description: "Run an agent in code mode, hand it off to computer use only when the DOM won't cooperate" --- Some page interactions don't survive a trip through the DOM. Pointer-based drag-and-drop is the common one: a library like `dnd-kit` or `SortableJS` needs a real sequence of mousemove events to recognize a drag, and a DOM-ref tool that jumps straight from press to release never crosses that threshold. The tool call still reports success — the card just doesn't move. @@ -20,24 +20,24 @@ then drag every item card to the Done column. Filtering and navigating work fine through the DOM. Dragging a card onto the board doesn't — `magnitasks.com`'s board uses pointer-based drag-and-drop, so it's a small, honest example of the failure mode this cookbook exists for. -## Playwright mode vs. CUA mode +## Code mode vs. computer use `browser-loop` compiles two tool catalogs for the same browser session: -| | playwright mode | CUA mode | +| | code mode | computer use | | --- | --- | --- | | tools | `browser_*` (`loop.toolsets.browser()`) | `computer_*` (`loop.toolsets.computer()`) | | driven by | an accessibility snapshot, resolved by ref | a screenshot, read pixel by pixel | | a click is | one DOM lookup + one CDP dispatch | a vision call, then an OS-level click at a coordinate — see [Computer Controls](/browsers/computer-controls) | | cost | cheap, fast, deterministic when the DOM cooperates | a full model round trip per action, but works on anything the page renders regardless of DOM structure | -Playwright mode should be your default: it's faster and cheaper per action. Reach for CUA mode — Kernel's [computer use](/integrations/computer-use/overview) surface — for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. +Code mode should be your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. ## Picking a harness -The snippets below use [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness` — currently the only agent harness `browser-loop` binds to. It's the right choice for this pattern specifically because of one method: `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript. The model that picks up in CUA mode still has the entire playwright-mode conversation as context — it knows what it already tried and why it stopped. +The snippets below use [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness` — currently the only agent harness `browser-loop` binds to. It's the right choice for this pattern specifically because of one method: `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript. The model that picks up in computer use still has the entire code-mode conversation as context — it knows what it already tried and why it stopped. -If your agent already runs on a different harness, you likely won't get that live swap; most agent SDKs don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +If your agent already runs on a different harness, you likely won't get that live swap; most agent SDKs don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run code mode to completion or failure, then start a fresh computer-use call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. ## Setup @@ -52,16 +52,16 @@ npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core t Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): ```bash -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx code-then-computer-use.ts ``` ## Three ways to hand off -All three run playwright mode first and switch to CUA mode once it stops making progress. They differ in what you need to know about the task at integration time: +All three run code mode first and switch to computer use once it stops making progress. They differ in what you need to know about the task at integration time: | you know | approach | integration cost | | --- | --- | --- | -| the specific playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | +| the specific code-mode tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | | the general shape of the task, not the specific risky action | [flat step-count budget](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | | little to nothing — tasks arrive from an end user at runtime | [mixed toolset + steering prompt](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | @@ -69,7 +69,7 @@ If you can name the risky tool, do — it's the only one of the three that's bot ### Approach A: trigger-tool budget -Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. +Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific code-mode tool this task needs computer use to take over for. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. ```ts import Kernel from "@onkernel/sdk"; @@ -83,13 +83,13 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// The playwright-mode tool this task depends on and expects to be unreliable. +// The code-mode tool this task depends on and expects to be unreliable. // Swap per task — a different task might key on browser_fill for a // canvas-backed input, or nothing at all if no single tool is the known risk. const FALLBACK_TRIGGER_TOOLS = new Set(["browser_drag"]); const FALLBACK_TRIGGER_BUDGET = 2; -// Backstop: total tool calls playwright mode gets regardless of whether it +// Backstop: total tool calls code mode gets regardless of whether it // ever reaches the trigger tool, so a wrong selector or navigation failure // can't run forever. const ACTION_SAFETY_CAP = 20; @@ -104,21 +104,21 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua" }); + const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use" }); - const playwrightPair = kb.compile({ + const codePair = kb.compile({ model: MODEL, tools: loop.toolsets.browser(), }); const harness = new AgentHarness({ session, - model: playwrightPair.model, - models: playwrightPair.models, - tools: [...playwrightPair.tools], - activeToolNames: playwrightPair.tools.map((tool) => tool.name), + model: codePair.model, + models: codePair.models, + tools: [...codePair.tools], + activeToolNames: codePair.tools.map((tool) => tool.name), systemPrompt: "Use the supplied browser tools to complete the task efficiently.", }); - playwrightPair.activate(harness); + codePair.activate(harness); let actionTurns = 0; let triggerAttempts = 0; @@ -141,14 +141,14 @@ async function main(): Promise { unsubscribe(); if (final.stopReason === "aborted") { - const cuaPair = kb.compile({ + const computerUsePair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await cuaPair.apply(harness); + await computerUsePair.apply(harness); final = await harness.prompt( - "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Code-mode drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -171,9 +171,9 @@ void main(); ### Approach B: flat step-count budget -Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so playwright mode gets a fixed total before the handoff regardless of what those actions turn out to be. +Use this when you know the general category of task but not the specific action likely to need computer use, so naming one tool up front isn't realistic. `CODE_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so code mode gets a fixed total before the handoff regardless of what those actions turn out to be. -Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs CUA. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. +Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs computer use. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. ```ts import Kernel from "@onkernel/sdk"; @@ -187,10 +187,10 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// How many tool calls of any kind playwright mode gets before handing off. +// How many tool calls of any kind code mode gets before handing off. // Size it to comfortably cover the task's normal setup plus a couple of // attempts at whatever turns out to be the hard part. -const PLAYWRIGHT_ACTION_BUDGET = 10; +const CODE_ACTION_BUDGET = 10; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; @@ -202,28 +202,28 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-stepcount" }); + const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use-stepcount" }); - const playwrightPair = kb.compile({ + const codePair = kb.compile({ model: MODEL, tools: loop.toolsets.browser(), }); const harness = new AgentHarness({ session, - model: playwrightPair.model, - models: playwrightPair.models, - tools: [...playwrightPair.tools], - activeToolNames: playwrightPair.tools.map((tool) => tool.name), + model: codePair.model, + models: codePair.models, + tools: [...codePair.tools], + activeToolNames: codePair.tools.map((tool) => tool.name), systemPrompt: "Use the supplied browser tools to complete the task efficiently.", }); - playwrightPair.activate(harness); + codePair.activate(harness); let actionTurns = 0; let budgetExhausted = false; const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; actionTurns += 1; - if (actionTurns >= PLAYWRIGHT_ACTION_BUDGET && !budgetExhausted) { + if (actionTurns >= CODE_ACTION_BUDGET && !budgetExhausted) { budgetExhausted = true; void harness.abort(); } @@ -233,14 +233,14 @@ async function main(): Promise { unsubscribe(); if (final.stopReason === "aborted") { - const cuaPair = kb.compile({ + const computerUsePair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await cuaPair.apply(harness); + await computerUsePair.apply(harness); final = await harness.prompt( - "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Code-mode drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -263,9 +263,9 @@ void main(); ### Approach C: mixed toolset and steering prompt -Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both playwright-mode and CUA-mode tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs computer use. `loop.toolsets.mixed()` hands the model both code-mode and computer-use tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. -The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job (`computer_drag`) over reassembling the effect from lower-level primitives (`computer_mouse_down` / `computer_move` / `computer_mouse_up`). +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with code-mode tools before trying computer use, and which computer-use tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job (`computer_drag`) over reassembling the effect from lower-level primitives (`computer_mouse_down` / `computer_move` / `computer_mouse_up`). ```ts import Kernel from "@onkernel/sdk"; @@ -289,7 +289,7 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-mixed" }); + const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use-mixed" }); const compiled = kb.compile({ model: MODEL, tools: loop.toolsets.mixed() }); const harness = new AgentHarness({ @@ -339,13 +339,13 @@ As a starting rule: reach for **A** first, even if it costs you one exploratory ## Notes -- **None of the three hand control back.** Once a run switches to CUA mode, playwright mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA mode re-establishes progress. +- **None of the three hand control back.** Once a run switches to computer use, code mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once computer use re-establishes progress. - **Swap models freely.** All three default to `anthropic:claude-sonnet-5` via `LOOP_MODEL`. Use `listLoopModels()` from `@onkernel/browser-loop/pi` to see everything else the catalog supports for both toolsets. -- **Record the whole handoff on one video.** Wrap the run in [`client.browsers.replays.start()` / `.stop()`](/browsers/replays) so the playwright-mode attempt and the CUA-mode recovery land on the same replay. +- **Record the whole handoff on one video.** Wrap the run in [`client.browsers.replays.start()` / `.stop()`](/browsers/replays) so the code-mode attempt and the computer-use recovery land on the same replay. ## Next steps -- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA mode drives +- [Computer Controls](/browsers/computer-controls) — the OS-level API computer use drives - [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on Kernel more generally - [Replays](/browsers/replays) — record a session end to end - [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place diff --git a/docs.json b/docs.json index 522be28..00c7c05 100644 --- a/docs.json +++ b/docs.json @@ -132,7 +132,7 @@ "browsers/ssh", "browsers/computer-controls", "browsers/playwright-execution", - "browsers/cua-fallback" + "browsers/code-mode-fallback" ] }, { From 8e9d59f63a58860131251600b22a676832b2f2e2 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:43:44 +0000 Subject: [PATCH 04/13] Tone down harness section to explanation, not endorsement State plainly that this cookbook uses browser-loop and why its apply() swap enables the mid-session handoff, and note the same approach applies to any harness or toolset driving the browser. --- browsers/code-mode-fallback.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/browsers/code-mode-fallback.mdx b/browsers/code-mode-fallback.mdx index 40d28e5..80fe126 100644 --- a/browsers/code-mode-fallback.mdx +++ b/browsers/code-mode-fallback.mdx @@ -35,9 +35,9 @@ Code mode should be your default: it's faster and cheaper per action. Reach for ## Picking a harness -The snippets below use [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness` — currently the only agent harness `browser-loop` binds to. It's the right choice for this pattern specifically because of one method: `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript. The model that picks up in computer use still has the entire code-mode conversation as context — it knows what it already tried and why it stopped. +The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up in computer use still has the entire code-mode conversation as context, and knows what it already tried. -If your agent already runs on a different harness, you likely won't get that live swap; most agent SDKs don't expose an equivalent mid-session tool-catalog change. Implement the handoff as two sequential calls instead: run code mode to completion or failure, then start a fresh computer-use call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run code mode to completion or failure, then start a fresh computer-use call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. ## Setup From 37cb336ddd0c19fee3c28f90640617d93871fc89 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:52:44 +0000 Subject: [PATCH 05/13] Restore original browser-loop code snippets, keep code mode as general framing Introduces code mode / computer use as the general concept, then establishes playwright mode / CUA mode as browser-loop's specific terms and uses them for the rest of the doc, matching the restored code. Code snippets are restored close to their original source, with internal-run-specific claims (exact tool-call counts, "verified directly" framing) generalized. --- browsers/code-mode-fallback.mdx | 258 +++++++++++++++++++++++++------- 1 file changed, 207 insertions(+), 51 deletions(-) diff --git a/browsers/code-mode-fallback.mdx b/browsers/code-mode-fallback.mdx index 80fe126..990a027 100644 --- a/browsers/code-mode-fallback.mdx +++ b/browsers/code-mode-fallback.mdx @@ -22,22 +22,22 @@ Filtering and navigating work fine through the DOM. Dragging a card onto the boa ## Code mode vs. computer use -`browser-loop` compiles two tool catalogs for the same browser session: +`browser-loop` compiles two tool catalogs for the same browser session. In general, an agent should run in **code mode** by default — resolving elements in the DOM and acting on them by reference — and fall back to **computer use** — driving the browser from screenshots, the way a person would — only for the interactions code mode can't reliably drive. `browser-loop` specifically calls these **playwright mode** (`loop.toolsets.browser()`) and **CUA mode** (`loop.toolsets.computer()`); the rest of this cookbook uses those two names, since that's what the code below is switching between. -| | code mode | computer use | +| | playwright mode | CUA mode | | --- | --- | --- | -| tools | `browser_*` (`loop.toolsets.browser()`) | `computer_*` (`loop.toolsets.computer()`) | +| tools | `browser_*` | `computer_*` | | driven by | an accessibility snapshot, resolved by ref | a screenshot, read pixel by pixel | | a click is | one DOM lookup + one CDP dispatch | a vision call, then an OS-level click at a coordinate — see [Computer Controls](/browsers/computer-controls) | | cost | cheap, fast, deterministic when the DOM cooperates | a full model round trip per action, but works on anything the page renders regardless of DOM structure | -Code mode should be your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. +Playwright mode should be your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. ## Picking a harness -The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up in computer use still has the entire code-mode conversation as context, and knows what it already tried. +The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up in CUA mode still has the entire playwright-mode conversation as context, and knows what it already tried. -The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run code mode to completion or failure, then start a fresh computer-use call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. ## Setup @@ -52,16 +52,18 @@ npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core t Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): ```bash -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx code-then-computer-use.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts ``` ## Three ways to hand off -All three run code mode first and switch to computer use once it stops making progress. They differ in what you need to know about the task at integration time: +All three run playwright mode first and switch to CUA mode once it stops making progress. They differ in what you need to know about the task at integration time: | you know | approach | integration cost | | --- | --- | --- | -| the specific code-mode tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | +| the specific playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | | the general shape of the task, not the specific risky action | [flat step-count budget](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | | little to nothing — tasks arrive from an end user at runtime | [mixed toolset + steering prompt](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | @@ -69,9 +71,59 @@ If you can name the risky tool, do — it's the only one of the three that's bot ### Approach A: trigger-tool budget -Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific code-mode tool this task needs computer use to take over for. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. +Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. ```ts +/** + * Cookbook: playwright mode first, fall back to CUA mode after the playwright + * tool(s) this task depends on have had their shot and haven't finished the + * job. + * + * `browser_drag` (the atomic tool @onkernel/browser-loop compiles for + * "playwright execution mode") drags by resolving two DOM/viewport points and + * dispatching exactly one CDP `mousePressed` -> `mouseMoved` -> `mouseReleased` + * sequence between them. Kanban boards built on pointer-based drag-and-drop + * (dnd-kit, SortableJS, most React DnD setups) need several intermediate + * `mousemove` events to cross their drag-activation threshold, so a single + * jump often never registers as a drag at all -- the card just gets a click. + * The tool call itself still reports success: there's no error to catch, only + * a card that never moved. + * + * `computer_drag` (CUA/computer-use mode) takes a multi-point pixel path and + * dispatches a real mousemove sequence along it, which is enough motion for + * those same libraries to pick up the gesture. + * + * This script runs a task in playwright mode and hands off to CUA mode once + * playwright mode has spent FALLBACK_TRIGGER_BUDGET attempts on + * FALLBACK_TRIGGER_TOOLS and the task still isn't done. The trigger keys on + * that specific tool rather than a generic action count, so setup steps like + * navigation and filtering never eat into the budget before the action that + * actually needs CUA gets a turn. FALLBACK_TRIGGER_TOOLS is `browser_drag` + * here because drag-and-drop is this task's known-unreliable action; adapt it + * to whatever playwright tool your own task expects to need CUA's help for + * (a file upload, a canvas interaction, anything DOM-ref execution doesn't + * reliably drive). An action safety cap is a backstop for pages that never + * reach a trigger tool at all. + * + * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag + * cards between columns, filter by assignee). Its board uses pointer-based + * drag-and-drop, so `browser_drag` reliably fails to move cards while + * `computer_drag` succeeds -- a good, honest example of the failure mode this + * fallback exists for. + * + * The session is recorded end to end with Kernel's replay API, so both modes + * -- the failed playwright-mode drags and the cua-mode recovery -- are on one + * video. The replay view URL prints as soon as recording starts and again + * once it's stopped and finished processing. + * + * Usage: + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts + * + * Env: + * KERNEL_API_KEY required, Kernel browser API key + * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 + * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) + */ import Kernel from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; @@ -83,15 +135,19 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// The code-mode tool this task depends on and expects to be unreliable. -// Swap per task — a different task might key on browser_fill for a -// canvas-backed input, or nothing at all if no single tool is the known risk. +// Primary trigger: the playwright-mode tool(s) this task depends on and known +// to be unreliable. Swap this per task -- browser_drag here because +// drag-and-drop is what this task needs CUA's help with; a different task +// might key on browser_fill for a canvas-backed input, or nothing at all if +// no single tool is the known risk. These tools report no execution error +// even when they don't produce the intended effect, so this is a count of +// attempts -- an attempt that doesn't finish the task is itself the signal. const FALLBACK_TRIGGER_TOOLS = new Set(["browser_drag"]); const FALLBACK_TRIGGER_BUDGET = 2; -// Backstop: total tool calls code mode gets regardless of whether it -// ever reaches the trigger tool, so a wrong selector or navigation failure -// can't run forever. +// Backstop: total tool calls playwright mode gets regardless of whether it +// ever reaches a drag, so a page that never triggers one (wrong selector, +// navigation failure) can't run forever. const ACTION_SAFETY_CAP = 20; async function main(): Promise { @@ -101,24 +157,26 @@ async function main(): Promise { const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); + const replay = await client.browsers.replays.start(browser.session_id); + console.log(`recording started: ${replay.replay_view_url}`); const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua" }); - const codePair = kb.compile({ + const playwrightPair = kb.compile({ model: MODEL, tools: loop.toolsets.browser(), }); const harness = new AgentHarness({ session, - model: codePair.model, - models: codePair.models, - tools: [...codePair.tools], - activeToolNames: codePair.tools.map((tool) => tool.name), + model: playwrightPair.model, + models: playwrightPair.models, + tools: [...playwrightPair.tools], + activeToolNames: playwrightPair.tools.map((tool) => tool.name), systemPrompt: "Use the supplied browser tools to complete the task efficiently.", }); - codePair.activate(harness); + playwrightPair.activate(harness); let actionTurns = 0; let triggerAttempts = 0; @@ -126,29 +184,36 @@ async function main(): Promise { const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; actionTurns += 1; - if (FALLBACK_TRIGGER_TOOLS.has(event.toolName)) triggerAttempts += 1; + const isTrigger = FALLBACK_TRIGGER_TOOLS.has(event.toolName); + if (isTrigger) triggerAttempts += 1; + console.log(`[playwright ${actionTurns}] ${event.toolName} error=${event.isError}${isTrigger ? ` (trigger attempt ${triggerAttempts}/${FALLBACK_TRIGGER_BUDGET})` : ""}`); if (budgetExhausted) return; if (triggerAttempts >= FALLBACK_TRIGGER_BUDGET) { budgetExhausted = true; + console.log("[playwright] trigger tool budget exhausted, aborting run to switch to CUA mode"); void harness.abort(); } else if (actionTurns >= ACTION_SAFETY_CAP) { budgetExhausted = true; + console.log("[playwright] action safety cap reached without a trigger tool attempt, aborting run to switch to CUA mode"); void harness.abort(); } }); + console.log(`model=${MODEL} mode=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); let final = await harness.prompt(TASK_PROMPT); unsubscribe(); if (final.stopReason === "aborted") { - const computerUsePair = kb.compile({ + console.log("[cua] playwright mode did not finish in budget, switching to computer-use tools"); + const cuaPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await computerUsePair.apply(harness); + await cuaPair.apply(harness); + console.log(`model=${MODEL} mode=cua`); final = await harness.prompt( - "Code-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -162,6 +227,8 @@ async function main(): Promise { } } finally { await kb.dispose(); + await client.browsers.replays.stop(replay.replay_id, { id: browser.session_id }); + console.log(`replay ready: ${replay.replay_view_url}`); await client.browsers.deleteByID(browser.session_id); } } @@ -171,11 +238,44 @@ void main(); ### Approach B: flat step-count budget -Use this when you know the general category of task but not the specific action likely to need computer use, so naming one tool up front isn't realistic. `CODE_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so code mode gets a fixed total before the handoff regardless of what those actions turn out to be. +Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so playwright mode gets a fixed total before the handoff regardless of what those actions turn out to be. -Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs computer use. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. +Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs CUA. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. ```ts +/** + * Cookbook variant B: playwright mode first, fall back to CUA mode after a + * flat step-count budget -- for when you know the general shape of the task + * (browser automation against an unfamiliar app) without knowing which + * specific playwright tool is going to be the unreliable one. Use this over + * the trigger-tool variant (playwright-then-cua.ts) when you can't name the + * risky action in advance; use it over the mixed-toolset variant + * (playwright-then-cua-mixed.ts) when you want a deterministic, bounded + * playwright-mode budget instead of trusting a system prompt to self-limit. + * + * The budget counts every completed tool call, so it has to be generous + * enough to survive whatever setup (navigation, filtering, form-filling) the + * task legitimately needs before it reaches the action that actually needs + * CUA. Too tight and it burns out on setup before ever attempting the risky + * action. Picking the right number takes running the task and looking; that + * imprecision is the tradeoff for skipping the step of naming which tool + * will fail. + * + * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag + * cards between columns, filter by assignee). See playwright-then-cua.ts for + * why browser_drag reliably fails here and computer_drag reliably succeeds. + * + * The session is recorded end to end with Kernel's replay API, so both modes + * land on one video. + * + * Usage: + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts + * + * Env: + * KERNEL_API_KEY required, Kernel browser API key + * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 + * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) + */ import Kernel from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; @@ -187,10 +287,13 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// How many tool calls of any kind code mode gets before handing off. -// Size it to comfortably cover the task's normal setup plus a couple of -// attempts at whatever turns out to be the hard part. -const CODE_ACTION_BUDGET = 10; +// How many tool calls of any kind playwright mode gets before we hand off to +// CUA mode. Size this to comfortably cover the task's normal setup steps plus +// a couple of attempts at whatever turns out to be the hard part -- too tight +// and it can exhaust the budget before ever reaching the action that needs +// CUA. There's no per-tool signal here, so this is deliberately generous +// compared to the trigger-tool variant's budget of 1-2. +const PLAYWRIGHT_ACTION_BUDGET = 10; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; @@ -199,48 +302,55 @@ async function main(): Promise { const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); + const replay = await client.browsers.replays.start(browser.session_id); + console.log(`recording started: ${replay.replay_view_url}`); const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use-stepcount" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-stepcount" }); - const codePair = kb.compile({ + const playwrightPair = kb.compile({ model: MODEL, tools: loop.toolsets.browser(), }); const harness = new AgentHarness({ session, - model: codePair.model, - models: codePair.models, - tools: [...codePair.tools], - activeToolNames: codePair.tools.map((tool) => tool.name), + model: playwrightPair.model, + models: playwrightPair.models, + tools: [...playwrightPair.tools], + activeToolNames: playwrightPair.tools.map((tool) => tool.name), systemPrompt: "Use the supplied browser tools to complete the task efficiently.", }); - codePair.activate(harness); + playwrightPair.activate(harness); let actionTurns = 0; let budgetExhausted = false; const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; actionTurns += 1; - if (actionTurns >= CODE_ACTION_BUDGET && !budgetExhausted) { + console.log(`[playwright ${actionTurns}/${PLAYWRIGHT_ACTION_BUDGET}] ${event.toolName} error=${event.isError}`); + if (actionTurns >= PLAYWRIGHT_ACTION_BUDGET && !budgetExhausted) { budgetExhausted = true; + console.log("[playwright] action budget exhausted, aborting run to switch to CUA mode"); void harness.abort(); } }); + console.log(`model=${MODEL} mode=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); let final = await harness.prompt(TASK_PROMPT); unsubscribe(); if (final.stopReason === "aborted") { - const computerUsePair = kb.compile({ + console.log("[cua] playwright mode did not finish in budget, switching to computer-use tools"); + const cuaPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await computerUsePair.apply(harness); + await cuaPair.apply(harness); + console.log(`model=${MODEL} mode=cua`); final = await harness.prompt( - "Code-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -254,6 +364,8 @@ async function main(): Promise { } } finally { await kb.dispose(); + await client.browsers.replays.stop(replay.replay_id, { id: browser.session_id }); + console.log(`replay ready: ${replay.replay_view_url}`); await client.browsers.deleteByID(browser.session_id); } } @@ -263,11 +375,48 @@ void main(); ### Approach C: mixed toolset and steering prompt -Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs computer use. `loop.toolsets.mixed()` hands the model both code-mode and computer-use tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both playwright-mode and CUA-mode tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. -The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with code-mode tools before trying computer use, and which computer-use tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job (`computer_drag`) over reassembling the effect from lower-level primitives (`computer_mouse_down` / `computer_move` / `computer_mouse_up`). +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. ```ts +/** + * Cookbook variant C: give the model both playwright-mode and CUA-mode tools + * up front (loop.toolsets.mixed()) and let it choose per action, steered by a + * system prompt toward playwright tools by default. Use this over the two + * scripted-handoff variants (playwright-then-cua.ts, + * playwright-then-cua-stepcount.ts) when you don't want to write any handoff + * logic at all -- appropriate for a general-purpose agent whose tasks aren't + * known ahead of time, where you can't pre-decide a trigger tool or a + * reasonable step budget. + * + * The tradeoff: with both toolsets available, a model can try browser_* + * tools first, take a screenshot, notice a drag hasn't landed, and switch to + * computer_* tools on its own -- but it may never call the purpose-built + * computer_drag tool. It can reassemble a drag by hand from + * computer_mouse_down / computer_move / computer_mouse_up primitives + * instead, taking far more tool calls -- and therefore model round trips -- + * for the same result. A system prompt is a soft constraint on both when it + * switches and which tool it reaches for once it does; naming computer_drag + * explicitly in a handoff message (as the scripted variants do) is what + * keeps CUA mode fast once it's in control. Nothing here enforces that a + * real handoff even happens -- there's no hard bound on how long the model + * sticks with browser_* tools before trying computer_* ones. + * + * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag + * cards between columns, filter by assignee). See playwright-then-cua.ts for + * why browser_drag reliably fails here and computer_drag reliably succeeds. + * + * The session is recorded end to end with Kernel's replay API. + * + * Usage: + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts + * + * Env: + * KERNEL_API_KEY required, Kernel browser API key + * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 + * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) + */ import Kernel from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; @@ -286,10 +435,12 @@ async function main(): Promise { const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); + const replay = await client.browsers.replays.start(browser.session_id); + console.log(`recording started: ${replay.replay_view_url}`); const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "code-then-computer-use-mixed" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-mixed" }); const compiled = kb.compile({ model: MODEL, tools: loop.toolsets.mixed() }); const harness = new AgentHarness({ @@ -308,11 +459,14 @@ async function main(): Promise { }); compiled.activate(harness); + let actionTurns = 0; harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; - console.log(`${event.toolName} error=${event.isError}`); + actionTurns += 1; + console.log(`[${actionTurns}] ${event.toolName} error=${event.isError}`); }); + console.log(`model=${MODEL} mode=mixed prompt=${JSON.stringify(TASK_PROMPT)}`); const final = await harness.prompt(TASK_PROMPT); console.log(`final stopReason: ${final.stopReason}`); for (const block of final.content) { @@ -320,6 +474,8 @@ async function main(): Promise { } } finally { await kb.dispose(); + await client.browsers.replays.stop(replay.replay_id, { id: browser.session_id }); + console.log(`replay ready: ${replay.replay_view_url}`); await client.browsers.deleteByID(browser.session_id); } } @@ -339,13 +495,13 @@ As a starting rule: reach for **A** first, even if it costs you one exploratory ## Notes -- **None of the three hand control back.** Once a run switches to computer use, code mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once computer use re-establishes progress. +- **None of the three hand control back.** Once a run switches to CUA mode, playwright mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA mode re-establishes progress. - **Swap models freely.** All three default to `anthropic:claude-sonnet-5` via `LOOP_MODEL`. Use `listLoopModels()` from `@onkernel/browser-loop/pi` to see everything else the catalog supports for both toolsets. -- **Record the whole handoff on one video.** Wrap the run in [`client.browsers.replays.start()` / `.stop()`](/browsers/replays) so the code-mode attempt and the computer-use recovery land on the same replay. +- **The whole handoff is recorded on one video.** Each script wraps the run in `client.browsers.replays.start()` / `.stop()`, so the playwright-mode attempt and the CUA-mode recovery land on the same replay. The replay URL prints as soon as recording starts and again once it's stopped and finished processing. ## Next steps -- [Computer Controls](/browsers/computer-controls) — the OS-level API computer use drives +- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA mode drives - [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on Kernel more generally - [Replays](/browsers/replays) — record a session end to end - [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place From 608db5cd0c77445aa16e6712b20a56bfc4de079c Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:57:20 +0000 Subject: [PATCH 06/13] Reframe code mode as the general approach browser-loop implements Playwright mode is browser-loop's specific implementation of the general code-mode approach, not a synonym introduced alongside it. --- browsers/code-mode-fallback.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/browsers/code-mode-fallback.mdx b/browsers/code-mode-fallback.mdx index 990a027..5d58414 100644 --- a/browsers/code-mode-fallback.mdx +++ b/browsers/code-mode-fallback.mdx @@ -22,7 +22,7 @@ Filtering and navigating work fine through the DOM. Dragging a card onto the boa ## Code mode vs. computer use -`browser-loop` compiles two tool catalogs for the same browser session. In general, an agent should run in **code mode** by default — resolving elements in the DOM and acting on them by reference — and fall back to **computer use** — driving the browser from screenshots, the way a person would — only for the interactions code mode can't reliably drive. `browser-loop` specifically calls these **playwright mode** (`loop.toolsets.browser()`) and **CUA mode** (`loop.toolsets.computer()`); the rest of this cookbook uses those two names, since that's what the code below is switching between. +`browser-loop` compiles two tool catalogs for the same browser session. **Code mode** is the general approach: an agent resolves elements in the DOM and acts on them by reference, rather than driving the browser from pixels the way a person would. `browser-loop` implements code mode by making **playwright mode** available (`loop.toolsets.browser()`); its computer-use counterpart is **CUA mode** (`loop.toolsets.computer()`). The rest of this cookbook uses those two specific names, since that's what the code below is switching between. | | playwright mode | CUA mode | | --- | --- | --- | From 3ed9482950739b875cf088ae6ffddfb67362d2d0 Mon Sep 17 00:00:00 2001 From: AnnaXWang <6621137+AnnaXWang@users.noreply.github.com> Date: Thu, 3 Sep 2026 00:02:17 +0000 Subject: [PATCH 07/13] Rename fallback cookbook for Playwright --- ...llback.mdx => playwright-cua-fallback.mdx} | 110 +++++++++--------- docs.json | 2 +- 2 files changed, 57 insertions(+), 55 deletions(-) rename browsers/{code-mode-fallback.mdx => playwright-cua-fallback.mdx} (77%) diff --git a/browsers/code-mode-fallback.mdx b/browsers/playwright-cua-fallback.mdx similarity index 77% rename from browsers/code-mode-fallback.mdx rename to browsers/playwright-cua-fallback.mdx index 5d58414..cd2616e 100644 --- a/browsers/code-mode-fallback.mdx +++ b/browsers/playwright-cua-fallback.mdx @@ -1,13 +1,17 @@ --- -title: "Using Code Mode with Computer Use Fallback" -description: "Run an agent in code mode, hand it off to computer use only when the DOM won't cooperate" +title: "Using Playwright with CUA Fallback" +description: "Start with fast, DOM-based Playwright tools, then fall back to screenshot-based CUA tools for interactions they cannot complete" --- -Some page interactions don't survive a trip through the DOM. Pointer-based drag-and-drop is the common one: a library like `dnd-kit` or `SortableJS` needs a real sequence of mousemove events to recognize a drag, and a DOM-ref tool that jumps straight from press to release never crosses that threshold. The tool call still reports success — the card just doesn't move. +[Playwright](https://github.com/microsoft/playwright) is Microsoft's open-source browser automation framework, first released in 2020. It controls Chromium, Firefox, and WebKit through one API, with resilient element locators, automatic waiting, isolated browser contexts, and detailed tracing when something goes wrong. -The fix isn't to run every task through the slower, pixel-based path. It's to run fast by default and fall back to computer use only for the step that actually needs it, without losing the agent's place in the task when it does. +Higher-level browser-agent tools often wrap Playwright in model-friendly abstractions. Those tools can be useful, but Playwright remains a strong starting point: it is widely adopted, well documented, and gives developers direct, inspectable control over the browser. For agents, DOM-based Playwright actions are usually faster, cheaper, and more predictable than reasoning from screenshots and clicking screen coordinates. -This cookbook covers three ways to wire that handoff with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), Kernel's tool package for browser agents, plus the code for each. +Some page interactions still cannot be completed reliably through simplified DOM-based tools. Pointer-driven drag-and-drop is a common example. Libraries such as `dnd-kit` and `SortableJS` may require several intermediate `mousemove` events before they recognize a drag. A tool that jumps directly from pressing to releasing can report success even though the page never recognized the gesture and the card did not move. + +The answer is not to run the entire task through slower, screenshot-based computer use. Start with Playwright for navigation, reading, clicking, and form filling. Switch to CUA only for the interaction that needs realistic pointer movement, then continue without losing the browser's open tabs, page state, or the agent's conversation context. + +This cookbook demonstrates three ways to make that handoff with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), Kernel's tool package for browser agents, with complete code for each approach. ## The example task @@ -20,24 +24,24 @@ then drag every item card to the Done column. Filtering and navigating work fine through the DOM. Dragging a card onto the board doesn't — `magnitasks.com`'s board uses pointer-based drag-and-drop, so it's a small, honest example of the failure mode this cookbook exists for. -## Code mode vs. computer use +## Playwright vs. CUA -`browser-loop` compiles two tool catalogs for the same browser session. **Code mode** is the general approach: an agent resolves elements in the DOM and acts on them by reference, rather than driving the browser from pixels the way a person would. `browser-loop` implements code mode by making **playwright mode** available (`loop.toolsets.browser()`); its computer-use counterpart is **CUA mode** (`loop.toolsets.computer()`). The rest of this cookbook uses those two specific names, since that's what the code below is switching between. +`browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **CUA tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. -| | playwright mode | CUA mode | +| | Playwright | CUA | | --- | --- | --- | | tools | `browser_*` | `computer_*` | | driven by | an accessibility snapshot, resolved by ref | a screenshot, read pixel by pixel | | a click is | one DOM lookup + one CDP dispatch | a vision call, then an OS-level click at a coordinate — see [Computer Controls](/browsers/computer-controls) | | cost | cheap, fast, deterministic when the DOM cooperates | a full model round trip per action, but works on anything the page renders regardless of DOM structure | -Playwright mode should be your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. +Make Playwright your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. ## Picking a harness -The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up in CUA mode still has the entire playwright-mode conversation as context, and knows what it already tried. +The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up with CUA still has the entire Playwright conversation as context, and knows what it already tried. -The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run playwright mode to completion or failure, then start a fresh CUA-mode call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh CUA call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. ## Setup @@ -59,11 +63,11 @@ KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts ## Three ways to hand off -All three run playwright mode first and switch to CUA mode once it stops making progress. They differ in what you need to know about the task at integration time: +All three run Playwright first and switch to CUA once it stops making progress. They differ in what you need to know about the task at integration time: | you know | approach | integration cost | | --- | --- | --- | -| the specific playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | +| the specific Playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | | the general shape of the task, not the specific risky action | [flat step-count budget](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | | little to nothing — tasks arrive from an end user at runtime | [mixed toolset + steering prompt](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | @@ -71,16 +75,16 @@ If you can name the risky tool, do — it's the only one of the three that's bot ### Approach A: trigger-tool budget -Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. +Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific Playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. ```ts /** - * Cookbook: playwright mode first, fall back to CUA mode after the playwright + * Cookbook: Playwright first, fall back to CUA after the Playwright * tool(s) this task depends on have had their shot and haven't finished the * job. * * `browser_drag` (the atomic tool @onkernel/browser-loop compiles for - * "playwright execution mode") drags by resolving two DOM/viewport points and + * the Playwright toolset) drags by resolving two DOM/viewport points and * dispatching exactly one CDP `mousePressed` -> `mouseMoved` -> `mouseReleased` * sequence between them. Kanban boards built on pointer-based drag-and-drop * (dnd-kit, SortableJS, most React DnD setups) need several intermediate @@ -89,18 +93,18 @@ Use this when you're integrating against a known, fixed target and you already k * The tool call itself still reports success: there's no error to catch, only * a card that never moved. * - * `computer_drag` (CUA/computer-use mode) takes a multi-point pixel path and + * `computer_drag` (from the CUA toolset) takes a multi-point pixel path and * dispatches a real mousemove sequence along it, which is enough motion for * those same libraries to pick up the gesture. * - * This script runs a task in playwright mode and hands off to CUA mode once - * playwright mode has spent FALLBACK_TRIGGER_BUDGET attempts on + * This script runs a task with Playwright and hands off to CUA once + * Playwright has spent FALLBACK_TRIGGER_BUDGET attempts on * FALLBACK_TRIGGER_TOOLS and the task still isn't done. The trigger keys on * that specific tool rather than a generic action count, so setup steps like * navigation and filtering never eat into the budget before the action that * actually needs CUA gets a turn. FALLBACK_TRIGGER_TOOLS is `browser_drag` * here because drag-and-drop is this task's known-unreliable action; adapt it - * to whatever playwright tool your own task expects to need CUA's help for + * to whatever Playwright tool your own task expects to need CUA's help for * (a file upload, a canvas interaction, anything DOM-ref execution doesn't * reliably drive). An action safety cap is a backstop for pages that never * reach a trigger tool at all. @@ -111,10 +115,9 @@ Use this when you're integrating against a known, fixed target and you already k * `computer_drag` succeeds -- a good, honest example of the failure mode this * fallback exists for. * - * The session is recorded end to end with Kernel's replay API, so both modes - * -- the failed playwright-mode drags and the cua-mode recovery -- are on one - * video. The replay view URL prints as soon as recording starts and again - * once it's stopped and finished processing. + * Kernel's replay API records the failed Playwright drags and the CUA recovery + * on one video. The replay view URL prints as soon as recording starts and + * again once it's stopped and finished processing. * * Usage: * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts @@ -135,7 +138,7 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// Primary trigger: the playwright-mode tool(s) this task depends on and known +// Primary trigger: the Playwright tool(s) this task depends on and known // to be unreliable. Swap this per task -- browser_drag here because // drag-and-drop is what this task needs CUA's help with; a different task // might key on browser_fill for a canvas-backed input, or nothing at all if @@ -145,7 +148,7 @@ const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic const FALLBACK_TRIGGER_TOOLS = new Set(["browser_drag"]); const FALLBACK_TRIGGER_BUDGET = 2; -// Backstop: total tool calls playwright mode gets regardless of whether it +// Backstop: total tool calls Playwright gets regardless of whether it // ever reaches a drag, so a page that never triggers one (wrong selector, // navigation failure) can't run forever. const ACTION_SAFETY_CAP = 20; @@ -190,30 +193,30 @@ async function main(): Promise { if (budgetExhausted) return; if (triggerAttempts >= FALLBACK_TRIGGER_BUDGET) { budgetExhausted = true; - console.log("[playwright] trigger tool budget exhausted, aborting run to switch to CUA mode"); + console.log("[playwright] trigger tool budget exhausted, aborting run to switch to CUA"); void harness.abort(); } else if (actionTurns >= ACTION_SAFETY_CAP) { budgetExhausted = true; - console.log("[playwright] action safety cap reached without a trigger tool attempt, aborting run to switch to CUA mode"); + console.log("[playwright] action safety cap reached without a trigger tool attempt, aborting run to switch to CUA"); void harness.abort(); } }); - console.log(`model=${MODEL} mode=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); + console.log(`model=${MODEL} toolset=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); let final = await harness.prompt(TASK_PROMPT); unsubscribe(); if (final.stopReason === "aborted") { - console.log("[cua] playwright mode did not finish in budget, switching to computer-use tools"); + console.log("[cua] Playwright did not finish in budget, switching to CUA tools"); const cuaPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); await cuaPair.apply(harness); - console.log(`model=${MODEL} mode=cua`); + console.log(`model=${MODEL} toolset=cua`); final = await harness.prompt( - "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Playwright drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -238,20 +241,20 @@ void main(); ### Approach B: flat step-count budget -Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so playwright mode gets a fixed total before the handoff regardless of what those actions turn out to be. +Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so Playwright gets a fixed total before the handoff regardless of what those actions turn out to be. Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs CUA. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. ```ts /** - * Cookbook variant B: playwright mode first, fall back to CUA mode after a + * Cookbook variant B: Playwright first, fall back to CUA after a * flat step-count budget -- for when you know the general shape of the task * (browser automation against an unfamiliar app) without knowing which - * specific playwright tool is going to be the unreliable one. Use this over + * specific Playwright tool is going to be the unreliable one. Use this over * the trigger-tool variant (playwright-then-cua.ts) when you can't name the * risky action in advance; use it over the mixed-toolset variant * (playwright-then-cua-mixed.ts) when you want a deterministic, bounded - * playwright-mode budget instead of trusting a system prompt to self-limit. + * Playwright budget instead of trusting a system prompt to self-limit. * * The budget counts every completed tool call, so it has to be generous * enough to survive whatever setup (navigation, filtering, form-filling) the @@ -265,8 +268,7 @@ Size the budget generously enough to survive the task's legitimate setup — nav * cards between columns, filter by assignee). See playwright-then-cua.ts for * why browser_drag reliably fails here and computer_drag reliably succeeds. * - * The session is recorded end to end with Kernel's replay API, so both modes - * land on one video. + * Kernel's replay API records both toolsets on one video. * * Usage: * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts @@ -287,8 +289,8 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// How many tool calls of any kind playwright mode gets before we hand off to -// CUA mode. Size this to comfortably cover the task's normal setup steps plus +// How many tool calls of any kind Playwright gets before we hand off to +// CUA. Size this to comfortably cover the task's normal setup steps plus // a couple of attempts at whatever turns out to be the hard part -- too tight // and it can exhaust the budget before ever reaching the action that needs // CUA. There's no per-tool signal here, so this is deliberately generous @@ -331,26 +333,26 @@ async function main(): Promise { console.log(`[playwright ${actionTurns}/${PLAYWRIGHT_ACTION_BUDGET}] ${event.toolName} error=${event.isError}`); if (actionTurns >= PLAYWRIGHT_ACTION_BUDGET && !budgetExhausted) { budgetExhausted = true; - console.log("[playwright] action budget exhausted, aborting run to switch to CUA mode"); + console.log("[playwright] action budget exhausted, aborting run to switch to CUA"); void harness.abort(); } }); - console.log(`model=${MODEL} mode=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); + console.log(`model=${MODEL} toolset=playwright prompt=${JSON.stringify(TASK_PROMPT)}`); let final = await harness.prompt(TASK_PROMPT); unsubscribe(); if (final.stopReason === "aborted") { - console.log("[cua] playwright mode did not finish in budget, switching to computer-use tools"); + console.log("[cua] Playwright did not finish in budget, switching to CUA tools"); const cuaPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); await cuaPair.apply(harness); - console.log(`model=${MODEL} mode=cua`); + console.log(`model=${MODEL} toolset=cua`); final = await harness.prompt( - "Playwright-mode drag actions were not registering on this page's drag-and-drop board -- " + + "Playwright drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + "task using the computer-use tools: click and drag by pixel coordinates instead of DOM " + "refs. For computer_drag, use a path with several intermediate waypoints between the start " + @@ -375,15 +377,15 @@ void main(); ### Approach C: mixed toolset and steering prompt -Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both playwright-mode and CUA-mode tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both Playwright and CUA tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. -The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with Playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. ```ts /** - * Cookbook variant C: give the model both playwright-mode and CUA-mode tools + * Cookbook variant C: give the model both Playwright and CUA tools * up front (loop.toolsets.mixed()) and let it choose per action, steered by a - * system prompt toward playwright tools by default. Use this over the two + * system prompt toward Playwright tools by default. Use this over the two * scripted-handoff variants (playwright-then-cua.ts, * playwright-then-cua-stepcount.ts) when you don't want to write any handoff * logic at all -- appropriate for a general-purpose agent whose tasks aren't @@ -399,7 +401,7 @@ The tradeoff is that a system prompt is a soft constraint on both axes that matt * for the same result. A system prompt is a soft constraint on both when it * switches and which tool it reaches for once it does; naming computer_drag * explicitly in a handoff message (as the scripted variants do) is what - * keeps CUA mode fast once it's in control. Nothing here enforces that a + * keeps CUA fast once it's in control. Nothing here enforces that a * real handoff even happens -- there's no hard bound on how long the model * sticks with browser_* tools before trying computer_* ones. * @@ -466,7 +468,7 @@ async function main(): Promise { console.log(`[${actionTurns}] ${event.toolName} error=${event.isError}`); }); - console.log(`model=${MODEL} mode=mixed prompt=${JSON.stringify(TASK_PROMPT)}`); + console.log(`model=${MODEL} toolset=mixed prompt=${JSON.stringify(TASK_PROMPT)}`); const final = await harness.prompt(TASK_PROMPT); console.log(`final stopReason: ${final.stopReason}`); for (const block of final.content) { @@ -495,13 +497,13 @@ As a starting rule: reach for **A** first, even if it costs you one exploratory ## Notes -- **None of the three hand control back.** Once a run switches to CUA mode, playwright mode doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA mode re-establishes progress. +- **None of the three hand control back.** Once a run switches to CUA, Playwright doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA re-establishes progress. - **Swap models freely.** All three default to `anthropic:claude-sonnet-5` via `LOOP_MODEL`. Use `listLoopModels()` from `@onkernel/browser-loop/pi` to see everything else the catalog supports for both toolsets. -- **The whole handoff is recorded on one video.** Each script wraps the run in `client.browsers.replays.start()` / `.stop()`, so the playwright-mode attempt and the CUA-mode recovery land on the same replay. The replay URL prints as soon as recording starts and again once it's stopped and finished processing. +- **The whole handoff is recorded on one video.** Each script wraps the run in `client.browsers.replays.start()` / `.stop()`, so the Playwright attempt and the CUA recovery land on the same replay. The replay URL prints as soon as recording starts and again once it's stopped and finished processing. ## Next steps -- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA mode drives +- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA drives - [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on Kernel more generally - [Replays](/browsers/replays) — record a session end to end - [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place diff --git a/docs.json b/docs.json index 00c7c05..687c326 100644 --- a/docs.json +++ b/docs.json @@ -132,7 +132,7 @@ "browsers/ssh", "browsers/computer-controls", "browsers/playwright-execution", - "browsers/code-mode-fallback" + "browsers/playwright-cua-fallback" ] }, { From 167fdb26c8a875eaa53d6213fef5756230ea2b50 Mon Sep 17 00:00:00 2001 From: AnnaXWang <6621137+AnnaXWang@users.noreply.github.com> Date: Thu, 3 Sep 2026 00:26:37 +0000 Subject: [PATCH 08/13] Define browser agent toolsets --- browsers/playwright-cua-fallback.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/browsers/playwright-cua-fallback.mdx b/browsers/playwright-cua-fallback.mdx index cd2616e..63d8e9c 100644 --- a/browsers/playwright-cua-fallback.mdx +++ b/browsers/playwright-cua-fallback.mdx @@ -5,7 +5,7 @@ description: "Start with fast, DOM-based Playwright tools, then fall back to scr [Playwright](https://github.com/microsoft/playwright) is Microsoft's open-source browser automation framework, first released in 2020. It controls Chromium, Firefox, and WebKit through one API, with resilient element locators, automatic waiting, isolated browser contexts, and detailed tracing when something goes wrong. -Higher-level browser-agent tools often wrap Playwright in model-friendly abstractions. Those tools can be useful, but Playwright remains a strong starting point: it is widely adopted, well documented, and gives developers direct, inspectable control over the browser. For agents, DOM-based Playwright actions are usually faster, cheaper, and more predictable than reasoning from screenshots and clicking screen coordinates. +Higher-level browser-agent frameworks often wrap Playwright in model-friendly abstractions. Those frameworks can be useful, but Playwright remains a strong starting point: it is widely adopted, well documented, and gives developers direct, inspectable control over the browser. For agents, DOM-based Playwright actions are usually faster, cheaper, and more predictable than reasoning from screenshots and clicking screen coordinates. Some page interactions still cannot be completed reliably through simplified DOM-based tools. Pointer-driven drag-and-drop is a common example. Libraries such as `dnd-kit` and `SortableJS` may require several intermediate `mousemove` events before they recognize a drag. A tool that jumps directly from pressing to releasing can report success even though the page never recognized the gesture and the card did not move. @@ -26,6 +26,8 @@ Filtering and navigating work fine through the DOM. Dragging a card onto the boa ## Playwright vs. CUA +An agent tool is a callable operation that lets the model read browser state or take an action. The model selects a tool and supplies its inputs; the harness executes the operation and returns the result to the model. A **toolset** is the collection of tools available during a run. + `browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **CUA tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. | | Playwright | CUA | From b1a59b12aeca49511b704e889833c06c68931e6a Mon Sep 17 00:00:00 2001 From: Anna Wang Date: Wed, 2 Sep 2026 21:19:10 -0700 Subject: [PATCH 09/13] start the rename --- browsers/playwright-cua-fallback.mdx | 85 +++++++++++++++------------- 1 file changed, 46 insertions(+), 39 deletions(-) diff --git a/browsers/playwright-cua-fallback.mdx b/browsers/playwright-cua-fallback.mdx index 63d8e9c..a1d04fc 100644 --- a/browsers/playwright-cua-fallback.mdx +++ b/browsers/playwright-cua-fallback.mdx @@ -1,17 +1,17 @@ --- -title: "Using Playwright with CUA Fallback" -description: "Start with fast, DOM-based Playwright tools, then fall back to screenshot-based CUA tools for interactions they cannot complete" +title: "Using Playwright with Computer Use Fallback" +description: "Run an agent with fast, DOM-based Playwright tools, then fall back to Computer Use only when the DOM won't cooperate" --- -[Playwright](https://github.com/microsoft/playwright) is Microsoft's open-source browser automation framework, first released in 2020. It controls Chromium, Firefox, and WebKit through one API, with resilient element locators, automatic waiting, isolated browser contexts, and detailed tracing when something goes wrong. +[Playwright](https://github.com/microsoft/playwright) is Microsoft's open-source browser automation framework. It controls Chromium, Firefox, and WebKit through one API, with resilient element locators, automatic waiting, isolated browser contexts, and detailed tracing when something goes wrong. Higher-level browser-agent frameworks often wrap Playwright in model-friendly abstractions. Those frameworks can be useful, but Playwright remains a strong starting point: it is widely adopted, well documented, and gives developers direct, inspectable control over the browser. For agents, DOM-based Playwright actions are usually faster, cheaper, and more predictable than reasoning from screenshots and clicking screen coordinates. -Some page interactions still cannot be completed reliably through simplified DOM-based tools. Pointer-driven drag-and-drop is a common example. Libraries such as `dnd-kit` and `SortableJS` may require several intermediate `mousemove` events before they recognize a drag. A tool that jumps directly from pressing to releasing can report success even though the page never recognized the gesture and the card did not move. +Some page interactions still cannot be completed reliably through DOM-based tools. Pointer-driven drag-and-drop is a common example. Libraries such as `dnd-kit` and `SortableJS` may require several intermediate `mousemove` events before they recognize a drag. A tool that jumps directly from pressing to releasing can report success even though the page never recognized the gesture and the card did not move. -The answer is not to run the entire task through slower, screenshot-based computer use. Start with Playwright for navigation, reading, clicking, and form filling. Switch to CUA only for the interaction that needs realistic pointer movement, then continue without losing the browser's open tabs, page state, or the agent's conversation context. +The answer is not to run the entire task through slower, screenshot-based computer use. Start with Playwright for navigation, reading, clicking, and form filling. Switch to computer use only for the interaction that needs realistic pointer movement, then continue without losing the browser's open tabs, page state, or the agent's conversation context. -This cookbook demonstrates three ways to make that handoff with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), Kernel's tool package for browser agents, with complete code for each approach. +This cookbook demonstrates three ways to make that handoff between Playwright and Computer Use with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), KERNEL's tool package for browser agents. ## The example task @@ -24,11 +24,11 @@ then drag every item card to the Done column. Filtering and navigating work fine through the DOM. Dragging a card onto the board doesn't — `magnitasks.com`'s board uses pointer-based drag-and-drop, so it's a small, honest example of the failure mode this cookbook exists for. -## Playwright vs. CUA +## Playwright vs. Computer Use Toolsets An agent tool is a callable operation that lets the model read browser state or take an action. The model selects a tool and supplies its inputs; the harness executes the operation and returns the result to the model. A **toolset** is the collection of tools available during a run. -`browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **CUA tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. +`browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **Computer Use Agent (CUA) tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. | | Playwright | CUA | | --- | --- | --- | @@ -37,13 +37,15 @@ An agent tool is a callable operation that lets the model read browser state or | a click is | one DOM lookup + one CDP dispatch | a vision call, then an OS-level click at a coordinate — see [Computer Controls](/browsers/computer-controls) | | cost | cheap, fast, deterministic when the DOM cooperates | a full model round trip per action, but works on anything the page renders regardless of DOM structure | -Make Playwright your default: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. +Make Playwright your default toolset: it's faster and cheaper per action. Reach for [computer use](/integrations/computer-use/overview) for the specific interactions that don't hold up to DOM-ref execution: drag-and-drop on a pointer-sensor library, canvas-drawn UI, a file picker's native dialog, anything a screenshot can see that the accessibility tree can't reliably resolve. ## Picking a harness -The snippets below drive the browser with `browser-loop`, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff below a single method call — the model that picks up with CUA still has the entire Playwright conversation as context, and knows what it already tried. +The snippets below drive the browser with KERNEL's `browser-loop` harness, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. -The same ideas apply to whatever harness or tools you use to drive a browser. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh CUA call and carry forward what happened as plain-text context in the new prompt, the way approach A's handoff message does below. +`compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff between Playwright and CUA tools a single method call. The model that picks up with CUA tools still has the entire Playwright conversation as context, and knows what it already tried. + +If you choose a different harness than `browser-loop`, the same concepts apply. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh CUA call and carry forward what happened as plain-text context in the new prompt. We demonstrate that below in Approach A. ## Setup @@ -52,7 +54,7 @@ npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core t ``` -`browser-loop` pins an exact `@onkernel/sdk` version as a peer dependency. Installing a newer `@onkernel/sdk` elsewhere in the same project can hoist a mismatched version, which produces two incompatible `Kernel` client types — `attach({ client, browser })` won't type-check. Run `npm ls @onkernel/sdk` if you hit a type error there. +`browser-loop` pins an exact `@onkernel/sdk` version as a peer dependency. Installing a newer `@onkernel/sdk` elsewhere in the same project can hoist a mismatched version, which produces two incompatible `KERNEL` client types — `attach({ client, browser })` won't type-check. Run `npm ls @onkernel/sdk` if you hit a type error there. Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): @@ -63,19 +65,24 @@ KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.t KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts ``` -## Three ways to hand off +## Three ways to hand off between Playwright and Computer Use + +We recommend three different approaches: +- Per-Tool Attempt Limit +- Total Tool-Call Limit +- Model-Directed Handoff -All three run Playwright first and switch to CUA once it stops making progress. They differ in what you need to know about the task at integration time: +All three run Playwright first and switch to Computer Use once it stops making progress. They differ in what you need to know about the task at integration time: -| you know | approach | integration cost | +| what you you know | approach | integration cost | | --- | --- | --- | -| the specific Playwright tool likely to be unreliable for this task | [trigger-tool budget](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | -| the general shape of the task, not the specific risky action | [flat step-count budget](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | -| little to nothing — tasks arrive from an end user at runtime | [mixed toolset + steering prompt](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | +| the specific Playwright tool likely to be unreliable for this task | [Per-Tool Attempt Limit](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | +| the general shape of the task, not the specific risky action | [Total Tool-Call Limit](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | +| little to nothing — tasks arrive from an end user at runtime | [Model-Directed Handoff](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | -If you can name the risky tool, do — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. +If you can name the risky tool, go with a Per-Tool Attempt Limit — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. -### Approach A: trigger-tool budget +### Approach A: Per-Tool Attempt Limit Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific Playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. @@ -117,7 +124,7 @@ Use this when you're integrating against a known, fixed target and you already k * `computer_drag` succeeds -- a good, honest example of the failure mode this * fallback exists for. * - * Kernel's replay API records the failed Playwright drags and the CUA recovery + * KERNEL's replay API records the failed Playwright drags and the CUA recovery * on one video. The replay view URL prints as soon as recording starts and * again once it's stopped and finished processing. * @@ -125,11 +132,11 @@ Use this when you're integrating against a known, fixed target and you already k * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts * * Env: - * KERNEL_API_KEY required, Kernel browser API key + * KERNEL_API_KEY required, KERNEL browser API key * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) */ -import Kernel from "@onkernel/sdk"; +import KERNEL from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; @@ -160,7 +167,7 @@ async function main(): Promise { if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); requireLoopEnvApiKeyForModel(MODEL); - const client = new Kernel({ apiKey: kernelApiKey }); + const client = new KERNEL({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); const replay = await client.browsers.replays.start(browser.session_id); console.log(`recording started: ${replay.replay_view_url}`); @@ -243,9 +250,9 @@ void main(); ### Approach B: flat step-count budget -Use this when you know the general category of task but not the specific action likely to need CUA, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so Playwright gets a fixed total before the handoff regardless of what those actions turn out to be. +Use this when you know the general category of task but not the specific action likely to need Computer Use, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so Playwright gets a fixed total before the handoff regardless of what those actions turn out to be. -Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs CUA. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. +Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs Computer Use. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. ```ts /** @@ -270,17 +277,17 @@ Size the budget generously enough to survive the task's legitimate setup — nav * cards between columns, filter by assignee). See playwright-then-cua.ts for * why browser_drag reliably fails here and computer_drag reliably succeeds. * - * Kernel's replay API records both toolsets on one video. + * KERNEL's replay API records both toolsets on one video. * * Usage: * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts * * Env: - * KERNEL_API_KEY required, Kernel browser API key + * KERNEL_API_KEY required, KERNEL browser API key * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) */ -import Kernel from "@onkernel/sdk"; +import KERNEL from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; @@ -304,7 +311,7 @@ async function main(): Promise { if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); requireLoopEnvApiKeyForModel(MODEL); - const client = new Kernel({ apiKey: kernelApiKey }); + const client = new KERNEL({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); const replay = await client.browsers.replays.start(browser.session_id); console.log(`recording started: ${replay.replay_view_url}`); @@ -379,9 +386,9 @@ void main(); ### Approach C: mixed toolset and steering prompt -Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs CUA. `loop.toolsets.mixed()` hands the model both Playwright and CUA tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs Computer Use. `loop.toolsets.mixed()` hands the model both Playwright and CUA tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. -The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with Playwright tools before trying CUA, and which CUA tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with Playwright tools before trying Computer Use, and which Computer Use tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. ```ts /** @@ -411,17 +418,17 @@ The tradeoff is that a system prompt is a soft constraint on both axes that matt * cards between columns, filter by assignee). See playwright-then-cua.ts for * why browser_drag reliably fails here and computer_drag reliably succeeds. * - * The session is recorded end to end with Kernel's replay API. + * The session is recorded end to end with KERNEL's replay API. * * Usage: * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts * * Env: - * KERNEL_API_KEY required, Kernel browser API key + * KERNEL_API_KEY required, KERNEL browser API key * LOOP_MODEL optional, defaults to anthropic:claude-sonnet-5 * (needs the matching provider API key, e.g. ANTHROPIC_API_KEY) */ -import Kernel from "@onkernel/sdk"; +import KERNEL from "@onkernel/sdk"; import { AgentHarness, InMemorySessionRepo, type AgentHarnessEvent } from "@earendil-works/pi-agent-core"; import { loop } from "@onkernel/browser-loop"; import { attach, requireLoopEnvApiKeyForModel, type LoopModelRef } from "@onkernel/browser-loop/pi"; @@ -437,7 +444,7 @@ async function main(): Promise { if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); requireLoopEnvApiKeyForModel(MODEL); - const client = new Kernel({ apiKey: kernelApiKey }); + const client = new KERNEL({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); const replay = await client.browsers.replays.start(browser.session_id); console.log(`recording started: ${replay.replay_view_url}`); @@ -499,13 +506,13 @@ As a starting rule: reach for **A** first, even if it costs you one exploratory ## Notes -- **None of the three hand control back.** Once a run switches to CUA, Playwright doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once CUA re-establishes progress. +- **None of the three hand control back.** Once a run switches to computer-use tools, Playwright doesn't get another turn in that session. Fine for a single bounded task; worth revisiting for a longer-running agent that could benefit from returning to the cheaper tools once computer-use tools re-establishes progress. - **Swap models freely.** All three default to `anthropic:claude-sonnet-5` via `LOOP_MODEL`. Use `listLoopModels()` from `@onkernel/browser-loop/pi` to see everything else the catalog supports for both toolsets. -- **The whole handoff is recorded on one video.** Each script wraps the run in `client.browsers.replays.start()` / `.stop()`, so the Playwright attempt and the CUA recovery land on the same replay. The replay URL prints as soon as recording starts and again once it's stopped and finished processing. +- **The whole handoff is recorded on one video.** Each script wraps the run in `client.browsers.replays.start()` / `.stop()`, so the Playwright attempt and the computer-use recovery land on the same replay. The replay URL prints as soon as recording starts and again once it's stopped and finished processing. ## Next steps - [Computer Controls](/browsers/computer-controls) — the OS-level API CUA drives -- [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on Kernel more generally +- [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on KERNEL more generally - [Replays](/browsers/replays) — record a session end to end - [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place From 5f33aa2439f22fd333f55a99c0d2928ac3ab2a4a Mon Sep 17 00:00:00 2001 From: AnnaXWang <6621137+AnnaXWang@users.noreply.github.com> Date: Thu, 3 Sep 2026 04:31:37 +0000 Subject: [PATCH 10/13] Standardize computer use handoff terminology --- ...x => playwright-computer-use-fallback.mdx} | 250 +++++++++--------- docs.json | 2 +- 2 files changed, 121 insertions(+), 131 deletions(-) rename browsers/{playwright-cua-fallback.mdx => playwright-computer-use-fallback.mdx} (64%) diff --git a/browsers/playwright-cua-fallback.mdx b/browsers/playwright-computer-use-fallback.mdx similarity index 64% rename from browsers/playwright-cua-fallback.mdx rename to browsers/playwright-computer-use-fallback.mdx index a1d04fc..6bc0d6b 100644 --- a/browsers/playwright-cua-fallback.mdx +++ b/browsers/playwright-computer-use-fallback.mdx @@ -1,6 +1,6 @@ --- title: "Using Playwright with Computer Use Fallback" -description: "Run an agent with fast, DOM-based Playwright tools, then fall back to Computer Use only when the DOM won't cooperate" +description: "Run an agent with fast, DOM-based Playwright tools, then fall back to computer use only when the DOM won't cooperate" --- [Playwright](https://github.com/microsoft/playwright) is Microsoft's open-source browser automation framework. It controls Chromium, Firefox, and WebKit through one API, with resilient element locators, automatic waiting, isolated browser contexts, and detailed tracing when something goes wrong. @@ -11,7 +11,7 @@ Some page interactions still cannot be completed reliably through DOM-based tool The answer is not to run the entire task through slower, screenshot-based computer use. Start with Playwright for navigation, reading, clicking, and form filling. Switch to computer use only for the interaction that needs realistic pointer movement, then continue without losing the browser's open tabs, page state, or the agent's conversation context. -This cookbook demonstrates three ways to make that handoff between Playwright and Computer Use with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), KERNEL's tool package for browser agents. +This cookbook demonstrates three ways to make that handoff between Playwright and computer use with [`@onkernel/browser-loop`](https://www.npmjs.com/package/@onkernel/browser-loop), KERNEL's tool package for browser agents. ## The example task @@ -24,13 +24,13 @@ then drag every item card to the Done column. Filtering and navigating work fine through the DOM. Dragging a card onto the board doesn't — `magnitasks.com`'s board uses pointer-based drag-and-drop, so it's a small, honest example of the failure mode this cookbook exists for. -## Playwright vs. Computer Use Toolsets +## Playwright vs. computer use An agent tool is a callable operation that lets the model read browser state or take an action. The model selects a tool and supplies its inputs; the harness executes the operation and returns the result to the model. A **toolset** is the collection of tools available during a run. -`browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **Computer Use Agent (CUA) tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. +`browser-loop` provides two toolsets for controlling the same browser session. **Playwright tools** (`loop.toolsets.browser()`) find page elements through the DOM and act on them by reference. **Computer-use tools** (`loop.toolsets.computer()`) read screenshots and control the pointer using screen coordinates. -| | Playwright | CUA | +| | Playwright | computer use | | --- | --- | --- | | tools | `browser_*` | `computer_*` | | driven by | an accessibility snapshot, resolved by ref | a screenshot, read pixel by pixel | @@ -41,11 +41,11 @@ Make Playwright your default toolset: it's faster and cheaper per action. Reach ## Picking a harness -The snippets below drive the browser with KERNEL's `browser-loop` harness, bound here to [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. +The snippets below use `browser-loop` with [`@earendil-works/pi-agent-core`](https://www.npmjs.com/package/@earendil-works/pi-agent-core)'s `AgentHarness`. -`compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff between Playwright and CUA tools a single method call. The model that picks up with CUA tools still has the entire Playwright conversation as context, and knows what it already tried. +`compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff between Playwright and computer-use tools a single method call. The model that picks up with computer-use tools still has the entire Playwright conversation as context, and knows what it already tried. -If you choose a different harness than `browser-loop`, the same concepts apply. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh CUA call and carry forward what happened as plain-text context in the new prompt. We demonstrate that below in Approach A. +If you use a different agent harness, the same concepts apply. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh call with computer-use tools and carry forward what happened as plain-text context in the new prompt. The Per-Tool Attempt Limit example below shows what to include in that handoff message. ## Setup @@ -60,37 +60,32 @@ npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core t Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): ```bash -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-attempt-limit.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-total-tool-call-limit.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-model-directed-handoff.ts ``` -## Three ways to hand off between Playwright and Computer Use +## Three ways to hand off between Playwright and computer use -We recommend three different approaches: -- Per-Tool Attempt Limit -- Total Tool-Call Limit -- Model-Directed Handoff +All three run Playwright first and switch to computer use once it stops making progress. They differ in what you need to know about the task at integration time: -All three run Playwright first and switch to Computer Use once it stops making progress. They differ in what you need to know about the task at integration time: - -| what you you know | approach | integration cost | +| what you know | approach | integration cost | | --- | --- | --- | -| the specific Playwright tool likely to be unreliable for this task | [Per-Tool Attempt Limit](#approach-a-trigger-tool-budget) | lowest — one config line, no tuning | -| the general shape of the task, not the specific risky action | [Total Tool-Call Limit](#approach-b-flat-step-count-budget) | needs a per-task number, found by testing | -| little to nothing — tasks arrive from an end user at runtime | [Model-Directed Handoff](#approach-c-mixed-toolset-and-steering-prompt) | no handoff logic, but no bound on the model's own switching either | +| the specific Playwright tool likely to be unreliable for this task | [Per-Tool Attempt Limit](#per-tool-attempt-limit) | lowest — one config line, no tuning | +| the general shape of the task, not the specific risky action | [Total Tool-Call Limit](#total-tool-call-limit) | needs a per-task number, found by testing | +| little to nothing — tasks arrive from an end user at runtime | [Model-Directed Handoff](#model-directed-handoff) | no handoff logic, but no bound on the model's own switching either | -If you can name the risky tool, go with a Per-Tool Attempt Limit — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. +If you can name the risky tool, use **Per-Tool Attempt Limit** — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. -### Approach A: Per-Tool Attempt Limit +### Per-Tool Attempt Limit -Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific Playwright tool this task needs CUA's help with. `FALLBACK_TRIGGER_TOOLS` names it; the harness counts completed attempts against that tool (most DOM-ref tools report success even when they don't produce the intended effect, so a completed attempt against a still-unfinished task is itself the signal) and hands off once `FALLBACK_TRIGGER_BUDGET` is spent. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches the trigger tool at all. +Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific Playwright tool needs help from computer-use tools. `PLAYWRIGHT_TOOLS_WITH_ATTEMPT_LIMITS` names those tools; the harness counts completed attempts for each one and hands off when any tool reaches `PER_TOOL_ATTEMPT_LIMIT`. Most DOM-ref tools report success even when they don't produce the intended effect, so an attempt against a still-unfinished task is itself the signal. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches a per-tool limit. ```ts /** - * Cookbook: Playwright first, fall back to CUA after the Playwright - * tool(s) this task depends on have had their shot and haven't finished the - * job. + * Cookbook: Playwright first, fall back to computer-use tools after a + * task-critical Playwright tool reaches its per-tool attempt limit without + * finishing the job. * * `browser_drag` (the atomic tool @onkernel/browser-loop compiles for * the Playwright toolset) drags by resolving two DOM/viewport points and @@ -102,21 +97,18 @@ Use this when you're integrating against a known, fixed target and you already k * The tool call itself still reports success: there's no error to catch, only * a card that never moved. * - * `computer_drag` (from the CUA toolset) takes a multi-point pixel path and - * dispatches a real mousemove sequence along it, which is enough motion for - * those same libraries to pick up the gesture. + * `computer_drag` (from the computer-use toolset) takes a multi-point pixel + * path and dispatches a real mousemove sequence along it, which is enough + * motion for those same libraries to pick up the gesture. * - * This script runs a task with Playwright and hands off to CUA once - * Playwright has spent FALLBACK_TRIGGER_BUDGET attempts on - * FALLBACK_TRIGGER_TOOLS and the task still isn't done. The trigger keys on - * that specific tool rather than a generic action count, so setup steps like - * navigation and filtering never eat into the budget before the action that - * actually needs CUA gets a turn. FALLBACK_TRIGGER_TOOLS is `browser_drag` - * here because drag-and-drop is this task's known-unreliable action; adapt it - * to whatever Playwright tool your own task expects to need CUA's help for - * (a file upload, a canvas interaction, anything DOM-ref execution doesn't - * reliably drive). An action safety cap is a backstop for pages that never - * reach a trigger tool at all. + * This script hands off to computer-use tools once any tool in + * PLAYWRIGHT_TOOLS_WITH_ATTEMPT_LIMITS reaches PER_TOOL_ATTEMPT_LIMIT and the + * task still isn't done. Each selected tool has its own attempt count, so + * setup steps like navigation and filtering never consume another tool's + * limit. `browser_drag` is selected here because drag-and-drop is this task's + * known-unreliable action; adapt the set to the Playwright tools your task + * expects to need computer-use help for. An action safety cap is a backstop + * for pages that never reach a per-tool limit. * * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag * cards between columns, filter by assignee). Its board uses pointer-based @@ -124,12 +116,12 @@ Use this when you're integrating against a known, fixed target and you already k * `computer_drag` succeeds -- a good, honest example of the failure mode this * fallback exists for. * - * KERNEL's replay API records the failed Playwright drags and the CUA recovery - * on one video. The replay view URL prints as soon as recording starts and - * again once it's stopped and finished processing. + * KERNEL's replay API records the failed Playwright drags and the computer-use + * recovery on one video. The replay view URL prints as soon as recording + * starts and again once it's stopped and finished processing. * * Usage: - * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua.ts + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-attempt-limit.ts * * Env: * KERNEL_API_KEY required, KERNEL browser API key @@ -147,15 +139,12 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// Primary trigger: the Playwright tool(s) this task depends on and known -// to be unreliable. Swap this per task -- browser_drag here because -// drag-and-drop is what this task needs CUA's help with; a different task -// might key on browser_fill for a canvas-backed input, or nothing at all if -// no single tool is the known risk. These tools report no execution error -// even when they don't produce the intended effect, so this is a count of -// attempts -- an attempt that doesn't finish the task is itself the signal. -const FALLBACK_TRIGGER_TOOLS = new Set(["browser_drag"]); -const FALLBACK_TRIGGER_BUDGET = 2; +// Playwright tools this task depends on and that are known to be unreliable. +// Each selected tool gets its own attempt count. These tools report no +// execution error even when they don't produce the intended effect, so an +// attempt that doesn't finish the task is itself the signal. +const PLAYWRIGHT_TOOLS_WITH_ATTEMPT_LIMITS = new Set(["browser_drag"]); +const PER_TOOL_ATTEMPT_LIMIT = 2; // Backstop: total tool calls Playwright gets regardless of whether it // ever reaches a drag, so a page that never triggers one (wrong selector, @@ -174,7 +163,7 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-computer-use-per-tool-attempt-limit" }); const playwrightPair = kb.compile({ model: MODEL, @@ -191,22 +180,28 @@ async function main(): Promise { playwrightPair.activate(harness); let actionTurns = 0; - let triggerAttempts = 0; - let budgetExhausted = false; + const attemptsByTool = new Map(); + let handoffRequested = false; const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; actionTurns += 1; - const isTrigger = FALLBACK_TRIGGER_TOOLS.has(event.toolName); - if (isTrigger) triggerAttempts += 1; - console.log(`[playwright ${actionTurns}] ${event.toolName} error=${event.isError}${isTrigger ? ` (trigger attempt ${triggerAttempts}/${FALLBACK_TRIGGER_BUDGET})` : ""}`); - if (budgetExhausted) return; - if (triggerAttempts >= FALLBACK_TRIGGER_BUDGET) { - budgetExhausted = true; - console.log("[playwright] trigger tool budget exhausted, aborting run to switch to CUA"); + let attemptStatus = ""; + let reachedAttemptLimit = false; + if (PLAYWRIGHT_TOOLS_WITH_ATTEMPT_LIMITS.has(event.toolName)) { + const attempts = (attemptsByTool.get(event.toolName) ?? 0) + 1; + attemptsByTool.set(event.toolName, attempts); + attemptStatus = ` (attempt ${attempts}/${PER_TOOL_ATTEMPT_LIMIT})`; + reachedAttemptLimit = attempts >= PER_TOOL_ATTEMPT_LIMIT; + } + console.log(`[playwright ${actionTurns}] ${event.toolName} error=${event.isError}${attemptStatus}`); + if (handoffRequested) return; + if (reachedAttemptLimit) { + handoffRequested = true; + console.log("[playwright] per-tool attempt limit reached, aborting run to switch to computer-use tools"); void harness.abort(); } else if (actionTurns >= ACTION_SAFETY_CAP) { - budgetExhausted = true; - console.log("[playwright] action safety cap reached without a trigger tool attempt, aborting run to switch to CUA"); + handoffRequested = true; + console.log("[playwright] action safety cap reached, aborting run to switch to computer-use tools"); void harness.abort(); } }); @@ -216,14 +211,14 @@ async function main(): Promise { unsubscribe(); if (final.stopReason === "aborted") { - console.log("[cua] Playwright did not finish in budget, switching to CUA tools"); - const cuaPair = kb.compile({ + console.log("[computer-use] Playwright did not finish within its limits, switching toolsets"); + const computerPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await cuaPair.apply(harness); + await computerPair.apply(harness); - console.log(`model=${MODEL} toolset=cua`); + console.log(`model=${MODEL} toolset=computer`); final = await harness.prompt( "Playwright drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + @@ -248,39 +243,38 @@ async function main(): Promise { void main(); ``` -### Approach B: flat step-count budget +### Total Tool-Call Limit -Use this when you know the general category of task but not the specific action likely to need Computer Use, so naming one tool up front isn't realistic. `PLAYWRIGHT_ACTION_BUDGET` counts every completed tool call, not just attempts at one tool, so Playwright gets a fixed total before the handoff regardless of what those actions turn out to be. +Use this when you know the general category of task but not the specific action likely to need computer use, so naming one tool up front isn't realistic. `TOTAL_TOOL_CALL_LIMIT` counts every completed Playwright tool call and hands off when the total reaches that limit, regardless of which tools the agent used. -Size the budget generously enough to survive the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs Computer Use. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. +Set the limit high enough to cover the task's legitimate setup — navigation, filtering, form-filling — before it reaches the action that needs computer use. There's no principled way to pick the number without running the task and looking; that's the direct cost of not needing to know which tool will fail. ```ts /** - * Cookbook variant B: Playwright first, fall back to CUA after a - * flat step-count budget -- for when you know the general shape of the task - * (browser automation against an unfamiliar app) without knowing which - * specific Playwright tool is going to be the unreliable one. Use this over - * the trigger-tool variant (playwright-then-cua.ts) when you can't name the - * risky action in advance; use it over the mixed-toolset variant - * (playwright-then-cua-mixed.ts) when you want a deterministic, bounded - * Playwright budget instead of trusting a system prompt to self-limit. + * Total Tool-Call Limit: Playwright first, fall back to computer-use tools after + * a total tool-call limit. Use this when you know the general shape of the + * task without knowing which specific Playwright tool will be unreliable. Use + * this over the per-tool attempt limit when you can't name the risky action in + * advance; use it over model-directed handoff when you want a deterministic, + * bounded Playwright run instead of trusting a system prompt to self-limit. * - * The budget counts every completed tool call, so it has to be generous + * The limit counts every completed tool call, so it has to be generous * enough to survive whatever setup (navigation, filtering, form-filling) the * task legitimately needs before it reaches the action that actually needs - * CUA. Too tight and it burns out on setup before ever attempting the risky - * action. Picking the right number takes running the task and looking; that - * imprecision is the tradeoff for skipping the step of naming which tool - * will fail. + * computer use. Too tight and it stops during setup before ever attempting + * the risky action. Picking the right number takes running the task and + * looking; that imprecision is the tradeoff for skipping the step of naming + * which tool will fail. * * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag - * cards between columns, filter by assignee). See playwright-then-cua.ts for - * why browser_drag reliably fails here and computer_drag reliably succeeds. + * cards between columns, filter by assignee). See + * playwright-computer-use-per-tool-attempt-limit.ts for why browser_drag + * reliably fails here and computer_drag reliably succeeds. * * KERNEL's replay API records both toolsets on one video. * * Usage: - * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-stepcount.ts + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-total-tool-call-limit.ts * * Env: * KERNEL_API_KEY required, KERNEL browser API key @@ -298,13 +292,10 @@ const TASK_PROMPT = const MODEL = (process.env.LOOP_MODEL as LoopModelRef | undefined) ?? "anthropic:claude-sonnet-5"; -// How many tool calls of any kind Playwright gets before we hand off to -// CUA. Size this to comfortably cover the task's normal setup steps plus -// a couple of attempts at whatever turns out to be the hard part -- too tight -// and it can exhaust the budget before ever reaching the action that needs -// CUA. There's no per-tool signal here, so this is deliberately generous -// compared to the trigger-tool variant's budget of 1-2. -const PLAYWRIGHT_ACTION_BUDGET = 10; +// Total Playwright tool calls allowed before the handoff to computer-use tools. +// Set this high enough to cover normal setup plus a few attempts at the action +// that needs computer use. +const TOTAL_TOOL_CALL_LIMIT = 10; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; @@ -318,7 +309,7 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-stepcount" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-computer-use-total-tool-call-limit" }); const playwrightPair = kb.compile({ model: MODEL, @@ -334,15 +325,15 @@ async function main(): Promise { }); playwrightPair.activate(harness); - let actionTurns = 0; - let budgetExhausted = false; + let toolCalls = 0; + let handoffRequested = false; const unsubscribe = harness.subscribe((event: AgentHarnessEvent) => { if (event.type !== "tool_execution_end") return; - actionTurns += 1; - console.log(`[playwright ${actionTurns}/${PLAYWRIGHT_ACTION_BUDGET}] ${event.toolName} error=${event.isError}`); - if (actionTurns >= PLAYWRIGHT_ACTION_BUDGET && !budgetExhausted) { - budgetExhausted = true; - console.log("[playwright] action budget exhausted, aborting run to switch to CUA"); + toolCalls += 1; + console.log(`[playwright ${toolCalls}/${TOTAL_TOOL_CALL_LIMIT}] ${event.toolName} error=${event.isError}`); + if (toolCalls >= TOTAL_TOOL_CALL_LIMIT && !handoffRequested) { + handoffRequested = true; + console.log("[playwright] total tool-call limit reached, aborting run to switch to computer-use tools"); void harness.abort(); } }); @@ -352,14 +343,14 @@ async function main(): Promise { unsubscribe(); if (final.stopReason === "aborted") { - console.log("[cua] Playwright did not finish in budget, switching to CUA tools"); - const cuaPair = kb.compile({ + console.log("[computer-use] Playwright did not finish within its limit, switching toolsets"); + const computerPair = kb.compile({ model: MODEL, tools: loop.toolsets.computer(), }); - await cuaPair.apply(harness); + await computerPair.apply(harness); - console.log(`model=${MODEL} toolset=cua`); + console.log(`model=${MODEL} toolset=computer`); final = await harness.prompt( "Playwright drag actions were not registering on this page's drag-and-drop board -- " + "the cards weren't moving. Take a screenshot to see where things stand, then finish the " + @@ -384,22 +375,20 @@ async function main(): Promise { void main(); ``` -### Approach C: mixed toolset and steering prompt +### Model-Directed Handoff -Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs Computer Use. `loop.toolsets.mixed()` hands the model both Playwright and CUA tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. +Use this when the task isn't known at integration time at all — a general-purpose agent product where an end user's request determines the site, the workflow, and whether anything needs computer use. `loop.toolsets.mixed()` gives the model both Playwright and computer-use tools from the start; a system prompt steers it toward the cheaper `browser_*` tools by default and toward `computer_*` tools once a `browser_*` action doesn't produce the expected effect. There's no handoff code to write. -The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with Playwright tools before trying Computer Use, and which Computer Use tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. +The tradeoff is that a system prompt is a soft constraint on both axes that matter: how long the model sticks with Playwright tools before trying computer use, and which computer-use tool it reaches for once it does. Nothing here enforces a hard bound, and nothing guarantees it picks the purpose-built tool for the job over reassembling the effect from lower-level primitives. ```ts /** - * Cookbook variant C: give the model both Playwright and CUA tools - * up front (loop.toolsets.mixed()) and let it choose per action, steered by a - * system prompt toward Playwright tools by default. Use this over the two - * scripted-handoff variants (playwright-then-cua.ts, - * playwright-then-cua-stepcount.ts) when you don't want to write any handoff - * logic at all -- appropriate for a general-purpose agent whose tasks aren't - * known ahead of time, where you can't pre-decide a trigger tool or a - * reasonable step budget. + * Model-Directed Handoff: give the model both Playwright and computer-use tools up + * front (loop.toolsets.mixed()) and let it choose per action, steered by a + * system prompt toward Playwright tools by default. Use model-directed handoff + * when you don't want to write handoff logic for a general-purpose agent whose + * tasks aren't known ahead of time, so you can't preselect a limited tool or + * set a reasonable total tool-call limit. * * The tradeoff: with both toolsets available, a model can try browser_* * tools first, take a screenshot, notice a drag hasn't landed, and switch to @@ -410,18 +399,19 @@ The tradeoff is that a system prompt is a soft constraint on both axes that matt * for the same result. A system prompt is a soft constraint on both when it * switches and which tool it reaches for once it does; naming computer_drag * explicitly in a handoff message (as the scripted variants do) is what - * keeps CUA fast once it's in control. Nothing here enforces that a + * keeps computer use fast once it's in control. Nothing here enforces that a * real handoff even happens -- there's no hard bound on how long the model * sticks with browser_* tools before trying computer_* ones. * * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag - * cards between columns, filter by assignee). See playwright-then-cua.ts for - * why browser_drag reliably fails here and computer_drag reliably succeeds. + * cards between columns, filter by assignee). See + * playwright-computer-use-per-tool-attempt-limit.ts for why browser_drag + * reliably fails here and computer_drag reliably succeeds. * * The session is recorded end to end with KERNEL's replay API. * * Usage: - * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-then-cua-mixed.ts + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-model-directed-handoff.ts * * Env: * KERNEL_API_KEY required, KERNEL browser API key @@ -451,7 +441,7 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-then-cua-mixed" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-computer-use-model-directed-handoff" }); const compiled = kb.compile({ model: MODEL, tools: loop.toolsets.mixed() }); const harness = new AgentHarness({ @@ -498,11 +488,11 @@ void main(); | approach | needs knowing | integration effort | switching behavior | | --- | --- | --- | --- | -| A: trigger-tool budget | the exact risky tool | one config line, no tuning | fires precisely once the named tool has had its shot | -| B: flat step-count budget | enough about the task to size a number | a number to find and re-check per task | fires on total action count, blind to which tool it was | -| C: mixed toolset + prompt | nothing in advance | no handoff logic at all | left entirely to the model's own judgment, with no hard bound | +| Per-Tool Attempt Limit | the exact risky tool | one config line, no tuning | fires when a selected tool reaches its attempt limit | +| Total Tool-Call Limit | enough about the task to size a number | a number to find and re-check per task | fires on total tool-call count, regardless of which tools ran | +| Model-Directed Handoff | nothing in advance | no handoff logic at all | left entirely to the model's own judgment, with no hard bound | -As a starting rule: reach for **A** first, even if it costs you one exploratory run to find the tool name. Fall back to **B** only when the task genuinely varies enough that naming a trigger tool isn't realistic. Reach for **C** only when you can't write task-specific logic at all — and if you do, consider pairing it with a hard action cap as a backstop against its unbounded soft constraint. +As a starting rule, reach for **Per-Tool Attempt Limit** first, even if it costs you one exploratory run to find the tool name. Use **Total Tool-Call Limit** only when the task varies enough that naming a specific tool isn't realistic. Use **Model-Directed Handoff** only when you can't write task-specific logic at all — and if you do, consider pairing it with a hard action cap as a backstop against its unbounded soft constraint. ## Notes @@ -512,7 +502,7 @@ As a starting rule: reach for **A** first, even if it costs you one exploratory ## Next steps -- [Computer Controls](/browsers/computer-controls) — the OS-level API CUA drives +- [Computer Controls](/browsers/computer-controls) — the OS-level API that computer-use tools drive - [Computer Use overview](/integrations/computer-use/overview) — running computer-use models on KERNEL more generally - [Replays](/browsers/replays) — record a session end to end - [Stealth mode](/browsers/bot-detection/stealth) — reduce how often a page notices the automation in the first place diff --git a/docs.json b/docs.json index 687c326..d89c716 100644 --- a/docs.json +++ b/docs.json @@ -132,7 +132,7 @@ "browsers/ssh", "browsers/computer-controls", "browsers/playwright-execution", - "browsers/playwright-cua-fallback" + "browsers/playwright-computer-use-fallback" ] }, { From 575fd95adeeb32c548d61f24d5b2047f3e0e3718 Mon Sep 17 00:00:00 2001 From: AnnaXWang <6621137+AnnaXWang@users.noreply.github.com> Date: Thu, 3 Sep 2026 04:38:23 +0000 Subject: [PATCH 11/13] Shorten per-tool approach name --- browsers/playwright-computer-use-fallback.mdx | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/browsers/playwright-computer-use-fallback.mdx b/browsers/playwright-computer-use-fallback.mdx index 6bc0d6b..f70688d 100644 --- a/browsers/playwright-computer-use-fallback.mdx +++ b/browsers/playwright-computer-use-fallback.mdx @@ -45,7 +45,7 @@ The snippets below use `browser-loop` with [`@earendil-works/pi-agent-core`](htt `compiled.apply(harness)` swaps a *running* harness onto a new (model, tools) pair without resetting browser refs, open tabs, or the conversation transcript, which is what makes the mid-session handoff between Playwright and computer-use tools a single method call. The model that picks up with computer-use tools still has the entire Playwright conversation as context, and knows what it already tried. -If you use a different agent harness, the same concepts apply. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh call with computer-use tools and carry forward what happened as plain-text context in the new prompt. The Per-Tool Attempt Limit example below shows what to include in that handoff message. +If you use a different agent harness, the same concepts apply. If your setup doesn't expose an equivalent live tool-catalog swap, implement the handoff as two sequential calls instead: run Playwright to completion or failure, then start a fresh call with computer-use tools and carry forward what happened as plain-text context in the new prompt. The Per-Tool Limit example below shows what to include in that handoff message. ## Setup @@ -60,7 +60,7 @@ npm install @onkernel/browser-loop @onkernel/sdk @earendil-works/pi-agent-core t Every script needs a `KERNEL_API_KEY` and a provider key for whichever model `LOOP_MODEL` points at (`anthropic:claude-sonnet-5` by default, so `ANTHROPIC_API_KEY`): ```bash -KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-attempt-limit.ts +KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-limit.ts KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-total-tool-call-limit.ts KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-model-directed-handoff.ts ``` @@ -71,19 +71,19 @@ All three run Playwright first and switch to computer use once it stops making p | what you know | approach | integration cost | | --- | --- | --- | -| the specific Playwright tool likely to be unreliable for this task | [Per-Tool Attempt Limit](#per-tool-attempt-limit) | lowest — one config line, no tuning | +| the specific Playwright tool likely to be unreliable for this task | [Per-Tool Limit](#per-tool-limit) | lowest — one config line, no tuning | | the general shape of the task, not the specific risky action | [Total Tool-Call Limit](#total-tool-call-limit) | needs a per-task number, found by testing | | little to nothing — tasks arrive from an end user at runtime | [Model-Directed Handoff](#model-directed-handoff) | no handoff logic, but no bound on the model's own switching either | -If you can name the risky tool, use **Per-Tool Attempt Limit** — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. +If you can name the risky tool, use **Per-Tool Limit** — it's the only one of the three that's both precise and free of tuning. The other two exist for when the task genuinely isn't known until runtime. -### Per-Tool Attempt Limit +### Per-Tool Limit Use this when you're integrating against a known, fixed target and you already know — or can find in one test run — which specific Playwright tool needs help from computer-use tools. `PLAYWRIGHT_TOOLS_WITH_ATTEMPT_LIMITS` names those tools; the harness counts completed attempts for each one and hands off when any tool reaches `PER_TOOL_ATTEMPT_LIMIT`. Most DOM-ref tools report success even when they don't produce the intended effect, so an attempt against a still-unfinished task is itself the signal. `ACTION_SAFETY_CAP` is a backstop for a run that never reaches a per-tool limit. ```ts /** - * Cookbook: Playwright first, fall back to computer-use tools after a + * Per-Tool Limit: Playwright first, fall back to computer-use tools after a * task-critical Playwright tool reaches its per-tool attempt limit without * finishing the job. * @@ -121,7 +121,7 @@ Use this when you're integrating against a known, fixed target and you already k * starts and again once it's stopped and finished processing. * * Usage: - * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-attempt-limit.ts + * KERNEL_API_KEY=... ANTHROPIC_API_KEY=... npx tsx playwright-computer-use-per-tool-limit.ts * * Env: * KERNEL_API_KEY required, KERNEL browser API key @@ -163,7 +163,7 @@ async function main(): Promise { const kb = attach({ client, browser }); try { - const session = await new InMemorySessionRepo().create({ id: "playwright-computer-use-per-tool-attempt-limit" }); + const session = await new InMemorySessionRepo().create({ id: "playwright-computer-use-per-tool-limit" }); const playwrightPair = kb.compile({ model: MODEL, @@ -268,7 +268,7 @@ Set the limit high enough to cover the task's legitimate setup — navigation, f * * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag * cards between columns, filter by assignee). See - * playwright-computer-use-per-tool-attempt-limit.ts for why browser_drag + * playwright-computer-use-per-tool-limit.ts for why browser_drag * reliably fails here and computer_drag reliably succeeds. * * KERNEL's replay API records both toolsets on one video. @@ -405,7 +405,7 @@ The tradeoff is that a system prompt is a soft constraint on both axes that matt * * Demo target: magnitasks.com, a Kanban-style task board (Tasks page, drag * cards between columns, filter by assignee). See - * playwright-computer-use-per-tool-attempt-limit.ts for why browser_drag + * playwright-computer-use-per-tool-limit.ts for why browser_drag * reliably fails here and computer_drag reliably succeeds. * * The session is recorded end to end with KERNEL's replay API. @@ -488,11 +488,11 @@ void main(); | approach | needs knowing | integration effort | switching behavior | | --- | --- | --- | --- | -| Per-Tool Attempt Limit | the exact risky tool | one config line, no tuning | fires when a selected tool reaches its attempt limit | +| Per-Tool Limit | the exact risky tool | one config line, no tuning | fires when a selected tool reaches its attempt limit | | Total Tool-Call Limit | enough about the task to size a number | a number to find and re-check per task | fires on total tool-call count, regardless of which tools ran | | Model-Directed Handoff | nothing in advance | no handoff logic at all | left entirely to the model's own judgment, with no hard bound | -As a starting rule, reach for **Per-Tool Attempt Limit** first, even if it costs you one exploratory run to find the tool name. Use **Total Tool-Call Limit** only when the task varies enough that naming a specific tool isn't realistic. Use **Model-Directed Handoff** only when you can't write task-specific logic at all — and if you do, consider pairing it with a hard action cap as a backstop against its unbounded soft constraint. +As a starting rule, reach for **Per-Tool Limit** first, even if it costs you one exploratory run to find the tool name. Use **Total Tool-Call Limit** only when the task varies enough that naming a specific tool isn't realistic. Use **Model-Directed Handoff** only when you can't write task-specific logic at all — and if you do, consider pairing it with a hard action cap as a backstop against its unbounded soft constraint. ## Notes From c9b3c7e2705e7a8d7183e3ffeb8968a0d415de30 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:32:52 +0000 Subject: [PATCH 12/13] Link Control intro to the Playwright/computer-use fallback cookbook Reframes the agent recommendation as starting with playwright execution and falling back to computer use, with a link to the new cookbook that shows the pattern. --- introduction/control.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/introduction/control.mdx b/introduction/control.mdx index 9c4e6a7..3be9d74 100644 --- a/introduction/control.mdx +++ b/introduction/control.mdx @@ -3,7 +3,7 @@ title: "Control" description: "Drive the browser with computer use, playwright execution, CDP, or WebDriver BiDi" --- -Kernel browsers expose four ways to drive a session. For agents, we recommend [computer use](/browsers/computer-controls) or [playwright execution](/browsers/playwright-execution) — both run co-located with the browser and avoid the bot-detection surface a direct CDP connection introduces. +Kernel browsers expose four ways to drive a session. For agents, we recommend starting with [playwright execution](/browsers/playwright-execution) and falling back to [computer use](/browsers/computer-controls) for the interactions it can't reliably drive — see our [cookbook](/browsers/playwright-computer-use-fallback) for the pattern. Both run co-located with the browser and avoid the bot-detection surface a direct CDP connection introduces. From 0bb82814bf60085941d6610622e63cd03684d917 Mon Sep 17 00:00:00 2001 From: dprevoznik <58714078+dprevoznik@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:41:42 +0000 Subject: [PATCH 13/13] Use exact wording for Control intro cookbook link --- introduction/control.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/introduction/control.mdx b/introduction/control.mdx index 3be9d74..06abc09 100644 --- a/introduction/control.mdx +++ b/introduction/control.mdx @@ -3,7 +3,9 @@ title: "Control" description: "Drive the browser with computer use, playwright execution, CDP, or WebDriver BiDi" --- -Kernel browsers expose four ways to drive a session. For agents, we recommend starting with [playwright execution](/browsers/playwright-execution) and falling back to [computer use](/browsers/computer-controls) for the interactions it can't reliably drive — see our [cookbook](/browsers/playwright-computer-use-fallback) for the pattern. Both run co-located with the browser and avoid the bot-detection surface a direct CDP connection introduces. +Kernel browsers expose four ways to drive a session. For agents, we recommend starting with playwright execution and falling back to computer use, here's our guide: [playwright w/ computer use fallback](/browsers/playwright-computer-use-fallback). + +Both run co-located with the browser and avoid the bot-detection surface a direct CDP connection introduces.