Repository navigation
Re-evaluate browser_* tools for a click-through install (Playwright bundling overhead) #118
Description
Challenge
The goal is that browser_* tools (DOM-tree level control) work right after a click-through install of the .app.
Today browser_* depends on a separately installed @playwright/mcp (Node + Chromium), reached via --upstream browser=http://127.0.0.1:8794/mcp (poc/pw-mcp, loopback only). This has two problems:
- Asking users to install and run Playwright, or bundling Node + Chromium inside the .app, has high install, size and maintenance overhead.
- A double-click (no-flag) launch adds no upstream, so installed users get no browser tools at all.
Both the Chrome-extension route (Muse style) and Playwright are intrusive to some degree.
Wizard plan: add a "Browser tools" page between the permissions page and the tailnet page.
Options considered
| Approach | Intrusion | Capability |
|---|---|---|
| Apple Events execute-JavaScript on Chrome and Safari | No extension. User enables Chrome "View > Developer > Allow JavaScript from Apple Events" (Safari: Develop menu) plus a one-time macOS Automation consent. Uses the user's real logged-in browser. | Can read document.documentElement.outerHTML, query, click and fill. No event subscription, limited cross-frame detail. |
| Chrome remote debugging opt-in toggle (our survey noted Chrome 144+; version not verified on a real machine) | No extension, no separate profile. User enables it at chrome://inspect and confirms a per-connect dialog. Chrome 136+ blocks remote debugging on the default profile otherwise. |
Full CDP: DOM, a11y tree, network. |
Accessibility API web a11y tree (existing ui_snapshot / ui_action) |
Zero install. Safari exposes the web AX tree by default; Chrome needs renderer accessibility enabled. | Semantic tree, not raw DOM (no CSS selectors, no hidden nodes). |
| Embedded WKWebView in our own window | No intrusion on the user's browser. | Full DOM control, but no user cookies or logins. |
| safaridriver / Firefox WebDriver BiDi | Standard WebDriver. Needs "Allow remote automation" and a separate automation window. Intrusion close to Playwright. | Standard WebDriver / BiDi capabilities. |
| Chrome extension + native messaging host (Muse approach: Apple Events on Chrome plus an extension relaying over a Unix socket with an auth secret) | Works with the real profile, but needs an extension install and a Web Store listing. | Full in-page DOM access through the extension. |
| Playwright (current) | Isolated headed Chromium with a persistent profile. Heavy to install or bundle. | Full DOM, 32 tools today. |
Current leaning (not decided)
This is a leaning, not a decision.
A layered approach:
- Default tier (zero install): Apple Events JS plus the Accessibility API.
- Advanced tier (opt-in): Chrome remote debugging.
- Optional: the extension and Playwright.
Security notes to keep:
- Owner profile only (the
observeprofile sees none of this). - Playwright stays loopback-only, with no tailnet mapping.
- No secrets or tokens in logs.
Open questions / next steps
- Verify the Chrome remote-debugging toggle behaviour on a real Chrome, and find its minimum version.
- Verify Apple Events JS on current Chrome and Safari, and check what the macOS consent prompts look like.
- Decide how
browser_*tool names map onto the lighter tiers, i.e. which of the 32 Playwright tools can be supported. - A design draft already exists at
.agents/tasks/browser-tools-install/(extension-first, under review). It would be revised based on the outcome here.
Activity
chaodu-agent commentedon Nov 14, 2026
Decision update
Owner decision: we will not build a Chrome extension. The "extension + native messaging host" option is out.
Decision council result: option 1, the layered approach, with these amendments:
browser_*tool sits behind a single internal backend interface.tools/listshows only the tools the active tier supports.ui_snapshot/ui_action. A wizard page walks the user through the "Allow JavaScript from Apple Events" toggle and the macOS Automation consent, then runs a live probe.127.0.0.1:8794, behind--upstream, no tailnet mapping).Rationale: no Web Store listing, no publisher account, no bundled Node or Chromium. It matches the zero-install goal. It is reversible and medium risk.
Next steps
browser_*tool names the Apple Events tier can support.The earlier extension-first design draft in
.agents/tasks/browser-tools-install/is superseded.chaodu-agent commentedon Nov 14, 2026
Spike results and second council decision
Spike results
Source: the browser-tools-install spike report.
Chrome remote debugging (CDP)
chrome://inspecttoggle works on the default profile from Chrome 144.--remote-debugging-porton the default profile.Apple Events JavaScript
alert()blocks execution.Playwright
Live testing was skipped because it needs human consent dialogs.
Still unverified
NSAppleEventsUsageDescriptionis needed in Info.plist.Second council decision (option 4)
Sol and Opus agreed:
browser_*backend. Its consent is collected once in the install wizard.browser_*tools sit behind one internal backend interface.tools/listshows only what the active tier supports.Risk: medium. Reversible.
Next
A design workflow covering:
chaodu-agent commentedon Nov 14, 2026
Clarification: on-demand Playwright install in the wizard
The wizard gets an OPTIONAL step to install Playwright on demand.
Nothing is bundled. The .app contains no Node or Chromium. Nothing is downloaded and no files are left on disk unless the user presses the install button.
If the user presses it, the plan is:
~/Library/Application Support/oab-computer/, never inside the .app.127.0.0.1:8794only, and connect it through--upstreamtobrowser_*.This is a design direction, not final.
Open points handed to the design workflow:
The Apple Events default tier needs no download.
Unattended use by tier:
chaodu-agent commentedon Nov 14, 2026
Recording owner decisions:
Note: the design draft in
.agents/tasks/browser-tiers-design/assumed nothing is downloaded unless chosen, and Apple Events first. It will be revised.docs: browser_* tiers ADR and requirements (#118) (#120)