Skip to content

feat(dialog): let agents accept, dismiss and fill JavaScript dialogs - #378

Closed
ArcueidMP wants to merge 18 commits into
Tencent:mainfrom
ArcueidMP:codex/issue-359-js-dialogs
Closed

ArcueidMP wants to merge 18 commits into
Tencent:mainfrom
ArcueidMP:codex/issue-359-js-dialogs

Conversation

@ArcueidMP

@ArcueidMP ArcueidMP commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Agents need to inspect, accept or dismiss native JavaScript dialogs and supply prompt input. This adds explicit dialog commands and integrates them with the default human-agent cooperation flow.

Fixes #359.

  • Add bsk dialog status, bsk dialog accept [text] and bsk dialog dismiss, with matching DSH actions. Omitted prompt text preserves the default; an empty string clears it.
  • Shared tabs leave all four dialog types pending. Accept/dismiss requires the observed dialog ID, without a DOM context or renderer read that a modal can block. Decisions respect human editing, hold, Stop and tab ownership, including a final check before the native command. Subsequent page actions need a fresh cooperation sync.
  • Outside shared control, confirm/prompt require a decision; alert/beforeunload retain automatic handling unless session start --no-auto-dialog is set. Dismissing beforeunload stays on the page.
  • Return dialog_pending promptly when a dialog blocks a tool, including lifecycle waits. Status and decisions can run while the session is busy, follow the task-bound tab, and preserve user cancellation. Track the actual native command until it settles, preserve cleanup without replaying input, and handle manual closure, consecutive dialogs and debugger detach.
  • Advertise protocol 1.5, reject unsupported opt-out requests on older components, and update schemas, skill guidance and regressions.

The pending response ends the original CLI wait, not execution inside Chrome. Handling the dialog may resume the already-sent native operation; it is never replayed. The original evaluation return value is not recovered. Callers inspect the page afterwards and wait while execution_pending is true.

The video regression waits for frame presentation after seeking before sampling pixels; its existing color assertions are unchanged.

Validation:

  • Rust formatting, workspace clippy with warnings denied, and all workspace tests.
  • Frontend lint, extension TypeScript compile, extension unit tests and build; DSH typecheck, tests, build and npm archive verification.
  • Localization and VOM tests; Node script tests, LF/CRLF skill budgets and Cargo skill package verification.
  • Nine isolated Chrome input/dialog regressions selected by the existing *click*.browser.test.ts CI glob.
  • Full CLI/daemon/extension/DSH cooperation regression in isolated Chrome for Testing, including nine new shared-dialog decisions covering alert, confirm, prompt text/default/empty input, and beforeunload stay/leave; old DOM contexts cannot replay the triggering action.
  • Native-dispatch regressions for a late human hold, stopped/editing states, task-tab binding and cancellation during a pending modal.
  • Video encoder/recovery and capture/export browser regressions. The headed preview test remains opt-in.

Resolve the SKILL.md reference-table conflict by listing both JS dialogs
and the new element waits for the interaction-details reference. The
entry point stays within its byte budget.
Keep both JavaScript dialog control and the new video tool: register both
DSH tools and schemas, mark both as control-plane methods, and gate dialog
decisions and video start on user interrupts.

Tighten the DSH skill entry point so the combined dialog and video guidance
stays within its 4,500-byte budget after Windows CRLF checkout.
Expose both wait-for-element and dialog on the DSH browser_page tool, keeping
each action's guidance in the tool description and both in the expected
action maps.
Keep JavaScript dialogs alongside structured extraction: register both DSH
tools and list both sets of error reasons. In the CLI skill, take main's
singular ref wording but keep the shorter reference-loading sentence so the
entry point stays within its 7,100-byte budget after Windows CRLF checkout.
The large-frame encoder check failed on a final-frame colour sample. The test only bundles video modules and type-only dialog changes; rerun to separate CI timing from a branch regression.
Adopt the DSH bsk_* tool names for the dialog action: tool maps, skill
entry point, interaction reference and session-resolution label now use
bsk_page/bsk_session. List dialog in the README's bsk_page actions, and
keep both the Added dialog entry and the Changed namespace entry.
Keep --no-auto-dialog alongside the new background-by-default focus
handling: the CLI sends both focused and no_auto_dialog, and the DSH start
tool maps noFocus to --focus/--no-focus and noAutoDialog to
--no-auto-dialog. Keep the dialog parse test before main's renamed
background session test.
The focus mapping cases from Tencent#447 still looked up browser_session after Tencent#446 renamed the tool, so they failed on main as well.
Main now assigns protocol 1.4 to incognito Agent Windows, so move dialog
control to protocol 1.5. A 1.4 extension supports incognito but not
dialog control, and must keep rejecting --no-auto-dialog and dialog
commands. Bump the daemon and extension protocol to 1.5, gate dialogs on
1.5, and update the tests and docs that named the dialog protocol.

Keep both session start options (no_auto_dialog and incognito) through the
CLI, daemon, schema and extension, with both capability checks. Take main's
version-agnostic handshake tests and derive the "newer" protocol in the
connection controller tests from PROTOCOL_VERSION.
The capture regression sampled preview frames as soon as seeking finished, the same race the encoder regression already waits out. Wait for frame presentation too, bounded so a seek that stays on the displayed frame cannot hang the test.
Keep the dialog parse test beside the global output flag test from Tencent#411,
and list the Unreleased Added dialog entry before the Fixed CLI entry.
Parse errors pick their output format from raw argv. `dialog accept --text`
accepts hyphen-leading prompt text, so `--text --json` passes the literal
"--json" as text, yet a later parse failure was reported as JSON. Skip the
value after any option that allows hyphen values, read from the clap
definition.
Expose both noAutoDialog and the new incognito option on bsk_session start,
forwarding --no-auto-dialog and --incognito independently.
@ArcueidMP ArcueidMP closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Give agents control over JS dialogs (confirm/prompt/beforeunload): dismiss/text/status instead of forced auto-accept

1 participant