A verified computer-use agent, for the browser and for native macOS apps. Footwork puts a cheap, calibrated guard from TypeSafe Jev in front of whoever drives: every claimed completion is checked against observed state and the trajectory before it counts, every irreversible action passes a gate, and every action comes back with a receipt saying what visibly changed. On top of that guard, Jev can also take the fast, mechanical steps itself (System 1) while the driver (System 2) keeps the reading, comparing and deciding. The driver can be an LLM loop (browser-use 0.13.10 as a library, not a fork) or the operator at the keyboard: a person, or an agent such as a Claude Code session, one command at a time. Native apps are driven in the background through the Cua Driver's accessibility tree, beside a person still using the same Mac.
The Python package is jevdual (in jevdual/); Footwork is the project. Every number below is
from a run directory under jevdual/results/, and every experiment was pre-registered in
jevdual/docs/experiments/ before it ran.
Optional browser inspection utilities collect redacted, timestamped evidence through an existing tab/CDP adapter. They are dependency-free ESM helpers, separate from the Python driver and verification gates. Their contract is tested offline; live capture fidelity must be calibrated on the selected browser.
footwork start "Compute 48 times 13" --app Calculator --require "The display shows 624"
footwork look # the menu of what can be done now, with ids, and the last receipt
footwork s1 --act # System 1's proposal on this observation (one Jev call), dispatched
footwork do click 12 # through the gate; settle, reobserve, print the receipt
footwork done "624" # the verifier judges the claim against the window and the trailAlso do type | append | key | hotkey | menu | scroll | enter, status, and footwork run for the whole
loop: by default the measured guarded arm (an LLM decides every step behind the guard), or --arm dual
for Jev as System 1 with the LLM on escalation. Nothing dispatches on an id from an older
observation; text comes from the task or from named secrets ({NAME} read from $FOOTWORK_SECRET_NAME,
never printed); --authorize "<label>" names the one change the operator approved. Sessions default to
background_only: the person's front app, pointer and keyboard are left alone (--mode grants more).
start --browser drives footwork's own Chrome profile through the Driver's DevTools route instead of the
accessibility tree. Needs macOS with Accessibility granted to the process that runs it, and a TypeSafe key.
For private visual inspection, use a numeric ID from the current native observation:
footwork inspect-target 12 --operation click --postcondition "The selected editor enters crop mode"This records a fresh window PNG, target-bound manifest and static HTML preview under the session's
inspection/ directory. The preview outlines the offered target over the unchanged original image
and displays the operation and required postcondition. It makes no model calls, executes no action
and does not verify the stated effect.
Sessions that have handled secrets withhold screenshots before capture. Missing or unqualified
capture geometry, truncated observations and changed target identities refuse inspection.
Native AX projections may be partial: the manifest records that limit, and label-based inspection
requires a complete observation. Browser inspection through this command is not yet qualified.
footwork visual-check --request request.json --session SESSION prepares one native advisory
document/reference or crop/subject check. It records the original capture and a structured receipt.
Explicit --provider decisions --disclose enables one bounded provider request using an existing
key; the default contacts no provider. Visual answers grant no action or completion authority.
See visual check inputs, limits and escalation.
Measured this way on 2026-09-25 (docs/plans/legible-harness.md):
Calculator 48 × 13 in 30 s (System 1 proposed every step at p ≥ 0.94, verifier accept 0.98); TextEdit
append-and-save in 9 s, the save confirmed by the window title losing its edited mark.
Three arms are compared on the same tasks: stock browser-use with the LLM alone, guarded (the LLM behind Jev verification and the gate, no Jev actions), and dual (the guard plus Jev taking the mechanical steps). System 2 in every run is Muse Spark 1.3 Contributor over the Meta Model API, chosen because it is cheap and fast and therefore the hardest baseline for Jev to beat on cost.
Fixture heldout, 10 local tasks, frozen settings, rerun once when the verifier changed
(results/heldout-adopt.md):
| arm | pass | false completions | driver steps | Jev calls | est. cost USD | wall s |
|---|---|---|---|---|---|---|
| stock | 9/10 | 1 (deleted an account when told to reach the page) | 33 | 0 | 0.049 | 708 |
| dual | 10/10 | 0 (paused before the click) | 24 | 76 | 0.049 | 522 |
Live sites, 55 tasks on Wikipedia, MDN, docs.python.org, arXiv, GitHub, PyPI and three practice sites,
three same-code runs (results/q9-live-dev.md,
results/q9b-coherent.md):
| arm | pass | false completions | driver steps | Jev calls | est. cost USD | cost per verified pass |
|---|---|---|---|---|---|---|
| guarded | 50 to 52 / 55 | 0 | 226 to 260 | 80 to 84 | 0.385 to 0.412 | 0.0086 to 0.0100 |
| dual | 51 to 52 / 55 | 0 | 161 to 193 | 360 to 391 | 0.449 to 0.492 | 0.0100 to 0.0117 |
| stock (one run) | 51 / 55 | 1 (its own LLM judge accepted it) | 265 | 0 | 0.382 |
What that says, in order of how sure we are:
- The guard is the product. Zero false completions on the guarded arm on every split and run, on these
predicates, at about two Jev calls per task, and at the LLM's own cost. The guard is not infallible: in
the Q8 run the dual arm clicked an authorized Delete twice and the verifier accepted it
(
results/q8-consent.md, the one false completion). The LLM alone deletes the account when asked to reach a page, and its own judge marks that a success. - Jev acting saves driver work, not money yet. Dual makes 23 to 29% fewer driver requests than the guarded arm across two same-night samples and runs its own steps at a median 1.4 s against 15.7 s for an LLM step, but pays for that in Jev volume; task wall time between the arms is not distinguishable on live sites, and dual matches stock's cost only on the fixture heldout so far.
- System 2 delegating bounded subgoals to Jev did not pay in three measurements
(
Q9): more cost, no more completions. The executor reached 5 of 30 assignments; an audit then found software defects behind most of those failures, and the repaired build was measured again (results/q9e-delegate-fixed.md): 36 verified against 41 for the same-day guarded arm, $0.644 against $0.424, 3 of 27 assignments reached. Delegation stays available as a tool, off by default. - Declaring the contract to the driver did not pay (
Q11): model requests rose 21% per task with completions unchanged. Neither did receipts for every action (Q12,results/q12-live-dev.md): repeated no-op clicks fell 18% but requests per unit of progress did not move (−0.02 [−0.14, +0.10], over the 53 tasks with some progress in both arms), and verified passes fell by 4 pooled. Receipts are kept as an honesty measure, not as a saving; the input defect they exposed is fixed first.
- The arbiter's target confidence ranks wrong System 1 steps on accessibility-tree menus
(
Q10,results/q10-native-dev.md): AUROC 0.90 [0.85, 0.94] on Calculator and TextEdit, against 0.80 on the web. That is a ranking result on two apps, whose agreed wrong steps come from two tasks; it is not a calibrated probability, and it does not yet say anything about unfamiliar apps. After verifier and task fixes, dual is within one task of guarded (13 against 15 of 18,results/q10b-native-dev.md); the gap is one repeated-sequence task. Native stays experimental. - A session runs beside a person without taking their work, input or focus
(
Q15): 15 of 15 coexistence cases passed every check (another app, another document of the same process, a foreground step beside an idle or moving person, a conflicting edit). In none of three attempts did the person's move overlap a foreground step, so that race is covered by a unit test, not a live one.background_onlyis the default;foreground_permittedholds the front for 36 to 65 ms and only after 3 s of idle.
task ─────────────────────────────────────────────┐
▼
┌──────────── System 2: the LLM driver (browser-use loop) ────────────────┐
│ reads, compares, decides, composes; may call delegate_subgoal / │
│ find_evidence; proposes actions and eventually `done` │
└──────────────────────────────┬──────────────────────────────────────────┘
│ every action, either system
▼
┌──────────── the guard (code + Jev nouls, ~1 call per check) ────────────┐
│ destructive gate: pause before delete / pay / send unless the task │
│ authorises that action; done verification: `complete` + one │
│ `unmet` noul per requirement, judged on the page AND the trajectory, │
│ claims graded against evidence atoms, a per-run ledger by kind │
└──────────────────────────────┬──────────────────────────────────────────┘
│ optional: dual mode
▼
┌──────────── System 1: Jev (one speculative fan-out per step) ───────────┐
│ Choice: operation · Choice: target (conditional per operation) │
│ Nouls: goal_done · stuck · needs_reasoning · destructive · login · │
│ bot_check; a code-owned arbiter (arbiter.toml) acts, escalates, │
│ retries the next-best target, or asks for confirmation │
└─────────────────────────────────────────────────────────────────────────┘
- Verification (
verify.py,ledger.py): a done is accepted only when thecompletenoul and every requirement'sunmetnoul are inside the accept band; answers must quote evidence atoms found on the full page text; requirements that name an action are judged from the trajectory once the page has moved on; a requirement judged met stays met unless it is a current-state requirement and the page later contradicts it. - Gate (
agent.py): keyword and noul rules on the target, the URL, the focused element for Enter, andevaluate; scoped by the task'sauthorized_actions. - Arbiter (
arbiter.py,arbiter.toml): confidence floors, noul thresholds, streak and no-effect rules. Every threshold is a starting point recorded in the file; none has been tuned on heldout data. - Rust (
crates/jevdual-core): paint-order occlusion, snapshot lookup, element hashes and evidence matching as PyO3 ports with pure-Python twins and equality tests. Honest note: the DOM pipeline is not the bottleneck; Jev and the driver are.
evals/: armsstock,s1_only,dual,guarded,delegate,delegate_evidence; predicates on observed state (URL, page text, returned answer, URL checkpoints for multistep tasks), never on the agent's claim; browser-use's own LLM judge recorded per row as a comparison, never as the grade.- Task sets: a local fixture site (
dev,heldout), 55 live dev tasks and 24 live heldout tasks, and 42 judge-graded tasks imported from browser-use's agent tasks and Mind2Web. Heldout splits are never debugged. - Method: pre-registered questions with predictions and decision rules, paired bootstrap intervals over
tasks (
evals/paired.py), results written after every task run, resumable, secrets redacted. - browser-use's own CI modules run against this package with the patches active
(
scripts/upstream_tests.sh); one of them caught a real bug in the paint-order port.
git clone --recurse-submodules https://github.com/Tom-R-Main/Footwork.git
cd Footwork/jevdual
uv sync # builds the PyO3 crate; Rust 1.95 via rust-toolchain.toml
uv run pytest # unit and equality tests, no keys neededTo run the agent or the evals you need a TypeSafe key (TYPESAFE_API_KEY) and a System 2 key
(MODEL_API_KEY for the Meta Model API, or wire another provider in evals/runner.py):
uv run python -m evals.runner --split dev --arm stock --arm guarded --arm dual --llm meta
uv run python -m evals.runner --split live-dev --arm guarded --llm meta --task lw-python-creator
uv run python -m evals.paired results/<before> results/<after> --arm dualUse the guard alone with any browser-use agent by constructing DualProcessAgent with an ArbiterHook
verifier and no S1 policy; see evals/runner.py (guarded arm) for the exact wiring.
| path | what |
|---|---|
jevdual/python/jevdual/ |
the package: agent, policy, menu, arbiter, verify, ledger, receipts, secrets, tools; native arm (native, desktop, posture, coexist); operator surface (cli, session) |
jevdual/crates/jevdual-core/ |
Rust hot paths (PyO3) with pure twins under python/jevdual/_pure/ |
jevdual/evals/ |
runner, predicates, report, metrics, paired comparison, task files, fixture site |
jevdual/results/ |
every run directory and its write-up; the README's numbers come from here |
jevdual/docs/experiments/ |
pre-registered questions Q1 to Q15 with results and decisions |
jevdual/docs/ |
design notes, plans (plans/legible-harness.md is the current one), the Cua survey, the Rust boundary rule, upstream reuse inventory |
browser-use/ |
upstream pinned as a submodule at the studied commit; the package imports the PyPI release |
Research code with a working product core. The guard and the reactive dual loop are measured on the
browser; the native arm and the operator session are measured on a small native split and a coexistence
suite and stay experimental. Delegation and evidence selection are off by default. The
status audit of 2026-09-26 reproduced five contract
breaks offline (operator sessions following the active browser tab, field receipts that overstate, a secret
in a receipt, the browser path continuing when its destructive judgment fails, short documents replaced
unpaused). Open, in order (plan): those fixes, authorization parity
across routes, the input loss after navigation, then receipts again (Q12b), verifier evidence and recovery,
a narrower declaration (Q11b), and one contract across browser and native (Q13). Whether the browser should
go through the accessibility tree too (Q14) has a working route but no comparison yet. Calibration claims
wait on the human audit sheets. Read CONTRIBUTING.md before opening a change.
Ideas were taken from six MIT-licensed sibling projects (jev-ultrafast, fastbrowse, jev-browser,
jev-for-chrome, public-browser) and, ideas only, from browserclaw (AGPL); browser-use's test fixtures and
judge are reused under MIT (docs/upstream-reuse.md). MIT license.