Skip to content

feat(standard): run a desktop benchmark with an agent of the Agents page, through a computer port and its kit - #188

Open
pierreleca wants to merge 49 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/agent-computer
Open

pierreleca wants to merge 49 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/agent-computer

Conversation

@pierreleca

@pierreleca pierreleca commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #187

What

An agent of the Agents page could not run OSWorld: the entry asked for agent.computer/v1, a port with no page, no server and no kit. This adds the three.

  • The port: the benchmark's runner serves the desktop to the agent as an MCP server computer of fifteen tools. Five read (info, screenshot, read, python, upload), eight act (click, move, drag, scroll, type, key, wait, step) and two end the task (done, fail). One acting call is one step, counted by the benchmark's own loop.
  • The kit: a small desktop in a container and eight checks: four basic (see, click, type, limit) and four skills (drag, scroll, recover, batch), with a null control and a reference.
  • OSWorld's agent side: an agent that holds the port runs OSWorld's tasks through the upstream's own loop and evaluator, unmodified, with the options Steps and Pause.
  • The core: what a driver needs for this (driver.participant, driver.unit_end, limits.setup_seconds, an option for agents alone), and conform going on past a kit that cannot run on the machine.

The pages have the detail: docs/ports/agent-computer-v1.md, docs/standard.md and the OSWorld entry's page.

Read first: the port's page, then computer_port.py (the server and its bridge to a loop), the kit's checks.py and driver.py, then OSWorld's runner.py and the new part of its driver.py. More than half of the diff is tests.

Two limits: OSWorld's agent side ran on the local backend alone, and the port bounds an agent's steps, not its model calls.

Touches

backend and the standard · templates and static · docs and site · catalogue

Before and after

OSWorld's Agent step in the builder, with an agent that holds the port chosen: the options Steps and Pause. Each picture opens at full size.

Before After
The Agent step on main The Agent step with Steps and Pause

Verified

  • The kit's own proof (prove), on a machine with Docker: the null control passes 0 of 8 checks and the reference 8 of 8. The kit's tests that need Docker pass there (100 tests). The words of see and scroll are too short for the leak check to look for; a test checks that the task's text holds none of them.
  • A real agent on the kit: codex 2.2.2 with gpt-6-luna passes the four basic checks and shows the four skills, each in three runs of three (conform --repeat 3, the container backend).
  • OSWorld on the upstream's real VM, through the port, with a scripted agent: every tool answered as the page says, and the steps as the port counts them equal the upstream's own trace. A turn with no done, a fail, a used-up budget and a stop each left no VM, container or process. Two units at a time kept their desktops apart.
  • Unchanged: OSWorld with its own agents, and the null control, give the same records as on main.
  • Tests: the backend and route suites, each module in its own process, pass but for two modules that fail the same way on main in the sandbox used here.
  • Not verified: the container backend for OSWorld's agent side, and OSWorld's tasks run by a model-driven agent of the Agents page.

Risk

A run with no agent of the Agents page takes the paths it took: the new hooks act only for a driver that asks for them (the comparison above). The kit needs Docker; on a machine without it conform reports that kit as refused and goes on. The port's page is a contract: agents built against it would break if it changed. Undo with a revert.

Pass participant kinds and agent setup limits to drivers that request them. Notify opted-in drivers once each agent unit ends, including failed and stopped units.

Apply agent-only benchmark options in runs, recipes and the builder. Keep leak checks on the control setup contract and document the hooks.
Report benchmark-specific refusals per port and continue checking the remaining candidates. Save the proofs collected by the command while preserving existing records for refused kits.

Keep explicit port refusals and command-wide failures fatal. Distinguish benchmark prerequisites and installs from agent installs, and validate provider credentials before checking kits.
…loop

A desktop benchmark's runner serves one MCP server, computer, whose tools
are the desktop's basic operations; the agent is a client alone. The
benchmark's loop runs as it is and is handed a Bridge: each predict gives
it the actions of the agent's next acting call, and the reading calls are
served before it, one desktop operation at a time, in the order they came.
An acting call is answered once its actions and their pause have run.

- computer_port: the Desktop protocol (python, upload, read, capture,
  password), the Bridge (the step budget, the refusal of every call past
  it, the turn's end), the Server over a Unix socket (every connection of
  the session, one call log line per tool call: tool, times, failure),
  files sent once for a run by their sha256, and the runner's ready file,
  lifeline and one stop.
- computer_pipe: the session's stdio pipe to the socket. A read of the
  input is counted under the lock the exit decision takes; the input's
  end is seen by its writer's hang-up while a full queue holds the reader;
  the pipe's end has one deadline from the input's end.
- Tests on a fake desktop with a scripted MCP client, a stand-in loop that
  counts steps as the upstream's does, and the upstream's own
  run_single_example where a checkout of it is at hand.
A page for agent.computer/v1: its fifteen tools, what a step is, the step
budget, the turn's end, uploads by content, the call log, and what a
desktop benchmark's runner gives the shared code. The standard's row of
the port links to it, and the docs folder map names ports/.
…dicts

The computer port's conformance kit needs a small real desktop on which an
agent shows that it sees the screen, points and types, decided by a program.
This adds the desktop and the fixture the agent acts on; the kit's driver,
runner and standard.json come with the port's server.

- desktop.Dockerfile: an environment of the kit on the base image, tagged
  nullius/env-agent-computer-v1-desktop:<hash> as container.env_image tags
  any environment. Xvfb at 1920 x 1080 on :0, Openbox, a session bus for
  uid 1000, Python 3 with Tk, pyautogui and Pillow pinned by version and
  hash (desktop-requirements.txt). Two users: user (uid 1000, no sudo) and
  fixture.
- session.py: the container's command; the display, its cookie, the bus and
  the window manager, then a ready file.
- desktop.py: start a desktop (no network, no new privileges), the five
  operations a benchmark's side gives the port's server (python, upload,
  read, capture, password), one action run as user with its start and end
  read on the desktop's clock by a root process, the fixture's start and log,
  and removal.
- fixture.py: a Tk program that draws codes and labels as bitmaps, with
  Tk's send removed, and logs each press, release, scroll and key with its
  time and what it hit, in a folder only fixture reads.
- checks.py: the values each check draws, and the verdicts of see, click and
  type from the fixture's log, the actions' times and the last message;
  click and type fail on any input outside an action's time.
- pyproject.toml: the kits' Dockerfiles and requirement files go in the
  package.

Tests: the verdicts on hand-written logs, the action's times taken from the
desktop's reply, the image's declaration; the docker tier starts real
desktops and judges a click and a word sent by actions right, and the same
click sent through python wrong.
A run stopped after a unit passed its stop check, and before its setup request was written, still sent that unit's end to the driver. A unit now counts as set up once its setup line is written.

A request cut short by a stop leaves its late reply in the driver's queue, where a forced end would read it as its own. Such a driver is no longer sent an end.

Document that a driver closed after a request got no reply, or one left with an unanswered request, is not sent an end. Test the forced wait after a stop, and replace a test whose mock passed for the wrong reason.
The setup time limit was turned into a float, so the live feed recorded the env_setup phase's timeout as 600.0 for every run, where it had recorded 600. The limit is now kept as given.

Test the env_setup phase's timeout in the live feed for an agent and a control, with and without limits.setup_seconds.
An option whose setups named agent beside another scope passed the check, though the builder never showed it in a setup's tab. One on a capability benchmark passed too, where no agent ever runs it.

The check now accepts agent only on its own, and only on a benchmark that agents run.
Conform without a port named went on past a kit refused for what its benchmark needs of the machine, and then exited 0 as if every port had been proven. When every kit was refused it still saved the definition and exited 0.

A refused kit is now returned as refused beside the proofs gained, which are written, and the command exits 1. When every kit is refused, nothing is written and the check is refused, with exit status 2.

The key is checked before the first kit only when some port has a kit: a command that names only ports without one says so and needs no key.

Document the refusals of conform and its exit statuses.
…ny client

- Each connection's answers are written by a thread of its own, at most
  OUT_BYTES waiting: a client that stops reading holds up neither the
  loop nor another connection, and a close never waits for a write.
  A connection accepted as the server closes is closed, and a closed one
  leaves the server's list.
- One table gives what tools/list says and what a call may hold. A call
  is checked in full, an upload's base64 decoded, before it is queued;
  the JSON-RPC envelope is checked before any tool, a frame's length
  with its newline, and a number past any integer is refused and
  recorded. An answer over the bound is recorded as failed.
- The turn's end stops the actions of a step not yet run and refuses
  that call; a done or fail taken before it still reaches the loop.
- Uploads: an append goes through a name of the server's own, removed
  on every path; a file taken by its sha256 starts the file, and append
  is refused with it; the chunks being kept are the unit's own, dropped
  when the file starts again and at the unit's end, up to KEEP_BYTES; a
  kept file is written through an exclusive temporary file and read only
  when it is no link.
- The last resort gives its own callback LAST_SECONDS before it exits.
- The budget test holds the first action behind an event in place of a
  250 ms wait.
Once the server's side ends, the pipe reads its input on to its end and
sends it, by its one deadline; it exits 0 only when it saw that end and
sent everything, and 1 otherwise. The queue test waits for the first
send and measures bytes, assuming no read's size, and runs again with
reads of 4 KiB.
…nd bounds

The page says that a call is checked against its listed schema before it
waits, what the turn's end does to a step not yet run and to a done or
fail taken before it, that a session may connect again, the bounds on
a connection's waiting answers and on a kept file, what an error is, and
that done and fail answer before the loop runs them.
… cookie and no key repeat

The desktop's X server ran as root with access control off, and a key held
from one action to the next repeated between them, which the verdicts count
as input outside every action.

- The server runs as `display`, an account of its own, with access control
  on: root, user and fixture connect with the cookie the session writes.
- The core key repeat is turned off, and the server's repeat delay is the
  longest XKB allows, which also covers the input-test device.
- The session ends, and the container with it, when the display, the session
  bus or the window manager ends, not the display alone.
- The image adds the accessibility bus (at-spi2-core), which the session bus
  starts on demand for programs that look for it.
…il loudly

A pass must come from input sent inside actions, and a correct agent must not
be failed by the desktop's own timing.

- An action answers only once the verdicts' slack has passed on the desktop,
  so a request sent after it cannot fall inside its time.
- The action's times and the fixture's events are read on one monotonic
  clock: a step of the wall clock no longer fails a correct action.
- click fails on a press of any mouse button on another button; Enter in the
  field is input whose time is checked, as keys are.
- The fixture says ready only when its window is mapped at the screen's size
  and holds the focus within its deadline.
- read gives a regular file alone, at most 12 MiB, and refuses the rest as
  the agent's error.
- A start that fails or is interrupted removes the container it made, its log
  read or not; remove checks the engine's answer and that the container is
  gone; a fixture log that cannot be read, or holds a line that is no event,
  is an error.
- The docker tier's desktop tests run wherever an engine answers and build
  the image first; the removal test's probe is bounded and must succeed.
…hers

When conform passed over a refused kit and wrote the other kits' proofs, it dropped every record of a known port that was not in the offers. A port proven earlier whose offer had been left out lost its proof although its kit never ran.

The record of a port whose kit was refused in the same command now stays as it was.
… ends

The driver.unit_end row said that a driver closed after a request got no reply has its process group ended with it. A child that redirects its streams can outlive that. The row keeps saying that such a driver is not sent an end.
…red-request test

The test stopped the run from a timer started before setup was sent; on a slow machine the stop could come before the line was written and fail the test for the wrong reason. The stop now comes from the request's on_sent call, and the test waits for the driver to read all it was sent instead of sleeping.
For a unit an agent runs (driver.participant), setup starts runner.py
behind the start gate. The runner builds OSWorld's DesktopEnv as the null
control does, serves the computer port's MCP server over a Desktop of
five methods on the upstream's controllers, and calls the upstream's own
run_single_example, unmodified, with the port's bridge as the loop's
agent. Setup answers once the loop asks for the agent's first step, with
the prompt and the session's server computer: the port's pipe, whose path
the driver takes from the package it imports, joined to the runner's
socket. Score sends the turn's end on the runner's lifeline, waits for
the runner within its bound and reads the upstream's result.txt; end
(driver.unit_end) reaps the runner.

- The bounds follow from the unit's limits (limits.setup_seconds,
  score's and end's); the runner's clean-up, now the bridge, the server
  and the desktop's bounded close, fits inside the driver's reap. A reap
  the driver's end cuts is handed to a reaper of its own.
- The options Steps (100) and Pause (3.0 s) apply to an agent's run
  alone; driver.command names runner.py for a host's copy of the entry.
- A unit with no agent runs as before.
…erywhere

- The last resort's word goes to stderr from a thread it waits for 0.1 s
  at most: an exit never waits on a pipe nobody reads.
- Every outgoing frame is bounded, the answers of any method and the
  errors standing in for them, with no id when even that is over.
- NaN, Infinity and -Infinity are no JSON. Each tool's schema lists
  every argument it takes (additionalProperties false), and any other
  argument is refused.
- The listed patterns use explicit classes, which Python and JSON
  Schema's regular expressions read alike; a final $ is the text's end.
- upload's data is base64 by its pattern, and its listed bound is the
  check's: at the longest length the padding says the chunk is not over.
- A failed append removes its part as far as the desktop answers, and a
  kept file's temporary file and descriptor go on every path.
- The tests close the wires they make, and the queue test's helper ends
  the pipe before it closes an input a feeder may still write.
The computer port had a server and the kit a desktop; this joins them so
that an agent can prove agent.computer/v1 like any other port.

- The kit agent-computer-v1: eight checks, four basic (see, click, type,
  limit) and four skills (drag, scroll, recover, batch), each decided by
  the fixture's log, the actions' times and the server's call log. Every
  check fails on input that came outside an acting call.
- driver.py: each unit's desktop and fixture, the port's server over it in
  the driver's process, and a loop of the kit's own that runs each step's
  actions and waits the pause. score ends the turn, judges and removes the
  desktop and the socket. A unit that never reaches score is let go by the
  next setup or the driver's end; reaper.py, started in a session of its
  own, removes what a killed driver left.
- The reference makes the right calls through the unit's own server; the
  null control passes nothing. The kit's proof is recorded in its
  standard.json.
- ports.py names the kit; the port's page gets its section, the ports
  table its row, and the generated agent pages the port's column.
- fake_computer_agent.py: a scripted ACP agent that looks and clicks once,
  for the tests and a scripted conformance.
A runner's snapshot for a remote host takes a benchmark's server command
only from files of its own folder, and refused the kit's path to the port's
pipe two folders up. pipe.py in the kit's folder runs that pipe.

The kit's proof is recorded again on this tree, and the recipe test now
expects the computer kit among the kits an ACP agent takes.
- A unit's VMs are removed in a thread the driver watches; when its
  input ends meanwhile, a reaper in a session of its own takes the
  removal and the socket's folder.
- After a kill, and always in the reaper, the unit's containers are
  looked for again 2, 8 and 20 s on within the removal's bound, as the
  setups' reaper does, for a create the engine had accepted.
- A runner stays in the driver's list until its reaper has started;
  when none can start, the driver kills its group and removes its VMs
  within 2 s.
- The start gate reads the provider's start line from this launch's
  part of the runner's log alone.
- The gate test's stand-in writes its ready file beside and renames it.
- The definition says both ways a task is run, tells the agent's turn
  apart from setup's and scoring's time, and states the known limits of
  the agent side; the page is made again.
… a stop

A pass must come from the desktop's records, the kit's reference must go
the way an agent's session goes, and a stop, a kill or a hang must leave
no container, folder or process.

- limit: every acting call sent once the steps are taken is counted, one
  whose arguments do not hold included, and a call is counted and its step
  taken in one hold of the bridge's lock.
- batch: each change of a field's text is recorded, however it came.
- recover: the error goes to the first acting call the loop takes,
  whatever it is, before any part of it runs; only a call sent after the
  error's answer can earn the check.
- An action the desktop did not end in time answers its call with an
  error; the next call goes on.
- The reference reads each code from the screenshot it gets, with a PNG
  reader of the standard library, and fails when a call does not answer
  as a correct agent's would.
- The engine's clients still running are ended and waited for when the
  driver lets its units go. The socket folder is named to the reaper
  before it is made, and the reaper is told when a desktop's start has
  ended: a container whose start had not ended is looked for again, up to
  a bound, until it is there and removed.
- The killed-driver test reads with a bound and waits for its own
  reaper under a label of its own; a test runs a session of the scripted
  ACP agent through the pipe.
- The kit's proof is recorded again on this tree.
Admit acting calls under the bridge lock and keep their arrival time through validation, so delayed checks cannot change budget or recovery evidence. Reject the first recovery call before terminal tools can end the task.

Drain engine clients even after scoring and serialize their launch with shutdown. Keep unsuccessful desktop starts pending so the reaper can remove containers created after a timeout.

Derive Docker test deadlines from supported operation limits and require successful click and prompt replies. Add deterministic regressions for admission, recovery, shutdown, late container creation and the test harness.
…es in the unit's record

A failed check could not be read once its desktop was gone: the unit's
record kept the verdict's counts, not what the fixture saw. At scoring the
driver now writes the fixture's events (computer-events.jsonl) and where its
targets were drawn (computer-boxes.json) beside the call log, in the unit's
verifier folder.
Write the event and box records through new files in the verifier folder before replacing their destination entries. Ignore only operating-system errors from these optional copies while retaining scoring and cleanup errors.

Cover write and close failures and replacement of symlinks and FIFOs with bounded scoring tests.
Describe the screen returned after each action and pause. Name screenshot pixels and ask for a screenshot before the first pointing action.

State that the final tools answer before their actions and return no image.
Remove H from codes and exclude the previous character at each draw. Keep twelve characters and every glyph used by labels.

Check many draws and read drawn codes from pixels at both scales. Describe the focused typing field and the conditions checked at the step budget.
Draw distinct common nouns for the see and scroll checks to make screen
reading reliable. Keep the glyph font and add word gaps, with larger words
at the end of the scroll list.

Accept the displayed words in order across case and punctuation while
rejecting missing or additional vocabulary words. Read the reference answer
from screenshot pixels and keep the prompts independent of the sampled words.
Accept the drawn words when they occur consecutively among at most twelve
vocabulary occurrences in the final message. Allow repeated answers and
other vocabulary words before or after the drawn sequence.

Replace screen, desktop and action terms with everyday nouns while
preserving the spelling and fixture label constraints. Update the verdict
descriptions and cover both reading checks, the occurrence limit and the
reference reader's vocabulary.
…ched launch

The test of a launch that meets a shutdown replaces subprocess.Popen for the whole process. On a machine with Docker the engine's environment is read through subprocess.run, which then met the stand-in and failed before the launch under test. The test now gives the launch a fixed environment.
@pierreleca
pierreleca marked this pull request as draft November 17, 2026 12:47
Record the kit v1 computer port proof for Codex 2.2.2. All four basic
checks and four skills passed all three runs. Preserve the run metadata
and the counts with the version's other proofs.

Regenerate the supported agent pages to show the proven port and OSWorld1
compatibility.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: an agent of the Agents page cannot run a computer-use benchmark

1 participant