Repository navigation
feat(standard): run a desktop benchmark with an agent of the Agents page, through a computer port and its kit - #188
Open
pierreleca wants to merge 49 commits into
Open
pierreleca wants to merge 49 commits into
pierreleca wants to merge 49 commits into
Conversation
Pass participant kinds and agent setup limits to drivers that request them. Notify opted-in drivers once each agent unit ends, including failed and stopped units. Apply agent-only benchmark options in runs, recipes and the builder. Keep leak checks on the control setup contract and document the hooks.
Report benchmark-specific refusals per port and continue checking the remaining candidates. Save the proofs collected by the command while preserving existing records for refused kits. Keep explicit port refusals and command-wide failures fatal. Distinguish benchmark prerequisites and installs from agent installs, and validate provider credentials before checking kits.
…loop A desktop benchmark's runner serves one MCP server, computer, whose tools are the desktop's basic operations; the agent is a client alone. The benchmark's loop runs as it is and is handed a Bridge: each predict gives it the actions of the agent's next acting call, and the reading calls are served before it, one desktop operation at a time, in the order they came. An acting call is answered once its actions and their pause have run. - computer_port: the Desktop protocol (python, upload, read, capture, password), the Bridge (the step budget, the refusal of every call past it, the turn's end), the Server over a Unix socket (every connection of the session, one call log line per tool call: tool, times, failure), files sent once for a run by their sha256, and the runner's ready file, lifeline and one stop. - computer_pipe: the session's stdio pipe to the socket. A read of the input is counted under the lock the exit decision takes; the input's end is seen by its writer's hang-up while a full queue holds the reader; the pipe's end has one deadline from the input's end. - Tests on a fake desktop with a scripted MCP client, a stand-in loop that counts steps as the upstream's does, and the upstream's own run_single_example where a checkout of it is at hand.
A page for agent.computer/v1: its fifteen tools, what a step is, the step budget, the turn's end, uploads by content, the call log, and what a desktop benchmark's runner gives the shared code. The standard's row of the port links to it, and the docs folder map names ports/.
…dicts The computer port's conformance kit needs a small real desktop on which an agent shows that it sees the screen, points and types, decided by a program. This adds the desktop and the fixture the agent acts on; the kit's driver, runner and standard.json come with the port's server. - desktop.Dockerfile: an environment of the kit on the base image, tagged nullius/env-agent-computer-v1-desktop:<hash> as container.env_image tags any environment. Xvfb at 1920 x 1080 on :0, Openbox, a session bus for uid 1000, Python 3 with Tk, pyautogui and Pillow pinned by version and hash (desktop-requirements.txt). Two users: user (uid 1000, no sudo) and fixture. - session.py: the container's command; the display, its cookie, the bus and the window manager, then a ready file. - desktop.py: start a desktop (no network, no new privileges), the five operations a benchmark's side gives the port's server (python, upload, read, capture, password), one action run as user with its start and end read on the desktop's clock by a root process, the fixture's start and log, and removal. - fixture.py: a Tk program that draws codes and labels as bitmaps, with Tk's send removed, and logs each press, release, scroll and key with its time and what it hit, in a folder only fixture reads. - checks.py: the values each check draws, and the verdicts of see, click and type from the fixture's log, the actions' times and the last message; click and type fail on any input outside an action's time. - pyproject.toml: the kits' Dockerfiles and requirement files go in the package. Tests: the verdicts on hand-written logs, the action's times taken from the desktop's reply, the image's declaration; the docker tier starts real desktops and judges a click and a word sent by actions right, and the same click sent through python wrong.
A run stopped after a unit passed its stop check, and before its setup request was written, still sent that unit's end to the driver. A unit now counts as set up once its setup line is written. A request cut short by a stop leaves its late reply in the driver's queue, where a forced end would read it as its own. Such a driver is no longer sent an end. Document that a driver closed after a request got no reply, or one left with an unanswered request, is not sent an end. Test the forced wait after a stop, and replace a test whose mock passed for the wrong reason.
The setup time limit was turned into a float, so the live feed recorded the env_setup phase's timeout as 600.0 for every run, where it had recorded 600. The limit is now kept as given. Test the env_setup phase's timeout in the live feed for an agent and a control, with and without limits.setup_seconds.
An option whose setups named agent beside another scope passed the check, though the builder never showed it in a setup's tab. One on a capability benchmark passed too, where no agent ever runs it. The check now accepts agent only on its own, and only on a benchmark that agents run.
Conform without a port named went on past a kit refused for what its benchmark needs of the machine, and then exited 0 as if every port had been proven. When every kit was refused it still saved the definition and exited 0. A refused kit is now returned as refused beside the proofs gained, which are written, and the command exits 1. When every kit is refused, nothing is written and the check is refused, with exit status 2. The key is checked before the first kit only when some port has a kit: a command that names only ports without one says so and needs no key. Document the refusals of conform and its exit statuses.
…ny client - Each connection's answers are written by a thread of its own, at most OUT_BYTES waiting: a client that stops reading holds up neither the loop nor another connection, and a close never waits for a write. A connection accepted as the server closes is closed, and a closed one leaves the server's list. - One table gives what tools/list says and what a call may hold. A call is checked in full, an upload's base64 decoded, before it is queued; the JSON-RPC envelope is checked before any tool, a frame's length with its newline, and a number past any integer is refused and recorded. An answer over the bound is recorded as failed. - The turn's end stops the actions of a step not yet run and refuses that call; a done or fail taken before it still reaches the loop. - Uploads: an append goes through a name of the server's own, removed on every path; a file taken by its sha256 starts the file, and append is refused with it; the chunks being kept are the unit's own, dropped when the file starts again and at the unit's end, up to KEEP_BYTES; a kept file is written through an exclusive temporary file and read only when it is no link. - The last resort gives its own callback LAST_SECONDS before it exits. - The budget test holds the first action behind an event in place of a 250 ms wait.
Once the server's side ends, the pipe reads its input on to its end and sends it, by its one deadline; it exits 0 only when it saw that end and sent everything, and 1 otherwise. The queue test waits for the first send and measures bytes, assuming no read's size, and runs again with reads of 4 KiB.
…nd bounds The page says that a call is checked against its listed schema before it waits, what the turn's end does to a step not yet run and to a done or fail taken before it, that a session may connect again, the bounds on a connection's waiting answers and on a kept file, what an error is, and that done and fail answer before the loop runs them.
… cookie and no key repeat The desktop's X server ran as root with access control off, and a key held from one action to the next repeated between them, which the verdicts count as input outside every action. - The server runs as `display`, an account of its own, with access control on: root, user and fixture connect with the cookie the session writes. - The core key repeat is turned off, and the server's repeat delay is the longest XKB allows, which also covers the input-test device. - The session ends, and the container with it, when the display, the session bus or the window manager ends, not the display alone. - The image adds the accessibility bus (at-spi2-core), which the session bus starts on demand for programs that look for it.
…il loudly A pass must come from input sent inside actions, and a correct agent must not be failed by the desktop's own timing. - An action answers only once the verdicts' slack has passed on the desktop, so a request sent after it cannot fall inside its time. - The action's times and the fixture's events are read on one monotonic clock: a step of the wall clock no longer fails a correct action. - click fails on a press of any mouse button on another button; Enter in the field is input whose time is checked, as keys are. - The fixture says ready only when its window is mapped at the screen's size and holds the focus within its deadline. - read gives a regular file alone, at most 12 MiB, and refuses the rest as the agent's error. - A start that fails or is interrupted removes the container it made, its log read or not; remove checks the engine's answer and that the container is gone; a fixture log that cannot be read, or holds a line that is no event, is an error. - The docker tier's desktop tests run wherever an engine answers and build the image first; the removal test's probe is bounded and must succeed.
…hers When conform passed over a refused kit and wrote the other kits' proofs, it dropped every record of a known port that was not in the offers. A port proven earlier whose offer had been left out lost its proof although its kit never ran. The record of a port whose kit was refused in the same command now stays as it was.
… ends The driver.unit_end row said that a driver closed after a request got no reply has its process group ended with it. A child that redirects its streams can outlive that. The row keeps saying that such a driver is not sent an end.
…red-request test The test stopped the run from a timer started before setup was sent; on a slow machine the stop could come before the line was written and fail the test for the wrong reason. The stop now comes from the request's on_sent call, and the test waits for the driver to read all it was sent instead of sleeping.
For a unit an agent runs (driver.participant), setup starts runner.py behind the start gate. The runner builds OSWorld's DesktopEnv as the null control does, serves the computer port's MCP server over a Desktop of five methods on the upstream's controllers, and calls the upstream's own run_single_example, unmodified, with the port's bridge as the loop's agent. Setup answers once the loop asks for the agent's first step, with the prompt and the session's server computer: the port's pipe, whose path the driver takes from the package it imports, joined to the runner's socket. Score sends the turn's end on the runner's lifeline, waits for the runner within its bound and reads the upstream's result.txt; end (driver.unit_end) reaps the runner. - The bounds follow from the unit's limits (limits.setup_seconds, score's and end's); the runner's clean-up, now the bridge, the server and the desktop's bounded close, fits inside the driver's reap. A reap the driver's end cuts is handed to a reaper of its own. - The options Steps (100) and Pause (3.0 s) apply to an agent's run alone; driver.command names runner.py for a host's copy of the entry. - A unit with no agent runs as before.
…erywhere - The last resort's word goes to stderr from a thread it waits for 0.1 s at most: an exit never waits on a pipe nobody reads. - Every outgoing frame is bounded, the answers of any method and the errors standing in for them, with no id when even that is over. - NaN, Infinity and -Infinity are no JSON. Each tool's schema lists every argument it takes (additionalProperties false), and any other argument is refused. - The listed patterns use explicit classes, which Python and JSON Schema's regular expressions read alike; a final $ is the text's end. - upload's data is base64 by its pattern, and its listed bound is the check's: at the longest length the padding says the chunk is not over. - A failed append removes its part as far as the desktop answers, and a kept file's temporary file and descriptor go on every path. - The tests close the wires they make, and the queue test's helper ends the pipe before it closes an input a feeder may still write.
The computer port had a server and the kit a desktop; this joins them so that an agent can prove agent.computer/v1 like any other port. - The kit agent-computer-v1: eight checks, four basic (see, click, type, limit) and four skills (drag, scroll, recover, batch), each decided by the fixture's log, the actions' times and the server's call log. Every check fails on input that came outside an acting call. - driver.py: each unit's desktop and fixture, the port's server over it in the driver's process, and a loop of the kit's own that runs each step's actions and waits the pause. score ends the turn, judges and removes the desktop and the socket. A unit that never reaches score is let go by the next setup or the driver's end; reaper.py, started in a session of its own, removes what a killed driver left. - The reference makes the right calls through the unit's own server; the null control passes nothing. The kit's proof is recorded in its standard.json. - ports.py names the kit; the port's page gets its section, the ports table its row, and the generated agent pages the port's column. - fake_computer_agent.py: a scripted ACP agent that looks and clicks once, for the tests and a scripted conformance.
…ake desktop's limit plainly
A runner's snapshot for a remote host takes a benchmark's server command only from files of its own folder, and refused the kit's path to the port's pipe two folders up. pipe.py in the kit's folder runs that pipe. The kit's proof is recorded again on this tree, and the recipe test now expects the computer kit among the kits an ACP agent takes.
- A unit's VMs are removed in a thread the driver watches; when its input ends meanwhile, a reaper in a session of its own takes the removal and the socket's folder. - After a kill, and always in the reaper, the unit's containers are looked for again 2, 8 and 20 s on within the removal's bound, as the setups' reaper does, for a create the engine had accepted. - A runner stays in the driver's list until its reaper has started; when none can start, the driver kills its group and removes its VMs within 2 s. - The start gate reads the provider's start line from this launch's part of the runner's log alone. - The gate test's stand-in writes its ready file beside and renames it. - The definition says both ways a task is run, tells the agent's turn apart from setup's and scoring's time, and states the known limits of the agent side; the page is made again.
… a stop A pass must come from the desktop's records, the kit's reference must go the way an agent's session goes, and a stop, a kill or a hang must leave no container, folder or process. - limit: every acting call sent once the steps are taken is counted, one whose arguments do not hold included, and a call is counted and its step taken in one hold of the bridge's lock. - batch: each change of a field's text is recorded, however it came. - recover: the error goes to the first acting call the loop takes, whatever it is, before any part of it runs; only a call sent after the error's answer can earn the check. - An action the desktop did not end in time answers its call with an error; the next call goes on. - The reference reads each code from the screenshot it gets, with a PNG reader of the standard library, and fails when a call does not answer as a correct agent's would. - The engine's clients still running are ended and waited for when the driver lets its units go. The socket folder is named to the reaper before it is made, and the reaper is told when a desktop's start has ended: a container whose start had not ended is looked for again, up to a bound, until it is there and removed. - The killed-driver test reads with a bound and waits for its own reaper under a label of its own; a test runs a session of the scripted ACP agent through the pipe. - The kit's proof is recorded again on this tree.
Admit acting calls under the bridge lock and keep their arrival time through validation, so delayed checks cannot change budget or recovery evidence. Reject the first recovery call before terminal tools can end the task. Drain engine clients even after scoring and serialize their launch with shutdown. Keep unsuccessful desktop starts pending so the reaper can remove containers created after a timeout. Derive Docker test deadlines from supported operation limits and require successful click and prompt replies. Add deterministic regressions for admission, recovery, shutdown, late container creation and the test harness.
…es in the unit's record A failed check could not be read once its desktop was gone: the unit's record kept the verdict's counts, not what the fixture saw. At scoring the driver now writes the fixture's events (computer-events.jsonl) and where its targets were drawn (computer-boxes.json) beside the call log, in the unit's verifier folder.
Write the event and box records through new files in the verifier folder before replacing their destination entries. Ignore only operating-system errors from these optional copies while retaining scoring and cleanup errors. Cover write and close failures and replacement of symlinks and FIFOs with bounded scoring tests.
Describe the screen returned after each action and pause. Name screenshot pixels and ask for a screenshot before the first pointing action. State that the final tools answer before their actions and return no image.
Remove H from codes and exclude the previous character at each draw. Keep twelve characters and every glyph used by labels. Check many draws and read drawn codes from pixels at both scales. Describe the focused typing field and the conditions checked at the step budget.
Draw distinct common nouns for the see and scroll checks to make screen reading reliable. Keep the glyph font and add word gaps, with larger words at the end of the scroll list. Accept the displayed words in order across case and punctuation while rejecting missing or additional vocabulary words. Read the reference answer from screenshot pixels and keep the prompts independent of the sampled words.
Accept the drawn words when they occur consecutively among at most twelve vocabulary occurrences in the final message. Allow repeated answers and other vocabulary words before or after the drawn sequence. Replace screen, desktop and action terms with everyday nouns while preserving the spelling and fixture label constraints. Update the verdict descriptions and cover both reading checks, the occurrence limit and the reference reader's vocabulary.
…ched launch The test of a launch that meets a shutdown replaces subprocess.Popen for the whole process. On a machine with Docker the engine's environment is read through subprocess.run, which then met the stand-in and failed before the launch under test. The test now gives the launch a fixed environment.
pierreleca
marked this pull request as draft
November 17, 2026 12:47
Record the kit v1 computer port proof for Codex 2.2.2. All four basic checks and four skills passed all three runs. Preserve the run metadata and the counts with the version's other proofs. Regenerate the supported agent pages to show the proven port and OSWorld1 compatibility.
pierreleca
marked this pull request as ready for review
November 17, 2026 13:30
This was referenced Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #187
What
An agent of the Agents page could not run OSWorld: the entry asked for
agent.computer/v1, a port with no page, no server and no kit. This adds the three.computerof fifteen tools. Five read (info,screenshot,read,python,upload), eight act (click,move,drag,scroll,type,key,wait,step) and two end the task (done,fail). One acting call is one step, counted by the benchmark's own loop.see,click,type,limit) and four skills (drag,scroll,recover,batch), with a null control and a reference.driver.participant,driver.unit_end,limits.setup_seconds, an option for agents alone), andconformgoing on past a kit that cannot run on the machine.The pages have the detail:
docs/ports/agent-computer-v1.md,docs/standard.mdand the OSWorld entry's page.Read first: the port's page, then
computer_port.py(the server and its bridge to a loop), the kit'schecks.pyanddriver.py, then OSWorld'srunner.pyand the new part of itsdriver.py. More than half of the diff is tests.Two limits: OSWorld's agent side ran on the local backend alone, and the port bounds an agent's steps, not its model calls.
Touches
backend and the standard · templates and static · docs and site · catalogue
Before and after
OSWorld's Agent step in the builder, with an agent that holds the port chosen: the options Steps and Pause. Each picture opens at full size.
Verified
prove), on a machine with Docker: the null control passes 0 of 8 checks and the reference 8 of 8. The kit's tests that need Docker pass there (100 tests). The words ofseeandscrollare too short for the leak check to look for; a test checks that the task's text holds none of them.codex2.2.2 withgpt-6-lunapasses the four basic checks and shows the four skills, each in three runs of three (conform --repeat 3, the container backend).done, afail, a used-up budget and a stop each left no VM, container or process. Two units at a time kept their desktops apart.main.mainin the sandbox used here.Risk
A run with no agent of the Agents page takes the paths it took: the new hooks act only for a driver that asks for them (the comparison above). The kit needs Docker; on a machine without it
conformreports that kit as refused and goes on. The port's page is a contract: agents built against it would break if it changed. Undo with a revert.