Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7e1a5d5
feat(standard): add opt-in driver lifecycle hooks
pierreleca Nov 16, 2026
34589a0
fix(standard): retain proofs when a candidate kit is refused
pierreleca Nov 16, 2026
7cc74c1
feat(standard): serve the computer port to a desktop benchmark's own …
pierreleca Nov 16, 2026
54027f1
docs(standard): the computer port's tools and rules
pierreleca Nov 16, 2026
570cb40
feat(standard): add the computer kit's desktop, fixture and first ver…
pierreleca Nov 16, 2026
c99c850
fix(standard): send a unit's end only once its setup was sent
pierreleca Nov 16, 2026
27eaac2
fix(standard): keep a unit's setup time limit a whole number
pierreleca Nov 16, 2026
acc16c7
fix(standard): accept an agent-only option alone and on agent benchmarks
pierreleca Nov 16, 2026
281c678
fix(standard): exit conform with 1 when a kit was refused
pierreleca Nov 16, 2026
d2baa07
fix(standard): harden the computer port's server and bridge against a…
pierreleca Nov 16, 2026
6eabd5d
fix(standard): exit 0 from the computer pipe only after its input's end
pierreleca Nov 16, 2026
a406a60
docs(standard): state the computer port's checked calls, turn's end a…
pierreleca Nov 16, 2026
674fb52
docs(standard): plainer wording for the computer port's turn's end an…
pierreleca Nov 16, 2026
f031bd2
test(standard): hold the late connection open until the server closes it
pierreleca Nov 16, 2026
eeb23dc
fix(standard): run the computer kit's X server unprivileged, with its…
pierreleca Nov 16, 2026
e288af2
fix(standard): earn the computer kit's passes and make its desktop fa…
pierreleca Nov 16, 2026
17e8194
fix(standard): keep a refused port's proof when conform writes the ot…
pierreleca Nov 16, 2026
60d5881
docs(standard): drop the promise that a closed driver's process group…
pierreleca Nov 16, 2026
a7a628c
test(standard): stop the run from the setup's own send in the unanswe…
pierreleca Nov 16, 2026
12f9b78
Merge branch 'feat/driver-hooks' into feat/osworld1-agents
pierreleca Nov 16, 2026
d237476
Merge branch 'feat/computer-port' into feat/osworld1-agents
pierreleca Nov 16, 2026
a3eff8c
merge(standard): bring the computer port's server into the kit's branch
pierreleca Nov 16, 2026
dfa439b
feat(catalogue): run OSWorld1 with an agent through the computer port
pierreleca Nov 16, 2026
355e763
fix(standard): bound every frame and read the port's schemas alike ev…
pierreleca Nov 16, 2026
8413506
docs(standard): state that a computer tool takes only the arguments i…
pierreleca Nov 16, 2026
a189905
Merge branch 'feat/computer-port' into feat/osworld1-agents
pierreleca Nov 16, 2026
b157116
feat(standard): give the computer port its conformance kit
pierreleca Nov 16, 2026
9959de6
test(catalogue): bound the OSWorld1 score test's wait and state the f…
pierreleca Nov 16, 2026
6151e9d
fix(standard): start the computer kit's pipe from the kit's own folder
pierreleca Nov 16, 2026
0b49eac
Merge branch 'feat/computer-port' into feat/computer-kit
pierreleca Nov 16, 2026
d4c5f84
fix(catalogue): let OSWorld1's reaper finish what the driver's end cuts
pierreleca Nov 16, 2026
a70fa48
fix(standard): earn the computer kit's passes and leave nothing after…
pierreleca Nov 16, 2026
c12294f
fix(standard): preserve computer kit admission and shutdown ordering
pierreleca Nov 16, 2026
003d2fd
feat(standard): keep the computer kit's fixture events and target box…
pierreleca Nov 16, 2026
68ceb15
fix(standard): preserve computer scores when record copies fail
pierreleca Nov 16, 2026
60b9fe3
Merge branch 'feat/computer-kit' into local/computer-integration
pierreleca Nov 16, 2026
5cc1164
Merge branch 'feat/osworld1-agents' into local/computer-integration
pierreleca Nov 16, 2026
cefc912
fix(standard): describe computer tool screenshots and coordinates
pierreleca Nov 16, 2026
ced74f3
Merge branch 'feat/computer-port' into feat/computer-kit
pierreleca Nov 16, 2026
0e98b4c
Merge branch 'feat/computer-port' into feat/osworld1-agents
pierreleca Nov 16, 2026
5c12eef
Merge branch 'feat/computer-kit' into local/computer-integration
pierreleca Nov 16, 2026
c3a78f1
fix(standard): avoid ambiguous and repeated computer kit code characters
pierreleca Nov 16, 2026
11303da
Merge branch 'feat/osworld1-agents' into local/computer-integration
pierreleca Nov 16, 2026
a07ead3
fix(standard): read common words in computer checks
pierreleca Nov 17, 2026
78fd4a3
fix(standard): allow repeated answers in computer reading checks
pierreleca Nov 17, 2026
a212a59
Merge branch 'feat/computer-kit' into local/computer-integration
pierreleca Nov 17, 2026
2d35e5b
Merge remote-tracking branch 'origin/main' into local/computer-integr…
pierreleca Nov 17, 2026
6a403a5
test(standard): keep the engine's own environment lookup out of a pat…
pierreleca Nov 17, 2026
8fde45d
feat(catalogue): record Codex computer port proof
pierreleca Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,13 @@ and its resources. The source layout stays under `app/`.
`feed`: the live feed any Harbor writes, from its own trial hooks),
`importers/` (`atif` ATIF, `otel` OpenTelemetry, and the `cli`), `judge/` (`questions`, `state`, `client`, `store`, `batch`, `reconcile`, `lanes`),
`standard/` (the Nullius standard, `docs/standard.md`: `ports`, `names` (capability vocabulary and legacy aliases), `definitions` (with an agent's or capability's versions), `match`,
`jsonl`, `acp`, `cancellation` (interruptible protocol waits), `relay`, `ledger` (a run's spending cap), `pins` (image digests checked against registry tags), `install`, `run`, `container` (the container backend), `setup_container` (an Original setup's driver and its Linux dependencies in Docker), `forward` (a licence server's one door from it, or a benchmark's MCP server's), `web_services` (HTTP fixtures a driver's setup declares, each behind a unit-local Docker door), `mcp_machine` and `mcp_bridge` (a benchmark's MCP server run on the machine, in its driver's environment, and the agent's bridge to it), `network_requirements` (an environment's declared public web access: the schema and the connectivity check), `hostreq` (what a benchmark's options need of the machine its driver runs on: the check and the install) and `egress` (an own image's check's proxy to public addresses), `job`, `conform`, `fidelity` (ACP against the agent's own CLI), `prove` (a benchmark's controls and proofs), `shard` and `merge` (one run over many machines), `recipe` (the builder's standard recipe), `howto` (the How to add steps) and the `cli`; its
`jsonl`, `acp`, `cancellation` (interruptible protocol waits), `relay`, `ledger` (a run's spending cap), `pins` (image digests checked against registry tags), `install`, `run`, `container` (the container backend), `setup_container` (an Original setup's driver and its Linux dependencies in Docker), `forward` (a licence server's one door from it, or a benchmark's MCP server's), `web_services` (HTTP fixtures a driver's setup declares, each behind a unit-local Docker door), `mcp_machine` and `mcp_bridge` (a benchmark's MCP server run on the machine, in its driver's environment, and the agent's bridge to it), `computer_port` and `computer_pipe` (the computer port's MCP server, served by a desktop benchmark's runner over the benchmark's own loop, and the session's stdio pipe to it), `network_requirements` (an environment's declared public web access: the schema and the connectivity check), `hostreq` (what a benchmark's options need of the machine its driver runs on: the check and the install) and `egress` (an own image's check's proxy to public addresses), `job`, `conform`, `fidelity` (ACP against the agent's own CLI), `prove` (a benchmark's controls and proofs), `shard` and `merge` (one run over many machines), `recipe` (the builder's standard recipe), `howto` (the How to add steps) and the `cli`; its
`payload` snapshots the running backend and effective catalogue for a remote host;
conformance kits under `standard/kits/`, each a benchmark: a `driver` and, for the
tools kit, an MCP server `mcp_kit`) - the definitions it reads are in the catalogue:
a benchmark's `standard.json` and `driver` in its folder (τ³-bench's also the MCP
server of its domains' tools, `mcp_server`: the upstream's own, run on the machine; OSWorld1's also its null control's
agent, `nop`), `catalogue/agents/<key>/`
agent, `nop`, and the `runner` of a unit an agent runs: the upstream's loop with the computer port's bridge), `catalogue/agents/<key>/`
(the reference `agent`; JiuwenSwarm's `agent` and mini-swe-agent's `run`, the wrappers that start them) and `catalogue/capabilities/<key>/` (the keyword-memory `capability`); `workspace` (and
`workspace_cli`, its shell command),
`cli` and `__main__` (the unified `nullius` and `python -m nullius` commands),
Expand Down
35 changes: 35 additions & 0 deletions app/nullius/catalogue/agents/codex/agent.json
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,41 @@
"kit_version": 1,
"backend": "docker",
"job": "2026-11-12__10-24-26__kit__agent-search-v1__codex-2.2.2"
},
"agent.computer/v1": {
"passed": "4/4",
"on": "2026-11-17",
"model": "gpt-6-luna",
"provider": "openai",
"version": "2.2.2",
"kit_version": 1,
"backend": "container",
"job": "2026-11-17__12-53-08__kit__agent-computer-v1__codex-2.2.2",
"repeat": 3,
"checks": {
"see": "3/3",
"click": "3/3",
"type": "3/3",
"limit": "3/3"
},
"skills": {
"drag": {
"passed": "3/3",
"ok": true
},
"scroll": {
"passed": "3/3",
"ok": true
},
"recover": {
"passed": "3/3",
"ok": true
},
"batch": {
"passed": "3/3",
"ok": true
}
}
}
}
},
Expand Down
13 changes: 11 additions & 2 deletions app/nullius/catalogue/benchmarks/osworld1/benchmark.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
],
"origin": "official",
"good_to_know": [
"An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. Steps (100) and Pause (3.0 seconds) set that loop.",
"Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with.",
"Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others.",
"Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call.",
Expand All @@ -25,10 +26,18 @@
"Each VM's ports are open on every network interface; one runs any command in the VM. Bind Docker's ports to 127.0.0.1, or keep the machine off untrusted networks."
],
"limitations": [
"No agent from the Agents page can run it yet.",
"An agent needs the computer port, proven by its conformance kit. The agent side has been checked on a fake desktop alone.",
"Preparing the VM image (its lock and its download) is not bounded by a unit's setup time; the core's own limit on setup ends it.",
"A VM start that hangs holds the start gate until its unit's setup time ends; units waiting for the gate fail at the end of theirs.",
"On the local backend an agent runs as you and can write where you can, the runner's folder included.",
"A runner let go at the end of scoring's time that then ends by itself is scored from its complete result.",
"A stop during the runner's last clean-up keeps the result already written.",
"A driver killed outright leaves the runner to stop on its own; a kill during a VM's start can leave that VM.",
"A reaper checks a runner's start time before it ends the runner's group; a process number reused in between is not caught.",
"The agent's python and upload calls do not mark OSWorld's desktop as used; each unit's desktop is reset once, before the agent's first call.",
"Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach.",
"The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested.",
"A task still running after 2 hours is stopped and gets no score."
"A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score."
],
"datasets": [
{
Expand Down
Loading
Loading