Skip to content

feat(traces): rework the trial page around what a trial recorded - #173

Merged
aitianshi merged 14 commits into
mainfrom
refactor/trial-evidence-strip
Nov 17, 2026
Merged

aitianshi merged 14 commits into
mainfrom
refactor/trial-evidence-strip

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #179

What

A trial's page spent its first screen on parts that said nothing or said it twice, and hid some of what the trial recorded. This reworks it into a summary of the trial.

  • Head. The name is smaller. One row under it says what ran (the benchmark and its runner, or the importer of an imported trial; the agent with its mark, name, version and model) and how it went (the dataset's checks, the reward or the score, the agent's time, the steps). The Tool calls card and the evidence strip (Trajectory, Trace, Reward, Rate limits) are gone. A value the trial has not has no cell and no dash.
  • Overview. "Shape of the run" is gone. "Where the time went" is a compact tile in a grid that other tiles can join: its total in the heading, no paragraph, a chip for every phase the trial recorded however short, each with a hover saying what the phase holds on that kind of run. A panel with nothing to show is not drawn.
  • Durations. Two decimals wherever one is drawn: the secs filter (every page that uses it), the run page's script, a span's minutes (nsdur), nullius runs show. One decimal turned 0.05 s into "0.1s".
  • Artifacts. The tab lists only the records the trial wrote, each with Read and Download. A record's page has Copy and Download of the whole record, redacted as the reader shows it (/trial/…/artifact/<key>/download), and a back button to the Artifacts tab. A record past the reader's 16 MiB is saved as <name>.partial.<ext>.
  • Tabs. Artifacts, Conversation and the judge tabs no longer repeat their tab's name as a heading. The judge tabs keep the experimental badge (who answered, the question set, the date and the tokens in its hover) and say a refusal, an earlier vocabulary or a truncated state only when there is one.
  • Removed. The "benchmark's own judge" card, with the code that read verifier/judge.json: nothing in the application ever wrote that file. Its two hand-made fixture copies are deleted with it, which is why the judge's script floors this at severe.

Commits nine and ten correct what an independent review of the first eight found: cells colliding in the head at mid widths, sideways scroll at phone width, an imported trial said to be Harbor's, dashes where the page claimed none, three places still at one decimal, a silent cut of large downloads, Copy saying copied when a browser refused the write, and tests whose names no longer matched what they asserted.

The last three take main (ten commits, among them a task's attempts and an attempt's tries, each with tabs on this page):

  • The head reads: the name, what ran, then the attempt and try tabs, then that attempt's numbers. What is above the tabs is every attempt's, so a tab changes what is under it and the tabs stay put. A trial with no tab keeps its one row.
  • All attempts has the same head (one macro, traces/_head.html) in the place of its large name and subtitle.
  • Download now reads an earlier try's records, as the record page does; it answered 404 there.

Touches

  • templates and static: traces/trial.html, traces/_head.html (new), traces/attempts.html, _judgement.html, _conversation.html, artifact.html, runs/_execution.html (its script's secs), base.html (the copy handler), style.css, traces/conversation.css
  • routes: web/traces.py (the download route, the importer's name), web/filters.py (secs, nsdur), web/judge.py
  • backend: runs/artifacts.py (whole), runs/cli.py (two decimals)
  • docs: field-guide.md (n_rate_limited_retries), cli/pages.md (the download route), the agent marks README
  • tests: route tests (test_attempt_routes.py among them), and a new ui_tests/test_trial_head.py

Before and after

main and this branch serving the same job trees, dark, 1280 px wide, rail collapsed.

Overview, a SWE-bench trial

Before After
Before: cards, evidence strip, title, bar and paragraph After: one row of blocks and a time tile

Artifacts tab

Before After
Before: nine rows, six of them not recorded After: the three records written, each with Read and Download

A record

Before After
Before: a back link and the page After: a back button, Copy and Download

Judge tab

Before After
Before: the benchmark judge card, a heading and a description After: the badge and the cards

Verified

  • On the merge with main (49d32d1): backend suite 2229 OK, route suite 562 OK. With fix(standard): time the agent's setup apart from its run #174 merged on top as well: 2236 and 562 OK.
  • New tests: the head for a Harbor trial, a trial with nothing to show, a SWE-bench trial with and without its report, an imported trial, a scored trial; the cut download; the judge tab's notes; a span's minutes against secs; the run page's script against the template (run in node); a bar shorter than a second; the shell's phases against the page's.
  • Attempts: a retried attempt's earlier try downloads its own record (the test fails without the fix); the attempt and try tabs sit at the same height on every attempt of a task and on All attempts, at 820, 1024 and 1280 px, with no sideways scroll or clipped text at 390.
  • Browser, measured in the page: a six-cell SWE-bench trial at 390, 820, 1024 and 1280 px has no cell wider than its box and no sideways scroll; the Artifacts tab and a three-page record at 390 px fit; dark and light at 1280 px.
  • Copy: the whole record reaches the clipboard (22,990 characters of a record shown 8,000 a page) through each of its three paths, checked with the clipboard stubbed, and a real click reads "copied".
  • An independent review of the first eight commits: twelve findings, each fixed or answered in the last two commits.
  • Not verified: Safari (the Copy path written for it could not be run here), the judge tabs with a real judgement (a seeded record in a test), the nightly UI sweep.

Risk

  • Every duration drawn through secs gains a second decimal, on every page that uses it. A stage time recorded to one decimal reads "41.20s".
  • The trial page no longer shows the count of retried rate limits; the job table's 429 ×N badge keeps it.
  • Deleting the two judge.json fixtures removes the only trace of a card nothing could fill; the judge's script floors a deleted test file at severe.
  • Merges before fix(standard): time the agent's setup apart from its run #174, which no longer touches traces/trial.html: the two merge cleanly in that order.
  • Undone by a revert: no stored data changes shape.

The strip repeated what the tabs and the Reward card already say. The
time bar loses its paragraph: the total moves to the heading, a phase
that took no time has no chip, and each chip's tooltip says what that
phase holds on a Harbor or a standard run. Other on a standard run is the
cleanup after the check, not a wait for the judge, which is in Verify.
@github-actions

This comment has been minimized.

A check that took under 0.05 s lost its chip, so an instant check read as
none. A recorded phase now keeps its chip; only a segment needs time to
draw, and Other, which is computed, still shows only when there is some.
@github-actions

This comment has been minimized.

One decimal turned 0.05 s into 0.1s and hid differences the clock had
measured. Seconds and minutes now carry two decimals.
The "Shape of the run" title goes, and the time bar becomes a tile in a
grid that other summaries can join.
@github-actions

This comment has been minimized.

A trajectory with no per-step usage, no reasoning total and no tool call
opened the panel and left it empty.
@github-actions

This comment has been minimized.

The Artifacts tab lists only the records a trial wrote, each with Read
and Download. A record's page gains Copy and Download, both of the whole
record redacted as the reader shows it, and its back button returns to
the Artifacts tab rather than the overview.
@github-actions

This comment has been minimized.

…ting or empty parts

The Artifacts and Conversation tabs lose the heading their tab already
gives. The benchmark's judge card goes: nothing writes the record it
read. The judge tabs keep their experimental badge, with who answered
and when as its tooltip, and drop the description, the clean hygiene
line and the flag summary; a flag's question lines up under its name.
…al's name

The trial's name gets smaller. The benchmark and its runner share a card,
the agent with its mark, version and model another, and the reward, the
agent's time and the steps a strip beside them. Tool calls leave the
head, and a value the trial does not have is not drawn.
@github-actions

This comment has been minimized.

- The head's strip let its cells shrink under their text: six cells on a
  SWE-bench trial ran into each other between 820 and 1180 px and at
  phone width. Cells keep their width now; the strip moves under the
  cards, then wraps.
- The Artifacts tab and a record's tools scrolled the page sideways at
  phone width: the buttons wrap.
- An imported trial was said to be Harbor's, its importer's name labelled
  a benchmark. It says where it was imported from, and nothing else.
- A value the trial has not has no cell, as the comment claimed: no
  Reward, check or total shown as a dash, no strip with nothing in it. A
  graded score has a cell of its own.
- Two decimals were not everywhere: the run page's script redrew stage
  times with the old rule a moment after load, a span's minutes kept one
  decimal beside the head's two, and the shell's show kept one.
- Download and Copy cut a record at the reader's 16 MiB with no word of
  it: the saved name says partial. The route answers nosniff.
- Copy could say copied with nothing on the clipboard where a browser
  drops the click across the fetch: the clipboard is handed the pending
  text inside the click, and a failure is said.
- The judge tab ran its notes together (2 refused state truncated).
- A runner's Other was called the work after the check; on an older
  runner's trial it is the wait for the judge. It is what the phases
  leave of the trial, on both.
- Left behind and now gone: the second judge.json fixture, the card and
  summary styles, four values the judge panel computed for a line that
  is not drawn.
- Tests said more than they held: one renamed, one docstring made true,
  the page half of an absent trajectory restored. The head, the cut
  download, the duration twins and the judge tab's notes have tests.
The segments' flex weights were the phases' seconds: under a second in
all they summed below 1, and the bar drew only that share of its width.
The weights are whole milliseconds now.
@aitianshi aitianshi changed the title refactor(traces): drop the trial's evidence strip and trim its time bar feat(traces): rework the trial page around what a trial recorded Nov 17, 2026
@github-actions

This comment has been minimized.

Ten commits landed on main since this branch left it; three files conflicted.

- traces/trial.html: main put the attempt and try tabs of a task run several
  times between the subtitle and the cards, both of which this branch replaced
  with the trial head. The tabs stay, above the head: which attempt, then what
  ran and how it went, the order main reads them in.
- web/traces.py: both sides added an import on the same line; both stay.
- docs/field-guide.md: main added the n_retries row above the row this branch
  rewords; both stay.
main keeps a retried attempt's earlier tries, each with a page of its own. The record page reads them; the download route this branch adds did not, so Download answered 404 on a try's Artifacts tab and Copy said it failed. It resolves a try as the record page does.
… on All attempts

main's attempt tabs came in above the trial's head, and its All attempts page kept the large name and the subtitle this branch replaced on the trial page: the tabs sat 60 px lower there than on an attempt's page, and the name changed size from one tab to the next.

Both pages now read the same way: the name, what ran (one macro, traces/_head.html), then the attempt and try tabs. On a trial's page that attempt's numbers follow the tabs; a trial with no tab keeps its one row. What is above the tabs is the same for every attempt of the task, so a tab changes what is under it and the tabs stay where they are. The title row is as tall with its outcome tag as without.
@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 17, 2026
#173 merges first and words each phase's hover by the kind of run, a standard run's setup included; this branch's own wording of the same line would only conflict with it.
@github-actions

Copy link
Copy Markdown

🔴 PR judge · 7 of 8 yes · #173 ↔ #179

Merge risk ⛔ SEVERE a test file deleted: app/tests/fixtures/native/lme-arm/q0/verifier/judge.json
Priority 🟠 HIGH the deepest path is app/nullius/runs/artifacts.py
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ⚠️ partly
Before and after shown · 8 images · the diff touches what is visible ✅ yes
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 The evidence strip and the "Shape of the run" title are gone from the trial page. ✅ met (cut)
2 Under the trial's name, one row of blocks says what ran (the benchmark and its runner, or the importer of an imported t… ✅ met (cut)
3 "Where the time went" is a tile with its total in the heading and no paragraph: a chip for every phase the trial record… ✅ met (cut)
4 A panel with nothing to show is not drawn; the benchmark judge card and the code that read verifier/judge.json are re… ✅ met (cut)
5 A duration reads to two decimals wherever one is drawn: the pages, the run page's script, a span's minutes, `nullius ru… ✅ met (cut)
6 The Artifacts tab lists only the records the trial wrote, each with Read and Download. A record's page has Copy and Dow… ✅ met (cut)
7 The Artifacts, Conversation and judge tabs do not repeat their tab's name as a heading; the judge tabs keep the experim… ✅ met (cut)
8 No sideways scroll and no text running into its neighbour at 390, 820, 1024 and 1280 px, dark and light. ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit babad27 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the refactor/trial-evidence-strip branch November 17, 2026 23:43
aitianshi added a commit that referenced this pull request Nov 17, 2026
Closes #180

## What

A standard trial of an agent driven over ACP had no agent setup: the
agent's process start, `initialize` and `session/new` (which also starts
the benchmark's MCP servers) ran inside the window written as
`agent_execution`. The trial page read "Setup —" and its Agent time
carried the start.

The run now notes when the agent's session opens and writes Harbor's
`agent_setup`, from the end of the environment's setup to the open
session. `agent_execution`, and the window of the trial's trace, start
there.

The second commit corrects what an independent review of the first
found:

- A start that ends with no session (the agent never answered, exited,
or the run was stopped) was still written as the agent's run. It is its
setup, with no `agent_execution`, as Harbor writes a failed agent setup.
- A capability got a setup of a millisecond: the time to spawn a process
that answers nothing until its first request. A capability, a command
agent, an Original setup and a control have no setup phase.
- A task's internet door opened inside the agent's setup and closed
inside its run. The agent's phases begin once it is open and end before
it closes.
- The live feed still sent one duration, timed from before the setup,
and no `agent_setup` phase, so the live board disagreed with the trial
page. It emits the phase, starts the agent's limit when the session
opens, and takes every duration from the trial's result.
- The trace-window test could not fail. It pins the window now, and the
run's side has tests of its own.

The last two commits take `main` (ten commits: a unit's VM, forced time
limits, attempts and retries among them):

- The unit loop reads: the unit's VM, the task's door, then the agent's
setup and its run.
- `main` renamed the turn's limit to the agent's whole work on a unit
(`agent_limit`) and starts its clock once the session is open. The
board's `agent_run` phase is emitted there with that same limit, a run's
`--timeout agent` included; the setup keeps ACP's two start limits. The
line this branch had added still called the old name and merged without
a conflict: every ACP unit would have ended in an error.
- The trial page's Setup hover is left to #173, which merges first and
words it for a standard run.

Installing the agent stays a stage of the run, not of a trial.

## Touches

- the standard: `standard/run.py` (the unit loop, `drive_agent`),
`standard/job.py` (`write_trial`, `feed_durations`), `standard/acp.py`
(its two start limits, named)
- docs: `field-guide.md` (the phase rows, shorter)
- tests: `test_standard_trace.py`, `test_standard_run.py`

## Before and after

The same recipe (`agent-task-v1`, the reference agent, a stand-in
endpoint answering in 2 s) run on #173, which merges first, and on #173
with this branch: the trial page this lands on. Before, the agent's
start sits in Agent; after, it is Setup.

| Before | After |
| --- | --- |
| <img width="440" alt="Before: no Setup phase"
src="https://github.com/user-attachments/assets/0bb4e268-444d-43ce-8ac4-737e3809b048"
/> | <img width="440" alt="After: Setup 0.06s apart from Agent"
src="https://github.com/user-attachments/assets/429ff985-957c-4ff0-b124-41308ce3fe15"
/> |

## Verified

- On the merge with `main` (`49d32d1`): backend suite 2236 OK, route
suite 547 OK. With #173 merged first: 2236 and 562 OK.
- Seven of the new or extended tests fail on the first commit and pass
on the second: the four windows on shared stamps with the feed told the
same seconds; a start that never answers; a capability; a conversation
the upstream starts again; the task's door (a stand-in door that takes
0.4 s to open and to close); a failed start and a trial with no clock in
`write_trial`. The trace's window is pinned as well: it starts at the
open session, and a model call made during the setup widens it and is
never clamped.
- After the merge, a new test forces `--timeout agent` and reads the
feed: the setup carries ACP's 300 s, the run the forced limit. With the
old limit's name left in, the setup test ends in an error.
- A real local run on the merged code (`nullius standard run
agent-task-v1 reference --limit 1 --unsandboxed --timeout agent=45`,
stand-in endpoint): `result.json` holds `environment_setup` 0.046 s,
`agent_setup` 0.054 s, `agent_execution` 2.056 s, `verifier` 0.001 s,
back to back; `events.jsonl` holds the phases `env_setup` (600 s),
`agent_setup` (300 s), `agent_run` (45 s), `verify` (600 s) and a
`trial.end` with the same four durations; the trace's session is 2.056
s.
- Not verified: the docker tier and the Boat backend were not run, so a
real container's internet door and a unit's VM were not timed; the
door's order is held by the test with the stand-in door, and the VM
wraps the same block.

## Risk

- Agent time on standard runs written before and after this differs by
the agent's start; a comparison across that line reads slightly
different. Stored results are not rewritten, and the field guide says
so.
- On a task with an internet door, the environment's time grows by the
door opening and the agent's run shrinks by its opening and closing.
- A failed start has no `agent_execution`. Readers already meet a trial
without one (a unit that fails in its environment's setup).
- The live board shows "agent setup" while the agent starts, with a
limit of 300 s (ACP's two start requests), then the agent's limit from
its open session.
- Merges after #173. Merged alone, the Setup hover on a standard trial
would still read "installing the agent" until #173 lands.
- Undone by a revert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trial page: empty and repeated parts crowd out what a trial recorded, and a record cannot be saved or copied whole

1 participant