Repository navigation
feat(runs): fold a launch's shards into one row per participant - #137
Merged
Merged
Conversation
A launch run over several machines drew one row per shard: an agent on
two machines was two rows, each with its own rate, and its shards' rows
were mixed with the other agents' in the ranking.
Fold each agent's or control's shards into one row when it has two or
more and `cli merge` would merge them. The row shows the merged job's
numbers: the trials of every shard, and the job record from
`merge.result_of`, now shared with `merge`. The rate is over every trial
with a verdict. Units that no shard ran stand beside the count ("2 of 8
not run"), never among the misses. The state is the most telling one of
the shards, with a note such as "1 of 2 shards stopped".
The row unfolds into one row per shard, named by shard and machine, each
with its Open. Controls fold the same way and stay unranked. The
task-check line is judged over every shard: "oracle not run on shard 2".
A fold reads every trial record of its shards. It is kept for the shard
jobs the workspace cache hands out, and read again only when the
workspace parses a shard again. A new set of shards replaces each
kept fold that shares a shard with it, so the cache holds one fold per
shard at most.
aitianshi
added a commit
that referenced
this pull request
Nov 16, 2026
No linked issue: the owner asked for v0.1.1 as a release, so that the Hugging Face endpoint setting (#129) reaches PyPI. ## What Prepares v0.1.1. The version becomes `0.1.1`, and `docs/releases/v0.1.1.md` gives the highlights since v0.1.0, a Contributors section, the full changelog of every pull request in `v0.1.0..main` (#114, #117, #120, #122, #125, #127, #129, #132, #137, and #135 among the test and CI changes) and the installation notes, in the shape of v0.1.0's. Install examples, the release guide, the docs' release-notes link and menu, and the workflows' default tags name 0.1.1. The release tooling already accepts final `vX.Y.Z` tags and does not change. ## Touches CI and skills (the release and publish workflows' default tags) · docs and site ## Verified - Backend suite (`python3 app/tests/parallel.py`): 1991 tests in 116 modules, OK. Route suite: 428 tests in 41 modules, OK. - `catalogue_docs.py --write` and `diagrams.py --write` left no change. - `python3 app/packaging/release.py build --tag v0.1.1` built and verified `nullius_eval-0.1.1-py3-none-any.whl`, `nullius_eval-0.1.1.tar.gz` and `SHA256SUMS`; `release.py verify --tag v0.1.1` and `verify.py` on the wheel passed, including the installed-tool and profile-preservation checks. - The workflows have not run; they run only by hand on `main`. ## Risk Version strings and docs only. A wrong default tag in a workflow is an input the person running it can change, and `release.py` refuses a tag that does not match `pyproject.toml` or has no notes. Nothing is tagged, drafted or published by this change. Undo with a revert.
3 of 11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #136
What
On Runs → By launch, an agent or control with 2 or more shards is now one row, when
cli mergewould merge those shards. The row shows the numbers of the merged job.launches.foldmakes that job in memory: the trials of every shard, and the job record frommerge.result_of.result_ofis the code thatmergeused inline to compute the job record, now a shared function.merge.checkkeeps the trial records that it already reads.merge.checkand stand beside the count ("2 of 8 not run"). They are never misses. The notes ofmergeare the tooltip.The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of
disclosures.js, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2."A participant with one shard keeps its plain row. Shards that
mergerefuses also keep their rows. Job pages,cli mergeand merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards").A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most.
Touches
backend · the standard (
merge.py) · routes · templates and static · docs and siteBefore and after
One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px.
Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2."
Verified
Fold.test_a_fold_reads_as_the_merged_job: three shards, one absent, and one with a planned unit that did not run. The folded job equalsparse_jobof themerge.mergeoutput on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those ofmerge. Shards thatmergerefuses are not folded.Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again: a second fold of the same cached jobs does not callmerge.check. After a trial record changes, it callsmerge.checkagain. After a third shard appears, the cache holds only the fold of the three.test_launch_shards: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows.c7af06d5: backend suitepython3 app/tests/parallel.py(1989 tests) and route suitepython3 app/tests/parallel.py app/ui_tests(328 tests) pass in a clean container./runs?view=launch, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after.Risk
If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly.
mergewrites the sameresult.jsonas before throughresult_of, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also changemerge.py: the second to merge must rebase. Revert the commit to undo.