Repository navigation
Description
Priority
low. The defects are in test code only. They fail the backend suite on a loaded machine, in changes that do not touch these tests.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Tests only (app/tests): execution, execution recovery, kit check isolation, standard payload, standard definition and constraints tests.
Evidence
The backend suite, python3 app/tests/parallel.py, on ce8b39d, in rootless Podman containers (Python 3.12) limited with --cpus 1, two containers at a time, 12 runs. parallel.py sizes its pool from the host's 16 CPUs, so the capped run is overloaded. Each test below gives another process a fixed time, and the load exceeds it. 9 of the 12 runs passed.
| Test | Failed | Fixed wait | Error |
|---|---|---|---|
test_execution KeyCutShort.test_a_harbor_run_gets_the_agent_keys_and_the_standard_none ([host], runner='harbor') |
1 of 12 | wait(), 10 s for the run's end |
AssertionError: True is not false : the run did not end |
test_execution_recovery CrashAtLaunch.test_a_part_started_before_its_pid_was_saved_runs_once ([a]) |
1 of 12 | wait(), 10 s for the run's end |
AssertionError: True is not false : the run did not end |
test_kit_check_isolation WorkspaceKit.test_a_planted_link_is_not_followed |
1 of 12 | 10 s for the driver's score reply |
nullius.standard.jsonl.Failed: score: no reply in 10 s |
test_standard_payload Snapshot.test_installed_local_execution_has_one_backend_provider_and_keeps_helpers_and_proof_writes |
0 of 12, then 2 of 12 | thread.join(15) for each of 3 runs, 120 s for the whole driver |
AssertionError: fixture did not finish |
test_standard_definitions Problems.test_catalogue_check_details_cover_the_driver_units (agent-tools-v1, agent-workspace-v1) |
0 of 12, then 2 of 24 | 10 s for the driver's units reply |
nullius.standard.jsonl.Failed: units: no reply in 10 s |
test_constraints Scope.test_a_unit_past_its_timeout_is_killed_with_its_group |
0 of 12, then 1 of 48 | 0.2 s for the killed group's background child to go | AssertionError: ProcessLookupError not raised |
wait() is the shared helper in app/tests/execution_support.py. The other WorkspaceKit tests give each driver request the same 10 s. The last three tests passed the 12 runs on main. They failed in further runs of the same suite and load, made while the fixes for other rows were applied one by one; those fixes do not touch them. The counts after "then" are of the further runs made while the test itself was unchanged.
The wait() failures:
File "/work/app/tests/test_execution.py", line 411, in test_a_harbor_run_gets_the_agent_keys_and_the_standard_none
File "/work/app/tests/execution_support.py", line 29, in wait
AssertionError: True is not false : the run did not end
File "/work/app/tests/test_execution_recovery.py", line 325, in test_a_part_started_before_its_pid_was_saved_runs_once
File "/work/app/tests/execution_support.py", line 29, in wait
AssertionError: True is not false : the run did not end
The driver failure:
File "/work/app/tests/test_kit_check_isolation.py", line 80, in test_a_planted_link_is_not_followed
self.driver().request("score", unit="fix", dir=self.ws, transcript="", answer="", timeout=10)
File "/work/app/nullius/standard/jsonl.py", line 63, in request
raise Failed(f"{op}: no reply in {int(timeout)} s")
nullius.standard.jsonl.Failed: score: no reply in 10 s
The snapshot failure, from the driver that the test runs in a subprocess:
File "/work/app/tests/test_standard_payload.py", line 367, in test_installed_local_execution_has_one_backend_provider_and_keeps_helpers_and_proof_writes
self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
AssertionError: 1 != 0 : Traceback (most recent call last):
File "<string>", line 20, in <module>
AssertionError: fixture did not finish
Screenshot
No response
Reproduction
On main (ce8b39d):
- Run
python3 app/tests/parallel.pyin a container limited with--cpus 1(podman run --init --cpus 1 ...), two containers at a time, 12 times. - One of the first three tests fails in about one run of four. Without a CPU limit the suite passes.
Use --init. Without it, Python is PID 1 and does not reap orphaned processes. The background child of test_a_unit_past_its_timeout_is_killed_with_its_group then stays a zombie, os.kill(child, 0) succeeds, and that test fails at any load. That failure comes from the container, not from the load.
Expected
The backend suite passes on a loaded machine at 1 CPU. A test fails only when the behaviour it checks is wrong.
Acceptance
-
wait()inapp/tests/execution_support.pywaits for the run's end until a load-tolerant deadline, not a fixed 10 s, and still fails when the run does not end. - The
WorkspaceKittests give each driver reply a load-tolerant time, and still fail when the driver does not reply. - The installed snapshot test in
test_standard_payloadgives each run a load-tolerant time, and still fails when a run does not end. -
test_catalogue_check_details_cover_the_driver_unitsgives each driver'sunitsreply a load-tolerant time, and still fails when a driver does not reply. -
test_a_unit_past_its_timeout_is_killed_with_its_groupwaits until the child is gone or a deadline passes, not a fixed 0.2 s, and still fails when the child outlives its group. - A passing test takes no longer than before.
- The backend suite has no fixed-wait failure in 12 runs at
--cpus 1.
Dependencies and related
- Tests: the judge batch overlap test fails on wall time under load #109 (closed): the judge batch overlap test failed on wall time under load. Fixed by test(judge): check the batch overlap with a barrier, not wall time #135, which checks the overlap with a barrier, not wall time.
- Tests: the SIGTERM test of a memory run gives its cleanup a fixed 12 s #112 (closed): the SIGTERM test of a memory run gave its cleanup a fixed 12 s. Fixed by test(standard): give a stopped memory run a load-tolerant time to clean up #142, which gives it a load-tolerant 60 s.
- Tests: backend tests race with the processes they watch #16 (closed): the earlier set of backend tests that raced with the processes they watch.
Activity