Skip to content

Tests: six backend tests fail at 1 CPU on fixed waits #169

Description

@harenome

Priority

low. The defects are in test code only. They fail the backend suite on a loaded machine, in changes that do not touch these tests.

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Tests only (app/tests): execution, execution recovery, kit check isolation, standard payload, standard definition and constraints tests.

Evidence

The backend suite, python3 app/tests/parallel.py, on ce8b39d, in rootless Podman containers (Python 3.12) limited with --cpus 1, two containers at a time, 12 runs. parallel.py sizes its pool from the host's 16 CPUs, so the capped run is overloaded. Each test below gives another process a fixed time, and the load exceeds it. 9 of the 12 runs passed.

Test Failed Fixed wait Error
test_execution KeyCutShort.test_a_harbor_run_gets_the_agent_keys_and_the_standard_none ([host], runner='harbor') 1 of 12 wait(), 10 s for the run's end AssertionError: True is not false : the run did not end
test_execution_recovery CrashAtLaunch.test_a_part_started_before_its_pid_was_saved_runs_once ([a]) 1 of 12 wait(), 10 s for the run's end AssertionError: True is not false : the run did not end
test_kit_check_isolation WorkspaceKit.test_a_planted_link_is_not_followed 1 of 12 10 s for the driver's score reply nullius.standard.jsonl.Failed: score: no reply in 10 s
test_standard_payload Snapshot.test_installed_local_execution_has_one_backend_provider_and_keeps_helpers_and_proof_writes 0 of 12, then 2 of 12 thread.join(15) for each of 3 runs, 120 s for the whole driver AssertionError: fixture did not finish
test_standard_definitions Problems.test_catalogue_check_details_cover_the_driver_units (agent-tools-v1, agent-workspace-v1) 0 of 12, then 2 of 24 10 s for the driver's units reply nullius.standard.jsonl.Failed: units: no reply in 10 s
test_constraints Scope.test_a_unit_past_its_timeout_is_killed_with_its_group 0 of 12, then 1 of 48 0.2 s for the killed group's background child to go AssertionError: ProcessLookupError not raised

wait() is the shared helper in app/tests/execution_support.py. The other WorkspaceKit tests give each driver request the same 10 s. The last three tests passed the 12 runs on main. They failed in further runs of the same suite and load, made while the fixes for other rows were applied one by one; those fixes do not touch them. The counts after "then" are of the further runs made while the test itself was unchanged.

The wait() failures:

File "/work/app/tests/test_execution.py", line 411, in test_a_harbor_run_gets_the_agent_keys_and_the_standard_none
File "/work/app/tests/execution_support.py", line 29, in wait
AssertionError: True is not false : the run did not end
File "/work/app/tests/test_execution_recovery.py", line 325, in test_a_part_started_before_its_pid_was_saved_runs_once
File "/work/app/tests/execution_support.py", line 29, in wait
AssertionError: True is not false : the run did not end

The driver failure:

File "/work/app/tests/test_kit_check_isolation.py", line 80, in test_a_planted_link_is_not_followed
    self.driver().request("score", unit="fix", dir=self.ws, transcript="", answer="", timeout=10)
File "/work/app/nullius/standard/jsonl.py", line 63, in request
    raise Failed(f"{op}: no reply in {int(timeout)} s")
nullius.standard.jsonl.Failed: score: no reply in 10 s

The snapshot failure, from the driver that the test runs in a subprocess:

File "/work/app/tests/test_standard_payload.py", line 367, in test_installed_local_execution_has_one_backend_provider_and_keeps_helpers_and_proof_writes
    self.assertEqual(ran.returncode, 0, ran.stdout + ran.stderr)
AssertionError: 1 != 0 : Traceback (most recent call last):
  File "<string>", line 20, in <module>
AssertionError: fixture did not finish

Screenshot

No response

Reproduction

On main (ce8b39d):

  1. Run python3 app/tests/parallel.py in a container limited with --cpus 1 (podman run --init --cpus 1 ...), two containers at a time, 12 times.
  2. One of the first three tests fails in about one run of four. Without a CPU limit the suite passes.

Use --init. Without it, Python is PID 1 and does not reap orphaned processes. The background child of test_a_unit_past_its_timeout_is_killed_with_its_group then stays a zombie, os.kill(child, 0) succeeds, and that test fails at any load. That failure comes from the container, not from the load.

Expected

The backend suite passes on a loaded machine at 1 CPU. A test fails only when the behaviour it checks is wrong.

Acceptance

  • wait() in app/tests/execution_support.py waits for the run's end until a load-tolerant deadline, not a fixed 10 s, and still fails when the run does not end.
  • The WorkspaceKit tests give each driver reply a load-tolerant time, and still fail when the driver does not reply.
  • The installed snapshot test in test_standard_payload gives each run a load-tolerant time, and still fails when a run does not end.
  • test_catalogue_check_details_cover_the_driver_units gives each driver's units reply a load-tolerant time, and still fails when a driver does not reply.
  • test_a_unit_past_its_timeout_is_killed_with_its_group waits until the child is gone or a deadline passes, not a fixed 0.2 s, and still fails when the child outlives its group.
  • A passing test takes no longer than before.
  • The backend suite has no fixed-wait failure in 12 runs at --cpus 1.

Dependencies and related

Activity

  1. added
    bugSomething isn't working
    P3When convenient: wording, layout, tests
    on Nov 16, 2026
  2. added theissue type on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3When convenient: wording, layout, testsbugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions