Skip to content

test: wait for runs, drivers and killed processes until a load-tolerant deadline - #170

Draft
harenome wants to merge 5 commits into
ThinkFlowLab:mainfrom
harenome:fix/fixed-wait-tests
Draft

harenome wants to merge 5 commits into
ThinkFlowLab:mainfrom
harenome:fix/fixed-wait-tests

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #169

What

Six backend tests give a run, a kit driver or a killed process a fixed time to end, to reply or to go: 10 s, 15 s or 0.2 s. On a loaded machine this can take longer, and the test fails. Each wait now has an upper limit of 60 s, the limit that #142 gives the SIGTERM test of a memory run, or 30 s where 60 s is too long for the test.

Each wait stops on its condition: Thread.join returns when the run ends, jsonl.Proc.request returns at the reply, and gone() returns when the process is gone. Thus a passing test takes no longer than before. Only the upper limit changes.

  • wait() in app/tests/execution_support.py: 60 s for the run's end, where it was 10 s. This is the shared helper of the execution tests. Calls that give their own time do not change.
  • test_kit_check_isolation WorkspaceKit: 60 s for each setup and score reply, where it was 10 s.
  • test_standard_payload installed snapshot test: 60 s for each of its three runs, where it was 15 s. The upper limit of its driver subprocess is now 240 s, where it was 120 s, so that the three runs fit.
  • test_standard_definitions test_catalogue_check_details_cover_the_driver_units: 60 s for each driver's units reply, where it was 10 s.
  • test_constraints test_a_unit_past_its_timeout_is_killed_with_its_group: polls with gone() from execution_support, as the other process tests do, for up to 30 s, where it slept 0.2 s and then checked once. The limit stays under the child's own sleep 60, so a child that the run leaves running still fails the test. gone() counts a zombie as gone, because a zombie runs nothing.

Touches

backend · the standard (tests only)

  • app/tests/execution_support.py
  • app/tests/test_kit_check_isolation.py
  • app/tests/test_standard_payload.py
  • app/tests/test_standard_definitions.py
  • app/tests/test_constraints.py

Verified

The backend suite, python3 app/tests/parallel.py, in rootless Podman containers (python:3.12, --init --cpus 1), two containers at a time. Main: 12 runs on ce8b39d. This branch: 18 runs at its tip, 12 on b11083b and 6 on 411b985. These bases change nothing that the tests wait for.

Test main this branch
KeyCutShort.test_a_harbor_run_gets_the_agent_keys_and_the_standard_none 1 0
CrashAtLaunch.test_a_part_started_before_its_pid_was_saved_runs_once 1 0
WorkspaceKit.test_a_planted_link_is_not_followed 1 0
Snapshot.test_installed_local_execution_has_one_backend_provider_and_keeps_helpers_and_proof_writes 0 (2 in 12 further runs) 0
Problems.test_catalogue_check_details_cover_the_driver_units 0 (2 in 24 further runs) 0
Scope.test_a_unit_past_its_timeout_is_killed_with_its_group 0 (1 in 48 further runs) 0
Runs that passed 9 of 12 17 of 18

The further runs used the same suite and load, while the fixes for the other tests were applied one by one. Each count is of the runs made while that test itself was unchanged.

Other tests failed in some runs. This change does not touch them, and they have other causes:

  • test_standard_run RunTest.test_a_cap_per_unit_cuts_off_the_unit_that_reaches_it_and_the_run_goes_on, 1 of the 18 runs on this branch: a unit other than the capped one was not resolved. Its outcome was not kept.
  • test_run_delete RunDelete.test_it_asks_what_goes, 2 of the further runs on b11083b: ['b', 'a'] != ['a', 'b']. The test expects workspaces a and b in creation order. workspace.list_all orders them by opened, a timestamp in whole seconds, most recent first. Under load the two creations fall in different seconds.
  • test_execution_stop StopAtLaunch.test_a_part_started_by_a_launcher_cut_short_here_is_stopped, 1 of the further runs: the part ran to its end after the Stop. The part sleeps 3 s, and the Stop must reach it before then. This is a race in the checked behaviour, not an upper limit on a wait.

Each test still fails when the behaviour it checks is wrong. Each check used a local change that is not in this branch, and ran in a container at --cpus 1:

Local change Test Result
The run thread never ends (threading.Event().wait() at the start of Execution._main) KeyCutShort harbor test the run did not end, after 60 s for each of 3 subtests
The same CrashAtLaunch the run did not end, after 60 s for each of 2 subtests
The same installed snapshot test fixture did not finish, after 60 s
A Harbor run gets no agent key (_agent_keys = {}) KeyCutShort harbor test 'cursor=none' != 'cursor=cur-...', at once
A restart launches the part again, not follows it (follow always false) CrashAtLaunch the log has no following a again line, at once
The kit driver follows a planted link (no O_NOFOLLOW) test_a_planted_link_is_not_followed (True, 'nullius-ok') != (False, None), at once
The kit driver does not reply to score test_a_planted_link_is_not_followed score: no reply in 60 s
A kit driver does not reply to units test_catalogue_check_details_cover_the_driver_units units: no reply in 60 s
A kit driver lists an extra unit test_catalogue_check_details_cover_the_driver_units ['write', 'fix'] != ['write', 'fix', 'extra'], at once
A timed-out unit kills only its direct child, not its group (p.kill() in place of _kill_group(p) in run_group) test_a_unit_past_its_timeout_is_killed_with_its_group the background child outlived its group, after 30 s

Without a CPU limit, at this branch: backend suite 2046 tests in 119 modules: OK, route suite 353 tests in 42 modules: OK.

Risk

Low. The change is in test code only. Undo it with a revert.

  • When a run does not end or a driver does not reply, the test now fails after 60 s, where it failed after 10 s or 15 s. When a killed group's child stays, the test fails after 30 s.
  • A machine that needs more than 60 s still fails these tests.

The shared wait() gave a run 10 s to end. On a loaded machine a run that works can take longer. join() returns when the run ends, so a passing test takes the same time as before.
The kit isolation tests gave each setup and score request 10 s. On a loaded machine the driver can take longer. request() returns at the reply, so a passing test takes the same time as before.
The test gave each of its three runs 15 s to end, and the whole driver 120 s. On a loaded machine a run that works can take longer. Each run now has 60 s and the driver 240 s, room for the three runs. join() and run() return at the end, so a passing test takes the same time as before.
The check details test gave each kit driver 10 s to reply to units. On a loaded machine a driver that works can take longer. request() returns at the reply, so a passing test takes the same time as before.
The scope test gave the child of a killed group 0.2 s to go. On a loaded machine it can take longer. The test now polls with gone(), as the other process tests do, for up to 30 s. The limit stays under the child's own 60 s sleep, so a child that the run leaves running still fails the test. gone() returns when the child is gone, so a passing test takes no longer than before.
@harenome
harenome marked this pull request as draft November 17, 2026 10:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests: six backend tests fail at 1 CPU on fixed waits

1 participant