Repository navigation
fix(standard): stop a run when SIGTERM reaches a thread other than main - #144
Merged
harenome merged 1 commit intoNov 16, 2026
Merged
Conversation
Python runs a signal handler only in the main thread. During a run, that thread waited in as_completed with no timeout. POSIX lets the kernel give a process's SIGTERM to any thread. Linux gives it to the main thread when that thread can take it, so the defect is rare in practice, but a SIGTERM that reached another thread did not stop the run. The main thread now waits for the units in a loop with a 0.5 s timeout, so it wakes up and runs the handler. It returns at once when a unit raises. The signal mask stays as it is: child processes that the units start must still receive SIGTERM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #143
Related to #112
What
Stop sends SIGTERM to the run process, and Python runs the handler only in the main thread. The main thread waited for the units in
as_completedwith no timeout, so a SIGTERM that reached another thread did not stop the run. Linux gives a process-directed signal to the main thread when that thread can take it, so this is rare on Linux; POSIX lets any thread take it, andtgkillorpthread_killcause it every time.Run.startnow waits for the units withconcurrent.futures.wait(..., timeout=0.5, return_when=FIRST_EXCEPTION)in a loop. Each timeout wakes the main thread, which then runs the handler. A unit that raises still ends the wait at once, so the cleanup inexceptstarts as early as before.Run.startis the only place where the run's main thread waits on the units' threads. The other blocking waits of the main thread, for example an image build before the first unit, end on their own. If SIGTERM reaches another thread during one of them, the run stops when that wait ends.A signal mask (SIGTERM blocked in the worker threads) does not fix the defect, and it adds one. Measured with SIGTERM sent by
tgkill, with the mask set in the pool's initializer:SigBlk: 0000000000004002).subprocesskeeps the mask of the thread that starts the child.Touches
backend · the standard
app/nullius/standard/run.pyapp/tests/test_audit_regressions.pyVerified
In rootless Podman containers (Python 3.12,
--init), on 4648c75:tgkillto one thread.test_sigterm_given_to_a_thread_other_than_main_stops_the_run. One of the run's own threads sends SIGTERM to itself withsignal.pthread_killwhen the memory process has started. It uses the same checks astest_sigterm_stops_a_real_memory_process_and_writes_cancelled_results. On main it fails withTimeoutExpiredafter 12 s. On this branch it passes.as_completed0.12 s and 0.50 s, this loop 0.08 s and 0.42 s.python3 app/tests/parallel.py:1992 tests in 116 modules: OK. Route suite,python3 app/tests/parallel.py app/ui_tests:331 tests in 41 modules: OK.Risk
Low. Undo it with a revert.
resultsbeforerun_againcan change, so only the order of the rate-limited units that run again can change. The job's results are rebuilt in the original order infinally.