Skip to content

fix(standard): stop a run when SIGTERM reaches a thread other than main - #144

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/stop-ignored-in-worker-thread
Nov 16, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/stop-ignored-in-worker-thread

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #143
Related to #112

What

Stop sends SIGTERM to the run process, and Python runs the handler only in the main thread. The main thread waited for the units in as_completed with no timeout, so a SIGTERM that reached another thread did not stop the run. Linux gives a process-directed signal to the main thread when that thread can take it, so this is rare on Linux; POSIX lets any thread take it, and tgkill or pthread_kill cause it every time.

Run.start now waits for the units with concurrent.futures.wait(..., timeout=0.5, return_when=FIRST_EXCEPTION) in a loop. Each timeout wakes the main thread, which then runs the handler. A unit that raises still ends the wait at once, so the cleanup in except starts as early as before.

Run.start is the only place where the run's main thread waits on the units' threads. The other blocking waits of the main thread, for example an image build before the first unit, end on their own. If SIGTERM reaches another thread during one of them, the run stops when that wait ends.

A signal mask (SIGTERM blocked in the worker threads) does not fix the defect, and it adds one. Measured with SIGTERM sent by tgkill, with the mask set in the pool's initializer:

  • A thread outside the pool does not have the mask, and the kernel can still give it the signal. With SIGTERM sent to that thread, the run is still running after 90 s.
  • The memory process that a unit starts has SIGTERM and SIGINT blocked (SigBlk: 0000000000004002). subprocess keeps the mask of the thread that starts the child.

Touches

backend · the standard

  • app/nullius/standard/run.py
  • app/tests/test_audit_regressions.py

Verified

In rootless Podman containers (Python 3.12, --init), on 4648c75:

  • The reproduction: a run whose memory process does not answer, with SIGTERM sent by tgkill to one thread.
    • main: to the main thread, exit 143 after 3.1 s. To another thread, still running after 90 s.
    • This branch: to the main thread, exit 143 after 3.1 s. To another thread, exit 143 after 3.5 s.
  • New test, test_sigterm_given_to_a_thread_other_than_main_stops_the_run. One of the run's own threads sends SIGTERM to itself with signal.pthread_kill when the memory process has started. It uses the same checks as test_sigterm_stops_a_real_memory_process_and_writes_cancelled_results. On main it fails with TimeoutExpired after 12 s. On this branch it passes.
  • Both SIGTERM tests, 20 times in a row: 20 of 20 passed.
  • CPU time of the main thread's wait over 5,000 and 20,000 short units on 8 threads: as_completed 0.12 s and 0.50 s, this loop 0.08 s and 0.42 s.
  • Backend suite, python3 app/tests/parallel.py: 1992 tests in 116 modules: OK. Route suite, python3 app/tests/parallel.py app/ui_tests: 331 tests in 41 modules: OK.

Risk

Low. Undo it with a revert.

  • While units run, the main thread wakes up two times each second and does nothing else.
  • Units that end normally are collected at each timeout, not one by one. The order of results before run_again can change, so only the order of the rate-limited units that run again can change. The job's results are rebuilt in the original order in finally.

Python runs a signal handler only in the main thread. During a run, that
thread waited in as_completed with no timeout. POSIX lets the kernel give
a process's SIGTERM to any thread. Linux gives it to the main thread when
that thread can take it, so the defect is rare in practice, but a SIGTERM
that reached another thread did not stop the run.

The main thread now waits for the units in a loop with a 0.5 s timeout,
so it wakes up and runs the handler. It returns at once when a unit
raises. The signal mask stays as it is: child processes that the units
start must still receive SIGTERM.
@harenome
harenome merged commit 1cc37ac into ThinkFlowLab:main Nov 16, 2026
2 checks passed
@harenome
harenome deleted the fix/stop-ignored-in-worker-thread branch November 16, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: Stop is ignored when SIGTERM reaches a thread other than main

1 participant