Skip to content

Exit forked processes immediately to avoid a SQLite mutex deadlock at exit - #801

Merged
rosa merged 2 commits into
mainfrom
exit-immediately-from-forked-processes
Aug 31, 2026
Merged

Exit forked processes immediately to avoid a SQLite mutex deadlock at exit#801
rosa merged 2 commits into
mainfrom
exit-immediately-from-forked-processes

Conversation

@rosa

@rosa rosa commented Aug 31, 2026

Copy link
Copy Markdown
Member

Fixes the orphaned/zombie Solid Queue processes on SQLite: the long-standing Puma plugin hang, several of the sqlite-only oddities catalogued in #797 (the fiber term supervisor outliving its teardown, the first-test CI hangs), and the ~1-in-10 zombie scheduler left behind by the plugin's "supervisor dies" test.

Root cause (found via gdb on a live zombie, then reproduced deterministically): killing a Ruby thread while it waits in SQLite's busy handler leaks the connection mutex — the sqlite3 gem invokes the handler with a bare rb_funcall, so the kill unwinds through SQLite's C frames and sqlite3_step never releases the mutex. Ruby kills leftover threads at process exit (a heartbeat/maintenance timer mid-query, a pool thread past shutdown_timeout) and then finalizes every remaining object, so the exit deadlocks inside sqlite3_close_v2:

ruby_cleanup → rb_objspace_call_finalizer → rb_data_free → sqlite3_close_v2 → pthread_mutex_lock (never returns)

Rails' SQLite adapter uses a Ruby-level busy handler in every configuration except 7.x's plain timeout: (retries: on 7.x, busy_handler_timeout from 7.2), so this bites exactly when several processes contend on one database — which is what the integration tests do, and what a multi-process production deployment on SQLite does. Kill-a-thread-mid-wait reproduces the hang 2/2 in a 25-line script with no Rails involved; the control (no kill) exits cleanly 2/2. The gem-level bug is being reported upstream to sqlite3-ruby separately.

The fix: forked processes exit with exit!(0) once their block has completed — at-exit hooks and finalizers are skipped, exactly like Puma's cluster workers and our own QUIT handler. Shutdown work (deregistering, releasing claims, lifecycle hooks) has all run by then. An error raised out of the block still propagates and exits non-zero, unchanged. The Puma plugin's supervisor fork gets the same treatment, which is what lets Puma's Process.wait return on shutdown.

Second commit: the plugin test configs still set port 3000, so every test Puma bound the default port in addition to the dynamically allocated one from #733, colliding with anything else on :3000.

Verification: new unit test (fork exits with status 0 without running inherited at_exit hooks); full sqlite suite 356 green; forked/fiber/async lifecycle suites green (they assert the forks' exit statuses); Puma plugin tests green; the "supervisor dies" test run 15× leaves 0 Solid Queue processes behind (verified via /proc cmdline scan), vs ~1 in 10 before.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41

rosa and others added 2 commits August 31, 2026 10:17
When a thread is killed while it waits in SQLite's busy handler, the
kill unwinds through SQLite's C frames and leaves the connection mutex
locked (the sqlite3 gem invokes the handler with a bare rb_funcall).
Ruby kills every leftover thread at process exit -- a heartbeat or
maintenance timer mid-query, a pool thread running a job past the
shutdown timeout -- and then finalizes every remaining object, so a
forked process on SQLite under write contention could deadlock inside
sqlite3_close_v2 during exit and linger forever instead of exiting.
That's where the orphaned processes in the test suite and the
long-standing Puma plugin hang on SQLite came from: about 1 in 10 runs
of the plugin's "supervisor dies" test left a zombie scheduler behind,
stuck with this backtrace:

    ruby_cleanup -> rb_objspace_call_finalizer -> rb_data_free
      -> sqlite3_close_v2 -> pthread_mutex_lock (never returns)

Exit forked processes with exit!, like Puma's cluster workers and our
own QUIT handler already do, skipping at-exit hooks and finalizers:
everything the process needs to do on shutdown has already run by then.
Only the success path exits this way, so an error raised out of the
fork's block still gets reported and exits non-zero. The Puma plugin's
supervisor fork gets the same treatment, which is what lets Puma's
Process.wait return during shutdown.

With this, that same test leaves 0 orphans in 15 runs, and the full
suite, the three lifecycle suites (which assert the forks' exit
statuses) and the Puma plugin tests all stay green.

The busy-handler mutex leak itself is a sqlite3-ruby bug (reported
separately with a standalone reproduction).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41
The test harness passes -b tcp://127.0.0.1:<free port> since the dynamic
port allocation was added, but the plugin test configs still set port
3000, so every test Puma bound the default port as well and collided
with whatever else was using it on the same machine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41
@rosa
rosa merged commit 4f77fb6 into main Aug 31, 2026
228 of 232 checks passed
@rosa
rosa deleted the exit-immediately-from-forked-processes branch August 31, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant