Exit forked processes immediately to avoid a SQLite mutex deadlock at exit - #801
Merged
Conversation
When a thread is killed while it waits in SQLite's busy handler, the
kill unwinds through SQLite's C frames and leaves the connection mutex
locked (the sqlite3 gem invokes the handler with a bare rb_funcall).
Ruby kills every leftover thread at process exit -- a heartbeat or
maintenance timer mid-query, a pool thread running a job past the
shutdown timeout -- and then finalizes every remaining object, so a
forked process on SQLite under write contention could deadlock inside
sqlite3_close_v2 during exit and linger forever instead of exiting.
That's where the orphaned processes in the test suite and the
long-standing Puma plugin hang on SQLite came from: about 1 in 10 runs
of the plugin's "supervisor dies" test left a zombie scheduler behind,
stuck with this backtrace:
ruby_cleanup -> rb_objspace_call_finalizer -> rb_data_free
-> sqlite3_close_v2 -> pthread_mutex_lock (never returns)
Exit forked processes with exit!, like Puma's cluster workers and our
own QUIT handler already do, skipping at-exit hooks and finalizers:
everything the process needs to do on shutdown has already run by then.
Only the success path exits this way, so an error raised out of the
fork's block still gets reported and exits non-zero. The Puma plugin's
supervisor fork gets the same treatment, which is what lets Puma's
Process.wait return during shutdown.
With this, that same test leaves 0 orphans in 15 runs, and the full
suite, the three lifecycle suites (which assert the forks' exit
statuses) and the Puma plugin tests all stay green.
The busy-handler mutex leak itself is a sqlite3-ruby bug (reported
separately with a standalone reproduction).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41
The test harness passes -b tcp://127.0.0.1:<free port> since the dynamic port allocation was added, but the plugin test configs still set port 3000, so every test Puma bound the default port as well and collided with whatever else was using it on the same machine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the orphaned/zombie Solid Queue processes on SQLite: the long-standing Puma plugin hang, several of the sqlite-only oddities catalogued in #797 (the fiber
termsupervisor outliving its teardown, the first-test CI hangs), and the ~1-in-10 zombie scheduler left behind by the plugin's "supervisor dies" test.Root cause (found via gdb on a live zombie, then reproduced deterministically): killing a Ruby thread while it waits in SQLite's busy handler leaks the connection mutex — the sqlite3 gem invokes the handler with a bare
rb_funcall, so the kill unwinds through SQLite's C frames andsqlite3_stepnever releases the mutex. Ruby kills leftover threads at process exit (a heartbeat/maintenance timer mid-query, a pool thread pastshutdown_timeout) and then finalizes every remaining object, so the exit deadlocks insidesqlite3_close_v2:Rails' SQLite adapter uses a Ruby-level busy handler in every configuration except 7.x's plain
timeout:(retries:on 7.x,busy_handler_timeoutfrom 7.2), so this bites exactly when several processes contend on one database — which is what the integration tests do, and what a multi-process production deployment on SQLite does. Kill-a-thread-mid-wait reproduces the hang 2/2 in a 25-line script with no Rails involved; the control (no kill) exits cleanly 2/2. The gem-level bug is being reported upstream to sqlite3-ruby separately.The fix: forked processes exit with
exit!(0)once their block has completed — at-exit hooks and finalizers are skipped, exactly like Puma's cluster workers and our own QUIT handler. Shutdown work (deregistering, releasing claims, lifecycle hooks) has all run by then. An error raised out of the block still propagates and exits non-zero, unchanged. The Puma plugin's supervisor fork gets the same treatment, which is what lets Puma'sProcess.waitreturn on shutdown.Second commit: the plugin test configs still set
port 3000, so every test Puma bound the default port in addition to the dynamically allocated one from #733, colliding with anything else on :3000.Verification: new unit test (fork exits with status 0 without running inherited
at_exithooks); full sqlite suite 356 green; forked/fiber/async lifecycle suites green (they assert the forks' exit statuses); Puma plugin tests green; the "supervisor dies" test run 15× leaves 0 Solid Queue processes behind (verified via/proccmdline scan), vs ~1 in 10 before.🤖 Generated with Claude Code
https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41