[laughingman7743]

WHAT

Each filesystem has one request pool, which its files and parallel operations share:

• S3FileSystem owns a request pool of max_workers threads. max_workers now defaults to the max_pool_connections of the S3 client (10 unless configured, or that of the connection's client), instead of cpu_count() * 5. A client that the filesystem builds takes max_pool_connections from an explicit max_workers unless config_kwargs sets it.
• S3FileSystem._create_executor(max_workers) returns a scope on that pool instead of a new pool. At most max_workers calls of a scope are in the pool at once; the others wait in the scope, in submission order, with the context variables of their submitter, A call counts toward max_workers only once the pool has accepted it, so an interrupt of the submitter at any point leaves no place taken and none freed twice; an interrupt cancels the call unless it has started, and each interrupt after the pool accepted a call that has started can let one more call run at once. A call that ends lets the next waiting calls enter from its pool thread, before it ends there, so a release of the pool waits for them; once the pool takes no more calls, as at interpreter exit, that thread runs them itself. Calls cancelled while they wait never enter the pool. Shutting a scope down waits only for its own calls, so a file that is closed drains its own requests, also inside a transaction, and leaves the pool to the others. The files of open(), rm, recursive chmod, clear_multipart_uploads, and multipart copies take their scopes from it, so the max_workers of a file or call still bounds its requests, and the pool bounds all of them together.
• S3FileSystem.close() and its with block release the idle threads of the request pool; the filesystem stays usable, as with AioS3FileSystem.
• AioS3FileSystem:
  • It sizes its pool for blocking calls from the internal S3FileSystem, so its max_workers also defaults to the client's connection pool (it used to default to cpu_count() * 5).
  • The requests that it fans out run in the request pool through scopes: part copies and annotation copies of multipart copies (at most the copy's max_workers at once), and the DeleteObjects batches of _rm() (at most the filesystem's max_workers, as S3FileSystem.rm() sends, so that a large deletion does not queue ahead of the files' requests). So do the range reads and parts of open_async() files, at most their max_workers at once.
  • The calls that wait for those requests stay in its own pool: completions and aborts of copies, and commits and aborts of streamed files. The request pool runs no call that waits for the request pool.
  • Its close() also releases the request pool.
• The pandas and s3fs result sets release the threads of their filesystem when they close.
• S3Executor gains run(), run_shielded(), and wait_shielded(), moved from S3ThreadPool; they use only submit().
• In a forked child process, a file that the parent opened (S3File, AioS3File, AioS3StreamedFile) raises RuntimeError from its reads, writes, flushes, commit(), and discard(). Closing it, also at garbage collection, only marks it closed, without flushing, committing, or aborting, since the upload belongs to the parent.
• A call that a thread of the request pool submits to a scope of that pool, such as one of a file that garbage collection closes there, runs in that thread instead of waiting for a place that the thread holds. A place that frees passes to the oldest waiting call under the lock that submitters take, so calls enter in submission order also with concurrent submitters, and calls cancelled while they wait free their arguments at once and do not accumulate.
• S3File and AioS3StreamedFile built directly take max_workers from the filesystem (max_workers=None), and an S3File without an executor takes a scope of the request pool instead of a pool of its own.
• The cursors' s3_max_workers (pandas and Polars, sync, async, and aio) defaults to None, so their result sets' filesystems size the request pool from the client's connection pool.
• AioS3StreamedFile.close(), and leaving its async with block, wait for its range reads, commit, and abort also when the task is cancelled, any number of times, and raise the cancellation after they end; they used to return while a read request or an abort still ran.
• A parallel read that fails, or that an interrupt stops at any point, cancels the ranges that have not started and keeps no reference to the data of the others.
• A scope's shutdown and a streamed file's close wait only for calls that can still run: calls that entered the pool, wait for a place, run in a pool thread, or have started. They wait through done callbacks and recheck the futures every second, so a cancelled call ends the wait at once, and no placement of interrupts makes them wait forever.
• docs/filesystem.md and docs/aio.md state the interrupt guarantees. They hold for one interrupt at a time. If a further interrupt arrives during a cleanup, calls still do not wait forever, but data can stay referenced until garbage collection and a multipart upload can be left incomplete (I1 amendment).
• docs/filesystem.md, docs/aio.md, the benchmark notes, and the docstrings describe the request pool, the sizing, close(), and the files inherited across a fork.

Release note (behavior changes):

• The default max_workers of S3FileSystem and AioS3FileSystem is the max_pool_connections of the S3 client (10 by default) instead of cpu_count() * 5, and a client that the filesystem builds gets max_pool_connections = max_workers when max_workers is given and config_kwargs does not set it.

• The files and operations of a filesystem share its request pool of max_workers threads: concurrent files and operations together run at most that many requests, where each used to have threads of its own. With the defaults, a single large read or write runs 10 requests at once instead of cpu_count() * 5; docs/filesystem.md says how to raise max_workers and the client's max_pool_connections.

• S3FileSystem has close() and supports with.

• In a forked child process, files opened in the parent raise RuntimeError when used, and closing them does nothing else; before, a write there could commit or abort the parent's upload, and a file with requests in flight at the fork hung.

• The default s3_max_workers of the pandas and Polars cursors is None instead of cpu_count() * 5: their result sets read with the client's connection pool size (10 by default) unless it is given.

• S3File and AioS3StreamedFile constructed directly default to the filesystem's max_workers, and an S3File without executor shares the filesystem's request pool.

• A cancelled AioS3StreamedFile.close() waits for its requests, commit, and abort before it raises CancelledError.

WHY

Part of #1332: step 2b of the implementation decisions, decisions 2, 3, and 6 of the design, after the pool types were merged in #1357.

Pools per file and per call left the threads unbounded across concurrent files, and max_workers was not reconciled with the client's connection pool of 10 (#1322 question 5). One request pool per filesystem bounds them, and the scopes keep the per-file and per-call bounds that callers rely on (GH-954). It also gives the batch operations of #1322 (step 4) one pool to fan out into. The rejection of files inherited across a fork also covers a write session with parts in flight at the fork, which the review of #1353 found to hang.

TEST

Tested commit: f18ee10.

The design changes after ddc302b follow the specification, its decisions, its update, and the I1 amendment.

• just lint and just docs lint passed.
• AWS suites: uv run --env-file .env pytest -n 8 tests/pyathena/filesystem/ tests/pyathena/s3fs/ tests/pyathena/pandas/ tests/pyathena/polars/ tests/pyathena/aio/, including the AWS integration tests: 2449 passed, 14 skipped at f18ee10. An earlier run at f18ee10 had 4 errors from an expired local credential refresh, and one at 48fef9a failed test_info_file, which compares the local clock with S3's LastModified and passed in three reruns.
• Offline, --noconftest: the unit tests of test_s3_executor.py, test_s3.py, test_s3_async.py, test_s3_writer.py, and the s3fs and pandas result sets pass (971 passed). The only failures are 2 pandas tests that need AWS. The interrupt-injection tests also pass on Python 3.12.
• Interrupt injection (tests/pyathena/util.py: OpcodeInterrupter, interrupt_each_opcode) raises KeyboardInterrupt at each point where CPython checks for signals in S3ThreadPool.submit(), submit_to(), _S3PoolScope.submit() (entering and waiting), S3File._fetch_range(), and CPython's Future.cancel(). A second interrupt is also tested in the read cleanup, after a first interrupt of the wait. OpcodeInterrupter(action=...) interleaves a shutdown at each point of the scope's hand-on. After each interrupt the tests check the outcome, the bounds, and that the data is freed. Real-SIGINT tests use sigint_when().
• New tests:
  • TestS3PoolScope: test_window, test_waiting_call_context, test_shutdown_waits_for_own_calls, test_cancel_waiting_call, test_interrupted_submit, test_interrupted_submit_after_acceptance, test_waiting_calls_run_after_pool_shutdown, test_release_waits_for_waiting_calls, test_waiting_calls_enter_in_order, test_shutdown_waits_for_call_from_pool_thread, test_cancelled_waiting_call_frees_arguments, test_cancelled_waiting_calls_do_not_accumulate, test_shutdown_interrupted_while_waiting, test_submit_from_pool_thread, test_waiting_call_rejected_by_pool, test_submit_interrupted_at_each_opcode, test_rejected_submit_interrupted_at_each_opcode, test_shutdown_during_hand_on_keeps_waiting_call, test_shutdown_returns_once_calls_are_cancelled, test_shutdown_after_interrupted_cancel; TestS3ThreadPool: test_submit_interrupted_at_each_opcode, test_submit_to_interrupted_at_each_opcode.
  • TestS3FileSystem: test_request_pool_size, test_request_pool_size_of_connection, test_files_share_request_pool, test_file_without_executor_shares_request_pool, test_close_releases_threads, test_file_inherited_across_fork, test_read_interrupted_at_each_opcode_frees_ranges, test_read_interrupted_by_signal_frees_ranges, test_read_interrupted_twice_frees_ranges.
  • TestAioS3FileSystem::test_close_releases_request_threads; TestAioS3StreamedFile: test_file_inherited_across_fork, test_close_waits_for_cancelled_read (also cancels the close), test_exit_cancelled_while_aborting, test_close_interrupted_after_queuing_commit, test_exit_interrupted_while_queuing_abort, test_default_max_workers.
  • TestAthenaPandasResultSetMetadataReader::test_filesystem_request_pool_size; test_s3_max_workers_default for the pandas and Polars cursors.
• Updated tests: the streamed-file tests of cancelled commits and flushes now check that the close waits for the commit. The tests of freed read data wait, bounded, for the threads to leave the read frames. Tests that patched the task pool to watch part copies, parts, or the commit now patch the scope or submit_to. The mock connections carry a max_pool_connections (mock_connection()). test_copy_object_with_multipart_upload_interrupted_creation uses sigint_when(), with the same assertions.
• Mutation check: each new or changed test fails under its mutation, for example no window, LIFO entry, an untracked inline call, a drain that stops on cancellation, an untracked abort, a drain or shutdown that waits for an unqueued or cancelled call, no compaction, a new pool per file, a constant default size, or closing an inherited file normally.

Deferred (recorded on #1332):

• Step 3 (the fan-out primitive): an unclosed multipart file that garbage collection finalizes on the only thread of its pool, while one of its parts is queued in that pool, waits for that part.
• An interrupt of S3ThreadPool.submit() between tracking a future and adding its callback keeps the future in the pool's set until another release or shutdown. This is within I2.
• Under the I1 amendment, a second interrupt inside Future.cancel() during the read cleanup can keep one future's data until the traceback is dropped. This is a documented limit.
• The streamed drain's one-second recheck has no file-level test; test_shutdown_after_interrupted_cancel tests the same mechanism in the scope.

Not run:

• just benchmark test locally (Three test_fleet.py tests fail locally while passing in CI #1354 fixed the local failures; the Benchmark tooling workflow runs it).
• The AWS CI jobs, which start when the PR is marked Ready.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round 1 (implementation behavior): FINDINGS, repaired in acfe321 and 404490b.

Scope: git diff 892b213773e2abe09668e55b53b3795930d0fb1b..a0d908e1 (base = merge-base with master), all 12 files: s3_executor.py, s3.py, s3_async.py, the docs, the benchmark notes, and the tests. Perspectives: behavior and failure paths (interrupts, cancellation, rejection, release, transactions, fork, GC), resource bounds (threads, places, queued arguments), the callers of _create_executor and of the moved run helpers, simplicity and test quality (/code-review high, then /simplify with four angles).

Repaired here (acfe321):

• An interrupt after a call reserved its place and before it reached the pool, or between pool.submit and the place's callback, kept the place for good and left a future that shutdown(wait=True) waited for forever. A call now hands its place on in its pool thread (_run, _next), and an interrupt before the pool takes it cancels it and frees its place. test_interrupted_submit interrupts at the scope's first callback and fails against a0d908e.
• S3ThreadPool.release() could end the threads between two calls of a scope, after which the next waiting call started threads that nothing released; the next call now enters before the current one ends in the pool. test_release_waits_for_waiting_calls fails against a0d908e.
• A rejected call recursed through _leave/_enter once per waiting call; _next is a loop. Cancelled waiting calls are settled without entering the pool.

Kept: one more copy_context() per call (a few µs against ms-scale requests); the duplicated _inherited/_check_process of S3File and AioS3StreamedFile (a shared mixin would be a class restructuring); the max_workers check of S3File, which is the only check when a caller passes an executor of its own; rm(max_workers=) reaches _delete_objects through **kwargs, so that parameter is not dead.

[laughingman7743]

Self-review round 1: repaired in acfe321. _rm() put every DeleteObjects batch straight into the shared request pool, so a large deletion (2,000 batches for 2,000,000 keys) queued ahead of the range reads and parts of every open file. It now runs through a scope of the filesystem's max_workers, as S3FileSystem.rm() does. In 404490b the annotation copies of a multipart copy also run through a scope (of the copy's max_workers) instead of an asyncio semaphore around the pool; a failed copy still keeps the waiting ones from starting through its stop event.

[laughingman7743]

Self-review round 1: repaired in acfe321. A result set's filesystem (skip_instance_cache=True) was never closed, so its request-pool threads stayed for the life of the result set, where the per-file pools used to end with each file. The pandas and s3fs result sets now call close() on close, which releases the idle threads; a read that still runs keeps its threads until it ends.

Deferred (in the PR body): the cursors pass an explicit s3_max_workers of cpu_count() * 5 (pandas/result_set.py:871, polars/cursor.py:84, async_cursor.py:77), which decision 3 uses as given, so their pools can exceed the client's connection pool, as their per-file pools did. A follow-up issue will propose the cursors' default.

[laughingman7743]

Self-review round 2 (claims, callers, operations, evidence): one correction to the PR body; no code change.

Scope: git diff 892b213773e2abe09668e55b53b3795930d0fb1b..404490b7850b9fa83ac3d9630b1eaada97727d95, the PR body, docstrings, comments, docs/filesystem.md, docs/aio.md, the benchmark notes, and the commit messages.

Claims checked:

• Sizing: the default is client.meta.config.max_pool_connections (botocore's default 10); a built client gets max_pool_connections = max_workers unless config_kwargs sets it (test_request_pool_size, four combinations; test_request_pool_size_of_connection). The old defaults were cpu_count() * 5 in both constructors, as the release note says.
• The documented members of the request pool match the code: sync part copies, recursive chmod, rm, clear_multipart_uploads, and files take scopes from _create_executor; aio part and annotation copies, _rm() batches, and streamed-file reads and parts use scopes on the same pool; aio completions, aborts, and streamed-file commits and discards stay in the task pool.
• Before this change, closing an inherited write file in a child ran flush(force=True) and the commit, which completes or puts the parent's upload; the release note's statement holds.
• Correction: "No pool runs a call that waits on that same pool" was too broad: a copy's _abort waits in the task pool for a completion that already started there (unchanged behavior). The PR body now says the request pool runs no call that waits for it, which is what the design requires.

Existing callers: max_workers stays an int attribute; _create_executor(max_workers) keeps its signature; S3File(executor=None) still builds a pool of its own; S3ThreadPool.run() and the other helpers remain available through S3Executor. Removed private attributes: AioS3StreamedFile._pool.

AWS operator: fewer threads by default, and requests of concurrent files and operations of one filesystem now stay within its client's connection pool; no retry or request shape changes.

Evidence limits: only offline runs (--noconftest), since the AWS login session expired; the AWS integration suite, just benchmark test, and CI are still to run before Ready.

[laughingman7743]

Independent review (relayed): Codex CLI 0.166.0, model gpt-6-astra, codex exec -s read-only (session 01a1a775-7d13-7563-a1bd-b631cc33d045), static review of a detached snapshot of 404490b7850b9fa83ac3d9630b1eaada97727d95 against merge base 892b213773e2abe09668e55b53b3795930d0fb1b, with the diff and the repository conventions, without the PR text or the self-review conclusions; no edits, tests, or network; both worktrees unchanged afterwards. Covered: request scheduling, write-session cancellation, transactions, custom executors, close/discard, fork, GC, interpreter exit, result sets, docs, and the changed tests. Verdict: FINDINGS (six).

Findings 1 (P1), 2 (P2), and 4 (P2): the interrupt windows of _S3PoolScope.submit().

1. An interrupt after _futures.add() and before the try left a pending future, possibly holding a place, and shutdown() hung.
2. An interrupt after the pool accepted a call and before entered was set freed its place twice (_running could go negative), and untracked a running call.
3. A submitter paused between reserving a place and submitting let a waiting call return, and a release then missed it.

Verified by reading the bytecode-level order. All three came from reserving a place before the pool accepted the call. Repaired in fd7a6b2: a call counts toward max_workers only once pool.submit has returned (it joins _entered under the lock), and leaves when it ends, so an interrupt at any point leaves no place taken and none freed twice. At worst, one more call runs at once for a while. The interrupt handler only cancels a call that has not started. test_interrupted_submit now interrupts pool.submit and fails without that cancellation (the shutdown hangs).

[laughingman7743]

Repair and independent follow-up 1 (relayed).

Self-review of the fd7a6b2 repair, both perspectives: each changed path was traced in _S3PoolScope (submit, _run, _next) against interrupts between bytecodes of the submitter, rejection, and release. The docstrings and the PR body describe the bound, the exit drain, and the queued-commit test as changed.

Follow-up 1: Codex gpt-6-astra, read-only, static, 404490b7..fd7a6b24 with the whole scope, the pool, the work item, the write session, and the changed tests. The first attempt (session 01a1a785-5d00-7480-b0b3-b6920ebd033f) failed with "Selected model is at capacity"; the retry is session 01a1a78a-ba13-7a60-ba44-63d08716be74. Findings 3, 5, and 6 were resolved, and no lock-order deadlock was found from calling pool.submit() under the scope lock. FINDINGS:

• (P1) An interrupt after the pool accepted a call that had not started cancelled it and marked it notified directly. When the pool ran it, _S3WorkItem.run() called set_running_or_notify_cancel() again, which raises on a notified future, so _run() skipped _next(). The place stayed taken, the next call waited forever, and shutdown() hung. Verified against CPython's Future.set_running_or_notify_cancel. Repaired in ddc302b: the handler settles the cancelled call through its work item (call.item.run()), which claims it once, so the pool's later run is a no-op, and _run() frees the place in a finally. test_interrupted_submit_after_acceptance interrupts right after acceptance while the pool's only thread is held; it fails at fd7a6b2.
• (P2) Repeated interrupts after acceptance of calls that have started let one uncounted call run per interrupt, so "at worst, one more call" overpromised. Repaired in ddc302b: the docstring states that each such interrupt can let one more call run at once.
• (P2, pre-existing) S3ThreadPool.submit() interrupted between tracking a future and adding its callback keeps the future in _futures, so a later release() waits for it forever and idle threads stay until another release or shutdown. This is the pool's own window since Own the AioS3FileSystem pool and keep aio DataFrame reads off the event loop #1209 (release() drops only futures that are done). The scope reaches it as files did before. Deferred: it is not introduced here and needs its own fix in S3ThreadPool; it will be listed in a follow-up issue.

A second follow-up review covers ddc302b.

[laughingman7743]

Repair and independent follow-up 2 (relayed).

Self-review of the ddc302b repair, both perspectives: the handler calls call.item.run() only after a successful future.cancel(). So it either loses the work item's claim, or finds the future cancelled and only notifies it. It never runs the user's callable in the submitter's thread. The docstrings state the per-interrupt allowance, and the offline suites pass twice in a row.

Follow-up 2 (Codex gpt-6-astra, read-only, session 01a1a794-616d-7af2-a5fc-468392da66df, static, fd7a6b24..ddc302bd with the whole scope, _S3WorkItem, and the new test): CLEAN.

• P1 is resolved for later runs by the pool and for draining the deque; _run() frees the place in finally.
• No callable runs in the submitter's thread.
• The docstrings are accurate.
• The test interrupts after acceptance while the only worker is held, and fails at fd7a6b2 because the duplicate notification raises and the next call times out.

[laughingman7743]

Independent review (relayed), finding 3 (P2): at interpreter exit ThreadPoolExecutor.submit() raises, so the calls waiting in a scope failed with RuntimeError instead of draining as the calls queued in a pool of their own used to.

Verified (concurrent.futures.thread._python_exit sets the shutdown flag before joining the workers). Repaired in fd7a6b2: when the pool rejects a waiting call, the thread of the call that ended runs it. test_waiting_calls_run_after_pool_shutdown fails without this.

[laughingman7743]

Independent review (relayed), finding 5 (P2): test_flush_cancelled_while_queued_then_exit_with_exception blocked a part in the request pool, but the commit now runs in the filesystem's pool, so it was not queued, and the tests waited on the file's scope, which does not cover the commit or the abort.

Verified: holding the thread before open_async() even blocked the open, which runs in that pool. Repaired in fd7a6b2: the test replaces the filesystem's pool with one thread and holds it after the write, so the commit really waits for a thread. Both commit tests wait for fs._thread_pool. Without the commit's reservation before it is queued, the test fails (the upload is aborted).

[laughingman7743]

Independent review (relayed), finding 6 (P3): an inherited streamed file's read(0) and a read at EOF returned before _fetch_range(), bypassing the RuntimeError.

Verified against fsspec's AbstractAsyncStreamedFile.read. Repaired in fd7a6b2: the public read() checks the process, and the fork test calls read(0), which fails without the check.

[laughingman7743]

Self-review round 1 after the specification (ddc302bd..80ad4f2b): FINDINGS, repaired.

Scope: the changes made against the specification and its decisions, across all changed files. The changes cover S3, S8, F2, F5, the cursor default, the injection helper, and the barrier cases. Perspectives: behavior under the failure matrix, consumers, and test quality. Ran /code-review (high) and /simplify (two agents).

/code-review (10 findings), all repaired in cebbe07:

• T1: a file closed by garbage collection on a request-pool thread waited for a place in the pool it held. A pool thread's call to its own pool's scope now runs inline (S3ThreadPool.owns_current_thread()).
• S5: a waiting call that the pool failed to take for a reason other than shutdown was lost; it now fails.
• S8: cancellation was O(n) per call under the lock; it is now O(1), and the arguments are freed at once.
• F1: S3File and AioS3StreamedFile built directly kept cpu_count() * 5, and S3File kept its own pool. max_workers=None raised in the streamed file.
• _create_executor() returns the scope type, so _drain() needs no cast.
• The fork check called os.getpid() on every read and write; it now compares a fork counter.
• The tests asserted private attributes; they now check observable behavior, except the documented barrier hook of S3.

Kept with reasons:

• Idle threads of a cached filesystem stay until close or GC (L3, L4).
• The polars chunked-read filesystem is collected (L4).
• No _ForkGuard mixin (a class restructuring).

/simplify: sigint_when/poll_until replace duplicated SIGINT and polling code (80ad4f2). A dict-as-FIFO and an int counter were rejected: popping the front of a dict degrades, and an int cannot skip uncounted calls.

Found while validating:

• the mock filesystem of the TestS3File helpers lacked a request pool, and test_s3.py hung (fixed);
• LocalBucketFileSystem lacked close() for the pandas result set (fixed, 7b77f37).

My own misjudgments, corrected before commit:

• the injection model first injected inside with statements, then at the next instruction past a try boundary; both gave false positives;
• I called the concurrent.futures.wait() waiter behaviour a CPython bug before verifying it. It only delays freeing until the cyclic GC, so it is recorded as such and not reported upstream.

[laughingman7743]

Self-review round 2 after the specification (ddc302bd..80ad4f2b): CLEAN after the corrections below.

Claims checked:

• docs/usage.md: Polars uses s3_max_workers for CSV results read without chunksize and for UNLOAD results read in chunks through PyArrow without read options (polars/result_set.py:1237-1240, :874). The old text said UNLOAD results do not use it, which was wrong before this change.
• The default None reaches S3FileSystem(max_workers=None), which takes the connection client's max_pool_connections. test_filesystem_request_pool_size checks the default path, and test_s3_max_workers_default checks the six cursors.
• F2: a cancelled close() now raises CancelledError only after the commit or the read requests end (test_commit_cancelled, test_close_waits_for_cancelled_read). This is a behaviour change for callers, listed in the PR body.
• F5: on a real Ctrl-C during the wait, concurrent.futures.wait() leaves its waiter on the futures, so the data is freed by the cyclic GC (verified with CPython alone). This is stated in the test and will be added to the spec.
• S9: the final return of _S3PoolScope.submit() is not a point where CPython checks signals, so the gap listed in the spec cannot occur. The spec will be updated.

Evidence: offline suites stable (961 passed and the two AWS-only failures); the affected AWS suites passed at 7b77f37 (2435 passed, 14 skipped); every new test fails under its mutation.

[laughingman7743]

Independent review (relayed): Codex CLI (codex exec -s read-only, default model), session 01a1aac8-b428-7872-86be-272f100cc2a9. Static review of ddc302bd668b8990982656889251a631efc2c38a..80ad4f2b4641abc948978faa01ff01f0df11316c against the specification on #1332, from a detached snapshot (left clean). Coverage: pool and scope lifecycle, cancellation, interrupts, rejection, exit, fork, GC, file cleanup, defaults and callers, test helpers, tests, docs. Verdict: FINDINGS (13).

Repaired in dee243f:

1. F2/I1: a cancelled close() returned while a read request ran. The drain now waits through repeated cancellations and then raises (test_close_waits_for_cancelled_read cancels the close).
2. F2/I3: an abort running after a cancelled exit, and a commit started by an interrupted submit(), were untracked. Both are now futures tracked before queuing (submit_to). New test: test_exit_cancelled_while_aborting.
3. S1/S3/S5: an inline call was not tracked by the scope's shutdown. It is now tracked; test_shutdown_waits_for_call_from_pool_thread.
4. F4: an at-fork hook registered earlier could close an inherited file before the counter moved. Reverted to the PID comparison.
5. F5: a second interrupt during the read cleanup kept the futures. They are now dropped in a finally, and popped so no frame variable holds one.
6. S4: an inline call now runs in the captured context.
7. S8: tombstones are compacted; test_cancelled_waiting_calls_do_not_accumulate.
8. Instruction.jump_target is 3.13+; the injector uses argval, and the injection tests pass on 3.12.
9. The submit_to injection test no longer notifies the future itself.
10. The pool injection test now bounds overlap at max_workers + 1 (I2), and the scope test requires cancellation of an interrupted waiting call (S9).
11. The timed barrier test was replaced by a deterministic FIFO test; a place now passes to the oldest waiting call under the submitters' lock.

Each new or changed test fails under its mutation.

Deferred:

• 4 (T1): a garbage-collected unclosed multipart file, finalized on the only pool thread while one of its parts is queued in that pool, waits for the part. Inline execution covers new submissions only. This needs the commit to stop waiting on its own pool from a pool thread, which belongs to the fan-out primitive (step 3, B1-B6). It will be recorded on Consolidate the filesystem executors before parallel batch operations #1332.
• 10: limits of the injection model, which stay limits and go into the spec. It approximates CPython's signal checks: interrupts at a return f() inside a try are injected before the call, and backward-jump targets also fire on their first forward entry (extra points, which only add cases).

[laughingman7743]

Independent follow-up (relayed): Codex CLI (codex exec -s read-only), session 01a1aae6-8fe1-7b22-891b-4b231270bf05, static review of 80ad4f2b..dee243f7. Verdict: FINDINGS (1 P1, 5 P2). Repaired in db9c9e0:

• P1 (F2/I1): an abort whose queuing raised left _discard_future pending, and the drain waited forever. It is now cancelled; new test test_exit_interrupted_while_queuing_abort, run in a thread so that a regression fails instead of hanging.
• P2 (F5): a second interrupt during the cancellation kept the first error, as its __context__, together with the futures of the wait. clear_frames() now runs first in a finally. New test test_read_interrupted_twice_frees_ranges: a wait that holds its own set of the futures, as CPython's does, is interrupted, then a second interrupt is injected at each point.
• P2 (P6): the interrupted pool call now runs while the others overlap it. The test fails with a pool of max_workers + 1 threads.
• P2 (S5): the test waits until the shutdown has begun.
• P2 (F2/I3): new test test_close_interrupted_after_queuing_commit interrupts after the commit has started. It fails when the commit is tracked after its queuing.
• P2 (S3): accepted, and my earlier claim is corrected. The place was already handed on under one lock, and no submitter enters while a call waits, so the old code was FIFO too. Freeing the place under the same lock only tidies the code. There is never a free place while a call waits, so the concurrent-submitter rule holds by construction and cannot be raced in a test. test_waiting_calls_enter_in_order checks the order of waiting calls (it fails under LIFO), and its comment no longer claims more.

Each new or changed test fails under its mutation. Offline suites: 967 passed; the 2 failures need AWS.

[laughingman7743]

Independent follow-ups 2 and 3 (relayed): Codex CLI (codex exec -s read-only), static review.

Follow-up 2: dee243f7..db9c9e0f, session 01a1aae6-… second run. Verdict FINDINGS: 2 nested-interrupt items and 3 timing-dependent tests.

• The tests were made independent of timing in 770541b.
• The nested-interrupt items led to the I1 amendment, decided by the maintainer. Guarantees hold for one interrupt at a time; under further interrupts, only "no call waits forever" holds. The amendment also compares other libraries.
• 9898f17: the streamed drain waits only for a commit or abort whose queuing returned or that has started. The docs state the guarantees.
• The Future.cancel() nested-interrupt retention is a documented limit.

Follow-up 3: db9c9e0f..9898f176. Verdict FINDINGS. Repaired in b9763f8:

• P1 (I1/S5): a pool-rejected submission, interrupted before its cleanup cancelled it, left a pending future that the scope's shutdown and the drain waited for forever. The scope now waits only for calls that have entered the pool, wait for a place, run inline, or have started, and cancels any other. New test test_rejected_submit_interrupted_at_each_opcode; under the mutation it fails by timeout.
• Docs: the cancellation on interrupt is now narrowed to parallel range reads. rm's DeleteObjects batches are not cancelled; that belongs to the step 3 primitive (B3). docs/aio.md now states the limit for an interrupt during the cleanup of an interrupted commit.

Not changed, with the reason: three test findings say a scheduling stall of 100 ms or more could let a mutant pass. They cannot make a test fail spuriously, and every test fails under its mutation in normal runs. Making them deterministic would need hooks into private state.

[laughingman7743]

Independent follow-ups 4–7 (relayed): Codex CLI (codex exec -s read-only), static reviews against the amended I1.

• Follow-up 4 (9898f176..b9763f80): FINDINGS. Repaired in 6a6e737:
  • S7, no interrupt needed: a waiting call that a pool thread runs inline after the pool rejected it was untracked, so a concurrent shutdown could cancel it. It is now tracked in _inline. New test test_shutdown_during_hand_on_keeps_waiting_call uses a new OpcodeInterrupter(action=...) to interleave a shutdown at each point of _next(); it fails under the mutation.
  • I1: a cancelled future left unnotified by a nested interrupt is no longer waited for.
• Follow-up 5 (b9763f80..6a6e7371): FINDINGS. A shutdown already waiting in concurrent.futures.wait() still waited for such a future. Repaired in 48fef9a: the shutdown waits through done callbacks, which cancel() runs at once. New test test_shutdown_returns_once_calls_are_cancelled: a shutdown no longer waits for the pool to reach a cancelled call. The SIGINT test now signals once the shutdown has added its callback.
• Follow-up 6 (6a6e7371..48fef9af): FINDINGS. An interrupt inside CPython's Future.cancel(), between setting the state and running the callbacks, skips them. Repaired in 524b22b: the scope shutdown and the streamed drain also recheck done() every second, as fsspec's sync() does. New test test_shutdown_after_interrupted_cancel injects at each point of Future.cancel(); it times out under the mutation. The drain uses the same mechanism without a file-level test.
• Follow-up 7 (48fef9af..524b22b0): the repair holds; no lost wake-up beyond the recheck. The two remaining items are intended behaviour: an interrupt of the wait in close() or shutdown() stops the wait, while running requests finish, as test_shutdown_interrupted_while_waiting checks. f18ee10 documents this in docs/filesystem.md. The review loop ends here.