Add s3fs as an API of the filesystem benchmark suite by laughingman7743 · Pull Request #1319 · pyathena-dev/PyAthena

laughingman7743 · 2026-10-23T13:16:55Z

WHAT

The filesystem suite of the benchmark project gains a third API, s3fs, which runs the suite's transfers through s3fs.S3FileSystem, and two operations, get and put, which transfer the object through a local file with get_file() and put_file().

Case.api accepts s3fs for the filesystem suite, and --api s3fs selects it. The case's max_workers becomes s3fs's max_concurrency and its pool_connections the max_pool_connections of s3fs's own client, created from the session's frozen credentials with the trial's region.
get downloads to a file in the trial's temporary directory, which the supervisor now passes to the trial process, measures as the peak temp dir, and removes, and checks the local file's size; the trial removes the file after the event loop's lag sampling stops; put uploads a sparse local file of the object's size, so that no disk read bounds the upload, and checks the object as writes do. PyAthena's get_file()/put_file() take block_size, s3fs's take chunksize; the aio API runs both in asyncio.to_thread() as it runs reads and writes.
s3fs copies objects above 150 MiB (s3fs.core.MANAGED_COPY_THRESHOLD) as a multipart copy with parts of block, one part at a time, so the single-CopyObject threshold that marks a copy unsupported is now per API (SINGLE_COPY_THRESHOLDS), and the reason names it.
The s3fs client is connected at setup, as the PyAthena filesystems take the connection's client when they are created, so no transfer measures a client's creation. A run in an environment without s3fs records the s3fs API's cases as unsupported.
The s3fs API runs in its own environment. The benchmark's new s3fs dependency group joins the uv conflict set with dev, polars1, and polars2, because aiobotocore pins botocore to a few patch releases; .venv-s3fs beside the default environment holds it, the CloudFormation host syncs /opt/pyathena/.venv-s3fs, and just benchmark test runs the tests there too. jobs marks the s3fs API's jobs for the s3fs environment without a suffix to their IDs, since the API names it, and rejects --environment with another environment for them.
The README documents the API, the operations, the environment, and the disk bound of get.

The locked versions of the dev, --no-dev, polars1, and polars2 resolutions are unchanged from the base commit. The s3fs environment resolves aiobotocore 3.9.2 with the same botocore 1.43.102 as the other environments.

WHY

Refs #1316, part of #1315. The comparison with s3fs needs its transfers in the same suite, host, and selection as PyAthena's filesystems, and s3fs's parallel paths are get_file(), put_file(), and the managed copy, which the suite did not run.

TEST

Tested at the PR's head commit.

just benchmark lint and just benchmark test: 283 passed, 2 skipped in .venv and .venv-polars2; 284 passed, 1 skipped in .venv-s3fs, where the test that checks SINGLE_COPY_THRESHOLDS["s3fs"] against s3fs.core.MANAGED_COPY_THRESHOLD runs.
just lint and markdownlint-cli2 benchmarks/README.md: clean.
uv export --locked of the dev, --no-dev, polars1, and polars2 resolutions, for Python 3.11, 3.12, 3.13, and 3.15, equals the base commit's (02730a2a) in every version.
Each new test fails when the behavior it names is removed: the chunksize mapping, the s3fs environment marking of jobs, the s3fs copy threshold, the local size check of get, and the missing-s3fs error each failed exactly one test.
Live, in .venv-s3fs against the CI bucket with a temporary script: read, get, write, and put of a 16 MiB object through filesystem() and transfer() for the s3fs, sync, and aio APIs, and the s3fs copy of a 160 MiB object (20 parts of 8 MiB); each transfer returned and HeadObject reported the object's size. Repeated for the s3fs and sync APIs after the s3fs client became connected at setup. Timings are not meaningful from this network.
Not run: a fleet run on EC2, which is Measure the filesystem suite against s3fs on c8gn.2xlarge #1318, so the CloudFormation user data change is checked by cfn-lint only; PyAthena's copies above 5 GiB, which the change does not touch.

🤖 Generated with Claude Code

The filesystem suite gains the s3fs API, which runs the same transfers through s3fs.S3FileSystem, and the get and put operations, which transfer one object through a local file with get_file() and put_file(), the paths that s3fs parallelizes with max_concurrency. The case's max_workers is s3fs's max_concurrency, and its connection pool size is s3fs's max_pool_connections. s3fs copies objects above 150 MiB as a multipart copy, so the single-copy threshold is per API. s3fs depends on aiobotocore, which pins botocore to a few patch releases, so the benchmark's s3fs dependency group joins the uv conflict set and is installed in its own environment, benchmarks/.venv-s3fs locally and /opt/pyathena/.venv-s3fs on the benchmark hosts. The jobs command marks the s3fs API's jobs for that environment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… the README (GH-1316) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 · 2026-10-23T13:19:36Z

    return operation()


+def filesystem(settings: Settings, case: Case, session_: Any, connection: Any) -> Any:


Self-review round one (behavior and implementation)

Base 02730a2a88ec3babeca114aaeac694a0d168db2d, head fb8b384faba03524fdc58777c89b9e77cdeedcca (initial pass at c7270b8b, repair at fb8b384f).

Covered: benchmarks/pyathena_bench/{cases,filesystem,fleet,__main__}.py, the s3fs dependency group and uv conflict set (pyproject.toml, benchmarks/pyproject.toml, uv.lock), justfile, the CloudFormation user data, benchmarks/README.md, and the tests in benchmarks/tests/{test_config,test_filesystem,test_fleet}.py.

Verified against the resolved s3fs 2026.9.0: cp_file(block=...) reaches _copy_managed(block=...) only above MANAGED_COPY_THRESHOLD (150 MiB), which the per-API unsupported rule guarantees, and a live 160 MiB copy ran 20 part copies; get_file(chunksize=...) and put_file(chunksize=...) bind s3fs's chunksize parameters; the constructor accepts max_concurrency, config_kwargs, and client_kwargs. AioS3FileSystem._get_file/_put_file run the synchronous filesystem's methods in the filesystem's thread pool, as the docstring says. The frozen session credentials are trial-scoped, and the sparse upload file is created with truncate(), which no disk read bounds.

Result: FINDINGS, both in the README, repaired in fb8b384f; see the threads on benchmarks/README.md. No code finding. Not run: an EC2 fleet run (#1318).

Round one repair follow-up (narrow, git range-diff 02730a2a..c5ccb8a4 02730a2a..1acf85d9) for the independent-review repairs in 1acf85d9. Traced hunks: filesystem() now calls s3fs's connect() (a sync_wrapper(set_session) run on fsspec's IO loop; the live smoke of s3fs/sync read, get, write, and put passed afterwards); run_filesystem() takes the supervisor's temp_dir, writes object there, and unlinks it in the inner finally after the heartbeat stops, also when the transfer raised; supervise() passes {**payload, 'temp_dir': ...} to the child only, so the trial_start checkpoint and the parent's payload['trial'] uses are unchanged; worker() ignores the field for query cases; run() records missing_api_reason(case) as unsupported before the completed check, consistent with the other unsupported records. Tests: test_s3fs_cases_are_unsupported_where_s3fs_is_not_installed, the payload_temp_dir assertion of the supervisor test, the connected assertion, and the file-removal assertions of TestRunFilesystem exercise each hunk; just benchmark test passed in the three environments (283/283/284 passed). Result: CLEAN for the repair.

laughingman7743 · 2026-10-23T13:19:36Z

- `write`: writes through a file opened with `block_size`, which uploads the object as a multipart upload with parts of that size;
- `copy`: `cp_file()`, or `_cp_file()` of `AioS3FileSystem`, which copies an object over 5 GiB as a multipart copy with parts of `block_size`; the copies of smaller objects, single CopyObject requests, are recorded as unsupported; the copies exclude tags and annotations, which the generated objects do not have.
+- `read`: one `read()` of the whole object from a file opened with `block_size`; PyAthena's files read it as parallel range requests of that size and hold the parts and the joined object, about twice the object's size, in memory, while s3fs's file reads it as one stream through fsspec's read-ahead cache;
+- `get`: `get_file()` to a local file in a temporary directory of the trial; PyAthena's reads one block at a time through a file opened with `block_size`, whose `bytes` cache requests the next block with the first read and one block at a time afterwards, and s3fs's reads parallel ranges of `chunksize` with `max_concurrency` requests at a time;


Round one F1 (factual claim, repaired in fb8b384f): the initial text said PyAthena's get downloads "through a file opened with block_size" without saying how it reads. S3FileSystem.get_file() reads remote.read(remote.blocksize) in a loop (pyathena/filesystem/s3.py, the while data := remote.read(...) loop), and the default bytes cache fetches the block plus the next one on the first read and one block at a time afterwards (fsspec.caching.BytesCache._fetch), so its range requests are sequential after the first fetch, unlike the whole-object read(), which _fetch_range() splits into parallel requests. The text now says so; the comparison with s3fs's parallel get_file() depends on it. This read pattern is a candidate improvement for #1315, not a change for this PR.

laughingman7743 · 2026-10-23T13:19:36Z

 Both event loops keep Python's default executor, which runs only the trial's `asyncio.to_thread()` call.
 Before #1159, the parallel requests of the `aio` API ran in the default executors, whose `min(32, os.cpu_count() + 4)` threads bound them.
+The `s3fs` API calls s3fs on the trial's thread; each call waits for a coroutine on fsspec's IO loop, where aiobotocore sends the requests, so the trial's event loop lag is not sampled.
+Its cases run in the `s3fs` environment: `jobs` marks them for it, without a suffix to their IDs, and rejects `--environment` with another environment for them; a `run` of them in another environment records each trial as failed.


Round one F2 (factual claim, repaired in fb8b384f): the initial text said run and plan need the s3fs environment's interpreter. plan only expands the matrix and never constructs a filesystem, so it runs anywhere; a run of s3fs cases in another environment raises RuntimeError from filesystem() inside each trial process, which the runner records as a failed trial. The text now describes that behavior.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 · 2026-10-23T13:22:53Z

 polars1 = ["polars>=1.39.0,<2"]
 polars2 = ["polars>=2,<3"]
+# The filesystem suite's s3fs API; aiobotocore pins botocore, so the group resolves apart.
+s3fs = ["s3fs>=2026.9.0"]


Self-review round two (claims, callers, operations, evidence)

Base 02730a2a88ec3babeca114aaeac694a0d168db2d, head c5ccb8a4d036b428d55d2371b041e8f5816cd2c6 (full claim audit at fb8b384f, repair at c5ccb8a4).

Claims checked:

PR body: the dev resolution's certifi moved with this change. False, corrected. The base 02730a2a is the merge of Bump certifi from 2024.12.14 to 2026.10.19 #1314, which bumped certifi to 2026.10.19; my first comparison used a stale checkout (c0a938ec). Against the base, uv export --locked of the dev, --no-dev, polars1, and polars2 resolutions for Python 3.11, 3.12, 3.13, and 3.15 is unchanged in every version; the PR body now says so.

aiobotocore pins botocore to a few patch releases. PyPI metadata of aiobotocore 3.10.0: botocore>=1.43.115,<1.43.118. The lock's s3fs fork resolves aiobotocore 3.9.2 with the botocore 1.43.102 of the other environments, so the conflict set keeps them equal today.

s3fs reads a whole read() as one stream. default_cache_type="readahead" (s3fs/core.py), and fsspec.caching.ReadAheadCache._fetch requests [start, end + blocksize) with one fetcher call, one GetObject.

s3fs's file uploads one part at a time; its managed copy copies one part at a time above 150 MiB. S3File._upload_chunk loops over self._call_s3("upload_part", ...); MANAGED_COPY_THRESHOLD = 150 * 2**20; _copy_managed awaits each upload_part_copy in sequence (the source comments that gather caused part-size errors).

README: a get is bounded by the volume's throughput. Unverified: the written file goes through the page cache, whose dirty limits exceed the 1 GiB objects of the selection, so the volume need not bound it. Narrowed in c5ccb8a4.

A put reads a sparse file that no disk bounds. truncate() creates holes that read as zeros; the live 16 MiB put uploaded the full size through every API.

Existing callers. transfer() gained a required local argument; its callers are run_filesystem() and the tests. run_filesystem(), Case, and the IDs of the existing sync/aio read/write/copy cases are unchanged, so the 4.1.0 and 4.2.0 history entries stay comparable.

Trial loop. fsspec's sync() raises only when called on its own IO loop's thread; the trial's loop is another, and the live smoke ran the s3fs calls inside asyncio.run().

Evidence. Five mutations (the chunksize mapping, the environment marking, the s3fs threshold, the local size check of get, the missing-s3fs error) each failed exactly one new test; just benchmark test passed in the three environments at c7270b8b, and the later commits touch the README only.

Limits: s3fs keeps its own retry loop (retries=5) above botocore's, which the suite does not equalize; the CloudFormation user data is checked by cfn-lint only until #1318 deploys it; just benchmark test now syncs .venv-s3fs in CI, which adds its install time.

Result: FINDINGS (1 and 5), both repaired; no code change.

Round two repair follow-up (claims and evidence of 1acf85d9). Claims checked: (a) the s3fs client is connected at setup, as the PyAthena filesystems take the connection's client when created — S3FileSystem.__init__ reads connection.s3_client (pyathena/filesystem/s3.py), which builds the boto3 client on first use, so both sides create their clients during setup_seconds; (b) the parent measures the local file as the peak temp dir — supervise() samples directory_bytes(temp_dir) of the directory it now passes; st_size of the sparse put file counts its full size, which the README's peak temp dir description covers; (c) a run without s3fs records the cases as unsupported — tested through run() with pyathena_bench.filesystem.s3fs set to None and supervise forbidden; (d) /tmp is tmpfs on the hosts — stated by the README's existing measurement note for Amazon Linux 2023, not re-measured here; (e) s3fs sends one PutObject below twice chunksize — _put_file in s3fs 2026.9.0: if size < min(5 * 2**30, 2 * chunksize). PR body and commit message updated accordingly. Result: CLEAN for the repair; the EC2 behavior of the trial directory on tmpfs remains to be observed in #1318.

laughingman7743 · 2026-10-23T13:22:53Z

+Each case sets the filesystem's `max_workers` and the S3 client's `max_pool_connections`, through the connection's `s3_config` for PyAthena, and as `max_concurrency` and `config_kwargs` for s3fs, which creates its own client from the session's credentials.
 The `[filesystem]` section of `config.toml` sets the swept values, 8 and 40 workers and pools of 10 and 40 by default, and a `block_size` of 8 MiB.
 The suite's scales are the object sizes in bytes in `[filesystem] sizes`, not the row counts in `[scales]`: 64 MiB (small), 256 MiB (medium), 1 GiB (large), and 6 GiB (xlarge) by default.
+A `get` also writes the object to a file on the host's volume, through the page cache; a `put` reads a sparse file, which no disk bounds.


Round two claim 5 (repaired in c5ccb8a4): the sentence said the host's volume throughput bounds a get. Linux absorbs the local writes in the page cache until the dirty-page limits, which are above the 1 GiB objects the README selects for gets, so the volume may not bound them; the sentence now states the write path without a bound.

laughingman7743 · 2026-10-23T13:22:53Z

- `read`: one `read()` of the whole object from a file opened with `block_size`, which reads it as parallel range requests of that size and holds the parts and the joined object, about twice the object's size, in memory;
- `write`: writes through a file opened with `block_size`, which uploads the object as a multipart upload with parts of that size;
- `copy`: `cp_file()`, or `_cp_file()` of `AioS3FileSystem`, which copies an object over 5 GiB as a multipart copy with parts of `block_size`; the copies of smaller objects, single CopyObject requests, are recorded as unsupported; the copies exclude tags and annotations, which the generated objects do not have.
+- `read`: one `read()` of the whole object from a file opened with `block_size`; PyAthena's files read it as parallel range requests of that size and hold the parts and the joined object, about twice the object's size, in memory, while s3fs's file reads it as one stream through fsspec's read-ahead cache;


Round two claims 3 and 4 (verified, no change): s3fs's open().read() is one GetObject through its default readahead cache, and its open("wb") uploads parts sequentially in S3File._upload_chunk; both from the s3fs 2026.9.0 source that the lock resolves.

…'s directory (GH-1316) The s3fs filesystem created its aiobotocore client on its first request, inside the transfer measurement, while the PyAthena filesystems take the connection's client when they are created; the s3fs API now connects at setup. The local file of a get or put lived in a directory only the trial process removed, which a terminated trial left behind, and its removal ran while the event loop's lag was still sampled; the file now lives in the trial's directory, which the supervisor passes to the child, measures, and removes, and the trial removes the file after the sampling stops. A run where s3fs is not installed records the s3fs API's cases as unsupported instead of stopping at their first trial. The README describes the tmpfs of the hosts, s3fs's single PutObject below twice its chunk size, and the peak temp dir's new content. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 · 2026-10-23T13:44:10Z

+    return None
+
+
+def filesystem(settings: Settings, case: Case, session_: Any, connection: Any) -> Any:


Independent review (relayed): Codex CLI 0.164.0, model gpt-6-astra, session 01a168ee-76d1-7260-a13b-4a1f4e398a3d, read-only sandbox, static review of base 02730a2a88ec3babeca114aaeac694a0d168db2d..head c5ccb8a4d036b428d55d2371b041e8f5816cd2c6 in a detached snapshot, prompt without the PR text or prior findings. Covered: the diff (uv.lock excluded), s3fs 2026.9.0 source, fsspec, the PyAthena transfer paths, trial supervision and reporting, fleet dispatch, packaging and CI wiring, the README, and all 15 added or changed tests. Verdict: FINDINGS (five P2, one P3, three P3 test notes); the API argument mapping, copy thresholds, environment assignment, and conflict declarations were found correct.

P2, s3fs client creation inside the measurement (this line): s3fs created its aiobotocore client on the first request, after the timer started, while S3FileSystem.__init__ takes connection.s3_client at setup. Verified and repaired in 1acf85d9: filesystem() calls fs.connect(); the constructor test asserts it.

P2, terminated downloads left files behind: the trial's TemporaryDirectory was removed only by the child, which supervise() may kill. Repaired: the supervisor passes its own trial directory to the child as temp_dir, the local file lives there, and the parent measures and removes it (see the thread on runner.py).

P2, aio loop lag included the file removal: the directory was removed before the heartbeat stopped. Repaired: the file is removed after stop.set(); await monitor (thread below).

P2, README: a default-environment run stops at the first s3fs trial instead of recording each as failed, since run() stops after a failed trial. Verified; repaired by behavior: run() records the s3fs API's cases as unsupported where s3fs is not installed (missing_api_reason), so the documented run commands complete in the default environment; the README says so.

P2, README said a get writes to the volume while /tmp is tmpfs on the hosts. Verified (the README's own note on tmpfs); the get text now says the object stays in memory outside the RSS, in the trial directory counted as the peak temp dir.

P3, s3fs uploads one PutObject below twice chunksize (_put_file, size < min(5 GiB, 2 * chunksize)). Verified; documented in the put bullet. The default sizes are at least eight blocks, so no default case hits it.
Test notes: the sparse-file test now asserts st_blocks == 0; the thread test asserts exactly one recorded call; the upload test uses a HeadObject size that differs from the helper's. Limits stated by the reviewer: lock resolution and installation were not executed; the fakes do not establish real-library acceptance, which the live smoke covers.

Independent follow-up (relayed): Codex CLI 0.164.0, model gpt-6-astra, session 01a16901-c318-75d3-a747-a1782baef935, read-only static review of the repair c5ccb8a4..1acf85d9 with the range-diff against the base and the affected contracts. Covered: supervise() → worker() → run_filesystem() including the temp directory's sampling and cleanup after termination, the client setup and transfer timing, the heartbeat shutdown and local-file cleanup, the missing-library classification and runner continuation, the report and README descriptions, and both test files. Verdict: FINDINGS, one P3 in the README's new PutObject sentence, repaired in 5a185343 (thread on the put bullet); no other actionable finding.

laughingman7743 · 2026-10-23T13:44:10Z

                await monitor
-            # Checked after the measurement: the written or copied object is complete.
-            if case.transport != "read":
+                local.unlink(missing_ok=True)


Repair of independent-review finding 3: the local file is removed here, after the heartbeat task has stopped, so a long removal of a downloaded file no longer appears in loop_lag_seconds. It is also removed on a failed transfer, and the supervisor removes the directory in every case.

laughingman7743 · 2026-10-23T13:44:10Z

    parent, child = context.Pipe()
-    process = context.Process(target=target, args=(child, payload))
    temp_dir = Path(tempfile.mkdtemp(prefix="pyathena-bench-trial-"))
+    process = context.Process(target=target, args=(child, {**payload, "temp_dir": str(temp_dir)}))


Repair of independent-review finding 2: the trial process receives the supervisor's directory as temp_dir in its payload, so the filesystem suite's local file is measured as the peak temp dir and removed by the parent even when the child is terminated. The trial_start checkpoint written before supervise() does not carry the field; a resumed job gets a new directory.

laughingman7743 · 2026-10-23T13:44:10Z

    for scale in scales:
        for case in cases:
-            unsupported = case.unsupported_reason(settings, scale)
+            unsupported = case.unsupported_reason(settings, scale) or missing_api_reason(case)


Repair of independent-review finding 4: a case whose API's library is missing in this environment is recorded as unsupported with its reason, so a run of the filesystem suite in the default environment completes the PyAthena cases. The fleet still routes the s3fs jobs to the s3fs environment, where the reason is None.

laughingman7743 · 2026-10-23T13:44:10Z

+Each case sets the filesystem's `max_workers` and the S3 client's `max_pool_connections`, through the connection's `s3_config` for PyAthena, and as `max_concurrency` and `config_kwargs` for s3fs, which creates its own client from the session's credentials.
 The `[filesystem]` section of `config.toml` sets the swept values, 8 and 40 workers and pools of 10 and 40 by default, and a `block_size` of 8 MiB.
 The suite's scales are the object sizes in bytes in `[filesystem] sizes`, not the row counts in `[scales]`: 64 MiB (small), 256 MiB (medium), 1 GiB (large), and 6 GiB (xlarge) by default.
+A `get` also writes the object to the trial's temporary directory, which the parent process measures as the peak temp dir and removes; on the Amazon Linux 2023 hosts, that directory is on the tmpfs `/tmp`, so the object stays in memory outside the process's RSS. A `put` reads a sparse file, which no disk bounds.


Repair of independent-review finding 5: the trial directory is under /tmp, tmpfs on the Amazon Linux 2023 hosts, so a get holds the object in memory outside the RSS; the sentence and the xlarge selection note now say so.

laughingman7743 · 2026-10-23T13:44:10Z

+- `read`: one `read()` of the whole object from a file opened with `block_size`; PyAthena's files read it as parallel range requests of that size and hold the parts and the joined object, about twice the object's size, in memory, while s3fs's file reads it as one stream through fsspec's read-ahead cache;
+- `get`: `get_file()` to a local file in the trial's temporary directory; PyAthena's reads one block at a time through a file opened with `block_size`, whose `bytes` cache requests the next block with the first read and one block at a time afterwards, and s3fs's reads parallel ranges of `chunksize` with `max_concurrency` requests at a time;
+- `write`: writes through a file opened with `block_size`, which uploads the object as a multipart upload with parts of that size; PyAthena's files upload up to `max_workers` parts at a time, and s3fs's file one part at a time;
+- `put`: `put_file()` from a sparse local file of the object's size, so that no disk read bounds the upload; PyAthena's uploads up to `max_workers` parts of `block_size` at a time, and s3fs's `max_concurrency` parts of `chunksize`, or one PutObject for an object below twice `chunksize`;


Repair of independent-review finding 6: s3fs's put_file() sends one PutObject for an object below twice chunksize; the default sizes are at least eight blocks.

Independent follow-up P3, repaired in 5a185343: the sentence omitted s3fs's second limit; _put_file sends one PutObject when size < min(5 GiB, 2 * chunksize), so a 5 GiB object with 3 GiB blocks, which the configuration accepts, is a multipart upload. The bullet now names both limits.

laughingman7743 · 2026-10-23T13:44:10Z

+    assert put_object(memory_fs, "/bucket/object", tmp_path / "up", 10) == 10
+    assert os.stat(tmp_path / "up").st_size == 10
+    # Sparse: the file holds no block, so an upload reads no disk.
+    assert os.stat(tmp_path / "up").st_blocks == 0


Independent-review test note: the sparse-file contract is now asserted (st_blocks == 0); writing the zero bytes instead of truncate() fails this test.

laughingman7743 · 2026-10-23T13:44:10Z


+    def test_uploaded_object_is_checked_with_head_object_and_deleted(self, s3_client):
+        # The upload reports 10 bytes; the object's size comes from S3.
+        s3_client.size = 11


Independent-review test note: the HeadObject size differs from the helper's return value, so the recorded bytes proves the check uses S3's size.

…-1316) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 and others added 2 commits October 23, 2026 22:16

Describe the get_file() read pattern and the s3fs environment need in…

fb8b384

… the README (GH-1316) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 commented Oct 23, 2026

View reviewed changes

Describe the local write of a get without a disk bound claim (GH-1316)

c5ccb8a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 commented Oct 23, 2026

View reviewed changes

State both limits of s3fs's single PutObject upload in the README (GH…

5a18534

…-1316) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

laughingman7743 marked this pull request as ready for review October 23, 2026 13:56

laughingman7743 merged commit 2e66e94 into master Oct 23, 2026
15 checks passed

laughingman7743 deleted the feat/1316-s3fs-benchmark-api branch October 23, 2026 14:08

laughingman7743 mentioned this pull request Oct 23, 2026

Add s3fs as an API of the filesystem benchmark suite #1316

Closed

		return operation()


		def filesystem(settings: Settings, case: Case, session_: Any, connection: Any) -> Any:

		return None


		def filesystem(settings: Settings, case: Case, session_: Any, connection: Any) -> Any:

Conversation

laughingman7743 commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHAT

WHY

TEST

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

laughingman7743 commented Oct 23, 2026 •

edited

Loading