Repository navigation
Add s3fs as an API of the filesystem benchmark suite #1316
Description
Part of #1315.
Summary
Add s3fs as a third API of the filesystem suite of the benchmark project, so that the suite's reads, writes, and copies of one object run through s3fs.S3FileSystem under the same host, object, block size, and connection pool as PyAthena's S3FileSystem (sync) and AioS3FileSystem (aio).
Environment
s3fs depends on aiobotocore, which pins botocore to a narrow window (aiobotocore 3.10.0: botocore>=1.43.115,<1.43.118), and s3fs pins fsspec to its own version. Install it through a dependency group of benchmarks/pyproject.toml with its own environment, as the polars2 group has benchmarks/.venv-polars2 (#1292); the jobs --environment selection then routes the s3fs cases to that environment. The lock must keep the PyAthena environment's botocore unchanged. Record the s3fs, aiobotocore, and botocore versions of the s3fs environment in the history entry.
Cases
Case.api gains s3fs. The filesystem is constructed from the trial's boto3 session credentials with config_kwargs={"max_pool_connections": pool} and max_concurrency=max_workers, which is s3fs's per-file concurrency for get_file, put_file, and managed copies (default 10). skip_instance_cache=True, as the PyAthena cases use.
The operations map as follows; the s3fs code paths differ from PyAthena's, so the suite records which method each case calls:
| Transport | PyAthena (sync, aio) |
s3fs |
|---|---|---|
read |
open(path, "rb", block_size).read(): parallel range requests through the executor |
open(path, "rb", block_size).read(): single stream through fsspec's read cache, no parallelism |
read (new variant, both) |
get_file to a local file |
get_file: parallel ranges of block_size with max_concurrency batches |
write |
open(path, "wb", block_size) with parts in flight bounded by max_workers |
open(path, "wb", block_size): S3File._upload_chunk uploads the parts one at a time |
write (new variant, both) |
put_file from a local file |
put_file: parallel parts with max_concurrency |
copy |
cp_file with max_workers part copies |
cp_file: managed multipart copy above 5 GiB with max_concurrency |
The local-file variants need a scratch file of the object's size on the host's disk; run them up to the large scale (1 GiB) and leave xlarge to the in-memory variants, since the host has 16 GiB of memory and the EBS volume is the only disk.
loop_lag_seconds is sampled for aio only; s3fs runs on fsspec's IO thread, so the trial's event loop is not involved, and the sample stays None.
Out of scope
Scenarios with many objects are #1317. The measurement is #1318.
Activity
Describe the get_file() read pattern and the s3fs environment need in…
Describe the local write of a get without a disk bound claim (GH-1316)
Connect the s3fs client at setup and keep the local file in the trial…
State both limits of s3fs's single PutObject upload in the README (GH…
laughingman7743 commentedon Oct 23, 2026
Done in #1319, merged as 2e66e94. The PR body referenced this issue with "Refs", so it did not close automatically.