Repository navigation
Benchmark chunked reads of a result larger than host memory #1162
Description
Summary
Measure chunked reads of a result larger than the host's memory, the case of #61 (PandasCursor reading a 10–30 GB result until MemoryError). The runs so far stopped at 10 million rows, about 2.9 GB of CSV.
Evidence so far
In the run of the 4.1.0 release commit (#1157, benchmarks/history/2026-10-13-437276e1.md), chunked reads kept almost the same peak from 1 million to 10 million flat rows, while unchunked reads grew with the result:
| Reader, 10,000-row chunks | 1 million rows | 10 million rows | 10 million rows without chunks |
|---|---|---|---|
| PandasCursor, CSV | 345 MiB | 363 MiB | 5,132 MiB |
PandasCursor, unload=True |
353 MiB | 604 MiB | 5,376 MiB |
| PolarsCursor, CSV | 613 MiB | 874 MiB | 6,854 MiB |
PolarsCursor, unload=True |
365 MiB | 659 MiB | 3,687 MiB |
This suggests that the peak does not grow with the result, but nothing beyond 10 million rows was measured.
Proposal
Run a bounded selection at 100 million flat rows, about 29 GB of CSV, on a host whose memory is smaller than the unchunked result, such as r7g.xlarge (32 GiB):
- PandasCursor and PolarsCursor with
chunksize=100000, from CSV and UNLOAD results, and PandasCursor UNLOAD withdtype_backend="pyarrow"; - S3FSCursor rows, which read the CSV file in streaming blocks;
- Wrangler CSV and UNLOAD with
chunksize=100000as the reference; - one unchunked PandasCursor CSV read, expected to stop at the harness's memory limit, to confirm that the result does not fit.
The scale can be added in the run's configuration ([scales] xxlarge = 100000000) without a harness change. Record:
- whether the chunked peaks stay near the 10-million-row values;
- the number and size of the UNLOAD output files at that size;
- the run time, to size
timeout_seconds(S3FSCursor took 329 seconds for 10 million rows).
Out of scope: a Dask integration. The Parquet files of unload=True stay in S3 and can be read by Dask directly.
Activity
laughingman7743 commentedon Oct 14, 2026
Plan: this run is part of the single combined benchmark run on the commit of the release that contains #1158, #1159, #1160, and #1161 (planned as 4.2.0), on the same
r7g.2xlargestack as the cursor re-measurements, instead of a separater7g.xlargehost.Unchunked reads of 100 million flat rows are still expected to exceed the harness's memory limit there. The limit is
memory_fraction0.8 of 61.7 GiB, about 49 GiB. Extrapolating the 10-million-row peaks of 5.1 GiB (PandasCursor CSV) and 6.8 GiB (PolarsCursor CSV) gives about 50 to 68 GiB, so one unchunked PandasCursor CSV case still shows that the result does not fit. If it fits after all, the entry records that, and the chunked results still answer #61.laughingman7743 commentedon Oct 18, 2026
Plan update: the combined benchmark is now tracked in #1288, with #1287 and #1159. It measures the 4.2.0 release candidate on master before the release, and 4.2.0 is tagged on the measured commit after the results are reviewed. Before, the plan was to measure the tagged release.
laughingman7743 commentedon Oct 19, 2026
Measured in the combined 4.2.0 run (#1288), recorded in #1310 (
benchmarks/history/2026-10-19-9e99caa8.md, section "100 million rows").r7g.2xlargestack of the cursor cases instead ofr7g.xlarge, as planned above.chunksize=100000, the peaks stayed near the 10-million-row values. PandasCursor peaked at 462 MiB from CSV and 978–986 MiB from UNLOAD, PolarsCursor at 1,001–1,051 MiB with Polars 1.44.2 and 2.0.0, and Wrangler at 390–1,173 MiB.The results for #61 are posted there.