[laughingman7743]

WHAT

A chunked PolarsCursor read of a CSV result reads the timestamp columns as text and converts them in the scan's query plan, truncating fractions finer than the Datetime time unit, as a whole read already does when parsing them fails.

• _scan_csv_chunks() applies _to_datetimes() to the pl.scan_csv() LazyFrame before collect_batches(). It previously passed the Datetime dtypes to pl.scan_csv(), so a chunked read of a timestamp with more fractional digits than the time unit raised OperationalError.
• _to_datetimes() takes a LazyFrame as well as a DataFrame. It selects each column with cs.by_name(name, require_all=False) instead of checking df.columns, so a column the frame does not have is still skipped without resolving a LazyFrame's schema.
• _get_timestamp_dtypes() becomes _get_csv_schema_overrides(), which also returns the schema_overrides with the timestamp columns as String; the chunked read and the whole read's fallback use it.
• The columns are not read as text when execute() is given schema, as with schema_overrides and with_column_names: Polars ignores schema_overrides when given a schema, so the columns have that schema's types.
• The whole read is unchanged: it parses the columns as Datetime and reads the data again with them as text only on ComputeError.

WHY

Fixes #1191.

A retry on ComputeError, as the whole read does, cannot work for chunks: a value that fails to parse can be in a later chunk, after the rows of the earlier chunks have been returned.

The whole read keeps parsing natively because reading the columns as text holds the text column and the converted one at once. Reading a local 5-million-row CSV file (a timestamp column and a string column, ~180-190 MB, pl.Datetime (us), each run in a fresh process), text reading made the whole read 2-4.5x faster but raised its peak RSS by about 35-40% (Polars 1.39.0: 545-577 → 756-798 MiB; 2.0.0: 479-525 → 645-700 MiB). Converting in the query plan was chosen for chunks over converting each batch; it was faster than per-batch conversion in the measurements below.

Chunked reads with this PR, same data, chunk_size=100_000 (seconds and peak RSS over 3 runs, 3 and 6 fractional digits):

Polars	native Datetime parse (master)	text + conversion in the plan
1.39.0	1.15–1.19 s, 395–411 MiB	0.19–0.23 s, 554–611 MiB
2.0.0	0.43–0.51 s, 412–461 MiB	0.19–0.21 s, 523–548 MiB

The peak RSS of a chunked read grows with the text: Polars 1.39.0 parses the whole file before collect_batches() returns the first batch, so the growth there covers the whole file. Converting each batch instead measured 0.24–0.39 s / 554–592 MiB on 1.39.0 and 0.25–0.28 s / 478–508 MiB on 2.0.0.

The converted values and dtypes equal those of the Datetime parse for Athena's timestamp text (0, 1, 3, and 6 fractional digits, years 0001 to 9999, NULL as an empty field or ""), with Datetime (us), Datetime("ns"), Datetime("us", "UTC"), and Datetime("ms", "Asia/Tokyo"), on Polars 1.39.0 and 2.0.0. With ignore_errors=True given to execute(), a chunked read now returns a value with more fractional digits than the time unit truncated instead of null; text that is not a timestamp is still read as null, as _to_datetimes() takes strict, which the readers set from ignore_errors.

TEST

Tested commit: eee0346.

• just lint: passed.
• uv run --env-file .env pytest -n 4 tests/pyathena/polars tests/pyathena/aio/polars (Python 3.15, Polars 2.0.0, with AWS, including the managed result work group): 216 passed.
• pytest tests/pyathena/polars/test_result_set.py tests/pyathena/polars/test_converter.py on Python 3.13 with Polars 1.39.0: 81 passed (79 with Polars 1.44.2 at 80424b2).
• test_read_csv_truncates_timestamps covers both readers, including new_columns, schema, and with_column_names; its _iter_csv_chunks cases without options and with new_columns fail on 0622449 (master). scan_csv() does not take columns, so that case covers the whole read only.
• test_to_datetimes covers a DataFrame and a LazyFrame, including a column the frame does not have; test_to_datetimes_not_strict covers strict=False.
• test_iter_csv_chunks_ignore_errors reads a chunk with ignore_errors=True and text that is not a timestamp; it fails at 80424b2.
• New integration test test_iter_chunks_result_file_values compares the chunks of RESULT_FILE_VALUES_QUERY read with chunksize=1 with the whole result, including the TIMESTAMP(12) column.
• Not run: just test sqla / sqla-async (no SQLAlchemy changes) and the other cursors' tests (no changes outside the Polars result set and converter).

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): FINDINGS, repaired.

Scope: merge-base 06224494788043a4887afcb112a96c94bbcb02f1..bc95db9a6441caf4e112754b16c0631106583d16; pyathena/polars/result_set.py (_get_csv_schema_overrides(), _read_csv(), _iter_csv_chunks(), _scan_csv_chunks()), tests/pyathena/polars/test_result_set.py, tests/pyathena/polars/test_cursor.py. Callers traced: the sync and async PolarsCursor share AthenaPolarsResultSet; the S3 result file and the managed GetQueryResults rows both reach _read_csv().

Finding: with schema given to execute(), Polars ignores schema_overrides (checked with read_csv() and scan_csv()), so a timestamp column keeps the dtype from that schema. bc95db9 still converted it with _to_datetimes(): a String column became Datetime("us"), and a Datetime("ms") column failed with OperationalError, for both readers. On master both kept the schema's dtype; the old fallback ran only after a ComputeError.

Repair (f5023e8): schema joins schema_overrides and with_column_names as a reason not to read the columns as text. test_read_csv_truncates_timestamps gains a schema case per reader; both fail without the repair.

Checked without findings: errors from _to_datetimes() are inside each reader's try and become OperationalError; NULL as an empty field or ""; columns, new_columns (renamed after the conversion), .txt results without a header, and tz-aware or ns Datetime dtypes from a custom converter.

Validation: just lint; pytest tests/pyathena/polars/test_result_set.py tests/pyathena/polars/test_converter.py: 76 passed. The AWS Polars suites ran at bc95db9 (209 passed) and are rerun before Ready.

[laughingman7743]

Round one, follow-up after the design change (80424b2): CLEAN for the changed scope (git range-diff-equivalent: f5023e8..80424b2 touches _read_csv(), _scan_csv_chunks(), _to_datetimes(), and their tests).

Round two measured that reading the timestamp columns of a whole read as text raised its peak RSS by about 35-40%. After consulting Codex and claude-fable-5-1, the maintainer chose to keep the whole read as on master (native parse, text re-read only on ComputeError) and to convert the chunked read's text columns in the scan_csv() query plan.

Checked: the whole read matches master except that a ComputeError with schema given re-raises without the futile re-read (the outcome is the same OperationalError); a conversion error in a chunk surfaces from collect_batches() inside the try as OperationalError; cs.by_name(require_all=False) matches names literally (not as a regex), skips a column the frame lacks without resolving a LazyFrame's schema, and keeps the column order; collect_batches() keeps maintain_order=True; a header-only file yields no batches as before; new_columns is applied after the conversion.

Validation at 80424b2: just lint; Polars 2.0.0 with AWS tests/pyathena/polars tests/pyathena/aio/polars: 214 passed; unit tests on Polars 1.39.0 and 1.44.2: 79 passed each.

[laughingman7743]

Self-review round two (claims, compatibility, operational effects): FINDINGS, repaired.

Scope: merge-base 06224494788043a4887afcb112a96c94bbcb02f1..80424b2eb029ce5a731dd41137b8c6b68155f3c9 (full pass of the PR body, commit messages, docstrings, comments, and docs/polars.md).

Findings:

1. Operational: the PR body claimed only a speed-up for reading timestamps as text. Measuring peak RSS (fresh process per run, 5M rows, ~180-190 MB) showed whole reads grew by ~35-40% (1.39.0: 545-577 → 756-798 MiB; 2.0.0: 479-525 → 645-700 MiB). Repaired in 80424b2 by keeping the whole read as on master (maintainer decision after consulting Codex and claude-fable-5-1), and converting chunks in the query plan.
2. Claim: the PR body said the text-reading conditions were unchanged; f5023e8 added schema. Corrected in the PR body.
3. Compatibility (from the Codex consultation, verified on 1.39.0 and 2.0.0): with ignore_errors=True, a chunked read used to return null for a value with excess digits; it now returns it truncated, and non-timestamp text raises. Documented in the PR body; Athena does not write such text for a timestamp column.

Claims verified: schema makes Polars ignore schema_overrides for read_csv() and scan_csv() (1.39.0, 1.44.2, 2.0.0); value/dtype equality of text conversion and native parsing (0/1/3/6 digits, years 0001-9999, NULL, us/ns/UTC/Asia/Tokyo; 1.39.0 and 2.0.0); Polars 1.39.0 parses the whole file before collect_batches() returns the first batch (first batch at 0.42 s, 530 MiB, equal to the total) while 2.0.0 streams (0.02 s, 163 MiB); the chunked timings and RSS in the PR table (3 runs each, bench4.py importing PyAthena's converter in both modes). docs/polars.md has no statement this change makes obsolete. The first commit's message (bc95db9) describes the superseded design; later commits record the change.

Limits: one data shape (one timestamp and one string column) on local macOS; with more columns the relative differences shrink. AWS validation is local; CI runs on Ready.

[laughingman7743]

Independent review (relayed): FINDINGS.

Reviewer: Codex CLI 0.162.1, model gpt-6-astra, reasoning effort max, codex exec -s read-only, session 01a141e0-50a5-77d3-83a6-59fd6e533f3f. Static review of a detached snapshot at 80424b2eb029ce5a731dd41137b8c6b68155f3c9, diff from merge-base 06224494788043a4887afcb112a96c94bbcb02f1; the prompt held the intended behavior, conventions, and diff only (no PR text, commit messages, or prior findings). The reviewer ran in-memory Polars probes under /tmp with 1.39.0 and 2.0.0; no repository tests, builds, edits, AWS, or GitHub access. The snapshot remained unchanged.

Covered: whole-read parse and fallback; chunked lazy conversion and OperationalError translation; NULLs, empty/header-only results, duplicate names, projections, new_columns; schema, schema_overrides, with_column_names, null_values, ignore_errors; Polars 1.39.0/2.0.0 selectors, units, batch ordering, typing, and test assertions.

Finding (P2): _to_datetimes() stayed strict regardless of ignore_errors. With timestamp metadata and CSV t\n2020-01-02 03:04:05.123\ninvalid\n, a chunked read with ignore_errors=True used to return the datetime then NULL; at 80424b2 it raised InvalidOperationError → OperationalError, aborting iteration.

Verified and repaired in eee0346: _to_datetimes() takes strict, which _scan_csv_chunks() and the whole read's fallback set from ignore_errors. test_iter_csv_chunks_ignore_errors fails at 80424b2 and passes now; test_to_datetimes_not_strict covers the converter. Both self-review perspectives on the repair: strict only affects str.to_datetime (not the cast); the PR body's statement that non-timestamp text raises with ignore_errors was corrected. Validation at eee0346: just lint; Polars 2.0.0 with AWS tests/pyathena/polars tests/pyathena/aio/polars: 216 passed; unit tests on Polars 1.39.0: 81 passed. An independent follow-up review of the repair follows.

[laughingman7743]

Independent follow-up review (relayed): CLEAN.

Reviewer: Codex CLI 0.162.1, gpt-6-astra, reasoning effort max, codex exec -s read-only, session 01a141ed-b626-7582-bb88-e0bfd275efa1. Static review of a detached snapshot at eee03465f89838bcb6c128009f8065916b7e3ff2, covering 80424b2e..eee03465 with the overall diff from merge-base 06224494788043a4887afcb112a96c94bbcb02f1 available; in-memory Polars 1.39.0 / 2.0.0 probes under /tmp only. The snapshot remained unchanged.

Covered: _to_datetimes() and both callers, kwargs forwarding, whole-read fallback and lazy batching, cast/timezone, NULL and empty strings, fractional truncation, non-boolean ignore_errors values (Polars rejects them before the conversion), and the regression assertions.

Notes from the reviewer, not actionable: the new tests do not exercise the whole read's fallback with ignore_errors (with that option the native parse does not raise, so the fallback is not reached); whole reads with ignore_errors=True return null for excess fractional digits while chunked reads truncate them, which predates this change.

[laughingman7743]

On hold for #1216 under the temporal contract in #1212; do not merge as is.

Reading chunks as text routes signed years through _to_datetimes(), whose fixed-length cut then returns a wrong fraction instead of failing: with Datetime("us"), -0001-01-02 03:04:05.123456789012 becomes .123450 (#1201). On master, a chunked read of such a value raises. The contract requires DataError for years outside 1 to 9999.

The lazy conversion in the scan plan and the measurements here (whole reads kept native after +35-40% peak RSS for text reads) carry over to #1216. This PR will be reworked there to enforce the year range.

[laughingman7743]

Superseded by #1234, which rebuilds this change on the current master (after #1226), keeps the same chunked-read design, also reads timestamps of whole reads with ignore_errors as text (Polars reads more than nine fractional digits as null there), and covers the chunked read in the temporal conformance test.