[laughingman7743]

WHAT

A chunked PolarsCursor read of a CSV result now reads the timestamp columns as text and converts them in the scan's query plan. Fractions finer than the Datetime time unit are truncated, as a whole read already does when parsing fails. A whole read with ignore_errors now does the same instead of returning null.

• _scan_csv_chunks() applies _to_datetimes() to the pl.scan_csv() LazyFrame before collect_batches(). It used to pass the Datetime dtypes to pl.scan_csv(), so chunked reads raised OperationalError on a timestamp with more than nine fractional digits (Athena's TIMESTAMP(10) to TIMESTAMP(12)). Polars parses up to nine digits into a Datetime of any time unit, truncating them, and fails on more (Polars 1.39.0 and 2.0.0).
• _to_datetimes() accepts a LazyFrame as well as a DataFrame. It selects each column with cs.by_name(name, require_all=False) instead of checking df.columns, so a missing column is still skipped and a LazyFrame doesn't have to resolve its schema.
• _to_datetimes() takes strict, which both readers set from ignore_errors, so text that is not a timestamp is read as null when ignore_errors=True, as Polars reads it.
• _get_timestamp_dtypes() is renamed to _get_csv_schema_overrides(). It also returns the schema_overrides with the timestamp columns set to String; the chunked read and the whole read's fallback both use it.
• The timestamp columns are not read as text when execute() is given schema, the same as with schema_overrides and with_column_names. Polars ignores schema_overrides when given schema, so the columns take that schema's types.
• Without ignore_errors, the whole read is unchanged: it parses the columns as Datetime and reads the data again with them as text only on ComputeError.
• With ignore_errors given to execute(), the whole read reads the timestamp columns as text at once. Polars reads a timestamp with more than nine fractional digits as null with ignore_errors, raising no ComputeError, so the whole read returned null where the chunked read now returns the truncated value.

This replaces Draft PR #1194, which was written against an older master. The chunked read is implemented as #1194 did (the design was agreed there). This PR is rebuilt on the current master, adds the whole read with ignore_errors, and covers the chunked read in the #1213 temporal conformance test.

WHY

Fixes #1191 (part of #1212, contract items 1, 2, and 8).

Retrying on ComputeError, as the whole read does, cannot work for chunks: a value that fails to parse can be in a later chunk, after the rows of earlier chunks have been returned.

The whole read keeps parsing natively. Reading the columns as text would hold the text column and the converted one at the same time. Measured for #1194, text reading made whole reads 2-4.5x faster but raised their peak RSS by about 35-40%.

Years outside 1 to 9999 are out of scope (#1212, contract item 3). The fixed-length truncation in _to_datetimes() assumes four-digit years, so it is correct for every year in range.

Chunked reads of a local 5-million-row CSV file (one timestamp column and one string column, 179-194 MB, pl.Datetime (us), chunk_size=100_000, each run in a fresh process, 3 runs each with 3 and 6 fractional digits), measured at 9fb2b4a:

Polars	native Datetime parse (master)	text + conversion in the plan
1.39.0	1.25–1.64 s, 382–401 MiB	0.25–0.57 s, 543–575 MiB
2.0.0	0.58–1.22 s, 407–447 MiB	0.22–0.27 s, 509–519 MiB

The peak RSS of a chunked read grows with the text. Polars 1.39.0 parses the whole file before collect_batches() returns the first batch, so on 1.39.0 the growth covers the whole file. For #1194, converting each batch instead measured slower than converting in the plan on both versions (1.39.0: 0.24–0.39 s, 554–592 MiB; 2.0.0: 0.25–0.28 s, 478–508 MiB).

Behavior change: with ignore_errors=True given to execute(), whole and chunked reads now truncate a timestamp with more than nine fractional digits instead of returning null; text that is not a timestamp is still null. A whole read with ignore_errors holds the text column and the converted one at once, so its peak RSS grows as measured above for text reading.

TEST

Tested commit: 3276a3f (the chunked read benchmark at 9fb2b4a, whose read path is the same).

• just lint: passed.
• uv run --env-file .env pytest -n 2 tests/pyathena/polars tests/pyathena/aio/polars (Python 3.15, Polars 2.0.0, with AWS): 219 passed, 1 xfailed (test_temporal_types[unload], UNLOAD timestamps outside 1677-2262 are read as wrong values #1214).
• pytest tests/pyathena/polars/test_result_set.py tests/pyathena/polars/test_converter.py on Python 3.13 with Polars 1.39.0: 82 passed.
• New integration test test_temporal_types_chunked reads the Add temporal type tests for each synchronous cursor #1213 temporal query with chunksize=1 and checks the dtypes and values against the contract, including TIMESTAMP(9) and TIMESTAMP(12). On master (9e2bf6e) it fails with could not parse `"2020-01-02 03:04:05.123456789012"` as dtype `datetime[μs]` at column 'ts_p12'.
• test_read_csv_truncates_timestamps covers both readers, with new_columns, schema, and with_column_names. Its _iter_csv_chunks cases with no options and with new_columns fail on master. scan_csv() does not take columns, so that case covers only the whole read.
• test_read_csv_ignore_errors covers both readers with ignore_errors=True, a 12-digit timestamp, and text that is not a timestamp; both cases fail on master.
• test_to_datetimes covers a DataFrame and a LazyFrame, including a column the frame does not have. test_to_datetimes_not_strict covers strict=False.
• Not run: just test sqla / sqla-async (no SQLAlchemy changes) and the other cursors' tests (no changes outside the Polars result set and converter).

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (behavior and implementation): FINDINGS, repaired

Base 9e2bf6eebfa96f7a1f747aae51b30637b1bd8296, head 9fb2b4a6fca154e7f493f2e2e6ec709cf5a3f349.

Covered: _to_datetimes() (DataFrame/LazyFrame, missing columns, strict), _get_csv_schema_overrides() (header, schema/schema_overrides/with_column_names, custom converters with non-Datetime dtypes), _read_csv() (result file and GetQueryResults rows), _iter_csv_chunks()/_scan_csv_chunks() (new_columns, ignore_errors, error wrapping into OperationalError), callers (only these two readers; AsyncPolarsCursor uses the same result set), fetch methods over chunks (unchanged; they read the converted DataFrames), and the tests.

Finding (pre-existing on master, exposed by this PR): with ignore_errors=True, pl.read_csv() reads a timestamp with more fractional digits than the time unit as null instead of raising ComputeError (Polars 1.39.0 and 2.0.0: "2020-01-02 03:04:05.123456789012" → None), so the whole read never reached the text fallback and returned null, while the chunked read now truncates it. That contradicts #1191's expected behavior and #1212 contract item 8.

Repair in 2a2a61c: with ignore_errors, _read_csv() reads the timestamp columns as text at once and converts them with strict=False. test_read_csv_ignore_errors now covers both readers; both cases fail on master 9e2bf6e. Unit tests pass on Polars 2.0.0 and 1.39.0 (82 passed each).

No other findings.

[laughingman7743]

The branch was rewritten after round two to correct the commit messages (same trees): 9fb2b4a → e2006b3, 2a2a61c → 7d5584e. git diff 2a2a61c2 7d5584ef is empty.

[laughingman7743]

Self-review round two (claims, callers, operations): FINDINGS, repaired

Base 9e2bf6eebfa96f7a1f747aae51b30637b1bd8296, head 7d5584ef (round one's head after rewording), repaired at 3276a3f472bd3996c154f42f2142c0c6d2eb74a2.

Claims checked (PR body, docstrings, comments, commit messages, issue premise):

• "Polars does not parse more fractional digits than the time unit" (docstrings, PR body "such as TIMESTAMP(9)", commit messages, Chunked PolarsCursor CSV reads fail on timestamps with more fractional digits than the time unit #1191's title): false. On Polars 1.39.0 and 2.0.0, read_csv/scan_csv with Datetime(\"us\") or (\"ns\") parse 3, 6, and 9 digits (9 truncated to us) and raise ComputeError only at 12; the live test_temporal_types_chunked failure on master names ts_p12, not ts_p9. Repaired: docstrings in 3276a3f, commit messages rewritten (same trees; force-pushed with lease), PR body corrected to "more than nine (TIMESTAMP(10) to (12))".
• "The whole read is unchanged": stale after round one's repair; PR body now says it is unchanged without ignore_errors and describes the ignore_errors path, including its peak-RSS implication.
• scan_csv() has no columns parameter (1.39.0, 2.0.0): confirmed by signature.
• Polars ignores schema_overrides when given schema: confirmed (schema of String wins over schema_overrides Int64).
• Fixed-length cut correct for years 1-9999: Athena writes four-digit years in range (Define and enforce the contract for Athena temporal types across cursors #1212 facts); outside is out of scope (contract item 3).
• Benchmark table: re-measured at 9fb2b4a (same read path as the head), local, noisy machine; ranges stated as measured. Per-batch numbers and the whole-read RSS figures are cited from Read the timestamp columns of chunked Polars CSV results as text #1194.

Callers: _scan_csv_chunks()/_get_csv_schema_overrides() are private with no callers outside result_set.py; AsyncPolarsCursor uses the same result set (aio tests pass). No AWS request count changes: the same single result file read per query; the live test adds one SELECT.

Evidence at 3276a3f: just lint passed; pytest -n 2 tests/pyathena/polars tests/pyathena/aio/polars with AWS (Py 3.15, Polars 2.0.0): 219 passed, 1 xfailed (#1214); unit tests on Py 3.13 + Polars 1.39.0: 82 passed. Not run: SQLAlchemy suites and other cursors (untouched).

[laughingman7743]

Independent review (relayed): Codex CLI 0.163.0, model gpt-6-astra, sandbox read-only, reasoning effort high, session 01a14803-ee7c-7a33-9d78-0d8bb2f56ec4. Static review of git diff 9e2bf6eebfa96f7a1f747aae51b30637b1bd8296..3276a3f472bd3996c154f42f2142c0c6d2eb74a2 in a detached snapshot (unchanged afterwards); no PR text, commit messages, or prior findings given. It ran small Polars probes under /tmp on 1.39.0 and 2.0.0; no repository tests, AWS, or GitHub access.

Covered (reviewer): whole-read native parsing and fallback; GetQueryResults CSV serialization; lazy chunk conversion and exception wrapping; NULLs and empty/header-only results; duplicate names and renaming; reader kwargs; typing, docstrings, and regression assertions. Valid timestamps with 0-12 fractional digits matched between whole and chunked reads across years and time units. Result: FINDINGS (2, both P2).

Finding 1 (P2), here: "Truncation hides malformed timestamp suffixes: 2020-01-02 03:04:05.123456garbage is cut to a valid timestamp before validation, so strict and ignore_errors accept it."

Author verdict: rejected. The text of a timestamp column comes from Athena, which writes only YYYY-MM-DD HH:MM:SS[.f{1,12}] for years 1-9999 (#1212 facts); a suffix after the fraction cannot occur. The cut predates this PR (master's whole-read fallback), and dropping it was #1201, closed not planned.

[laughingman7743]

Independent review (relayed), finding 2 (P2), here: "Eligibility ignores the effective has_header kwarg: with execute(has_header=False, skip_rows=1, ignore_errors=True) on a .csv result whose timestamp column is named column_1 (Polars 1.39) / column_0 (2.0), the native whole read returns null for a 12-digit value while this change returns a truncated datetime, violating the headerless exception."

Author verdict: rejected. _get_csv_schema_overrides() only replaces the Datetime entries of the schema_overrides that the readers already pass with String, and _to_datetimes() converts the same names. With has_header=False, Polars applies those overrides by its default names, so the columns read as text are exactly the ones it would otherwise parse as Datetime, and a column the frame does not have is skipped. In the scenario, the truncated value is what #1212 contract items 2 and 8 require; the null was the ignore_errors defect fixed in 7d5584e. Honoring the execute() has_header here would make that case raise OperationalError (without ignore_errors) or return null instead. The docstring's "Without one" refers to the has_header argument, which comes from the result file type.

No repair, so no follow-up review is needed.