[laughingman7743]

WHAT

_to_datetimes() (pyathena/polars/converter.py) no longer cuts the timestamp text at a fixed length before str.to_datetime(). PolarsCursor now truncates the fraction to the time unit for years Athena writes with a sign: +11020-01-02 03:04:05.123 reads as .123 with ms, not .100.

_to_datetimes() converts the timestamp columns of a chunked CSV read and of a whole read whose native parse fails. ArrowCursor's _to_timestamp() keeps its cut at _TIMESTAMP_TEXT_LENGTHS, because pyarrow fails with ArrowInvalid on a signed or five-digit year before the cut matters.

Stacked on #1194 (base fix/1191-polars-chunked-timestamps), which changes _to_datetimes(); this PR will be rebased onto master after #1194 merges.

WHY

Fixes #1201.

The fixed length assumed a four-digit year. A live Athena query writes +11020-01-02 03:04:05.123 and -0001-01-02 03:04:05.123456789012 in both the CSV result file and the GetQueryResults rows.

str.to_datetime("%Y-%m-%d %H:%M:%S%.f", time_unit=...) parses up to 12 fractional digits, truncating those the unit does not hold (.999999999999 → .999 for ms), and signed years, on Polars 1.39.0, 1.44.2, and 2.0.0. Over 300,000 random four-digit-year values with 0 to 12 fractional digits, plus "" and null, the result was equal with and without the cut for ms, us, and ns on those three versions.

Without the cut, converting 5 million values of one column took 0.055–0.063 s instead of 0.074–0.108 s (local, best of 3, ms and us, 3/6/12 fractional digits, Polars 1.39.0 and 2.0.0).

TEST

Tested commit: 34142c6.

• just lint: passed.
• uv run --env-file .env pytest -n 4 tests/pyathena/polars tests/pyathena/aio/polars (Python 3.15, Polars 2.0.0, with AWS): 220 passed.
• pytest tests/pyathena/polars/test_result_set.py tests/pyathena/polars/test_converter.py on Python 3.13 with Polars 1.39.0: 83 passed.
• New test_to_datetimes_years_outside_four_digits (unit) and TestPolarsCursor.test_timestamp_years_outside_four_digits (AWS, whole and chunksize=1) fail on eee0346 and pass now.
• Not run: the other cursors' tests (no changes outside the Polars converter); just test sqla / sqla-async (no SQLAlchemy changes).

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): CLEAN.

Scope: merge-base eee03465f89838bcb6c128009f8065916b7e3ff2 (stacked on fix/1191-polars-chunked-timestamps)..34142c639a4c3235deb1b6d9bbcd05c4aae78a1e; pyathena/polars/converter.py (_to_datetimes()), tests/pyathena/polars/test_converter.py, tests/pyathena/polars/test_cursor.py. Callers traced: _read_csv() fallback and _scan_csv_chunks() (LazyFrame) in pyathena/polars/result_set.py, shared by the sync and async PolarsCursor; ArrowCursor's _to_timestamp() is untouched and still imports _TIMESTAMP_TEXT_LENGTHS.

Checked: "" stays null through when(text != ""); strict=False still yields null for non-timestamp text; up to 12 fractional digits truncate (not round) for ms/us/ns; signed and five-digit years parse; the cast to tz-aware dtypes is unchanged. ns cannot represent years outside 1677-2262 and Polars wraps them silently, but its own CSV parse does the same, so that is not introduced here. The new unit test asserts the year and the sub-second nanoseconds, since Python datetime cannot hold these years; it and the AWS test (whole read and chunksize=1) fail on eee0346.

Validation: just lint; Polars AWS suites 220 passed; Polars 1.39.0 unit tests 83 passed.

[laughingman7743]

Self-review round two (claims, compatibility, operational effects): CLEAN.

Scope: same base/head as round one; PR body, commit message, the changed docstring, and issue #1201.

Claims verified: Athena writes +11020-01-02 03:04:05.123 and -0001-01-02 03:04:05.123456789012 in both the CSV result file and GetQueryResults (live query); str.to_datetime() with the explicit format parses 12 digits with truncation and signed years on Polars 1.39.0, 1.44.2, 2.0.0; 300,000 random four-digit-year values (0-12 digits, plus "" and null) give equal results with and without the cut for ms/us/ns on those versions; conversion timings (0.055-0.063 s vs 0.074-0.108 s per 5M values, best of 3); pyarrow 22.0.0 (the floor) and 26.0.0 raise ArrowInvalid casting +11020-... and -0001-... to a timestamp, so ArrowCursor's cut does not matter for these years.

Compatibility: _to_datetimes() is private; its signature is unchanged. No docs mention the cut. Limits: timings are local and cover the conversion expression only.

[laughingman7743]

Independent review (relayed): CLEAN.

Reviewer: Codex CLI 0.162.1, model gpt-6-astra, codex exec -s read-only, session 01a14540-f786-7193-aa58-a77b69072b88. Static review of a detached snapshot at 34142c639a4c3235deb1b6d9bbcd05c4aae78a1e, diff from eee03465f89838bcb6c128009f8065916b7e3ff2; the prompt held the intended behavior and diff only. The reviewer ran library-only snippets under /tmp with Polars 1.39.0, 1.44.2, and 2.0.0; no repository tests, AWS, or GitHub access. The snapshot remained unchanged.

Covered: eager and LazyFrame conversion, whole-read fallback and chunked callers, custom units and time zones, strictness, Arrow's separate cut, and the regression assertions. Reported: identical results before and after for ordinary timestamps (no fraction, 1-12 digits, ms/us/ns, pre-epoch, "", null, tz-aware dtypes, both strict settings), truncation without rounding, correct signed and five-/six-digit years, and that the new tests fail on the old expression (for example 123450000 ns instead of 123456000 ns for the negative year). Limit: runtime evidence covers those three Polars versions.

[laughingman7743]

Closing: this approach conflicts with the temporal contract in #1212.

It relies on str.to_datetime() accepting years written with a sign or five or more digits, while the contract makes those raise DataError. Parsing timestamp text by Athena's format, including the fixed-length cut, is now handled in #1216.

Measurements kept for #1216: dropping the cut gave equal results for four-digit years (300,000 random values, 0 to 12 fractional digits, ms/us/ns, Polars 1.39.0, 1.44.2, and 2.0.0) and was 25-40% faster than cutting.