Skip to content

Give PandasCursor's temporal columns one dtype per Athena type - #1273

Merged
laughingman7743 merged 13 commits into
masterfrom
fix/1219-pandas-temporal-dtypes
Oct 18, 2026
Merged

laughingman7743 merged 13 commits into
masterfrom
fix/1219-pandas-temporal-dtypes

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

WHAT

Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type, as in the contract table of #1212. AthenaPandasResultSet is shared by PandasCursor, AsyncPandasCursor, and AioPandasCursor.

Athena type DataFrame dtype Fetch methods
timestamp datetime64[us], fractions beyond six digits truncated datetime
timestamp with time zone datetime64[us, UTC] aware datetime in UTC
date datetime64[us], from CSV and UNLOAD date
time, time with time zone unchanged: object (time) unchanged

NULL is NaT in the DataFrame and None in the fetch methods.

  • CSV results, read by every engine (C, python, PyAthena's PyArrow reader, and pandas' PyArrow engine), whole, in chunks, and as managed results: timestamp, date, and timestamp with time zone columns are read as text and converted per column after reading. Timestamps are cut to 26 characters, which truncates the fraction for years 1 to 9999. Zone names are localized once per zone with ambiguous and nonexistent raising, offsets are subtracted, and the column is converted to UTC. pyarrow-backed text is parsed with pyarrow compute, and the zones are split with regular expressions that pandas runs in pyarrow, so no Python object is made per value.
  • UNLOAD results: date columns are cast to datetime64[us], unless dtype_backend or coerce_int96_timestamp_unit is given to execute().
  • Fetch methods: the temporal columns that PyAthena converted are returned as datetime, date, or aware datetime, and NULL as None, converted per block of rows. Columns that a caller's parse_dates made keep pandas' values.
  • Errors (the PandasCursor part of Name the column in DataError for time zone values in every cursor #1217): a value that cannot be converted, such as one with an ambiguous, nonexistent, or unknown zone, raises DataError naming the column, in whole and chunked reads, instead of an OperationalError that wraps it in whole reads or a pandas or pyarrow error. When the caller's options leave the columns to pandas, the default timestamp with time zone converter runs per value as at the base, and an unknown zone raises OperationalError without the column name. Local times before a zone's first transition, such as 1500-01-01 00:00:00.000 America/New_York, are converted one value at a time.
  • Precedence: when parse_dates, dtype, or converters is given to execute(), PyAthena does not convert the temporal columns after reading. pandas reads them with those options and the values PyAthena sets for the rest, as at the base: PyAthena's parse_dates for date, time, and timestamp columns unless the caller gave parse_dates, and the converter mapping, whose default converts timestamp with time zone values in their zones, unless the caller gave converters. The fetch methods return pandas' values for these columns. This follows the documented rule that dtype replaces PyAthena's whole column type mapping and converters replace the converter mapping, without resolving their keys. A Converter mapping of a temporal type to another function replaces PyAthena's conversion of that type. The default timestamp with time zone converter is recognized by identity, so DefaultPandasTypeConverter.mappings is unchanged.
  • docs/pandas.md describes the dtypes, the fetched types (also from PandasDataFrameIterator.iterrows() and itertuples()), the precedence, and the errors.

Breaking changes, for the 4.2.0 migration notes (#1221):

  • timestamp is always datetime64[us]. It was [ns] for 7 or more fractional digits, [s] when all NULL or with engine="pyarrow" and no fraction, [ns] with engine="pyarrow" and any fraction, and str for a column that also held values outside the nanosecond range.
  • timestamp with time zone is datetime64[us, UTC]. It had the data's zone, or was object for several zones or an all-NULL column.
  • UNLOAD date is datetime64[us]. It was object (date).
  • The fetch methods return datetime and date, and None for NULL. They returned pd.Timestamp and NaT.
  • Time zone errors raise DataError naming the column.
  • engine="pyarrow" is now used for results with timestamp with time zone columns; their default converter had forced the C engine.

WHY

Closes #1219. Part of #1212: the dtypes depended on the values, the engine, and the read mode, and the fetch methods returned pandas values instead of the types of the other cursors.

TEST

Python 3.15.0, pandas 3.0.6, pyarrow 26.0.0. At e3f8da4, on e086eac:

  • just format: no changes. just lint: passed. just docs lint: passed.
  • Offline, with --noconftest and placeholder settings: the offline tests of tests/pyathena/pandas and tests/pyathena/aio/pandas: 268 passed. Thirteen new tests in pandas/test_result_set.py read the full temporal corpus of tests/pyathena/temporal.py through every read path, mixed digit counts and years in one column across chunks, UNLOAD types and options, time zone errors, precedence, local times before a zone's first transition, conversion errors, and engine-independent time zone columns, NULL in the PyArrow engines, and the caller's parse_dates, dtype, and converters with the C, python, and PyArrow engines, with and without future.infer_string. The PyArrow cases assert that the PyArrow reader ran.
  • The two strict #1219 xfails of pandas/test_cursor.py::TestPandasCursor::test_temporal_types are removed, and its fetch check compares types too.
  • Mutation checks, offline; git diff -- pyathena/ was empty after each. Each of these fails at least one test: removing the 26-character cut, as_unit, or tz_convert; flipping the offset sign; removing the UNLOAD date cast or its option check; removing the fetch conversion; ignoring each precedence rule; unwrapping the DataError; removing the ambiguity check; and putting back each round-one repair.

Performance, offline over local result files, 1M rows, the median of 3 runs in fresh processes; the peak RSS is the increase over the process after imports. The CSV and time zone rows were measured at 2090397 against e086eac, the 9-digit and fetchall() rows at eafd63d, and the UNLOAD rows at 7b0a17b4 against 3dda0dc:

Case Base s Branch s Δ time Base peak RSS MiB Branch peak RSS MiB Δ RSS
CSV, C engine (timestamp, date, two timestamp with time zone, time, integer) 8.35 2.81 -66% 414 402 -3%
CSV, PyArrow engine (timestamp, date, integer) 0.50 0.13 -74% 230 207 -10%
CSV, chunks of 100,000 rows 8.44 2.60 -69% 120 134 +12%
CSV, timestamp with 9 digits 0.68 0.58 -15% 167 181 +8%
CSV, one timestamp with time zone column of America/New_York values 4.92 1.05 -79% 124 253 +104%
The same with one value of 1500, before the zone's first transition 4.90 1.04 -79% 135 262 +94%
UNLOAD, whole 0.06 0.12 +83% 101 106 +5%
UNLOAD, chunks of 100,000 rows 0.05 0.10 +97% 43 43 -1%
fetchall() 6.23 2.02 -68% 608 490 -19%

The CSV increases come from holding the temporal text after read_csv() until it is converted, where the C engine converted each value during the read. For a result of only a timestamp with time zone column, that text is most of the peak: pandas' C engine needs 223 MiB to read the 1M values as text. Reading them as object instead raised the peak of every case (the time zone column to 350 MiB) and was not kept. The UNLOAD cases spend about 0.05 s casting the date column.

  • Live, at aaabc3c, with no Test workflow or other live run active: uv run --env-file .env pytest -n 4 tests/pyathena/pandas tests/pyathena/aio/pandas: 421 passed, 1 failed, in 237 s. The failure, test_complex_unload_as_pandas_pyarrow, expected the new datetime64 dtype on the binary column instead of the UNLOAD date column; fixed in the next commit, the test passed live alone.

  • CI after Ready, at e3f8da4: Test run 39099490355 passed (test / run (3.15), lint); test-sqla and test-sqla-async were skipped by the changes path filter.

  • Rebased onto c0bcb6d after Build the pandas result-set tests through the constructor #1268, with conflicts only in the import lines of pandas/test_result_set.py: at 3b62b8d, just lint passed, the offline pandas and aio pandas tests passed (271), and Test run 39101849701 passed (test / run (3.15), lint).

Not run: the live mutations of the covers that are live tests.

🤖 Generated with Claude Code

return pd.to_datetime(text, format="ISO8601").dt.as_unit("us")


def _to_utc_timestamps(values: Series) -> Series:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1 (implementation behavior, contracts, simplicity, regression coverage) of e086eac7bb2120e434c08c2eababccd9d0167fb0..9d6fcbadbb7e827083d434cfdfda9f800820bc2f, with /code-review (high) at 5668c88. Repairs in 1d15de5 (code), e99ce54 (tests), eafd63d (docs), 9d6fcba (warning filter); each repair's test fails when the old code is put back offline, and git diff -- pyathena/ stayed empty.

FINDINGS 1 and 6, repaired: named-zone values before the zone's first transition (1500-01-01 ... America/New_York, 0001-01-01 ... Asia/Tokyo) failed tz_localize(..., nonexistent="raise") and re-raised pandas' ValueError; they are now converted one value at a time, and test_time_zone_before_first_transition covers them. The zones are grouped once instead of a full mask per zone; shifting the group bounds by one fails 5 tests.

@@ -787,7 +999,13 @@ def _get_csv_engine(
is_compatible = (
effective_chunksize is None

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 FINDING 2, repaired: engine="pyarrow" ignored the time zone converter even when the column would not be converted after reading, so user converters reached pandas' PyArrow engine and a dtype entry naming the column returned raw text. The engine choice and the post-read conversion now use one condition; test_time_zone_results_do_not_depend_on_engine (3 cases) fails without it.

return converted if converted.isna().any() else converted.infer_objects()


class _TemporalConverter:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 FINDINGS 4 and 7, repaired: conversion failures other than DataError (pandas ValueError, pyarrow ArrowInvalid, TypeError) escaped raw; any conversion failure now raises DataError naming the column (test_temporal_conversion_error, 6 cases). The column / MultiIndex level / index-name dispatch shared with _JSONConverter is one helper; the index-name branch fails 2 json tests when broken, and the MultiIndex branches are covered only by the live multi-index case of pandas/test_cursor.py.

if d[1] in ("timestamp", "date")
}

def _get_csv_temporal_converters(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 FINDING 5, repaired: the precedence for timestamp with time zone differed from the docs (a parse_dates naming it did not exempt it; any converters did). Now one rule for every temporal type: a parse_dates given to execute() turns all of PyAthena's temporal conversion off, and a dtype or converters entry exempts only the column it names (test_time_zone_options_take_precedence). Not changed, with reason: FINDING 3 (fetch of ArrowDtype timestamp columns) cannot be reached: pandas 3 rejects types_mapper in to_pandas_kwargs, and the other options give NumPy datetime columns, whose fetch conversion is unchanged.

pytest.param({"dtype": "str"}, id="dtype"),
],
)
def test_time_zone_results_do_not_depend_on_engine(self, tmp_path, kwargs):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 FINDING 8, repaired in the docs: PandasDataFrameIterator.iterrows()/itertuples() return the same types as the fetch methods, now stated in docs/pandas.md. The pandas ParserWarning for a caller's dtype together with PyAthena's converter is filtered here, as the other converter-and-dtype tests do. Offline at 9d6fcba: pandas/test_result_set.py 210 passed; with the converter, reader, util, read-options, and offline cursor tests, 253 passed. The benchmark moved within 3 points of the PR body's table.

Comment thread docs/pandas.md Outdated

Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument.
For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns.
PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 2 (claims, existing callers, AWS operation, evidence) of e086eac7bb2120e434c08c2eababccd9d0167fb0..9d6fcbadbb7e827083d434cfdfda9f800820bc2f.

FINDINGS in the PR body, corrected in the description: it still described the precedence as before round 1 (any parse_dates naming a column, and a converters entry exempting the time zone column), the time zone errors only, 240 offline tests, and seven new tests. It now states one precedence rule for every temporal type, DataError for any conversion failure, local times before a zone's first transition, the same condition for engine="pyarrow" and the post-read conversion, 253 offline tests, and eleven new tests.

Checked claims: docs/pandas.md matches the code for the dtypes, the fetched types (also from iterrows()/itertuples()), the precedence list, the errors, the PyArrow engine condition, and the UNLOAD options; just docs lint passed. Existing callers: AsyncPandasCursor and AioPandasCursor read through the same AthenaPandasResultSet, so they get the same dtypes and fetched types; the breaking changes are listed in the PR body for #1221. AWS operation: unchanged; the change reads the same files with the same requests. Evidence: the benchmark is offline over local files; live tests run once at the end.

Comment thread pyathena/pandas/result_set.py Outdated
"header": header,
"names": names,
"dtype": self.dtypes,
"dtype": {**self.dtypes, **dict.fromkeys(temporal, "str")},

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d0d-3ef9-70c3-aa48-02016e5535e0. Static review of e086eac7..9d6fcbad in a clean detached snapshot, with the diff, the #1212 contract, the maintainer's decisions, and the test policy, and no PR text or author findings; no edits, builds, tests, network, or GitHub access. Covered: CSV whole, chunked, and managed results with the C, python, and both PyArrow paths, .txt results, truncation, pre-epoch values, year bounds, NULL and all-NULL columns, mixed zones, offsets, historical zones, DST failures, string inference, index and duplicate labels, precedence, fetch and iterator conversion, async/aio callers, UNLOAD, json/binary/time interactions, failure and close paths, tests, and docs. Result: FINDINGS (5).

Finding 1 (P2): with engine="pyarrow", encoding="utf-8" (pandas' own PyArrow engine), future.infer_string=False, and a file over 100 bytes, pandas applies the injected str dtype and stringifies NULL in a timestamp with time zone column, so _to_utc_timestamps() raises DataError; the C engine and PyAthena's PyArrow reader keep NULL.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 2090397: reproduced, and a NULL date raised DataError too. The pandas PyArrow engine now reads the temporal columns as string[pyarrow], which keeps NULL as NULL; the pandas-pyarrow case of test_temporal_types without future.infer_string (the corpus has NULL timestamp, date, and time zone columns) fails when the fix is reverted (offline).

Comment thread pyathena/pandas/result_set.py Outdated
continue
if "dtype" in self._kwargs and (type(dtype) is not dict or label in dtype):
continue
if label in converters:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review finding 2 (P2): positional converters keys ({0: f} for a first column ts) are not checked by the exemption, so pandas applies f and PyAthena then converts its output and raises DataError; a named key works.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 2090397: dtype and converters entries that name a column by its position in the file exempt it, including with duplicate column names; a parse_dates already turns all conversion off. test_temporal_options_by_position_and_mapping (converter and dtype cases) fails without it. Not covered by a test: positional keys on header-less .txt results.

Comment thread pyathena/pandas/result_set.py Outdated
continue
elif type_ in mappings:
continue
if "dtype" in self._kwargs and (type(dtype) is not dict or label in dtype):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review finding 3 (P2): type(dtype) is not dict treats a mapping subclass such as OrderedDict({"n": "int64"}) as a non-mapping dtype, which exempts every temporal column, so a 9-digit timestamp keeps [ns] and fetch returns Timestamp; a plain dict with the same entries converts it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 2090397: any mapping counts as a mapping, and a defaultdict applies to every column, as pandas applies its default, as docs/pandas.md now states. The OrderedDict and defaultdict cases of test_temporal_options_by_position_and_mapping fail without it. MappingProxyType is not covered: pandas already fails to pickle it.

Comment thread pyathena/pandas/result_set.py Outdated
# pandas reports local times before a zone's first transition as
# nonexistent, so the values are converted one at a time, which raises the
# DataError of a value without one instant.
texts = text.to_numpy(dtype=object)[positions]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review finding 4 (P2): one value before its zone's first transition sends the whole zone group (for example a million modern America/New_York values) through _to_datetime_with_tz() one value at a time, and each failing group materializes the whole text column; this breaks the vectorized-read contract.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 2090397: a zone's values are localized together, and only the values that pandas cannot localize are converted one at a time; test_time_zone_before_first_transition checks that only the two pre-transition values take that path. Benchmark, 1M America/New_York values with one 1500 value: 5.25 s and 347 MiB at 9d6fcba, 1.04 s and 262 MiB now (base 4.90 s and 135 MiB). The remaining peak of a column of only time zone values is pandas' C engine holding the text after read_csv() (223 MiB measured); reading it as object instead raised every case's peak (csv-ny 350 MiB) and was not kept.

tmp_path: The directory of the result files.
kwargs: The result set's options.
"""
columns = temporal.columns()

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review finding 5 (P2): temporal.columns() includes time with time zone, whose default converter keeps the PyArrow engine from being chosen, so the pyarrow and pandas-pyarrow cases read with the C engine despite the size assertion, and a break only in either PyArrow path, such as finding 1, is not caught.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 2090397: the two PyArrow cases drop the time with time zone columns, whose default converter needs the C engine, and assert which reader ran; putting those columns back fails all 4 PyArrow cases (offline).

Comment thread docs/pandas.md Outdated

Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument.
For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns.
PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the independent-review repair, 9d6fcbad..20903970 (one added commit, same base).

Round 1 (behavior and test quality): CLEAN. Each finding's test fails with the repair reverted, offline; git diff -- pyathena/ stayed empty. The per-value fallback now runs only for the values that pandas cannot localize, so a zone group stays vectorized. Positional keys are mapped to labels by file position, as pandas maps them.

Round 2 (claims and operation): FINDINGS, corrected in the PR body: the precedence now includes positional keys and any mapping type (a defaultdict applies to every column); the tests are twelve and 257 offline tests pass at 2090397; the benchmark table adds a column of only time zone values (+104% peak, the text that pandas' C engine holds) and records that reading as object was tried and not kept. just lint and just docs lint passed.

Comment thread pyathena/pandas/result_set.py Outdated
]
self._csv_temporal_converters = self._get_csv_temporal_converters(
[
(i, label, d)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d28-1eee-7490-ad6f-22ef37a8b060. Static review of 9d6fcbad..20903970 in a clean detached snapshot. It found the PyArrow NULL fix, the engine coverage, and the limited per-value fallback sound. Result: FINDINGS (4).

Finding 1 (P2): with engine="python", usecols=["ts"], and converters={0: f} on columns n,ts, pandas resolves position 0 against the selected columns, but PyAthena checks file position 1, keeps its conversion, and raises DataError on f's output.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 791b4f9 and 9db7952 with one key lookup, shared by the post-read conversion and the PyArrow engine condition, that follows pandas' rule per engine (probed): in every engine a key equal to a column label names that column, and a dtype key also covers the duplicates that pandas renames, such as ts.1; with the C engine an integer key also names the column at that position in the file; with the python engine an integer key that is not a label is a position among the columns read; the PyArrow engine takes no positions. docs/pandas.md states the rule. This input: test_temporal_converters_position_with_usecols (python and C engines) fails with the repair reverted (offline).

Comment thread pyathena/pandas/result_set.py Outdated
not isinstance(dtype, Mapping)
or isinstance(dtype, defaultdict)
or label in dtype
or position in dtype

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up finding 2 (P2): every integer key is treated as a position even when it is a label: with names=[1, 0] and dtype={0: "str"}, both columns are exempted, so label 1 keeps nanoseconds and fetches as Timestamp.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 791b4f9: an integer that is a label names that label. test_temporal_options_by_key[python-dtype_label] fails with the repair reverted; with the C engine, which also takes the key as a file position, both columns are named, as pandas applies the dtype to both.

Comment thread pyathena/pandas/result_set.py Outdated
# name.
not isinstance(dtype, Mapping)
or isinstance(dtype, defaultdict)
or label in dtype

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up finding 3 (P2): with duplicate ts,ts and dtype=OrderedDict(ts="str"), pandas applies the dtype to ts.1 too, but PyAthena still converts ts.1, which truncates its fraction and changes its fetched type.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 791b4f9: a dtype key covers the duplicates that pandas applies it to. test_temporal_options_by_key[*-dtype_duplicates] (C and python engines) fails with the repair reverted.

"2020-01-01T18:04:05",
]
np.testing.assert_array_equal(values, np.array(expected, dtype="datetime64[us]"))
assert [c.args for c in spy.call_args_list] == [

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up finding 4 (P2): the fallback test distinguishes the repair only through the exact arguments of a patched private helper; restoring the whole-zone fallback passes every public assertion.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repaired in 791b4f9: test_time_zone_before_first_transition checks only the values and the dtype, without patching a private helper. The vectorization evidence is the benchmark: the column of 1M America/New_York values with one 1500 value takes 0.99 s and 260 MiB at 9db7952, against 5.25 s when the whole zone was converted one value at a time (9d6fcba).

Comment thread docs/pandas.md Outdated

Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument.
For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns.
PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the follow-up repair, 20903970..9db79526 (two added commits, same base).

Round 1 (behavior and test quality): CLEAN. One key lookup resolves dtype and converters keys per engine as pandas does (probed for the C, python, and PyArrow engines), and both the post-read conversion and the PyArrow engine condition use it. Each new test fails with its repair reverted, offline; git diff -- pyathena/ stayed empty. Out-of-range python positions raise pandas' own IndexError, which passes through. A caller's converter on a column that PyAthena's parse_dates lists (a time column) is skipped by pandas' python engine, as at the base.

Round 2 (claims): FINDINGS, corrected in the PR body: the precedence text now states the per-engine key rule; 267 offline tests pass at 9db7952 with fourteen new tests; the vectorization evidence for the pre-transition fallback is the benchmark (0.99 s for 1M values with one 1500 value). just lint and just docs lint passed.

Comment thread pyathena/pandas/result_set.py Outdated
labels = {label for _, label, _ in columns}
# Without names given to execute(), pandas renames duplicates of a name, and
# applies the name's dtype to them.
renamed = "names" not in self._kwargs

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review 2 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d35-3004-7801-b9ce-67bfcbd80817. Static review of 20903970..9db79526, including pandas 3.0.6's python and PyArrow parser source. It found that the new python-label, duplicate-dtype, python-usecols, and PyArrow-position tests fail against 2090397. Result: FINDINGS (3), each in the key lookup: names together with duplicate dtype inheritance (names=["ts", "ts.1"], dtype={"ts": "str"}), a shortened names with python positions (names=["ts"], usecols=[1], dtype={0: "str"}, also for header-less .txt), and negative python positions (dtype={-1: "str"}).

Decision (maintainer): emulating pandas' resolution of dtype and converters keys per engine keeps finding new edge cases, and it follows pandas internals that can change. The precedence becomes one rule without key resolution: when parse_dates, dtype, or converters is given to execute(), PyAthena leaves the temporal columns to pandas, as at the base, and the fetch methods return pandas' values for them. This matches the documented behavior that dtype replaces PyAthena's whole column type mapping and converters replace the converter mapping. A Converter mapping of a temporal type to another function keeps its per-type precedence. The key lookup and its tests are removed in the next commit.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in f7a5e96: the key lookup (positions, renamed duplicates, defaultdict handling) and its tests are removed. When parse_dates, dtype, or converters is given to execute(), PyAthena converts no temporal column after reading, pandas reads them with PyAthena's parse_dates (unless given) and the converter mapping (unless converters are given), and the fetch methods return pandas' values. test_temporal_options_read_by_pandas (C, python, and PyArrow engines) and test_temporal_options_read_by_pandas_with_pyarrow check it; dropping dtype, converters, or parse_dates from the rule fails 3 cases each, and removing the per-type Converter precedence fails test_temporal_options_take_precedence[converter] (offline). The benchmark is unchanged (csv-c 2.70 s / 398 MiB). Kept as at the base: with those options, the default time zone converter runs per value, and an unknown zone raises OperationalError without the column name.

Comment thread docs/pandas.md

Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument.
For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns.
With `parse_dates`, `dtype`, or `converters` passed here, PyAthena does not convert the `date`, `timestamp`, and `timestamp with time zone` columns after reading. pandas reads them with these options and the values PyAthena sets for the others: `parse_dates` for date, time, and timestamp columns, and the converter mapping, which converts `timestamp with time zone` values in their zones. The fetch methods return their values as pandas holds them. A conversion function that the converter maps to one of these types also replaces PyAthena's conversion of the columns of that type.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the rule change, 9db79526..f7a5e96d (one added commit, same base).

Round 1 (behavior and test quality): CLEAN. _get_csv_temporal_converters returns no converters when parse_dates, dtype, or converters is given, and the PyArrow engine condition follows the same rule; the removed code was only the key lookup. The new tests check the rule through the DataFrame and fetched values for each engine, and fail under each mutation of the rule (offline).

Round 2 (claims): FINDINGS, corrected in the PR body: the precedence bullet states the rule and the base behavior that remains with those options, including the OperationalError without the column name for an unknown zone in the per-value converter; the tests are twelve and 261 offline tests pass at f7a5e96. This docs paragraph matches the code and keeps the earlier sentences that dtype replaces the whole mapping and converters replace the converter mapping. just lint and just docs lint passed.

and not self.converters
# pandas' PyArrow engine takes no converters. Those of the temporal
# columns that are converted after reading are not used.
and self._kwargs.get("converters") is None

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review 3 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d53-3b64-7f81-905e-ea1c73155ebe. Static review of 9db79526..f7a5e96d. It found that the option-presence rule reaches the CSV conversion and the fetch paths with no key resolver left, and that docs/pandas.md agrees with it. Result: FINDINGS (2), both repaired in 4a7df0a and aaabc3c.

Finding 1 (P2): an explicit converters=None forced the C engine and counted as given, unlike the base: with engine="pyarrow", converters=None, and parse_dates=[], the base used the PyArrow engine. Repaired: None, pandas' default, counts as not given for parse_dates, dtype, and converters, in the rule and in the PyArrow engine condition. test_temporal_options_none (each option, C and PyArrow) fails when None counts as given, and the converters_none case of test_temporal_options_read_by_pandas_with_pyarrow fails when the engine condition counts it (offline).

)
row = result_set.fetchone()
assert row == expected
# pandas keeps the zone of the time zone values.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review 3, finding 2 (P2): the options test compared aware values only by instant, so a time zone value normalized to UTC passed. Repaired: it also compares the UTC offsets of the aware values; returning UTC values from the per-value converter fails 4 cases (offline).

Self-review of the repair: round 1 CLEAN (225 offline tests in pandas/test_result_set.py pass; just lint passed; git diff -- pyathena/ empty after each mutation); round 2: no claim in the PR body or docs changes.

# objects, as pandas' PyArrow engine does.
pytest.param({"parse_dates": ["ts", "d"]}, id="parse_dates"),
# converters of None, their default, do not rule out the PyArrow engine.
pytest.param({"parse_dates": ["ts", "d"], "converters": None}, id="converters_none"),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review 4 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d5b-302d-70c3-b75f-c1f39489fbf8. Static review of f7a5e96d..aaabc3ce. It found that the source conditions fix the None handling, that the offset comparison detects UTC normalization, and no production regression. Result: FINDINGS (1).

P2: the converters_none case shows the engine choice only through pyarrow.csv.read_csv being called; it proposes asserting that a date column left out of parse_dates is fetched as a date object, which the C engine returns as a string.

Not repaired, with reason: that value difference is pandas' PyArrow engine reading such a column as date objects, not a value that PyAthena defines, so asserting it would test pandas (the testing guide's library-determined values). The engine choice has no PyAthena-defined value of its own; the observable is that pyarrow's public reader runs, which reverting the engine condition fails.

@@ -1034,7 +1027,7 @@ def test_complex_unload_as_pandas_pyarrow(self, pandas_cursor, parquet_engine):
str,
str,
np.datetime64,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Live run and repair, aaabc3ce..e3f8da48. Live at aaabc3c, with no Test workflow or other live run active: tests/pyathena/pandas and tests/pyathena/aio/pandas, 421 passed and 1 failed. test_complex_unload_as_pandas_pyarrow expected the new datetime64 dtype on col_binary instead of the UNLOAD col_date; the expectation moves one entry up, and the test passed live alone at e3f8da4.

Self-review of the repair: round 1 CLEAN (test-only; the expected dtypes now follow the selected columns: col_timestamp and col_date datetime64, col_binary object); round 2: the PR body records the live run, and 268 offline tests pass at {H[:8]}.

@@ -1034,7 +1027,7 @@ def test_complex_unload_as_pandas_pyarrow(self, pandas_cursor, parquet_engine):
str,
str,
np.datetime64,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review 5 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d65-9619-72b3-9e96-2b7dab09c502. Static review of aaabc3ce..e3f8da48. Result: CLEAN. The 16 expected dtypes match the selected columns in order (timestamp and date datetime64; binary, array, map, struct, and decimal object), and the expected row already uses pd.Timestamp for both temporal values of the DataFrame.

@laughingman7743
laughingman7743 marked this pull request as ready for review October 18, 2026 05:07
laughingman7743 and others added 13 commits October 18, 2026 14:47
PandasCursor reads timestamp, date, and timestamp with time zone columns
as text and converts each column at once after reading: timestamp and
date become datetime64[us], fractions beyond six digits truncated, and
timestamp with time zone becomes datetime64[us, UTC], localized once per
zone. The dtypes no longer depend on the values, the engine, chunking,
or managed results, and UNLOAD date columns are datetime64[us] too.
pyarrow-backed text is parsed with pyarrow compute, so no Python object
is made per value.

The fetch methods return datetime, date, aware datetime in UTC, and None
for NULL for the columns that PyAthena converts; columns that a caller's
parse_dates made keep pandas' values. Time zone errors raise DataError
naming the column, in whole and chunked reads. A caller's parse_dates,
dtype or converters entry, or converter mapping still replaces
PyAthena's conversion.

Closes #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atch

A named zone's local times before its first transition, which pandas
reports as nonexistent, are converted one at a time, and only a value
without one instant raises DataError. The values of each zone are
grouped once by a stable sort of the zone codes instead of a mask per
zone, and any conversion failure raises DataError naming the column.
The json and temporal converters find a column or an index level by the
same helper.

The PyArrow engine is chosen only when converters given to execute()
and the converters of the columns not converted after reading would not
be needed, so the results of a read do not depend on the engine. The
fetch methods convert only NumPy-backed datetime columns, as pyarrow-
backed ones keep their values. parse_dates given to execute() keeps
PyAthena from converting any temporal column, and a dtype or converters
entry keeps only the column it names.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne columns, and options

The temporal tests cover a named zone's local times before its first
transition, a value that fails to convert in each temporal type, time
zone columns read with engine="pyarrow" as with the C engine under
converters or dtype, and parse_dates and converters for time zone
columns. The fetch methods again check the datetime64 dtype of a column,
as every converted column is NumPy-backed.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocalize zones together

The PyArrow engine reads the temporal text as pyarrow strings, as pandas'
PyArrow engine turns NULL into "None" with the NumPy str dtype. dtype and
converters entries name a column by its label or its position, as in
pandas, any mapping other than a defaultdict names its columns, and a
defaultdict applies to every column. The values of a zone are localized
together, and only those that pandas does not localize, such as local
times before the zone's first transition, are converted one at a time.
The PyArrow cases of the temporal types test leave out the time with
time zone columns, so they read with PyAthena's and pandas' PyArrow
readers, which the test checks.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dtype or converters key names the column with that label, and a dtype
key also the columns that pandas renamed from duplicates of that name.
The C engine also takes an integer key as a position in the file, the
Python engine, if no column has that label, as a position among the
columns read, and the PyArrow engine takes no positions. The test of
local times before a zone's first transition checks the values and the
dtype only.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umns

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With any of these options given to execute(), PyAthena converts no
temporal column after reading: pandas reads them with PyAthena's
parse_dates and converter mapping unless the options replace them, and
the fetch methods return pandas' values. The resolution of dtype and
converters keys and its tests are removed.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pandas' defaults for these options are None, so a None passed to
execute() keeps PyAthena's temporal conversion and the PyArrow engine.
The options test also checks that pandas keeps the zone of the time
zone values.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expectation had marked the binary column instead of the date column.

Refs #1219

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@laughingman7743
laughingman7743 force-pushed the fix/1219-pandas-temporal-dtypes branch from e3f8da4 to 3b62b8d Compare October 18, 2026 05:48
import threading
from collections import defaultdict
from datetime import UTC, datetime
from datetime import UTC, date, datetime

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase onto c0bcb6d after #1268 (c0bcb6d) changed tests/pyathena/pandas/test_result_set.py. The patch series is the same 13 commits; git range-diff e086eac7..e3f8da48 c0bcb6d6..3b62b8d6 shows changes only in commits 1, 7, 8, and 10, where the conflicts were import lines of that test file (resolved as their union, with unused imports removed) and context from #1268 (_InMemoryS3Object, S3FileSystem, the converter argument of _local_result_set). Upstream changes checked for this PR's contracts: #1268 moved the pandas result-set tests onto the constructor, which this PR's tests already use; #1269 changed the S3 multipart copy, which the pandas reads do not use.

Re-checked at 3b62b8d: just lint passed; offline, the pandas and aio pandas tests pass (271), including #1268's moved tests through PyAthena's temporal conversion; tests/pyathena collects 3,607 tests.

import threading
from collections import defaultdict
from datetime import UTC, datetime
from datetime import UTC, date, datetime

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review of the rebase (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d8e-e38a-7e10-bb0d-ee41206ca89a, with the rebased diff and the range-diff against e3f8da4. Result: CLEAN. All 13 temporal tests and #1268's moved and new tests keep their bodies, parameters, and decorators; the json date tests still use pandas' parse_dates, which turns the new temporal conversion off; the open-options, range, and Parquet tests use non-temporal columns; no stale helper references or missing imports.

@laughingman7743
laughingman7743 merged commit 5aecab9 into master Oct 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type

1 participant