Repository navigation
Give PandasCursor's temporal columns one dtype per Athena type - #1273
Conversation
| return pd.to_datetime(text, format="ISO8601").dt.as_unit("us") | ||
|
|
||
|
|
||
| def _to_utc_timestamps(values: Series) -> Series: |
There was a problem hiding this comment.
Self-review round 1 (implementation behavior, contracts, simplicity, regression coverage) of e086eac7bb2120e434c08c2eababccd9d0167fb0..9d6fcbadbb7e827083d434cfdfda9f800820bc2f, with /code-review (high) at 5668c88. Repairs in 1d15de5 (code), e99ce54 (tests), eafd63d (docs), 9d6fcba (warning filter); each repair's test fails when the old code is put back offline, and git diff -- pyathena/ stayed empty.
FINDINGS 1 and 6, repaired: named-zone values before the zone's first transition (1500-01-01 ... America/New_York, 0001-01-01 ... Asia/Tokyo) failed tz_localize(..., nonexistent="raise") and re-raised pandas' ValueError; they are now converted one value at a time, and test_time_zone_before_first_transition covers them. The zones are grouped once instead of a full mask per zone; shifting the group bounds by one fails 5 tests.
| @@ -787,7 +999,13 @@ def _get_csv_engine( | |||
| is_compatible = ( | |||
| effective_chunksize is None | |||
There was a problem hiding this comment.
Round 1 FINDING 2, repaired: engine="pyarrow" ignored the time zone converter even when the column would not be converted after reading, so user converters reached pandas' PyArrow engine and a dtype entry naming the column returned raw text. The engine choice and the post-read conversion now use one condition; test_time_zone_results_do_not_depend_on_engine (3 cases) fails without it.
| return converted if converted.isna().any() else converted.infer_objects() | ||
|
|
||
|
|
||
| class _TemporalConverter: |
There was a problem hiding this comment.
Round 1 FINDINGS 4 and 7, repaired: conversion failures other than DataError (pandas ValueError, pyarrow ArrowInvalid, TypeError) escaped raw; any conversion failure now raises DataError naming the column (test_temporal_conversion_error, 6 cases). The column / MultiIndex level / index-name dispatch shared with _JSONConverter is one helper; the index-name branch fails 2 json tests when broken, and the MultiIndex branches are covered only by the live multi-index case of pandas/test_cursor.py.
| if d[1] in ("timestamp", "date") | ||
| } | ||
|
|
||
| def _get_csv_temporal_converters( |
There was a problem hiding this comment.
Round 1 FINDING 5, repaired: the precedence for timestamp with time zone differed from the docs (a parse_dates naming it did not exempt it; any converters did). Now one rule for every temporal type: a parse_dates given to execute() turns all of PyAthena's temporal conversion off, and a dtype or converters entry exempts only the column it names (test_time_zone_options_take_precedence). Not changed, with reason: FINDING 3 (fetch of ArrowDtype timestamp columns) cannot be reached: pandas 3 rejects types_mapper in to_pandas_kwargs, and the other options give NumPy datetime columns, whose fetch conversion is unchanged.
| pytest.param({"dtype": "str"}, id="dtype"), | ||
| ], | ||
| ) | ||
| def test_time_zone_results_do_not_depend_on_engine(self, tmp_path, kwargs): |
There was a problem hiding this comment.
Round 1 FINDING 8, repaired in the docs: PandasDataFrameIterator.iterrows()/itertuples() return the same types as the fetch methods, now stated in docs/pandas.md. The pandas ParserWarning for a caller's dtype together with PyAthena's converter is filtered here, as the other converter-and-dtype tests do. Offline at 9d6fcba: pandas/test_result_set.py 210 passed; with the converter, reader, util, read-options, and offline cursor tests, 253 passed. The benchmark moved within 3 points of the PR body's table.
|
|
||
| Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument. | ||
| For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns. | ||
| PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide: |
There was a problem hiding this comment.
Self-review round 2 (claims, existing callers, AWS operation, evidence) of e086eac7bb2120e434c08c2eababccd9d0167fb0..9d6fcbadbb7e827083d434cfdfda9f800820bc2f.
FINDINGS in the PR body, corrected in the description: it still described the precedence as before round 1 (any parse_dates naming a column, and a converters entry exempting the time zone column), the time zone errors only, 240 offline tests, and seven new tests. It now states one precedence rule for every temporal type, DataError for any conversion failure, local times before a zone's first transition, the same condition for engine="pyarrow" and the post-read conversion, 253 offline tests, and eleven new tests.
Checked claims: docs/pandas.md matches the code for the dtypes, the fetched types (also from iterrows()/itertuples()), the precedence list, the errors, the PyArrow engine condition, and the UNLOAD options; just docs lint passed. Existing callers: AsyncPandasCursor and AioPandasCursor read through the same AthenaPandasResultSet, so they get the same dtypes and fetched types; the breaking changes are listed in the PR body for #1221. AWS operation: unchanged; the change reads the same files with the same requests. Evidence: the benchmark is offline over local files; live tests run once at the end.
| "header": header, | ||
| "names": names, | ||
| "dtype": self.dtypes, | ||
| "dtype": {**self.dtypes, **dict.fromkeys(temporal, "str")}, |
There was a problem hiding this comment.
Independent review (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d0d-3ef9-70c3-aa48-02016e5535e0. Static review of e086eac7..9d6fcbad in a clean detached snapshot, with the diff, the #1212 contract, the maintainer's decisions, and the test policy, and no PR text or author findings; no edits, builds, tests, network, or GitHub access. Covered: CSV whole, chunked, and managed results with the C, python, and both PyArrow paths, .txt results, truncation, pre-epoch values, year bounds, NULL and all-NULL columns, mixed zones, offsets, historical zones, DST failures, string inference, index and duplicate labels, precedence, fetch and iterator conversion, async/aio callers, UNLOAD, json/binary/time interactions, failure and close paths, tests, and docs. Result: FINDINGS (5).
Finding 1 (P2): with engine="pyarrow", encoding="utf-8" (pandas' own PyArrow engine), future.infer_string=False, and a file over 100 bytes, pandas applies the injected str dtype and stringifies NULL in a timestamp with time zone column, so _to_utc_timestamps() raises DataError; the C engine and PyAthena's PyArrow reader keep NULL.
There was a problem hiding this comment.
Repaired in 2090397: reproduced, and a NULL date raised DataError too. The pandas PyArrow engine now reads the temporal columns as string[pyarrow], which keeps NULL as NULL; the pandas-pyarrow case of test_temporal_types without future.infer_string (the corpus has NULL timestamp, date, and time zone columns) fails when the fix is reverted (offline).
| continue | ||
| if "dtype" in self._kwargs and (type(dtype) is not dict or label in dtype): | ||
| continue | ||
| if label in converters: |
There was a problem hiding this comment.
Independent review finding 2 (P2): positional converters keys ({0: f} for a first column ts) are not checked by the exemption, so pandas applies f and PyAthena then converts its output and raises DataError; a named key works.
There was a problem hiding this comment.
Repaired in 2090397: dtype and converters entries that name a column by its position in the file exempt it, including with duplicate column names; a parse_dates already turns all conversion off. test_temporal_options_by_position_and_mapping (converter and dtype cases) fails without it. Not covered by a test: positional keys on header-less .txt results.
| continue | ||
| elif type_ in mappings: | ||
| continue | ||
| if "dtype" in self._kwargs and (type(dtype) is not dict or label in dtype): |
There was a problem hiding this comment.
Independent review finding 3 (P2): type(dtype) is not dict treats a mapping subclass such as OrderedDict({"n": "int64"}) as a non-mapping dtype, which exempts every temporal column, so a 9-digit timestamp keeps [ns] and fetch returns Timestamp; a plain dict with the same entries converts it.
There was a problem hiding this comment.
Repaired in 2090397: any mapping counts as a mapping, and a defaultdict applies to every column, as pandas applies its default, as docs/pandas.md now states. The OrderedDict and defaultdict cases of test_temporal_options_by_position_and_mapping fail without it. MappingProxyType is not covered: pandas already fails to pickle it.
| # pandas reports local times before a zone's first transition as | ||
| # nonexistent, so the values are converted one at a time, which raises the | ||
| # DataError of a value without one instant. | ||
| texts = text.to_numpy(dtype=object)[positions] |
There was a problem hiding this comment.
Independent review finding 4 (P2): one value before its zone's first transition sends the whole zone group (for example a million modern America/New_York values) through _to_datetime_with_tz() one value at a time, and each failing group materializes the whole text column; this breaks the vectorized-read contract.
There was a problem hiding this comment.
Repaired in 2090397: a zone's values are localized together, and only the values that pandas cannot localize are converted one at a time; test_time_zone_before_first_transition checks that only the two pre-transition values take that path. Benchmark, 1M America/New_York values with one 1500 value: 5.25 s and 347 MiB at 9d6fcba, 1.04 s and 262 MiB now (base 4.90 s and 135 MiB). The remaining peak of a column of only time zone values is pandas' C engine holding the text after read_csv() (223 MiB measured); reading it as object instead raised every case's peak (csv-ny 350 MiB) and was not kept.
| tmp_path: The directory of the result files. | ||
| kwargs: The result set's options. | ||
| """ | ||
| columns = temporal.columns() |
There was a problem hiding this comment.
Independent review finding 5 (P2): temporal.columns() includes time with time zone, whose default converter keeps the PyArrow engine from being chosen, so the pyarrow and pandas-pyarrow cases read with the C engine despite the size assertion, and a break only in either PyArrow path, such as finding 1, is not caught.
There was a problem hiding this comment.
Repaired in 2090397: the two PyArrow cases drop the time with time zone columns, whose default converter needs the C engine, and assert which reader ran; putting those columns back fails all 4 PyArrow cases (offline).
|
|
||
| Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument. | ||
| For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns. | ||
| PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide: |
There was a problem hiding this comment.
Self-review of the independent-review repair, 9d6fcbad..20903970 (one added commit, same base).
Round 1 (behavior and test quality): CLEAN. Each finding's test fails with the repair reverted, offline; git diff -- pyathena/ stayed empty. The per-value fallback now runs only for the values that pandas cannot localize, so a zone group stays vectorized. Positional keys are mapped to labels by file position, as pandas maps them.
Round 2 (claims and operation): FINDINGS, corrected in the PR body: the precedence now includes positional keys and any mapping type (a defaultdict applies to every column); the tests are twelve and 257 offline tests pass at 2090397; the benchmark table adds a column of only time zone values (+104% peak, the text that pandas' C engine holds) and records that reading as object was tried and not kept. just lint and just docs lint passed.
| ] | ||
| self._csv_temporal_converters = self._get_csv_temporal_converters( | ||
| [ | ||
| (i, label, d) |
There was a problem hiding this comment.
Independent follow-up review (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d28-1eee-7490-ad6f-22ef37a8b060. Static review of 9d6fcbad..20903970 in a clean detached snapshot. It found the PyArrow NULL fix, the engine coverage, and the limited per-value fallback sound. Result: FINDINGS (4).
Finding 1 (P2): with engine="python", usecols=["ts"], and converters={0: f} on columns n,ts, pandas resolves position 0 against the selected columns, but PyAthena checks file position 1, keeps its conversion, and raises DataError on f's output.
There was a problem hiding this comment.
Repaired in 791b4f9 and 9db7952 with one key lookup, shared by the post-read conversion and the PyArrow engine condition, that follows pandas' rule per engine (probed): in every engine a key equal to a column label names that column, and a dtype key also covers the duplicates that pandas renames, such as ts.1; with the C engine an integer key also names the column at that position in the file; with the python engine an integer key that is not a label is a position among the columns read; the PyArrow engine takes no positions. docs/pandas.md states the rule. This input: test_temporal_converters_position_with_usecols (python and C engines) fails with the repair reverted (offline).
| not isinstance(dtype, Mapping) | ||
| or isinstance(dtype, defaultdict) | ||
| or label in dtype | ||
| or position in dtype |
There was a problem hiding this comment.
Follow-up finding 2 (P2): every integer key is treated as a position even when it is a label: with names=[1, 0] and dtype={0: "str"}, both columns are exempted, so label 1 keeps nanoseconds and fetches as Timestamp.
There was a problem hiding this comment.
Repaired in 791b4f9: an integer that is a label names that label. test_temporal_options_by_key[python-dtype_label] fails with the repair reverted; with the C engine, which also takes the key as a file position, both columns are named, as pandas applies the dtype to both.
| # name. | ||
| not isinstance(dtype, Mapping) | ||
| or isinstance(dtype, defaultdict) | ||
| or label in dtype |
There was a problem hiding this comment.
Follow-up finding 3 (P2): with duplicate ts,ts and dtype=OrderedDict(ts="str"), pandas applies the dtype to ts.1 too, but PyAthena still converts ts.1, which truncates its fraction and changes its fetched type.
There was a problem hiding this comment.
Repaired in 791b4f9: a dtype key covers the duplicates that pandas applies it to. test_temporal_options_by_key[*-dtype_duplicates] (C and python engines) fails with the repair reverted.
| "2020-01-01T18:04:05", | ||
| ] | ||
| np.testing.assert_array_equal(values, np.array(expected, dtype="datetime64[us]")) | ||
| assert [c.args for c in spy.call_args_list] == [ |
There was a problem hiding this comment.
Follow-up finding 4 (P2): the fallback test distinguishes the repair only through the exact arguments of a patched private helper; restoring the whole-zone fallback passes every public assertion.
There was a problem hiding this comment.
Repaired in 791b4f9: test_time_zone_before_first_transition checks only the values and the dtype, without patching a private helper. The vectorization evidence is the benchmark: the column of 1M America/New_York values with one 1500 value takes 0.99 s and 260 MiB at 9db7952, against 5.25 s when the whole zone was converted one value at a time (9d6fcba).
|
|
||
| Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument. | ||
| For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns. | ||
| PyAthena converts `date`, `timestamp`, and `timestamp with time zone` columns after reading, except for the columns that these options decide: |
There was a problem hiding this comment.
Self-review of the follow-up repair, 20903970..9db79526 (two added commits, same base).
Round 1 (behavior and test quality): CLEAN. One key lookup resolves dtype and converters keys per engine as pandas does (probed for the C, python, and PyArrow engines), and both the post-read conversion and the PyArrow engine condition use it. Each new test fails with its repair reverted, offline; git diff -- pyathena/ stayed empty. Out-of-range python positions raise pandas' own IndexError, which passes through. A caller's converter on a column that PyAthena's parse_dates lists (a time column) is skipped by pandas' python engine, as at the base.
Round 2 (claims): FINDINGS, corrected in the PR body: the precedence text now states the per-engine key rule; 267 offline tests pass at 9db7952 with fourteen new tests; the vectorization evidence for the pre-transition fallback is the benchmark (0.99 s for 1M values with one 1500 value). just lint and just docs lint passed.
| labels = {label for _, label, _ in columns} | ||
| # Without names given to execute(), pandas renames duplicates of a name, and | ||
| # applies the name's dtype to them. | ||
| renamed = "names" not in self._kwargs |
There was a problem hiding this comment.
Independent follow-up review 2 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d35-3004-7801-b9ce-67bfcbd80817. Static review of 20903970..9db79526, including pandas 3.0.6's python and PyArrow parser source. It found that the new python-label, duplicate-dtype, python-usecols, and PyArrow-position tests fail against 2090397. Result: FINDINGS (3), each in the key lookup: names together with duplicate dtype inheritance (names=["ts", "ts.1"], dtype={"ts": "str"}), a shortened names with python positions (names=["ts"], usecols=[1], dtype={0: "str"}, also for header-less .txt), and negative python positions (dtype={-1: "str"}).
Decision (maintainer): emulating pandas' resolution of dtype and converters keys per engine keeps finding new edge cases, and it follows pandas internals that can change. The precedence becomes one rule without key resolution: when parse_dates, dtype, or converters is given to execute(), PyAthena leaves the temporal columns to pandas, as at the base, and the fetch methods return pandas' values for them. This matches the documented behavior that dtype replaces PyAthena's whole column type mapping and converters replace the converter mapping. A Converter mapping of a temporal type to another function keeps its per-type precedence. The key lookup and its tests are removed in the next commit.
There was a problem hiding this comment.
Applied in f7a5e96: the key lookup (positions, renamed duplicates, defaultdict handling) and its tests are removed. When parse_dates, dtype, or converters is given to execute(), PyAthena converts no temporal column after reading, pandas reads them with PyAthena's parse_dates (unless given) and the converter mapping (unless converters are given), and the fetch methods return pandas' values. test_temporal_options_read_by_pandas (C, python, and PyArrow engines) and test_temporal_options_read_by_pandas_with_pyarrow check it; dropping dtype, converters, or parse_dates from the rule fails 3 cases each, and removing the per-type Converter precedence fails test_temporal_options_take_precedence[converter] (offline). The benchmark is unchanged (csv-c 2.70 s / 398 MiB). Kept as at the base: with those options, the default time zone converter runs per value, and an unknown zone raises OperationalError without the column name.
|
|
||
| Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument. | ||
| For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns. | ||
| With `parse_dates`, `dtype`, or `converters` passed here, PyAthena does not convert the `date`, `timestamp`, and `timestamp with time zone` columns after reading. pandas reads them with these options and the values PyAthena sets for the others: `parse_dates` for date, time, and timestamp columns, and the converter mapping, which converts `timestamp with time zone` values in their zones. The fetch methods return their values as pandas holds them. A conversion function that the converter maps to one of these types also replaces PyAthena's conversion of the columns of that type. |
There was a problem hiding this comment.
Self-review of the rule change, 9db79526..f7a5e96d (one added commit, same base).
Round 1 (behavior and test quality): CLEAN. _get_csv_temporal_converters returns no converters when parse_dates, dtype, or converters is given, and the PyArrow engine condition follows the same rule; the removed code was only the key lookup. The new tests check the rule through the DataFrame and fetched values for each engine, and fail under each mutation of the rule (offline).
Round 2 (claims): FINDINGS, corrected in the PR body: the precedence bullet states the rule and the base behavior that remains with those options, including the OperationalError without the column name for an unknown zone in the per-value converter; the tests are twelve and 261 offline tests pass at f7a5e96. This docs paragraph matches the code and keeps the earlier sentences that dtype replaces the whole mapping and converters replace the converter mapping. just lint and just docs lint passed.
| and not self.converters | ||
| # pandas' PyArrow engine takes no converters. Those of the temporal | ||
| # columns that are converted after reading are not used. | ||
| and self._kwargs.get("converters") is None |
There was a problem hiding this comment.
Independent follow-up review 3 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d53-3b64-7f81-905e-ea1c73155ebe. Static review of 9db79526..f7a5e96d. It found that the option-presence rule reaches the CSV conversion and the fetch paths with no key resolver left, and that docs/pandas.md agrees with it. Result: FINDINGS (2), both repaired in 4a7df0a and aaabc3c.
Finding 1 (P2): an explicit converters=None forced the C engine and counted as given, unlike the base: with engine="pyarrow", converters=None, and parse_dates=[], the base used the PyArrow engine. Repaired: None, pandas' default, counts as not given for parse_dates, dtype, and converters, in the rule and in the PyArrow engine condition. test_temporal_options_none (each option, C and PyArrow) fails when None counts as given, and the converters_none case of test_temporal_options_read_by_pandas_with_pyarrow fails when the engine condition counts it (offline).
| ) | ||
| row = result_set.fetchone() | ||
| assert row == expected | ||
| # pandas keeps the zone of the time zone values. |
There was a problem hiding this comment.
Independent follow-up review 3, finding 2 (P2): the options test compared aware values only by instant, so a time zone value normalized to UTC passed. Repaired: it also compares the UTC offsets of the aware values; returning UTC values from the per-value converter fails 4 cases (offline).
Self-review of the repair: round 1 CLEAN (225 offline tests in pandas/test_result_set.py pass; just lint passed; git diff -- pyathena/ empty after each mutation); round 2: no claim in the PR body or docs changes.
| # objects, as pandas' PyArrow engine does. | ||
| pytest.param({"parse_dates": ["ts", "d"]}, id="parse_dates"), | ||
| # converters of None, their default, do not rule out the PyArrow engine. | ||
| pytest.param({"parse_dates": ["ts", "d"], "converters": None}, id="converters_none"), |
There was a problem hiding this comment.
Independent follow-up review 4 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d5b-302d-70c3-b75f-c1f39489fbf8. Static review of f7a5e96d..aaabc3ce. It found that the source conditions fix the None handling, that the offset comparison detects UTC normalization, and no production regression. Result: FINDINGS (1).
P2: the converters_none case shows the engine choice only through pyarrow.csv.read_csv being called; it proposes asserting that a date column left out of parse_dates is fetched as a date object, which the C engine returns as a string.
Not repaired, with reason: that value difference is pandas' PyArrow engine reading such a column as date objects, not a value that PyAthena defines, so asserting it would test pandas (the testing guide's library-determined values). The engine choice has no PyAthena-defined value of its own; the observable is that pyarrow's public reader runs, which reverting the engine condition fails.
| @@ -1034,7 +1027,7 @@ def test_complex_unload_as_pandas_pyarrow(self, pandas_cursor, parquet_engine): | |||
| str, | |||
| str, | |||
| np.datetime64, | |||
There was a problem hiding this comment.
Live run and repair, aaabc3ce..e3f8da48. Live at aaabc3c, with no Test workflow or other live run active: tests/pyathena/pandas and tests/pyathena/aio/pandas, 421 passed and 1 failed. test_complex_unload_as_pandas_pyarrow expected the new datetime64 dtype on col_binary instead of the UNLOAD col_date; the expectation moves one entry up, and the test passed live alone at e3f8da4.
Self-review of the repair: round 1 CLEAN (test-only; the expected dtypes now follow the selected columns: col_timestamp and col_date datetime64, col_binary object); round 2: the PR body records the live run, and 268 offline tests pass at {H[:8]}.
| @@ -1034,7 +1027,7 @@ def test_complex_unload_as_pandas_pyarrow(self, pandas_cursor, parquet_engine): | |||
| str, | |||
| str, | |||
| np.datetime64, | |||
There was a problem hiding this comment.
Independent follow-up review 5 (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d65-9619-72b3-9e96-2b7dab09c502. Static review of aaabc3ce..e3f8da48. Result: CLEAN. The 16 expected dtypes match the selected columns in order (timestamp and date datetime64; binary, array, map, struct, and decimal object), and the expected row already uses pd.Timestamp for both temporal values of the DataFrame.
PandasCursor reads timestamp, date, and timestamp with time zone columns as text and converts each column at once after reading: timestamp and date become datetime64[us], fractions beyond six digits truncated, and timestamp with time zone becomes datetime64[us, UTC], localized once per zone. The dtypes no longer depend on the values, the engine, chunking, or managed results, and UNLOAD date columns are datetime64[us] too. pyarrow-backed text is parsed with pyarrow compute, so no Python object is made per value. The fetch methods return datetime, date, aware datetime in UTC, and None for NULL for the columns that PyAthena converts; columns that a caller's parse_dates made keep pandas' values. Time zone errors raise DataError naming the column, in whole and chunked reads. A caller's parse_dates, dtype or converters entry, or converter mapping still replaces PyAthena's conversion. Closes #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atch A named zone's local times before its first transition, which pandas reports as nonexistent, are converted one at a time, and only a value without one instant raises DataError. The values of each zone are grouped once by a stable sort of the zone codes instead of a mask per zone, and any conversion failure raises DataError naming the column. The json and temporal converters find a column or an index level by the same helper. The PyArrow engine is chosen only when converters given to execute() and the converters of the columns not converted after reading would not be needed, so the results of a read do not depend on the engine. The fetch methods convert only NumPy-backed datetime columns, as pyarrow- backed ones keep their values. parse_dates given to execute() keeps PyAthena from converting any temporal column, and a dtype or converters entry keeps only the column it names. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne columns, and options The temporal tests cover a named zone's local times before its first transition, a value that fails to convert in each temporal type, time zone columns read with engine="pyarrow" as with the C engine under converters or dtype, and parse_dates and converters for time zone columns. The fetch methods again check the datetime64 dtype of a column, as every converted column is NumPy-backed. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocalize zones together The PyArrow engine reads the temporal text as pyarrow strings, as pandas' PyArrow engine turns NULL into "None" with the NumPy str dtype. dtype and converters entries name a column by its label or its position, as in pandas, any mapping other than a defaultdict names its columns, and a defaultdict applies to every column. The values of a zone are localized together, and only those that pandas does not localize, such as local times before the zone's first transition, are converted one at a time. The PyArrow cases of the temporal types test leave out the time with time zone columns, so they read with PyAthena's and pandas' PyArrow readers, which the test checks. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dtype or converters key names the column with that label, and a dtype key also the columns that pandas renamed from duplicates of that name. The C engine also takes an integer key as a position in the file, the Python engine, if no column has that label, as a position among the columns read, and the PyArrow engine takes no positions. The test of local times before a zone's first transition checks the values and the dtype only. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umns Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With any of these options given to execute(), PyAthena converts no temporal column after reading: pandas reads them with PyAthena's parse_dates and converter mapping unless the options replace them, and the fetch methods return pandas' values. The resolution of dtype and converters keys and its tests are removed. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pandas' defaults for these options are None, so a None passed to execute() keeps PyAthena's temporal conversion and the PyArrow engine. The options test also checks that pandas keeps the zone of the time zone values. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expectation had marked the binary column instead of the date column. Refs #1219 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e3f8da4 to
3b62b8d
Compare
| import threading | ||
| from collections import defaultdict | ||
| from datetime import UTC, datetime | ||
| from datetime import UTC, date, datetime |
There was a problem hiding this comment.
Rebase onto c0bcb6d after #1268 (c0bcb6d) changed tests/pyathena/pandas/test_result_set.py. The patch series is the same 13 commits; git range-diff e086eac7..e3f8da48 c0bcb6d6..3b62b8d6 shows changes only in commits 1, 7, 8, and 10, where the conflicts were import lines of that test file (resolved as their union, with unused imports removed) and context from #1268 (_InMemoryS3Object, S3FileSystem, the converter argument of _local_result_set). Upstream changes checked for this PR's contracts: #1268 moved the pandas result-set tests onto the constructor, which this PR's tests already use; #1269 changed the S3 multipart copy, which the pandas reads do not use.
Re-checked at 3b62b8d: just lint passed; offline, the pandas and aio pandas tests pass (271), including #1268's moved tests through PyAthena's temporal conversion; tests/pyathena collects 3,607 tests.
| import threading | ||
| from collections import defaultdict | ||
| from datetime import UTC, datetime | ||
| from datetime import UTC, date, datetime |
There was a problem hiding this comment.
Independent follow-up review of the rebase (relayed): Codex CLI 0.163.0, model gpt-6-astra, ChatGPT login, codex exec -s read-only, session 01a14d8e-e38a-7e10-bb0d-ee41206ca89a, with the rebased diff and the range-diff against e3f8da4. Result: CLEAN. All 13 temporal tests and #1268's moved and new tests keep their bodies, parameters, and decorators; the json date tests still use pandas' parse_dates, which turns the new temporal conversion off; the open-options, range, and Parquet tests use non-temporal columns; no stale helper references or missing imports.
WHAT
Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type, as in the contract table of #1212.
AthenaPandasResultSetis shared byPandasCursor,AsyncPandasCursor, andAioPandasCursor.timestampdatetime64[us], fractions beyond six digits truncateddatetimetimestamp with time zonedatetime64[us, UTC]datetimein UTCdatedatetime64[us], from CSV and UNLOADdatetime,time with time zoneobject(time)NULL is
NaTin the DataFrame andNonein the fetch methods.timestamp,date, andtimestamp with time zonecolumns are read as text and converted per column after reading. Timestamps are cut to 26 characters, which truncates the fraction for years 1 to 9999. Zone names are localized once per zone withambiguousandnonexistentraising, offsets are subtracted, and the column is converted to UTC. pyarrow-backed text is parsed with pyarrow compute, and the zones are split with regular expressions that pandas runs in pyarrow, so no Python object is made per value.datecolumns are cast todatetime64[us], unlessdtype_backendorcoerce_int96_timestamp_unitis given toexecute().datetime,date, or awaredatetime, and NULL asNone, converted per block of rows. Columns that a caller'sparse_datesmade keep pandas' values.DataErrornaming the column, in whole and chunked reads, instead of anOperationalErrorthat wraps it in whole reads or a pandas or pyarrow error. When the caller's options leave the columns to pandas, the defaulttimestamp with time zoneconverter runs per value as at the base, and an unknown zone raisesOperationalErrorwithout the column name. Local times before a zone's first transition, such as1500-01-01 00:00:00.000 America/New_York, are converted one value at a time.parse_dates,dtype, orconvertersis given toexecute(), PyAthena does not convert the temporal columns after reading. pandas reads them with those options and the values PyAthena sets for the rest, as at the base: PyAthena'sparse_datesfordate,time, andtimestampcolumns unless the caller gaveparse_dates, and the converter mapping, whose default convertstimestamp with time zonevalues in their zones, unless the caller gaveconverters. The fetch methods return pandas' values for these columns. This follows the documented rule thatdtypereplaces PyAthena's whole column type mapping andconvertersreplace the converter mapping, without resolving their keys. AConvertermapping of a temporal type to another function replaces PyAthena's conversion of that type. The defaulttimestamp with time zoneconverter is recognized by identity, soDefaultPandasTypeConverter.mappingsis unchanged.docs/pandas.mddescribes the dtypes, the fetched types (also fromPandasDataFrameIterator.iterrows()anditertuples()), the precedence, and the errors.Breaking changes, for the 4.2.0 migration notes (#1221):
timestampis alwaysdatetime64[us]. It was[ns]for 7 or more fractional digits,[s]when all NULL or withengine="pyarrow"and no fraction,[ns]withengine="pyarrow"and any fraction, andstrfor a column that also held values outside the nanosecond range.timestamp with time zoneisdatetime64[us, UTC]. It had the data's zone, or wasobjectfor several zones or an all-NULL column.dateisdatetime64[us]. It wasobject(date).datetimeanddate, andNonefor NULL. They returnedpd.TimestampandNaT.DataErrornaming the column.engine="pyarrow"is now used for results withtimestamp with time zonecolumns; their default converter had forced the C engine.WHY
Closes #1219. Part of #1212: the dtypes depended on the values, the engine, and the read mode, and the fetch methods returned pandas values instead of the types of the other cursors.
TEST
Python 3.15.0, pandas 3.0.6, pyarrow 26.0.0. At e3f8da4, on e086eac:
just format: no changes.just lint: passed.just docs lint: passed.--noconftestand placeholder settings: the offline tests oftests/pyathena/pandasandtests/pyathena/aio/pandas: 268 passed. Thirteen new tests inpandas/test_result_set.pyread the full temporal corpus oftests/pyathena/temporal.pythrough every read path, mixed digit counts and years in one column across chunks, UNLOAD types and options, time zone errors, precedence, local times before a zone's first transition, conversion errors, and engine-independent time zone columns, NULL in the PyArrow engines, and the caller'sparse_dates,dtype, andconverterswith the C, python, and PyArrow engines, with and withoutfuture.infer_string. The PyArrow cases assert that the PyArrow reader ran.#1219xfails ofpandas/test_cursor.py::TestPandasCursor::test_temporal_typesare removed, and its fetch check compares types too.git diff -- pyathena/was empty after each. Each of these fails at least one test: removing the 26-character cut,as_unit, ortz_convert; flipping the offset sign; removing the UNLOAD date cast or its option check; removing the fetch conversion; ignoring each precedence rule; unwrapping theDataError; removing the ambiguity check; and putting back each round-one repair.Performance, offline over local result files, 1M rows, the median of 3 runs in fresh processes; the peak RSS is the increase over the process after imports. The CSV and time zone rows were measured at 2090397 against e086eac, the 9-digit and
fetchall()rows at eafd63d, and the UNLOAD rows at 7b0a17b4 against 3dda0dc:timestamp,date, twotimestamp with time zone,time, integer)timestamp,date, integer)timestampwith 9 digitstimestamp with time zonecolumn ofAmerica/New_Yorkvaluesfetchall()The CSV increases come from holding the temporal text after
read_csv()until it is converted, where the C engine converted each value during the read. For a result of only atimestamp with time zonecolumn, that text is most of the peak: pandas' C engine needs 223 MiB to read the 1M values as text. Reading them asobjectinstead raised the peak of every case (the time zone column to 350 MiB) and was not kept. The UNLOAD cases spend about 0.05 s casting thedatecolumn.Live, at aaabc3c, with no Test workflow or other live run active:
uv run --env-file .env pytest -n 4 tests/pyathena/pandas tests/pyathena/aio/pandas: 421 passed, 1 failed, in 237 s. The failure,test_complex_unload_as_pandas_pyarrow, expected the newdatetime64dtype on the binary column instead of the UNLOAD date column; fixed in the next commit, the test passed live alone.CI after Ready, at e3f8da4: Test run 39099490355 passed (
test / run (3.15), lint);test-sqlaandtest-sqla-asyncwere skipped by thechangespath filter.Rebased onto c0bcb6d after Build the pandas result-set tests through the constructor #1268, with conflicts only in the import lines of
pandas/test_result_set.py: at 3b62b8d,just lintpassed, the offline pandas and aio pandas tests passed (271), and Test run 39101849701 passed (test / run (3.15), lint).Not run: the live mutations of the covers that are live tests.
🤖 Generated with Claude Code