Skip to content

Give PandasCursor's temporal columns one dtype per Athena type #1273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laughingman7743 merged 13 commits into from
Oct 18, 2026
Merged

Give PandasCursor's temporal columns one dtype per Athena type #1273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
0967010
Give PandasCursor's temporal columns one dtype per Athena type
laughingman7743 Oct 18, 2026
b752a34
Describe PandasCursor's temporal dtypes and fetched types
laughingman7743 Oct 18, 2026
b765f4d
Convert early local times, group zones once, and share the label disp…
laughingman7743 Oct 18, 2026
d9a4ba0
Test early local times, conversion errors, engine-independent time zo…
laughingman7743 Oct 18, 2026
244ae20
Describe the temporal conversion precedence, errors, and iterator rows
laughingman7743 Oct 18, 2026
7afc37d
Ignore pandas' converter-and-dtype warning in the engine test
laughingman7743 Oct 18, 2026
acf5baa
Keep NULL in every engine, take positional and mapping options, and l…
laughingman7743 Oct 18, 2026
3da05c4
Resolve dtype and converters keys of temporal columns as pandas does
laughingman7743 Oct 18, 2026
bab3840
Test that the PyArrow engine takes no dtype positions of temporal col…
laughingman7743 Oct 18, 2026
36b23f0
Leave temporal columns to pandas with parse_dates, dtype, or converters
laughingman7743 Oct 18, 2026
b59362f
Treat a None parse_dates, dtype, or converters as not given
laughingman7743 Oct 18, 2026
d39163c
Check that converters of None keep the PyArrow engine
laughingman7743 Oct 18, 2026
3b62b8d
Expect datetime64 for the UNLOAD date column of the complex pyarrow test
laughingman7743 Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions docs/pandas.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,9 @@ for row in cursor:
print(row)
```

The DATE and TIMESTAMP of Athena's data type are returned as [pandas.Timestamp](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.html) type.
In a DataFrame, `timestamp` and `date` columns are `datetime64[us]`, and `timestamp with time zone` columns are `datetime64[us, UTC]`, with fractional digits beyond microseconds truncated.
The fetch methods return their values as `datetime`, `date`, and aware `datetime` objects in UTC, and NULL as `None`.
A value that cannot be converted, such as a `timestamp with time zone` value with an unknown zone or with a local time that its zone repeats or skips, raises `DataError` naming the column.

```python
from pyathena import connect
Expand All @@ -231,7 +233,7 @@ cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
cursor_class=PandasCursor).cursor()

cursor.execute("SELECT col_timestamp FROM one_row_complex")
print(type(cursor.fetchone()[0])) # <class 'pandas._libs.tslibs.timestamps.Timestamp'>
print(type(cursor.fetchone()[0])) # <class 'datetime.datetime'>
```

Execution information of the query can also be retrieved.
Expand Down Expand Up @@ -390,6 +392,7 @@ awsathena+pandas://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_sta
When this option is used, the object returned by the as_pandas method is a `PandasDataFrameIterator` object.
This object is an iterator of DataFrames, like the pandas `TextFileReader` object.
It also provides `get_chunk()`, `as_pandas()`, `iterrows()`, `itertuples()`, and `close()`, and can be used as a context manager.
Its `iterrows()` and `itertuples()` return the values of temporal columns as the fetch methods do.

```python
from pyathena import connect
Expand Down Expand Up @@ -566,7 +569,7 @@ PandasCursor reads a `json` column as text and applies the converter mapping's `
A `json` column with NULL in the result or chunk holds objects, with `None` for NULL; otherwise it has the dtype that pandas infers from the converted values.
Converters passed to `execute()` replace the converter mapping and are applied by pandas.

With `engine="pyarrow"`, PandasCursor uses the PyArrow engine only when the result is not a tab-separated `.txt` file, pyarrow is installed, no chunksize is set (explicitly or by `auto_optimize_chunksize`), `quoting` is the default, the result has no columns that need a converter (`boolean`, `decimal`, `varbinary`, `json`, `time with time zone`, and `timestamp with time zone` with the default converter), and the result file is at least `AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES` bytes.
With `engine="pyarrow"`, PandasCursor uses the PyArrow engine only when the result is not a tab-separated `.txt` file, pyarrow is installed, no chunksize is set (explicitly or by `auto_optimize_chunksize`), `quoting` is the default, the result has no columns that need a converter (`boolean`, `decimal`, `varbinary`, `json`, `time with time zone`, columns of a type that the converter maps to a conversion function, and `timestamp with time zone` columns that PyAthena does not convert after reading), no `converters` are given to `execute()`, and the result file is at least `AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES` bytes.
Otherwise, it falls back to the C engine.
With `engine="pyarrow"`, tab-separated `.txt` results from DDL statements such as `SHOW TABLES`, `SHOW COLUMNS`, and `DESCRIBE` use the C engine to preserve leading zeros, exponent notation, and padding in string values.
With the PyArrow engine, when `keep_default_na` and `na_values` are the defaults and the only pandas.read_csv() options passed to `execute()` are `dtype` as a mapping and `parse_dates` as a list, PyAthena reads the file with `pyarrow.csv`, allowing newlines in quoted values, and converts it to a DataFrame as `pandas.read_csv(engine="pyarrow")` does, except that columns with a string dtype are read as text, and without `future.infer_string`, a column with dtype `str` keeps NULL as a missing value.
Expand All @@ -575,6 +578,7 @@ It also changes the values of a column with a string dtype whose values all look

Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument.
For example, `dtype` replaces the whole column type mapping, and `parse_dates` replaces the list of date, time, and timestamp columns.
With `parse_dates`, `dtype`, or `converters` passed here, PyAthena does not convert the `date`, `timestamp`, and `timestamp with time zone` columns after reading. pandas reads them with these options and the values PyAthena sets for the others: `parse_dates` for date, time, and timestamp columns, and the converter mapping, which converts `timestamp with time zone` values in their zones. The fetch methods return their values as pandas holds them. A conversion function that the converter maps to one of these types also replaces PyAthena's conversion of the columns of that type.

@laughingman7743 laughingman7743 Oct 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the rule change, 9db79526..f7a5e96d (one added commit, same base).

Round 1 (behavior and test quality): CLEAN. _get_csv_temporal_converters returns no converters when parse_dates, dtype, or converters is given, and the PyArrow engine condition follows the same rule; the removed code was only the key lookup. The new tests check the rule through the DataFrame and fetched values for each engine, and fail under each mutation of the rule (offline).

Round 2 (claims): FINDINGS, corrected in the PR body: the precedence bullet states the rule and the base behavior that remains with those options, including the OperationalError without the column name for an unknown zone in the per-value converter; the tests are twelve and 261 offline tests pass at f7a5e96. This docs paragraph matches the code and keeps the earlier sentences that dtype replaces the whole mapping and converters replace the converter mapping. just lint and just docs lint passed.


(pandas-s3-read-options)=

Expand Down Expand Up @@ -604,7 +608,8 @@ With `storage_options` given to `execute()`, or `filesystem` for an UNLOAD resul

PandasCursor also supports the unload option, as does {ref}`arrow-cursor`.
Reading the unloaded Parquet files requires pyarrow (`pip install PyAthena[Pandas,Arrow]`), and `engine` must be `auto` or `pyarrow`.
Timestamps are read at microseconds, as `datetime64[us]`, unless `coerce_int96_timestamp_unit` is given to `execute()`.
Timestamps are read at microseconds, as `datetime64[us]`, and dates as `datetime64[us]`, as from CSV results.
With `coerce_int96_timestamp_unit` or `dtype_backend` given to `execute()`, the columns have the dtypes that these options give, and the fetch methods return their values as pandas holds them.

See {ref}`arrow-unload-options` for more information.

Expand Down Expand Up @@ -768,7 +773,9 @@ print(df.describe())
print(df.head())
```

The DATE and TIMESTAMP of Athena's data type are returned as [pandas.Timestamp](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.html) type.
In a DataFrame, `timestamp` and `date` columns are `datetime64[us]`, and `timestamp with time zone` columns are `datetime64[us, UTC]`, with fractional digits beyond microseconds truncated.
The fetch methods return their values as `datetime`, `date`, and aware `datetime` objects in UTC, and NULL as `None`.
A value that cannot be converted, such as a `timestamp with time zone` value with an unknown zone or with a local time that its zone repeats or skips, raises `DataError` naming the column.

```python
from pyathena import connect
Expand All @@ -780,7 +787,7 @@ cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",

query_id, future = cursor.execute("SELECT col_timestamp FROM one_row_complex")
result_set = future.result()
print(type(result_set.fetchone()[0])) # <class 'pandas._libs.tslibs.timestamps.Timestamp'>
print(type(result_set.fetchone()[0])) # <class 'datetime.datetime'>
```

As with AsyncCursor, you need a query ID to cancel a query.
Expand Down
Loading
Loading