Repository navigation
Test the temporal types of each synchronous cursor - #1226
Conversation
| # timestamp_tz_ambiguous with the text of the second row replaced (#1220). | ||
| Case("timestamp_tz_nonexistent", None), | ||
| Case( | ||
| "timestamp_tz_unknown_zone", |
There was a problem hiding this comment.
Self-review round one (implementation behavior). Base b2d7aa24b00a204e9e150952c708e3bb1126f9e2, head 895d1c5bf15a69c81473581f91edd3d2b2892713. Result: FINDINGS. One finding, repaired in 1389f37.
Inventory covered:
fake_aws.py: Athena paging (header row on the first page,NextToken), UNLOAD manifest and locations, managed storage without an output location, S3 ranged GET, HEAD, ListObjectsV2.corpus.py: capture and write, including managed storage and the case edited by hand.expected.py: the hand-written values.test_conformance.py: modes, read cache, endpoint and time zone contexts, gap functions, and assertions.test_corpus.py: the live comparison.
Finding (test isolation). _time_zone_database() hides the whole time zone database during reads of timestamp_tz_unknown_zone. The neighboring rows of that case used UTC, which resolved only because importing expected.py had already cached ZoneInfo("UTC"). If the cache were cold, the error would name UTC instead of Pacific/Chatham, or the test would fail for an unrelated reason. Repair: the neighboring rows now use the offsets +05:30 and -08:00, which need no time zone database. The case was captured again from Athena.
Verified, no change needed:
- The simulation works. In a fresh process, Cursor and PandasCursor return a naive
datetimeforPacific/Chathamand aware values for the offsets, which is today's behavior that Define time zone conversion for values whose zone cannot be represented #1220 changes. - The row-group mode serves 11 row groups, where Athena's file has 1 (fake_aws.py:98).
- Chunked modes produce several chunks: the per-chunk dtype lists in the failures show them.
fetchall,fetchmany,fetchone, and iteration take the same paths in the sync and asyncio cursors.
Checks after the repair:
just lintpassed.pytest --noconftest tests/pyathena/temporal/ -n 8: 4,603 passed and 1,486 xfailed.
|
|
||
| The cursors read the frozen corpus from a local endpoint, without AWS. The dtype, the | ||
| values, the NULLs, and the errors are separate tests, so that a known gap fails only the | ||
| checks that it breaks. Known gaps are strict xfails that name the issue that fixes them; |
There was a problem hiding this comment.
Self-review round two (claims, callers, operations). Base b2d7aa24b00a204e9e150952c708e3bb1126f9e2, head e3d0f5937bf5a34a4f1334efd5cba889323e1122. Result: FINDINGS. Three corrections, no change in behavior.
Claims checked against evidence:
- "18 queries captured, plus one case edited by hand":
CASEShas 19 cases, 18 with a query. - "Athena writes one row group for this corpus": all 8 UNLOAD files have 1 row group.
- "200 KB":
duoftests/resources/temporal. - "1,486 strict xfails": with
--runxfail, exactly 1,486 tests fail. - "Failures are the same across sync, futures-based, and asyncio forms, and across fetch methods": measured again with
--runxfailafter the time zone cases were added. No differences. - "Parquet bytes differ between runs": observed when the corpus was captured twice. The decoded values and
created_bywere identical. - The regeneration command in the
corpus.pydocstring works and skips the case edited by hand. - Name the column in DataError for time zone values in every cursor #1217 comment, Arrow and Polars
dateoutside the range: matches the failing tests (Arrow CSV reads year 0 and raisesOperationalErrorfor signed years; Polars CSV and both UNLOAD readers read all three). - Supported Python versions: 3.11, 3.13, and 3.15 give the same results (Polars 1.44.2 and 2.0.0, pyarrow 25.0.1 and 26.0.0).
Corrections:
- The module docstring referenced a removed
_known_gap(). It now names the_gap_<issue>()functions (e3d0f59). - The PR body said pyarrow is "the only patched class", but the unknown-zone case also hides the time zone database. It now says "the only class patched to reach the endpoint". The TEST section was refreshed for e3d0f59.
- The Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219 comment said chunked reads give
objector[ns]chunks. They can also bestr. The comment was edited.
AWS operator: the live test adds 7 queries in the test work group and 6 in the managed work group per run, all on literals. The conformance tests make no AWS calls; their os.environ changes are scoped to each read and restored. Each xdist worker is a separate process.
97deeac to
4936167
Compare
|
|
||
|
|
||
| @contextlib.contextmanager | ||
| def _time_zone_database(case: str) -> Iterator[None]: |
There was a problem hiding this comment.
Independent review (relayed). Reviewer: Codex CLI 0.162.1, model gpt-6-astra, codex exec -s read-only, session 01a14767-8556-7d03-b5e0-43b26654eaf6. Static review only: no tests or builds were run. Snapshot: detached worktree at e3d0f5937bf5a34a4f1334efd5cba889323e1122, base b2d7aa24b00a204e9e150952c708e3bb1126f9e2. Prompt: the diff and the contract, without the PR text or the earlier findings. Coverage, as the reviewer reported it: all six files, the mode and xfail matrix, the fake Athena and S3, endpoint and cache isolation, capture and write, the live comparison, and the expected values against the queries and frozen data. Result: FINDINGS (6 findings, 1 minor suggestion).
1. [P2] Missing-zone simulation leaks converter cache state. _parse_time_zone() is cached separately from ZoneInfo. The unknown-zone read caches Pacific/Chatham → None, which survives teardown. A previously cached valid zone would also bypass the simulation. Suggested: clear the converter cache before and after, and restore the saved TZPATH instead of the default.
There was a problem hiding this comment.
Verified and repaired in 97deeac, then revised in 4936167 after rebasing onto master with #1224. The context clears _parse_time_zone before and after the read and restores the saved TZPATH. #1224 made pandas show that hiding the whole tzdata package was unrealistic: it raised ModuleNotFoundError, and tzdata is now a required dependency. The simulation therefore puts a tzdata package whose zoneinfo directory is empty first on sys.path, and removes it afterwards. The other rows of the case are offsets (1389f37), so they do not depend on cached zones.
| if isinstance(e, KeyboardInterrupt | SystemExit): | ||
| raise | ||
| # The traceback would keep the cursors of the read alive. | ||
| _reads[key] = Read(error=e.with_traceback(None)) |
There was a problem hiding this comment.
2. [P2] Cached exceptions retain unclosed cursor resources (relayed from the independent review above). _execute() closed connections but not cursors or the result sets that futures return. Cached exceptions kept their tracebacks, and with them the cursors and executors, for the worker's lifetime.
There was a problem hiding this comment.
Verified and repaired in 97deeac. Cursors are now context-managed (sync, futures-based, and async with for asyncio), and the result set that a future returns is closed. Cached errors drop their tracebacks, and _reads is cleared at module teardown. After about 15 asynchronous reads, including failing ones, the thread count stays at 6 (it was 2 before the reads) instead of growing per read.
| "session": session, | ||
| "cursor_class": mode.cursor_class, | ||
| # Athena rejects an output location with managed query result storage. | ||
| "s3_staging_dir": "" |
There was a problem hiding this comment.
3. [P2] Managed modes use a configuration that real Athena rejects (relayed). Every mode passed an S3 staging directory, which sets ResultConfiguration.OutputLocation, and that conflicts with managed results (docs/usage.md:103). The fake accepted it.
There was a problem hiding this comment.
Verified and repaired in 97deeac. Managed modes pass s3_staging_dir="", as docs/usage.md describes. The fake now rejects ResultConfiguration.OutputLocation in the managed work group with InvalidRequestException, the same message Athena gives; a probe with a staging directory raised DatabaseError from that error.
| assert output.csv == case.csv | ||
| if name == "timestamp": | ||
| # The file names are the capture's, and only their count differs. | ||
| ignored = {"files"} |
There was a problem hiding this comment.
4. [P2] Live UNLOAD comparison treats file layout as temporal drift (relayed). Requiring the same file count and manifest order fails if Athena splits identical rows differently. Suggested: compare combined rows by id and the physical schemas separately.
There was a problem hiding this comment.
Verified and repaired in 97deeac. The live check compares the UNLOAD statement types, ColumnInfo, and rows, but not the capture-generated file names. It then compares the set of physical schemas and the combined rows sorted by id (INT96 read at microseconds). It passed against Athena (12 passed).
|
|
||
| @pytest.mark.parametrize( | ||
| ("mode", "case", "column", "method", "athena_type"), | ||
| _value_params("nulls", _frame_and_fetch_methods), |
There was a problem hiding this comment.
5. [P2] NULL assertions omit the other fetch methods (relayed). test_nulls covered only fetchall, and test_values skips NULL positions. A fix limited to fetchall would go undetected in fetchone, fetchmany, and iteration.
| return nulls | ||
|
|
||
|
|
||
| def _names(error: BaseException, column: str) -> bool: |
There was a problem hiding this comment.
6. [P2] Column-name assertion accepts generic date errors (relayed). The column d is a substring of messages like "invalid date", so an error without the column name could XPASS. Suggested: match whole words.
Minor (relayed): fake_aws.py should return 416 for unsatisfiable ranges. The reviewer found no path that sends one.
There was a problem hiding this comment.
Verified and repaired in 97deeac: _names() matches the column as a whole word (\b<column>\b), for out-of-range and time zone errors. The minor 416 suggestion is deferred: no cursor path sends an unsatisfiable range to this corpus, and an invalid response would fail visibly.
| _reads: dict[tuple[str, str, str], Read] = {} | ||
|
|
||
|
|
||
| def _without_tracebacks(error: BaseException) -> BaseException: |
There was a problem hiding this comment.
Independent follow-up review (relayed). Reviewer: Codex CLI 0.162.1, codex exec -s read-only, static only. Snapshot: detached worktree at 4936167db86983f41284e5d921045d6f8127f59f, new base ff4e468c499148829a007dc819cae8ce82c38e0c. Package: the range-diff of both patch series, the repair diff since e3d0f59, and the upstream changes to pyathena/ between the bases (#1224's converter changes). Coverage, as the reviewer reported it: all six repairs; module, environment, and TZPATH restoration; cursor and result-set lifetimes; managed requests and fake responses; UNLOAD normalization; NULL coverage; column-name assertions; and the _gap_* rules against the rebased code. It reported that the temporary tzdata package exercises ZoneInfo's missing-zone fallback, that cleanup restores the original modules and the saved TZPATH, and that the xfails referencing only #1217 match. Result: FINDINGS (2).
1. [P2] Chained exception tracebacks remain cached. with_traceback(None) cleared only the outer traceback. For example, PandasCursor's whole-read OperationalError from DataError (pandas/result_set.py:1111) kept its cause's traceback, with the partly built result set and reader.
Repair (4d97e96): _without_tracebacks() clears the tracebacks of the error and of every __cause__ and __context__, with cycle protection.
There was a problem hiding this comment.
Independent follow-up on the repair (relayed): CLEAN. Reviewer: Codex CLI 0.162.1, codex exec -s read-only. Static review with local data inspection; no tests run. Snapshot: 4d97e96f7e1e1604083a9bfe5677c9f773853523. Scope: the diff 4936167d..4d97e96f and the frozen date data. Coverage: stale d/d_null references in the sources and frozen data; the _gap_* rules, parameters, expected values, and UNLOAD projections; the JSON, CSV, and Parquet of all four date cases; and the cause/context traversal, including shared nodes and cycles. Both findings are resolved, with no new findings.
| return nulls | ||
|
|
||
|
|
||
| def _names(error: BaseException, column: str) -> bool: |
There was a problem hiding this comment.
2. [P2] Whole-word matching still mistakes %d for column d (relayed from the follow-up review above). \bd\b matches the d in '%Y-%m-%d', so the negative-year date message without a column name would satisfy the check once it became DataError.
Repair (4d97e96): the date columns are now date_col and date_null, captured again from Athena (4 cases), so no format code or parser word matches them. This avoids prescribing #1217's message format. _names(DataError("time data '-0001-01-02' does not match format '%Y-%m-%d'"), 'date_col') is False.
Validation at 4d97e96: just lint passed. Offline on Python 3.15 and 3.11: 5,419 passed and 1,558 xfailed. With conftest.py: 5,431 passed and 1,558 xfailed. The live corpus test: 12 passed.
Each synchronous cursor runs one SELECT whose columns cover the temporal formats Athena writes: 0 to 12 fractional digits, years 1, 1600, 2300, and 9999, NULL, UTC, a named zone, an offset, dates, and times. Cursors that read UNLOAD results also run its UNLOAD form. The values, dtypes, and time zones are checked against the contract of #1212, and the current gaps are strict xfails that name the issue that fixes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4d97e96 to
5065d62
Compare
PandasCursor holds aware values in UTC, so the zone of the fetched values is not checked, and the expected DataFrame values come from a helper instead of a nested conditional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| return converter | ||
|
|
||
|
|
||
| def _pandas_value(column: temporal.Column): |
There was a problem hiding this comment.
Self-review round one (implementation, reworked design). Base 18c1eea1 (master), head 25569f1a065ff9c322eade38529dc1b297fe01f2. Result: FINDINGS. Two findings, repaired in 25569f1.
Inventory: tests/pyathena/temporal.py (columns, UNLOAD subset, query, expected row, zones), and test_temporal_types in the five sync cursor test classes.
- The pandas test checked the zone of fetched aware values. PandasCursor holds them in UTC (Define and enforce the contract for Athena temporal types across cursors #1212, Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219), so the check would fail once Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219 lands, although that behavior is correct. Repair: pandas compares aware values by instant only.
- The expected DataFrame values came from a nested conditional expression. Repair: the
_pandas_value()helper.
Verified, no change needed:
- Each xfail fails at its dtype check with
--runxfail. - The UNLOAD form selects only
TIMESTAMP(3)anddate. zones()comparesZoneInfoandtimezoneobjects by equality, sotimezone.utcandZoneInfo('UTC')differ.- The NULL columns compare as
Nonein fetch and as the container's null in frames. pd.Timestamp == dateis False, so a fetchedTimestampfordatefails as intended.
| ] | ||
|
|
||
| # UNLOAD writes only TIMESTAMP(3) and date of these types. | ||
| UNLOAD_COLUMNS = [ |
There was a problem hiding this comment.
Self-review round two (claims and operations). Head 25569f1a065ff9c322eade38529dc1b297fe01f2. Result: CLEAN after the round-one repairs.
Claims checked against evidence:
- The gap table in the PR: measured per value on live Athena before the tests were written, and each xfail reproduces it with
--runxfail. - "UNLOAD writes only
TIMESTAMP(3)anddate": the Define and enforce the contract for Athena temporal types across cursors #1212 facts. Other precisions failed withNOT_SUPPORTEDin the earlier audit. - "The async cursors and managed results return the same values": the earlier measurement for Test the temporal types of each synchronous cursor #1226's first design found no difference between sync, futures-based, and asyncio forms, and managed GetQueryResults text equals the CSV text.
- Python 3.11 with Polars 1.44.2 and pyarrow 25.0.1 gives the same 3 passed and 5 xfailed.
AWS operator: 14 queries per run, all on literals (8 SELECTs and 6 UNLOADs). No new fixtures or session resources.
- Fetch from the execution whose DataFrame or table was checked, so each test parameter runs one query. - Add timestamp with time zone values in years 1 and 9999, and a time with time zone value with 12 fractional digits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| "03:04:05.123+05:30", | ||
| ), | ||
| Column( | ||
| "time_tz_p12", |
There was a problem hiding this comment.
Independent review (relayed). Reviewer: Codex CLI 0.162.1, codex exec -s read-only, static only. Snapshot: detached worktree at 25569f1a065ff9c322eade38529dc1b297fe01f2, base 18c1eea1. Prompt: the diff, the contract, and the repository conventions, without the PR text or the earlier findings. Coverage, as the reviewer reported it: all six files; SQL and expected values; precision and year boundaries; NULLs; pandas equality (Timestamp, date, NaT); container dtypes; fetched zones; xfail placement; and conventions. It found no contradiction between the SQL and the expected values. Result: FINDINGS (3).
1. [P2] time with time zone with 12 fractional digits was missing. A path that rejects or rounds TIME(12) WITH TIME ZONE would pass.
Repair (66bfa6a): added time_tz_p12. The fetch methods expect .123456+05:30, and Arrow and Polars expect Athena's text 03:04:05.123456789012+05:30.
There was a problem hiding this comment.
Independent follow-up on the repair (relayed): CLEAN. Reviewer: Codex CLI 0.162.1, codex exec -s read-only, static only. Snapshot: 66bfa6a47079c7a34d25e8ac7d2df0b1deb53e29. Scope: the diff 25569f1a..66bfa6a4. All three findings are resolved. The new expected values match their SQL, and the Arrow and Polars text matches Athena's. The reviewer confirmed in the cursor code that unchunked pandas, Arrow, and Polars reads return cached frames without advancing their fetch iterators, with and without UNLOAD, so fetching from the same execution is valid.
| "2020-01-02 03:04:05.123456789012 UTC", | ||
| ), | ||
| Column( | ||
| "tstz_min", |
There was a problem hiding this comment.
2. [P2] Aware timestamps lacked years 1 and 9999 (relayed). A conversion through nanosecond storage would pass with 2020 values only.
Repair (66bfa6a): added tstz_min and tstz_max in UTC.
| _pandas_value(c) for c in columns | ||
| ] | ||
| # The fetch methods read the same DataFrame, whose aware values are in UTC. | ||
| row = pandas_cursor.fetchone() |
There was a problem hiding this comment.
3. [P2] The DataFrame tests ran the query twice (relayed), for the frame and again for the fetch methods.
Repair (66bfa6a): the DataFrame, Arrow, and Polars tests fetch from the execution they read. A probe confirmed that fetchone() after as_pandas() or as_arrow() returns the row, both with and without UNLOAD. Each test parameter now runs one query (8 per run).
Validation at 66bfa6a: just lint passed. Live Athena on Python 3.15 and 3.11: 3 passed and 5 xfailed. With --runxfail, each xfail fails at its dtype check.
WHAT
Adds one temporal type test to each synchronous cursor, checking its values against the contract of #1212 with live Athena output.
tests/pyathena/temporal.pydefines one SELECT whose columns cover the temporal formats Athena writes:timestampwith 0, 3, 6, 9, and 12 fractional digits, a fraction before 1970, and years 1, 1600, 2300, and 9999.datein years 2020, 1, and 9999.timestamp with time zonewithUTC,Asia/Tokyo,+05:30, 12 fractional digits, and years 1 and 9999 inUTC.timeandtime with time zone(with an offset), each with 3 and 12 fractional digits.Each column lists the value that the fetch methods return and, for the types that ArrowCursor and PolarsCursor keep as text, Athena's text. The UNLOAD form selects only the
TIMESTAMP(3)anddatecolumns, the only temporal types UNLOAD writes.test_temporal_typesinTestCursor,TestS3FSCursor,TestPandasCursor,TestArrowCursor, andTestPolarsCursorruns the query once. The DataFrame cursors read the DataFrame or table and then fetch from the same execution.timezoneor aZoneInfo. PandasCursor's aware values are compared by instant, because its DataFrame holds them in UTC.unload.The current gaps are strict xfails that name the issue that fixes them:
[ns],str, and[s]dtypes, zones in the dtype, fetchedNaTandTimestamp[ns];dateisobjectdateistimestamp[ms]; NULL text is''nsThe output depends on the read path, UNLOAD or CSV. The asynchronous cursors and managed results return the same values as the synchronous cursors, so they have no tests of their own. Years outside 1 to 9999 are out of scope (#1212).
WHY
Closes #1213. Part of #1212.
TEST
Tested commit: 66bfa6a.
just lint: passed.uv run --env-file .env pytest -n 4 -p no:randomly tests/pyathena -k test_temporal_types(live Athena): 3 passed and 5 xfailed in 38 s. The same result on Python 3.11 with Polars 1.44.2 and pyarrow 25.0.1.--runxfail, each of the 5 xfails fails at its dtype check.🤖 Generated with Claude Code