Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/development-workflow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Prefer existing fixtures and integration classes; do not build a parallel mock f
| SQLAlchemy dialect | Relevant PyAthena tests in `tests/pyathena/sqlalchemy/` plus `just test sqla`; also `tests/pyathena/aio/sqlalchemy/` and `just test sqla-async` for shared reflection or async paths. Both PyAthena directories are included in `just test pyathena`. |
| Filesystem | Affected S3 filesystem tests, including listing and path boundaries |
| Documentation | `just docs lint`; `just docs build` for rendered documentation changes |
| Agent skills | `just docs lint` and `mise exec -- rumdl check .agents/skills`; inspect links and walk through realistic workflow scenarios |
| Agent skills | `just docs lint`; inspect links and walk through realistic workflow scenarios |
| Dependency, packaging, or CI configuration | Affected build/install checks and supported Python/dependency combinations from `pyproject.toml`, `uv.lock`, and CI |

The `tests/pyathena/` directory includes AWS integration tests despite the recipe's "unit tests" label.
Expand Down
12 changes: 11 additions & 1 deletion .rumdl.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ include = [
"scripts/**/*.md",
"cloudformation/**/*.md",
"/*.md",
".agents/skills/**/*.md",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result; static review)

  • Reviewer: Codex CLI 0.166.0 (codex exec --sandbox read-only), model gpt-6-astra, session 01a1b461-da50-7af3-8709-c02ec9404044
  • Base: 2a60ecc1f2a7a6bd0d9293525a41ea0d642a0082
  • Head: 6cb01956bb79e540567632677c463c825f151cd7
  • Package: a detached snapshot of the head, the literal diff, and repository conventions. The prompt left out the PR number, PR description, commit messages and self-review findings.
  • Constraints: read-only; no edits, builds, tests, commits, pushes or GitHub access. Read-only rumdl checks were allowed.
  • After the review, the snapshot and the PR worktree were both clean and still at the head.

Result: CLEAN

Reviewer output:

Covered all 20 changed files: rumdl configuration and file discovery; documentation meaning, identifiers, links, MyST targets, headings and tables; related instructions and CI; and license-checker behavior and remaining test coverage.

CLEAN

  • rumdl 0.3.2 passes all 30 selected files with caching disabled and configuration warnings treated as errors. The include change adds exactly four canonical skill files and excludes no previously selected files.
  • MD044 recognizes the configured options and preserves code spelling. Removing the three suppression comments in stdin-only lint checks produces exactly the intended lowercase-target violations. Dropping MD043 is consistent with its previously empty heading requirements.
  • Documentation edits preserve identifiers, option values, link destinations and reference targets. No introduced meaning or structural defects, or stale related instructions, were found.
  • No .jsonc files or dangling references remain. The deleted test row covered the removed syntax; retained formats and the fallback remain represented in existing tests.

No actionable pre-existing issues were identified within scope.

Validation was read-only inspection and permitted rumdl checks. No builds, tests, edits or GitHub access occurred; rendered-output assessment was static.

]
# Run the rule set of markdownlint 0.38 (used by markdownlint-cli2 0.18.1,
# the previous linter) instead of every rule rumdl enables by default.
Expand All @@ -24,14 +25,15 @@ enable = [
"MD012", "MD018", "MD019", "MD020", "MD021", "MD022", "MD023", "MD024",
"MD025", "MD026", "MD027", "MD028", "MD029", "MD030", "MD031", "MD032",
"MD033", "MD034", "MD035", "MD036", "MD037", "MD038", "MD039", "MD040",
"MD042", "MD043", "MD044", "MD045", "MD046", "MD047", "MD048", "MD049",
"MD042", "MD044", "MD045", "MD046", "MD047", "MD048", "MD049",
"MD050", "MD051", "MD052", "MD053", "MD054", "MD055", "MD056", "MD058",
"MD059",
# Not enabled:
# MD013: docs have long URLs and code lines that would be awkward to wrap
# MD014: `$ command` style is intentional in docs/testing.md and README shell snippets
# MD041: MyST `(label)=` ref targets must come before the first heading, so the
# first line of many docs/*.md files is not an H1
# MD043: the docs share no single heading structure to require
]

# Sphinx renders docs/ with MyST, so directive fences such as ```{toctree}
Expand Down Expand Up @@ -80,3 +82,11 @@ ignore = ["x"]

[MD053]
ignored-definitions = ["//"]

# Spell project and library names consistently in prose; code is exempt.
# rumdl also checks MyST `(label)=` targets, so the labels that contain one of
# these names disable MD044 for their line.
[MD044]
names = ["PyAthena", "Athena", "SQLAlchemy", "fsspec", "Polars", "PyArrow", "pandas"]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior)

  • Base: 2a60ecc1f2a7a6bd0d9293525a41ea0d642a0082 (merge-base with master)
  • Head: 6cb01956bb79e540567632677c463c825f151cd7
  • Covered: all 20 changed files — .rumdl.toml (include, enable, MD044), AGENTS.md, the development-workflow skill, scripts/check_license_headers.py and its test, and the 16 Markdown files changed for MD044.

Result: CLEAN

Checks:

  • rumdl check . lints 30 files, including the four .agents/skills files, with no issues; the skills' .claude/skills links are not added to include, so they are not linted twice.
  • MD044 skips inline code and code blocks (checked with a scratch file), so PyAthena[Pandas], engine="pyarrow" and import paths stay as they are; whole-words = true excludes AWS_ATHENA_* in docs/usage.md.
  • The rumdl-disable-next-line MD044 comments cover only the MyST targets (pyathena)=, (polars)= and (sqlalchemy)=. Without them, just docs format (rumdl check --fix) would rewrite the labels and break {ref} targets. A plain sphinx-build of this head and of the base produces the same warnings apart from paths and line numbers, and the anchors in index.html, polars.html and sqlalchemy.html are unchanged.
  • docs/polars.md and docs/sqlalchemy.md are in unheaded-files; a leading HTML comment does not affect scripts/check_license_headers.py (exit 0).
  • Removing the .jsonc entry removes only behavior that has no files; pytest scripts/tests/: 111 passed.
  • benchmarks/RESULTS.md is maintained by hand (no generator writes it), so the PyArrow spelling will not be reverted.

Noted for round two: docs/pandas.md's H1 becomes pandas, which changes the page title in the navigation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to round one (found after merge, db5edc9)

Round one said that, without the rumdl-disable-next-line MD044 comments, just docs format would rewrite the labels and break {ref} targets.
The rewrite does not break the references: Sphinx normalizes label names to lowercase.
Checked with a plain sphinx-build of db5edc9 after removing the comments and changing (sqlalchemy)= to (SQLAlchemy)= and (polars)= to (Polars)=. No new warnings appeared, and {ref}sqlalchemy`` in docs/introduction.md and `docs/pandas.md` still links to `sqlalchemy.html#sqlalchemy`. The anchors stay `#sqlalchemy` and `#polars`.

The disable comments are still needed to keep MD044 from reporting the labels. Without them, the auto-fix makes a label's spelling differ from its {ref} uses.
The cause, rumdl not recognizing MyST targets under the MyST flavor, is reported as rvben/rumdl#983. The comments can be removed once a fixed rumdl is pinned.
The PR description and .rumdl.toml do not contain the incorrect statement.

code-blocks = false
whole-words = true
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ uv run --env-file .env pytest -n 1 tests/pyathena/test_cursor.py -v

### Markdown Lint

`docs/**/*.md`, `benchmarks/*.md`, `scripts/**/*.md`, `cloudformation/**/*.md`, and project-root `*.md` files are linted with [rumdl](https://github.com/rvben/rumdl). The config lives at `.rumdl.toml`. CI runs lint + Sphinx build on PRs that touch docs (`.github/workflows/docs-lint.yaml`).
`docs/**/*.md`, `benchmarks/*.md`, `scripts/**/*.md`, `cloudformation/**/*.md`, `.agents/skills/**/*.md`, and project-root `*.md` files are linted with [rumdl](https://github.com/rvben/rumdl). The config lives at `.rumdl.toml`. CI runs lint + Sphinx build on PRs that touch docs (`.github/workflows/docs-lint.yaml`).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, callers, operational effects)

  • Base: 2a60ecc1f2a7a6bd0d9293525a41ea0d642a0082
  • Head: 6cb01956bb79e540567632677c463c825f151cd7
  • Covered: PR body, the three commit messages, the .rumdl.toml comments, AGENTS.md's Markdown Lint section, the development-workflow skill row, and the issue premises repeated by the change.

Result: FINDINGS (PR description only; no code or docs repair)

Claims checked:

  • "rumdl 0.3.2 applies MD044 to MyST target lines": reproduced. With [per-file-flavor] docs/**/*.md = "myst", (pyathena)= and (sqlalchemy)= were reported with the four-name list.
  • "whole-words = true keeps MD044 from matching AWS_ATHENA_S3_STAGING_DIR": without it, docs/usage.md:796/799 were reported; with it, they were not.
  • "Inline code and code blocks are exempt": a scratch file reported only the prose polars. The fenced import polars as pl examples in the docs produce no reports.
  • "With only the first four names, MD044 reports nothing but the MyST targets": two reports, both labels.
  • "just docs build builds committed refs only": its sources were read from a /tmp/.../<commit sha>/ checkout, so the plain sphinx-build comparison was used.
  • AGENTS.md's file list now matches .rumdl.toml's include. No other doc describes the lint file set or the removed rumdl check .agents/skills step.
  • Documentation reader: the remaining lowercase pyarrow/polars in the docs are code, URLs or import statements. Link URLs and {ref} targets are unchanged, and heading_anchors = 0, so no heading-slug anchor depends on the # pandas text.
  • Contributor/operator: just docs format (rumdl check --fix) now rewrites the listed names in new prose. The disable comments keep it from rewriting the three labels.

Corrections made to the PR body:

  1. It now states that docs/pandas.md's H1 change renames the page in the navigation.
  2. It now marks as not verified the issue's premise that the workflow's **.md filter triggers on a skills-only PR. Add shared development and review skills for Codex and Claude Code #778, the only earlier skills-only change, also touched AGENTS.md, and this PR touches root files, so its own run cannot show this either.


`rumdl` is pinned in `.mise.toml`, so [`mise`](https://mise.jdx.dev/) installs the exact version used in CI. Run locally:

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Extra packages:
|---------------|-----------------------------------------|----------|
| SQLAlchemy | `pip install PyAthena[SQLAlchemy]` | >=2.0.0 |
| AioSQLAlchemy | `pip install PyAthena[AioSQLAlchemy]` | >=2.0.0 |
| Pandas | `pip install PyAthena[Pandas]` | >=3.0.0 |
| pandas | `pip install PyAthena[Pandas]` | >=3.0.0 |
| Arrow | `pip install PyAthena[Arrow]` | >=22.0.0 |
| Polars | `pip install PyAthena[Polars]` | >=1.39.0 |

Expand Down
4 changes: 2 additions & 2 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,8 +352,8 @@ When the temporary directory is a tmpfs, as `/tmp` is on Amazon Linux 2023, file
| Capability | Treatment |
| --- | --- |
| Cursor, DictCursor, S3FS | Row batches with arraysize 100 and 1000 |
| Pandas CSV | Full DataFrame or streamed chunks |
| Pandas UNLOAD | Full DataFrame or chunks, each also with `dtype_backend="pyarrow"` |
| pandas CSV | Full DataFrame or streamed chunks |
| pandas UNLOAD | Full DataFrame or chunks, each also with `dtype_backend="pyarrow"` |
| Arrow CSV and UNLOAD | Eager table loading; arraysize changes row extraction, not S3 streaming |
| Polars CSV and UNLOAD | Full result or chunk iterator |
| SQLAlchemy (`sqla_*`) | `pandas.read_sql_query()` through an engine of each PyAthena driver, whole or with `chunksize` |
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8g

| Need | Use | Measured |
| --- | --- | --- |
| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.2 to 19.8 seconds through UNLOAD, against 55.2 to 84.2 seconds through CSV; nested rows in 33.8 to 36.4 seconds with ArrowCursor, PolarsCursor, or PandasCursor with pyarrow dtypes, against 211.3 to 280.3 seconds through CSV |
| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.2 to 19.8 seconds through UNLOAD, against 55.2 to 84.2 seconds through CSV; nested rows in 33.8 to 36.4 seconds with ArrowCursor, PolarsCursor, or PandasCursor with PyArrow dtypes, against 211.3 to 280.3 seconds through CSV |
| A pandas DataFrame with nested columns | PandasCursor with `unload=True` and `dtype_backend="pyarrow"` | 10 million nested rows in 34.2 seconds and 5,820 MiB, against 88.0 seconds and 19,498 MiB with the default dtypes |
| Bounded memory, including results larger than memory | `chunksize` with PandasCursor or PolarsCursor, with CSV or UNLOAD results | 100 million flat rows, about 29 GB of CSV, at 462 to 1,051 MiB with 100,000-row chunks; 10 million rows at 362 to 917 MiB with 10,000-row chunks, against 3,765 to 19,498 MiB without chunks; larger chunks use more memory |
| A whole CSV result in PolarsCursor | Polars 2.0, or `chunksize`; with Polars 1.x, the read holds a copy of the result file (#1307) | 10 million flat rows peaked at 7,547 MiB with Polars 2.0.0 and 9,641 MiB with Polars 1.44.2 |
Expand Down
4 changes: 2 additions & 2 deletions docs/aio.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,8 @@ Native asyncio versions are available for all cursor types:
| Cursor | Module | Result format |
|--------|--------|---------------|
| {ref}`AioPandasCursor <aio-pandas-cursor>` | `pyathena.aio.pandas.cursor` | pandas DataFrame |
| {ref}`AioArrowCursor <aio-arrow-cursor>` | `pyathena.aio.arrow.cursor` | pyarrow Table |
| {ref}`AioPolarsCursor <aio-polars-cursor>` | `pyathena.aio.polars.cursor` | polars DataFrame |
| {ref}`AioArrowCursor <aio-arrow-cursor>` | `pyathena.aio.arrow.cursor` | PyArrow Table |
| {ref}`AioPolarsCursor <aio-polars-cursor>` | `pyathena.aio.polars.cursor` | Polars DataFrame |
| {ref}`AioS3FSCursor <aio-s3fs-cursor>` | `pyathena.aio.s3fs.cursor` | Row tuples (lightweight) |
| {ref}`AioSparkCursor <aio-spark-cursor>` | `pyathena.aio.spark.cursor` | PySpark execution |

Expand Down
4 changes: 2 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ api/aio
- {ref}`api_sqlalchemy` - SQLAlchemy dialect implementations
- {ref}`api_pandas` - pandas DataFrame integration
- {ref}`api_arrow` - Apache Arrow columnar data integration
- {ref}`api_polars` - Polars DataFrame integration (no pyarrow required)
- {ref}`api_s3fs` - Lightweight S3FS-based cursor (no pandas/pyarrow required)
- {ref}`api_polars` - Polars DataFrame integration (no PyArrow required)
- {ref}`api_s3fs` - Lightweight S3FS-based cursor (no pandas/PyArrow required)
- {ref}`api_spark` - Apache Spark integration for big data processing
- {ref}`api_aio` - Native asyncio cursor implementations
14 changes: 7 additions & 7 deletions docs/arrow.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
## ArrowCursor

ArrowCursor directly handles the CSV file of the query execution result output to S3.
This cursor is to download the CSV file after executing the query, and then loaded into [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).
This cursor is to download the CSV file after executing the query, and then loaded into [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).
Performance is better than fetching data with Cursor.

You can use the ArrowCursor by specifying the `cursor_class`
Expand Down Expand Up @@ -49,7 +49,7 @@ cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
region_name="us-west-2").cursor(ArrowCursor)
```

The as_arrow method returns a [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).
The as_arrow method returns a [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).

```python
from pyathena import connect
Expand Down Expand Up @@ -123,7 +123,7 @@ print(cursor.service_processing_time_in_millis)
print(cursor.output_location)
```

If you want to customize the [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) types, create a converter class like this:
If you want to customize the [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) types, create a converter class like this:

```python
import pyarrow as pa
Expand Down Expand Up @@ -290,16 +290,16 @@ cursor = connect(
).cursor(ArrowCursor, connect_timeout=10.0, request_timeout=30.0)
```

The timeout parameters accept float values in seconds and apply to the pyarrow S3 filesystem that reads the result CSV and Parquet files.
The pyarrow S3 filesystem uses the credentials of the connection's boto3 session, including the credentials of an assumed role.
The timeout parameters accept float values in seconds and apply to the PyArrow S3 filesystem that reads the result CSV and Parquet files.
The PyArrow S3 filesystem uses the credentials of the connection's boto3 session, including the credentials of an assumed role.
The other S3 requests, the HeadObject request for the result file size and the GetObject request for the UNLOAD data manifest,
use the connection's boto3 client and its botocore configuration.

(async-arrow-cursor)=

## AsyncArrowCursor

AsyncArrowCursor is an AsyncCursor that can handle [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).
AsyncArrowCursor is an AsyncCursor that can handle [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html).
This cursor directly handles the CSV of query results output to S3 in the same way as ArrowCursor.

You can use the AsyncArrowCursor by specifying the `cursor_class`
Expand Down Expand Up @@ -408,7 +408,7 @@ result_set = future.result()
print(result_set.fetchall())
```

This object also has an as_arrow method that returns a [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) similar to the ArrowCursor.
This object also has an as_arrow method that returns a [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) similar to the ArrowCursor.

```python
from pyathena import connect
Expand Down
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ See LICENSE or https://opensource.org/licenses/MIT.
SPDX-License-Identifier: MIT
-->

<!-- rumdl-disable-next-line MD044 -->
(pyathena)=

![PyAthena](_static/header.png)
Expand Down
4 changes: 2 additions & 2 deletions docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Extra packages:
|---------------|-----------------------------------------|----------|
| SQLAlchemy | `pip install PyAthena[SQLAlchemy]` | >=2.0.0 |
| AioSQLAlchemy | `pip install PyAthena[AioSQLAlchemy]` | >=2.0.0 |
| Pandas | `pip install PyAthena[Pandas]` | >=3.0.0 |
| pandas | `pip install PyAthena[Pandas]` | >=3.0.0 |
| Arrow | `pip install PyAthena[Arrow]` | >=22.0.0 |
| Polars | `pip install PyAthena[Polars]` | >=1.39.0 |

Expand All @@ -47,7 +47,7 @@ PyAthena provides comprehensive support for Amazon Athena's data types and featu

- **DB API 2.0 Compliance**: Full PEP 249 compatibility for database operations
- **SQLAlchemy Integration**: Native dialect support with table reflection and ORM capabilities
- **Multiple Cursor Types**: Standard, Pandas, Arrow, Polars, S3FS and Spark cursor implementations
- **Multiple Cursor Types**: Standard, pandas, Arrow, Polars, S3FS and Spark cursor implementations
- **Async Support**: Asynchronous query execution for non-blocking operations

**Data Type Support:**
Expand Down
4 changes: 2 additions & 2 deletions docs/null_handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ id,name,description

Most CSV parsers handle these differently:

- **Pandas** (used by `PandasCursor`): By default, treats both unquoted empty and quoted empty as missing values
- **pandas** (used by `PandasCursor`): By default, treats both unquoted empty and quoted empty as missing values
- **PyArrow** (used by `ArrowCursor`): With `quoted_strings_can_be_null=False`, treats both as empty strings
- **Polars** (used by `PolarsCursor`): Correctly distinguishes quoted empty strings from unquoted NULL values
- **AthenaCSVReader** (used by `S3FSCursor`): Correctly distinguishes NULL (returns `None`) from empty strings
Expand Down Expand Up @@ -76,7 +76,7 @@ With the default CSV settings and converters, pandas and Arrow cursors distingui
values when reading CSV results: `fetchone()`, `fetchmany()`, and `fetchall()` return `None`
for NULL and `b''` for an empty binary value.
This also applies to their asynchronous variants and pandas chunked reads.
Pandas DataFrames preserve the same values.
pandas DataFrames preserve the same values.
Arrow Tables retain the CSV hexadecimal strings, with NULL represented as an Arrow null;
fetch methods convert the hexadecimal strings to Python bytes.

Expand Down
Loading
Loading