From f1367e9361de6f58a2359d0e0cd4c98fd3744695 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Sat, 7 Nov 2026 13:57:22 +0900 Subject: [PATCH 1/3] Lint the agent skills with the rest of the Markdown (GH-1368) `just docs lint` and the Docs Lint workflow skipped `.agents/skills`, so the separate `rumdl check .agents/skills` step in the development workflow was not enforced. Include the skills in `.rumdl.toml` and drop the separate step. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/development-workflow/SKILL.md | 2 +- .rumdl.toml | 1 + AGENTS.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.agents/skills/development-workflow/SKILL.md b/.agents/skills/development-workflow/SKILL.md index 43cd27056..245c16e67 100644 --- a/.agents/skills/development-workflow/SKILL.md +++ b/.agents/skills/development-workflow/SKILL.md @@ -51,7 +51,7 @@ Prefer existing fixtures and integration classes; do not build a parallel mock f | SQLAlchemy dialect | Relevant PyAthena tests in `tests/pyathena/sqlalchemy/` plus `just test sqla`; also `tests/pyathena/aio/sqlalchemy/` and `just test sqla-async` for shared reflection or async paths. Both PyAthena directories are included in `just test pyathena`. | | Filesystem | Affected S3 filesystem tests, including listing and path boundaries | | Documentation | `just docs lint`; `just docs build` for rendered documentation changes | -| Agent skills | `just docs lint` and `mise exec -- rumdl check .agents/skills`; inspect links and walk through realistic workflow scenarios | +| Agent skills | `just docs lint`; inspect links and walk through realistic workflow scenarios | | Dependency, packaging, or CI configuration | Affected build/install checks and supported Python/dependency combinations from `pyproject.toml`, `uv.lock`, and CI | The `tests/pyathena/` directory includes AWS integration tests despite the recipe's "unit tests" label. diff --git a/.rumdl.toml b/.rumdl.toml index 7b3e8f823..8f58daffb 100644 --- a/.rumdl.toml +++ b/.rumdl.toml @@ -16,6 +16,7 @@ include = [ "scripts/**/*.md", "cloudformation/**/*.md", "/*.md", + ".agents/skills/**/*.md", ] # Run the rule set of markdownlint 0.38 (used by markdownlint-cli2 0.18.1, # the previous linter) instead of every rule rumdl enables by default. diff --git a/AGENTS.md b/AGENTS.md index 20a8879fd..1b9605742 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,7 +102,7 @@ uv run --env-file .env pytest -n 1 tests/pyathena/test_cursor.py -v ### Markdown Lint -`docs/**/*.md`, `benchmarks/*.md`, `scripts/**/*.md`, `cloudformation/**/*.md`, and project-root `*.md` files are linted with [rumdl](https://github.com/rvben/rumdl). The config lives at `.rumdl.toml`. CI runs lint + Sphinx build on PRs that touch docs (`.github/workflows/docs-lint.yaml`). +`docs/**/*.md`, `benchmarks/*.md`, `scripts/**/*.md`, `cloudformation/**/*.md`, `.agents/skills/**/*.md`, and project-root `*.md` files are linted with [rumdl](https://github.com/rvben/rumdl). The config lives at `.rumdl.toml`. CI runs lint + Sphinx build on PRs that touch docs (`.github/workflows/docs-lint.yaml`). `rumdl` is pinned in `.mise.toml`, so [`mise`](https://mise.jdx.dev/) installs the exact version used in CI. Run locally: From 7e65d1e304a83310420067c8faed4b0efdbef544 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Sat, 7 Nov 2026 13:57:22 +0900 Subject: [PATCH 2/3] Remove the license header entry for .jsonc files (GH-1368) Its only file, `.markdownlint-cli2.jsonc`, was removed with markdownlint-cli2, and no tracked `.jsonc` file remains. Co-Authored-By: Claude Opus 5.5 --- scripts/check_license_headers.py | 1 - scripts/tests/test_check_license_headers.py | 1 - 2 files changed, 2 deletions(-) diff --git a/scripts/check_license_headers.py b/scripts/check_license_headers.py index 00e2d8bac..b64f162c2 100644 --- a/scripts/check_license_headers.py +++ b/scripts/check_license_headers.py @@ -52,7 +52,6 @@ def _block(opening: str | None, prefix: str, closing: str | None) -> re.Pattern[ ".css": (_block("/*", " * ", " */"),), ".html": (JINJA, HTML), ".jinja2": (JINJA,), - ".jsonc": (_block(None, "// ", None),), ".md": (HTML,), ".rst": (_block("..", " ", None),), } diff --git a/scripts/tests/test_check_license_headers.py b/scripts/tests/test_check_license_headers.py index 4bcdcda8e..4679f6ad7 100644 --- a/scripts/tests/test_check_license_headers.py +++ b/scripts/tests/test_check_license_headers.py @@ -65,7 +65,6 @@ class TestHasLicenseHeader: (".html", HTML + "

\n"), (".md", "---\n---\n" + HTML), (".css", "/*\n" + prefixed(" * ") + " */\n"), - (".jsonc", prefixed("// ") + "\n{}\n"), ], ) def test_accepted(self, suffix, text): From 6cb01956bb79e540567632677c463c825f151cd7 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Sat, 7 Nov 2026 13:57:23 +0900 Subject: [PATCH 3/3] Configure MD044 proper names and drop MD043 (GH-1368) Both rules were enabled without `names` or `headings`, so neither reported anything. MD044 now checks the spelling of PyAthena, Athena, SQLAlchemy, fsspec, Polars, PyArrow and pandas outside code, and the docs follow it: library names in prose are capitalized as their projects write them, and module paths, driver names and option values are code. rumdl also checks MyST `(label)=` targets, so the three labels with a lowercase name disable MD044 for their line. MD043 is dropped because the docs share no single heading structure. Co-Authored-By: Claude Opus 5.5 --- .rumdl.toml | 11 ++++++++++- README.md | 2 +- benchmarks/README.md | 4 ++-- benchmarks/RESULTS.md | 2 +- docs/aio.md | 4 ++-- docs/api.md | 4 ++-- docs/arrow.md | 14 +++++++------- docs/index.md | 1 + docs/introduction.md | 4 ++-- docs/null_handling.md | 4 ++-- docs/pandas.md | 20 ++++++++++---------- docs/polars.md | 35 ++++++++++++++++++----------------- docs/s3fs.md | 8 ++++---- docs/sqlalchemy.md | 37 +++++++++++++++++++------------------ docs/temporal_types.md | 2 +- docs/usage.md | 4 ++-- 16 files changed, 84 insertions(+), 72 deletions(-) diff --git a/.rumdl.toml b/.rumdl.toml index 8f58daffb..7bed8a8b8 100644 --- a/.rumdl.toml +++ b/.rumdl.toml @@ -25,7 +25,7 @@ enable = [ "MD012", "MD018", "MD019", "MD020", "MD021", "MD022", "MD023", "MD024", "MD025", "MD026", "MD027", "MD028", "MD029", "MD030", "MD031", "MD032", "MD033", "MD034", "MD035", "MD036", "MD037", "MD038", "MD039", "MD040", - "MD042", "MD043", "MD044", "MD045", "MD046", "MD047", "MD048", "MD049", + "MD042", "MD044", "MD045", "MD046", "MD047", "MD048", "MD049", "MD050", "MD051", "MD052", "MD053", "MD054", "MD055", "MD056", "MD058", "MD059", # Not enabled: @@ -33,6 +33,7 @@ enable = [ # MD014: `$ command` style is intentional in docs/testing.md and README shell snippets # MD041: MyST `(label)=` ref targets must come before the first heading, so the # first line of many docs/*.md files is not an H1 + # MD043: the docs share no single heading structure to require ] # Sphinx renders docs/ with MyST, so directive fences such as ```{toctree} @@ -81,3 +82,11 @@ ignore = ["x"] [MD053] ignored-definitions = ["//"] + +# Spell project and library names consistently in prose; code is exempt. +# rumdl also checks MyST `(label)=` targets, so the labels that contain one of +# these names disable MD044 for their line. +[MD044] +names = ["PyAthena", "Athena", "SQLAlchemy", "fsspec", "Polars", "PyArrow", "pandas"] +code-blocks = false +whole-words = true diff --git a/README.md b/README.md index 466cda0bd..64a388995 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Extra packages: |---------------|-----------------------------------------|----------| | SQLAlchemy | `pip install PyAthena[SQLAlchemy]` | >=2.0.0 | | AioSQLAlchemy | `pip install PyAthena[AioSQLAlchemy]` | >=2.0.0 | -| Pandas | `pip install PyAthena[Pandas]` | >=3.0.0 | +| pandas | `pip install PyAthena[Pandas]` | >=3.0.0 | | Arrow | `pip install PyAthena[Arrow]` | >=22.0.0 | | Polars | `pip install PyAthena[Polars]` | >=1.39.0 | diff --git a/benchmarks/README.md b/benchmarks/README.md index 4edc602eb..8137046cd 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -352,8 +352,8 @@ When the temporary directory is a tmpfs, as `/tmp` is on Amazon Linux 2023, file | Capability | Treatment | | --- | --- | | Cursor, DictCursor, S3FS | Row batches with arraysize 100 and 1000 | -| Pandas CSV | Full DataFrame or streamed chunks | -| Pandas UNLOAD | Full DataFrame or chunks, each also with `dtype_backend="pyarrow"` | +| pandas CSV | Full DataFrame or streamed chunks | +| pandas UNLOAD | Full DataFrame or chunks, each also with `dtype_backend="pyarrow"` | | Arrow CSV and UNLOAD | Eager table loading; arraysize changes row extraction, not S3 streaming | | Polars CSV and UNLOAD | Full result or chunk iterator | | SQLAlchemy (`sqla_*`) | `pandas.read_sql_query()` through an engine of each PyAthena driver, whole or with `chunksize` | diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index 597051bf8..2dd779018 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -28,7 +28,7 @@ These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8g | Need | Use | Measured | | --- | --- | --- | -| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.2 to 19.8 seconds through UNLOAD, against 55.2 to 84.2 seconds through CSV; nested rows in 33.8 to 36.4 seconds with ArrowCursor, PolarsCursor, or PandasCursor with pyarrow dtypes, against 211.3 to 280.3 seconds through CSV | +| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.2 to 19.8 seconds through UNLOAD, against 55.2 to 84.2 seconds through CSV; nested rows in 33.8 to 36.4 seconds with ArrowCursor, PolarsCursor, or PandasCursor with PyArrow dtypes, against 211.3 to 280.3 seconds through CSV | | A pandas DataFrame with nested columns | PandasCursor with `unload=True` and `dtype_backend="pyarrow"` | 10 million nested rows in 34.2 seconds and 5,820 MiB, against 88.0 seconds and 19,498 MiB with the default dtypes | | Bounded memory, including results larger than memory | `chunksize` with PandasCursor or PolarsCursor, with CSV or UNLOAD results | 100 million flat rows, about 29 GB of CSV, at 462 to 1,051 MiB with 100,000-row chunks; 10 million rows at 362 to 917 MiB with 10,000-row chunks, against 3,765 to 19,498 MiB without chunks; larger chunks use more memory | | A whole CSV result in PolarsCursor | Polars 2.0, or `chunksize`; with Polars 1.x, the read holds a copy of the result file (#1307) | 10 million flat rows peaked at 7,547 MiB with Polars 2.0.0 and 9,641 MiB with Polars 1.44.2 | diff --git a/docs/aio.md b/docs/aio.md index 5bf72b50b..b72e38d42 100644 --- a/docs/aio.md +++ b/docs/aio.md @@ -217,8 +217,8 @@ Native asyncio versions are available for all cursor types: | Cursor | Module | Result format | |--------|--------|---------------| | {ref}`AioPandasCursor ` | `pyathena.aio.pandas.cursor` | pandas DataFrame | -| {ref}`AioArrowCursor ` | `pyathena.aio.arrow.cursor` | pyarrow Table | -| {ref}`AioPolarsCursor ` | `pyathena.aio.polars.cursor` | polars DataFrame | +| {ref}`AioArrowCursor ` | `pyathena.aio.arrow.cursor` | PyArrow Table | +| {ref}`AioPolarsCursor ` | `pyathena.aio.polars.cursor` | Polars DataFrame | | {ref}`AioS3FSCursor ` | `pyathena.aio.s3fs.cursor` | Row tuples (lightweight) | | {ref}`AioSparkCursor ` | `pyathena.aio.spark.cursor` | PySpark execution | diff --git a/docs/api.md b/docs/api.md index fd58cccb4..30064b4b6 100644 --- a/docs/api.md +++ b/docs/api.md @@ -51,7 +51,7 @@ api/aio - {ref}`api_sqlalchemy` - SQLAlchemy dialect implementations - {ref}`api_pandas` - pandas DataFrame integration - {ref}`api_arrow` - Apache Arrow columnar data integration -- {ref}`api_polars` - Polars DataFrame integration (no pyarrow required) -- {ref}`api_s3fs` - Lightweight S3FS-based cursor (no pandas/pyarrow required) +- {ref}`api_polars` - Polars DataFrame integration (no PyArrow required) +- {ref}`api_s3fs` - Lightweight S3FS-based cursor (no pandas/PyArrow required) - {ref}`api_spark` - Apache Spark integration for big data processing - {ref}`api_aio` - Native asyncio cursor implementations diff --git a/docs/arrow.md b/docs/arrow.md index 6262106bc..5d0941ae0 100644 --- a/docs/arrow.md +++ b/docs/arrow.md @@ -7,7 +7,7 @@ ## ArrowCursor ArrowCursor directly handles the CSV file of the query execution result output to S3. -This cursor is to download the CSV file after executing the query, and then loaded into [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). +This cursor is to download the CSV file after executing the query, and then loaded into [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). Performance is better than fetching data with Cursor. You can use the ArrowCursor by specifying the `cursor_class` @@ -49,7 +49,7 @@ cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(ArrowCursor) ``` -The as_arrow method returns a [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). +The as_arrow method returns a [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). ```python from pyathena import connect @@ -123,7 +123,7 @@ print(cursor.service_processing_time_in_millis) print(cursor.output_location) ``` -If you want to customize the [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) types, create a converter class like this: +If you want to customize the [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) types, create a converter class like this: ```python import pyarrow as pa @@ -290,8 +290,8 @@ cursor = connect( ).cursor(ArrowCursor, connect_timeout=10.0, request_timeout=30.0) ``` -The timeout parameters accept float values in seconds and apply to the pyarrow S3 filesystem that reads the result CSV and Parquet files. -The pyarrow S3 filesystem uses the credentials of the connection's boto3 session, including the credentials of an assumed role. +The timeout parameters accept float values in seconds and apply to the PyArrow S3 filesystem that reads the result CSV and Parquet files. +The PyArrow S3 filesystem uses the credentials of the connection's boto3 session, including the credentials of an assumed role. The other S3 requests, the HeadObject request for the result file size and the GetObject request for the UNLOAD data manifest, use the connection's boto3 client and its botocore configuration. @@ -299,7 +299,7 @@ use the connection's boto3 client and its botocore configuration. ## AsyncArrowCursor -AsyncArrowCursor is an AsyncCursor that can handle [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). +AsyncArrowCursor is an AsyncCursor that can handle [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). This cursor directly handles the CSV of query results output to S3 in the same way as ArrowCursor. You can use the AsyncArrowCursor by specifying the `cursor_class` @@ -408,7 +408,7 @@ result_set = future.result() print(result_set.fetchall()) ``` -This object also has an as_arrow method that returns a [pyarrow.Table object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) similar to the ArrowCursor. +This object also has an as_arrow method that returns a [`pyarrow.Table` object](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) similar to the ArrowCursor. ```python from pyathena import connect diff --git a/docs/index.md b/docs/index.md index a893a6ad4..46bb58632 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,6 +7,7 @@ See LICENSE or https://opensource.org/licenses/MIT. SPDX-License-Identifier: MIT --> + (pyathena)= ![PyAthena](_static/header.png) diff --git a/docs/introduction.md b/docs/introduction.md index 448206257..f27643d11 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -33,7 +33,7 @@ Extra packages: |---------------|-----------------------------------------|----------| | SQLAlchemy | `pip install PyAthena[SQLAlchemy]` | >=2.0.0 | | AioSQLAlchemy | `pip install PyAthena[AioSQLAlchemy]` | >=2.0.0 | -| Pandas | `pip install PyAthena[Pandas]` | >=3.0.0 | +| pandas | `pip install PyAthena[Pandas]` | >=3.0.0 | | Arrow | `pip install PyAthena[Arrow]` | >=22.0.0 | | Polars | `pip install PyAthena[Polars]` | >=1.39.0 | @@ -47,7 +47,7 @@ PyAthena provides comprehensive support for Amazon Athena's data types and featu - **DB API 2.0 Compliance**: Full PEP 249 compatibility for database operations - **SQLAlchemy Integration**: Native dialect support with table reflection and ORM capabilities -- **Multiple Cursor Types**: Standard, Pandas, Arrow, Polars, S3FS and Spark cursor implementations +- **Multiple Cursor Types**: Standard, pandas, Arrow, Polars, S3FS and Spark cursor implementations - **Async Support**: Asynchronous query execution for non-blocking operations **Data Type Support:** diff --git a/docs/null_handling.md b/docs/null_handling.md index e71a5a30b..6f3c9ef3b 100644 --- a/docs/null_handling.md +++ b/docs/null_handling.md @@ -38,7 +38,7 @@ id,name,description Most CSV parsers handle these differently: -- **Pandas** (used by `PandasCursor`): By default, treats both unquoted empty and quoted empty as missing values +- **pandas** (used by `PandasCursor`): By default, treats both unquoted empty and quoted empty as missing values - **PyArrow** (used by `ArrowCursor`): With `quoted_strings_can_be_null=False`, treats both as empty strings - **Polars** (used by `PolarsCursor`): Correctly distinguishes quoted empty strings from unquoted NULL values - **AthenaCSVReader** (used by `S3FSCursor`): Correctly distinguishes NULL (returns `None`) from empty strings @@ -76,7 +76,7 @@ With the default CSV settings and converters, pandas and Arrow cursors distingui values when reading CSV results: `fetchone()`, `fetchmany()`, and `fetchall()` return `None` for NULL and `b''` for an empty binary value. This also applies to their asynchronous variants and pandas chunked reads. -Pandas DataFrames preserve the same values. +pandas DataFrames preserve the same values. Arrow Tables retain the CSV hexadecimal strings, with NULL represented as an Arrow null; fetch methods convert the hexadecimal strings to Python bytes. diff --git a/docs/pandas.md b/docs/pandas.md index 1ecaecb7e..e388721c9 100644 --- a/docs/pandas.md +++ b/docs/pandas.md @@ -1,6 +1,6 @@ (pandas)= -# Pandas +# pandas (as-dataframe)= @@ -109,11 +109,11 @@ cursor.execute("SHOW PARTITIONS YOUR_TABLE") print(cursor.fetchall()) ``` -`to_sql` writes the data as Parquet with pyarrow, so it requires `pip install PyAthena[Pandas,Arrow]`. +`to_sql` writes the data as Parquet with PyArrow, so it requires `pip install PyAthena[Pandas,Arrow]`. With the default `flavor="spark"`, timestamps are written as INT96. Athena reads them correctly. -pyarrow, of any version, reads the timestamps before 1970 in files that pyarrow 22 wrote as wrong values. -pyarrow 23.0.0 and later write them correctly and are recommended. +PyArrow, of any version, reads the timestamps before 1970 in files that PyArrow 22 wrote as wrong values. +PyArrow 23.0.0 and later write them correctly and are recommended. The Spark flavor also replaces spaces, tabs, newlines, and the characters `,;{}()=` in column names with `_`, and the table declares the replaced names. The partition keys keep their names. Conversion to Parquet and upload to S3 use [ThreadPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) by default. @@ -344,7 +344,7 @@ NOTE: PandasCursor handles the CSV file on memory. Pay attention to the memory c ### Chunksize options -The Pandas cursor can read the CSV file for each specified number of rows by using the chunksize option. +The pandas cursor can read the CSV file for each specified number of rows by using the chunksize option. This option should reduce memory usage. With the unload option, the cursor reads the Parquet files of the result in chunks of the specified number of rows, unless `execute()` is given `pandas.read_parquet()` options other than `columns` and `dtype_backend`, in which case the whole result is read. @@ -566,7 +566,7 @@ cursor.execute("SELECT * FROM typed_table", Common performance options: -- `engine`: CSV parsing engine ('auto', 'c', 'python', 'pyarrow'); 'auto' uses the C engine +- `engine`: CSV parsing engine (`'auto'`, `'c'`, `'python'`, `'pyarrow'`); `'auto'` uses the C engine - `low_memory`: Parse the file in internal chunks to reduce memory use (C engine only) - `dtype`: Explicit column data types - `parse_dates`: Columns to parse as dates @@ -575,11 +575,11 @@ PandasCursor reads a `json` column as text and applies the converter mapping's ` A `json` column with NULL in the result or chunk holds objects, with `None` for NULL; otherwise it has the dtype that pandas infers from the converted values. Converters passed to `execute()` replace the converter mapping and are applied by pandas. -With `engine="pyarrow"`, PandasCursor uses the PyArrow engine only when the result is not a tab-separated `.txt` file, pyarrow is installed, no chunksize is set (explicitly or by `auto_optimize_chunksize`), `quoting` is the default, the result has no columns that need a converter (`boolean`, `decimal`, `varbinary`, `json`, `time with time zone`, columns of a type that the converter maps to a conversion function, and `timestamp with time zone` columns that PyAthena does not convert after reading), no `converters` are given to `execute()`, and the result file is at least `AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES` bytes. +With `engine="pyarrow"`, PandasCursor uses the PyArrow engine only when the result is not a tab-separated `.txt` file, PyArrow is installed, no chunksize is set (explicitly or by `auto_optimize_chunksize`), `quoting` is the default, the result has no columns that need a converter (`boolean`, `decimal`, `varbinary`, `json`, `time with time zone`, columns of a type that the converter maps to a conversion function, and `timestamp with time zone` columns that PyAthena does not convert after reading), no `converters` are given to `execute()`, and the result file is at least `AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES` bytes. Otherwise, it falls back to the C engine. With `engine="pyarrow"`, tab-separated `.txt` results from DDL statements such as `SHOW TABLES`, `SHOW COLUMNS`, and `DESCRIBE` use the C engine to preserve leading zeros, exponent notation, and padding in string values. With the PyArrow engine, when `keep_default_na` and `na_values` are the defaults and the only pandas.read_csv() options passed to `execute()` are `dtype` as a mapping and `parse_dates` as a list, PyAthena reads the file with `pyarrow.csv`, allowing newlines in quoted values, and converts it to a DataFrame as `pandas.read_csv(engine="pyarrow")` does, except that columns with a string dtype are read as text, and without `future.infer_string`, a column with dtype `str` keeps NULL as a missing value. -Otherwise, pandas reads the file, and its PyArrow engine raises an error or returns wrong values when a quoted value containing a newline crosses one of pyarrow's read blocks. +Otherwise, pandas reads the file, and its PyArrow engine raises an error or returns wrong values when a quoted value containing a newline crosses one of PyArrow's read blocks. It also changes the values of a column with a string dtype whose values all look like numbers, such as `"007"` to `"7.0"`, and without `future.infer_string`, turns NULL in a column with dtype `str` into the string `"nan"`. Apart from PandasCursor's own options such as `engine` and `chunksize`, an option passed here replaces the value PyAthena sets for the same pandas.read_csv() argument. @@ -607,13 +607,13 @@ df = cursor.execute("SELECT * FROM many_rows").as_pandas() Without `block_size` or `cache_type`, a CSV result file is read in 32 MiB blocks with the `"background"` cache, which reads the next block while pandas parses the current one and keeps two blocks. With either option, the file is read with the given values and the filesystem's defaults for the other, `"bytes"` and 5 MiB blocks. -When an UNLOAD result is read in chunks of `chunksize` rows, a Parquet file of up to 32 MiB is read at once, and a larger one is read with the given `cache_type`, or without it with the `"none"` cache, which requests only the ranges that pyarrow reads. +When an UNLOAD result is read in chunks of `chunksize` rows, a Parquet file of up to 32 MiB is read at once, and a larger one is read with the given `cache_type`, or without it with the `"none"` cache, which requests only the ranges that PyArrow reads. With `storage_options` given to `execute()`, or `filesystem` for an UNLOAD result, pandas reads the result files with those options instead of these. ### Unload options PandasCursor also supports the unload option, as does {ref}`arrow-cursor`. -Reading the unloaded Parquet files requires pyarrow (`pip install PyAthena[Pandas,Arrow]`), and `engine` must be `auto` or `pyarrow`. +Reading the unloaded Parquet files requires PyArrow (`pip install PyAthena[Pandas,Arrow]`), and `engine` must be `auto` or `pyarrow`. Timestamps are read at microseconds, as `datetime64[us]`, and dates as `datetime64[us]`, as from CSV results. With `coerce_int96_timestamp_unit` or `dtype_backend` given to `execute()`, the columns have the dtypes that these options give, and the fetch methods return their values as pandas holds them. `coerce_int96_timestamp_unit=None` is the same as no `coerce_int96_timestamp_unit`. diff --git a/docs/polars.md b/docs/polars.md index 6ae9c779e..9695578eb 100644 --- a/docs/polars.md +++ b/docs/polars.md @@ -1,3 +1,4 @@ + (polars)= # Polars @@ -7,12 +8,12 @@ ## PolarsCursor PolarsCursor directly handles the CSV file of the query execution result output to S3. -This cursor downloads the CSV file after executing the query and loads it into a [polars.DataFrame object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). +This cursor downloads the CSV file after executing the query and loads it into a [`polars.DataFrame` object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). Performance is better than fetching data with Cursor. PolarsCursor uses [Polars](https://pola.rs/) native reading capabilities (`pl.read_csv`, `pl.read_parquet`) and does not require PyArrow as a dependency; for UNLOAD results with timestamps read with Polars 1.x, see {ref}`polars-unload-options`. -CSV results read without the chunksize option, and UNLOAD results read with it when pyarrow is +CSV results read without the chunksize option, and UNLOAD results read with it when PyArrow is installed and `execute()` is given no Polars read options, use PyAthena's own S3FileSystem (fsspec compatible), and other reads use Polars' native S3 access, so s3fs is also not required. @@ -20,8 +21,8 @@ Options passed to `execute()`, other than PolarsCursor's own options such as `ch `schema=None` is the same as no `schema`. Column types for a CSV result are given as `schema_overrides`, which replaces PyAthena's column types. Polars' deprecated alias `dtypes` is not supported: a CSV result read with it fails with an error. -`pl.read_csv()` reads the CSV result with pyarrow when `use_pyarrow=True` is given with `schema_overrides=None`, which replaces PyAthena's column types, and Polars' other conditions for it hold. -That read raises an error or returns wrong values when a quoted value containing a newline crosses one of pyarrow's read blocks. +`pl.read_csv()` reads the CSV result with PyArrow when `use_pyarrow=True` is given with `schema_overrides=None`, which replaces PyAthena's column types, and Polars' other conditions for it hold. +That read raises an error or returns wrong values when a quoted value containing a newline crosses one of PyArrow's read blocks. The default Polars CSV reader reads such values. You can use the PolarsCursor by specifying the `cursor_class` @@ -63,7 +64,7 @@ cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(PolarsCursor) ``` -The as_polars method returns a [polars.DataFrame object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). +The as_polars method returns a [`polars.DataFrame` object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). ```python from pyathena import connect @@ -153,7 +154,7 @@ print(table.num_columns) print(table.schema) ``` -If you want to customize the polars.DataFrame dtypes, create a converter class like this: +If you want to customize the `polars.DataFrame` dtypes, create a converter class like this: ```python import polars as pl @@ -225,10 +226,10 @@ See {ref}`arrow-unload-options` for more information. Timestamps of UNLOAD results are read at microseconds, as `Datetime("us")`. Polars 2.0 and later read them with Polars. -Polars 1.x reads them at nanoseconds ([pola-rs/polars#29184](https://github.com/pola-rs/polars/issues/29184)), which gives wrong values outside 1677-09-21 to 2262-04-11; they are cast to microseconds, and a warning is logged. -The exception is a read with the chunksize option, with pyarrow installed and without Polars read options given to `execute()`: the chunks are read with pyarrow at microseconds, without the cast or the warning. +Polars 1.x reads them at nanoseconds ([`pola-rs/polars#29184`](https://github.com/pola-rs/polars/issues/29184)), which gives wrong values outside 1677-09-21 to 2262-04-11; they are cast to microseconds, and a warning is logged. +The exception is a read with the chunksize option, with PyArrow installed and without Polars read options given to `execute()`: the chunks are read with PyArrow at microseconds, without the cast or the warning. A `schema` given to `execute()` is passed to Polars as given, and the timestamps that it reads are not converted. -With such a schema, Polars 2.0.0 reads the timestamps in a `Map` dtype at nanoseconds whatever its time unit ([pola-rs/polars#30031](https://github.com/pola-rs/polars/issues/30031)), and those in a map read as a `List` of `Struct`s of its key and value at the schema's time unit. +With such a schema, Polars 2.0.0 reads the timestamps in a `Map` dtype at nanoseconds whatever its time unit ([`pola-rs/polars#30031`](https://github.com/pola-rs/polars/issues/30031)), and those in a map read as a `List` of `Struct`s of its key and value at the schema's time unit. The unload option can be enabled by specifying it in the `cursor_kwargs` argument of the connect method or as an argument to the cursor method. An argument to the cursor method takes precedence over the same key in `cursor_kwargs`. @@ -339,9 +340,9 @@ Polars limits the read-ahead by the number of CPU cores, so memory usage does no If reading a chunk fails, iteration raises `OperationalError` instead of ending early. The chunked iteration also works with the unload option. -With pyarrow installed, the Parquet files of the result are read one at a time through PyAthena's S3 filesystem, -and each chunk is converted from pyarrow, so memory usage depends on the size of each chunk and of the row group being read, not on the size of the result. -With read options given to `execute()`, or without pyarrow, Polars' `scan_parquet()` reads the files with those options; +With PyArrow installed, the Parquet files of the result are read one at a time through PyAthena's S3 filesystem, +and each chunk is converted from PyArrow, so memory usage depends on the size of each chunk and of the row group being read, not on the size of the result. +With read options given to `execute()`, or without PyArrow, Polars' `scan_parquet()` reads the files with those options; it decodes whole row groups and reads ahead, so `chunksize` does not limit memory usage: ```python @@ -372,7 +373,7 @@ df = cursor.execute("SELECT * FROM many_rows").as_polars() # All chunks in a si ``` Apart from returning an empty DataFrame when there are no chunks, this is equivalent to using -[polars.concat](https://docs.pola.rs/api/python/stable/reference/api/polars.concat.html) on the chunks from `iter_chunks()`: +[`polars.concat`](https://docs.pola.rs/api/python/stable/reference/api/polars.concat.html) on the chunks from `iter_chunks()`: ```python import polars as pl @@ -404,8 +405,8 @@ cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", df = cursor.execute("SELECT * FROM many_rows").as_polars() ``` -They apply to the reads through PyAthena's filesystem: CSV results read without the chunksize option, which use `"bytes"` and 5 MiB blocks by default, and the Parquet files of UNLOAD results read with the chunksize option when pyarrow is installed and `execute()` is given no Polars read options. -Of the latter, a file of up to 32 MiB is read at once, and a larger one is read with the given `cache_type`, or without it with the `"none"` cache, which requests only the ranges that pyarrow reads. +They apply to the reads through PyAthena's filesystem: CSV results read without the chunksize option, which use `"bytes"` and 5 MiB blocks by default, and the Parquet files of UNLOAD results read with the chunksize option when PyArrow is installed and `execute()` is given no Polars read options. +Of the latter, a file of up to 32 MiB is read at once, and a larger one is read with the given `cache_type`, or without it with the `"none"` cache, which requests only the ranges that PyArrow reads. With `storage_options` given to `execute()`, the given options replace PyAthena's S3 settings, and `block_size` and `cache_type` do not apply. For a CSV result read without the chunksize option, the options are passed to `fsspec.open()`, which uses PyAthena's S3FileSystem; for the other reads, they are passed to the Polars read function, which uses its native S3 access by default and fsspec with `use_pyarrow=True`. @@ -413,7 +414,7 @@ For a CSV result read without the chunksize option, the options are passed to `f ## AsyncPolarsCursor -AsyncPolarsCursor is an AsyncCursor that can handle [polars.DataFrame object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). +AsyncPolarsCursor is an AsyncCursor that can handle [`polars.DataFrame` object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html). This cursor directly handles the CSV of query results output to S3 in the same way as PolarsCursor. You can use the AsyncPolarsCursor by specifying the `cursor_class` @@ -522,7 +523,7 @@ result_set = future.result() print(result_set.fetchall()) ``` -This object also has an as_polars method that returns a [polars.DataFrame object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) similar to the PolarsCursor. +This object also has an as_polars method that returns a [`polars.DataFrame` object](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) similar to the PolarsCursor. ```python from pyathena import connect diff --git a/docs/s3fs.md b/docs/s3fs.md index 24769f469..d5091ed4a 100644 --- a/docs/s3fs.md +++ b/docs/s3fs.md @@ -7,12 +7,12 @@ ## S3FSCursor S3FSCursor is a lightweight cursor that directly handles the CSV file of the query execution result output to S3. -Unlike ArrowCursor or PandasCursor, this cursor does not require pandas or pyarrow dependencies, +Unlike ArrowCursor or PandasCursor, this cursor does not require pandas or PyArrow dependencies, making it ideal for environments where installing these libraries is not desirable. **Key features:** -- No pandas or pyarrow dependencies required +- No pandas or PyArrow dependencies required - Lightweight CSV parsing (custom parser or Python's built-in `csv` module) - Lower memory footprint for simple query results - Full DB API 2.0 compatibility @@ -271,7 +271,7 @@ S3FSCursor has some limitations compared to ArrowCursor or PandasCursor: S3FSCursor is recommended when: -- You want to minimize dependencies (no pandas/pyarrow required) +- You want to minimize dependencies (no pandas/PyArrow required) - You're working in a constrained environment (e.g., AWS Lambda with size limits) - You only need simple row-by-row result processing - Memory efficiency is important and results don't need columnar operations @@ -283,7 +283,7 @@ For large-scale data processing or analytical workloads, consider using ArrowCur ## AsyncS3FSCursor AsyncS3FSCursor is an AsyncCursor that uses the same lightweight CSV parsing as S3FSCursor. -This cursor is useful when you need to execute queries asynchronously without pandas or pyarrow dependencies. +This cursor is useful when you need to execute queries asynchronously without pandas or PyArrow dependencies. You can use the AsyncS3FSCursor by specifying the `cursor_class` with the connect method or connection object. diff --git a/docs/sqlalchemy.md b/docs/sqlalchemy.md index fbad00aae..b09539ee0 100644 --- a/docs/sqlalchemy.md +++ b/docs/sqlalchemy.md @@ -1,3 +1,4 @@ + (sqlalchemy)= # SQLAlchemy @@ -160,7 +161,7 @@ awsathena+aiorest://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_st `Date` and `DateTime` values render as `DATE` and `TIMESTAMP` literals, both as bound parameters and as inline literals, and `TIMESTAMP` literals keep microseconds as described in {ref}`usage-query-with-parameters`. A `cast()` to `DateTime`, and the casts that ARRAY, MAP and ROW values use, render `TIMESTAMP(6)`, because a bare `TIMESTAMP` is `timestamp(3)` in Athena. -Their results are `timestamp(6)` whatever the value, so `CREATE TABLE AS SELECT` into a Hive table and `UNLOAD`, including the `unload=True` option of the arrow, pandas and polars drivers, reject them as output columns. +Their results are `timestamp(6)` whatever the value, so `CREATE TABLE AS SELECT` into a Hive table and `UNLOAD`, including the `unload=True` option of the `arrow`, `pandas` and `polars` drivers, reject them as output columns. `AthenaTimestamp(precision=p)`, with `p` from 0 to 6, casts to `TIMESTAMP(p)` and truncates its bound values and literals, including values compared with a column of that type, to `p` fractional digits: ```python @@ -178,26 +179,26 @@ Column definitions in `CREATE TABLE` render `TIMESTAMP` for `DateTime` and for ` ### Sync -| Dialect | Driver | Schema | Cursor | -|-----------|--------|------------------|------------------------| -| awsathena | rest | awsathena | DefaultCursor | -| awsathena | rest | awsathena+rest | DefaultCursor | -| awsathena | pandas | awsathena+pandas | {ref}`pandas-cursor` | -| awsathena | arrow | awsathena+arrow | {ref}`arrow-cursor` | -| awsathena | polars | awsathena+polars | {ref}`polars-cursor` | -| awsathena | s3fs | awsathena+s3fs | {ref}`s3fs-cursor` | +| Dialect | Driver | Schema | Cursor | +|-------------|----------|--------------------|----------------------| +| `awsathena` | `rest` | `awsathena` | DefaultCursor | +| `awsathena` | `rest` | `awsathena+rest` | DefaultCursor | +| `awsathena` | `pandas` | `awsathena+pandas` | {ref}`pandas-cursor` | +| `awsathena` | `arrow` | `awsathena+arrow` | {ref}`arrow-cursor` | +| `awsathena` | `polars` | `awsathena+polars` | {ref}`polars-cursor` | +| `awsathena` | `s3fs` | `awsathena+s3fs` | {ref}`s3fs-cursor` | ### Async Requires `pip install PyAthena[aiosqlalchemy]`. -| Dialect | Driver | Schema | Cursor | -|-----------|-----------|---------------------|------------------------------| -| awsathena | aiorest | awsathena+aiorest | DefaultCursor (async) | -| awsathena | aiopandas | awsathena+aiopandas | {ref}`pandas-cursor` (async) | -| awsathena | aioarrow | awsathena+aioarrow | {ref}`arrow-cursor` (async) | -| awsathena | aiopolars | awsathena+aiopolars | {ref}`polars-cursor` (async) | -| awsathena | aios3fs | awsathena+aios3fs | {ref}`s3fs-cursor` (async) | +| Dialect | Driver | Schema | Cursor | +|-------------|-------------|-----------------------|------------------------------| +| `awsathena` | `aiorest` | `awsathena+aiorest` | DefaultCursor (async) | +| `awsathena` | `aiopandas` | `awsathena+aiopandas` | {ref}`pandas-cursor` (async) | +| `awsathena` | `aioarrow` | `awsathena+aioarrow` | {ref}`arrow-cursor` (async) | +| `awsathena` | `aiopolars` | `awsathena+aiopolars` | {ref}`polars-cursor` (async) | +| `awsathena` | `aios3fs` | `awsathena+aios3fs` | {ref}`s3fs-cursor` (async) | ## Dialect options @@ -716,8 +717,8 @@ The `on_start_query_execution` callback is supported by all PyAthena SQLAlchemy - `awsathena` and `awsathena+rest` (default cursor) - `awsathena+pandas` (pandas cursor) -- `awsathena+arrow` (arrow cursor) -- `awsathena+polars` (polars cursor) +- `awsathena+arrow` (Arrow cursor) +- `awsathena+polars` (Polars cursor) - `awsathena+s3fs` (S3FS cursor) - `awsathena+aiorest`, `awsathena+aiopandas`, `awsathena+aioarrow`, `awsathena+aiopolars`, and `awsathena+aios3fs` (aio cursors) diff --git a/docs/temporal_types.md b/docs/temporal_types.md index e56b79257..696aac874 100644 --- a/docs/temporal_types.md +++ b/docs/temporal_types.md @@ -42,7 +42,7 @@ Their fetch methods convert these values to `datetime` and `time` objects. Typed values cover years 1 to 9999. PyAthena does not check for other years. -For them, the result is whatever Python, pandas, pyarrow, or Polars returns or raises. +For them, the result is whatever Python, pandas, PyArrow, or Polars returns or raises. `PandasCursor` holds `timestamp with time zone` values in UTC, so for them the range applies to the year in UTC. For example, the fetch methods raise an error for `0001-01-01 00:00:00.000 Asia/Tokyo`, which is in year 0 in UTC. diff --git a/docs/usage.md b/docs/usage.md index 2d79926fd..8852cb554 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -239,7 +239,7 @@ You can find more information about the [considerations and limitations of param ## Execution options The `execute()` method of every SQL cursor (`Cursor`, `AsyncCursor`, the aio cursors, and their -pandas/arrow/polars/s3fs variants) accepts the same set of shared keyword arguments, such as +pandas, Arrow, Polars, and S3FS variants) accepts the same set of shared keyword arguments, such as `work_group`, `s3_staging_dir`, `cache_size`, `cache_expiration_time`, `result_reuse_enable`, `result_reuse_minutes`, `paramstyle`, and `result_set_type_hints`. The synchronous and aio cursors also accept `on_start_query_execution`. @@ -717,7 +717,7 @@ column. You can mix both styles in the same dictionary. does not clearly delimit nested arrays. If your query returns nested arrays (e.g. `array(array(integer))`), use `CAST(... AS JSON)` in your query to get JSON-formatted output, which is parsed reliably. -- **Arrow, Pandas, and Polars cursors** — These cursors accept `result_set_type_hints` +- **Arrow, pandas, and Polars cursors** — These cursors accept `result_set_type_hints` but their converters do not currently use the hints because they rely on their own type systems. The parameter is passed through for forward compatibility.