Skip to content

[SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar - #249

Closed
HyukjinKwon wants to merge 31 commits into
apache:masterfrom
HyukjinKwon:SPARK-60745-arrow-string
Closed

HyukjinKwon wants to merge 31 commits into
apache:masterfrom
HyukjinKwon:SPARK-60745-arrow-string

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This replaces #145 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below.

createDataFrame(pandas_df, schema), for a pandas DataFrame of at least 10,000 rows whose columns are all kept in pandas' Arrow string storage (StringDtype with pyarrow storage, as string[pyarrow], including the NumPy-semantics variant of pandas 2.2 and the str dtype of pandas 3), with an explicit StructType of StringType (UTF8_BINARY) fields, now passes that storage to the Arrow importer as Arrow string columns (_arrow_string_table in python/pyspark/sql/_pandas_input.py), instead of materializing Python rows. pandas converts such a column for the row path from that same storage, with its missing values as None, so the values are the same.

Everything else keeps the row path: string[python] storage, mixed dtypes, a null in a non-nullable field, subclasses of pandas' Arrow string arrays, smaller frames, other schemas, and the cases that the guards of #208 (SPARK-60720) already decline.

Review changes on top of #145:

  • The storage is read through pandas' public API (Series.array, __arrow_array__), and only pandas' own array classes are trusted, instead of walking pandas' block manager (_mgr, iget_values, blknos, blklocs) and reading _pa_array behind about 200 lines of checks of pandas internals.
  • The table goes to the Arrow importer, as the int64 and date tables do, so it is uploaded as cached artifacts at spark.sql.session.localRelationCacheThreshold; [SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar #145 sent it inline, and a large inline relation fails (an HTTP/2 error above about 128 MB).
  • It works on pandas 2.2, the lower bound of the supported range, whose StringDtype has a pyarrow_numpy storage instead of na_value. test_arrow_string_input.py, which required pandas 2.3, is replaced by tests in test_arrow_handoff.py.
  • The architecture docs describe the path.

Why are the changes needed?

SPARK-60745: converting Arrow-backed strings into Python rows and back to Arrow dominates createDataFrame of such frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of createDataFrame of 200,000 rows of three string[pyarrow] columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 266 ms on the current master, 26 ms with this change (official PySpark 4.3.0: 23 ms). All give the same rows.

Does this PR introduce any user-facing change?

No, other than speed.

How was this patch tested?

  • New tests in python/tests/test_arrow_handoff.py, against the mock Arrow input service: every cell, field name, nullability and metadata of string[pyarrow], NumPy-semantics, large_string-backed and sliced columns, inline below the relation cache threshold and uploaded at it, without materializing rows; the same table as the row path; and each unsupported input keeps the row path.
  • Against the current master and a local Spark 4.3.0 server: the full suites (results below the merge).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5

This pull request and its description were written by Isaac.

dbtsai and others added 30 commits November 5, 2026 15:59
…mantics

Use explicit LongType schema names when constructing Arrow columns. Limit the
columnar path to concrete pandas/NumPy arrays and native schema descriptors,
including authentic converter aliases retained by already imported modules.
Keep nullable values in non-nullable fields on the native row path so local
validation still precedes configuration RPCs. Support warm Arrow importers that
resolve converters locally.

Add mock-RPC regressions for all emitted integer cells, schema metadata and
nullability, custom conversion hooks, and repeated warm construction.

Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and
assert its concrete label type. Pandas 3 otherwise normalizes constructor
labels to builtin strings before the unchanged integer helper sees them.

The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed
regression and full CI-selected Arrow/drop-in suites pass on both versions.
Runtime Python and native sources remain unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>
Avoid loading optional Arrow label modules for fallback inputs, and decline
integer Arrow conversion when the loaded importer AtomicType alias is absent.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Reuse inline IPC for explicit native string schemas after authenticating ordinary pandas Arrow string storage and normalization hooks. Retain the row path for customized and unsupported inputs.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…pache#145

Takes master's tree, which has apache#121 (SPARK-60676) and apache#208 (SPARK-60720) as they were merged; the next commit applies the change of apache#145 to it.

Co-authored-by: Isaac <no-reply@databricks.com>
…importer

The change of apache#145 by DB Tsai, on top of the current master. `prepare_string_input` returns,
for frames whose columns are all kept in pandas' Arrow string storage (`StringDtype` with
`pyarrow` storage, including the NumPy-semantics variant of pandas 2.2 and the `str` dtype of
pandas 3), an Arrow table of that storage instead of None, which the session passes to the
Arrow importer, as the int64 and date tables are passed. pandas converts such a column for the
row path from that storage, with its missing values as None, so the values are the same.

Review changes on top of apache#145:
- The storage is read through pandas' public API (`Series.array`, `__arrow_array__`), and only
  pandas' own array classes are trusted, rather than walking the block manager and reading
  `_pa_array` behind checks of pandas internals.
- The table goes to the importer, which uploads it at the relation cache threshold, rather than
  inline: a large inline relation failed (an HTTP/2 error above about 128 MB).
- It works on pandas 2.2, the lower bound of the supported range, whose `StringDtype` differs.
- The tests, in `test_arrow_handoff.py`, replace `test_arrow_string_input.py`.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants