Repository navigation
[SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar - #249
Closed
HyukjinKwon wants to merge 31 commits into
Closed
HyukjinKwon wants to merge 31 commits into
HyukjinKwon wants to merge 31 commits into
Conversation
…mantics Use explicit LongType schema names when constructing Arrow columns. Limit the columnar path to concrete pandas/NumPy arrays and native schema descriptors, including authentic converter aliases retained by already imported modules. Keep nullable values in non-nullable fields on the native row path so local validation still precedes configuration RPCs. Support warm Arrow importers that resolve converters locally. Add mock-RPC regressions for all emitted integer cells, schema metadata and nullability, custom conversion hooks, and repeated warm construction. Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and assert its concrete label type. Pandas 3 otherwise normalizes constructor labels to builtin strings before the unchanged integer helper sees them. The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed regression and full CI-selected Arrow/drop-in suites pass on both versions. Runtime Python and native sources remain unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Avoid loading optional Arrow label modules for fallback inputs, and decline integer Arrow conversion when the loaded importer AtomicType alias is absent. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Reuse inline IPC for explicit native string schemas after authenticating ordinary pandas Arrow string storage and normalization hooks. Retain the row path for customized and unsupported inputs. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…pache#145 Takes master's tree, which has apache#121 (SPARK-60676) and apache#208 (SPARK-60720) as they were merged; the next commit applies the change of apache#145 to it. Co-authored-by: Isaac <no-reply@databricks.com>
…importer The change of apache#145 by DB Tsai, on top of the current master. `prepare_string_input` returns, for frames whose columns are all kept in pandas' Arrow string storage (`StringDtype` with `pyarrow` storage, including the NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), an Arrow table of that storage instead of None, which the session passes to the Arrow importer, as the int64 and date tables are passed. pandas converts such a column for the row path from that storage, with its missing values as None, so the values are the same. Review changes on top of apache#145: - The storage is read through pandas' public API (`Series.array`, `__arrow_array__`), and only pandas' own array classes are trusted, rather than walking the block manager and reading `_pa_array` behind checks of pandas internals. - The table goes to the importer, which uploads it at the relation cache threshold, rather than inline: a large inline relation failed (an HTTP/2 error above about 128 MB). - It works on pandas 2.2, the lower bound of the supported range, whose `StringDtype` differs. - The tests, in `test_arrow_handoff.py`, replace `test_arrow_string_input.py`. Co-authored-by: Isaac <no-reply@databricks.com>
…pache#145 Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This replaces #145 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below.
createDataFrame(pandas_df, schema), for a pandas DataFrame of at least 10,000 rows whose columns are all kept in pandas' Arrow string storage (StringDtypewithpyarrowstorage, asstring[pyarrow], including the NumPy-semantics variant of pandas 2.2 and thestrdtype of pandas 3), with an explicitStructTypeofStringType(UTF8_BINARY) fields, now passes that storage to the Arrow importer as Arrow string columns (_arrow_string_tableinpython/pyspark/sql/_pandas_input.py), instead of materializing Python rows. pandas converts such a column for the row path from that same storage, with its missing values asNone, so the values are the same.Everything else keeps the row path:
string[python]storage, mixed dtypes, a null in a non-nullable field, subclasses of pandas' Arrow string arrays, smaller frames, other schemas, and the cases that the guards of #208 (SPARK-60720) already decline.Review changes on top of #145:
Series.array,__arrow_array__), and only pandas' own array classes are trusted, instead of walking pandas' block manager (_mgr,iget_values,blknos,blklocs) and reading_pa_arraybehind about 200 lines of checks of pandas internals.spark.sql.session.localRelationCacheThreshold; [SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar #145 sent it inline, and a large inline relation fails (an HTTP/2 error above about 128 MB).StringDtypehas apyarrow_numpystorage instead ofna_value.test_arrow_string_input.py, which required pandas 2.3, is replaced by tests intest_arrow_handoff.py.Why are the changes needed?
SPARK-60745: converting Arrow-backed strings into Python rows and back to Arrow dominates
createDataFrameof such frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time ofcreateDataFrameof 200,000 rows of threestring[pyarrow]columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 266 ms on the current master, 26 ms with this change (official PySpark 4.3.0: 23 ms). All give the same rows.Does this PR introduce any user-facing change?
No, other than speed.
How was this patch tested?
python/tests/test_arrow_handoff.py, against the mock Arrow input service: every cell, field name, nullability and metadata ofstring[pyarrow], NumPy-semantics,large_string-backed and sliced columns, inline below the relation cache threshold and uploaded at it, without materializing rows; the same table as the row path; and each unsupported input keeps the row path.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5
This pull request and its description were written by Isaac.