Skip to content

[SPARK-60745][CONNECT] Keep Arrow-backed pandas string input columnar - #145

Closed
dbtsai wants to merge 28 commits into
apache:masterfrom
dbtsai:codex/SPARK-60745-arrow-string
Closed

dbtsai wants to merge 28 commits into
apache:masterfrom
dbtsai:codex/SPARK-60745-arrow-string

Conversation

@dbtsai

@dbtsai dbtsai commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Current change after the master rebase

Keep eligible canonical Arrow-backed pandas StringDtype columns columnar under explicit native binary-collated StringType schemas with at least 10,000 rows. Validate Arrow buffers, cast to standard UTF-8 and reuse PR #144's inline IPC bridge; unsupported/customized input retains current pandas row conversion under the finite guarded-hook scope. Width-error regressions follow current master's structured AXIS_LENGTH_MISMATCH error and preserve failure-before-configuration/upload/delivery checks. The branch remains stacked on PR #144.

Validation of the rebased change

Validated exact head 52d466de7079c392a8ad84ddc6bff3c01aa06bcd against upstream master 014674e44470001f5c184846964c131d43f6e4fa. Formatting, 115 core and 419 API Rust unit tests, all API integration-target compilation with and without wasm-udf, the native extension build, and 1163 offline/mock Python tests passed (8 skips). Committed whitespace checks passed. Workspace Clippy remains advisory and fails on unmodified master; diagnostics are compared separately in the rebase evidence. Full live-Spark parity and CI for the rewritten head remain pending. No new performance, memory, or RPC-cost measurement is claimed; historical results do not validate this rebased head.

Historical pre-rebase description, validation and performance evidence

The complete previous description is preserved below as historical evidence. Every test count, benchmark number, source/dependency/revision ID, RPC observation, source-equivalence claim, and use of “current”, “final”, “fresh” or “new” in this appendix refers to its original pre-rebase context. Master has since changed local inference and conversion, cached schema access and Arrow configuration handling. The current scope above takes precedence over stale runtime, dependency and per-call RPC statements below. None of these results validates or measures the November 5 rebased head. Prior failures, guard/cold-path costs, RPC/memory observations and live-server limitations remain preserved within their stated historical scope.

What changes were proposed in this pull request?

Keep canonical Arrow-backed pandas StringDtype columns columnar for an explicit native schema containing only binary-collated StringType fields and at least 10,000 rows. Validate Arrow buffers, cast to standard UTF-8, and reuse the inline IPC bridge. Guards authenticate a finite set of directly skipped normalization/storage hooks; customized or unsupported inputs retain row conversion. This does not guarantee arbitrary pandas customization equivalence.

Depends on PR #144, including its prerequisite stack. This narrow delta compares baseline d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723 with candidate 28f2b65cfcda7460459e67dcbac7e9437b385c4b: one production helper and two test files. Measurements do not attribute prerequisite-stack changes to this patch.

Why are the changes needed?

Arrow-backed string columns currently become Python objects and rows, discarding their columnar representation. Eligible inputs can retain Arrow storage through explicit-schema IPC.

Does this PR introduce any user-facing change?

Eligible inputs use columnar conversion. Tests preserve values, schemas and checked error types/messages/order against forced row conversion, covering customized direct hooks, manager caches, malformed buffers and public error/RPC ordering.

How was this patch tested?

The selected offline suites passed: pandas 2.3.3, 865 passed/9 skipped; pandas 3.1.0, 863 passed/11 skipped. All 184 added cases pass in each; all 690 baseline outcomes/skip reasons remain.

Original formatting, native client/core units (394/95), native/host-WASM test compilation and wheel builds are explicitly reused under identical production/build source bytes. Baseline Python results are reused; full candidate Python suites ran at the committed test head. Four-environment before/after captures verify source, wheels, installed packages, native and recorded dependency identities. Original/recovery-v2 failures remain FAILED; no fresh Clippy/live Spark result.

Runs use Python 3.12.14, NumPy 2.5.4, PyArrow 26.0.0, grpcio 1.84.1 and pandas 2.3.3/3.1.0 with frozen harness 1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4; Rust packages report 4.3.0, while separate official reference ca1514a2b4e72f9529871f34d09881cd4921df82 reports 5.0.0.dev0 and is measured only with pandas 2.3.3.

Three completed, independently reviewed and root-accepted groups contain 1,175 worker samples: explicit640, object160 and ordinary375. Cohorts remain separate; five samples per role provide descriptive distributions, not significance, confidence intervals or general neutrality.

Explicit640. All 16 eligible cells (two pandas versions × two profiles × two nominal missing slots × exact/Unicode-null-chunked variants) improve median elapsed 88.2–98.9% and CPU 89.3–98.9%. Candidate/baseline ratios are 0.010873–0.117645 and 0.010874–0.106766 respectively. Exact nominal pd_na medians, baseline → candidate:

pandas/profile elapsed ms CPU ms
2.3.3/warm 21.526 → 0.257 21.552 → 0.257
2.3.3/first 25.418 → 2.528 25.105 → 2.058
3.1.0/warm 23.213 → 0.252 23.234 → 0.253
3.1.0/first 26.510 → 2.153 26.172 → 2.148

Fallback/error costs are mixed: 26/64 elapsed and 26/64 CPU medians rise. Warm 3.1.0 hook-error/pd_na elapsed 0.0442 → 0.1036 ms (2.345×, +0.0594), CPU 0.0445 → 0.0772 (1.736×, +0.0327); warm 2.3.3 hook-error elapsed reaches 1.596×. First 3.1.0 hook-error/nan elapsed 1.0639 → 1.4113 ms (+347.38 µs, 1.327×), CPU 0.781 → 0.993. Warm 3.1.0 Python-storage/nan elapsed 20.865 → 23.965 (1.149×), CPU 20.889 → 23.431 (1.122×); first 2.3.3 Python-storage/pd_na elapsed 26.093 → 29.031 (+2.938 ms/+11.26%). First 2.3.3 threshold/nan elapsed 25.594 → 28.164 (1.100×), CPU 25.120 → 27.551 (1.097×). Generic-Arrow worst elapsed/CPU ratios are 1.033×/1.050× in different cells.

Eligible held-RSS medians fall 3,096,576–11,403,264 bytes. Warm 3.1.0 Python-storage/nan nevertheless has absolute held/post-validation increases +5,095,424/+4,571,136 bytes and signed held-minus-baseline +1,638,400 (4,784,128 → 6,422,528). First 2.3.3 generic-Arrow/nan has worst signed post-validation increase +2,555,904 (13,828,096 → 16,384,000), alongside absolute post-validation +2,883,584 and held +278,528. Eligible warm 2.3.3 chunked/pd_na absolute post-validation falls 5,505,024 while its signed increment rises 655,360.

Object160. Pandas 2.3.3 only, eight 10,000×3 fixtures/two profiles. Candidate/baseline median and distribution-maximum timing ratios follow; candidate-minus-baseline absolute held-RSS and signed held-minus-baseline median differences in bytes. Lifetime-peak medians equal held medians.

profile/fixture elapsed × CPU × elapsed max × CPU max × held Δ B signed Δ B
warm/exact 0.932 0.932 0.982 0.979 +65,536 -16,384
warm/all_null 0.983 0.973 0.971 0.970 +98,304 0
warm/early_subclass 1.053 1.049 1.071 1.054 -311,296 +49,152
warm/middle_subclass 0.991 0.991 0.884 0.944 -409,600 -98,304
warm/late_subclass 0.986 0.986 1.013 1.012 +262,144 +376,832
warm/early_surrogate 0.986 0.984 1.003 1.008 +147,456 +16,384
warm/middle_surrogate 1.061 1.061 1.085 1.085 +180,224 +16,384
warm/late_surrogate 1.005 1.006 1.004 1.004 -16,384 0
first/exact 1.010 1.023 0.943 1.078 0 +32,768
first/all_null 1.010 1.039 0.897 0.881 +212,992 -32,768
first/early_subclass 1.043 1.025 1.107 1.074 -294,912 -49,152
first/middle_subclass 1.085 1.045 1.083 1.046 -147,456 +32,768
first/late_subclass 0.998 1.001 0.989 1.013 -114,688 +81,920
first/early_surrogate 0.931 0.952 0.907 0.835 -98,304 +32,768
first/middle_surrogate 1.041 1.014 1.110 1.065 -32,768 -16,384
first/late_surrogate 1.002 1.004 0.943 0.940 +49,152 +65,536

Nine elapsed/ten CPU medians rise. Warm early-subclass elapsed/CPU +5.30%/+4.86%; middle-surrogate +6.06%/+6.11%. First middle-subclass elapsed 24.847 → 26.954 ms (+8.48%), CPU 24.844 → 25.970 (+4.53%). Worst baseline-RSS median increase: first all-null +524,288; held/lifetime-peak: warm late-subclass +262,144; signed increment: the latter +376,832 (5,603,328 → 5,980,160).

Distribution maxima also rise: first early-subclass elapsed +2.970 ms/CPU +2.007 ms; largest maximum ratios first middle-surrogate elapsed 1.110×/warm middle-surrogate CPU 1.085×. Warm early-subclass absolute held/lifetime-peak maxima rise +5,046,272 bytes, signed maxima +5,931,008. These compare cohort maxima, not matched samples or sampled construction peaks.

Ordinary375. Twenty-seven cohorts: 21 pandas-2.3.3 cohorts include official Python plus both Rust roles; six selected pandas-3.1.0 cohorts compare Rust roles only. Versions have different selected coverage; official Python is a reference, not this patch's baseline. All candidate/baseline median and distribution-maximum timing ratios:

pandas/profile/case elapsed × CPU × elapsed max × CPU max ×
233/first/explicit-10k×32 0.795 0.990 0.740 0.567
233/first/type-date-10k×3 0.998 1.011 1.022 1.020
233/first/type-string-10k×3 0.923 1.020 1.084 0.926
233/warm/arrow-explicit-10k×3 0.973 1.007 0.632 0.751
233/warm/ddl-10k×3 1.013 1.014 0.986 1.007
233/warm/explicit-10k×3 1.016 1.016 1.046 1.045
233/warm/explicit-10k×32 0.999 1.002 0.117 0.605
233/warm/inferred-10k×3 0.987 0.987 1.005 1.006
233/warm/scale-1M×3 1.017 1.003 0.500 0.844
233/warm/scale-100k×3 0.952 0.948 0.307 1.241
233/warm/scale-1000x3 0.834 1.037 0.179 0.710
233/warm/scale-100x3 1.025 1.026 1.136 1.138
233/warm/tuples-explicit-10k×3 0.996 0.997 0.980 0.980
233/warm/type-array-10k×3 0.997 0.997 0.974 0.974
233/warm/type-date-10k×3 1.002 1.002 1.026 1.023
233/warm/type-decimal-10k×3 1.020 1.020 1.104 1.060
233/warm/type-string-10k×3 0.949 0.949 0.961 0.966
233/warm/type-timestamp-10k×3 0.983 0.983 0.939 0.953
233/warm/return-100k×8-collect 0.999 1.015 0.853 0.979
233/warm/return-100k×8-toArrow 0.967 0.873 1.426 0.942
233/warm/udf-construct-capture-0 0.964 0.968 1.082 1.075
310/first/explicit-10k×32 0.758 0.996 0.693 0.929
310/first/type-date-10k×3 1.003 1.008 0.999 1.007
310/first/type-string-10k×3 0.061 0.058 0.075 0.058
310/warm/explicit-10k×32 0.985 1.016 0.963 1.301
310/warm/type-date-10k×3 0.997 0.996 0.986 0.986
310/warm/type-string-10k×3 0.021 0.021 0.022 0.023

Seven elapsed/14 CPU medians rise. Largest elapsed ratio: 2.3.3/warm/100×3 +2.5325%; CPU: 1000×3 +3.7037%. Decimal has largest absolute median increases: elapsed 24.831 → 25.325 ms (+1.99%), CPU +0.493 ms. Pandas-3.1.0 strings improve elapsed 22.904 → 0.481 ms warm and 22.198 → 1.345 first. Largest elapsed maximum increase: warm 2.3.3 toArrow 27.990 → 39.903 ms (+42.56%); CPU maximum ratio: warm 3.1.0 wide 1.301×; largest absolute CPU-maximum increase: decimal +1.512 ms.

Worst higher absolute RSS medians: baseline +442,368 bytes (2.3.3 first wide); held +507,904/lifetime-peak +606,208 bytes (3.1.0 warm wide); post-GC +475,136 (2.3.3 warm Arrow); sampled peak +884,736 (2.3.3 warm 1M). Worst signed median increases: held +229,376 (warm toArrow), post-GC +1,015,808 (3.1.0 warm strings, despite lower absolute post-GC), sampled peak +950,272 (warm 1M). Decimal has largest absolute held/sampled/lifetime-peak maxima increases +2,408,448, post-GC +2,473,984; signed maxima +2,179,072. Maxima compare distributions, not matched endpoints. Every ordinary cohort has some higher descriptive observation.

Scope and limits. Explicit/object warm uses 20 constructions/three warmups; first uses one/no constructor warmup after interpreter/input/session/server preparation. Ordinary warm uses one operation warmup; first uses zero after untimed spark.version; measured loops have one operation except warm 2.3.3 UDF (five). Automatic GC is not explicitly disabled. No cold-interpreter or per-sample helper-dispatch claim. Explicit/object loop/error-checking overhead is included; preparation/validation IPC decoding is outside construction timing. Ordinary setup/warmup/counters/RSS/validation are outside operation timing; process CPU brackets the wall timer. Explicit/object order alternates deterministically; ordinary retains rotated order, without randomization. Generic Arrow uses actual pd.NA in both nominal slots.

Independent explicit review decodes all 480 retained last-successful DataFrame payloads/14,399,760 cells and verifies 1,680 measured expected-error checks; preceding successful iterations are not individually decoded. Object independently decodes 100 retained UTF-8 IPC/protobuf payloads/3,000,000 cells, only last successful results. Its 630 surrogate attempts/hook totals remain sealed-worker assertions: printed totals/one error fingerprint, no retained per-iteration errors or independent replay. Object has aggregate stdout timing/RSS, without pre-validator construction-measurement files/nanosecond endpoints. Ordinary rehashes/cross-links all 330 typed-validator receipts/requests/payloads covering 72,149,500 cells plus 45 noninput guards; typed value/null-mask/nested-null/schema validation remains a frozen producer assertion, not fresh independent decoding.

Construction RPC zeros apply to explicit/object sole-loopback fixtures; frozen20 zeros are inferred separately. Ordinary directly observes all 20 counter keys in all 375 sample maps, including nonzero input Config traffic (four calls/512 bytes); all 135 Rust-pair maps match. Return/config traffic exists; there is no overall-zero or full-wire claim.

RSS baseline/held precedes validation; lifetime ru_maxrss includes preparation/warmups. Explicit post-validation RSS follows output/mock-holder release/one GC; inputs/expected/schema/session/server and native arenas/internal frames may remain. Ordinary post-GC includes additive validation allocations. Neither proves complete reclamation/construction-only retention. Ordinary external RSS targets 2 ms but records 1–59 observations/largest gap 5.520125 ms; observation starts before go and stops on measured message, not exact timer endpoints; short transients can be missed. Object has no post-GC/sampled-peak/count/gap fields. Signed deltas can be nonpositive; ratios require positive denominators; medians of signed deltas differ from differences of absolute medians.

Ordinary harness truncates stderr at 8,000 and exception stdout at 2,000 characters. Object streams/receipts exist before launch; interruption may leave RUNNING receipts/only written bytes. Selected dependency RECORD/PySpark-origin inventories do not establish shadow-import/loaded-dependency/continuous isolation. Inherited-group supervision does not cover arbitrary detach/setuid/host crash/realtime/external-supervisor-SIGKILL guarantees.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-6)

This pull request and its description were written by Isaac.

dbtsai and others added 28 commits November 5, 2026 15:59
…mantics

Use explicit LongType schema names when constructing Arrow columns. Limit the
columnar path to concrete pandas/NumPy arrays and native schema descriptors,
including authentic converter aliases retained by already imported modules.
Keep nullable values in non-nullable fields on the native row path so local
validation still precedes configuration RPCs. Support warm Arrow importers that
resolve converters locally.

Add mock-RPC regressions for all emitted integer cells, schema metadata and
nullability, custom conversion hooks, and repeated warm construction.

Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and
assert its concrete label type. Pandas 3 otherwise normalizes constructor
labels to builtin strings before the unchanged integer helper sees them.

The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed
regression and full CI-selected Arrow/drop-in suites pass on both versions.
Runtime Python and native sources remain unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>
Avoid loading optional Arrow label modules for fallback inputs, and decline
integer Arrow conversion when the loaded importer AtomicType alias is absent.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Reuse inline IPC for explicit native string schemas after authenticating ordinary pandas Arrow string storage and normalization hooks. Retain the row path for customized and unsupported inputs.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@dbtsai
dbtsai force-pushed the codex/SPARK-60745-arrow-string branch from 28f2b65 to 52d466d Compare November 6, 2026 02:09
@HyukjinKwon

Copy link
Copy Markdown
Member

Thanks for this. As agreed with the maintainers, I am leaving it open for DB Tsai. #144, which it is stacked on, is being replaced by a PR from a maintainer's fork, since these branches do not allow maintainer edits. Before this lands:

  • _prepare_arrow_string_input walks pandas' block manager (_mgr, iget_values, blknos, blklocs) and reads _pa_array and _dtype directly. The public API gives the same values: column.array.__arrow_array__(type=pa.string()), then .cast(pa.string()), matched the row path for string[pyarrow], sliced frames and ArrowDtype(large_string). Gating on type(dtype) is StringDtype and dtype.storage == "pyarrow" could replace about 200 lines.
  • On pandas 2.2, the lower bound of the supported range, the hook checks fail (StringDtype.name is a plain class attribute there), so the path declines a canonical pd.NA column, and test_arrow_string_input.py fails at StringDtype(na_value=...), which 2.2 does not take.
  • It encodes the IPC in Python, while [SPARK-60720][CONNECT] Encode guarded pandas string columns directly in Rust #144 adds a Rust encoder for strings; one of the two would do.

@HyukjinKwon

Copy link
Copy Markdown
Member

Thanks for this. As this branch does not allow edits by maintainers, it continues in #249, which keeps your commits and adds the merge with master and the review changes: the storage is read through pandas' public API instead of its block manager, the table goes to the Arrow importer so that large frames are uploaded rather than sent inline, and it works on pandas 2.2. Closing in favor of #249.

@HyukjinKwon HyukjinKwon closed this Nov 7, 2026
HyukjinKwon added a commit that referenced this pull request Nov 7, 2026
### What changes were proposed in this pull request?

This replaces #145 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below.

`createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least 10,000 rows whose columns are all kept in pandas' Arrow string storage (`StringDtype` with `pyarrow` storage, as `string[pyarrow]`, including the NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), with an explicit `StructType` of `StringType` (UTF8_BINARY) fields, now passes that storage to the Arrow importer as Arrow string columns (`_arrow_string_table` in `python/pyspark/sql/_pandas_input.py`), instead of materializing Python rows. pandas converts such a column for the row path from that same storage, with its missing values as `None`, so the values are the same.

Everything else keeps the row path: `string[python]` storage, mixed dtypes, a null in a non-nullable field, subclasses of pandas' Arrow string arrays, smaller frames, other schemas, and the cases that the guards of #208 (SPARK-60720) already decline.

Review changes on top of #145:
- The storage is read through pandas' public API (`Series.array`, `__arrow_array__`), and only pandas' own array classes are trusted, instead of walking pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and reading `_pa_array` behind about 200 lines of checks of pandas internals.
- The table goes to the Arrow importer, as the int64 and date tables do, so it is uploaded as cached artifacts at `spark.sql.session.localRelationCacheThreshold`; #145 sent it inline, and a large inline relation fails (an HTTP/2 error above about 128 MB).
- It works on pandas 2.2, the lower bound of the supported range, whose `StringDtype` has a `pyarrow_numpy` storage instead of `na_value`. `test_arrow_string_input.py`, which required pandas 2.3, is replaced by tests in `test_arrow_handoff.py`.
- The architecture docs describe the path.

### Why are the changes needed?

[SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): converting Arrow-backed strings into Python rows and back to Arrow dominates `createDataFrame` of such frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of `createDataFrame` of 200,000 rows of three `string[pyarrow]` columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 266 ms on the current master, 26 ms with this change (official PySpark 4.3.0: 23 ms). All give the same rows.

### Does this PR introduce _any_ user-facing change?

No, other than speed.

### How was this patch tested?

- New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service: every cell, field name, nullability and metadata of `string[pyarrow]`, NumPy-semantics, `large_string`-backed and sliced columns, inline below the relation cache threshold and uploaded at it, without materializing rows; the same table as the row path; and each unsupported input keeps the row path.
- Against the current master and a local Spark 4.3.0 server: the full suites (results below the merge).

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5

This pull request and its description were written by Isaac.

Closes #249 from HyukjinKwon/SPARK-60745-arrow-string.

Lead-authored-by: DB Tsai <dbtsai@dbtsai.com>
Co-authored-by: DB Tsai <db.tsai@databricks.com>
Co-authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants