Repository navigation
Conversation
…mantics Use explicit LongType schema names when constructing Arrow columns. Limit the columnar path to concrete pandas/NumPy arrays and native schema descriptors, including authentic converter aliases retained by already imported modules. Keep nullable values in non-nullable fields on the native row path so local validation still precedes configuration RPCs. Support warm Arrow importers that resolve converters locally. Add mock-RPC regressions for all emitted integer cells, schema metadata and nullability, custom conversion hooks, and repeated warm construction. Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and assert its concrete label type. Pandas 3 otherwise normalizes constructor labels to builtin strings before the unchanged integer helper sees them. The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed regression and full CI-selected Arrow/drop-in suites pass on both versions. Runtime Python and native sources remain unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Avoid loading optional Arrow label modules for fallback inputs, and decline integer Arrow conversion when the loaded importer AtomicType alias is absent. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Reuse inline IPC for explicit native string schemas after authenticating ordinary pandas Arrow string storage and normalization hooks. Retain the row path for customized and unsupported inputs. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
dbtsai
force-pushed
the
codex/SPARK-60745-arrow-string
branch
from
November 6, 2026 02:09
28f2b65 to
52d466d
Compare
Member
|
Thanks for this. As agreed with the maintainers, I am leaving it open for DB Tsai. #144, which it is stacked on, is being replaced by a PR from a maintainer's fork, since these branches do not allow maintainer edits. Before this lands:
|
This was referenced Nov 6, 2026
Member
|
Thanks for this. As this branch does not allow edits by maintainers, it continues in #249, which keeps your commits and adds the merge with master and the review changes: the storage is read through pandas' public API instead of its block manager, the table goes to the Arrow importer so that large frames are uploaded rather than sent inline, and it works on pandas 2.2. Closing in favor of #249. |
HyukjinKwon
added a commit
that referenced
this pull request
Nov 7, 2026
### What changes were proposed in this pull request? This replaces #145 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below. `createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least 10,000 rows whose columns are all kept in pandas' Arrow string storage (`StringDtype` with `pyarrow` storage, as `string[pyarrow]`, including the NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), with an explicit `StructType` of `StringType` (UTF8_BINARY) fields, now passes that storage to the Arrow importer as Arrow string columns (`_arrow_string_table` in `python/pyspark/sql/_pandas_input.py`), instead of materializing Python rows. pandas converts such a column for the row path from that same storage, with its missing values as `None`, so the values are the same. Everything else keeps the row path: `string[python]` storage, mixed dtypes, a null in a non-nullable field, subclasses of pandas' Arrow string arrays, smaller frames, other schemas, and the cases that the guards of #208 (SPARK-60720) already decline. Review changes on top of #145: - The storage is read through pandas' public API (`Series.array`, `__arrow_array__`), and only pandas' own array classes are trusted, instead of walking pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and reading `_pa_array` behind about 200 lines of checks of pandas internals. - The table goes to the Arrow importer, as the int64 and date tables do, so it is uploaded as cached artifacts at `spark.sql.session.localRelationCacheThreshold`; #145 sent it inline, and a large inline relation fails (an HTTP/2 error above about 128 MB). - It works on pandas 2.2, the lower bound of the supported range, whose `StringDtype` has a `pyarrow_numpy` storage instead of `na_value`. `test_arrow_string_input.py`, which required pandas 2.3, is replaced by tests in `test_arrow_handoff.py`. - The architecture docs describe the path. ### Why are the changes needed? [SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): converting Arrow-backed strings into Python rows and back to Arrow dominates `createDataFrame` of such frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of `createDataFrame` of 200,000 rows of three `string[pyarrow]` columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 266 ms on the current master, 26 ms with this change (official PySpark 4.3.0: 23 ms). All give the same rows. ### Does this PR introduce _any_ user-facing change? No, other than speed. ### How was this patch tested? - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service: every cell, field name, nullability and metadata of `string[pyarrow]`, NumPy-semantics, `large_string`-backed and sliced columns, inline below the relation cache threshold and uploaded at it, without materializing rows; the same table as the row path; and each unsupported input keeps the row path. - Against the current master and a local Spark 4.3.0 server: the full suites (results below the merge). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5 This pull request and its description were written by Isaac. Closes #249 from HyukjinKwon/SPARK-60745-arrow-string. Lead-authored-by: DB Tsai <dbtsai@dbtsai.com> Co-authored-by: DB Tsai <db.tsai@databricks.com> Co-authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current change after the master rebase
Keep eligible canonical Arrow-backed pandas StringDtype columns columnar under explicit native binary-collated StringType schemas with at least 10,000 rows. Validate Arrow buffers, cast to standard UTF-8 and reuse PR #144's inline IPC bridge; unsupported/customized input retains current pandas row conversion under the finite guarded-hook scope. Width-error regressions follow current master's structured AXIS_LENGTH_MISMATCH error and preserve failure-before-configuration/upload/delivery checks. The branch remains stacked on PR #144.
Validation of the rebased change
Validated exact head
52d466de7079c392a8ad84ddc6bff3c01aa06bcdagainst upstream master014674e44470001f5c184846964c131d43f6e4fa. Formatting, 115 core and 419 API Rust unit tests, all API integration-target compilation with and withoutwasm-udf, the native extension build, and 1163 offline/mock Python tests passed (8 skips). Committed whitespace checks passed. Workspace Clippy remains advisory and fails on unmodified master; diagnostics are compared separately in the rebase evidence. Full live-Spark parity and CI for the rewritten head remain pending. No new performance, memory, or RPC-cost measurement is claimed; historical results do not validate this rebased head.Historical pre-rebase description, validation and performance evidence
The complete previous description is preserved below as historical evidence. Every test count, benchmark number, source/dependency/revision ID, RPC observation, source-equivalence claim, and use of “current”, “final”, “fresh” or “new” in this appendix refers to its original pre-rebase context. Master has since changed local inference and conversion, cached schema access and Arrow configuration handling. The current scope above takes precedence over stale runtime, dependency and per-call RPC statements below. None of these results validates or measures the November 5 rebased head. Prior failures, guard/cold-path costs, RPC/memory observations and live-server limitations remain preserved within their stated historical scope.
What changes were proposed in this pull request?
Keep canonical Arrow-backed pandas
StringDtypecolumns columnar for an explicit native schema containing only binary-collatedStringTypefields and at least 10,000 rows. Validate Arrow buffers, cast to standard UTF-8, and reuse the inline IPC bridge. Guards authenticate a finite set of directly skipped normalization/storage hooks; customized or unsupported inputs retain row conversion. This does not guarantee arbitrary pandas customization equivalence.Depends on PR #144, including its prerequisite stack. This narrow delta compares baseline
d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723with candidate28f2b65cfcda7460459e67dcbac7e9437b385c4b: one production helper and two test files. Measurements do not attribute prerequisite-stack changes to this patch.Why are the changes needed?
Arrow-backed string columns currently become Python objects and rows, discarding their columnar representation. Eligible inputs can retain Arrow storage through explicit-schema IPC.
Does this PR introduce any user-facing change?
Eligible inputs use columnar conversion. Tests preserve values, schemas and checked error types/messages/order against forced row conversion, covering customized direct hooks, manager caches, malformed buffers and public error/RPC ordering.
How was this patch tested?
The selected offline suites passed: pandas 2.3.3, 865 passed/9 skipped; pandas 3.1.0, 863 passed/11 skipped. All 184 added cases pass in each; all 690 baseline outcomes/skip reasons remain.
Original formatting, native client/core units (394/95), native/host-WASM test compilation and wheel builds are explicitly reused under identical production/build source bytes. Baseline Python results are reused; full candidate Python suites ran at the committed test head. Four-environment before/after captures verify source, wheels, installed packages, native and recorded dependency identities. Original/recovery-v2 failures remain FAILED; no fresh Clippy/live Spark result.
Runs use Python 3.12.14, NumPy 2.5.4, PyArrow 26.0.0, grpcio 1.84.1 and pandas 2.3.3/3.1.0 with frozen harness
1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4; Rust packages report 4.3.0, while separate official referenceca1514a2b4e72f9529871f34d09881cd4921df82reports 5.0.0.dev0 and is measured only with pandas 2.3.3.Three completed, independently reviewed and root-accepted groups contain 1,175 worker samples: explicit640, object160 and ordinary375. Cohorts remain separate; five samples per role provide descriptive distributions, not significance, confidence intervals or general neutrality.
Explicit640. All 16 eligible cells (two pandas versions × two profiles × two nominal missing slots × exact/Unicode-null-chunked variants) improve median elapsed 88.2–98.9% and CPU 89.3–98.9%. Candidate/baseline ratios are 0.010873–0.117645 and 0.010874–0.106766 respectively. Exact nominal
pd_namedians, baseline → candidate:Fallback/error costs are mixed: 26/64 elapsed and 26/64 CPU medians rise. Warm 3.1.0 hook-error/pd_na elapsed 0.0442 → 0.1036 ms (2.345×, +0.0594), CPU 0.0445 → 0.0772 (1.736×, +0.0327); warm 2.3.3 hook-error elapsed reaches 1.596×. First 3.1.0 hook-error/nan elapsed 1.0639 → 1.4113 ms (+347.38 µs, 1.327×), CPU 0.781 → 0.993. Warm 3.1.0 Python-storage/nan elapsed 20.865 → 23.965 (1.149×), CPU 20.889 → 23.431 (1.122×); first 2.3.3 Python-storage/pd_na elapsed 26.093 → 29.031 (+2.938 ms/+11.26%). First 2.3.3 threshold/nan elapsed 25.594 → 28.164 (1.100×), CPU 25.120 → 27.551 (1.097×). Generic-Arrow worst elapsed/CPU ratios are 1.033×/1.050× in different cells.
Eligible held-RSS medians fall 3,096,576–11,403,264 bytes. Warm 3.1.0 Python-storage/nan nevertheless has absolute held/post-validation increases +5,095,424/+4,571,136 bytes and signed held-minus-baseline +1,638,400 (4,784,128 → 6,422,528). First 2.3.3 generic-Arrow/nan has worst signed post-validation increase +2,555,904 (13,828,096 → 16,384,000), alongside absolute post-validation +2,883,584 and held +278,528. Eligible warm 2.3.3 chunked/pd_na absolute post-validation falls 5,505,024 while its signed increment rises 655,360.
Object160. Pandas 2.3.3 only, eight 10,000×3 fixtures/two profiles. Candidate/baseline median and distribution-maximum timing ratios follow; candidate-minus-baseline absolute held-RSS and signed held-minus-baseline median differences in bytes. Lifetime-peak medians equal held medians.
Nine elapsed/ten CPU medians rise. Warm early-subclass elapsed/CPU +5.30%/+4.86%; middle-surrogate +6.06%/+6.11%. First middle-subclass elapsed 24.847 → 26.954 ms (+8.48%), CPU 24.844 → 25.970 (+4.53%). Worst baseline-RSS median increase: first all-null +524,288; held/lifetime-peak: warm late-subclass +262,144; signed increment: the latter +376,832 (5,603,328 → 5,980,160).
Distribution maxima also rise: first early-subclass elapsed +2.970 ms/CPU +2.007 ms; largest maximum ratios first middle-surrogate elapsed 1.110×/warm middle-surrogate CPU 1.085×. Warm early-subclass absolute held/lifetime-peak maxima rise +5,046,272 bytes, signed maxima +5,931,008. These compare cohort maxima, not matched samples or sampled construction peaks.
Ordinary375. Twenty-seven cohorts: 21 pandas-2.3.3 cohorts include official Python plus both Rust roles; six selected pandas-3.1.0 cohorts compare Rust roles only. Versions have different selected coverage; official Python is a reference, not this patch's baseline. All candidate/baseline median and distribution-maximum timing ratios:
Seven elapsed/14 CPU medians rise. Largest elapsed ratio: 2.3.3/warm/100×3 +2.5325%; CPU: 1000×3 +3.7037%. Decimal has largest absolute median increases: elapsed 24.831 → 25.325 ms (+1.99%), CPU +0.493 ms. Pandas-3.1.0 strings improve elapsed 22.904 → 0.481 ms warm and 22.198 → 1.345 first. Largest elapsed maximum increase: warm 2.3.3 toArrow 27.990 → 39.903 ms (+42.56%); CPU maximum ratio: warm 3.1.0 wide 1.301×; largest absolute CPU-maximum increase: decimal +1.512 ms.
Worst higher absolute RSS medians: baseline +442,368 bytes (2.3.3 first wide); held +507,904/lifetime-peak +606,208 bytes (3.1.0 warm wide); post-GC +475,136 (2.3.3 warm Arrow); sampled peak +884,736 (2.3.3 warm 1M). Worst signed median increases: held +229,376 (warm toArrow), post-GC +1,015,808 (3.1.0 warm strings, despite lower absolute post-GC), sampled peak +950,272 (warm 1M). Decimal has largest absolute held/sampled/lifetime-peak maxima increases +2,408,448, post-GC +2,473,984; signed maxima +2,179,072. Maxima compare distributions, not matched endpoints. Every ordinary cohort has some higher descriptive observation.
Scope and limits. Explicit/object warm uses 20 constructions/three warmups; first uses one/no constructor warmup after interpreter/input/session/server preparation. Ordinary warm uses one operation warmup; first uses zero after untimed
spark.version; measured loops have one operation except warm 2.3.3 UDF (five). Automatic GC is not explicitly disabled. No cold-interpreter or per-sample helper-dispatch claim. Explicit/object loop/error-checking overhead is included; preparation/validation IPC decoding is outside construction timing. Ordinary setup/warmup/counters/RSS/validation are outside operation timing; process CPU brackets the wall timer. Explicit/object order alternates deterministically; ordinary retains rotated order, without randomization. Generic Arrow uses actualpd.NAin both nominal slots.Independent explicit review decodes all 480 retained last-successful DataFrame payloads/14,399,760 cells and verifies 1,680 measured expected-error checks; preceding successful iterations are not individually decoded. Object independently decodes 100 retained UTF-8 IPC/protobuf payloads/3,000,000 cells, only last successful results. Its 630 surrogate attempts/hook totals remain sealed-worker assertions: printed totals/one error fingerprint, no retained per-iteration errors or independent replay. Object has aggregate stdout timing/RSS, without pre-validator construction-measurement files/nanosecond endpoints. Ordinary rehashes/cross-links all 330 typed-validator receipts/requests/payloads covering 72,149,500 cells plus 45 noninput guards; typed value/null-mask/nested-null/schema validation remains a frozen producer assertion, not fresh independent decoding.
Construction RPC zeros apply to explicit/object sole-loopback fixtures; frozen20 zeros are inferred separately. Ordinary directly observes all 20 counter keys in all 375 sample maps, including nonzero input Config traffic (four calls/512 bytes); all 135 Rust-pair maps match. Return/config traffic exists; there is no overall-zero or full-wire claim.
RSS baseline/held precedes validation; lifetime
ru_maxrssincludes preparation/warmups. Explicit post-validation RSS follows output/mock-holder release/one GC; inputs/expected/schema/session/server and native arenas/internal frames may remain. Ordinary post-GC includes additive validation allocations. Neither proves complete reclamation/construction-only retention. Ordinary external RSS targets 2 ms but records 1–59 observations/largest gap 5.520125 ms; observation starts before go and stops on measured message, not exact timer endpoints; short transients can be missed. Object has no post-GC/sampled-peak/count/gap fields. Signed deltas can be nonpositive; ratios require positive denominators; medians of signed deltas differ from differences of absolute medians.Ordinary harness truncates stderr at 8,000 and exception stdout at 2,000 characters. Object streams/receipts exist before launch; interruption may leave RUNNING receipts/only written bytes. Selected dependency RECORD/PySpark-origin inventories do not establish shadow-import/loaded-dependency/continuous isolation. Inherited-group supervision does not cover arbitrary detach/setuid/host crash/realtime/external-supervisor-SIGKILL guarantees.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
This pull request and its description were written by Isaac.