Repository navigation
Conversation
Separate the PyO3 dependency upgrade from the Arrow export change. Preserve the Python stable ABI and declare the extension crate minimum Rust version required by PyO3 0.30.1. Generated-by: OpenAI Codex
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Upgrade PyO3 from 0.28.3 to 0.30.1 in a standalone dependency change:
rust-version = "1.85"forpyspark-rs, matching PyO3 0.30.1's published requirement. The workspace default remains 1.82.extension-moduleandabi3-py39, and remove an outdated version-specific comment.There are no Arrow handoff, scalar conversion, or Python wrapper implementation changes. This is separate from #103, which remains unchanged for now.
Why are the changes needed?
Track the PyO3 upgrade under SPARK-60596, independently of the Arrow optimization in #103, so it can be reviewed and reverted separately.
Does this PR introduce any user-facing change?
The Python extension's declared minimum Rust version becomes 1.85. Its Python 3.9+ stable-ABI target is unchanged. No Python API changes are intended.
Performance concern: the isolated benchmark reproduces a
collect()slowdown with this upgrade. This draft is not ready to merge on performance grounds. The four selected collection workloads were about 15-31% slower in the initial run and 19-30% slower in a separate slot-swapped confirmation. The exact mechanism has not been diagnosed. The twotoArrow()workloads did not show a clear timing difference in this sample.How was this patch tested?
Built both unmodified upstream
fbd6af5763bae5cd572ba4438026e675165ba70aand the four-file upgrade offline as optimized release wheels (opt-level=3,abi3-py39) on macOS arm64, using Rust 1.99.1 and maturin 1.15.0.cargo test -p apache-spark-connect --release --lib: 388 passed.cargo test -p apache-spark-connect --release --no-run: all integration tests compiled.cargo test -p apache-spark-connect --release --features wasm-udf --no-run: all integration tests compiled with WASM enabled.cargo fmt --all --check: passed.python/tests/test_dropin_offline.pyagainst the built wheel: 273 passed on Python 3.9 / PyArrow 10, and 273 passed on Python 3.12 / PyArrow 26.scripts/run_official_tests.py --spark <Spark-4.2.0-tree> --remote sc://localhost:15084 --only pyspark.sql.tests.connect.test_parity_collection: 15 passed, 1 skipped, using the candidate extension and a local Spark Connect server. This exercises official PySpark through the Rust transport, not the entire native drop-in API.SparkSession.Hookgap remains;Column.to_planis intentionally absent. This is not a claim of complete behavioral/signature parity.cargo clippy --workspace --all-targets: not green. The same 150 error diagnostics, including their source locations, reproduce on unmodified upstream (workspace plus a separatepyspark-rscheck to finish collecting baseline errors).No new tests were added: this change updates the binding dependency without changing application logic, and uses the existing unit, integration, offline Python, and official parity suites.
Isolated benchmark
Used dbtsai/pyspark-connect-benchmark, with the same upstream source before and after except for this PR. Six fresh-process samples per client/case, matching Python 3.12.14, PyArrow 26.0.0, pandas 2.3.3, NumPy 2.5.4 and psutil 7.2.2. These public client actions receive canned localhost gRPC responses; they do not benchmark Spark execution.
Initial run (median time and sampled client-process peak RSS):
core-return-long-w4-b1000-collectcore-return-string-w4-b1000-collectcore-return-string-w4-b10000-collectallocation-first-100000x8-collectcore-return-long-w50-b10000-toArrowreturn-buffer-134217728-toArrowThe four collection cases were rerun with baseline/candidate harness slots swapped: slowdowns were respectively 30.2%, 19.1%, 20.6%, and 20.0%. Before/after timing ranges did not overlap for those cases in either run. All 180 samples across the three clients passed validation; Rust request counters matched and source/wheel identities remained unchanged. The runs are not pooled. RSS is sampled resident memory, not allocation counts, and may miss short spikes. This selected benchmark does not establish performance on every workload or platform.
Draft checklist
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (application version unavailable).