Skip to content

[DON'T MERGE][SPARK-60596][CONNECT] Upgrade PyO3 to 0.30.1 - #107

Closed
dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:codex/upgrade-pyo3-0301
Closed

dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:codex/upgrade-pyo3-0301

Conversation

@dbtsai

@dbtsai dbtsai commented Oct 27, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Upgrade PyO3 from 0.28.3 to 0.30.1 in a standalone dependency change:

  • Update the workspace dependency and five PyO3 lockfile entries, including the required Syn 3 dependency.
  • Declare rust-version = "1.85" for pyspark-rs, matching PyO3 0.30.1's published requirement. The workspace default remains 1.82.
  • Keep extension-module and abi3-py39, and remove an outdated version-specific comment.

There are no Arrow handoff, scalar conversion, or Python wrapper implementation changes. This is separate from #103, which remains unchanged for now.

Why are the changes needed?

Track the PyO3 upgrade under SPARK-60596, independently of the Arrow optimization in #103, so it can be reviewed and reverted separately.

Does this PR introduce any user-facing change?

The Python extension's declared minimum Rust version becomes 1.85. Its Python 3.9+ stable-ABI target is unchanged. No Python API changes are intended.

Performance concern: the isolated benchmark reproduces a collect() slowdown with this upgrade. This draft is not ready to merge on performance grounds. The four selected collection workloads were about 15-31% slower in the initial run and 19-30% slower in a separate slot-swapped confirmation. The exact mechanism has not been diagnosed. The two toArrow() workloads did not show a clear timing difference in this sample.

How was this patch tested?

Built both unmodified upstream fbd6af5763bae5cd572ba4438026e675165ba70a and the four-file upgrade offline as optimized release wheels (opt-level=3, abi3-py39) on macOS arm64, using Rust 1.99.1 and maturin 1.15.0.

  • cargo test -p apache-spark-connect --release --lib: 388 passed.
  • cargo test -p apache-spark-connect --release --no-run: all integration tests compiled.
  • cargo test -p apache-spark-connect --release --features wasm-udf --no-run: all integration tests compiled with WASM enabled.
  • cargo fmt --all --check: passed.
  • Existing python/tests/test_dropin_offline.py against the built wheel: 273 passed on Python 3.9 / PyArrow 10, and 273 passed on Python 3.12 / PyArrow 26.
  • scripts/run_official_tests.py --spark <Spark-4.2.0-tree> --remote sc://localhost:15084 --only pyspark.sql.tests.connect.test_parity_collection: 15 passed, 1 skipped, using the candidate extension and a local Spark Connect server. This exercises official PySpark through the Rust transport, not the entire native drop-in API.
  • Public API-name comparison across 12 classes: unchanged from the baseline. The existing SparkSession.Hook gap remains; Column.to_plan is intentionally absent. This is not a claim of complete behavioral/signature parity.
  • cargo clippy --workspace --all-targets: not green. The same 150 error diagnostics, including their source locations, reproduce on unmodified upstream (workspace plus a separate pyspark-rs check to finish collecting baseline errors).

No new tests were added: this change updates the binding dependency without changing application logic, and uses the existing unit, integration, offline Python, and official parity suites.

Isolated benchmark

Used dbtsai/pyspark-connect-benchmark, with the same upstream source before and after except for this PR. Six fresh-process samples per client/case, matching Python 3.12.14, PyArrow 26.0.0, pandas 2.3.3, NumPy 2.5.4 and psutil 7.2.2. These public client actions receive canned localhost gRPC responses; they do not benchmark Spark execution.

Initial run (median time and sampled client-process peak RSS):

Workload Time before / after Change Peak RSS before / after
core-return-long-w4-b1000-collect 12.18 / 15.89 ms +30.5% 137.12 / 139.41 MiB
core-return-string-w4-b1000-collect 22.50 / 26.40 ms +17.3% 144.67 / 146.52 MiB
core-return-string-w4-b10000-collect 21.10 / 25.23 ms +19.6% 146.58 / 147.80 MiB
allocation-first-100000x8-collect 23.36 / 26.90 ms +15.2% 157.56 / 159.23 MiB
core-return-long-w50-b10000-toArrow 29.49 / 29.39 ms -0.4% 268.93 / 270.27 MiB
return-buffer-134217728-toArrow 59.27 / 62.06 ms +4.7% 628.98 / 629.78 MiB

The four collection cases were rerun with baseline/candidate harness slots swapped: slowdowns were respectively 30.2%, 19.1%, 20.6%, and 20.0%. Before/after timing ranges did not overlap for those cases in either run. All 180 samples across the three clients passed validation; Rust request counters matched and source/wheel identities remained unchanged. The runs are not pooled. RSS is sampled resident memory, not allocation counts, and may miss short spikes. This selected benchmark does not establish performance on every workload or platform.

Draft checklist

  • Investigate the reproduced collection slowdown before merging.
  • Run the full official parity gate; only the collection suite was run locally.
  • Validate the declared Rust 1.85 minimum; local builds used 1.99.1.
  • Review CI results and the reproduced upstream Clippy/API gaps.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (application version unavailable).

Separate the PyO3 dependency upgrade from the Arrow export change. Preserve the Python stable ABI and declare the extension crate minimum Rust version required by PyO3 0.30.1.

Generated-by: OpenAI Codex
@dbtsai dbtsai changed the title [SPARK-60596][CONNECT] Upgrade PyO3 to 0.30.1 [DON'T MERGE][SPARK-60596][CONNECT] Upgrade PyO3 to 0.30.1 Oct 27, 2026
@dbtsai dbtsai closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant