Skip to content

fix: allow Rust consumers to link datafusion-python - #1722

Open
emecii wants to merge 1 commit into
apache:mainfrom
emecii:fix/1703-rust-link-feature
Open

fix: allow Rust consumers to link datafusion-python#1722
emecii wants to merge 1 commit into
apache:mainfrom
emecii:fix/1703-rust-link-feature

Conversation

@emecii

@emecii emecii commented Sep 10, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #1703.

Rationale for this change

Unconditionally enabling pyo3/extension-module prevents ordinary Rust executables from linking the core rlib against libpython. It also prevents CI from executing useful Rust tests.

What changes are included in this PR?

  • Expose a default-on extension-module crate feature and gate the extension-only build-script linker arguments with it.
  • Explicitly enable that feature for all wheel builds, including free-threaded builds that disable defaults to avoid ABI3.
  • Add a separate integration-test executable that constructs PySessionContext from Rust, initializes Python, executes SQL through the bindings, and asserts a result. Run core Rust tests in Linux CI with defaults disabled and Substrait enabled.
  • Document the test/dependency configuration and the workspace feature-unification caveat.

Are there any user-facing changes?

Rust executables can depend on datafusion-python with default-features = false and opt into features such as substrait. Default Python extension builds retain ABI3 and mimalloc; Python APIs are unchanged.

Validation

Local macOS arm64 validation:

  • The new rust_link executable fails before the fix with unresolved Py_* symbols and passes after it. Both the minimal no-default-feature command and cargo test --locked -p datafusion-python --no-default-features --features substrait execute and pass the test.
  • All-target Cargo checks with defaults and without defaults; no-default-feature Clippy with -D warnings; all-target/all-feature Clippy through pre-commit.
  • Built and installed the ABI3 wheel with maturin 1.13.3. SQL/Arrow smoke check and the complete Python tests/doctests pass: 1,387 passed, 7 skipped.
  • Built and installed the CPython 3.14t wheel with defaults disabled and extension-module,mimalloc,substrait enabled. SQL/Arrow smoke check passes and the GIL remains disabled after import and execution.
  • The composite action's actual argument script passes 54 combinations plus GIL-interpreter rejection in its Windows branch (branch logic exercised locally, not a Windows-host build).
  • Nightly rustfmt, Taplo, native Actionlint 1.7.12, and the remaining full pre-commit hooks pass. Docker is not installed locally, so the Docker Actionlint hook was replaced by the same pinned native version.

Generated with OpenAI Codex.

Gate PyO3 extension-module behind a default-on crate feature, retain it in wheel builds, and run a separate executable linking regression in CI.

Generated-by: OpenAI Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate pyo3/extension-module so the crate can be linked as a Rust dependency and tested

1 participant