Skip to content

[SPARK-62039][CONNECT][DOCS] Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface - #868

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62039-rust-migration-4.3-4.4
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62039-rust-migration-4.3-4.4

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the Rust API's documentation (items S1 and S7 of the review of the Rust migration notes).

1. The Rust migration notes are split by release line.

2. The claim that the native Rust API mirrors PySpark's surface is qualified (docs/compatibility.md, docs/index.md).

Why are the changes needed?

So that a Rust user upgrading from 4.2 to 4.3 reads what 4.3 changed, a user of the next release reads what 4.4 changes, and the docs do not overstate the Rust API's surface.

Does this PR introduce any user-facing change?

No, docs only.

How was this patch tested?

  • Every sentence of the old section appears verbatim in one of the two new sections, apart from three that were edited on purpose: an intro line, an antecedent, and a split paragraph. No sentence is in both. This was checked by script.
  • Every link and anchor of docs/*.md resolves (checked with Python-Markdown's toc slugs). mkdocs build --strict runs in CI's "Validate docs" job; mkdocs is not installed here.
  • scripts/check_doc_snippets.py: 60 snippets, 0 failed.
  • python3.11 -m unittest discover -s scripts/tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface

- "Rust migration for the 4.3 line" is the changes from v4.2.0 to v4.3.0, at v4.3.0's meaning,
  with the corrections of SPARK-61996 (the typed map keys and the new variants, the Pipeline
  stages, PipelineModel's fields, Raw and MAX_PLAN_DEPTH, the seed).
- "Rust migration for the 4.4 line (unreleased)" holds every Rust-API change since v4.3.0, moved
  as it was, and those that the notes lacked (the new error kinds, the hidden expression nodes,
  the ML, client and connection settings, the catalog and reader changes).
- The claim that the native Rust API mirrors PySpark's surface names the APIs that need Python
  itself, of Python functions, pandas or Python's implicit session, and their Rust counterparts.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant