Repository navigation
[SPARK-62039][CONNECT][DOCS] Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface - #868
Closed
HyukjinKwon wants to merge 1 commit into
Conversation
…4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface - "Rust migration for the 4.3 line" is the changes from v4.2.0 to v4.3.0, at v4.3.0's meaning, with the corrections of SPARK-61996 (the typed map keys and the new variants, the Pipeline stages, PipelineModel's fields, Raw and MAX_PLAN_DEPTH, the seed). - "Rust migration for the 4.4 line (unreleased)" holds every Rust-API change since v4.3.0, moved as it was, and those that the notes lacked (the new error kinds, the hidden expression nodes, the ML, client and connection settings, the catalog and reader changes). - The claim that the native Rust API mirrors PySpark's surface names the APIs that need Python itself, of Python functions, pandas or Python's implicit session, and their Rust counterparts. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the Rust API's documentation (items S1 and S7 of the review of the Rust migration notes).
1. The Rust migration notes are split by release line.
## Rust migration for the 4.3 linemixed what v4.3.0 shipped (tagged b2656b6, 2026-11-16) with the Rust-API changes merged after it, which ship in 4.4.0.git show v4.3.0:docs/compatibility.mddiffers only in that line and in the corrections of [SPARK-61996][CONNECT][RUST] Make the Rust API's replace, repartitions, approx_quantile and stream reader send and check as PySpark's, and add jdbc, the stream xml and PySpark's shortcuts #846 (SPARK-61996): the typed map keys and the newValue,DataType,SparkErrorKindandLogicalPlanvariants,Pipeline::set_stages/stages,PipelineModel's fields,RawandMAX_PLAN_DEPTH, and the seed. Each was checked against the code at v4.3.0, not only the docs.bucket_byis unchanged and sends the count it is given". The new section says what changed.## Rust migration for the 4.4 line (unreleased)holds every Rust-API change since v4.3.0:BuiltinAttributeError,BuiltinAssertionError), the hidden expression nodes (PlanBuildOrigin,CastTarget::Shown), the ML changes ([SPARK-61865][CONNECT][ML] Send and convert the ML params as PySpark does: lists of ints of the type of their first element, ints beyond their type, and the copy of the validator models #784, [SPARK-61873][CONNECT][ML] Fit, load and copy OneVsRest as PySpark does #787, [SPARK-61884][CONNECT][ML] Log the server's warning of an ML fit as PySpark does, and give it to the Rust API's fitted model #793), the client and connection settings ([SPARK-61847][CONNECT][RUST] Bound the TLS handshake of a direct connection by grpcio's 20 s connect deadline, as through a proxy #769, [SPARK-61856][CONNECT][PYTHON] Give SparkSession.client the public API of PySpark Connect's SparkConnectClient, over the Rust transport #779, [SPARK-61915][CONNECT][RUST] Send the user agent in the client type only, give a connection one 20 s deadline, and keep the backoff of no maximum as grpcio and PySpark do #802, [SPARK-61941][CONNECT][PYTHON] Take any bytes-like blob in cache_artifact, raise PySpark's errors of the return types of register_udf and register_udtf, make global_user_context_extensions PySpark's list attribute, and document what stays different #815, [SPARK-61991][CONNECT][PYTHON] Check the session_id of the connection string as PySpark's uuid.UUID(session_id, version=4), with its messages #842, [SPARK-62030][CONNECT][PYTHON] Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr #864), and the catalog, reader andcreate_dataframechanges ([SPARK-61853][CONNECT] Send the schema and the wholetext option of the reads as PySpark's readers do #776, [SPARK-61863][CONNECT][PYTHON] Let other threads and Ctrl-C interrupt the native conversion of createDataFrame, and follow PySpark's pyarrow width errors and the cache threshold of the Rust API #782, [SPARK-61876][CONNECT][RUST] Make the catalog and ML golden tests compare with goldens of PySpark 4.3.0, and send PySpark's requests from the Rust API's create_external_table, variant_set and tuple_union functions #788, [SPARK-61896][CONNECT][RUST] Send PySpark's requests for star column names, the catalog's scalar operations, the removal of checkpoints, columns of no rows and errorifexists #798, [SPARK-61980][CONNECT][PYTHON] Refuse the types that have no proto or Arrow type where PySpark refuses them, and send PySpark's JSON of the schema of local data #836, [SPARK-62012][CONNECT][PYTHON] Send local rows and pandas string columns of PySpark's large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache threshold with pyarrow's nbytes #855);2. The claim that the native Rust API mirrors PySpark's surface is qualified (
docs/compatibility.md,docs/index.md).toPandas,pandas_api,plot,GroupedData.apply,Catalog.registerFunction,udf.register,udtf.register,dataSource.register,F.udf/udtf/arrow_udtf,SparkSession.Hook,active/getActiveSession, andSparkSession.client.register_function,register_table_function,register_data_sourceandapply_in_pandas, andset_execute_plan_hook.Why are the changes needed?
So that a Rust user upgrading from 4.2 to 4.3 reads what 4.3 changed, a user of the next release reads what 4.4 changes, and the docs do not overstate the Rust API's surface.
Does this PR introduce any user-facing change?
No, docs only.
How was this patch tested?
docs/*.mdresolves (checked with Python-Markdown's toc slugs).mkdocs build --strictruns in CI's "Validate docs" job; mkdocs is not installed here.scripts/check_doc_snippets.py: 60 snippets, 0 failed.python3.11 -m unittest discover -s scripts/tests.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.