Repository navigation
[SPARK-62037][CONNECT][PYTHON] Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers - #867
Open
HyukjinKwon wants to merge 1 commit into
Open
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…er's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers - The options of the Connect builder are one shared state of its shallow copies, and its hook factories one shared list, as PySpark's __dict__; the channel builder is each builder's own; __traverse__ reports the list and the channel builder. - SparkSession._lock is held for getOrCreate, around the default session of create, and for stop, and Builder._lock for config, remote, appName, channelBuilder and _registerHook, as PySpark holds them, through their __enter__/__exit__; getOrCreate of a new session applies its options again, as PySpark's. - SparkSession, Catalog and DataSourceRegistration are subclassable, with PySpark's __init__ and pickle formats. - DataFrameReader and DataStreamReader load PySpark's pickles of them (argument-less __new__ and a __setstate__ of PySpark's state). - LogicalPlan.to_proto, as PySpark's. - The docs: the builder copies, the locks, the subclasses, a channel builder without a host, the rendering of deep plans, PySpark's hang at exit of a session that is not stopped, the port of the channel builder and the extra names of pyspark.sql.connect.client. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62037-builder-locks-subclasses-reader-pickles
branch
from
November 18, 2026 21:15
b6f0b11 to
6f111e2
Compare
dongjoon-hyun
approved these changes
Nov 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the documented differences of the session (
docs/compatibility.md,dev/parity/known-differences.md) against PySpark 4.3.0. Each item was probed on both clients against a Spark 4.3.0 server, the requests through a recording proxy.1. Shallow copies of the Connect builder share its options and hooks, as PySpark's plain object shares the
_optionsdict and the_hook_factorieslist of its__dict__with its copy.remote,configor_registerHookof either one is seen by both.channelBuildersets the channel builder of the builder it is called on, as it rebinds an attribute of PySpark's.__traverse__reports the shared list and its own channel builder. The list reports the factories, so the strong-count rule of SPARK-61983 is no longer needed.2. PySpark's locks.
SparkSession._lock, an RLock, is held forgetOrCreate, around the default session ofcreate, and forstop(), where PySpark holds it. The builder'sBuilder._lockis held forconfig,remote,appName,channelBuilderand_registerHook, and while the options are applied.__enter__and__exit__, aswithdoes: the class attribute is read at each call, and an RLock waits without the GIL and is interrupted by a signal.SparkSession._lockholds off the drop-in's builders, as PySpark's. The Rust lock that made sessions before is gone.getOrCreateof a new session applies its options again aftercreate, as PySpark's does, so the server gets the Config of its options twice, as from PySpark.createinstalls the hooks before the session becomes the default, as PySpark's does.3. SparkSession, Catalog and DataSourceRegistration can be subclassed, as PySpark's plain classes.
__init__arguments,(connection, userId=None, hook_factories=None),(sparkSession)and(sparkSession), and copy and pickle in PySpark's formats.__init__did not run is backed by a closed session, whose requests raiseNO_ACTIVE_SESSIONand send nothing.4. PySpark's pickles of a
DataFrameReaderor aDataStreamReaderload in the drop-in, through an argument-less__new__and a__setstate__of PySpark's state (_client,_format,_schema,_options, and the stream reader's_source_name). They raisedTypeError: missing 'client'.5.
LogicalPlan.to_proto(session, debug=False), as PySpark's: thePlanof the relation ofplan(session), which a plugin or a tool calls.6. Docs.
default_port()is 15002 in both, as pyspark-client is remote-only._proto_to_stringrenders past about 125 levels, where PySpark'stext_formattruncates.SparkSession.__del__→Channel.close()), where the drop-in exits;DefaultChannelBuilder.portexists only in the drop-in;pyspark.sql.connect.client;DataFrame(plan, session)builds its plan when it is made.Why are the changes needed?
So that code which copies builders, takes
SparkSession._lock, subclasses the session, the catalog or the registration, or loads PySpark's pickles of readers, works as with PySpark.Does this PR introduce any user-facing change?
Yes, as above, including the second Config request of
getOrCreateof a new session, as PySpark sends it.How was this patch tested?
test_dropin_offline.py:test_shallow_copies_of_a_builder_share_its_options_and_hooks_as_pyspark;test_the_session_and_builder_locks_are_held_where_pyspark_holds_them;test_a_thread_that_holds_the_session_lock_holds_off_the_builders_and_stops_as_pyspark;test_sessions_catalogs_and_registrations_subclass_as_pyspark;test_pyspark_pickles_of_readers_catalogs_and_registrations_load_as_pyspark;test_logical_plans_give_the_plans_of_their_relations_of_to_proto_as_pyspark;coverage.ymlin its order, on pandas 3, on a UTC host withoutTZ=UTC. Livetest_dropin_e2e.py. The type-stub drift check and test.cargo test -p pyspark-rs --lib, linked asrust.ymllinks it.cargo fmt --all --check,cargo +1.100.0 clippy -p pyspark-rs --all-targets -- -D warnings, ruff 0.14.0,audit_no_stubs.shandscripts/tests.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.