Repository navigation
[SPARK-62023][CONNECT][PYTHON] Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read - #863
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
HyukjinKwon
force-pushed
the
SPARK-62023-client-as-pyspark
branch
from
November 18, 2026 18:52
992a7b7 to
fdf0b15
Compare
dongjoon-hyun
approved these changes
Nov 18, 2026
…rkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read The four commits of the PR, squashed for the rebase: the per-session client state and thread_local (C1, C10), the registrations' argument conversions and order (C2, C3), PySpark's _session_id, _closed, _user_id, _use_reattachable_execute, _get_operation_statuses and _builder (C6, C7, N5), and the offline tests that keep the Python coverage above its floor. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62023-client-as-pyspark
branch
from
November 18, 2026 20:39
fdf0b15 to
fb07419
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Four commits under one ticket (
jira-SPARK-62023-client-as-pyspark.txt):1. The client's Python state, one per session, as PySpark's.
PySpark's session has one
SparkConnectClient, a plain object. Itsthread_localis athreading.local(client/core.py:885). Itsglobal_user_context_extensionsand their lock are attributes (:961-962), and its requests serialize the extensions when they are built (:1500-1516). The drop-in'sspark.clientwas a new object once the last reference was gone, with a list and a lock of its own. Itsthread_localwas a plain object of read-only properties. This commit:WeakKeyDictionarykeyed by the session holds. The session itself is in a slot. So an attribute set onspark.client, its lock and its list are there the next time. A list held before an extension was added is the one that it was added to. A list that the attribute is no longer set to is not read. This fixes the stale-list regression of [SPARK-61941][CONNECT][PYTHON] Take any bytes-like blob in cache_artifact, raise PySpark's errors of the return types of register_udf and register_udtf, make global_user_context_extensions PySpark's list attribute, and document what stays different #815.thread_localis athreading.localsubclass, one per session:tagsanduser_context_extensionsexist once a thread uses them.Anyobjects and strs as given.SharedTags::replaceandSharedUserContextExtensions::replace, and the threads that are given a list or set share it.add_tag,remove_tag,get_tags,clear_tags,_throw_if_invalid_tagand the extension methods are PySpark's, onthread_local.SparkSession.addTag,removeTag,getTagsandclearTagsdelegate to the client, as PySpark's do. SogetTags()is the live set.pyspark.util.inheritable_thread_targetandInheritableThreadare PySpark's code ofthread_local. The threads of a decorated function share the copies of its attributes.Anythat is changed in place is sent as it is.SparkConnectClient::set_user_context_refresher).interrupt::run_interruptiblecalls it on the calling thread, with the GIL held, before it waits on the server.docs/compatibility.mdis corrected ("SparkSession.client"). Theget_tagsentry ofknown-differences.mdgoes, and an entry describes the classes that stay different.2. The arguments of the registrations and the executions, in PySpark's order.
register_udfsets the name anddeterministicin the fields ofpyspark.sql.connect.protomessages first, asCommonInlineUserDefinedFunction.to_plan_udfdoes. It then parses and converts the return type, setseval_type(PythonUDF.to_plan), and pickles last. The runtime raises protobuf's errors of each field: an int is a bool, None is'NoneType' object cannot be interpreted as an integer, a bool is not an int32, and an out-of-range value is aValueError. This also covers the kd-udf-session item C3.register_javasets the name, the class name, the return type and thenaggregate, of no return type only, asJavaUDF.to_plandoes.register_udtfreads__name__, then sets the name anddeterministic. The binding'seval_typeis converted after the return type and before the pickle (_registerPythonUdtf), so its errors come in PySpark's order. This sits after SPARK-61995's conversion of the return type.spark.udf.register,registerJavaFunctionandregisterJavaUDAFregister throughclient.register_udfandclient.register_java, as PySpark'sconnect/udf.pydoes.UDFRegistration._register_udfis removed.to_table,to_pandasandto_table_as_iteratorreadobservationsas PySpark's:[], none is set;name in observationsis evaluated when observed metrics arrive, of itsTypeErrorfor None or an int.copyFromLocalToFsrunsurlparseof the destination first, then the client'scopy_from_local_to_fs. That one makes the checks of PySpark'sArtifactManagerand names the artifact as it does, of theTypeErrorof a destination that is not a str, and uploads through a new client-only binding entry,_copy_from_local_to_fs.docs/compatibility.mddescribes them.3. The members that PySpark's tests and libraries read. PySpark's client keeps them as attributes (
client/core.py):_session_idis the session ID that the requests read, so a set makes the next response of another server-side session close the client.test_connect_session.py:280relies on it.client_session_id), whichset_client_session_idchanges for it and its clones.verify_responsechecks against it, whilesession_id()stays the session's, as PySpark's session keeps its own.ArtifactManagerkeeps the ID that the client was built with._closedset to False opens the session again (reopen)._user_idand_use_reattachable_executeare properties of the session's values.spark.client._builderis aDefaultChannelBuilderof the session's connection string, of its endpoint, user agent and parameters. The core keeps the channel builder of the connection, with the session ID popped for a new session and set for a clone, as PySpark's._get_operation_statusessends PySpark's GetStatus request (get_status_proto) under the newRpcDeadlines.get_statusdeadline.4. Tests of the members that the offline suite did not reach. The offline suite of
coverage.ymldid not reach a part of the new code of the first three commits, which lowered the job's Python coverage below its floor of 93% (92.89%). The tests of this commit reach it, each asserting PySpark's behavior:_get_operation_statusessends PySpark's GetStatusRequest and gives the GetStatusResponse. The mock server of the offline suite answers GetStatus.thread_localare a plain set and a plain list.inheritable_thread_targetdeep-copies the other attributes ofthread_local.observations._closedreads the state that a set gives.The unused
handleparameter of_ThreadTagsand_ThreadExtensionsis removed.Why are the changes needed?
So that
SparkSession.clientis PySpark's:thread_localis live;_session_id(test_connect_session.py:280) and_builder.endpoint(:259);spark.udf.registerraises the client's errors.Does this PR introduce any user-facing change?
Yes, as pyspark-client 4.3.0, in each row of the two ticket drafts:
the attributes, the lock and the lists of
spark.clientare those of its session;thread_localand its tags and extensions behave as PySpark's;getTags()is the live set;an
Anychanged in place is sent;the registrations convert their arguments in PySpark's order;
deterministic=1andaggregate=1are accepted;to_table(plan, None)works;copyFromLocalToFsraises PySpark's errors of a destination that is not a str._session_id,_user_id,_use_reattachable_execute,spark.client._builderand_get_operation_statusesare PySpark's, and_closed = Falsereopens.What stays different, documented:
thread_localand of its set and lists are subclasses, of plain reprs, copies and pickles;copy.copyof a client is set to is not sent;_session_idthat is not a str raises when it is set;close()orstop()sends the requests, where PySpark's closed channel fails them;_user_idis read-only.How was this patch tested?
test_dropin_offline.py, on the mock server; each fails on master:test_the_attributes_of_the_clients_of_a_session_are_its_own_as_pyspark;test_the_thread_local_of_a_client_is_a_threading_local_of_live_tags_and_extensions_as_pyspark;test_client_registrations_convert_their_arguments_in_the_order_of_pyspark;test_copy_from_local_to_fs_raises_pysparks_errors_of_a_destination;test_client_executions_read_their_observations_as_pyspark;test_client_members_that_pysparks_tests_read_are_pysparks;test_client_state_is_one_per_session_and_its_thread_local_a_threading_local_as_pyspark,test_client_registrations_and_executions_take_their_arguments_as_pysparkandtest_client_members_that_pysparks_tests_read_as_pysparkintest_dropin_e2e.py, which also pass on pyspark-client 4.3.0;test_user_context_extensions_are_sent_as_pysparkoftest_ml_offline.pynow expects PySpark'sAttributeErrorin another thread;a_list_given_to_threads_is_theirs_and_its_changes_are_theirs;a_refresher_is_kept_for_the_client_and_its_clones_and_not_a_new_session;a_set_of_tags_given_to_threads_is_theirs_and_its_changes_are_theirs;a_client_session_id_is_sent_and_checked_while_the_session_id_stays;a_closed_client_reopens_and_the_status_rpc_has_a_deadline;a_client_without_session_tracking_passes_the_session_id_of_each_call_throughanda_set_client_session_id_is_that_of_the_calls_and_not_of_the_artifacts_as_pysparkinserver_session.rs;the_frames_keep_the_session_id_of_the_session_that_the_client_sets_another_ofinartifact_uploads.rs;test_dropin_offline.py:test_client_get_operation_statuses_sends_pysparks_get_status_request;test_the_tags_and_extensions_of_a_thread_copy_and_pickle_as_pysparks_set_and_list;test_the_tags_and_extensions_of_a_thread_that_has_none_are_pysparks;test_the_other_attributes_of_a_thread_local_are_copied_and_deleted_as_pysparks;test_the_thread_local_of_a_client_outlives_its_session_as_pysparks;test_client_executions_read_their_observations_as_pysparkandtest_client_members_that_pysparks_tests_read_are_pysparks.p_global_stale_min.py,p_client_identity.py;p_reg_order.py,p_reg_order2.py,p_udtf_vals.py,p_reg_cases.py,p_udfreg_deleg.py;p_paths_more.py;p_threadlocal.pyandp_globalext.py, apart from the class names, which are documented;p_c3_none.py;p_c6_c7_n5.py, apart from the two documented rows: a reopen afterclose(), and the error time of a_session_idthat is not a str.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core -p apache-spark-connect-macros -p apache-spark-connect-build --features apache-spark-connect/wasm-udf;cargo test -p apache-spark-connect-core: the lib (251 passed) and every integration test (server_session11,reattach_cleanup19,proxy17,deadlines17,artifact_uploads7,user_context4 passed;integration_test3 ignored, which need a server);cargo test -p apache-spark-connect --lib: 806 passed;cargo test -p pyspark-rs --lib, linked asrust.ymllinks it: 15 passed;test_ruff_excludes.py,audit_no_stubs.sh, the stub and dispatch generators, the guards andtest_type_stubs_offline.py(12 passed, 3 skipped);coverage.yml: 6421 passed, 77 skipped;coverage.yml, as it runs it: Python coverage 93.44% (lines 5060 of 5320, branches 1894 of 2122), from 92.91% of the first three commits, where the job measured 92.89% (floor 93.00%);test_dropin_e2e.pyagainst a Spark 4.3.0 server: 329 passed;pipelines-parity.ymlagainst a Spark 4.3.0 server: the officialpyspark/pipelines/testsover the transport plugin (78 passed), the pure-Python tests (5 passed) and the official CLI tests through the drop-in (73 passed);test_connect_session,test_connect_clone_session,test_connect_error,test_parity_errorsandtest_connect_basicover the transport plugin (run_official_tests.py --only): no unexpected failure (30 passed, the rest skipped in remote-only mode, as in CI).What remains:
spark.addArtifacts(Path(...))andclient.add_artifacts(Path(...))raise PyO3'sTypeErrorwhere PySpark raises theAttributeErrorofurlparse(p_paths_more.py). That belongs to the artifact paths of SPARK-61973.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.