Skip to content

[SPARK-62023][CONNECT][PYTHON] Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read - #863

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62023-client-as-pyspark

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Four commits under one ticket (jira-SPARK-62023-client-as-pyspark.txt):

1. The client's Python state, one per session, as PySpark's.

PySpark's session has one SparkConnectClient, a plain object. Its thread_local is a threading.local (client/core.py:885). Its global_user_context_extensions and their lock are attributes (:961-962), and its requests serialize the extensions when they are built (:1500-1516). The drop-in's spark.client was a new object once the last reference was gone, with a list and a lock of its own. Its thread_local was a plain object of read-only properties. This commit:

  • Per-session state. The clients of a session share their attributes through one instance dict per session, which a WeakKeyDictionary keyed by the session holds. The session itself is in a slot. So an attribute set on spark.client, its lock and its list are there the next time. A list held before an extension was added is the one that it was added to. A list that the attribute is no longer set to is not read. This fixes the stale-list regression of [SPARK-61941][CONNECT][PYTHON] Take any bytes-like blob in cache_artifact, raise PySpark's errors of the return types of register_udf and register_udtf, make global_user_context_extensions PySpark's list attribute, and document what stays different #815.
  • thread_local is a threading.local subclass, one per session:
    • Its tags and user_context_extensions exist once a thread uses them.
    • They are the set and the list that the requests of the thread carry: a list subclass and a set subclass that hold the Rust transport's shared list and set of them, of their Any objects and strs as given.
    • A change or an assignment is taken at once, through the new SharedTags::replace and SharedUserContextExtensions::replace, and the threads that are given a list or set share it.
    • Deleting them leaves the thread with none.
    • Another attribute is the thread's own.
  • The methods are PySpark's code. The client's add_tag, remove_tag, get_tags, clear_tags, _throw_if_invalid_tag and the extension methods are PySpark's, on thread_local. SparkSession.addTag, removeTag, getTags and clearTags delegate to the client, as PySpark's do. So getTags() is the live set.
  • Threads. pyspark.util.inheritable_thread_target and InheritableThread are PySpark's code of thread_local. The threads of a decorated function share the copies of its attributes.
  • Serialized for each call. The extensions are serialized again before each call that sends requests, so an Any that is changed in place is sent as it is.
    • The session keeps a refresher, an opaque handle that the core keeps (SparkConnectClient::set_user_context_refresher).
    • interrupt::run_interruptible calls it on the calling thread, with the GIL held, before it waits on the server.
    • No Python code runs under a lock of the core.
  • Docs. docs/compatibility.md is corrected ("SparkSession.client"). The get_tags entry of known-differences.md goes, and an entry describes the classes that stay different.

2. The arguments of the registrations and the executions, in PySpark's order.

  • register_udf sets the name and deterministic in the fields of pyspark.sql.connect.proto messages first, as CommonInlineUserDefinedFunction.to_plan_udf does. It then parses and converts the return type, sets eval_type (PythonUDF.to_plan), and pickles last. The runtime raises protobuf's errors of each field: an int is a bool, None is 'NoneType' object cannot be interpreted as an integer, a bool is not an int32, and an out-of-range value is a ValueError. This also covers the kd-udf-session item C3.
  • register_java sets the name, the class name, the return type and then aggregate, of no return type only, as JavaUDF.to_plan does.
  • register_udtf reads __name__, then sets the name and deterministic. The binding's eval_type is converted after the return type and before the pickle (_registerPythonUdtf), so its errors come in PySpark's order. This sits after SPARK-61995's conversion of the return type.
  • spark.udf.register, registerJavaFunction and registerJavaUDAF register through client.register_udf and client.register_java, as PySpark's connect/udf.py does. UDFRegistration._register_udf is removed.
  • to_table, to_pandas and to_table_as_iterator read observations as PySpark's:
    • of an object that is not a dict, such as None or [], none is set;
    • name in observations is evaluated when observed metrics arrive, of its TypeError for None or an int.
  • copyFromLocalToFs runs urlparse of the destination first, then the client's copy_from_local_to_fs. That one makes the checks of PySpark's ArtifactManager and names the artifact as it does, of the TypeError of a destination that is not a str, and uploads through a new client-only binding entry, _copy_from_local_to_fs.
  • Docs. The "SparkSession.client" bullet of docs/compatibility.md describes them.

3. The members that PySpark's tests and libraries read. PySpark's client keeps them as attributes (client/core.py):

  • _session_id is the session ID that the requests read, so a set makes the next response of another server-side session close the client. test_connect_session.py:280 relies on it.
    • The core client keeps the ID that its requests carry (client_session_id), which set_client_session_id changes for it and its clones.
    • The request entry points set it when the client tracks its session, and verify_response checks against it, while session_id() stays the session's, as PySpark's session keeps its own.
    • The requests of the artifacts keep the session's ID, as PySpark's ArtifactManager keeps the ID that the client was built with.
    • A client that does not track its session, the transport-injection stub's, passes the IDs of the reference client's requests through, as the grpcio stub does.
  • _closed set to False opens the session again (reopen).
  • _user_id and _use_reattachable_execute are properties of the session's values.
  • spark.client._builder is a DefaultChannelBuilder of the session's connection string, of its endpoint, user agent and parameters. The core keeps the channel builder of the connection, with the session ID popped for a new session and set for a clone, as PySpark's.
  • _get_operation_statuses sends PySpark's GetStatus request (get_status_proto) under the new RpcDeadlines.get_status deadline.

4. Tests of the members that the offline suite did not reach. The offline suite of coverage.yml did not reach a part of the new code of the first three commits, which lowered the job's Python coverage below its floor of 93% (92.89%). The tests of this commit reach it, each asserting PySpark's behavior:

  • _get_operation_statuses sends PySpark's GetStatusRequest and gives the GetStatusResponse. The mock server of the offline suite answers GetStatus.
  • The copies, deep copies and pickles of the tags and extensions of thread_local are a plain set and a plain list.
  • In a thread that has neither tags nor extensions, the removals and the clear leave it as PySpark's does.
  • Once the session is gone, the tags, the extensions and a global list that a caller kept behave as plain ones.
  • inheritable_thread_target deep-copies the other attributes of thread_local.
  • The observed metrics of the accumulators of the Python workers are not looked up in observations.
  • _closed reads the state that a set gives.

The unused handle parameter of _ThreadTags and _ThreadExtensions is removed.

Why are the changes needed?

So that SparkSession.client is PySpark's:

  • its state is per session, and its thread_local is live;
  • the members that PySpark's tests and libraries read exist, such as _session_id (test_connect_session.py:280) and _builder.endpoint (:259);
  • its registrations and executions raise PySpark's errors in PySpark's order;
  • spark.udf.register raises the client's errors.

Does this PR introduce any user-facing change?

Yes, as pyspark-client 4.3.0, in each row of the two ticket drafts:

  • the attributes, the lock and the lists of spark.client are those of its session;

  • thread_local and its tags and extensions behave as PySpark's;

  • getTags() is the live set;

  • an Any changed in place is sent;

  • the registrations convert their arguments in PySpark's order;

  • deterministic=1 and aggregate=1 are accepted;

  • to_table(plan, None) works;

  • copyFromLocalToFs raises PySpark's errors of a destination that is not a str.

  • _session_id, _user_id, _use_reattachable_execute, spark.client._builder and _get_operation_statuses are PySpark's, and _closed = False reopens.

What stays different, documented:

  • the classes of thread_local and of its set and lists are subclasses, of plain reprs, copies and pickles;
  • a tag that is neither a str nor bytes is refused when it is added;
  • the list that a copy.copy of a client is set to is not sent;
  • a _session_id that is not a str raises when it is set;
  • a reopen after close() or stop() sends the requests, where PySpark's closed channel fails them;
  • _user_id is read-only.

How was this patch tested?

  • New tests:
    • in test_dropin_offline.py, on the mock server; each fails on master:
      • test_the_attributes_of_the_clients_of_a_session_are_its_own_as_pyspark;
      • test_the_thread_local_of_a_client_is_a_threading_local_of_live_tags_and_extensions_as_pyspark;
      • test_client_registrations_convert_their_arguments_in_the_order_of_pyspark;
      • test_copy_from_local_to_fs_raises_pysparks_errors_of_a_destination;
      • test_client_executions_read_their_observations_as_pyspark;
      • test_client_members_that_pysparks_tests_read_are_pysparks;
    • test_client_state_is_one_per_session_and_its_thread_local_a_threading_local_as_pyspark, test_client_registrations_and_executions_take_their_arguments_as_pyspark and test_client_members_that_pysparks_tests_read_as_pyspark in test_dropin_e2e.py, which also pass on pyspark-client 4.3.0;
    • test_user_context_extensions_are_sent_as_pyspark of test_ml_offline.py now expects PySpark's AttributeError in another thread;
    • core tests:
      • a_list_given_to_threads_is_theirs_and_its_changes_are_theirs;
      • a_refresher_is_kept_for_the_client_and_its_clones_and_not_a_new_session;
      • a_set_of_tags_given_to_threads_is_theirs_and_its_changes_are_theirs;
      • a_client_session_id_is_sent_and_checked_while_the_session_id_stays;
      • a_closed_client_reopens_and_the_status_rpc_has_a_deadline;
      • a_client_without_session_tracking_passes_the_session_id_of_each_call_through and a_set_client_session_id_is_that_of_the_calls_and_not_of_the_artifacts_as_pyspark in server_session.rs;
      • the_frames_keep_the_session_id_of_the_session_that_the_client_sets_another_of in artifact_uploads.rs;
    • of the fourth commit, in test_dropin_offline.py:
      • test_client_get_operation_statuses_sends_pysparks_get_status_request;
      • test_the_tags_and_extensions_of_a_thread_copy_and_pickle_as_pysparks_set_and_list;
      • test_the_tags_and_extensions_of_a_thread_that_has_none_are_pysparks;
      • test_the_other_attributes_of_a_thread_local_are_copied_and_deleted_as_pysparks;
      • test_the_thread_local_of_a_client_outlives_its_session_as_pysparks;
      • more cases in test_client_executions_read_their_observations_as_pyspark and test_client_members_that_pysparks_tests_read_are_pysparks.
  • Probes of both clients against a Spark 4.3.0 server give identical output:
    • p_global_stale_min.py, p_client_identity.py;
    • p_reg_order.py, p_reg_order2.py, p_udtf_vals.py, p_reg_cases.py, p_udfreg_deleg.py;
    • the observations and destinations of p_paths_more.py;
    • p_threadlocal.py and p_globalext.py, apart from the class names, which are documented;
    • p_c3_none.py;
    • p_c6_c7_n5.py, apart from the two documented rows: a reopen after close(), and the error time of a _session_id that is not a str.
  • The gates, of the four commits on master 39f523f:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core -p apache-spark-connect-macros -p apache-spark-connect-build --features apache-spark-connect/wasm-udf;
    • in debug, as the coverage job builds them:
      • cargo test -p apache-spark-connect-core: the lib (251 passed) and every integration test (server_session 11, reattach_cleanup 19, proxy 17, deadlines 17, artifact_uploads 7, user_context 4 passed; integration_test 3 ignored, which need a server);
      • cargo test -p apache-spark-connect --lib: 806 passed;
    • cargo test -p pyspark-rs --lib, linked as rust.yml links it: 15 passed;
    • ruff 0.14.0, test_ruff_excludes.py, audit_no_stubs.sh, the stub and dispatch generators, the guards and test_type_stubs_offline.py (12 passed, 3 skipped);
    • the offline suite of coverage.yml: 6421 passed, 77 skipped;
    • the step of the offline suite under coverage of coverage.yml, as it runs it: Python coverage 93.44% (lines 5060 of 5320, branches 1894 of 2122), from 92.91% of the first three commits, where the job measured 92.89% (floor 93.00%);
    • test_dropin_e2e.py against a Spark 4.3.0 server: 329 passed;
    • the steps of pipelines-parity.yml against a Spark 4.3.0 server: the official pyspark/pipelines/tests over the transport plugin (78 passed), the pure-Python tests (5 passed) and the official CLI tests through the drop-in (73 passed);
    • the official test_connect_session, test_connect_clone_session, test_connect_error, test_parity_errors and test_connect_basic over the transport plugin (run_official_tests.py --only): no unexpected failure (30 passed, the rest skipped in remote-only mode, as in CI).

What remains: spark.addArtifacts(Path(...)) and client.add_artifacts(Path(...)) raise PyO3's TypeError where PySpark raises the AttributeError of urlparse (p_paths_more.py). That belongs to the artifact paths of SPARK-61973.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62023-client-as-pyspark branch from 992a7b7 to fdf0b15 Compare November 18, 2026 18:52
…rkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read

The four commits of the PR, squashed for the rebase: the per-session client state and thread_local (C1, C10), the registrations' argument conversions and order (C2, C3), PySpark's _session_id, _closed, _user_id, _use_reattachable_execute, _get_operation_statuses and _builder (C6, C7, N5), and the offline tests that keep the Python coverage above its floor.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62023-client-as-pyspark branch from fdf0b15 to fb07419 Compare November 18, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants