Skip to content

[SPARK-62037][CONNECT][PYTHON] Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers - #867

Open
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62037-builder-locks-subclasses-reader-pickles

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the documented differences of the session (docs/compatibility.md, dev/parity/known-differences.md) against PySpark 4.3.0. Each item was probed on both clients against a Spark 4.3.0 server, the requests through a recording proxy.

1. Shallow copies of the Connect builder share its options and hooks, as PySpark's plain object shares the _options dict and the _hook_factories list of its __dict__ with its copy.

  • A remote, config or _registerHook of either one is seen by both.
  • A channelBuilder sets the channel builder of the builder it is called on, as it rebinds an attribute of PySpark's.
  • The builder's __traverse__ reports the shared list and its own channel builder. The list reports the factories, so the strong-count rule of SPARK-61983 is no longer needed.

2. PySpark's locks.

  • SparkSession._lock, an RLock, is held for getOrCreate, around the default session of create, and for stop(), where PySpark holds it. The builder's Builder._lock is held for config, remote, appName, channelBuilder and _registerHook, and while the options are applied.
  • They are entered through the lock type's __enter__ and __exit__, as with does: the class attribute is read at each call, and an RLock waits without the GIL and is interrupted by a signal.
  • So a thread that holds SparkSession._lock holds off the drop-in's builders, as PySpark's. The Rust lock that made sessions before is gone.
  • getOrCreate of a new session applies its options again after create, as PySpark's does, so the server gets the Config of its options twice, as from PySpark. create installs the hooks before the session becomes the default, as PySpark's does.

3. SparkSession, Catalog and DataSourceRegistration can be subclassed, as PySpark's plain classes.

  • They take PySpark's __init__ arguments, (connection, userId=None, hook_factories=None), (sparkSession) and (sparkSession), and copy and pickle in PySpark's formats.
  • An object whose __init__ did not run is backed by a closed session, whose requests raise NO_ACTIVE_SESSION and send nothing.

4. PySpark's pickles of a DataFrameReader or a DataStreamReader load in the drop-in, through an argument-less __new__ and a __setstate__ of PySpark's state (_client, _format, _schema, _options, and the stream reader's _source_name). They raised TypeError: missing 'client'.

5. LogicalPlan.to_proto(session, debug=False), as PySpark's: the Plan of the relation of plan(session), which a plugin or a tool calls.

6. Docs.

  • The known differences of the builder copies and of the lock are removed. "Default port" is removed: default_port() is 15002 in both, as pyspark-client is remote-only.
  • "Missing URL error" is now "A channel builder without a host": the two builders are PySpark's, with its errors.
  • The rendering of deep plans is described as it is: _proto_to_string renders past about 125 levels, where PySpark's text_format truncates.
  • New entries:
    • PySpark 4.3.0 (grpcio 1.85.1) hangs at exit for a session that is not the default and is not stopped (SparkSession.__del__ → Channel.close()), where the drop-in exits;
    • DefaultChannelBuilder.port exists only in the drop-in;
    • the extra names of pyspark.sql.connect.client;
    • DataFrame(plan, session) builds its plan when it is made.

Why are the changes needed?

So that code which copies builders, takes SparkSession._lock, subclasses the session, the catalog or the registration, or loads PySpark's pickles of readers, works as with PySpark.

Does this PR introduce any user-facing change?

Yes, as above, including the second Config request of getOrCreate of a new session, as PySpark sends it.

How was this patch tested?

  • New tests in test_dropin_offline.py:
    • test_shallow_copies_of_a_builder_share_its_options_and_hooks_as_pyspark;
    • GC of cycles through builder copies;
    • test_the_session_and_builder_locks_are_held_where_pyspark_holds_them;
    • test_a_thread_that_holds_the_session_lock_holds_off_the_builders_and_stops_as_pyspark;
    • test_sessions_catalogs_and_registrations_subclass_as_pyspark;
    • test_pyspark_pickles_of_readers_catalogs_and_registrations_load_as_pyspark;
    • test_logical_plans_give_the_plans_of_their_relations_of_to_proto_as_pyspark;
    • the new objects in the test of objects traversed while they are created.
  • Probes of both clients against a Spark 4.3.0 server, identical:
    • the builder copies;
    • the locks, their order of enter and exit, reentrancy, a replaced lock, signals;
    • the builder lock;
    • the GC with builder copies;
    • the signatures;
    • the catalog and registration pickles in both directions;
    • the user ids and the Config requests through a recording proxy (2/1/1).
  • PySpark's own reader pickles load with the same state and session.
  • Python. The offline suite of coverage.yml in its order, on pandas 3, on a UTC host without TZ=UTC. Live test_dropin_e2e.py. The type-stub drift check and test.
  • Rust. cargo test -p pyspark-rs --lib, linked as rust.yml links it.
  • Lint. cargo fmt --all --check, cargo +1.100.0 clippy -p pyspark-rs --all-targets -- -D warnings, ruff 0.14.0, audit_no_stubs.sh and scripts/tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…er's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers

- The options of the Connect builder are one shared state of its shallow copies, and its hook
  factories one shared list, as PySpark's __dict__; the channel builder is each builder's own;
  __traverse__ reports the list and the channel builder.
- SparkSession._lock is held for getOrCreate, around the default session of create, and for
  stop, and Builder._lock for config, remote, appName, channelBuilder and _registerHook, as
  PySpark holds them, through their __enter__/__exit__; getOrCreate of a new session applies its
  options again, as PySpark's.
- SparkSession, Catalog and DataSourceRegistration are subclassable, with PySpark's __init__
  and pickle formats.
- DataFrameReader and DataStreamReader load PySpark's pickles of them (argument-less __new__ and
  a __setstate__ of PySpark's state).
- LogicalPlan.to_proto, as PySpark's.
- The docs: the builder copies, the locks, the subclasses, a channel builder without a host, the
  rendering of deep plans, PySpark's hang at exit of a session that is not stopped, the port of
  the channel builder and the extra names of pyspark.sql.connect.client.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62037-builder-locks-subclasses-reader-pickles branch from b6f0b11 to 6f111e2 Compare November 18, 2026 21:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants