Repository navigation
[SPARK-62030][CONNECT][PYTHON] Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr - #864
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr - proxy_for parses the proxy URI first, logs a value that is not a proxy for every host, as grpcio's core logs it of each channel, and fails an invalid address only for a host that the proxy applies to: loopback and no_proxy/no_grpc_proxy hosts connect directly. - A plaintext connection's stream (ServerStream, of a DirectConnector and of the proxy tunnel) fails a read with "connection attempt timed out before receiving SETTINGS frame" once the connect deadline passes before the server's first frame, so a silent server fails the call with UNAVAILABLE within the deadline, where it waited for the keepalive (160 s of a first plan). - grpc_log: the errors that grpcio's C core writes to stderr (a proxy URI that is not a proxy, a TLS roots file that cannot be read) go to a listener that the Python binding sets, which writes grpcio's line format to stderr unless GRPC_VERBOSITY is NONE; the roots file is read once per process, as grpcio reads it. - The docs give the reasons of the user agent's read time, the address racing and the TLS roots, the remaining fail-fast difference after a failed connection, and N6. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Nov 18, 2026
…4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface ### What changes were proposed in this pull request? From a review of the Rust API's documentation (items S1 and S7 of the review of the Rust migration notes). **1. The Rust migration notes are split by release line.** - `## Rust migration for the 4.3 line` mixed what v4.3.0 shipped (tagged b2656b6, 2026-11-16) with the Rust-API changes merged after it, which ship in 4.4.0. - The section is now the changes from v4.2.0 to v4.3.0, at v4.3.0's meaning, with an intro line. A diff against `git show v4.3.0:docs/compatibility.md` differs only in that line and in the corrections of #846 (SPARK-61996): the typed map keys and the new `Value`, `DataType`, `SparkErrorKind` and `LogicalPlan` variants, `Pipeline::set_stages`/`stages`, `PipelineModel`'s fields, `Raw` and `MAX_PLAN_DEPTH`, and the seed. Each was checked against the code at v4.3.0, not only the docs. - A sentence true of v4.3.0 and changed since stays there, such as "`bucket_by` is unchanged and sends the count it is given". The new section says what changed. - A new `## Rust migration for the 4.4 line (unreleased)` holds every Rust-API change since v4.3.0: - the passages that were there, moved as they were (#781, #792, #799, #800, #831, #837, #843, #845, #846); - those that the notes lacked, from the merged PRs and checked against the code: the new error kinds (`BuiltinAttributeError`, `BuiltinAssertionError`), the hidden expression nodes (`PlanBuildOrigin`, `CastTarget::Shown`), the ML changes (#784, #787, #793), the client and connection settings (#769, #779, #802, #815, #842, #864), and the catalog, reader and `create_dataframe` changes (#776, #782, #788, #798, #836, #855); - a sentence on the doc-hidden items. - The 4.3 heading is unchanged, so its anchor keeps working; no link pointed at it. **2. The claim that the native Rust API mirrors PySpark's surface is qualified** (`docs/compatibility.md`, `docs/index.md`). - After #845 and #846, a comparison of the public names of PySpark's classes with the crate's public API leaves only the APIs that need Python itself: `toPandas`, `pandas_api`, `plot`, `GroupedData.apply`, `Catalog.registerFunction`, `udf.register`, `udtf.register`, `dataSource.register`, `F.udf`/`udtf`/`arrow_udtf`, `SparkSession.Hook`, `active`/`getActiveSession`, and `SparkSession.client`. - These are of Python functions that the server runs, of pandas, or of Python's implicit session. The paragraph names each, with its Rust counterpart where there is one: the pickled commands of `register_function`, `register_table_function`, `register_data_source` and `apply_in_pandas`, and `set_execute_plan_hook`. ### Why are the changes needed? So that a Rust user upgrading from 4.2 to 4.3 reads what 4.3 changed, a user of the next release reads what 4.4 changes, and the docs do not overstate the Rust API's surface. ### Does this PR introduce _any_ user-facing change? No, docs only. ### How was this patch tested? - Every sentence of the old section appears verbatim in one of the two new sections, apart from three that were edited on purpose: an intro line, an antecedent, and a split paragraph. No sentence is in both. This was checked by script. - Every link and anchor of `docs/*.md` resolves (checked with Python-Markdown's toc slugs). `mkdocs build --strict` runs in CI's "Validate docs" job; mkdocs is not installed here. - `scripts/check_doc_snippets.py`: 60 snippets, 0 failed. - `python3.11 -m unittest discover -s scripts/tests`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. Closes #868 from HyukjinKwon/SPARK-62039-rust-migration-4.3-4.4. Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
From a review of the documented differences of the transport against PySpark 4.3.0 (grpcio 1.85.1). Each item was probed on both clients, with silent sockets, proxies and TLS roots files.
1. An invalid proxy address does not fail the hosts that the proxy does not apply to.
grpc_proxy(such ashttp://localhost:80837,p:notaport,[::1) and a host thatno_proxyorno_grpc_proxyexcludes, grpcio connects directly, where the drop-in failedcreate()withInvalid HTTP proxy address.ProxyEnv::proxy_fornow parses the proxy URI first, and fails an invalid address only for a host that the proxy applies to. Loopback andno_proxyhosts connect directly.create(), where grpcio makes the session and fails its first request with UNAVAILABLE; this is documented with its reason.2. A server that sends no HTTP/2 settings fails within the connect deadline.
ServerStream, of a newDirectConnectorand of the proxy tunnel) fails a read once the connect deadline, 20 s from the connector call, passes before the server's first frame has arrived, with grpcio's details:connection attempt timed out before receiving SETTINGS frame. The call fails with UNAVAILABLE.3. grpcio's errors of the proxy and the TLS roots reach stderr, as grpcio's C core writes them, unless
GRPC_VERBOSITYisNONE.E1118 16:53:41.241267 4026384 http_proxy_mapper.cc:159] 'https' scheme not supported in proxy URIof each channel. A TLS roots file that cannot be read getserror loading file, once per process.tracing, which the Python client never shows.grpc_logmodule of the core gives them to a listener that the Python binding sets at import, asretries::set_listenerdoes. The binding writes grpcio's line format to stderr, of the Python thread of the call.GRPC_DEFAULT_SSL_ROOTS_FILE_PATHand its file are now read once per process, as grpcio reads them, and an empty file is silent, as in grpcio.4. Docs.
docs/configuration.md: the invalid proxy with its reason, non-UTF-8 values, the TLS roots read once (and the reason for the system store: an empty store fails), and a new "Errors on stderr" section.docs/compatibility.md: the proxy and connect-deadline paragraphs, the user agent's read time with its reason, and the SETTINGS details.dev/parity/known-differences.md: the remaining fail-fast difference. N6: the CONNECT request has noUser-Agent, where grpcio sendsgrpc-httpcli/0.0, and PySpark's debug logs ofSPARK_CONNECT_LOG_LEVEL=debug(RPC deadlines, "Given up on retrying", "GRPC Error received").Why are the changes needed?
So that a bad proxy variable does not break the hosts it does not apply to, so that a silent server fails in tens of seconds rather than minutes, and so that grpcio's diagnostics are visible, as with PySpark.
Does this PR introduce any user-facing change?
Yes, as above. In the Rust API:
grpc_logmodule (a listener of these errors, not a stable API);How was this patch tested?
crates/spark-connect-core/tests/proxy.rs: an invalid proxy withno_proxyandno_grpc_proxyhosts connects directly; a silent server, directly and in a stalled tunnel, fails with the SETTINGS text after at least 15 s and within 40 s; servers that send their SETTINGS at 12 s, or only after the client's preface, connect.proxy.rs,client.rsandgrpc_log.rs.test_dropin_offline.py: the format and the thread of the stderr line, one line per connected session, and silence withGRPC_VERBOSITY=nonein a fresh process.no_proxy);GRPC_VERBOSITY;cargo test -p apache-spark-connect -p apache-spark-connect-core --release(1713 passed), and--features wasm-udf(1395), and in a debug build (1713);cargo test -p pyspark-rs --lib, linked asrust.ymllinks it. Live: the coreintegration_test,e2e_integration,e2e_dataframe.coverage.ymlin its order, on pandas 3 and a UTC host withoutTZ=UTC(6401 passed). Livetest_dropin_e2e.py.cargo fmt --all --check,cargo +1.100.0 clippy --all-targets -- -D warnings, ruff 0.14.0,audit_no_stubs.sh,scripts/testsandRUSTDOCFLAGS="-D warnings" cargo doc --no-depswith wasm-udf.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.