Skip to content

[SPARK-62030][CONNECT][PYTHON] Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr - #864

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62030-proxy-settings-deadline-grpc-log

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

From a review of the documented differences of the transport against PySpark 4.3.0 (grpcio 1.85.1). Each item was probed on both clients, with silent sockets, proxies and TLS roots files.

1. An invalid proxy address does not fail the hosts that the proxy does not apply to.

  • With an invalid grpc_proxy (such as http://localhost:80837, p:notaport, [::1) and a host that no_proxy or no_grpc_proxy excludes, grpcio connects directly, where the drop-in failed create() with Invalid HTTP proxy address.
  • ProxyEnv::proxy_for now parses the proxy URI first, and fails an invalid address only for a host that the proxy applies to. Loopback and no_proxy hosts connect directly.
  • For the other hosts, the drop-in keeps its error at create(), where grpcio makes the session and fails its first request with UNAVAILABLE; this is documented with its reason.

2. A server that sends no HTTP/2 settings fails within the connect deadline.

  • Against a server that accepts the connection and never answers, PySpark fails at 20.3 s: grpcio waits for the server's SETTINGS within its connect deadline. The drop-in waited for the keepalive, 80 s, and its first plan, the lookup of the plan compression and then the execution, 160 s.
  • A plaintext connection's stream (ServerStream, of a new DirectConnector and of the proxy tunnel) fails a read once the connect deadline, 20 s from the connector call, passes before the server's first frame has arrived, with grpcio's details: connection attempt timed out before receiving SETTINGS frame. The call fails with UNAVAILABLE.
  • The stream does not peek before the client's preface: a server that waits for the preface, or sends its SETTINGS late within the deadline, still connects (tests at 12 s, and after the preface).
  • What remains, documented with its reason: grpcio keeps a channel failed until a connection succeeds, so its later calls fail at once, and its first plan at 20.3 s. The drop-in reconnects for each call, so its first plan fails at 40 s. TLS connections have no such bound after their handshake, as tonic does TLS above the connector.

3. grpcio's errors of the proxy and the TLS roots reach stderr, as grpcio's C core writes them, unless GRPC_VERBOSITY is NONE.

  • A proxy URI that is not a proxy: grpcio writes E1118 16:53:41.241267 4026384 http_proxy_mapper.cc:159] 'https' scheme not supported in proxy URI of each channel. A TLS roots file that cannot be read gets error loading file, once per process.
  • The drop-in sent them to Rust's tracing, which the Python client never shows.
  • A new grpc_log module of the core gives them to a listener that the Python binding sets at import, as retries::set_listener does. The binding writes grpcio's line format to stderr, of the Python thread of the call.
  • GRPC_DEFAULT_SSL_ROOTS_FILE_PATH and its file are now read once per process, as grpcio reads them, and an empty file is silent, as in grpcio.

4. Docs.

  • docs/configuration.md: the invalid proxy with its reason, non-UTF-8 values, the TLS roots read once (and the reason for the system store: an empty store fails), and a new "Errors on stderr" section.
  • docs/compatibility.md: the proxy and connect-deadline paragraphs, the user agent's read time with its reason, and the SETTINGS details.
  • dev/parity/known-differences.md: the remaining fail-fast difference. N6: the CONNECT request has no User-Agent, where grpcio sends grpc-httpcli/0.0, and PySpark's debug logs of SPARK_CONNECT_LOG_LEVEL=debug (RPC deadlines, "Given up on retrying", "GRPC Error received").

Why are the changes needed?

So that a bad proxy variable does not break the hosts it does not apply to, so that a silent server fails in tens of seconds rather than minutes, and so that grpcio's diagnostics are visible, as with PySpark.

Does this PR introduce any user-facing change?

Yes, as above. In the Rust API:

  • a new grpc_log module (a listener of these errors, not a stable API);
  • the roots file is read once per process;
  • a silent plaintext server fails its connection at the connect deadline.

How was this patch tested?

  • New tests.
    • crates/spark-connect-core/tests/proxy.rs: an invalid proxy with no_proxy and no_grpc_proxy hosts connects directly; a silent server, directly and in a stalled tunnel, fails with the SETTINGS text after at least 15 s and within 40 s; servers that send their SETTINGS at 12 s, or only after the client's preface, connect.
    • Unit tests in proxy.rs, client.rs and grpc_log.rs.
    • test_dropin_offline.py: the format and the thread of the stderr line, one line per connected session, and silence with GRPC_VERBOSITY=none in a fresh process.
  • Probes of both clients:
    • the proxy matrix (seven invalid formats, with and without no_proxy);
    • silent sockets, direct and through a tunnel: grpcio 20.3 s / 20.0 s, the drop-in 20.0 s for a call, and 40 s for a first plan, against 160 s before;
    • the stderr lines, their counts per channel and GRPC_VERBOSITY;
    • TLS roots files.
  • Rust. cargo test -p apache-spark-connect -p apache-spark-connect-core --release (1713 passed), and --features wasm-udf (1395), and in a debug build (1713); cargo test -p pyspark-rs --lib, linked as rust.yml links it. Live: the core integration_test, e2e_integration, e2e_dataframe.
  • Python. The offline suite of coverage.yml in its order, on pandas 3 and a UTC host without TZ=UTC (6401 passed). Live test_dropin_e2e.py.
  • Lint. cargo fmt --all --check, cargo +1.100.0 clippy --all-targets -- -D warnings, ruff 0.14.0, audit_no_stubs.sh, scripts/tests and RUSTDOCFLAGS="-D warnings" cargo doc --no-deps with wasm-udf.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr

- proxy_for parses the proxy URI first, logs a value that is not a proxy for every host, as
  grpcio's core logs it of each channel, and fails an invalid address only for a host that the
  proxy applies to: loopback and no_proxy/no_grpc_proxy hosts connect directly.
- A plaintext connection's stream (ServerStream, of a DirectConnector and of the proxy tunnel)
  fails a read with "connection attempt timed out before receiving SETTINGS frame" once the
  connect deadline passes before the server's first frame, so a silent server fails the call
  with UNAVAILABLE within the deadline, where it waited for the keepalive (160 s of a first plan).
- grpc_log: the errors that grpcio's C core writes to stderr (a proxy URI that is not a proxy, a
  TLS roots file that cannot be read) go to a listener that the Python binding sets, which writes
  grpcio's line format to stderr unless GRPC_VERBOSITY is NONE; the roots file is read once per
  process, as grpcio reads it.
- The docs give the reasons of the user agent's read time, the address racing and the TLS roots,
  the remaining fail-fast difference after a failed connection, and N6.

Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon added a commit that referenced this pull request Nov 18, 2026
…4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface

### What changes were proposed in this pull request?

From a review of the Rust API's documentation (items S1 and S7 of the review of the Rust migration notes).

**1. The Rust migration notes are split by release line.**
- `## Rust migration for the 4.3 line` mixed what v4.3.0 shipped (tagged b2656b6, 2026-11-16) with the Rust-API changes merged after it, which ship in 4.4.0.
- The section is now the changes from v4.2.0 to v4.3.0, at v4.3.0's meaning, with an intro line. A diff against `git show v4.3.0:docs/compatibility.md` differs only in that line and in the corrections of #846 (SPARK-61996): the typed map keys and the new `Value`, `DataType`, `SparkErrorKind` and `LogicalPlan` variants, `Pipeline::set_stages`/`stages`, `PipelineModel`'s fields, `Raw` and `MAX_PLAN_DEPTH`, and the seed. Each was checked against the code at v4.3.0, not only the docs.
- A sentence true of v4.3.0 and changed since stays there, such as "`bucket_by` is unchanged and sends the count it is given". The new section says what changed.
- A new `## Rust migration for the 4.4 line (unreleased)` holds every Rust-API change since v4.3.0:
  - the passages that were there, moved as they were (#781, #792, #799, #800, #831, #837, #843, #845, #846);
  - those that the notes lacked, from the merged PRs and checked against the code: the new error kinds (`BuiltinAttributeError`, `BuiltinAssertionError`), the hidden expression nodes (`PlanBuildOrigin`, `CastTarget::Shown`), the ML changes (#784, #787, #793), the client and connection settings (#769, #779, #802, #815, #842, #864), and the catalog, reader and `create_dataframe` changes (#776, #782, #788, #798, #836, #855);
  - a sentence on the doc-hidden items.
- The 4.3 heading is unchanged, so its anchor keeps working; no link pointed at it.

**2. The claim that the native Rust API mirrors PySpark's surface is qualified** (`docs/compatibility.md`, `docs/index.md`).
- After #845 and #846, a comparison of the public names of PySpark's classes with the crate's public API leaves only the APIs that need Python itself: `toPandas`, `pandas_api`, `plot`, `GroupedData.apply`, `Catalog.registerFunction`, `udf.register`, `udtf.register`, `dataSource.register`, `F.udf`/`udtf`/`arrow_udtf`, `SparkSession.Hook`, `active`/`getActiveSession`, and `SparkSession.client`.
- These are of Python functions that the server runs, of pandas, or of Python's implicit session. The paragraph names each, with its Rust counterpart where there is one: the pickled commands of `register_function`, `register_table_function`, `register_data_source` and `apply_in_pandas`, and `set_execute_plan_hook`.

### Why are the changes needed?

So that a Rust user upgrading from 4.2 to 4.3 reads what 4.3 changed, a user of the next release reads what 4.4 changes, and the docs do not overstate the Rust API's surface.

### Does this PR introduce _any_ user-facing change?

No, docs only.

### How was this patch tested?

- Every sentence of the old section appears verbatim in one of the two new sections, apart from three that were edited on purpose: an intro line, an antecedent, and a split paragraph. No sentence is in both. This was checked by script.
- Every link and anchor of `docs/*.md` resolves (checked with Python-Markdown's toc slugs). `mkdocs build --strict` runs in CI's "Validate docs" job; mkdocs is not installed here.
- `scripts/check_doc_snippets.py`: 60 snippets, 0 failed.
- `python3.11 -m unittest discover -s scripts/tests`.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

Closes #868 from HyukjinKwon/SPARK-62039-rust-migration-4.3-4.4.

Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant