Skip to content

[SPARK-60727][CONNECT] Preserve customized pandas double conversion - #280

Closed
dbtsai wants to merge 3 commits into
apache:masterfrom
dbtsai:codex/postmerge-pr-207-ndarray-storage
Closed

dbtsai wants to merge 3 commits into
apache:masterfrom
dbtsai:codex/postmerge-pr-207-ndarray-storage

Conversation

@dbtsai

@dbtsai dbtsai commented Nov 7, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Preserve the existing row conversion of customized pandas float64 input in the fast path added by #207 (SPARK-60727).

  • A DataFrame constructed with copy=False can retain an ndarray subclass inside pandas' concrete NumPy wrapper. The column path skips that subclass's astype hook, which can change values or suppress an existing conversion error. Require exact ndarray storage before using this path.
  • Inspect public schema aliases through their namespace dictionary. A missing alias should decline the optimization without invoking a module's custom __getattr__ before row conversion.

Add three mock-service regression cases for transformed values, conversion errors, and a missing alias with a raising lazy hook. Ordinary ndarray input keeps the fast path; subclasses such as np.memmap take the row path.

Why are the changes needed?

These inputs are customized cases that #207 intended to keep on the existing row path. Both guard issues remain on current master. Official Spark 4.3 also bypasses the ndarray hook; this change preserves the Rust client's prior custom-input behavior rather than claiming an official parity fix.

Does this PR introduce any user-facing change?

It restores the prior values and errors for the affected customized input. Public signatures and inline transport policy are unchanged.

How was this patch tested?

  • Reproduced both issues with pandas 2.3.3 and 3.0.6, NumPy 2.5.4 and PyArrow 26.0.0.
  • Supplemental helper checks pass on both pandas versions, including ordinary ndarray float bits and null masks. These use private copies of native 4.2 schema classes and do not prove current-master native dispatch.
  • Python syntax and git diff --check pass.
  • Genuine local Claude Code reviewed the landed patch and follow-up; the final full review and all findings are recorded in the local review evidence.

Draft: required validation remains pending. The current environment has no usable Cargo/rustc toolchain, matching native extension, or protoc. The new public-API test attempt is blocked during collection; Rust compile/fmt/Clippy, complete native suites, API/official parity, and live Spark 4.3 checks have not passed for this candidate. Historical maintainer results are not claimed as validation of this follow-up.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex; reviewed with installed local Claude Code.

This pull request and its description were written by Isaac.

Keep NumPy subclass storage and missing schema aliases on the existing
row fallback before Arrow encoding. Cover transformed values, conversion
errors, and lazy module hooks.

Co-authored-by: Isaac <no-reply@databricks.com>
@dbtsai
dbtsai marked this pull request as ready for review November 7, 2026 05:24
HyukjinKwon and others added 2 commits November 7, 2026 16:53
…declines

Since SPARK-61079 such frames convert as PySpark's Arrow conversion does, which reads the configs and the raw storage of a NumPy subclass without calling its astype, rather than the row path.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants