You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
make_date can throw on unconsumed rows through PySpark and raw row iterators #7616
Partial consumers of raw QueryExecution.toRdd or PySpark DataFrame.rdd can encounter a make_date error from a later row that Spark would never evaluate. A native projection can evaluate that row as part of a batch before the consumer stops.
The existing Scala Dataset.rdd safeguards do not cover all of these entry points. The gap already exposes invalid-calendar errors with ANSI enabled. The Date32 fix in #7343 also exposes it with ANSI disabled: an overflowing valid date now raises ARITHMETIC_OVERFLOW instead of returning NULL.
Steps to reproduce
For the ANSI-off Date32 case, use #7343 at 8905cda0a or a revision containing its kernel fix. Use the regression shape from the earlier PySpark bridge test:
With Comet disabled, write a single-partition Parquet input with 4,096 rows. Set integer y=2024 on the first 4,095 rows and y=6000000 on the final row; retain id.
Set ANSI and AQE off and spark.comet.batchSize=8192.
Read the Parquet input and project:
unix_date(make_date(y, 1, 1)) AS d,
repeat(sha2(CAST(id AS STRING), 512), 512) AS padding
Consume only the first serialized batch through the JVM javaToPython().take(1) bridge used by PySpark. Compare Comet enabled and disabled. The large padding makes Python serialization emit a batch before reaching the overflowing final row; a two-row example can instead cause Spark to consume both rows.
Cover both the uncapped DataFrame and limit(4096), including when the physical plan is prepared before the RDD action. Add corresponding raw QueryExecution.toRdd and public PySpark partial-consumer coverage.
The integer unix_date output avoids unrelated Python date-range limits. The earlier JVM bridge regression provides the complete test setup.
Expected behavior
Match Spark when a raw or PySpark consumer stops before a throwing row, including reused/preplanned DataFrames and AQE on/off.
Cover ANSI-on invalid-calendar errors and Date32 overflow in both ANSI modes. Full consumption must still raise the required error.
Preserve existing Scala Dataset safeguards and native collect-limit caps.
Benchmark ordinary root projections, limits above one (including show()), and materializing/write consumers before selecting a broader fallback strategy.
Additional context
Requested in this review of #7343. The broader external-iterator routing was deferred from #7343 to keep the overflow fix focused; this issue tracks that remaining correctness work. It follows the consumer guards introduced in #7283.
Activity
kazuyukitanimura commentedon Nov 16, 2026
Thank you @pingzh
Assigned this to you for now. PySpark is experimental, but this looks important to fix.