fix: keep the dictionary hash fast path off nested and reseeded buffers

[viirya]

Which issue does this PR close?

Closes #5756.

Rationale for this change

The dictionary fast path hashes each distinct dictionary value once and reuses that result
for every key. It was selected by the column's position and restarted from a hardcoded seed:

let first_col = i == 0;
...
if !first_col { /* unpack and recurse */ } else {
    let mut dict_hashes = vec![42; dict_values.len()];

Two things are wrong with that:

• create_hashes_internal! also runs on recursion, so a dictionary nested in a list,
  struct or map arrives as the only column of its call and looks like a first column, even
  though the buffer already holds the hash accumulated for earlier elements of that row. That
  hash was discarded, so a dictionary-encoded list element hashed differently from the
  identical decoded value.
• The hardcoded 42 is wrong whenever the caller supplies its own seed, which
  hash(col, seed) and xxhash64(col, seed) allow, so even a genuine first column disagreed
  with its decoded form for a non-default seed.

For a shuffle partitioning key that means equal keys can reach different partitions, breaking
grouping and joins; the hash() and xxhash64() SQL functions are affected too.

What changes are included in this PR?

• The reuse is valid exactly when every row carries the same incoming hash, so that is what
  is checked now, rather than the column index.
• The per-value hashes start from the seed the buffer actually holds instead of an assumed 42.
• Both changes are in murmur3 and xxhash64, which share this structure.

A top-level dictionary column keeps the optimisation.

On the cost of the check. It is a scan of the hash buffer, and that is not free: running
it for every column measured 17% slower on an int column and 11% on a string column in a
local criterion benchmark. It is therefore done inside the dictionary arm, so only dictionary
columns pay for it. After moving it, the same benchmark is back at the unmodified timings.

How are these changes tested?

cargo test -p datafusion-comet-spark-expr passes 713 + 5 tests.

Three new tests, all failing without the change:

• a dictionary as a list element, compared against the decoded array — hashes 3853467749
  instead of 1401423033 before the fix
• the same for xxhash64
• a top-level dictionary column for both seed 42 and seed 7, which pins that the fast path
  survives and that a non-default seed is handled

Additional context

Found while reviewing nested hash partitioning keys (#5567), which makes this reachable from
shuffle partitioning, but the defect predates it and reproduces on main unchanged.

Worth recording how it was found, since it says something about the tests: comparing a batched
hash against a per-row hash cannot catch this, because both sides run the same faulty branch.
It took an independent leaf-by-leaf chaining oracle to surface it.

[sunchao]

Correctness

Reviewed head b53ebc16bd5503c3b197949cde9c2696a3101a87 against base bc74cc79fcf5eeed9610d9a20937375652cb74df. I found no P1/P2 issue in the three changed files.

The old dictionary path used column position to select hash reuse and restarted each dictionary value at 42. A nested list element re-enters the hasher as column zero with an accumulated row hash, so this could discard earlier elements. It also ignored a non-default Catalyst expression seed. The change checks the incoming buffer for uniformity and initializes dictionary hashes from that actual value. A nonuniform buffer uses the existing decoded path and preserves each row's seed.

This matches the relevant hash semantics in maintained Spark 3.5 (5947fd6e) and 4.0 (03f28fc4): null leaves the incoming hash unchanged, and arrays, structs and supported map paths chain the preceding hash into the next value. I traced Comet's list/map recursion, struct recursion, expression seed serialization and shuffle callers. Null dictionary keys preserve the row seed, and a key referencing a null value produces that same seed. The retained first-column condition is conservative and does not make reuse unsafe. This change does not alter type support, hash arithmetic or overflow behavior. Maintained Spark 3.4 and 4.1 branches were unavailable, so this is not a source compatibility claim for those versions.

Native CI checked out merge 5855232c2963df7d1d27d0bea99484d84ec8df43, whose raw parents are the exact base and head above. All 1,184 native tests passed, with four skipped. The five added regressions passed, covering nested dictionary elements in both algorithms, nonuniform seeds with null keys and null values in both algorithms, and Murmur3 reuse with seeds 42 and 7. The changed hash files, callers and lockfile match the reviewed head. This is CI merge execution, not a local rerun. The full snapshot at 2026-09-08 02:38:13.960 UTC had 65 successful checks and nine skipped checks.

Performance

The extra buffer scan runs only for a dictionary in the first position of its current call. Primitive and string columns do not pay it, and later columns short-circuit it. Uniform buffers still hash each dictionary value once. Nonuniform buffers use the existing allocation and decoding fallback because one cached hash per value cannot represent different incoming seeds. The scan adds linear work to eligible dictionary calls, but does not change their asymptotic cost. The author's Criterion comparison is reported evidence only, and I did not independently reproduce its timings or infer an end-to-end speedup.

Design

The fix puts the reuse condition beside dictionary dispatch and keeps the actual seed in the two algorithm-specific helpers. It handles recursive calls without adding a separate top-level/recursive mode or changing callers. A single-row recursive buffer remains eligible, which is safe because the helper now uses its accumulated hash. Existing fallback behavior also handles rows that already diverged after a preceding field. I found no design change needed before merge.

Abstraction & complexity

The shared macro applies the same guard to Murmur3 and xxHash64 across the existing dictionary key types. No new abstraction, persistent state or public API is introduced. The five focused tests separately exercise seed preservation and the nonuniform-buffer fallback, keeping the added complexity proportional to the two failure modes.

[viirya]

Thanks @sunchao !