fix: preserve non-canonical NaN payloads and distinguish -0.0 in v3 metadata

[barlowa124]

Summary

Third item of #4453: float fill values holding non-canonical NaN payloads do not round-trip, and ArrayV3Metadata.__eq__ conflates documents whose fill_values serialize differently.

Two bugs, one in each direction.

Parse to serialize loses NaN payloads. "0x7fc00001" parses correctly, but float_to_json_v3 flattened every NaN to "NaN" on write, so the document came back with the canonical payload. float_to_json_v3 now emits the hex bit pattern for any NaN that is not the canonical positive NaN (payload or sign differing), which is the case the spec's hex encoding exists to name: "the only way to specify a NaN value other than the specific NaN value denoted by 'NaN'". The old comment claiming NumPy cannot represent distinct NaNs was wrong. np.float32 holds payload bits. They were being discarded on write, not absent.

A related latent loss is fixed in the same place. float_from_json_v3 decoded hex via struct.unpack into Python float (64-bit), which cannot carry a float16 payload and narrows float32 payloads. Hex strings now decode through np.frombuffer into a same-width NumPy scalar, so all three widths round-trip.

Equality conflated distinct documents. ArrayV3Metadata.__eq__ compared to_dict() dictionaries, under which -0.0 equals 0.0 and every NaN payload equals "NaN" (pre-fix). __eq__ now compares json.dumps(to_dict(), sort_keys=True), the exact string __hash__ already hashed, so equal implies equal-hash by construction rather than by parallel convention.

For reviewers

float_from_json_v3 now returns np.floating for hex input instead of float. Both call sites, float.py and complex_float_from_json_v3, consume it through a cast or complex(), so there is no caller-visible change. "NaN" still denotes and round-trips the canonical positive NaN at each width. v2 serialization is untouched. Complex fill values go through the same helpers per component, so a complex64 real part holding 0x7fc00001 now writes and re-parses as "0x7fc00001" as well. That path was exercised but has no dedicated test. A float16 document declaring a wider hex fill still parses and casts as before. Payload bits survive only when the hex width matches the dtype width, which is what the spec requires anyway.

Author attestation

• I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md (no user-facing behavior beyond spec compliance)
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[d-v-b]

thanks for this, I will have a look in the next few days!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.71%. Comparing base (19e84dc) to head (d238278).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4486      +/-   ##
==========================================
+ Coverage   94.69%   94.71%   +0.01%     
==========================================
  Files          94       94              
  Lines       13619    13666      +47     
==========================================
+ Hits        12897    12944      +47     
  Misses        722      722              

Files with missing lines	Coverage Δ
src/zarr/core/dtype/npy/common.py	99.23% <100.00%> (+0.04%)	⬆️
src/zarr/core/metadata/v3.py	96.91% <100.00%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[d-v-b]

1. Compare all of a NaN's bits, sign included, before writing "NaN". Any other NaN, such as 0xffc00000, should be written as hex, as zarrs and tensorstore do. At the moment 0xffc00000 writes "NaN" and reads back as 0x7fc00000.
2. Make ArrayV3Metadata.__eq__ work without json.dumps(..., sort_keys=True). It raises TypeError when attribute keys mix int and str, and that reaches Array.__eq__ too.
3. Get the canonical NaN bits from numpy (np.asarray(np.nan, dtype=...) viewed as the matching uint) instead of hardcoded tables.
4. Use parametrize in test_noncanonical_nan_serializes_as_hex, and add tests for the 0xffc00000 round trip and the complex path.
5. note in the changelog that equality now compares the JSON text, so 1 and 1.0 in attributes no longer compare equal.

[barlowa124]

Pushed 8a2714a.

"NaN" now requires the value's full bit pattern to equal the canonical NaN, sign bit included, so 0xffc00000 round-trips as "0xffc00000" in line with zarrs and tensorstore. The canonical pattern is read from np.asarray(np.nan, dtype=data.dtype).view(uint_dtype). __eq__ and __hash__ no longer pass sort_keys to json.dumps, and attribute dicts mixing int and str keys compare without raising TypeError. The hex-payload test is parametrized over six cases covering both sign-set payloads, and a Complex64 test covers the complex path. The changelog states that equality compares the JSON text and that 1 versus 1.0 attributes compare unequal.

python -m pytest tests/test_metadata/ tests/test_dtype/ gives 2200 passed.

[barlowa124]

d238278 fixes the CI fallout: test_special_complex_fill_values_roundtrip expected inf * 1j to write "NaN" for its real part, but 0 * inf produces a NaN whose sign bit is platform-dependent (0xffc00000 on x86), so the expectation is now derived from the produced scalar's bits. ruff format applied on both branches.