Skip to content

fix: validate consolidated metadata listing keys and hierarchy - #4485

Open
barlowa124 wants to merge 5 commits into
zarr-developers:mainfrom
barlowa124:consolidated-metadata-4453
Open

barlowa124 wants to merge 5 commits into
zarr-developers:mainfrom
barlowa124:consolidated-metadata-4453

Conversation

@barlowa124

@barlowa124 barlowa124 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Second item of #4453. ConsolidatedMetadata.from_dict (and
GroupMetadata.from_dict, which delegates to it) accepted listings
the spec rules out, two of which broke reads.

Listing keys are node paths, so keys with an empty segment ("",
a//b) are now rejected with MetadataValidationError. Keys carrying
a reserved segment emit
a ZarrDeprecationWarning naming the key and advising a rename,
avoiding breakage of stores that work today.

Every non-root key also needs its ancestors present as group entries:
{"a/b": GROUP} used to surface as GroupNotFoundError on open,
making the root unopenable from one entry. It now fails with a clear
MetadataValidationError, as does a listing nested under an array.
v2 listings skip the name checks entirely. The remaining #4453 items
are untouched.

For reviewers

The reserved-name path moved from rejection to deprecation after
review, since stores written today with __-prefixed names still open
on main. It becomes a hard error once #4484 stops those names being
written. The empty-segment rule stays strict because consolidated keys
are node paths and node names cannot be empty. If #4484 lands first,
this can reuse its helper for the reserved-name part.

Author attestation

  • I am a human. These are my changes. I have reviewed and understood every change and can explain why each is correct.

TODO

Done:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md

Outstanding:

  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@github-actions github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 7, 2026
barlowa124 added a commit to barlowa124/zarr-python that referenced this pull request Oct 7, 2026
@barlowa124
barlowa124 force-pushed the consolidated-metadata-4453 branch from 668c547 to 7b01078 Compare October 7, 2026 21:08
@github-actions github-actions Bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Oct 7, 2026
@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.72%. Comparing base (069fd20) to head (9156127).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4485      +/-   ##
==========================================
+ Coverage   94.69%   94.72%   +0.03%     
==========================================
  Files          94       94              
  Lines       13606    13685      +79     
==========================================
+ Hits        12884    12963      +79     
  Misses        722      722              
Files with missing lines Coverage Δ
src/zarr/core/group.py 95.77% <100.00%> (+0.09%) ⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@d-v-b d-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Warn instead of raising when a v3 consolidated listing has a reserved node name (__-prefixed, zarr.json, ., ..). today we can write __a (create_array("__a") plus consolidate_metadata), and those stores open on main. With this PR, zarr.open_group(store) raises MetadataValidationError for the whole group, siblings included. Make it a deprecation warning that describes the listing and says to rename the node, and raise only after #4484 has stopped these names being written.
  2. Don't apply the reserved-name rules to v2 listings. The v2 spec doesn't reserve these names, and v2 .zmetadata goes through the same from_dict.
  3. Keep raising for a child listed without its parent and for a node listed below an array. Those listings already failed (KeyError, unopenable root), so a clear error is an improvement.
  4. Use single backticks in changes/4485.bugfix.md, and say which cases warn and which raise.

@d-v-b

d-v-b commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

we don't want to break stuff that works today, even if it violates a subtle part of the spec. so lets do this via deprecation

@barlowa124

Copy link
Copy Markdown
Contributor Author

fc0e8cf makes the changes.

  1. Reserved names in a v3 listing now emit a ZarrDeprecationWarning naming the key and advising a rename. create_array("__a") plus consolidate_metadata stores open with a warning instead of MetadataValidationError. Keys with empty path segments still raise, since no stored node can have an empty name.
  2. v2 listings skip the name checks entirely.
  3. Missing-parent and below-array listings still raise MetadataValidationError.
  4. The changelog uses single backticks and states which cases warn and which raise.

python -m pytest tests/test_metadata/test_consolidated.py tests/test_group.py gives 1028 passed.

@barlowa124

Copy link
Copy Markdown
Contributor Author

Done in fc0e8cf: reserved node names in a v3 consolidated listing now emit a ZarrDeprecationWarning instead of raising, so stores that write __a today keep opening while the name is flagged for rename. The raised checks that remain are the structural ones: empty path segments, a missing parent entry, a node listed under an array. Those produce stores that cannot be read back at all. 1913880 and 9156127 are format and test cleanup on top.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants