Repository navigation
Skip non-Zarr objects silently when listing group members - #4471
mayuriphad wants to merge 5 commits into
Conversation
LocalStore.list, list_prefix and list_dir returned the '<stem>.<uuid>.partial' files created by _atomic_write while a write was in progress. Readers then treated them as store keys and emitted ZarrUserWarning for each one. Filter files matching the temp-file pattern in _list_files and _list_dir and add a regression test. Fixes zarr-developers#4161.
|
i think we want the store to show these keys. we should remove the warning from member listing instead. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4471 +/- ##
==========================================
- Coverage 94.71% 94.71% -0.01%
==========================================
Files 94 94
Lines 13663 13662 -1
==========================================
- Hits 12941 12940 -1
Misses 722 722
🚀 New features to boost your workflow:
|
…ering LocalStore Per review: keep LocalStore listings faithful to what is on disk (including in-progress .partial temp files) and drop the ZarrUserWarning emitted when group member iteration encounters an object that is not a Zarr node.
|
Thanks @d-v-b, makes sense. I've reworked this: \LocalStore\ listings are back to reporting everything on disk (including in-progress .partial\ files), and group member iteration now skips objects that aren't Zarr nodes silently instead of emitting a \ZarrUserWarning. Updated \ est_group_members\ accordingly and the changelog entry. |
|
Can you update the pr title and description |
|
Thanks @d-v-b. I updated the title and the description to match the new approach. |
|
Updated as suggested: store listings are unchanged, the warning is removed from member listing instead, and the title and description now reflect that. Also merged main into the branch. |
Fixes #4161.
Problem
While
LocalStorewrites a key, it first writes to a temporary file named<name>.<uuid>.partialand then renames it into place. A reader that lists the store while another process is writing sees these files, andGroup.members()emitted aZarrUserWarning("Object at ... is not recognized as a component of a Zarr hierarchy") for each one. Any other object in the store that is not a Zarr node triggered the same warning.Change
As discussed in review, the store listings are unchanged and keep showing everything on disk. Instead, listing the members of a group now skips objects that are not Zarr nodes without a warning.
_iter_membersno longer warns in theKeyErrorbranch. It skips the object.test_group_membersnow checks that listing members does not warn about extra objects. The expected warning about consolidated metadata with Zarr format 3 is kept.test_use_consolidated_for_children_membersno longer expects the warning.changes/4161.bugfix.md.Testing
pytest tests/test_group.py tests/test_metadata/test_consolidated.py: 1010 passed, 209 skipped. The CI checks on this PR pass.