Skip to content

fix(storage): pickle the zarr.storage module by name - #4557

Open
d-v-b wants to merge 2 commits into
zarr-developers:mainfrom
d-v-b:fix/4029-storage-module-reduce
Open

d-v-b wants to merge 2 commits into
zarr-developers:mainfrom
d-v-b:fix/4029-storage-module-reduce

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This PR partially reverts the changes in #4505 and solves the underlying bug #4505 aimed to solve more directly by allowing the VerboseModule to be pickled (by defining __reduce__).

edit: I thought #4505 had been merged, but it hasn't. so this PR just closes #4029 directly without the removal component, which is slated for the 3.5.0 release

🤖 AI text below 🤖

The zarr.storage module can now be serialized with pickle and cloudpickle. Previously, code that referenced the module (for example a Dask task that used zarr.storage.ObjectStore) failed with TypeError: cannot pickle 'VerboseModule' object.

Closes #4029.

zarr.storage swaps its class for a VerboseModule subclass of ModuleType so it can warn when default_compressor is set. Serializers that dispatch on the exact type, such as cloudpickle (Dask's default), have no reduce for that subclass and fall through to object.__reduce_ex__, which raises. This PR gives VerboseModule a __reduce__ that re-imports the module by name, so both pickle and cloudpickle round-trip it to the same module object.

The deprecation warning for zarr.storage.default_compressor is unchanged, which keeps this a pure bugfix suitable for a patch release. #4505 fixes the same bug by removing VerboseModule and the warning altogether. The versioning policy in docs/contributing.md excludes behavior changes from patch releases, so that removal fits a minor release, and it can land there on top of this fix.

Testing:

  • New test test_storage_module_pickles_by_name in tests/test_store/test_core.py round-trips zarr.storage through the standard library's pickle, so it needs no new test dependency. It fails on main with the TypeError above.
  • cloudpickle round-trips zarr.storage with this change, checked in a throwaway environment.
  • tests/test_store/test_core.py, tests/test_v2.py, and tests/test_api.py pass, including the existing default_compressor deprecation-warning test. Ruff, mypy, and the other pre-commit hooks pass.

🤖 Generated with Claude Code

d-v-b added 2 commits October 22, 2026 22:42
`zarr.storage` swaps its class for a `VerboseModule` subclass to warn
when `default_compressor` is set. Serializers that dispatch on the exact
type, such as cloudpickle (Dask's default), had no reduce for the
subclass and raised `TypeError: cannot pickle 'VerboseModule' object`.
Give `VerboseModule` a `__reduce__` that re-imports the module by name,
so `pickle` and `cloudpickle` both round-trip it. The deprecation
warning is unchanged; removing it is left for a minor release.

Refs zarr-developers#4029.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b marked this pull request as ready for review October 22, 2026 20:47
@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.71%. Comparing base (957a5f6) to head (04363dc).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4557   +/-   ##
=======================================
  Coverage   94.71%   94.71%           
=======================================
  Files          94       94           
  Lines       13663    13667    +4     
=======================================
+ Hits        12941    12945    +4     
  Misses        722      722           
Files with missing lines Coverage Δ
src/zarr/storage/__init__.py 95.83% <100.00%> (+0.83%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VerboseModule cannot be pickled

1 participant