Skip to content

Remove VerboseModule so zarr.storage can be serialized - #4505

Open
Yi-111-a wants to merge 1 commit into
zarr-developers:mainfrom
Yi-111-a:fix/4029-pickle-storage-module
Open

Yi-111-a wants to merge 1 commit into
zarr-developers:mainfrom
Yi-111-a:fix/4029-pickle-storage-module

Conversation

@Yi-111-a

@Yi-111-a Yi-111-a commented Oct 9, 2026

Copy link
Copy Markdown

Closes #4029.

Problem

src/zarr/storage/__init__.py replaced the imported module with a VerboseModule subclass of types.ModuleType:

sys.modules[__name__].__class__ = VerboseModule

Its only purpose was to warn when zarr.storage.default_compressor was set.

Serializers dispatch on the exact type of an object. cloudpickle — which Dask uses by default — registers a reduce for ModuleType, but VerboseModule is not in that table, so serialization fell through to object.__reduce_ex__ and failed:

TypeError: cannot pickle 'VerboseModule' object

I reproduced this on the current main (before this change) and confirmed the fix removes it:

$ python -c "...sys.modules['zarr.storage'].__class__ = VerboseModule; cloudpickle.dumps(zarr.storage)"
TypeError: cannot pickle 'VerboseModule' object

Fix

Remove VerboseModule entirely, as suggested by @d-v-b in #4029 ("we can remove that verbosemodule thing -- it only existed to emit a warning for long-deprecated behavior"), along with the now-unused imports and the test_default_compressor_deprecation_warning test that covered the removed warning.

This also fixes plain pickle, which fails with cannot pickle 'VerboseModule' object today and cannot pickle 'module' object after — modules are not serializable with plain pickle in either case, so nothing regresses there. cloudpickle is what actually matters for the Dask use case in the issue, and it round-trips correctly again.

Behaviour change

Setting zarr.storage.default_compressor no longer emits a deprecation warning. zarr.config remains the supported way to configure array.v2_default_compressor.

Testing

  • New regression test tests/test_store/test_core.py::test_storage_module_is_picklable asserts type(zarr.storage) is ModuleType and that cloudpickle round-trips the module.
  • cloudpickle added to the test dependency group (it was already an indirect dependency; needed to be a direct test dep).
  • Release note in changes/4029.bugfix.md.
  • pytest tests/test_store/test_core.py tests/test_v2.py tests/test_api.py → 634 passed, 9 skipped (CuPy/GPU).
  • ruff check, ruff format --check, and mypy src/zarr/storage/__init__.py all clean.

🤖 Generated with Claude Code

`zarr.storage` was replaced at import time by `VerboseModule`, a
`types.ModuleType` subclass that only existed to warn about the
long-deprecated `zarr.storage.default_compressor` attribute.

Serializers that dispatch on the exact type of an object miss that subclass:
`cloudpickle` (used by Dask) registers a reduce for `ModuleType`, so any
attempt to serialize the module raised `TypeError: cannot pickle
'VerboseModule' object`, which made `zarr.storage.ObjectStore`-backed Dask
arrays impossible to send to a cluster.

Drop the wrapper as suggested in zarr-developers#4029 and add a regression test. Setting
`zarr.storage.default_compressor` no longer warns; `zarr.config` is the
supported way to configure `array.v2_default_compressor`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VerboseModule cannot be pickled

1 participant