Repository navigation
Conversation
`zarr.storage` was replaced at import time by `VerboseModule`, a `types.ModuleType` subclass that only existed to warn about the long-deprecated `zarr.storage.default_compressor` attribute. Serializers that dispatch on the exact type of an object miss that subclass: `cloudpickle` (used by Dask) registers a reduce for `ModuleType`, so any attempt to serialize the module raised `TypeError: cannot pickle 'VerboseModule' object`, which made `zarr.storage.ObjectStore`-backed Dask arrays impossible to send to a cluster. Drop the wrapper as suggested in zarr-developers#4029 and add a regression test. Setting `zarr.storage.default_compressor` no longer warns; `zarr.config` is the supported way to configure `array.v2_default_compressor`.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4029.
Problem
src/zarr/storage/__init__.pyreplaced the imported module with aVerboseModulesubclass oftypes.ModuleType:Its only purpose was to warn when
zarr.storage.default_compressorwas set.Serializers dispatch on the exact type of an object.
cloudpickle— which Dask uses by default — registers a reduce forModuleType, butVerboseModuleis not in that table, so serialization fell through toobject.__reduce_ex__and failed:I reproduced this on the current
main(before this change) and confirmed the fix removes it:Fix
Remove
VerboseModuleentirely, as suggested by @d-v-b in #4029 ("we can remove that verbosemodule thing -- it only existed to emit a warning for long-deprecated behavior"), along with the now-unused imports and thetest_default_compressor_deprecation_warningtest that covered the removed warning.This also fixes plain
pickle, which fails withcannot pickle 'VerboseModule' objecttoday andcannot pickle 'module' objectafter — modules are not serializable with plainpicklein either case, so nothing regresses there.cloudpickleis what actually matters for the Dask use case in the issue, and it round-trips correctly again.Behaviour change
Setting
zarr.storage.default_compressorno longer emits a deprecation warning.zarr.configremains the supported way to configurearray.v2_default_compressor.Testing
tests/test_store/test_core.py::test_storage_module_is_picklableassertstype(zarr.storage) is ModuleTypeand thatcloudpickleround-trips the module.cloudpickleadded to thetestdependency group (it was already an indirect dependency; needed to be a direct test dep).changes/4029.bugfix.md.pytest tests/test_store/test_core.py tests/test_v2.py tests/test_api.py→ 634 passed, 9 skipped (CuPy/GPU).ruff check,ruff format --check, andmypy src/zarr/storage/__init__.pyall clean.🤖 Generated with Claude Code