Repository navigation
Conversation
…ONTEXT scope Add definitions for the extensions zarr-python writes. zarr-extensions registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes` codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own `null_terminated_bytes` and `variable_length_bytes`, which no specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the default scope stays spec-pure. Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context. Fixes #423 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Resolve the conflict in test_pipelines.py: keep the unclaimed `acme.vlen` codec, at the codec-level location zarr-developers#4574 gives a codec's chunk problems. Rename CORE to CORE_CONTEXT in main's new test_rule_locations.py, and move the expectations in test_zarr_python_scope.py to zarr-developers#4574's locations: a codec's chunk problem at the codec, a missing key at the object that lacks it. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…ules Keep `CORE` and `CORE_AND_EXTENSIONS` importable from `zarr_metadata.v3` and `zarr_metadata.v3.definition` as the same scopes with a DeprecationWarning, so a consumer such as pydantic-zarr keeps working across the rename. Check that `vlen-utf8` encodes `string` and `vlen-bytes` encodes `bytes` (or zarr-python's `variable_length_bytes`), as zarr-extensions says, instead of accepting any data type whose values vary in size. Do not count trailing null bytes against the size of a `null_terminated_bytes` fill value, since they are padding. Spell a `fixed_length_utf32` fill value without its trailing U+0000 padding, so `"a"` and `"a\u0000"` have one canonical spelling. Assisted-by: ClaudeCode:claude-opus-5-5
Add zarr-python's `raw_bytes` data type (NumPy's `V`) to ZARR_PYTHON_CONTEXT, so the scope reads every data type zarr-python writes. Its names start with `ZarrPython`, since `RawBytes` already names the spec's raw bits. A fill value longer than `length_bytes` is a problem, because zarr-python cuts it short, and the canonical spelling pads it with null bytes, as zarr-python writes it. Rename CORE_V2 to CORE_V2_CONTEXT, so every scope's name ends in `_CONTEXT`. The old name keeps working with a DeprecationWarning, through the same `renamed_scope` helper as the v3 names, which moves to `zarr_metadata._renamed`. Share the base64 decoding and spelling of fill values in `_byte.py`, and say that the vlen codecs' `configuration` key is optional, since zarr-python writes an empty object. Assisted-by: ClaudeCode:claude-opus-5-5
Move CORE_V2_CONTEXT and its table of old names into `v2/_core.py`, as `v3/_core.py` holds the v3 scopes, so `zarr_metadata.v2` and `zarr_metadata.v2.definition` share one table. Say why a `fixed_length_utf32` fill value counts its trailing U+0000 code points, as the spec requires, while `null_terminated_bytes` follows zarr-python and does not, and test a value that only its padding makes too long. Assisted-by: ClaudeCode:claude-opus-5-5
…ers#4586) * ci(examples): run example scripts once in their own job Each example's PEP 723 header defines its environment, so running tests/test_examples.py in every pytest matrix job repeated the same work. Its header-rewriting helper also depended on the test environment's packaging and tomlkit, which broke the min_deps job on Python 3.12.3. A new `just examples` recipe uses `uv add --script` to point each header at the local checkout, and one CI job runs it. `uv add` resolves the header, and that showed two examples declaring requires-python >=3.11 although zarr requires 3.12. Those headers now say >=3.12. tomlkit was only used by the removed test, so it is dropped from the test dependency group. Assisted-by: ClaudeCode:claude-opus-5-5 * test(examples): run the example scripts through pytest The previous commit ran the examples from a bash loop, which stops at the first failing script, so one broken example hid the others. The examples now run from tests/test_examples.py again, one test per script. The tests are marked `examples` and skipped unless pytest gets --run-examples, as slow_hypothesis tests are with --run-slow-hypothesis, so the matrix jobs skip them. `just examples` passes the option, and the CI job runs it in the test.py3.13-minimal hatch environment. Assisted-by: ClaudeCode:claude-opus-5-5 * chore(just): rename the examples recipe to test-examples `just examples` could mean listing or building the examples. `test-examples` says what the recipe does, and follows the `<group>-<variant>` naming of recipes like `coverage-html` and `docs-build`. Assisted-by: ClaudeCode:claude-opus-5-5
…arr-developers#4575) * feat(zarr-metadata): add Hypothesis strategies for metadata fields Add `zarr_metadata.testing.strategies`, behind a new `testing` extra. It ships Hypothesis strategies that draw valid Zarr v3 metadata fields, one for every codec, data type, chunk grid and chunk key encoding the package defines that holds no other field. - Each kind's strategy takes what its configurations depend on: a data type's fill values take its configuration, a chunk grid takes the array shape, and a codec takes the `Chunk` it is handed. - `fields` draws every envelope spelling the spec allows, or only the canonical one. - Strategies are a mapping keyed by definition, so an extension author can add strategies for their own definitions. The tests read every draw back in `CORE_AND_EXTENSIONS` and require no problems. They also check: - that every definition in scope has a strategy, apart from the three that hold nested fields; - reach, with targets derived from each configuration's TypedDict: each optional key present and absent, each `Literal` value, and a non-canonical spelling where one exists; - fill-value spellings and envelope forms. Strategies follow the spec where the package is looser. Core data types are drawn only as bare strings, and regular-grid chunk lengths are never zero ("Chunk sizes must be greater than zero"), although the package's rules accept both today. Assisted-by: ClaudeCode:claude-opus-5-5 * chore(zarr-metadata): number the strategies changelog fragment after #414 Assisted-by: ClaudeCode:claude-opus-5-5 * chore(zarr-metadata): number the strategies changelog fragment after PR 4575 The fragment was numbered after fork PR 414, so the changelog linked to an unrelated upstream PR. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): draw a fresh empty configuration on every draw `st.just({})` (crc32c) and `st.just(EmptyConfiguration())` (every data type with nothing to configure) handed out one dict for every draw, and `fields` puts the configuration in the field as it is. A test that changed the configuration it drew changed every later draw, which Hypothesis reports as a flaky strategy. Both are now `st.builds(EmptyConfiguration)`. Export `Configurations` and `NOTHING_KNOWN` from `zarr_metadata.testing.strategies`: they appear in the signatures of `DefinitionStrategy.unconstrained` and `codec_configurations`. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): check every drawn field, and draw hex scale_offset values `fields` read a field only when asked for the canonical spelling, so a caller's configurations the definition refuses were drawn silently in the default spelling. Every field is now read, in a scope built once per call rather than once per draw, and a refused one raises in both spellings. A definition no Zarr v3 scope holds, such as a v2 codec, is a `TypeError` when `fields` is called, not a `StopIteration` on a draw. A float `scale_offset` codec's `offset` and `scale` are fill values of its type, so they are now also drawn as the hex string of their bits. The docstring no longer claims a non-zero scale makes decoding defined: a tiny one still overflows. The codec chunk test now draws chunks of an unknown data type and with unknown lengths, which every codec takes. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(zarr-metadata): draw scale_offset for every number type, and widen Strategies keys `scale_offset` drew nothing for complex numbers, numpy times and data types the package does not define, though its rules refuse only data types whose values are not numbers, and a numpy time "is compatible with any codec that supports arrays of signed 64-bit integers". A numpy time is now drawn as that integer, a complex number as a pair of finite floats, and a type the package does not define takes the identity configuration. None now means only what `CodecStrategy` says it means: no configuration takes the chunk. `Strategies` is keyed by `Any`: a mapping's key type is invariant, so a mapping an extension author builds, keyed by a `CodecDefinition`, was not a `Mapping[Definition[Any], ...]` to pyright. A rectilinear dimension of unknown length may now be drawn with no edges, which its rules accept. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(zarr-metadata): share the strategies' empty configuration and rank cap `EMPTY_CONFIGURATIONS` and `MAX_RANK` are defined once, in `_strategy`, rather than once in each module that draws them. A numpy time's `scale_offset` values are now drawn finite, never the integer numpy reads as `NaT`, as the docstring says. The `fields` docstring now says that a codec's chunk rules are not checked, since no chunk is known. Assisted-by: ClaudeCode:claude-opus-5-5 * test(zarr-metadata): cover scale_offset over a data type the package does not define The identity configuration `scale_offset` draws for a data type the package does not define had no test, since the chunk test draws only the package's own data types. A test now reads an extension `bfloat16` in a scope of its own and checks the codec takes every draw. Also splits the `_scope_of` error message across two literals and drops a doubled blank line in `_data_type`. Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(deps): bump cast-value.rs to >=0.5.2 * chore(deps): raise the runtime cast-value-rs floor to 0.5.1 The previous commit raised the packaging floor to 0.5.1 but left the runtime check at 0.4.2. cast-value-rs 0.5.0 makes casts exact at integer and float range limits; 0.5.0 was never published to PyPI, so the floor is 0.5.1. Also update the error message, the version-check test cases, and uv.lock. Assisted-by: ClaudeCode:claude-opus-5-5 * test(cast_value): derive version-check cases from the minimum The cases hard-coded versions near the floor, so every floor bump meant rewriting them. Build them from CAST_VALUE_RS_MIN_VERSION instead, and drop cases that repeated the same comparison. A floor bump now touches only pyproject.toml and the constant. Assisted-by: ClaudeCode:claude-opus-5-5
… definitions The Hypothesis strategies merged from main (zarr-developers#4575) imported `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the `_CONTEXT` names, and their coverage test requires a strategy for every definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each vlen codec draws nothing for a data type it does not read. Assisted-by: ClaudeCode:claude-opus-5-5
…strategies `DEFAULT_STRATEGIES` holds a strategy for each definition the package defines, but `null_terminated_bytes`, `variable_length_bytes` and `raw_bytes` had none, so `fields()` raised `LookupError` for them. Draw each, and check the strategies against `ZARR_PYTHON_CONTEXT`, which holds every definition. Say that the documents in test_zarr_python_scope.py leave out the compressors zarr-python adds. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…IONS_CONTEXT The strategies are now checked against `ZARR_PYTHON_CONTEXT`, which no longer proves that `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes` are in `CORE_AND_EXTENSIONS_CONTEXT`. Assert it directly. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each vlen codec strategy named its compatible data types a second time; draw nothing for exactly the chunks the codec's own chunk rules refuse, so the two cannot drift. Draw `fixed_length_utf32` fill values with trailing U+0000 padding too, which the canonical spelling removes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Add the
ZARR_PYTHON_CONTEXTscope, which reads what zarr-python writes:CORE_AND_EXTENSIONS_CONTEXTplus zarr-python's own data typesnull_terminated_bytes(NumPy'sS) andvariable_length_bytes, which no specification registers.CORE_AND_EXTENSIONS_CONTEXTnow also reads thefixed_length_utf32data type (NumPy'sU) and thevlen-utf8andvlen-bytescodecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.Rename the v3 scopes
COREandCORE_AND_EXTENSIONStoCORE_CONTEXTandCORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is aZarrV3Context.Fixes #423
What the agent checked
The agent generated v3 metadata with zarr-python 3.4.2 for
<U3,S4,strandVariableLengthBytes()arrays. All four now read with no problem and no unclaimed field inZARR_PYTHON_CONTEXT. Before this change,CORE_AND_EXTENSIONSleft thestringarray'svlen-utf8codec unclaimed as well as the data types.fixed_length_utf32CORE_AND_EXTENSIONS_CONTEXTvlen-utf8,vlen-bytescodecsCORE_AND_EXTENSIONS_CONTEXTnull_terminated_bytesZARR_PYTHON_CONTEXTvariable_length_bytesZARR_PYTHON_CONTEXTRules the new definitions check:
length_bytesis a positive multiple of 4 forfixed_length_utf32, and at least 1 fornull_terminated_bytes.null_terminated_bytesfill value drops trailing null bytes.fixed_length_utf32is multi-byte, so thebytescodec requires anendian.vlen-*codecs reject a data type of fixed size. This uses the data type's declared storage, so it does not tellstringfrombytes: a name check would refuse zarr-python's ownvariable_length_bytes+vlen-bytespairing.Not covered
raw_bytes(NumPyV): its type names would collide with theRawBytes*names thatr*already exports at the top level.structuredalias that earlier zarr-python 3.x releases wrote.numcodecs.*codecs.CORE_V2keeps its name.just checkpasses on Python 3.11 and the newest supported version, including the strict docs build.🤖 Generated with Claude Code