feat: builtin zarr:, zarr2: and zarr3: URL pipeline adapters

changes/4192.feature.md

@@ -0,0 +1,46 @@
+Added core support for URL pipelines (https://github.com/jbms/url-pipeline):
+`|`-chained URLs that address zarr data through nested storage layers, e.g.
+`s3://bucket/data.zip|zip:|zarr3:`. This includes the parser
+(`zarr.storage.parse_pipeline` / `zarr.storage.resolve_pipeline`), the
+single-method `zarr.abc.url_pipeline.URLPipelineAdapter` interface, and the
+`zarr.url_adapters` entry-point group through which third-party packages
+(e.g. Icechunk) register adapters for their own schemes. Adapters for a scheme
+are loaded lazily and individually. Builtin adapters (`zip:`,
+`zarr2:`/`zarr3:`) will follow separately.
+
+Behavior notes:
+
+- The `|` character is now reserved as the pipeline delimiter in every string
+  store specification, and no percent-escape is decoded; pass a `pathlib.Path`
+  to address a local file whose name contains `|`. URLs without a `|` (and
+  without a registered root adapter scheme) are handled exactly as before —
+  registered adapters cannot intercept zarr's native `file:`/`memory:`
+  routing, and fsspec chained URLs (`zip::s3://...`) keep flowing to fsspec.
+- Inside a pipeline, `memory:` and `file:` roots follow the URL pipeline
+  spec's spelling rules (`memory:` ≡ `memory:/` ≡ `memory://`; `file:` must
+  carry an absolute path with at most a `localhost` authority). Percent-escapes
+  are *not* decoded in `file:` roots, matching `LocalStore`; this is a
+  documented divergence from RFC 8089. When fsspec is installed, a plain
+  `memory://name` URL is still routed to fsspec's in-memory filesystem, whereas
+  a pipeline's `memory:` root is zarr's `ManagedMemoryStore`; the two do not
+  share data.
+- Wrapper adapters can open a local *file* root only in mode `"r"` (via
+  `PipelineContext.resolve_preceding(mode="r")`); other modes create a
+  directory at the root and fail. Writing through a container file is not yet
+  supported.
+- A `zarr.url_adapters` entry point whose scheme is already registered (e.g. by
+  a builtin adapter) is ignored with a `ZarrUserWarning`, as are duplicate
+  entry-point names; an entry point that fails to import raises
+  `URLPipelineError` naming it, and stays discoverable for a retry.
+- `zarr.create_array` and `Group.from_store` now default `zarr_format` to
+  `None`, resolving to the configured default (3) as before unless a URL
+  pipeline selects a format; `Array.open` gains a `zarr_format` parameter
+  mirroring `Group.open`. `Group.open` and `Array.open` keep their default of
+  3, so pass `zarr_format=None` to defer to a pipeline's format. A format
+  selected by a pipeline that conflicts with an explicit `zarr_format` raises
+  `ValueError` in every `open*`/`create*`/`save*` entry point.
+- Mode `"a"` (open-or-create, the `zarr.open` default) on a *read-only* store
+  now serves the "open" half instead of raising upfront, for all stores;
+  unambiguous write modes (`"w"`, `"w-"`, `"r+"`) still raise.
+- For root-adapter URLs (e.g. `gh://org/repo`), `storage_options` are handed
+  to the adapter and are not validated as used by `make_store`.

changes/4476.feature.md

@@ -0,0 +1,11 @@
+Added the builtin `zarr:`, `zarr2:` and `zarr3:` URL pipeline adapters, so a pipeline
+can address a Zarr node within a resource and select its format, e.g.
+`zarr.open_group("file:///data/archive|zarr2:path/to/group")`. The text after the colon
+is a path within the preceding resource; `zarr2:` / `zarr3:` select the Zarr format and
+`zarr:` auto-detects it. These schemes are adapter-only, so strings without `|` behave as
+before.
+
+`Group.open`, `Array.open`, `AsyncGroup.open` and `AsyncArray.open` no longer default to
+`zarr_format=3` when given a URL pipeline: if `zarr_format` is not passed, the pipeline's
+format segment decides (auto-detection when there is none). For every other store the
+default is still 3.

docs/api/zarr/abc/index.md

@@ -11,3 +11,4 @@ Abstract base classes for extending Zarr-Python.
 - **[zarr.abc.metadata](./metadata.md)** - Creating metadata classes compatible with the Zarr API
 - **[zarr.abc.numcodec](./numcodec.md)** - Protocols and classes for modeling codec interface used by numcodecs
 - **[zarr.abc.store](./store.md)** - ABC for implementing Zarr stores and managing getting and setting bytes in a store
+- **[zarr.abc.url_pipeline](./url_pipeline.md)** - ABC for implementing [URL pipeline](https://github.com/jbms/url-pipeline) adapters

docs/api/zarr/abc/url_pipeline.md

@@ -0,0 +1,5 @@
+---
+title: url_pipeline
+---
+
+::: zarr.abc.url_pipeline

docs/user-guide/extending.md

@@ -85,6 +85,70 @@ implementation.
 Custom stores can be created by implementing the [`zarr.abc.store.Store`][] interface.
 See [developing custom stores](storage.md#developing-custom-stores) for more information.
 
+## Custom URL pipeline adapters
+
+[URL pipelines](storage.md#user-guide-url-pipelines) are `|`-chained URLs such as
+`s3://bucket/data.zip|zip:|zarr3:`. Each sub-URL after the first names an *adapter*
+that reinterprets everything to its left. A package provides an adapter for a scheme by
+subclassing [`zarr.abc.url_pipeline.URLPipelineAdapter`][] and implementing its single
+classmethod, which resolves the adapter's segment into an open store and a residual path:
+
+```python test="true" session="url-adapter"
+import dataclasses
+
+from zarr.abc.url_pipeline import (
+    AdapterResolution,
+    PipelineContext,
+    PipelineSegment,
+    URLPipelineAdapter,
+)
+from zarr.storage import WrapperStore
+
+
+class MyAdapter(URLPipelineAdapter):
+    @classmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> AdapterResolution:
+        # A *wrapper* adapter opens the resource to its left and wraps it. Join the
+        # preceding residual path with this segment's own path, and keep every other
+        # field (e.g. zarr_format) via dataclasses.replace.
+        preceding = await context.resolve_preceding()
+        store = WrapperStore(preceding.store)
+        path = "/".join(part.strip("/") for part in (preceding.path, segment.body) if part)
+        return dataclasses.replace(preceding, store=store, path=path)
+```
+
+Wrapper adapters compose: in `memory://x|mypackage.myscheme:a|mypackage.myscheme:b` the
+inner adapter sees the outer one's residual path `a` and the pipeline resolves to `a/b`.
+
+```python test="true" session="url-adapter"
+import asyncio
+
+from zarr.registry import register_url_adapter
+from zarr.storage import resolve_pipeline
+
+register_url_adapter("mypackage.myscheme", MyAdapter)
+resolution = asyncio.run(resolve_pipeline("memory://x|mypackage.myscheme:a|mypackage.myscheme:b"))
+assert resolution.path == "a/b"
+```
+
+Register the class under the `zarr.url_adapters` entry-point group, using the URL
+scheme as the entry-point name. Nonstandard schemes should be vendor-prefixed
+(`vendor.scheme`), per the URL pipeline specification:
+
+```toml
+[project.entry-points."zarr.url_adapters"]
+"mypackage.myscheme" = "mypackage.zarr_adapter:MyAdapter"
+```
+
+Adapters are loaded lazily, only when a pipeline naming their scheme is resolved. An
+adapter may also be used as a *root* scheme (e.g. `mypackage.myscheme://org/repo`), in
+which case `context.preceding` is empty. The returned store must honor
+`context.read_only`, and `open_pipeline_segment` runs on zarr's internal event loop, so
+it must not block or call zarr's synchronous API. See
+[`zarr.abc.url_pipeline`][] for the full contract.
+
 ## Custom array buffers
 
 Zarr-python provides control over where and how arrays are stored in memory through

docs/user-guide/storage.md

@@ -100,9 +100,61 @@ print(group)
    group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r')
    ```
 
+- a [URL pipeline](#user-guide-url-pipelines) string containing `|`, such as
+  `s3://bucket/data.zip|zip:`, which is resolved through registered adapters.
+
 - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
   see explicit store creation below.
 
+## URL Pipelines {#user-guide-url-pipelines}
+
+Zarr supports [URL pipelines](https://github.com/jbms/url-pipeline): `|`-chained URLs
+that address zarr data through nested storage layers, read left to right. The first
+sub-URL locates a resource with a conventional URL; each subsequent sub-URL names an
+*adapter* that reinterprets everything to its left (e.g.
+`s3://bucket/data.zip|zip:|zarr3:`). Adapters are provided by packages through the
+`zarr.url_adapters` entry-point group — see
+[`zarr.abc.url_pipeline`][zarr.abc.url_pipeline] for the adapter interface. URLs
+without a `|` (and without a registered root scheme) are handled exactly as before.
+
+Zarr ships the `zarr:`, `zarr2:` and `zarr3:` format adapters. The text after the
+colon is a path within the resource to its left, and `zarr2:` / `zarr3:` select the
+Zarr format (`zarr:` auto-detects it). The colon is optional when the path is empty:
+
+```python exec="true" session="storage-url-pipeline" source="above" result="ansi"
+import tempfile
+
+import zarr
+
+root = tempfile.mkdtemp()
+group = zarr.open_group(f"{root}|zarr2:", mode="w")
+group.create_array("data", shape=(4,), dtype="int32")
+print(zarr.open_array(f"{root}|zarr2:data", mode="r").metadata.zarr_format)
+print(zarr.open_group(f"file:{root}|zarr", mode="r").metadata.zarr_format)
+```
+
+A `zarr_format` argument that conflicts with a `zarr2:` / `zarr3:` segment raises
+`ValueError`. `Group.open` and `Array.open` take their format from the pipeline when
+`zarr_format` is not passed.
+
+`storage_options` passed to `zarr.open` apply to the *root* sub-URL (e.g. fsspec
+options for `s3://...`); adapters may consume adapter-specific, namespaced keys.
+Non-dict forms of `storage_options` are reserved for future per-segment
+configuration.
+
+The `|` character is reserved as the pipeline delimiter in every string store
+specification, and no percent-escape is decoded: to address a local file whose
+*name* contains `|` (or `#`), pass a `pathlib.Path` instead of a string.
+Registered adapters cannot intercept zarr's native `file:` and `memory:` root
+schemes, and fsspec's chained-URL syntax (`zip::s3://...`) keeps flowing to
+fsspec.
+
+Inside a pipeline, a `memory:` root is zarr's managed in-memory store (`memory:`,
+`memory:/` and `memory://` are equivalent, and `memory:name` selects a named store).
+When fsspec is installed, a plain `memory://name` URL *without* a `|` is still routed to
+fsspec's in-memory filesystem, which is a different store. A `file:` root must carry an
+absolute path; percent-escapes are not decoded, matching the [local store](#local-store).
+
 ## Explicit Store Creation
 
 In some cases, it may be helpful to create a store instance directly. Zarr-Python offers

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.metadata</code>': api/zarr/abc/metadata.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.numcodec</code>': api/zarr/abc/numcodec.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.store</code>': api/zarr/abc/store.md
+        - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.url_pipeline</code>': api/zarr/abc/url_pipeline.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api</code>':
         - api/zarr/api/index.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api.asynchronous</code>': api/zarr/api/asynchronous.md