[jhamman]

Summary

Core support for URL pipelines (https://github.com/jbms/url-pipeline, ZEP 8): |-chained URLs that address zarr data through nested storage layers, e.g. s3://bucket/data.zip|zip:|zarr3:. Adapters are registered under the zarr.url_adapters entry-point group (entry-point name = URL scheme).

Following the review discussion below, this PR no longer changes what existing string store specifications mean. A plain string is never read as a pipeline. Pipelines reach zarr in two explicit ways:

• zarr.open_url(url, *, mode="r" | "r+", storage_options=None) opens the existing group or array a pipeline string addresses. It never creates or overwrites anything.
• zarr.storage.URLPipeline.from_url(url) builds a parsed, immutable pipeline object. It can be passed anywhere a StoreLike is accepted (open_group, create_array, xr.open_zarr(store=...), ...), which is how you create or write through a pipeline.

zarr.open("data|x.zarr") still opens a local directory of that name, whatever adapters are installed.

What's in the PR:

• zarr.abc.url_pipeline: only the URLPipelineAdapter ABC. Adapters receive a zarr.storage.PipelineSegment and zarr.storage.PipelineContext and return a zarr.storage.StorePath. To resolve the preceding pipeline with another mode or options, an adapter derives a context: dataclasses.replace(context, mode="r").resolve_preceding().
• zarr.storage.URLPipeline: holds every parsed segment and resolves via URLPipeline.resolve().
  • The root sub-URL must carry a scheme.
  • memory: and file: roots follow the spec's semantics. Every other root delegates to the existing make_store routing.
• Format segments (zarr2:/zarr3:) are read by zarr from the pipeline itself, not by adapters, and must be the last segment. The format they select is merged with the caller's zarr_format; an explicit conflict raises.
• zarr.registry.register_url_adapter registers an adapter without an entry point and warns when it replaces one. Lookup is lazy per scheme, thread-safe and case-insensitive, and is internal.
• StorePath is unchanged from main. There is no zarr_format attribute, and mode "a" on a read-only store raises as before. create_array and Group.from_store keep their default of 3.

Behavior notes are in changes/4192.feature.md, and the user guide has a URL pipelines section.

Why the change in scope

The earlier version reserved | in every string store spec. That is a breaking change for anyone with | in a local path or object key; see @d-v-b's review. Explicit entry points let the pipeline machinery ship now with no change to existing strings. The design here is @d-v-b's jhamman#7, merged with the current branch and main.

Planned stack

#	PR	Contents
1	this PR	URL pipeline core + zarr.open_url
2	#4477 (to be reworked)	builtin read-only zip: adapter; drops its format pseudo-adapters
3	follow-up	file-resource primitive for adapters, and writable zip:
4	follow-up	root semantics and API polish
5	follow-up	pipeline strings in the existing APIs (zarr.open("s3://bucket/repo|icechunk:"))

#4476 (format adapters) will be closed as superseded, since format segments are now read by zarr itself.

Proposal for PR 5:

• Rule: a string is treated as a pipeline only under a strict rule:
  • its root carries a URL scheme, so bare paths are never claimed;
  • every segment after a | names an installed adapter followed by :.
• 3.5: strings that match the rule keep their current meaning, with a FutureWarning that they will be read as URL pipelines starting in 3.6.
• 3.6: the behavior switches.
• Who is affected: only users whose existing paths match the rule. Their fallback is to construct the store explicitly (LocalStore, FsspecStore, pathlib.Path) or rename the path.
• Why the population is small: a literal | is not a valid URL character (RFC 3986), it's on AWS's "characters to avoid" list for S3 keys, GCS discourages it, and Windows forbids it in file names.

Author attestation

• I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.21429% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.77%. Comparing base (3d73e29) to head (d534d31).

Files with missing lines	Patch %	Lines
src/zarr/storage/_url_pipeline.py	98.24%	3 Missing ⚠️
src/zarr/api/asynchronous.py	95.23%	1 Missing ⚠️
src/zarr/core/group.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4192      +/-   ##
==========================================
+ Coverage   94.71%   94.77%   +0.06%     
==========================================
  Files          94       96       +2     
  Lines       13663    13923     +260     
==========================================
+ Hits        12941    13196     +255     
- Misses        722      727       +5     

Files with missing lines	Coverage Δ
src/zarr/__init__.py	97.82% <ø> (ø)
src/zarr/abc/url_pipeline.py	100.00% <100.00%> (ø)
src/zarr/api/synchronous.py	96.05% <100.00%> (+0.27%)	⬆️
src/zarr/core/array.py	98.44% <100.00%> (+<0.01%)	⬆️
src/zarr/errors.py	100.00% <100.00%> (ø)
src/zarr/registry.py	93.21% <100.00%> (+1.45%)	⬆️
src/zarr/storage/__init__.py	95.23% <100.00%> (+0.23%)	⬆️
src/zarr/storage/_common.py	93.51% <100.00%> (+0.48%)	⬆️
src/zarr/api/asynchronous.py	96.27% <95.23%> (-0.20%)	⬇️
src/zarr/core/group.py	95.59% <80.00%> (-0.09%)	⬇️
... and 1 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[d-v-b]

This is great! Here's some stuff I found with Claude:

🤖 AI text below 🤖

Review of c744f95. The parser and adapter ABC look solid (the spec corpus passes, ruff/mypy are clean, no import cycles, test order doesn't leak registry state), but the store hooks introduce some regressions and there are a few contract questions worth settling before the zip:/zarr3: follow-ups build on this. Everything below was reproduced locally unless marked otherwise.

Correctness / regressions

1. | is now reserved in every string store spec, with no escape, and str vs Path diverge (_url_pipeline.py:115). Any string containing | is routed to the pipeline machinery:

   zarr.open_group(f"{tmp}/a|b", mode="w")        # main: LocalStore. PR: URLPipelineError: no URL pipeline adapter is registered for scheme 'b'
   zarr.open_group(Path(f"{tmp}/a|b"), mode="w")  # still works

   Reserving | is presumably intended (it's the spec delimiter), but the changelog only says URLs without | are unchanged; it should say | is now reserved — and there's currently no escape, since %7C is not decoded for local paths (open_group(f"{tmp}/a%7Cb") roots at a literal a%7Cb). Same story for #: the _split_query error tells users to write %23, but nothing decodes it, so a local file with # or | in its name is unreachable through a pipeline in either spelling.

2. Local file roots can't be resolved by wrapper adapters except in mode "r" (_url_pipeline.py:161). The base case hands the root to make_store, which builds a LocalStore and mkdirs it (_local.py:169) for every mode but "r". With a minimal adapter that returns context.resolve_preceding():

   await make_store_path(f"{tmp}/data.zip|wrap:")   # mode None / "a" / "w" / "r+" / "w-" -> FileExistsError: [Errno 17] File exists: '.../data.zip'

   In "r" the wrapper gets a LocalStore rooted at the file, whose only useful op is the accidental get(""). And for a not-yet-existing archive, zarr.open("new.zip|zip:", mode="w") creates a directory named new.zip before the adapter runs. There's also no way for an adapter to ask for the root to be opened read-only/non-creating (resolve_preceding() takes no mode). The base case needs a file-vs-directory notion (or wrapper adapters need a different primitive) before the zip PR.

3. make_store(url, mode="r") returns a writable store when the adapter ignores context.read_only (_common.py:357-364). Every other make_store branch bakes read_only = mode == "r" into the constructor; the pipeline branch returns result.store untouched. make_store_path gets it right only incidentally via StorePath.open(mode="r") → with_read_only(True). The CLI (_cli/cli.py) uses make_store directly. And the reference adapter in tests/package_with_entrypoint/__init__.py:114 (MemoryStore.open(read_only=False)) itself violates the ABC's "must honor context.read_only" — so the one example adapter authors will copy is wrong. Suggest the resolver enforce it (with_read_only(True) or raise) rather than trusting adapters. (Note ZipStore doesn't implement with_read_only, so a zip adapter returning a writable store under "r" would hit NotImplementedError via make_store_path too.)

4. mode="a" → "r" downgrade only happens for pipeline strings, and produces a late, unclear error (_common.py:488). Every other StoreLike gets StorePath.open's explicit Store is read-only but mode is 'a'. Create a writable store or use 'r' mode.; here make_store_path returns a read-only StorePath while open_group/open_array still hold mode="a", take the create path when the node doesn't exist, and die in Store._check_writable:

   zarr.open_group("rooty://x", mode="a")   # ValueError: store was opened in read-only mode and does not support writing

   test_mode_a_downgrades_to_read_only_open asserts this deliberately, so it's a design choice — but either "a on a read-only store means open-only" belongs in StorePath.open for all stores, or it shouldn't exist and adapters own the decision (which is what the PipelineContext.mode docstring already says).

5. fsspec :: chained URLs get captured by a same-named root adapter (_url_pipeline.py:117-118 via _root_scheme). parse_store_url("zip::file:///t/d.zip") yields scheme='zip', so as soon as any adapter named zip is registered (the planned builtin, or an entry point), zarr.open("zip::file:///t/d.zip") — which works today through FsspecStore.from_url — is rerouted to the adapter as a root segment with body=':file:///t/d.zip', preceding=(). Same for tar::, simplecache::, etc. Root-adapter dispatch should exclude scheme:: roots (or treat them as opaque fsspec URLs).

6. make_store tests the raw residual path (_common.py:359). An adapter returning path="/" makes make_store raise "resolves to a path inside a store" while make_store_path of the same URL normalizes it to "". AdapterResolution.path says root is "", so "/" is technically off-contract, but spec bodies are slash-prefixed (zip:/path/..., n5:/..., json:/) so it's worth normalize_path-ing defensively.

7. Entry-point names are never lowercased (registry.py:329,343, minor). register_url_adapter lowercases, but get_url_adapter compares the lowercased scheme against the raw entry_point.name and list_url_adapter_schemes unions raw names. An entry point named MyScheme is advertised but never loadable or routable — get_url_adapter("myscheme") raises no URL pipeline adapter is registered for scheme 'myscheme'. Registered schemes: ['MyScheme'].

8. Root-adapter URLs skip make_store's storage_options validation (_common.py:357). For rooty://x, storage_options are handed to the adapter and never checked as used, so make_store's documented TypeError contract no longer holds for those strings — fine if intended, but worth stating. (Direct make_store also skips the mode assert for root adapters, but that's a bare assert on main too and StorePath.open still validates for zarr.open, so not much lost.)

9. get_url_adapter's lazy-load loop is racy across threads and can permanently drop a scheme (registry.py:338-347). It iterates lazy_load_list while entry_point.load() (an import; releases the GIL) runs, then slice-assigns [:] = remaining. Two threads resolving different schemes interleave → one thread's rebuilt list omits entries the other shifted, and its final write clobbers. Reproduced with fake entry points [b, a(slow), c]: afterwards c is gone from both list_url_adapter_schemes() and routing for the rest of the process. Only reachable via the async API from multiple threads/loops (the sync API funnels through one IO thread), so low exposure — but a lock or a name-keyed dict of pending entry points would make it moot.

10. Wrapper adapters can't actually "consume adapter-specific keys" from storage_options (docstring at abc/url_pipeline.py:110-113 vs _url_pipeline.py:161). _resolver closes over the caller's original storage_options and resolve_preceding() takes no arguments, so there's no way to strip adapter keys before the root sees them; for a local/memory root the unused-options TypeError fires before the adapter runs (which test_storage_options_forwarded_to_root asserts), and for an fsspec root unknown keys reach the filesystem constructor. Native (preceding_url) adapters are unaffected. Either let resolve_preceding(storage_options=...) override, or drop the docstring claim.

Spec conformance at the root

11. Root sub-URL semantics are delegated wholesale to legacy make_store, which doesn't implement the spec's memory:/file: definitions (_url_pipeline.py:161). The spec says memory:, memory:/, memory:// and memory:a/memory:/a/memory://a are pairwise equivalent; with fsspec installed, make_store gives four different stores (memory: and memory:a become MemoryFileSystem keys under cwd, memory://a → /a), and different again without fsspec. file://localhost/tmp/x (≡ file:/tmp/x per spec) builds LocalStore("tmp/x") relative to cwd; file:relative/path (spec MUST NOT) is accepted; a query on file: is silently dropped. All pre-existing in make_store, but the PR now makes these the spec-mandated root of every pipeline, and test_spec_examples.py only exercises parse_pipeline — so conformance is asserted lexically while resolution disagrees. Notably memory://x|adapter: (the idiom throughout the PR's tests) is the only memory spelling that works as a root. The base case probably wants a small spec-aware root resolver for memory:/file:.

12. ?/# handling on bare-path roots (_url_pipeline.py:56-63, 90-96). URL lexical rules are applied to schemeless local paths, where those bytes are ordinary filename chars: parse_pipeline("/tmp/d?v=1|zip:") yields body='/tmp/d', query='v=1' while the base case passes raw to make_store, which strips ?v=1 and opens /tmp/d — no error, adapters inspecting preceding[0].body see something else. Simulated win32: \\?\C:\x|zip: splits into body='\\\\', query='C:\\x'. Either skip query splitting for schemeless roots or document that local roots may not contain ?/#. Related: _root_scheme uses urlparse (which strips leading whitespace and \t\r\n) while the body slice uses the raw string, so "\tfile:/x|zip:" yields scheme='file', body='\tfile:/x'; and roots with exotic authorities that the fallback regex accepts (vendor.x://[authority]/path, celebrated in test_exotic_authority_root_scheme) leak a raw ValueError: 'authority' does not appear to be an IPv4 or IPv6 address from parse_store_url if they ever reach the base case, instead of URLPipelineError.

Design / API surface

13. Root-adapter routing by entry-point name extends install-time scheme override to zarr's builtin routes. is_url_pipeline routes on entry-point names (unloaded included) with no builtin-scheme exclusion. fsspec already lets any installed package clobber s3/https/gcs via fsspec.specs, so that part isn't new — but file: and (without fsspec) memory: are, and the interception now happens in zarr before its own dispatch. Worth deciding whether builtin schemes should be excluded or an explicit opt-in required. Relatedly, collisions are silent: register_url_adapter overwrites unconditionally, get_url_adapter skips lazy loading when the key is already registered (so a builtin zip permanently shadows a third-party zip entry point that list_url_adapter_schemes keeps advertising), and two entry points with the same name → iteration order wins. A warning on collision (or override=) would help.

14. AdapterResolution has no channel for what the headline |zarr3: example must return. It's store + path only; a format adapter as last segment cannot express zarr_format, and make_store_path has nowhere to put it (the description says "StorePath gains a zarr_format attribute" but there's no such change in the diff). Adding a zarr_format: ZarrFormat | None = None slot later is source-compatible for constructors, but every wrapper adapter written against this release that re-wraps as AdapterResolution(store=..., path=r.path) will silently drop it. Since the entry-point contract is being frozen now, the slot (or an extras mapping, plus "use dataclasses.replace") probably belongs in this PR.

15. open_pipeline_segment runs on zarr's IO-loop thread, and the ABC doesn't say so. Under zarr.open("x|myscheme:") the adapter coroutine executes inside sync(), so any sync zarr call from an adapter (zarr.open, Group.open, anything using sync()) dies with SyncError: Calling sync() from within a running loop — reproduced with a trivial adapter — and blocking I/O in the adapter (zipfile central-directory read, sync fsspec.open) stalls every concurrent zarr op. Same constraint as Store._open, but this is a new third-party extension point whose most obvious implementation ("open the preceding thing with zarr, wrap it") is exactly the failing pattern; the docstring should say "must be non-blocking async, must not call the sync API; use resolve_preceding()".

16. The pipeline check is bolted on ahead of the existing dispatch rather than being a branch of it, and it's duplicated with diverging semantics (_common.py:357 vs :482 — only one has the "a" downgrade, only one has the residual-path check, only one enforces read-only), plus len(segments) == 1 and scheme not in list_url_adapter_schemes() is copy-pasted between resolve_pipeline and _resolve. zarr.open("memory://x") runs is_url_pipeline twice, parse_store_url three times, and builds the adapter-scheme set twice before any store exists (microseconds, but it's the duplication that matters). Routing on scheme once inside the existing parsed.scheme dispatch and letting make_store_path fall through to make_store would fix 3, 8, and 16 at once.

17. PipelineContext shape (abc/url_pipeline.py:118-122, nits): read_only is a stored field always equal to mode == "r" (a @property can't drift); _resolver is a required private field on a public frozen dataclass that adapter authors must supply in tests; frozen+dict is unhashable. Since the context already holds preceding/mode/storage_options, resolve_preceding could call the module-level _resolve(...) and drop _resolver and the per-segment closure.

18. Half-public surface. zarr.abc.url_pipeline and zarr.registry.*_url_adapter* are documented public, but parse_pipeline/resolve_pipeline live in zarr.storage._url_pipeline and aren't re-exported. Icechunk/tooling that wants to introspect a pipeline without opening a node has to import the underscore module. Either export from zarr.storage (and document) or state that zarr.open is the only supported entry.

Smaller things

• docs/api/zarr/abc/index.md doesn't list the new url_pipeline page (nav was updated). No user-guide entry describes the | syntax or how to register an adapter, though the checklist ticks it — maybe deferred to the adapter PRs?
• tests/test_url_pipeline/test_resolver.py: _store_of is unused; TracingStore.segment is set but never asserted. test_windows_drive_path_is_not_a_scheme only asserts segment.raw == input (true by construction) — could assert scheme under a sys.platform branch.
• The parser happy-path tests could be one parametrized test rather than ten functions.
• The description's "StorePath gains a zarr_format attribute (populated by format segments in a follow-up)" isn't in the diff (see 14).

[d-v-b]

a few more findings, this time from codex:

🤖 AI text below 🤖

A few additional findings on 6e6c2f5 that do not appear to overlap with the earlier review:

1. Rejected make_store() resolutions leak the opened store (src/zarr/storage/_common.py:357-364).

   When an adapter returns an open store with a nonempty residual path, make_store() raises URLPipelineError without closing that store. Reproduced with a close-tracking MemoryStore subclass:

   URLPipelineError
   closed=False open=True
   

   This could leak file handles, archive handles, or client sessions. The error branch should close result.store before raising, with a regression test using a tracking store.

2. The new property tests classify URLs outside the specification grammar as valid (tests/test_url_pipeline/test_parser.py:105-148).

   _BODY and _QUERY draw from nearly all printable ASCII, including raw spaces, backslashes, and malformed percent escapes. The scheme strategy also generates arbitrary unqualified nonstandard schemes. For example, both of these are accepted:

   file:/tmp/a b|zip:
   file:/tmp/%ZZ|custom:
   

   The URL-pipeline specification restricts the permitted URI characters and requires nonstandard schemes to use vendor.scheme naming. The public adapter example’s myscheme and the test entry point’s entrypoint-scheme also do not follow that convention.

   If parse_pipeline() is intentionally a permissive segment splitter rather than a grammar validator, that is defensible, but the property tests should not describe these generated values as valid/conforming. The validation boundary should also be documented explicitly.

3. The reference wrapper adapter discards the preceding residual path (tests/test_url_pipeline/test_resolver.py:39-50).

   It calls context.resolve_preceding(), wraps only preceding.store, then replaces preceding.path with segment.body:

   preceding = await context.resolve_preceding()
   store = TracingStore(preceding.store)
   return AdapterResolution(store=store, path=segment.body)

   Therefore a chain such as:

   root|wrap:a|wrap:b
   

   loses a for actual store operations. The API can represent this correctly, but the only wrapper example demonstrates the lossy pattern and there is no nested-wrapper resolution test. Before the zip: follow-up, it would be useful to document how wrapper adapters must consume preceding.path and add a test that proves residual paths survive nested wrappers.

For reference, the updated focused suite passes locally: 313 passed in 5.07s.

[maxrjones]

3. Change url-pipeline to eliminate the potential for ambiguous strings

what's the technical downside of this approach?

I'm asking because I think it's worth exploring all options before trying to switch people to a new API. The value of url-pipeline is being able to have zarr just work simply across varied storage types. requiring code changes for use reduces that value.

[d-v-b]

3. Change url-pipeline to eliminate the potential for ambiguous strings

what's the technical downside of this approach?

I'm asking because I think it's worth exploring all options before trying to switch people to a new API. The value of url-pipeline is being able to have zarr just work simply across varied storage types. requiring code changes for use reduces that value.

are you proposing changing the url pipeline spec itself? the basic premise of the pipeline is that there's a delimiter string (or sequence of strings) separating the store segments. using that delimiter string as a delimiter means it can't be used as part of a path any more. So I don't see a transformation of the spec that cleanly separates URL pipeline declarations from possible s3 objects or file paths. lmk if this is not what you had in mind.

I'm asking because I think it's worth exploring all options before trying to switch people to a new API.

why? new API for a new feature seems like a good starting point to me, especially when the existing API is problematic and has immediate conflicts with the new feature.

see this PR against joe's fork that adds new API for the URL pipeline. by completely separating URL pipeline declarations from legacy string declarations, we can remove code that existed only the disambiguate the two. I think this is an easier starting point that managing the inherent conflict in zarr.open.

[d-v-b]

and here's a second PR that covers array / group creation with strings interpreted as URL pipelines: jhamman#8. That could be a follow-up to the PR here. Since it adds new API, it's completely safe to merge without breaking existing code, unlike the proposed changes to zarr.open.

[jhamman]

Thanks @d-v-b and @maxrjones for working through this. I'd like us to agree on a plan we can merge. I don't have strong opinions on the exact API, with one exception:

Xarray users have to be able to use pipelines through the existing store= argument, e.g. xr.open_zarr("s3://bucket/repo|icechunk://branch.main/"), with no new function or type in xarray's public API. The whole point of url-pipeline is that a single string works everywhere a store string works today. If xarray needs a parallel open_url, most users will never discover the feature. Beyond that, I'm open.

There's a lot I think we already agree on:

1. Strings that work today should keep working. A plain path or URL that resolves on main shouldn't start raising. I take the | concern. I'll also note that | is invalid in URLs under RFC 3986, it's on AWS's "characters to avoid" list for S3 keys, GCS discourages it, and Windows forbids it in filenames. So I expect the affected population to be close to, if not, zero. If you have evidence to the contrary, I'm open to hearing it.
2. StorePath shouldn't gain zarr_format. I agree the format belongs to the request, not the location. I'll drop that change, and revert the create_array / Group.from_store default change that only existed to support it.
3. An unambiguous explicit form is good to have. A URLPipeline object along the lines of refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings jhamman/zarr-python#7 is useful no matter what we decide about strings.

Here's a proposal that I think gets everyone what they need:

• Strict string detection. A string is treated as a pipeline only if:

  • it contains |; and
  • every segment after the first starts with a registered adapter scheme followed by :.

  Anything else goes through the existing code path unchanged. That covers foo/bar|baz, a|b, and any | in a path where no adapter is involved. (This is Max's rule.)

• Fallback for the remaining ambiguous case. That leaves strings like file://./foo|zip:/test.zarr. If the literal path already exists, open it as before and emit a FutureWarning that points to pathlib.Path / LocalStore. Otherwise, resolve it as a pipeline.

• Explicit forms are always unambiguous:

  • pathlib.Path and explicit Store objects are always literal;
  • URLPipeline.from_url(...) (or similar) is always a pipeline.

• Format selection travels with the request (AdapterResolution / URLPipeline), not with StorePath.

Concrete asks:

• @d-v-b: would strict detection plus the exists-on-disk fallback address your breakage concern? If not, would the opt-in flag?
• Everyone: any objection to dropping StorePath.zarr_format and taking the URLPipeline type from refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings jhamman/zarr-python#7 as part of this?

If we can settle on one this plan, I'll rework #4192 to match. I'll fold in d-v-b's PoC where it fits and keep the format-adapter and zip follow-ups (#4476, #4477) rebased on top.

[maxrjones]

Thanks @d-v-b and @maxrjones for working through this. I'd like us to agree on a plan we can merge. I don't have strong opinions on the exact API, with one exception:

Xarray users have to be able to use pipelines through the existing store= argument, e.g. xr.open_zarr("s3://bucket/repo|icechunk://branch.main/"), with no new function or type in xarray's public API. The whole point of url-pipeline is that a single string works everywhere a store string works today. If xarray needs a parallel open_url, most users will never discover the feature. Beyond that, I'm open.

There's a lot I think we already agree on:

1. Strings that work today should keep working. A plain path or URL that resolves on main shouldn't start raising. I take the | concern. I'll also note that | is invalid in URLs under RFC 3986, it's on AWS's "characters to avoid" list for S3 keys, GCS discourages it, and Windows forbids it in filenames. So I expect the affected population to be close to, if not, zero. If you have evidence to the contrary, I'm open to hearing it.
2. StorePath shouldn't gain zarr_format. I agree the format belongs to the request, not the location. I'll drop that change, and revert the create_array / Group.from_store default change that only existed to support it.
3. An unambiguous explicit form is good to have. A URLPipeline object along the lines of refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings jhamman/zarr-python#7 is useful no matter what we decide about strings.

Here's a proposal that I think gets everyone what they need:

• Strict string detection. A string is treated as a pipeline only if:

  • it contains |; and
  • every segment after the first starts with a registered adapter scheme followed by :.

  Anything else goes through the existing code path unchanged. That covers foo/bar|baz, a|b, and any | in a path where no adapter is involved. (This is Max's rule.)

• Fallback for the remaining ambiguous case. That leaves strings like file://./foo|zip:/test.zarr. If the literal path already exists, open it as before and emit a FutureWarning that points to pathlib.Path / LocalStore. Otherwise, resolve it as a pipeline.

• Explicit forms are always unambiguous:

  • pathlib.Path and explicit Store objects are always literal;
  • URLPipeline.from_url(...) (or similar) is always a pipeline.

• Format selection travels with the request (AdapterResolution / URLPipeline), not with StorePath.

Concrete asks:

• @d-v-b: would strict detection plus the exists-on-disk fallback address your breakage concern? If not, would the opt-in flag?
• Everyone: any objection to dropping StorePath.zarr_format and taking the URLPipeline type from refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings jhamman/zarr-python#7 as part of this?

If we can settle on one this plan, I'll rework #4192 to match. I'll fold in d-v-b's PoC where it fits and keep the format-adapter and zip follow-ups (#4476, #4477) rebased on top.

thanks, @jhamman. Your proposal sounds good to me. I don't have a strong opinion on StorePath.zarr_format, so no objection from me.

We really need this feature for getting virtual Icechunk stores into CMR, so I'm really excited to see this move forward.

[d-v-b]

Xarray users have to be able to use pipelines through the existing store= argument, e.g. xr.open_zarr("s3://bucket/repo|icechunk://branch.main/"), with no new function or type in xarray's public API. The whole point of url-pipeline is that a single string works everywhere a store string works today. If xarray needs a parallel open_url, most users will never discover the feature. Beyond that, I'm open.

So the central design requirement is something for xarray users, not zarr users. IMO that suggests the solution: xarray handles the URL pipeline inference, and branches internally on whether the user-supplied string is a url pipeline, or a plain path. When a user invokes xr.open_zarr(string_that_is_url_pipeline), xarray calls zarr's url-pipeline-native open_url method. When a user invokes xr.open_zarr(plain_path), xarray calls zarr.open.

to make this easy for xarray, zarr needs to export everything required to parse a string as a URL pipeline + construct a store from it.

this keeps zarr.open unchanged, and thereby this feature would have no effect on zarr users who might today be using paths that are ambiguous. IMO this is a better outcome than changing how zarr.open handles strings, even if we think the impact on ordinary usage is vanishingly small, I don't want to risk breaking people's workflows / data if we can avoid it. And here I believe we can avoid it.

[d-v-b]

to be clear, I'm not saying we should be slow to land this. I also want this feature out, and I'm happy to move quickly to land it. I'd be happy working on a 3.5.0 release next week with the zarr side of the url pipeline changes. But IMO the correct sequence is to isolate the zarr changes from xarray changes, which we would land after 3.5.0 ships

[maxrjones]

hey @d-v-b, do you mind responding to Joe's question about whether an opt-in flag would resolve your concerns about breakages?

FWIW I appreciate your concerns about not breaking things, even for super rare use-cases. I also generally prefer having a deliberate, forward progression in our existing APIs rather than asking people to move to new APIs for new features.

[d-v-b]

opt-in flag

can you sketch out how this would work? i might have missed that explanation in joe's proposal

[d-v-b]

also, I should give some context for why I am deeply averse to increasing / encouraging use of zarr.open by overloading it, instead of baking these features in better-designed functions.

zarr.open has a lot of problems that have caused issues / confusion in the past:

• it has subtle mutation behavior. It will mutate storage (or erase it entirely) depending on the mode parameter. see

  zarr-python/src/zarr/api/asynchronous.py

  Line 408 in bd0dc84

  # TODO: the mode check below seems wrong!

  . The better thing is to separate the mutability of the created store (a property of the store you get) from whether opening the store is allowed to delete existing stuff (an action we take before creating the store).
• it has subtle zarr_format handling, because this is also controlled by a keyword argument which has a default value that's sensitive to the global config, so it's possible to accidentally create v2 data when you meant to create v3 data, and vice versa. The better option is to separate v2 and v3 access routines entirely, so that it's not possible to mix them up.
• storage_options is abstraction leakage from fsspec, which just one of many possible store providers
• it hides all the other keyword parameters with **kwargs, which mixes keyword arguments for creating a group with the keyword arguments for creating an array. creating an array and a group are very separate and should have distinct top-level routines

zarr.open is legacy zarr 2.x code that we kept around for backwards compatibility, and overloaded. It has many problems, and it has been a major source of issues / bugs in the past. We are long past the need for zarr 2.x compatibility. We can do better than a function that mixes reading, writing, overwriting zarr v2 or zarr v3 arrays or groups. that's the content of my second PR against' joe's branch.

especially if we are talking about xarray, and not zarr users, we have even more flexibility here.

[jhamman]

I remember advocating that we deprecate zarr.open at least 2 years ago so you wont get any push back from me on that point.

I do also care about this as a feature within Zarr and I think this plan risks divergent behavior between open_group/open_array and open_url. But what I'm hearing from @d-v-b is that he prefers that than a behavior change for the | for posix file systems? Do I have that right?

I'm going to pursue the new api surface of open_url and will use that in Xarray once available.

[d-v-b]

can we use a Sentinel here?

[d-v-b]

we might even already have an unset sentinel value

[d-v-b]

code smell: this is just StorePath without the methods. Can we re-use StorePath?

...when we define the same (Store, path) accessory class twice, that's a sign that we should just put a path attribute on the Store itself. Obviously out of scope for this PR but I would love to stop paying tech debt in this area. Something to think about when we iterate the store API

[d-v-b]

we only need PipelineContext and AdapterResolution here for type checking, so can we conditionally import them from _url_pipeline.py under a if TYPE_CHECKING block? Because those two classes are not abstract base classes, and URLPipelineAdapter clearly belongs in _url_pipeline.py because it is importing _resolve from there in a naughty way

[d-v-b]

some more issues found by claude:

Race in get_url_adapter at registry.py:613. Two threads doing the first lookup of the same scheme: the second sees the pending entry point already removed and the class not yet registered, and raises “no adapter registered”. The existing concurrency tests only cover different schemes.

Default zarr_format=3 conflicts with a zarr2: pipeline. Reproduced: Group.open(URLPipeline.from_url("...|zarr2:")) with no format argument raises “conflicts with”. The tests pin this as intended, but it is the same default-versus-explicit ambiguity your Sentinel comment points at, and a sentinel default would fix it without the config-dependent default you objected to.

zarr.open(pipeline) with mode=None always picks "a", so a read-only adapter store fails with “Store is read-only but mode is ‘a’” where the same store object passed directly would open read-only. open_url sidesteps this by forcing "r", but the xarray store= path will not.

No subclass validation in register_url_adapter or the entry-point loader. A bad entry point surfaces as an AttributeError mid-resolution rather than a URLPipelineError.

Smaller items: AsyncArray._create now mkdirs the store before rejecting a bad dtype; the file/memory adapter block only applies at the root position, not later segments; a disobedient adapter’s writable store is left open after a successful read-only conversion.

I'd be fine handling these in a follow-up PR, but we should make sure they are resolved before we release

[maxrjones]

I understand not wanting to build on zarr.open(), but IMO the biggest weakness of the open* functions is the mode parameter, which is super confusing for users and developers. So, I'm a bit concerned about adding another function with a mode keyword argument.

Do others share this concern? If so, to get an MVP out, could zarr.open_url() be moved to a distinct PR such that the following would work immediately:

from zarr.storage import URLPipeline

pipeline = URLPipeline.from_url("memory://pipeline-demo|zarr2:")
group = zarr.open_group(pipeline, mode="w")

[d-v-b]

I understand not wanting to build on zarr.open(), but IMO the biggest weakness of the open* functions is the mode parameter, which is super confusing for users and developers. So, I'm a bit concerned about adding another function with a mode keyword argument.

Do others share this concern? If so, to get an MVP out, could zarr.open_url() be moved to a distinct PR such that the following would work immediately:

from zarr.storage import URLPipeline

pipeline = URLPipeline.from_url("memory://pipeline-demo|zarr2:")
group = zarr.open_group(pipeline, mode="w")

totally agree!

[jhamman]

@maxrjones and @d-v-b - to clarify, are you asking for the following scope changes:

1. open_url moves to a follow-on PR
2. open_* accepts URLPipeline objects but not "URL pipeline strings"

Anything else?