Add support for zarr multipart upload

[jjnesbitt]

• Supersedes Use multi-part upload on zarr chunks larger than 5GB #1839.
• Requires Re-work zarr multipart upload dandi-archive#2869.

What changed

Upload. multipart_upload() is factored out of LocalFileAsset.iter_upload() into dandi/files/bases.py, parameterized by upload_root (/uploads vs /zarr/uploads) and the matching init_fields, so Zarr entries reuse the blob machinery. No behavior change for asset blobs.

Digests. dandietag_nocache() and get_zarr_multipart_checksum() join md5file_nocache() and get_zarr_checksum(), sharing a tree walk via _zarr_checksum(). The Zarr's scheme selects between them everywhere an entry is digested — deciding what to re-upload, computing the checksum to report, and ZarrAsset.stat().

Download. A multipart Zarr's entries arrive with multipart ETags, which the previous code mis-verified as MD5. is_multipart_etag() now drives the per-entry algorithm, and the whole-Zarr comparison uses the matching checksummer.

Retries. The multipart and single-part batch loops were the same loop twice, one of them nested five levels deep inside iter_upload(). They are now one _upload_zarr_batch() generator differing only in a submit callback. Along the way: _retry_s3_upload (the 501/400 transient conditions) applies to blob part uploads too, and a part upload returning no ETag header raises an HTTPError carrying the response instead of a bare KeyError.

User-visible changes

get_digest(path, "zarr-checksum") is delineated based on zarr checksum scheme.

dandi digest -d zarr-checksum sample.zarr            # single-part Zarr (unchanged)
dandi digest -d zarr-checksum-multipart sample.zarr  # multipart Zarr

Since this command isn't in reference to a remote zarr at all (simply computing a local zarr checksum), we cannot assume a default. However dandi ls --metadata on a local Zarr now reports the multipart checksum, since that is what dandi upload will produce for it.

Compatibility

Against an archive without upload_type, the field is ignored on POST /zarr/, the response comes back without it, and the client uses single-part PUTs exactly as before. Backward compatibility is structural — the scheme is read from the server's response, not discovered by a failed probe. Uploading oversized content to a Zarr that is or must be single-part is rejected up front with an explanation rather than failing partway through.

Testing

The Zarr suite (test_files.py, test_upload.py, test_download.py, -k zarr) was run against both archives:

Archive	Result
dandiarchive/dandiarchive-api (current master)	42 passed
dandi-archive#2869 @ 947f55f4, built from dev/django-public.Dockerfile	42 passed

Verified from debug logs that each run took the path it should: against master, single-part presigned PUTs with Content-MD5, no call to /zarr/uploads/initialize; against the PR, every entry through initialize → complete → validate, with the server's ingested checksum matching the locally computed multipart checksum, and downloads verifying against multipart ETags.

Unit tests added for both digest helpers, both UploadItem digest behaviors, and the two dandi digest Zarr names.

Oversized entries, the threshold this PR exists to lift, are covered end-to-end by two tests over a Zarr whose single chunk is 1 MiB past S3_MAX_SINGLE_PART_UPLOAD:

Test	Asserts	Result
test_upload_zarr_oversized_entry	The 5 GiB + 1 MiB entry uploads; its ETag is a multipart ETag of more than one part, which a single-part PUT cannot produce; the archive's ingested checksum equals get_zarr_multipart_checksum() of the local tree	passed (61s)
test_upload_zarr_oversized_entry_to_singlepart_zarr	With the Zarr minted as singlepart beforehand (standing in for one created before the archive supported multipart), the upload is refused up front with UploadError	passed (1.6s)

Both are gated behind DANDI_TESTS_OVERSIZED_ZARR=1, since they cost several GiB of disk and need a multipart-capable archive; they skip in normal runs, including CI against the published archive image. Run with:

DANDI_TESTS_OVERSIZED_ZARR=1 pytest dandi/tests/test_upload.py -k oversized

The chunk is written directly to the array's chunk key rather than through zarr, so a single >5 GiB chunk never has to be held in memory, and it is filled with os.urandom so the digests under test are over incompressible, non-deduplicable content.

[codecov]

Codecov Report

❌ Patch coverage is 71.96262% with 90 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.69%. Comparing base (a069d5b) to head (b75b7fb).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/files/zarr.py	57.39%	49 Missing ⚠️
dandi/tests/test_upload.py	31.70%	28 Missing ⚠️
dandi/files/bases.py	82.45%	10 Missing ⚠️
dandi/consts.py	0.00%	2 Missing ⚠️
dandi/support/digests.py	97.43%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1925      +/-   ##
==========================================
- Coverage   77.94%   77.69%   -0.25%     
==========================================
  Files          91       91              
  Lines       13792    13980     +188     
==========================================
+ Hits        10750    10862     +112     
- Misses       3042     3118      +76     

Flag	Coverage Δ
unittests	77.69% <71.96%> (-0.25%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic-gitmate]

@yarikoptic folded the claude's review for @jjnesbitt 's agentic beast to expand:

Review of this PR together with dandi/dandi-archive#2869, since the digest contract spans both.

Summary: the digest design is sound and the refactoring backbone (extracting multipart_upload(), unifying the two batch retry loops, sharing _retry_s3_upload) is worth keeping. But I found two correctness regressions not mentioned in the description, a wrong-by-construction case for zero-byte entries, and a request-amplification concern that I think needs an answer before all new Zarrs are switched to multipart. I would not merge as-is.

What I verified as correct first

The core invariant holds. The archive never computes a Zarr checksum from client-supplied digests — _ingest_zarr_archive calls compute_zarr_checksum(yield_files_s3(...)) and yield_files_s3 uses obj["ETag"].strip('"') verbatim — so a Zarr's checksum is by definition the aggregate of whatever S3 stored. Reproducing that locally with DandiETag is the right approach, and the part sizing does agree: dandischema.PartGenerator.DEFAULT_PART_SIZE == s3_file_field.MultipartManager.part_size == 64 MiB, same 10k cap, same clamps (checked across every boundary size). validate_successful() compares client etag to the real object etag, so a divergence fails loudly. zarr_checksum needs no archive-side change — ZarrChecksum.digest is opaque and ZARR_DIGEST_PATTERN only parses directory digests. There is no cross-scheme digest comparison on any path I could find: new Zarr, re-upload into multipart, re-upload into single-part, both zarr_modes, replacing-a-non-Zarr, and the mkzarr() "already exists" fallback all stay consistent. Single-part batch behaviour is genuinely preserved through the retry rewrite (same budget, same worker halving, same backoff, same error text), and the new except BaseException: cancel futures is a strict improvement.

---

Blockers

1. The 409 handler was widened and now breaks asset-blob uploads. dandi/files/bases.py:370-384

On master the 409 catch wrapped only client.post("/uploads/initialize/"). It now wraps the whole multipart_upload generator. The archive returns 409 from two places:

• upload_initialize_view — with a Location header (dandiapi/api/views/upload.py:179-183)
• upload_validate_view — without one (:265-268, 'An identical blob has already been uploaded.', the two-clients-race case)

So a 409 from /validate/ — or from any part PUT or /complete/ — now reaches blob_id = e.response.headers["Location"] and raises KeyError: 'Location', masking the real error. The description says "No behavior change for asset blobs"; this is one. Fix: keep the 409 catch around initialize only.

2. The Zarr is never marked pending at multipart initialize. dandiapi/zarr/views/upload.py (archive PR)

The single-part path marks the Zarr pending the moment URLs are issued, with an explicit comment saying why (dandiapi/zarr/views/__init__.py:357-359: "Set status back to pending, since with these URLs the zarr could have been changed"). zarr_upload_initialize_view has no mark_pending() at all — it only happens at validate.

Sequence: initialize → parts uploaded → /complete/ writes the object into the live Zarr prefix → client dies before validate. The ZarrArchive keeps status=COMPLETE with a stale checksum/file_count/size, and nothing ever recomputes them — ZarrUpload.abort() is deliberately a no-op for a completed MPU, so GC won't either. Reachable on any re-upload into an existing COMPLETE Zarr. Suggest marking pending at initialize under the same select_for_update(of=['self']) that create_files uses.

3. Zero-byte entries produce a checksum no archive can ever match.

PartGenerator.for_file_size(0) returns zero parts, so dandietag_nocache() on an empty file yields d41d8cd98f00b204e9800998ecf8427e-0. But S3 stores an empty object under its plain MD5 under either scheme. So get_zarr_multipart_checksum is wrong by construction for any Zarr containing an empty entry, independent of upload — dandi digest --digest zarr-checksum-multipart, dandi ls --metadata, ZarrAsset.get_digest() and .stat() all emit a value nothing can reproduce.

On the upload path it fails earlier: contentSize = IntegerField(min_value=1) (dandiapi/zarr/views/upload.py:48) rejects it with a 400, which _upload_zarr_entry_multipart reports as "the archive may not support multipart upload of Zarr chunks" — misleading, though the appended Server response: does carry DRF's real message. Had it got past, zcc.add_leaf(..., it.digest) would drive iter_upload into RuntimeError("Unresolvable Zarr checksum mismatch") (dandi/files/zarr.py:940).

This works today on the single-part path (a presigned PUT of an empty body is fine, and create_files has no size validation), so it is a regression. Nothing filters empty files out anywhere on the path — I traced LocalZarrEntry.iterdir (skips only exclude_from_zarr names and empty directories) → register → _mkitem → UploadItem.from_entry → initialize.

The fix belongs in the digest function — return the plain MD5 for size 0 — not in the upload call. Related: _MULTIPART_ETAG_RE = r"[0-9a-f]{32}-[1-9][0-9]*\Z" (dandi/support/digests.py:118) excludes -0, which is effectively an undocumented workaround for this case. dandischema already owns the pattern as DandiETag.REGEX (-\d{1,5}), and digests.py already imports DandiETag.

Frequency caveat: I scanned the two real manifests available in dandi/zarr-manifests (266,659 entries) and found no zero-size entries, so this may well be rare in practice. Two Zarrs is a small sample though.

4. Request amplification — including ~4 S3 round trips per chunk made by the API server itself.

	today (single-part)	this PR (multipart)
DANDI API requests	1 per 255 chunks	3 per chunk (initialize, complete, validate)
S3 requests from the client	1 PUT per chunk	1 PUT per part + 1 POST complete
S3 requests from the API server	0 (presigning is local signing)	~4 per chunk
DB writes	1 mark_pending per 255	1 ZarrUpload INSERT + DELETE + 1 full-row ZarrArchive UPDATE per chunk
audit records	1 per 255, one record carrying all paths	1 per chunk, plus another on every retry

The server-side S3 row is the one I'd highlight: validate_successful() issues three separate HeadObjects (object_key_exists() → head_object, actual_size() → Object().content_length, actual_etag() → Object().e_tag), and initialize does a real CreateMultipartUpload.

For a 500k-chunk Zarr that is ~2 000 API calls today versus ~1.5 M, plus 500k audit rows and 2 M server-side S3 calls. Since all new Zarrs become multipart, every Zarr pays this, not just the ones with >5 GiB chunks. The design doc's requirement #3 covers checksum performance only; upload throughput for many-small-chunk Zarrs isn't addressed anywhere. I'd want a measured comparison on a realistic Zarr (≥100k chunks) before this lands.

Two client-side amplifiers of the same problem

• Session churn. multipart_upload() opens RESTFullAPIClient("http://nil.nil") at dandi/files/bases.py:739 — a fresh requests.Session per Zarr entry, so a new TLS handshake to S3 per chunk. The single-part path shares one storage session created once in iter_upload (dandi/files/zarr.py:816).
• Nested thread pools. _upload_zarr_batch uses jobs or 5 and multipart_upload uses jobs or 5 again, so 25 concurrent part PUTs per Zarr asset at defaults, and one ThreadPoolExecutor construction per entry. (Note --jobs 20 parses to (20, None), so jobs_per_file is None and both pools fall back to 5 — you need --jobs N:M to raise the inner one.)

On a possible "mixed" scheme

A size-threshold rule (≤ 64 MiB → plain MD5, > 64 MiB → multipart etag) is computable purely locally and needs no remote ETag lookups, so it avoids the performance objection the design doc raises against mixed. But it only works as a new third upload_type where the upload path picks by the same threshold — it cannot be applied to existing singlepart Zarrs, which legitimately store plain-MD5 ETags for entries all the way up to 5 GiB. Worth a sentence in the design doc either way.

---

Should fix before merge

• Every byte of an uploaded Zarr is hashed twice. EntryUploadTracker._mkitem / _cmp_digests compute the etag via dandietag_nocache(), then multipart_upload() recomputes it at dandi/files/bases.py:709 via get_dandietag(filepath). The latter is @checksums.memoize_path-decorated while the former is not and never populates the cache, so the second pass is a guaranteed full re-read plus one fscacher lookup+write per chunk — exactly what md5file_nocache's own docstring says fscacher was avoided for. Suggest expected_etag: str | None → etagger: DandiETag | None, since _upload_blob_part needs the object anyway (get_part(), get_part_etag()).

• The load-bearing claim has no automated coverage. dandi-cli's Zarr tests run against the published dandiarchive/dandiarchive-api image, which has no upload_type — so every existing Zarr upload test still takes the single-part branch, and the only multipart tests are gated behind DANDI_TESTS_OVERSIZED_ZARR. Archive-side, test_zarr_multipart_upload_initialize_and_complete lands an object in MinIO but never finalizes or ingests, so nothing checks that the ingested checksum equals the multipart tree. MinIO produces real multipart ETags, so a cheap archive test (two chunks via multipart → finalize → ingest_zarr_archive → compare against a locally built tree) would close this.

• LocalZarrEntry.get_digest() changed its return type from DigestType.md5 to DigestType.dandi_etag (dandi/files/zarr.py:352-365) — a public-API change on a documented class. Together with ZarrAsset.get_digest(), .stat() and cmd_ls.py:374 unconditionally reporting the multipart checksum, and migration zarr/0006 backfilling every existing Zarr to singlepart, this means these report a checksum matching nothing for the entire current archive. That's the majority case for the foreseeable future, not an edge case — worth at least a prominent CHANGELOG note, better an explicit scheme parameter.

• zarr_has_oversized_entry(): gate it behind not multipart (its only consumer at zarr.py:681 fires only when multipart is False, yet the walk runs unconditionally on every Zarr upload), move the check above the asset mint (currently it raises after POST .../assets/ / PUT, leaving a dangling empty Zarr asset), and reuse iterfiles() instead of a third os.walk implementation. Reusing iterfiles() also fixes a gap: os.walk does not descend symlinked directories while iterfiles does, so an oversized entry behind a symlink evades the guard, and os.path.getsize raises OSError on a broken symlink straight out of iter_upload.

Structure

The PR threads a bare multipart: bool through six call sites and writes dandietag_nocache(...) if multipart else md5file_nocache(...) out three times across two modules. DEVELOPMENT.md explicitly lists enum-based configuration as a key design pattern, and the archive already has exactly this enum (ZarrUploadType). Mirroring it in dandi/consts.py would collapse _zarr_is_multipart, the download.py scheme heuristic (the server exposes upload_type on both ZarrArchiveSerializer and ZarrListSerializer — no need to infer it from entry digests), the near-identical get_zarr_multipart_checksum / get_zarr_checksum pair with its copy-pasted docstring, the currently-dead ZARR_UPLOAD_TYPE_SINGLEPART constant, and the hand-rolled regex. Mostly deletion.

That also surfaces an inconsistency the duplication hides: the single-part path.is_file() branch uses the fscacher-memoized get_digest(path, "md5") while the multipart one uses uncached dandietag_nocache — for the one-off dandi digest <file> case both should be cached.

Smaller: docs/source/cmdline/digest.rst:13 still lists the old -d choices (the PR touches no docs at all, while adding a user-visible choice); DANDI_TESTS_OVERSIZED_ZARR bypasses the repo's dandi/tests/skip.py mechanism and isn't documented in DEVELOPMENT.md alongside the other DANDI_TESTS_* vars; _retry_s3_upload is now shared API imported across modules and should probably lose its underscore, as multipart_upload did; and UploadItem.base64_digest raising ValueError could use is_multipart_etag() rather than "-" in self.digest.

Archive-side, smaller (probably better raised on #2869)

• Migration 0034_backfill_audit_record_upload_type is one unbatched UPDATE ... jsonb_set(...) over an unindexed record_type, i.e. a seq scan plus full row rewrite in a single transaction on a growing table. Batch it, or skip the backfill and read details.get('upload_type', 'singlepart').
• clear_active_uploads (dandiapi/api/views/dandiset.py:670-673) deletes ZarrUpload rows without abort(), orphaning the S3 multipart uploads and destroying the rows GC would have used to abort them — which is what this PR added abort() to GC to prevent.
• Abandoned ZarrUpload rows now block unembargo. num_active_uploads() counts them, and the CLI abandons one on every retryable entry failure (RETRY_NEEDED re-initializes rather than aborting). With _url_expiration = timedelta(days=7), GC leaves them for a week, so kickoff_dandiset_unembargo can raise DandisetActiveUploadsError long after the upload succeeded. Before this PR Zarr uploads left no rows at all, so this failure mode is new.
• num_active_uploads() does 2 COUNTs (one joined) and is in VersionSerializer.fields, so it runs on every Dandiset listing row at page_size = 100. Both real consumers only need a boolean — EXISTS would do.
• _serialize_uploads sorts on the serialized created string; DRF omits the fractional part when microseconds are exactly 0, and '…:00Z' > '…:00.000001Z' lexicographically, so same-second ties can come back out of order. Sort on the datetimes already in page.

Nits

• Progress for a multipart entry only advances when the whole entry completes — _upload_zarr_entry discards the per-part statuses with for _status in ...: pass. For the single-multi-GiB-chunk Zarrs this PR exists to serve, the display sits frozen for the entire transfer. Not a regression, but newly visible in exactly the target case.
• The 400-from-initialize branch conflates at least five distinct causes (contentSize < 1, wrong digest algorithm, upload_type != MULTIPART, INGEST_ERROR_MSG, path traversal) into one "archive may not support multipart" message. Matching on the response body would be more honest. The 404 arm is also unreachable by construction: an archive without the endpoint also lacks upload_type, so the client never calls initialize.
• download.py:790-794 builds ETagHashlike(size) once outside the lambda, so retries reuse the same hasher (the hashlib branch builds a fresh one each attempt), and ETagHashlike.update raises ValueError when over-fed, which _download_file re-raises without retrying. Pre-existing, but before this PR that branch was only reached for blob assets — now it's taken for every entry of every multipart Zarr download.
• Per .autorc (intuit/auto) this needs its release label applied.

Out of scope, but caused by this PR

dandi/zarr_checksum's zarr_checksum local uses yield_files_local(), which computes a plain MD5 per file. Against a multipart Zarr it will silently produce a different checksum from the archive's, with no warning. It needs the same two-scheme treatment dandi digest just got — worth filing before this merges, since this is what makes that tool wrong. (dandi/zarr-manifests is fine — update_manifest.py feeds raw S3 ETags straight into ZarrChecksumTree, so it's already scheme-agnostic.)

Also worth a one-line correction in doc/design/multipart-zarr-chunks.md: it says the CLI "will check the upload_type field" for checksumming, whereas the implemented dandi digest has the user name the scheme — which is the right call, since no remote Zarr is in play there.

---

Checked and clean, so you're not left wondering: no URL collision between /api/zarr/uploads/... and the zarr router (lookup_value_regex is the UUID regex); no shared-index-name hazard from class Meta(BaseUpload.Meta) (Django deep-copies Meta.indexes, and zarr/0007 confirms distinct names); pre_upload_size_check/post_upload_size_check placement in multipart_upload is equivalent to master's; permissions on the three new endpoints mirror the existing blob ones exactly; and embargo tagging is correct, including unembargo_in_progress.

One more behaviour change worth noting explicitly since it isn't in the description: the refactor also adds retry_if=_retry_s3_upload to _upload_blob_part, so 501 "chunked encoding" and 400 "timeout period" responses on an asset blob part are now retried in place. An improvement, but it does change the blob path.

---

Generated by Claude Code