Add local and remote Dandiset directory browsing

[AtomicGlance]

Browsing one directory currently requires listing individual assets. This adds get_path() to local and remote Dandisets, using the existing BasePath interface for navigation, file/directory tests, recursive counts, sizes and explicit asset retrieval.

Remote paths use /assets/paths/. Listed children retain their metadata, so inspecting their types and aggregate sizes does not fetch each asset. get_asset() retrieves the full record separately. Local paths share one discovery snapshot, preserving DANDI's treatment of hidden files, empty directories and Zarr assets.

This revision follows the review's proposed first step: it removes get_subject_ids(), moves the organized-path patterns into consts.py, and documents a subject-directory recipe. A layout API and BIDS-specific subject semantics are deferred. CLI listing and download behavior are outside this change.

Tests cover pagination, nested paths, cached child properties, blob/Zarr retrieval, missing paths, permission errors, empty roots and local discovery. An Archive integration test compares local and remote listings using text_dandiset. New tests carry ai_generated markers.

Local validation: directory tests 10 passed, 2 skipped; existing API module 30 passed, 44 skipped; Dandiset/organization tests passed with platform skips and existing expected-failure markers. Mypy, flake8, codespell and Sphinx pass. Docker integration tests and directory symlinks could not run in this Windows environment; CI must verify those.

Related to #1457.

[codecov]

Codecov Report

❌ Patch coverage is 84.08304% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.65%. Comparing base (31807c3) to head (1be1ce2).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/dandiapi.py	62.68%	25 Missing ⚠️
dandi/dandiset.py	62.00%	19 Missing ⚠️
dandi/tests/test_dandiset_paths.py	98.83%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1917      +/-   ##
==========================================
+ Coverage   77.45%   77.65%   +0.19%     
==========================================
  Files          89       90       +1     
  Lines       13397    13684     +287     
==========================================
+ Hits        10377    10626     +249     
- Misses       3020     3058      +38     

Flag	Coverage Δ
unittests	77.65% <84.08%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic]

The remote implementation streams asset records ordered by path and
extracts only nested top-level sub-* prefixes

this sounds too heavy (try on 000026) -- there is a dedicated API endpoint

so you can quickly get them

❯ time curl --silent -X 'GET' 'https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/paths/' -H 'accept: application/json' | jq -r '.results' | grep path
    "path": "README",
    "path": "dataset_description.json",
    "path": "derivatives",
    "path": "participants.tsv",
    "path": "samples.tsv",
    "path": "sub-EXC022",
    "path": "sub-HCPA1",
    "path": "sub-I38",
    "path": "sub-I41",
    "path": "sub-I45",
    "path": "sub-I46",
    "path": "sub-I48",
    "path": "sub-I52",
    "path": "sub-I53",
    "path": "sub-I55",
    "path": "sub-I56",
    "path": "sub-I57",
    "path": "sub-I58",
    "path": "sub-I59",
    "path": "sub-I60",
    "path": "sub-I61",
    "path": "sub-I62",
    "path": "sub-I63",
    "path": "sub-I64",
    "path": "sub-KC001",
noglob curl --silent -X 'GET'  -H 'accept: application/json'  0.00s user 0.01s system 3% cpu 0.357 total

so check if we interface that endpoint already and use that instead of listing all assets

[AtomicGlance]

Good catch — I switched the remote implementation to /assets/paths/, so it now reads only the immediate children of the Dandiset root instead of streaming every asset. It also distinguishes directory entries from root-level assets before parsing sub-* names.

I added coverage for the endpoint call and missing-version handling. The six focused subject-discovery tests, lint, and typing pass locally, and I verified the method against production Dandiset 000026 (20 subject IDs). The fix is in 77c39ba0; CI is rerunning now.

[AtomicGlance]

The matrix has finished: 35 checks pass, including lint, typing, docs, CodeQL, both Codecov checks, and all other test environments. The Windows/Python 3.12 job's only failure was the existing test_get_content_url timing out while reading dandiarchive.s3.amazonaws.com; the subject-discovery tests passed in that job. I tried to rerun the failed job, but GitHub limits upstream workflow reruns to repository maintainers. Could someone rerun that job when convenient?

[yarikoptic]

overall, instruct your agent to follow the files (I guess we should symlink into AGENTS.md or alike)

CLAUDE.md:- Testing requirements, including the **mandatory `@pytest.mark.ai_generated` marker on any test
DEVELOPMENT.md:- `@pytest.mark.ai_generated` — **mandatory** on any test written with AI

and I will request agentic review now

[yarikoptic-gitmate]

Thanks — the behaviour is right and the switch to /assets/paths/ was the correct fix. My concern now is about what we are exposing, and it is a design question rather than a defect list, so let me put that first.

Do we need the endpoint? Yes — but what's missing is a directory listing, not a subject list

There is no efficient way to do this with today's interfaces, because dandi-cli has no interface to /assets/paths/ at all. Measured on 000026:

approach	requests	payload	wall clock
assets/paths/ (root)	1	25 records	~0.5 s
get_assets(order="path") — the only existing option	410 pages	40,982 records, ~12 MB	minutes

So the capability is warranted. But assets/paths/ is not a subject endpoint — it is "list one level of a virtual directory, with recursive aggregate_files / aggregate_size per entry". get_subject_ids() calls it and throws away everything that makes it worth calling:

{"path": "sub-EXC022", "aggregate_files": 75, "aggregate_size": 78143012714, "asset": null}
{"path": "derivatives",  "aggregate_files": 1823, "aggregate_size": 384663181865, "asset": null}

Four things in the tree want that primitive today, and subject discovery helps none of them:

1. dandi ls <dandiset-url> cannot list one level. cmd_ls.py:140 prints only the Dandiset record unless --recursive, and dandi ls dandi://.../000026/sub-I38/ fans out over every asset beneath. A remote ls is currently not expressible.
2. Download size/count denominator. aggregate_size gives the progress total for a directory in one request; download_directory() (dandiapi.py:1535) instead materializes list(get_assets_with_path_prefix(...)) first.
3. Interactive/programmatic browsing — arguably the actual thing behind #1457: see what is in a Dandiset without pulling 40k asset records.
4. Subject discovery, as a three-line derivation.

What to expose: reuse BasePath, don't invent a type

We already have the abstraction, it is already public and documented, and LocalZarrEntry (files/zarr.py:300) is already a subclass of it: dandi.misctypes.BasePath (misctypes.py:67) — parts, name, parent, /, joinpath, plus abstract exists(), is_file(), is_dir(), iterdir(), size. That is exactly this endpoint's shape.

# dandi/dandiapi.py
@dataclass
class RemoteDandisetPath(BasePath):
    dandiset: RemoteDandiset
    parts: tuple[str, ...]
    _asset: RemoteAsset | None          # None => directory
    aggregate_files: int
    aggregate_size: int

    def is_dir(self) -> bool: ...       # self._asset is None
    def is_file(self) -> bool: ...
    def iterdir(self) -> Iterator[RemoteDandisetPath]: ...  # one paginated GET, path_prefix=str(self)
    @property
    def size(self) -> int: ...          # aggregate_size
    def get_asset(self) -> RemoteAsset: ...   # raises on a directory

class RemoteDandiset:
    def get_path(self, path: str = "") -> RemoteDandisetPath: ...   # "" == root

A @dataclass like RemoteZarrEntry, not APIBase — APIBase is for records deserialized wholesale from the API and its docstring calls itself an implementation detail.

One honest caution: RemoteZarrEntry dropped BasePath in 0.48.0 (dandiapi.py:2221) because per-directory listing was an N+1. Avoid the repeat by having iterdir() fully populate its children from the single paginated response, and by not pretending exists()/is_dir() on an arbitrary un-walked path is cheap.

Local symmetry: Dandiset.get_path() over find_dandi_files / Dandiset.assets(), not raw Path.iterdir() — find_dandi_files already knows a Zarr directory is one asset and that dot-dirs are not data, so local and remote then agree on what a "file" is, and one consumer implementation serves ls, size estimation and browsing on both.

Where the sub-* policy belongs

Not in dandiapi.py, not in dandiset.py, and not in utils.py. We already own this vocabulary in two places — consts.dandi_layout_fields (sub-, _ses-, _tis-, _slice-, _cell-, _desc-, _probe-, _obj-) and organize.LABELREGEX / ORGANIZED_FOLDER_REGEX — and that is the embryonic entity grammar a pybids-like layer would stand on:

# dandi/layout.py  (new; imports only re + consts)
def parse_dandi_path(path: str) -> dict[str, str]     # {"subject": ..., "session": ...}

class DandiLayout:
    def __init__(self, dandiset: Dandiset | RemoteDandiset): ...   # BasePath only
    def get_subjects(self) -> list[str]: ...
    def get_sessions(self, subject: str | None = None) -> list[str]: ...
    def get(self, *, subject=..., session=..., suffix=...) -> list[BasePath]: ...

That is the one seam where a BIDS Dandiset can later be special-cased (participants.tsv, derivatives/) or delegated to pybids — without Dandiset.get_subject_ids() having already promised "top-level sub-* directories" forever. For the record the heuristic is correct today: on 000026 the top-level dirs, participants.tsv, and assetsSummary.numberOfSubjects all say 20. The concern is the promise, not the answer.

Recommendation: split the PR

• PR A, now: move LABELREGEX / ORGANIZED_*_REGEX into consts.py; add the BasePath subclasses and get_path() on both classes; integration-test against the text_dandiset docker fixture; rewire download_directory / ls if cheap.
• PR B, later: dandi/layout.py + get_subjects(), once the BIDS-vs-organize question is settled. #1457 is answered in the interim by a two-line recipe over iterdir() in the docs.

Defects in the current diff, independent of the above

1. The remote helper is fed a full path, not a basename. AssetPath.path is the full path — ?path_prefix=derivatives returns derivatives/sub-I38, not sub-I38. parse_dandi_subject_dirname(item["path"]) is therefore correct at the root by accident, and returns None for every entry at any other prefix. The local side passes path.name. Same function, two contracts.
2. Same method name, different semantics local vs remote. Local silently drops symlinked directories and directories containing no file anywhere; remote does neither. The same Dandiset answers differently before and after upload. (find_dandi_files at least warns when it skips a symlinked dir — files/__init__.py:124.)
3. any(p.is_file() for p in path.rglob("*")) is a recursive walk per subject directory, on the method advertised as the cheap one, and it is a third definition of "populated": find_dandi_files uses any(p.iterdir()) (files/__init__.py:132). Same root cause as #1915 — discovery policy implemented twice, and these two will drift.
4. The regex is duplicated verbatim. parse_dandi_subject_dirname's pattern is character-for-character ORGANIZED_FOLDER_REGEX (organize.py:1160). The circular-import note is accurate as far as it goes (organize.py:31 imports dandiset), but the fix is consts.py — zero intra-package imports, already holds dandi_layout_fields — not a copy. And dandi.utils is a grab-bag that is itself in the published modref; it should not own DANDI layout semantics.
5. docs/source/modref/dandiset.rst promotes dandi.dandiset.Dandiset and AssetView to documented public API. That is a larger commitment than the feature and shouldn't arrive as a side effect of it.
6. Tests:
   • @pytest.mark.ai_generated is missing on all six new tests. DEVELOPMENT.md:250 makes it mandatory — same item as 7. in the #1915 review.
   • mocker.patch.object(Path, "is_symlink", autospec=True, ...) monkeypatches stdlib globally to fake one symlink. Call os.symlink; test_organize.py:147 already shows the pattern, Windows guard included.
   • The remote tests mock client.paginate wholesale and assert the URL string — a change-detector that never touches the only contract that can actually break here (asset: null ⇒ directory). Every neighbour in test_dandiapi.py uses the real fixture; one text_dandiset test with sub-* assets would cover it.
   • The 16 hand-rolled mkdir/touch calls are mkpaths() (test_files.py:38) — same item as 4. in the #1915 review.

---

Generated by Claude Code

[AtomicGlance]

I followed the proposed directory-listing direction and rewrote the title and description around it. get_subject_ids() and its duplicate parser are removed. Both Dandiset classes now expose get_path() through BasePath subclasses, and the organized-path patterns live in consts.py.

The remote implementation uses the endpoint's full paths and caches each listed child's count, size and asset reference. The endpoint only returns an asset ID/URL, so get_asset() explicitly retrieves the full record. Local browsing shares one discovery snapshot. I added the text_dandiset comparison test and marked all new tests as required.

Local directory tests, the runnable API tests, mypy, lint and documentation pass. Docker integration is left to the fresh CI run. I kept the layout API and CLI/download consumers out of this revision so the directory interface can be reviewed on its own.

[AtomicGlance]

I pushed 5007b30 with the remaining fixture and review fixes:

• the pagination mock now preserves path_prefix, includes count, and runs with DANDI_PAGINATION_DISABLE_FALLBACK=1, matching CI;
• the local/remote parity fixture writes non-empty files before uploading to the test Archive;
• the absolute-path assertion now calls joinpath() explicitly so CodeQL can analyze the exercised operation.

Locally, the path-browsing module passes (10 passed, 2 platform-dependent skips), including the strict pagination regression; Ruff and compile checks also pass. The Docker-backed parity case is now waiting for CI. Please take another look once the new matrix completes.

[AtomicGlance]

Follow-up: the lowest-dependency CI job exposed a test-fixture compatibility issue in the empty path_prefix= query matcher. I replaced the dependency-specific matcher with a raw-URL query matcher that preserves blank values, so the pagination contract is tested consistently across supported
esponses versions.\n\nValidation: python -m pytest -p no:cacheprovider dandi/tests/test_dandiset_paths.py -q (10 passed, 2 skipped), focused pagination test passed, and Ruff plus diff checks passed. The fix is pushed in e280a59; the new CI run is in progress.

[AtomicGlance]

One final CI cleanup: the new test helper needed an explicit callable return annotation for the repository typing gate. I reproduced the failure locally, added the annotation, and pushed 4bf0e7f. The focused mypy check now passes with no issues; Ruff and formatting remain clean. The preceding 27-job test matrix was fully green.

[AtomicGlance]

The final full matrix reached 26/27 successful jobs. The only failure is unrelated to this PR: the Ubuntu 3.11 NFS job completed 1,062 tests over 18 minutes, then errored in dandi/tests/test_delete.py::test_delete_paths[paths1-remainder1] because fscacher reported NFS persistence taking 0.63s. No path-listing or pagination test failed. I’m rerunning the failed NFS job to clear this environment-sensitive failure. @yarikoptic

[yarikoptic]

Thank you @AtomicGlance for working on these, but I wonder -- are you trying to address some specific use-case here or just "itching a scratch"? FWIW, we also have already higher level dandi ls interface but it does use the "recursive" listing end point so not really the same. But the point of mine here, is that due to any new code (and now it is very easy to "produce" code with LLMs) and especially data structure and interfaces adding additional maintenance etc burden, I would prefer to be considerate of adding code which otherwise would not likely to be used as not addressing a specific demand: there is a good number of interfaces already (the website, datalad dandisets, neurosift) to navigate dandisets, so not sure if worth adding explicit interfaces to the client ATM. Moreover with then unrelated stuff like those patterns for filenames...

Unless there is a clear demand for this, I would prefer to not push this development further ATM.

[AtomicGlance]

Thanks for taking the time to explain this — I understand the concern, and I agree that a new public interface should solve a concrete problem rather than just add another way to browse Dandisets. My motivation was a narrow use case in MetricProof-NWB: comparing subject-level organization locally and remotely without downloading asset contents. But I agree that this alone does not demonstrate broader demand, especially with dandi ls, Datalad, and Neurosift already available. don’t want to add maintenance burden for an interface that may not be used. I’m happy to stop development here and close the PR unless you think the underlying directory-listing primitive would be useful for a future, clearly requested feature. I also agree that the filename-pattern work should not be included as unrelated scope.

[yarikoptic]

yeah, I would likely skip this one for now... if demand comes - we know where to find it! Also may be we would find some existing formalization to make it adapter too (IIRC there was some notion of rpath in fsspec or somewhere... need to find what is nagging in the back of my mind)

[AtomicGlance]

yeah, I would likely skip this one for now... if demand comes - we know where to find it! Also may be we would find some existing formalization to make it adapter too (IIRC there was some notion of rpath in fsspec or somewhere... need to find what is nagging in the back of my mind)

Thanks, Yarik that makes sense. I’ll leave this one aside for now rather than push an API without a clear demand. The fsspec/rpath idea is useful context too; if I revisit it, I’ll first look for an existing formalization and build around that rather than inventing another one.
Since I’m working in the NWB/DANDI space, I also wanted to mention a small project I’ve been developing: MetricProof-NWB. It creates reproducible evidence bundles around NWB validation file hashes, validator/version provenance, findings, and deterministic reports so validation results can travel with a dataset or analysis.
I’d really value your opinion on whether that fills a useful gap in the DANDI/NWB workflow or overlaps with something that already exists. What’s your preferred way for me to stay in touch or ask an occasional question GitHub, DANDI Discussions, email, or another channel? I recently finished my bachelor's degree and wanted to get into neuroscience for my graduate studies I would love to have discussions with you on what to do, if you have time.

[yarikoptic]

a small project I’ve been developing: MetricProof-NWB.

oh -- looks very interesting and relating to a number of ongoing efforts

• evidence collection by @satra @tekrajchhetri et al in their ongoing https://brainkb.org
• our unified validation collection (from nwbinspector, bids-validator, ...) and overall effort to improve cross standards validation , see Please submit pointers to your validator output formats con/validation#1 and pointers there in
• at least "self-containment" and "tracking" of our https://stamped-principles.org
• passed/postponed https://github.com/dandi/dandisets-healthstatus, which @CodyCBakerPhD I believe is to a degree reviving within https://github.com/dandi-compute as we would also need to add 'server-side validation' (as beyond just metadata records validation)
• DataLad dandisets https://github.com/dandisets as they carry ultimate versioning info even for draft trees
• @neurovium started to work on https://github.com/dandi/dandi-dqp where likely that "passport" is somewhat a generalization over the "proof" you are collecting?
• ... likely othes ...

[AtomicGlance]

a small project I’ve been developing: MetricProof-NWB.

oh -- looks very interesting and relating to a number of ongoing efforts

* evidence collection by @satra @tekrajchhetri et al in their ongoing https://brainkb.org

* our unified validation collection (from nwbinspector, bids-validator, ...) and overall effort to improve cross standards validation , see [Please submit pointers to your validator output formats con/validation#1](https://github.com/con/validation/issues/1) and pointers there in

* at least "self-containment" and "tracking" of our https://stamped-principles.org

* passed/postponed https://github.com/dandi/dandisets-healthstatus, which @CodyCBakerPhD I believe is to a degree reviving within https://github.com/dandi-compute as we would also need to add 'server-side validation' (as beyond just metadata records validation)

* DataLad dandisets https://github.com/dandisets as they carry ultimate versioning info even for draft trees

* @neurovium started to work on https://github.com/dandi/dandi-dqp where likely that "passport" is somewhat a generalization over the "proof" you are collecting?

* ... likely othes ...

Thanks, Yaroslav! I really appreciate you taking the time to look at MetricProof-NWB and put together these pointers. I’m still finding my feet in this ecosystem, and hearing where it connects with work you’re already doing is very encouraging.

The passport idea in dandi-dqp particularly caught my attention. Do you see a useful role for a small tool that produces file-specific validation evidence to feed into that broader passport? I’d like to understand which parts belong in MetricProof-NWB and which would be more useful as contributions to the existing projects.

I’ll work through the links you shared. If you’re open to it, I’d love to come back with a small example and get your thoughts once I have something concrete. Is GitHub the easiest place to continue, or is there a community channel you’d recommend joining? or probably a google meet with you if you have the time.

Thanks again for the guidance and for your patience throughout the PR reviews. I’ve learned a lot from working through your feedback.

[yarikoptic]

The passport idea in dandi-dqp particularly caught my attention. Do you see a useful role for a small tool that produces file-specific validation evidence to feed into that broader passport?

potentially! but better to ask on dandi-dqp issues or discussions

I’d like to understand which parts belong in MetricProof-NWB and which would be more useful as contributions to the existing projects.

1c: Frankly, although such proofs, similarly to provenance information, are very important, it is not a common practice to produce/rely on them. So, such "proof'ing" should be automated and then available when desired/needed. It is our approach e.g. with projects like https://github.com/con/tinuous/ https://github.com/con/duct and even https://github.com/duecredit and others -- automate collection of those, so when potentially useful later -- those materials are available! So, I think you might consider such "automation of collection and cataloging for potential future use" to be one of the driving design and interaction principles: existing projects might be reluctant to adopt etc. YOU are the one who could establish that extra layer of "knowledge" and make it available to those projects to demonstrate utility etc.

I’ll work through the links you shared. If you’re open to it, I’d love to come back with a small example and get your thoughts once I have something concrete. Is GitHub the easiest place to continue, or is there a community channel you’d recommend joining? or probably a google meet with you if you have the time.

We have https://github.com/dandi/helpdesk where I think you could share / seek feedback! Also we have a slack you are welcome to join/share at -- you should have been invited when created dandi account. I think it would be nice for us to establish some DANDI community hour (zoom) at some point, but time is indeed scarse...

With that , I will for now close this PR -- its discussion already derailed quite a bit ;)

[tekrajchhetri]

@yarikoptic just wanted to update some progress (not yet deployed). I am mostly working on knowledge representation and the core-foundations. It should be soon available.

[AtomicGlance]

@yarikoptic just wanted to update some progress (not yet deployed). I am mostly working on knowledge representation and the core-foundations. It should be soon available.

Thanks for sharing this, @tekrajchhetri! Nice to meet you, and thanks @yarikoptic for connecting the projects here.

I’m developing MetricProof-NWB, and your work on knowledge representation makes me curious about how its validation records could fit into BrainKB. One question I’m thinking through is how to associate a finding with the exact file version and validator run that produced it, so it remains interpretable when either the data or the validation rules change. Is there a representation you’re working toward that I could learn from?

I’m still early in this work and would enjoy comparing notes. Once your foundations are available, I’d be happy to try mapping a small NWB validation example to them and share what I find. Is this a good place to continue the conversation, or is there a project discussion channel you prefer? So we can stay in touch

[tekrajchhetri]

@AtomicGlance Nice to meet you too. Yes, I have the underlying representation aka ontology. I have also come up with some formal proofs that set the boundary. I would be happy to share. Let's meet over zoom and see how we can combine things. I could also share more information + UI thing.

[AtomicGlance]

@AtomicGlance Nice to meet you too. Yes, I have the underlying representation aka ontology. I have also come up with some formal proofs that set the boundary. I would be happy to share. Let's meet over zoom and see how we can combine things. I could also share more information + UI thing.

Thank you, Tekraj! I’d love to meet and hear more about the ontology, the formal proofs, and the UI. It would be great to explore where our work could fit together.

I’ve just followed you here as well. What’s the best way to reach you to arrange a Zoom call, email or another channel you prefer? I’m happy to work around your availability. Looking forward to it!

[tekrajchhetri]

@AtomicGlance we can arrange zoom call. my email is tekraj@mit.edu. Happy to meet sometime this week.

[AtomicGlance]

@AtomicGlance we can arrange zoom call. my email is tekraj@mit.edu. Happy to meet sometime this week.

Would love to, just emailed you.