Add DESIGN.md and IMPLEMENTATION.md for eval-under action - #6
Open
yarikoptic-gitmate wants to merge 1 commit into
Open
yarikoptic-gitmate wants to merge 1 commit into
yarikoptic-gitmate wants to merge 1 commit into
Conversation
Design docs only -- no code yet -- so the shape can be reviewed before anything is built. DESIGN.md (what and why): the action splits in two, because an action runs inside one job and looping backends inside it would serialize the mounts, collapse five verdicts into one, and lose per-cell re-runs. `actions/run` wraps one command under one backend; `actions/list` emits the backend catalog as a matrix and lets GitHub do the looping. Both speak the backend slug `backend_slug()` already produces. Also covers: capability tags on the backend rows (with the rule that tags scope a sweep and never predict its outcome); why that table is measured in our CI and committed rather than probed in consumers' CI; the three target consumers (fscacher, datalad, dandi-cli) and why the adoption order is feedback-loop length rather than value; the constraints that forced the shape (no `post:` on composite actions, no expressions in `uses:`, sudo must not wrap the action, the house rule against shell logic in workflow YAML); and act compatibility scoped to plumbing rather than filesystems, which is what `eval-under-none` makes achievable. IMPLEMENTATION.md (how): component sketches, plus the facts already established so nobody re-derives them -- each carrying its provenance. Notably: tox passes TMPDIR/HOME by default but no custom vars, so fscacher needs no tox.ini change while any extra var we set does need a passenv entry; the backends already run the wrapped command as the invoker, so the action must not be wrapped in sudo; and the vfat experiment where two simulations of the same filesystem gave opposite answers about fscacher going stale -- recorded as an unverified hypothesis, since this container's kernel has no vfat to settle it. That failure is also why `ctime-advances-on-write` is in the probed feature set: it is the field that decided the outcome, and nobody would have declared it by hand. CLAUDE.md gains the rule that the two documents are a pair: interface, constraint and ordering changes land in both in the same commit, DESIGN.md wins on disagreement, and recorded facts keep their [verified]/[unverified] provenance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TFwTzwKHoApnMTRfGpbWXf
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR establishes the design and implementation roadmap for packaging
eval-underas a reusable GitHub Action. It adds two complementary documentation files that define the feature scope, constraints, and sequencing for making eval-under available to downstream projects (fscacher, datalad, dandi-cli).Changes
DESIGN.md (276 lines): Defines the what and why of the feature
actions/run(wraps one command under one backend) andactions/list(emits filtered backend catalog)IMPLEMENTATION.md (308 lines): Defines the how, measurements, and constraints already established
actions/run/action.yml,actions/list/action.yml,bin/eval-under-none, feature probe, selftest target.claude/CLAUDE.md(updated): Adds guidance on keeping DESIGN.md and IMPLEMENTATION.md in syncNotable Details
[verified: read of bin/eval-under-{loop,nfs,beegfs}])https://claude.ai/code/session_01TFwTzwKHoApnMTRfGpbWXf