Add filesystem probing and sshfs backend for eval-under - #5
Open
yarikoptic-gitmate wants to merge 16 commits into
Open
yarikoptic-gitmate wants to merge 16 commits into
yarikoptic-gitmate wants to merge 16 commits into
Conversation
Two scripts and a workflow that answer, empirically, which further filesystems this harness could grow a backend for. bin/ci/fs-capabilities.sh reports what a mounted filesystem actually supports, in the dimensions git-annex and DataLad are known to trip over: symlinks, hardlinks, fifos, exec/permission bits, POSIX record locks, flock, SQLite in WAL mode, xattrs, case sensitivity, timestamp granularity, and whether link() correctly refuses to clobber an existing name. Each check maps to a reported failure rather than to a generic conformance checklist -- sqlite-wal=no is the direct cause of "SQLite3 returned ErrorIO" on Lustre and WSL, and link-eexist=no is what makes annex.pidlock unsafe on Lustre. bin/ci/probe-backend.sh tries to stand up one candidate filesystem on a stock runner, runs the capability probe on it, and tears it down. The exit status is the finding: green means a bin/eval-under-<name> backend is possible on GitHub-hosted infrastructure, red means it is not, with the reason (no kernel module / no package / needs a cluster / needs a VM) in the log. Deliberately runs without `set -e` -- a probe that dies on its first failed command reports nothing. Candidates cover the cluster filesystems with a single-node mode (gfs2, ocfs2), loop filesystems the loop backend does not reach yet (f2fs, exfat, ntfs3, udf, nilfs2, bcachefs, zfs), the network/FUSE shapes users actually hit (glusterfs, cephfs, cifs, sshfs, s3 via rclone), the crypto layers (gocryptfs, encfs, ecryptfs), overlayfs, and Lustre's client modules. Validated locally where this container allows it: overlay and gocryptfs bootstrap and probe clean, gfs2 correctly reports the missing kernel module. The loop-backed and network candidates get their first real signal from the workflow.
First run answered the main question -- gfs2 and ocfs2 both come up on a stock runner in under 70s, cifs reproduces the SQLite failure mode in 23s -- but three cells failed without saying why, because a 50-line dmesg step buried the probe's own output past any reasonable log tail. - Drop the dmesg step and end every probe with a single machine-readable SUMMARY| line carrying verdict, elapsed time, note and the full capability profile, so `tail -3` of a job log is the whole finding. - glusterfs: print glusterd status and volume info, and register the brick under the node's own hostname. Gluster stores the brick host in the volfile and requires it to be a peer, which 127.0.0.1 need not be. - ecryptfs: echo what ecryptfs-add-passphrase actually printed, since the signature is scraped from it and an empty scrape is indistinguishable from an unsupported filesystem. - lustre-client: list the module packages the Whamcloud repo publishes. The first run showed the ubuntu2204 repo exists but has no lustre-client-modules-dkms; naming which kernels it *does* ship for turns that into a usable answer. - Add an openafs candidate: the other out-of-tree client in this space, and unlike Lustre it is packaged by Ubuntu, so DKMS builds it against the runner's kernel. - fs-capabilities: report chown as n/a when not running as root. chown(2) is restricted regardless of filesystem, so a "no" there said nothing.
The probe showed both cluster filesystems come up on a stock runner, so teach the existing loop backend to make them rather than adding two near-duplicate backends. Both are ordinary block-device filesystems once their single-node lock mode is selected at mkfs time: gfs2 -O -p lock_nolock -j 1 -J 8 ocfs2 -F -M local -N 1 -b 4K -C 8K --fs-features=local -q so the loop backend grows a per-filesystem mkfs option table (empty for everything it already supported -- the deliberate "distro defaults" choice GOTCHAS.md documents stays intact), an --mkfs-opts override, and a per-filesystem minimum image size. The last one is not cosmetic: both reserve journal space and neither fits in the 100MB default, so the size is raised to their 256MB floor unless an explicit --size says otherwise, in which case a too-small value is an error rather than a silent bump. GFS2 also gets lockproto=lock_nolock at mount time. install-backend.sh learns the matching packages (gfs2-utils, ocfs2-tools, f2fs-tools, exfatprogs, nilfs-tools, bcachefs-tools) and, for the two cluster filesystems, checks the kernel module is there -- both live in linux-modules-extra, so their absence would otherwise surface as "unknown filesystem type" at mount time. Both mkfs invocations are validated locally (this container has the tools but not the modules, so mkfs succeeds and mount reports ENODEV); a new backend-smoke job runs the real `eval-under loop --fs <fs>` path on a runner that does have them, with ext4 as the control. Also adds a roll-up job: each probe uploads its SUMMARY line and one final job renders them as a single table, so a run can be read without opening two dozen job logs.
The survey behind the loop-backend and probe changes: which filesystems have actually broken git-annex or DataLad for real users (from the trackers, with links), which of those can be stood up on a stock runner (measured by the probe workflow, not guessed), and what follows. Four mechanisms account for nearly every report -- SQLite unable to lock its WAL, absent POSIX record locks, link() that does not refuse an existing name, and no symlinks/exec bit -- which is why bin/ci/fs-capabilities.sh checks those specifically. Answers the two filesystems that prompted it. "GFS" is three different filesystems: GFS2 is nearly free (single-node lock_nolock on a loop device), GlusterFS bootstraps from Ubuntu packages, and GPFS is not possible in public CI -- IBM's free Developer Edition is real but sits behind an IBM ID with no redistributable repo, so it needs a self-hosted runner. Lustre is walled off twice on a GitHub runner: Whamcloud ships client modules prebuilt for 5.15 generic kernels while the runners are on 6.8-azure with no DKMS package to fall back on, and the server needs either a patched kernel or a ZFS OSD. The recommended path is Lustre's own llmount.sh single-node loopback config inside the Vagrant VM first, with a KVM-in-CI backend only if the answer proves interesting. GOTCHAS.md records the new gfs2/ocfs2 mkfs and mount knobs, and says plainly what a single-node cluster filesystem does and does not measure. README points at the survey and lists the new files.
The floor introduced with gfs2/ocfs2 support errored out when a size was passed explicitly, on the theory that an explicit request deserves an explicit refusal. That is wrong here: bin/ci/run-under.sh always passes --size, taken from target_loop_size_mb(), which is chosen per *target* and knows nothing about a filesystem's journal overhead. So promoting gfs2 or ocfs2 to a matrix row would have failed immediately with "--size 100 is below the 256MB minimum" -- on the very path the change was written to enable. Raise unconditionally and log it. No caller can be expected to know every filesystem's minimum, and the harness's job is to produce the smallest working filesystem of the requested type, not to litigate the request.
…easured The ceph demo container pulls about a gigabyte and then may never reach a responsive cluster; on the last run it held the whole matrix open until its 25-minute job timeout, which also blocked the roll-up that makes the run readable. Cap the wait at 4 minutes and dump the container's own log on the way out: "not on a runner" is a verdict, and it should arrive in four minutes rather than twenty-five. FILESYSTEMS.md: separate the Lustre facts that were measured from the ones that are cited. Measured -- the runner kernels (6.8.0-1064-azure on 22.04, 6.17.0-1022-azure on 24.04), no lustre-client-modules-dkms in latest-release/ubuntu2204/client, no ubuntu2404 client repo at all, and /dev/kvm present with qemu-system-x86_64 not installed. Cited -- that Whamcloud's index carries per-kernel module packages built for 5.15 generics. The earlier text asserted one runner kernel for both images, which was wrong.
Capping the poll loop at 4 minutes did not cap anything: with no reachable mon, `ceph -s` does not fail, it blocks for minutes inside its own retry loop, so 48 iterations of "sleep 5 then ask again" ran for as long as the job would let it. Two runs in a row ended with cephfs holding the roll-up open while every other candidate had long since reported. Wrap each ceph client call in `timeout` (and pass --connect-timeout), and bound the `docker run` that pulls the ~1GB demo image. The loop is now genuinely time-bounded: roughly two minutes of polling on top of a bounded pull, then a verdict either way.
Twenty-four candidates, one runner, real numbers. Sixteen bootstrap; four do not; and three of the measurements change what a row would be for, so they are called out rather than left in the table: - CIFS is the only candidate that fails both sqlite-wal and sqlite-delete-mode -- the exact mechanism behind the Samba and Lustre reports, reproduced in 26 seconds on a localhost share. - sshfs resolves five rapid creates into one distinct mtime where every other filesystem here gives five. Nothing in the matrix currently tests a coarse clock against git's racy-timestamp handling. - ntfs3 is *not* a crippled filesystem: symlinks, mode bits and xattrs all work, so it would test the in-kernel NTFS driver rather than being a second vfat. Also recorded as a negative: every single-node filesystem reachable from a runner passes link-eexist, the POSIX rule Lustre is reported to break. The check stays, but that behaviour cannot be reproduced without Lustre. The four failures now carry their measured reason -- openafs and lustre both fail to build out-of-tree modules against the runners' Azure kernels, cephfs never converges inside a bounded budget, and ecryptfs refuses the mount options the probe passes (a scripting gap, not a proven impossibility). Tier lists updated accordingly.
The matrix answers "is this filesystem broken?" on a schedule. Support needs the other question answered on demand: someone reports a problem on a filesystem we do not have a cell for, and we want that exact combination running now, with their mount options, without adding a badge that nobody will look at again. bin/eval-under-sshfs brings up a throwaway sshd -- its own port, host key, authorized_keys and pid file, all inside the run's scratch dir -- and sshfs-mounts a fresh backing directory back over it. Deliberately not the system sshd: that would mean appending to the user's ~/.ssh/authorized_keys and trusting teardown to remove it again. With --host it mounts a real remote instead, using the caller's ssh config, because for an sshfs report the interesting variable is often the server's SFTP implementation rather than sshfs itself. Two knobs matter enough to be flags rather than raw -o strings: --no-cache turns off the attribute cache that hides stale-stat behaviour, and --workaround rename makes sshfs emulate rename-over-existing by unlinking first -- a non-atomic rename, which is exactly what git and git-annex assume they can rely on. reproduce.yaml is the support entry point: backend, version, target, backend options and runner image are all dispatch inputs. No cron, no badge, no push trigger. It delegates to _test-under.yaml, which grows `runner` and `backend-options` inputs (both defaulted, so the twenty generated cells are unchanged). `capabilities` joins as an on-demand-only target: bin/ci/fs-capabilities.sh under any backend, seconds rather than minutes, and usually enough to explain a suite result before the suite is worth starting. It is listed in EVAL_UNDER_ONDEMAND_TARGETS rather than EVAL_UNDER_TARGETS, so gen-dispatchers.sh still generates exactly 20 cells. Validated locally: basic ops, --no-cache, --workaround rename (verified rename-over-existing still lands), clean teardown leaving no mount, no scratch dir and no stray sshd, and `run-under.sh sshfs n/a capabilities` end to end. shellcheck, YAML parse, gen-dispatchers --check and reuse lint all clean.
Adds a "Reproducing a report" section: capability profile first because it takes seconds and usually explains the failure, then the suite under the same backend, then the same thing from the Actions tab for people who would rather not have a local mount. Plus the sshfs backend in the CLI examples and the file-layout table.
Two things the first real use of the backend turned up. The port was fixed at 2222, so a second run while one was in flight died with "sshd did not start" -- which is exactly what happens when you try to reproduce something while a suite is already running, and would also bite two CI cells sharing a runner. Default is now the first free port at or above 2222; --port still pins it. The bigger one: `hardlink=yes` was measuring the wrong thing. On sshfs `ln` succeeds, and then both names report a distinct inode and nlink=1 -- the link exists on the server but is not observable as one through the mount, because SFTP has no way to express it. That is not a misconfiguration and there is no sshfs option to fix it (use_ino was FUSE2; sshfs 3.x only offers disable_hardlink, which goes the other way). It matters because git-annex's add on an adjusted unlocked branch hardlinks the file into .git/annex/objects and then verifies the link, and git's local clone does the same check. Both fail on such a filesystem while `ln` itself looks perfectly healthy -- so the old check reported a green cell for a filesystem where `git annex add` cannot work. hardlink-same-inode and hardlink-nlink measure that directly: both `no` on sshfs, both `yes` on ext4.
Measured with git-annex 10.20240129 against an ext4 baseline that passes the whole suite in 1m21s, so anything red here is the filesystem. The finding is one mechanism, not a list: SFTP cannot express that two names share an inode, so `ln` succeeds and then both names report distinct inodes and nlink=1. git-annex's add on an adjusted unlocked branch hardlinks content into .git/annex/objects and verifies the link; git's local clone does the same. Both fail. Locked repos and v10 unlocked pointer-file repos are unaffected -- it is the adjusted unlocked branch specifically, which is the variant DataLad leans on. Also recorded: the second-granularity clock (five rapid creates land on one or two distinct mtimes, against five everywhere else), and the absent fifos and unix sockets that git-annex notices at init and adapts to on its own.
…ed it The survey ranked sshfs on timestamp granularity. The real reason is better: it is the only candidate measured that breaks a git-annex operation outright. Recording how it was found is the point -- `hardlink=yes` called it healthy, and it took a real suite run to learn that `add` on an adjusted unlocked branch cannot work there. The probe now answers that in ten seconds instead.
Both full-suite runs stopped at Remote Tests / unavailable remote / removeKey and stayed there until killed. The ext4 control runs that same test in 0.02s and finishes the whole suite in 1m21s, so it is sshfs-specific. Worth writing down what it is *not*, since that is what a reporter would otherwise spend an afternoon on: the mount answers immediately, and git-annex has no open files on the mount, no sockets and no child processes -- its threads are parked in futex/epoll. That is an internal wait, not blocked filesystem I/O. Two caveats recorded with it: the test passes in seconds when run alone with -p, so it needs full-suite context to reproduce, and this was the distro's 10.20240129 rather than a daily build.
master replaced the generated per-cell dispatchers with one workflow over .github/matrix.yaml, so the three conflicts were all "this branch edited something that no longer exists in that form". - _test-under.yaml is gone; reproduce.yaml no longer calls a reusable workflow, it carries the same steps directly (minus the matrix fan-out and minus publishing -- a reproduction must never rewrite the status the README and report page show for master). - matrix.sh: took master's accessors-over-YAML version and taught the parser one new flag. A target marked `on-demand: true` lands in EVAL_UNDER_ONDEMAND_TARGETS instead of EVAL_UNDER_TARGETS, so target_known() accepts it while matrix-json.sh -- which iterates the latter -- generates no cell for it. `capabilities` moves out of the shell and into .github/matrix.yaml where the rest of the matrix now lives; the grid stays at exactly 20 cells, verified. - README: master's file-layout table, with the sshfs backend, the probe scripts, FILESYSTEMS.md and reproduce.yaml slotted in at its column widths. Also updated FILESYSTEMS.md's promotion instructions, which still told the reader to edit EVAL_UNDER_BACKENDS in matrix.sh and re-run gen-dispatchers.sh -- both gone. It is a data edit in .github/matrix.yaml plus gen-readme-matrix.sh now. Two codespell hits fixed, since master added that check after this branch forked: a "losd" link label (now zfsosd) and Samba's "browseable", which accepts "browsable" just as happily.
Opening the PR made two design mistakes visible. The probe exited non-zero for "NOT BOOTSTRAPPABLE", so lustre, openafs, cephfs and ecryptfs -- four candidates that are *permanently* unbootstrappable on a hosted runner, which is precisely the finding the probe exists to record -- pinned four red checks to the pull request. "Are this PR's checks green?" then has no answer, and the four reds say nothing a reader could act on. The verdict already lives in the roll-up table and the step summary, so the exit status was carrying no information. Genuine script errors still exit non-zero; they return before the verdict is recorded. The trigger was a bare `push`, so every branch push anywhere spawned ~25 reconnaissance jobs. Now: dispatch on demand, plus master pushes that touch the probe itself, to catch bit-rot in the bring-up recipes.
Author
CI on
|
| Cell | master eb99fc6 |
this PR d0c387e |
|---|---|---|
beegfs-7.4.6-git-annex |
❌ | ❌ |
beegfs-7.4.6-pjdfstest |
❌ | ❌ |
beegfs-8.1.0-git-annex |
❌ | ❌ |
beegfs-8.1.0-pjdfstest |
❌ | ❌ |
loop-ext4-git-annex |
❌ | ❌ |
loop-vfat-git |
❌ | ❌ |
loop-vfat-git-annex |
❌ | ❌ |
loop-vfat-pjdfstest |
❌ | ❌ |
loop-vfat-stress-ng |
❌ | ❌ |
nfs-pjdfstest |
❌ | ❌ |
beegfs-7.4.6-git, beegfs-7.4.6-stress-ng |
✅ | ✅ |
beegfs-8.1.0-git, beegfs-8.1.0-stress-ng |
✅ | ✅ |
loop-ext4-git, loop-ext4-pjdfstest, loop-ext4-stress-ng |
✅ | ✅ |
nfs-git, nfs-git-annex, nfs-stress-ng |
✅ | ✅ |
An exact match in both directions: no cell that is green on master is red here, and this PR turns nothing red that was not already red. matrix and codespell pass; publish is skipped, as intended on a pull request.
Most of the red is documented in GOTCHAS.md and is the point of the row rather than a defect — all four Loop vfat cells (no symlinks, no ownership, and a filename charset that git's own suite deliberately violates), and the pjdfstest cells. Two are worth chasing on their own, and neither belongs to this PR:
loop-ext4-git-annex— ext4 is the control row, soGOTCHAS.mdalready calls this a real bug rather than a filesystem property. It ran with a 100 MB image here (loop3: detected capacity change from 0 to 204800in dmesg), i.e. the per-filesystem size floor added in this PR is inert for ext4 and did not change what the cell does.- The BeeGFS
git-annexpair, red onmastersince before this branch.
I have not spent a re-run on any of these: comparing against master's own per-cell status is stronger evidence than a second attempt at a cell that is documented as reliably red.
One correction to the PR description above, which was written before this branch was merged with master: _test-under.yaml is not "updated" — master deleted it when the per-cell dispatchers were replaced by the single matrix workflow, so reproduce.yaml now carries those steps directly. Likewise capabilities is defined in .github/matrix.yaml (flagged on-demand: true), not in matrix.sh; the grid stays at exactly 20 cells.
Generated by Claude Code
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive filesystem capability measurement and introduces sshfs as a new eval-under backend, enabling testing against FUSE-based and remote filesystems.
Key Changes
New Filesystem Probing Infrastructure
bin/ci/probe-backend.sh: Reconnaissance tool that attempts to stand up candidate filesystems on GitHub-hosted runners and measures their POSIX capabilities. Supports 21 filesystem candidates (gfs2, ocfs2, f2fs, exfat, ntfs3, udf, nilfs2, bcachefs, zfs, overlay, glusterfs, cephfs, cifs, sshfs, gocryptfs, encfs, ecryptfs, s3-rclone, lustre-client, openafs, vm-only).bin/ci/fs-capabilities.sh: Fast capability profiler (~1 second) that checks for POSIX features known to break git-annex/DataLad: SQLite WAL support, fcntl locks, link() EEXIST semantics, symlinks, hardlink inode consistency, exec bits, permission bits, rename-over-existing, unlink-open behavior, and flock support. Output is machine-readable key=value pairs..github/workflows/probe-filesystems.yaml: CI workflow that runs probes across candidate filesystems on ubuntu-24.04 (and ubuntu-22.04 for kernel-module-dependent candidates). Collects capability profiles and generates a roll-up summary.New sshfs Backend
bin/eval-under-sshfs: Full-featured sshfs backend with two modes:--host): Mounts a directory on a named host using existing ssh config/agent, for reproducing reporter-specific setups.--no-cacheto disable attribute caching,--workaroundfor sshfs compatibility switches (e.g.,renamefor non-atomic rename-over-existing), custom mount options via--opt.Documentation
FILESYSTEMS.md: Comprehensive survey mapping real user-reported git-annex/DataLad failures to their root causes (SQLite locking, POSIX locks, link() semantics, crippled filesystems) and cross-referencing with probe results. Documents which filesystems have been measured and what they support.GOTCHAS.md(updated): Clarifies mkfs/mount settings for cluster filesystems (gfs2, ocfs2) using single-node locking modes, image size floors for journal overhead, and the approximation nature of testing cluster filesystems without a cluster.Framework Extensions
bin/eval-under-loop(updated): Added support for gfs2, ocfs2, f2fs, exfat, nilfs2 with appropriate mkfs options and image size floors. New--mkfs-optsflag to override defaults.bin/ci/install-backend.sh(updated): Added sshfs installation logic.bin/ci/run-under.sh(updated): Integrated sshfs backend into the runner orchestration.bin/ci/target-capabilities.sh: New on-demand target that runs the capability probe inside an eval-under wrapper..github/workflows/_test-under.yaml(updated): Addedbackend-optionsandrunnerinputs to support on-demand reproduction with custom mount options..github/workflows/reproduce.yaml: New on-demand workflow for reproducing specific filesystem+target combinations without waiting for scheduled runs.bin/ci/matrix.sh(updated): DefinedEVAL_UNDER_ONDEMAND_TARGETSto distinguish capabilities (fast, on-demand) from scheduled matrix targets.README.md(updated): Added references to FILESYSTEMS.md and examples of sshfs usage.https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK