Add 9p backends: diod over TCP + QEMU virtio-9p via virtme-ng (vagrant/QEMU stack) - #7
Open
yarikoptic-gitmate wants to merge 6 commits into
Open
yarikoptic-gitmate wants to merge 6 commits into
yarikoptic-gitmate wants to merge 6 commits into
Conversation
Design notes for adding 9p as a matrix row: run the suite inside a virtme-ng guest against a QEMU -virtfs export (the actual vagrant-libvirt synced-folder stack), with a diod-over-TCP localhost variant mirroring the NFS backend where host kernels allow it. Covers backend-script knobs, matrix/CI wiring (per-row runs-on, pinned guest kernel), provisioning, a Phase-0 feasibility probe for the open runner questions (/dev/kvm, 9p modules on azure kernels), and PR sequencing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
Two new backends put 9p -- the filesystem behind Vagrant/libvirt "9p" synced folders, WSL2 drive shares, and QEMU -virtfs in general -- into the matrix, per the reviewed design in drafts/9p-backend-plan.md: - bin/eval-under-9p-tcp: diod (9P2000.L) exporting a fresh dir on 127.0.0.1, kernel v9fs mount -- the NFS backend's architectural sibling; no VM. diod runs unprivileged in its single-user mode by default; --run-as-root switches to the multi-user export for the privileged suites (the 9p analog of --no-root-squash). - bin/eval-under-9p-virtio: QEMU's virtfs server (-virtfs local,...) into a virtme-ng guest booted from the host's own rootfs, suite runs inside the guest; exit status and stdio come back over virtio-serial. Guest kernel pinned in CI (refs: 9p-kernel, handed over by run-under.sh -- the backend itself never reads matrix.yaml and defaults to the host kernel for local runs). Needs no host root for the mapped-xattr row. --shell drops into a live guest for debugging. Wiring: two matrix rows (9p-tcp/n/a, 9p-virtio/mapped) on ubuntu-24.04 via a new optional per-backend runs-on key (default ubuntu-22.04 keeps BeeGFS rows untouched); install_9p_tcp/_9p_virtio in install-backend.sh (incl. the KVM udev rule and a pre-warm boot that moves the ~180 MB pinned-kernel download outside the suite timeout); 9p cases in run-under.sh (needs-root wiring, VM timeout, --share-rw for git's test-results) and dump-failure-logs.sh; guest log teed to /var/log/eval-under-9p-virtio.log and added to the artifact list; GOTCHAS gains both settings tables and the semantics that make red 9p cells readable (mapped-xattr absorbs privileged ops; QEMU answers every TLOCK with success while diod takes whole-file flocks; diod caps msize at 64 KiB); provision installs the new deps. Validated end to end in a container: TCG boot of the pinned v6.8 mainline kernel, virtfs device visible (--disable-microvm), 9p mount + usability probe, argument fidelity through vng's double quoting layer (printf %q + a dq-escape pass), env trio on the mount, exit-status propagation. shellcheck-clean; README grid regenerated (28 cells); REUSE and codespell clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
yarikoptic-gitmate
changed the title
On the hosted runners (fs.protected_regular=2) the sequence root-mktemp -> chown-to-invoker -> root '>>' append in sticky /tmp is denied, so diod never started and every 9p-tcp cell failed with "diod did not bind". The chown was never needed: the current shell opens the log fd and diod only inherits it, whatever uid diod runs as. Reproduced and verified both ways by flipping fs.protected_regular locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
The 9p-virtio/git cell died in check-chainlint: vng's --rwdir shares are not uid-faithful (measured on the runner stack and reproduced locally) -- writes arrive host-side as the sharing daemon's identity, so a non-root guest process gets EACCES inside a directory it just created. Let the suite's in-tree writes (chainlinttmp, test-results) go to the guest's uid-faithful tmpfs overlay instead, and add --copy-out: stage2 ferries the requested guest paths back through the 9p export after the command (failed runs included), and the host half restores them to their real paths with ownership normalized to the invoker, so the artifact upload can read them. run-under.sh now passes --copy-out for the git target's t/test-results; --share-rw stays as a power tool with the caveat documented in its usage and in GOTCHAS. Verified end to end locally: a dropped-user guest write into the overlay lands back on the host with content intact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
Both the wrapped command's own in-guest timeout and the host-side --vm-timeout surface as exit 124; the first CI git-annex hang showed the message wrongly asserting the VM was killed when the guest had in fact reported cleanly (its dmesg tail was right there above). Say both possibilities and how to tell them apart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
All eight 9p cells ran their suites end-to-end on the PR's matrix run; every red is a finding, not a harness failure, and the legacy twenty cells match master's baseline exactly. Recorded per house rule: - git-annex test hangs identically on both servers (testremote init fails, then unavailable-remote/removeKey blocks until the timeout) -- the pre-registered locking suspect; - stress-ng: fstat/ftruncate ENOENT on open-but-unlinked files; fallocate fails only on diod (server divergence), with v6.8 guest kernel WARNs in the v9fs fid-lookup path on the QEMU row; - pjdfstest: long-pathname 03.t scripts, the open/06.t flags matrix, and unlink/14.t (shared with the NFS row) on diod; virtio tally in the artifact; - git testsuite: 194 (diod) / 177 (virtio) failed assertions, t1050-large diod-only, t1517/t0450 shared and green on ext4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7
Member
|
apparently 9p is "no good" for sqlite3 even! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 9p — the filesystem behind Vagrant/libvirt
type: "9p"synced folders, WSL2 drive shares, and QEMU-virtfsgenerally — to the matrix, as two backends plus their CI rows, docs, and provisioning.What's in here
Design doc, reviewed (
drafts/9p-backend-plan.md): the original plan was adversarially reviewed by two independent passes (systems mechanics; harness fit). The doc is now the decision record: what was decided, what the reviews corrected (several load-bearing facts), and what is deferred.bin/eval-under-9p-tcp— diod (LLNL's 9P2000.L server, Ubuntu universe) exports a fresh dir on127.0.0.1; the kernel v9fs client mounts it. Architecturally the NFS backend's sibling; no VM. diod runs unprivileged in its documented single-user mode by default (root only for mount/umount);--run-as-rootswitches to the multi-user export — the 9p analog of--no-root-squash, wired totarget_needs_root().bin/eval-under-9p-virtio— the real vagrant/QEMU stack: QEMU's virtfs server exporting into a virtme-ng guest booted from the host's own rootfs, with the suite running inside the guest (host-installed targets exist there unchanged; exit status and stdio return over virtio-serial). Highlights:refs: 9p-kernel: v6.8, Ubuntu mainline build) because the v9fs client is the kernel and its semantics moved in 6.4/6.8; the backend itself never readsmatrix.yaml—run-under.shhands the pin over, and the local default is the host kernel (offline-friendly).mappedrow;--shelldrops into a live guest with the mount up for morning-after-a-red-cell debugging.--disable-microvm(vng's microvm has no PCI bus stock guest kernels can see), host-side--vm-timeout(the in-guest per-target timeout can't catch a boot hang), guest log teed to/var/log/eval-under-9p-virtio.log(uploaded as an artifact), and--copy-outferrying suite outputs (git'st/test-results/**) out of the guest through the 9p export itself — vng's--rwdirshares turned out not to be uid-faithful (measured; documented in GOTCHAS).Matrix/CI: rows
9p tcp (diod)and9p virtio (mapped)(28 cells total), onubuntu-24.04via a new optional per-backendruns-onkey (default stays ubuntu-22.04, BeeGFS rows untouched);install_9p_tcp/install_9p_virtio(KVM udev rule + a pre-warm boot moving the ~180 MB pinned-kernel download outside the suite timeout);dump-failure-logs.shcases; GOTCHAS settings tables and known-red entries; README grid + docs regenerated; Vagrant VM provisioning for all new deps.Matrix outcome (full run on this branch)
The 20 pre-existing cells match master's baseline exactly — 10 green, 10 documented known-red; no regressions from the shared-file changes (4-field backend tuple, per-row
runs-on, install-backend now sourcing matrix.sh).All 8 new 9p cells ran their suites end-to-end; every red is a documented finding (GOTCHAS "Known-red cells"), the highlights being:
git-annex testhangs identically on both 9p servers:testremote type git.initfails ("Not initialized"), thenunavailable remote / removeKeyblocks until the per-target timeout — the pre-registered locking suspect (diod's docs themselves warn record locking "will deadlock").fstat/ftruncate→ ENOENT), with a clean server divergence:fallocatefails on diod only. The pinned v6.8 guest kernel logs WARNs inv9fs_fid_lookup_with_uidwhile these run.03.tscripts, theopen/06.tflags matrix, andunlink/14.t(the same assertion the NFS row flags).t1050-largediod-only;t1517-outside-repo/t0450shared and green on the ext4 control row.Three harness bugs were caught and fixed along the way (each reproduced locally before the fix was pushed): the diod-log chown vs
fs.protected_regularon runners; vng--rwdiruid-unfaithfulness (replaced by--copy-out); and an ambiguous exit-124 message.Validation done before each push
End-to-end local runs of
eval-under-9p-virtio(TCG boot of the pinned v6.8 mainline kernel, virtfs device, mount + probe, quoting round-trip through vng's layers, copy-out with a dropped-user write);shellcheck bin/ci/*.sh bin/eval-under*;matrix-json.sh/gen-readme-matrix.sh --check; codespell;reuse lint;run-under.shtranslation exercised for all four new paths.Deferred variants (passthrough rows including vagrant-libvirt's true default of passthrough-under-unprivileged-QEMU,
cache=loose, virtiofs sibling) are recorded in GOTCHAS "Not yet covered" and reachable locally via flags today.🤖 Generated with Claude Code
https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7