Repository navigation
image: Store filesystem snapshots in manifests - #406
daandemeyer wants to merge 7 commits into
Conversation
The manifest uses the wrong timestamp field and lacks the trailer that marks a complete sequence. Let's use mTimeNSec and require the trailer in readers and writers. Let's also move the format out of the image driver into a uapi16 package of its own. Other packages can then read and write manifests without depending on image/. Signed-off-by: Daan De Meyer <daan@amutable.com>
Buck directory artifacts lose inode metadata and hardlink identity. Let's keep complete filesystem manifests beside new content blobs and restore private roots before running commands, with declared hardlinks in the upper so index=off can't split them on copy-up. Signed-off-by: Daan De Meyer <daan@amutable.com>
Image layers lose package and operation metadata when Buck stores their directory artifacts. Let's capture snapshots in the producing action and restore them for image commands, solvers and terminals while keeping box layers on their direct mounting path. Signed-off-by: Daan De Meyer <daan@amutable.com>
Terminal archives and directory artifacts can lose metadata or mistake shared contents for shared inodes. Let's write archives from snapshots, keep terminal listings consistent with their outputs, and restore directory exports only after Buck has materialized their manifests and blobs. Signed-off-by: Daan De Meyer <daan@amutable.com>
Image reconstruction copies every regular file before commands run. Let's use private reflinks when available and metadata over shared blobs otherwise. Declared hardlink groups share private inodes. Capture uses upper inode identity when OverlayFS reports different inode numbers for paths linked by commands. Signed-off-by: Daan De Meyer <daan@amutable.com>
Box layers lose modes, hardlinks and xattrs when Buck stores them, so a child box installs over a parent that differs from what the package manager wrote. Let's capture every box layer as a snapshot and install a child box over its restored parent snapshot. The sandbox keeps mounting one plain directory per layer, which we export from the snapshot with only what Buck stores, so action startup doesn't change. Signed-off-by: Daan De Meyer <daan@amutable.com>
Buildroots and their BuildRequires deltas are still overlay uppers with whiteout and opaque markers encoded as files, while boxes and images use snapshots. Let's capture every install as a snapshot and restore it for plans and builds. rootfs then only needs bind mounts, and we can drop its marker translation. Every lower root of an install is now a restored snapshot, so the package installers no longer need to know whether they install over lower layers. Signed-off-by: Daan De Meyer <daan@amutable.com>
|
I did a strategic human review, sorted out some questions with Daan. Summary: upstream buck2 rejected our patches for mtime and hardlink support, but we want to eventually build with upstream buck. We don't care much about file owners and access mode, but hardlinks/mtimes require the framework anyway, so we might just as well cover everything. AFAICS file normalization is still in place (no suid binaries), the code just moved. At this size it needs bot support for reviewing. First round: pr-manifest-image-storage-review.md, and the referenced reproducer: repro2.py. Some findings we may just accept, e.g. discarding M4 seems reasonable to me; M9 is not that important either. I'd lift m1 to major, as the overlay whiteouts sound like an useful feature. One thing to watch out for: The default (Fedora) x86_64 test took 24.5 minutes here, but 15 mins on most recent PR and even less on the one before. arch took about 6 minutes more, debian is about the same. Let's see how long the next push takes, it may just have been a busy time? But the review above also found some performance regressions, in particular M2. Wrt. expected.json conflict: Feel free to break out "image: Write complete UAPI.16 sequences" into its own PR, that should be the only ones touching the digests, fairly independent, and quick to review. |
Buck directory artifacts lose inode metadata and cannot preserve every logical
filename. Let's capture complete UAPI.16 snapshots in image-producing actions
and store regular contents in immutable SHA-256 blobs. Derived images retain
parent stores as explicit inputs and add only newly introduced blobs. Metadata
changes affect cache keys without duplicating unchanged data. UAPI.16 records
use mTimeNSec and the required trailer.
Image roots first try explicit reflinks into temporary private files. Independent
files have separate inodes; declared hardlink groups share only their private
inode. Unsupported or denied reflinks discard the partial tree and select an
OverlayFS root with separate lower metadata and data-only blob layers. Writable
hardlink groups start as private upper files. Selection uses actual clone support,
so separate btrfs filesystems do not accidentally trigger full byte copies.
Read-only roots use either a read-only bind mount or a read-only overlay.
Unprivileged OverlayFS can report different inode numbers when a command adds a
hardlink to an inherited file. Capture uses the actual upper inode to preserve
that group in snapshots and exports. Repart and terminal readers that apply
tmpfiles use full private roots when their consumers need consistent raw inode
identity. Metadata restoration and later writes leave cached blobs unchanged.
Boxes retain their directory layers and direct startup path.
Archive writers consume manifests and blobs directly when possible. Terminal
listings describe the processed output, including package-database removal and
timestamp clamping. Directory outputs keep an authoritative manifest and provide
an exporter that restores metadata after Buck materialization.
Exports require a new destination under a parent owned by the caller or root,
with sticky protection when group or other write is allowed. Caller-owned
parents also work when Buck runs inside a user namespace.
Directory export needs read access to the kernel's UID overflow sysctl to
distinguish caller ownership from unmapped users.
On a same-filesystem btrfs input with 4,097 paths and 64 MiB of logical contents,
three-run median writable setup takes 162 ms with private reflinks and 139 ms
with a forced overlay. Chmod of 100 files takes 0.42 ms instead of 2.42 ms, and
100 partial writes take 13 ms instead of 20 ms. Read-only setup takes about
157 ms with either strategy. FIEMAP confirms shared extents for the private
files. Earlier non-reflinking tmpfs measurements cover memory copies rather
than ext4 disk I/O.
The native example digests change for the initrd, UKI, disk and three manifests.
The kernel, /usr filesystem, verity data, root hash and extension are unchanged.
Arm64 reference digests still require a native refresh. Device-node export
requires a filesystem and namespace that permit mknod; cpio retains whole-second
timestamps and cannot represent user extended attributes.