Skip to content

feat(init): Start the telemetry producer, embedded or from the initrd - #1

Draft
ananos wants to merge 1 commit into
stack/10-acpi-power-buttonfrom
feat/init-embedded-telem-capture
Draft

ananos wants to merge 1 commit into
stack/10-acpi-power-buttonfrom
feat/init-embedded-telem-capture

Conversation

@ananos

@ananos ananos commented Nov 16, 2026 •

Copy link
Copy Markdown

What

Both early-init roles start the in-guest telemetry producer (vmi's telem-capture) before the root is mounted, when the kernel offers /dev/telem, and wait for it to report ready. The producer can be embedded in urunit, so an introspecting initrd stays one file.

So far only urunc's legacy BusyBox /init started the producer, which is why brig-standalone-linux-priv still builds that layout instead of the single-binary initrd hull-assets ships.

How

  • make static TELEM_CAPTURE=<path> embeds a prebuilt static producer for the target architecture: .incbin into .rodata between urunit_telem_capture_blob and _end, assembled with --noexecstack. init.c declares the symbols weak, so a build without TELEM_CAPTURE leaves them null and nothing else changes; the init then falls back to /vmi-telem-capture in the initrd, and does nothing when neither exists.
  • The producer runs from a memfd (embedded) or the initrd file through fexecve, as a child of PID 1 while the initramfs is still the root, so switch_root deleting the file does not matter. The reaping loop collects it like the agent. Its output goes to /dev/kmsg.
  • tracefs is mounted first (the producer lays its probes out from the tracepoint format files). The producer gets --ready-fd on a pipe (NOFireAI/vmi#295) and the init waits up to 3 s for the byte: ready, exited first (EOF) or stalled (timeout) are each logged; the boot goes on in every case.
  • Gated on /dev/telem, so the same initrd serves a stock and an introspecting kernel.

Verification

  • make static check with glibc (Ubuntu, nofire) and musl (Alpine 3.21): PASS. New unit test test_capture_wait_ready covers byte, end of file and timeout.

  • End to end on nofire (x86_64, KVM): hull-assets introspection kernel 6.18.60 (NOFireAI/hull-assets#35 CI artifact, CONFIG_TELEM=y, modules off), cloud-hypervisor-vmi at the fork commit vmi 2594a64 pins, sidecar and producer from vmi-binaries at 2594a64. Initrd: urunit with the producer embedded, three links, mount-point dirs, 1.7 MB. Console:

    [    0.186356] telem: ready phys=0x1b400000 rings=1 ring_size=262144 header_bytes=4096 generation=1
    [    0.256926] capture: attached; forwarding events to /dev/telem -> export ring
    container-init: telemetry producer ready (pid 44, embedded)
    [    0.257560] capture: readiness reported on fd 5
    

    Sidecar: attached to the telemetry region, records=1845 drops=0 seq_gaps=0; 117 nofire.vmi.process.exec records naming the workload's ls, cat, sync, sleep and /bin/sh /work.sh.

Base

Stacked on stack/10-acpi-power-button, the branch hull-assets pins (48a61bd); main is 20 commits behind it.

Follow-ups

  • hull-assets: build-urunit.sh passes TELEM_CAPTURE from vmi-binaries at the pinned VMI_REF for the introspection bundle variant.
  • The producer's stdout/stderr go to kmsg; a quieter default once the path is routine.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HEbPad1TPYJvLXQDpypFKT

An introspecting guest runs vmi's telem-capture from boot: it fills the
export ring the kernel offers at /dev/telem, and is the source of every
rich event the host sidecar sees. So far only urunc's legacy BusyBox
/init started it, which is why brig-standalone-linux-priv still builds
that layout instead of the single-binary initrd hull-assets ships.

Both early-init roles now start it, right after /dev, /proc and /sys
are mounted and before the root, when /dev/telem exists. It comes from
one of two places: the bytes a build with TELEM_CAPTURE=<path> embeds in
urunit (.incbin into .rodata between two symbols init.c declares weak,
so a build without them leaves the symbols null and nothing else
changes), run from a memfd; or /vmi-telem-capture in the initrd. Either
way it is fexecve'd as a child of PID 1 while this initramfs is still
the root, so switch_root deleting the file does not matter, and the
reaping loop collects it like the agent.

tracefs is mounted first, since the producer lays its probes out from
the tracepoint format files. The producer is given --ready-fd on a pipe
and the init waits up to three seconds for its readiness byte, so the
workload starts under capture; a producer that exits first (end of file)
or stalls (timeout) is logged and the boot goes on without the rich
channel. A kernel without /dev/telem boots as before, so one initrd
serves a stock and an introspecting kernel.

Tested on an x86_64 KVM host with the hull-assets introspection kernel
(6.18.60, ring built in) under cloud-hypervisor-vmi: the producer is
ready 50 ms after fork, embedded, and the sidecar drains the workload's
exec and syscall records with no drops. Unit tests cover the readiness
wait: byte, end of file, timeout.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant