Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ What the tail names as the machine's state when the records stop:
Between those, cpu5 and then cpu0 repeat `usb-storage: 00:14.0 slot 5
transport broke on SCSI 0x28` and `SCSI 0x28 broke 3 times running; the
transport is not coming back on its own`: the boot stick's READ(10) transport
broke at 3.47 s, the root filesystem stopped answering, `logd` and `netd` died
broke at 3.47 s, the root filesystem stopped answering, `logkeeper` and `netstack` died
on page faults over file bytes the device would not read, the test runner
could not be loaded, and the disk went offline at 5.556 s. A boot with no
runner asks for no reboot; the deadline was the only bound left, and it fired
Expand Down
4 changes: 2 additions & 2 deletions issues/a-boot-with-no-kernel-log-is-not-no-boot-complete.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ opened: 2026-09-08
Run 22's hung mkdir image left the stick holding `attempts` and `loader.log` and
**no kernel log file at all** — where every earlier hung boot (run 19's three)
left a 325-line file ending at a spawn. `loader.log` shows an ordinary handoff,
so that kernel started; `logd` either never created its file or its first write
so that kernel started; `logkeeper` either never created its file or its first write
never completed. The stick after the power cut is
`t14-run22/stick-after-cycle.log`, outside the tree.

`tests/metal/src/lib.rs` reads that back as the absence of `Boot: complete`, which is the
same verdict it gives a boot that ran and stopped late. They are not the same
machine: a boot with no kernel log wedged before its
first durable record, which places the failure before `logd`'s first `SYS_FSYNC`
first durable record, which places the failure before `logkeeper`'s first `SYS_FSYNC`
returned rather than anywhere in the job list — the single most useful thing the
readback could say about the run-22 class of hang.

Expand Down
2 changes: 1 addition & 1 deletion issues/a-claims-own-refusals-are-read-by-nothing.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ A one-field mutation of any of them — an index bound compared `<=` rather than
green.

Owned by whoever next adds a boot config whose own test binary holds a claimable
function: netd holds this machine's only claim, and a test that makes it fail
function: netstack holds this machine's only claim, and a test that makes it fail
this way costs the machine its NIC for that boot. Exit condition: a guest arm in
which a test binary's claim is refused each of these ways, each one red against
a kernel whose corresponding check answers `Ok`.
18 changes: 9 additions & 9 deletions issues/a-client-cannot-tell-soundserver-it-paused.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,20 @@ kind: track
opened: 2026-08-01
---

# A client cannot tell soundd it paused, so soundd cannot tell paused from wedged
# A client cannot tell soundserver it paused, so soundserver cannot tell paused from wedged

soundd infers a dead client from its signal event alone: a ring answering
soundserver infers a dead client from its signal event alone: a ring answering
`Hungup` is a gone process, and any other ring coalesces into a signal the
client has not taken yet. That detects exactly one failure. A client that
stops making progress while holding its handle — a wedged thread, a deadlock,
a callback that never returns — takes no signal forever and is
indistinguishable from one that deliberately paused. The ambiguity is not a
soundd bug: pause is defined as the client's stream thread simply not reading,
soundserver bug: pause is defined as the client's stream thread simply not reading,
with no coordination, and the cpal backend confirms it — `pause()` stores a
state word and parks on a futex, and soundd is never told.
state word and parks on a futex, and soundserver is never told.

The cost is that `is_streaming` latches on the first delivered period and never
clears, so one paused client holds soundd out of idle for the life of the
clears, so one paused client holds soundserver out of idle for the life of the
process: a wake per period forever, the DMA engine running, the codec voice
open, and the client's shm, signal event and slot ring pinned.

Expand All @@ -33,18 +33,18 @@ deferral window the mix loop tolerates — a number to measure, not to invent.

**Blocked on the cpal fork**, which needs a clone beside the monorepo and
`.cargo/config.toml` path overrides — a file every agent in a working tree
shares. Landing the soundd and SDK halves alone is *worse* than not landing
shares. Landing the soundserver and SDK halves alone is *worse* than not landing
them: every cpal client that pauses would declare `STREAMING`, stop producing,
and be killed by the new deadline. That is an audible regression.

**One edge unblocks three things.** Suspending on no progress is not landable on
its own either, and the reason is worth keeping: soundd's wait has exactly two
its own either, and the reason is worth keeping: soundserver's wait has exactly two
sources plus a timeout, and its command channel carries only add/remove/set-volume
— **there is no resume message**. A suspended soundd with the device stopped and
— **there is no resume message**. A suspended soundserver with the device stopped and
`timeout = u64::MAX` waits for completions that never come while the resumed
client blocks forever on a signal byte nobody writes: a battery defect traded
for permanent silence. Killing wedged clients, suspending on no progress and
resuming from suspend all need the same client→soundd notification, sent from
resuming from suspend all need the same client→soundserver notification, sent from
cpal's `play()`.

The one variant that needs no client change is filed apart, at
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,9 @@ started. Nothing reaches userland when a device arrives or leaves.

Rulings that bound the stages: the ownership and event model (owner,
2026-10-13); PCIe hot-plug as stage 2, native only, with physical arrival
witnessed through `issues/toyos-supports-thunderbolt-and-usb4.md`, and the
claim-slot limit removed by stage 1 if cheap, otherwise by its own pull
request before stage 1's guest smoke
(`issues/a-cargo-run-boot-claims-every-pcidev-slot.md`) (orchestrator, on
#910); and one shared block service to which disk drivers offer their disks
witnessed through `issues/toyos-supports-thunderbolt-and-usb4.md`
(orchestrator, on #910); and one shared block service to which disk drivers
offer their disks
(orchestrator, 2026-10-20, replacing #901's `block@<location>` ports). The
supervisor starts the `drives` processes before the storage rows and tells
the block service how many disk drivers it started; an open of a partition no
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ and run 56's transcript opens, at 21:29:39Z, with
— a copy of `/dev/sda3` taken from outside the loop at 21:15Z, 35651584 bytes,
which is the log partition's 69632 sectors (`log p3 at 151552+69632`, the
loop's own line about the image) whole. Run 56's `flash` then overwrote it.
The copy held two files, `loader.log` and `attempts`, and no file `logd`
The copy held two files, `loader.log` and `attempts`, and no file `logkeeper`
wrote. Its `loader.log` is the loader's first pass alone, 105 lines ending in
`Loader log: the kernel handoff begins, so this file ends here`, and its line
20 reads
Expand All @@ -56,7 +56,7 @@ wrote. Its `loader.log` is the loader's first pass alone, 105 lines ending in
— the boot whose `WEDGED` record run 56's loader pass then printed
(`issues/a-120000-ms-boot-deadline-fired-132859-ms-late-on-the-t14.md`).
So for the one hung boot whose partition was captured, the flash would have
destroyed nothing `logd` wrote, because `logd` wrote nothing: that boot's ring
destroyed nothing `logkeeper` wrote, because `logkeeper` (then `logd`) wrote nothing: that boot's ring
tail carries `exit: logd pid=4 code=-1 cpu=2143ms` at 3.471 s. What the other
four partitions held is unknown, and the sequence above is why.

Expand Down
4 changes: 2 additions & 2 deletions issues/a-machine-without-an-iommu-refuses-every-claim.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ Exit, in both arms of `iommu_virtio_platform` (`tests/common/iommu.rs`):
`faults::refused_claim` reads, so a refusal for any other reason (no
MSI-X, no BAR window) fails the arm. Deleting the identity check so that
every claim goes through reds it. The non-image claimant is a
`tests/guest/rust-tests` binary, to be added, that init hands the NIC
function under `netcase`; netd is the in-image claimant.
`tests/guest/rust-tests` binary, to be added, that the supervisor hands the NIC
function under `netcase`; netstack is the in-image claimant.
- The kernel says "no isolation on this machine" at boot on
`Profile::HeadlessNoIommu` and never on `Profile::Headless`. Printing it on
every profile reds the `Headless` arm; printing it on none reds the other.
Expand Down
2 changes: 1 addition & 1 deletion issues/a-metal-session-runs-a-pre-flash-gate-first.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ that hold nothing against 7 ms finding everything that is there

The record this came from also warned about a NIC retry that looks like boot
cost; that warning is spent. `sdk/toyos/src/net.rs:272` says the hundred retries at
ten milliseconds are gone — a `netd` connector is live from a process's first
ten milliseconds are gone — a `netstack` connector is live from a process's first
instruction — so nothing after `Boot: complete` inflates a reading any more.

The panic path's own metal price belongs to the same session: the T14 measured
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ the binary in a ToyOS guest is this track's harness's job, not gbae's.
`pkg install <file>` stays for local and offline installs, checked against
the `SHA256SUMS` beside it.
- **Fetching is HTTPS.** GitHub serves releases only over TLS, so `pkg`
carries a TLS client; the network stack under it is netd's. A crate that
carries a TLS client; the network stack under it is netstack's. A crate that
does TLS is not our job to write and is widely used; it takes the fork
treatment every third-party crate takes. The redirect GitHub answers with
is followed once.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ userland spinlock around each stream's partial line, taken by every

- **A thread that runs at a real-time priority spins on a preempted holder.**
At smp 1 the holder cannot run until the spinner yields, and the spinner
never does. soundd's mix thread is clear of it (it writes a lane of its own,
never does. soundserver's mix thread is clear of it (it writes a lane of its own,
`claim_lane`); any other real-time thread that prints is not.
- **A panic raised inside `Held::with` re-enters it.** The panic's message is
a write to stderr, which takes the stream's `Held` again on the same thread
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ So the kernel ran the whole quiesce and made its last word durable; what
follows `Rebooting.` — `xhci::seal_shut`, the reset register — writes
nothing the stick keeps. Whether the reset was
written, whether firmware came back, and whether the host's Linux came up
without its network (the host is reached over the same I219 this boot's netd
without its network (the host is reached over the same I219 this boot's netstack
left faulted) are not separable from anything on the stick. A 454-line copy of
the same file that ends at 35.585 s was taken for the whole log; it is a
truncated read, not the file.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ No `PANIC:`, no `SEGFAULT`, no fault record, no `LOCK CONTENTION`, no
flushed`. Every one of those tripwires would have fired inside 40 s and the
machine was down for six minutes. The spawning CPU is `cpu6` in all three.

## The evidence is bounded by logd, and that is the whole difficulty
## The evidence is bounded by logkeeper, and that is the whole difficulty

**The stick's file is what `logd` flushed, not what the kernel emitted.** So a hang anywhere from that spawn through the reset register write
**The stick's file is what `logkeeper` flushed, not what the kernel emitted.** So a hang anywhere from that spawn through the reset register write
leaves exactly this tail, with everything after it still in the record ring and
never on the stick. The window is **[the reboot binary's spawn → the reset
register write]**, and the candidates in it are all consistent with what the
Expand All @@ -50,7 +50,7 @@ file shows:
program, through the same block path. A wedge in that path stays a candidate;
the 2 MiB read that matched the healthy 46–100 ms gap does not.
- `quiesce`'s `flush_disks` and `hand_back`, which landed at `88841f56` a few
minutes before the first hang; `hand_back` takes `XHCI` while `logd`'s own
minutes before the first hang; `hand_back` takes `XHCI` while `logkeeper`'s own
write to the same stick may still be in flight.

Nothing on the stick can separate them, and the black-box page was lost to the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Which wait left it set was never established.

## The sightings, all on paths since removed

- Once in logd's `io_uring_enter` on a KVM shard (nightly dispatch
- Once in logkeeper's `io_uring_enter` on a KVM shard (nightly dispatch
`31900050723`, job `95049280131`, 2026-08-15), 4.7 s after the boot stick
was pulled and the kernel's usb-storage gave the disk up. The same tree was
green on the same name minutes earlier; one guest per machine, so host
Expand Down
33 changes: 18 additions & 15 deletions issues/an-unlinked-open-file-answers-stale.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,28 @@ opened: 2026-10-22

# An unlinked open file answers Stale

A file the file server holds open answers `Stale` from the moment its name is
unlinked or renamed over (`system/fileserver/src/volume.rs`, the module
header), and a test asserts it. POSIX's file lives on until its last close or
unmap, and SQLite relies on that: it opens every temporary file and unlinks it
at once, and its last connection unlinks the `-shm` file while it is still
mapped. So any SQLite statement that spills to a temporary file fails today,
cargo's bundled one included.

**Ruled** (owner, 2026-10-21):
removing the last directory name prevents fresh opens through that name, and
existing handles and mappings keep the same file until their final release. A
new file made at the old path is a different file and never retargets them.
The file server owns the lifetime and the storage's reclaim; the kernel keeps
no filesystem identity or naming.

**Exit**: a file unlinked while open, and one renamed over, is read and written
through its open descriptor until the last goes, and its blocks are freed
then; a crash leaves the orphans for the next mount to reclaim; each asserted
by the file server's host volume tests, replacing today's `Stale` test
(`system/fileserver/src/data.rs`). A guest test reads a mapping of an
unlinked file once file mappings exist
(`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`).
Built for DATA and MEMORY: a file unlinked or renamed over while open is read
and written through its descriptor until the last close, and its blocks are
freed then (`unlink_and_replacement_keep_live_inode_identity` in
`system/fileserver/src/data.rs`, `unlink_and_replacement_keep_open_inode_identity`
in `memory.rs`, `persistent_unlink_preserves_dirty_bytes_until_the_last_holder`
in `pages/tests.rs`); a crash leaves the orphans for the next mount to release
(`namespace_operations_survive_a_crash_at_every_request` in
`lib/bcachefs/src/engine/tests/crash.rs`).

What is left:

- A FAT volume still answers `Stale`: its `unlink` marks the open file gone
(`system/fileserver/src/fat.rs`, `orphan`).
- No guest test reads a mapping of an unlinked file, which waits for file
mappings
(`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`).

**Exit**: the guest test of a mapping, and FAT keeping an unlinked open file.
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ opened: 2026-09-21

# Both boots of the manageability-read image found the T14 powered off

A read-only netd job — one read of the I219's `EXTCNF_CTRL` (§10.2.2.15,
A read-only netstack job — one read of the I219's `EXTCNF_CTRL` (§10.2.2.15,
`0x00F00`), no write, no PHY register touched, then `exit` with the reading
encoded in the code — was flashed to the T14 twice, as metal runs 58 and 62.
Both times the owner came back to a machine that was **off**: not frozen, not
wedged at a panic, powered down. Neither boot left a black box.

The persisted kernel log is the same on both runs: it ends at the first `logd`
The persisted kernel log is the same on both runs: it ends at the first `logkeeper`
batch, 192 lines, at around 0.6 s. So the stick carries nothing from the moment
the machine went away, and nothing recorded says what took the power.

Expand Down
4 changes: 2 additions & 2 deletions issues/client-ring-depth-is-the-devices-pipeline-depth.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ opened: 2026-08-01

# Every client's ring geometry is the device's pipeline depth

soundd sets `slot_count = num_buffers`, the kernel driver's `TX_INFLIGHT_MAX`.
soundserver sets `slot_count = num_buffers`, the kernel driver's `TX_INFLIGHT_MAX`.
So a device with a different pipeline depth silently changes the shm size, the
number of slots a client fills per wake, and the latency floor of *every*
client, through a constant no client sees and none can influence.
Expand All @@ -18,7 +18,7 @@ was assumed.

**What to build.** `slot_count` becomes a client property negotiated in
`MSG_STREAM_OPEN`/`MSG_STREAM_OPENED`, which already carries the format
negotiation, with soundd clamping to `[num_buffers.next_power_of_two(),
negotiation, with soundserver clamping to `[num_buffers.next_power_of_two(),
MAX_SLOTS]`. A client that asks for nothing gets today's value, so the default
is unchanged. `num_buffers` stays a device property and stops leaking.

Expand Down
10 changes: 5 additions & 5 deletions issues/deferred-release-outlives-its-syscall.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ So the sentence below is no longer the whole of it: this is a *semantic* event
riding a release the caller cannot wait for, which is what
`kernel/src/object/mod.rs`'s own header says must never happen — *"every
userland-visible lifecycle event rides `handle_count`"* — and the same shape
reaches soundd, whose cpal clients spend their lives parked in a signal-pipe
reaches soundserver, whose cpal clients spend their lives parked in a signal-pipe
read.

## Why it matters beyond a test
Expand Down Expand Up @@ -278,9 +278,9 @@ live process's `close` of its last handle, and `SYS_PROCESS_KILL` returning
before its victim's teardown (`kill_while_blocked`'s writes), both still
released on `ZERO_QUEUE`.

## A fourth witness: a device claim, and init waiting it out
## A fourth witness: a device claim, and the supervisor waiting it out

A service swap (`toyos-swap`, `/system/bin/init`) kills a service, `wait`s for
A service swap (`toyos-swap`, `/system/bin/supervisor`) kills a service, `wait`s for
it, and mints the service's device claims again for the binary that replaces
it. `Device` is a `deferred` row, so the `wait` returning is not the claim
being back. Measured on `wt/toyos-swap`, `tests/swapcase` (virtio-net, two
Expand All @@ -294,8 +294,8 @@ init: netd: pci:1af4:1041 is already claimed
```

The claim was asked for, and refused as held, before the release record — and
the release ran on the other CPU. The replacement netd started with no NIC and
exited, which init's probation caught and answered by restarting the old
the release ran on the other CPU. The replacement netstack started with no NIC and
exited, which the supervisor's probation caught and answered by restarting the old
binary five seconds later, when the claim was long back.

The supervisor's `CLAIM_RETURN`, `isa_row.rs`'s `claim_after` and
Expand Down
Loading