Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/src/architecture/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ All four layers run in userland.
block service.
- A partition has one holder at a time, every request is bounded to it before
the device sees it, and a flush answers for its own writer's writes.
- A driver answers every request a session publishes, or ends the session,
within one bound the block protocol states, counted from the request's
publication: the wait to be taken from the session is inside it, as are a
device's silence before the driver takes a command back and taking it back.
A driver shows that its worst case fits under the bound. A client that has
waited longer ends the session; it never waits on a driver's word alone.
- Caching belongs to file servers, never to the block layer.
- Partition editing and storage initialisation are userland operations shared
by disk tools and the installer. Editing the table, reading payloads and
Expand Down

This file was deleted.

25 changes: 0 additions & 25 deletions issues/a-read-or-write-answered-lost-is-never-answered.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
status: open
kind: defect
opened: 2026-11-11
severity: [data-loss]
---

# A USB disk's session can wait past the block silence bound behind other sessions

**Owner.** The storage track: usbd's mass-storage driver and the block
protocol.

The block protocol bounds a server's silence: while it owes a session an
answer it posts a completion, or ends the session, within
`toyos_blockring::SILENCE_BOUND_NS`, and a client ends a session silent
longer. The NVMe driver shows that its figure from a command's submission
fits (`system/nvme/src/nvme.rs`, `ANSWERED_WITHIN`), which is not yet the
whole path (below). usbd cannot as it is:

- One command at a time per device (Bulk-Only Transport 1.0 §5.1), from a
queue of up to `QUEUE` (32) commands shared by every session on the device
(`system/usbd/xhci/src/msc.rs`). Each of a command's transfers may take
`COMMAND_NS` (10 s) and still succeed, so one session's request can wait
behind other sessions' commands for about 32 × 30 s, 16 minutes, with no
failure and no completion of its own.
- A break climbs the recovery ladder (`system/usbd/xhci/src/ladder.rs`): class
reset, port reset and enumeration again, the last reset. Its steps add their
own bounds, whose sum is not derived.

A client that ends such a session reconnects to the same running driver,
which refuses the open `Held` while it still holds the request, and the
client's disk is gone for its life (`system/diskserver/src/disk.rs`). The
acknowledged writes the client keeps that no flush has covered are then
reported not durable by its next flush: accepted data later reported lost.

The NVMe driver's figure, `ANSWERED_WITHIN`, equals the bound with no slack and
counts from a command's submission to the device. It leaves out:

- the wait to be pulled: a session's request waits behind other sessions'
requests whenever the device is full (`toyos_blockserve::Sessions::pull`
takes the sessions in order), and behind a reset, during which nothing is
pulled;
- the serve loop's lag between a command reaching `COMMAND_SILENCE` and the
reset;
- the reset's own register and page work;
- posting the reset's answers and ringing the sessions after it;
- the client's clock, which starts just before it publishes.

**Exit.** Each block driver's worst case from a request's publication to its
completion, the wait behind other sessions included, is shown to fit under
`SILENCE_BOUND_NS`, with a host test of usbd's; or the protocol's bound is
redefined so a session's silence does not count other sessions' work, and
both drivers show they keep it.
31 changes: 0 additions & 31 deletions issues/nothing-reopens-a-diskserver-session-over-a-used-page.md

This file was deleted.

9 changes: 9 additions & 0 deletions issues/smaller-data-rows-trail-main-by-up-to-half.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,14 @@ The larger gaps have records of their own
Evidence: #1152's evidence, `storage-integration/r10-known-gaps`
(`osbench-a1f07627e/` and `osbench-table-a1f07627e.md`).

- **Owed on metal:** both benches, `osbench_disk`'s `fs.disk.*` rows in one
A/B of main before and after #1236, which reads a DATA listing's inodes in
one pass (`issues/a-data-directory-listing-is-read-whole-for-every-readdir-request.md`).
On the AMD bench, unmeasured, main after must show `readdir.1000` below main
before on every boot. On both, no other row may resolve against main after:
in #1236's T14 A/B `open_close` and `stat` did (1.8 and 8 %), though the
code on their path is the same but for its addresses, so a second A/B is
where they recur or not.

**Exit.** Each row within 1.1 times main's on both benches, or its cause
shown to be inherent.
13 changes: 4 additions & 9 deletions issues/the-kernel-is-small-interrupts-post-and-threads-wait.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,17 +72,14 @@ Beside that order, each waiting on something outside it:
`what_a_flush_calls_durable_is_on_the_medium` and
`what_was_on_the_wire_at_the_end_is_refused_and_what_was_not_waits` for
the transport, for diskserver's part by `wire::Opened::over`, whose type
makes a server's ends before its answer, and by the client test
`issues/nothing-reopens-a-diskserver-session-over-a-used-page.md` owes, a
session reopened over the page a killed server left, by fileserver's
makes a server's ends before its answer, and by a client that opens each
session over a region no server has held, by fileserver's
`a_directory_rename_is_whole_under_one_name_wherever_the_server_is_killed`
for DATA, and for LOG by the FAT test step 10 owes. **Partly done**
(#525, #536, #971): NVMe and the stick are driven from userland, and the
kernel's NVMe driver, USB storage, block layer and refusal chain are gone;
the `nvme` driver and usbd offer their disks to diskserver, the block
service. **Left**: the page cache, with step 9's file half, and the client
test `issues/nothing-reopens-a-diskserver-session-over-a-used-page.md`
owes.
service. **Left**: the page cache, with step 9's file half.
4. **Where the data filesystem lives** — owner ruling, 2026-09-26, from the
roasted proposal: userland file servers, the VFS a client library, and a
program's view a set of directory capabilities the supervisor hands it,
Expand Down Expand Up @@ -257,9 +254,7 @@ Beside that order, each waiting on something outside it:
same under a write to the NVMe. #971's first metal boots
came back with no `logkeeper` file on the stick, which every metal
row reads; QEMU's stick boot (`the_log_lands_on_the_boot_stick`) has
it. And the client test
`issues/nothing-reopens-a-diskserver-session-over-a-used-page.md`
owes, which `/log`'s evidence names.
it.

5. **USB by userland**, with discovery and recovery written once as
straight-line code: step 10, and this exit. **Exit**: step 10's, and no
Expand Down
21 changes: 18 additions & 3 deletions issues/the-nvme-driver-resets-a-drive-that-is-slow-and-healthy.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ opened: 2026-09-26
# The NVMe driver resets a drive that is slow and healthy

The NVMe driver reclaims a command the controller has not answered in
`COMMAND_SILENCE`, ten seconds (`system/nvme/src/main.rs`), by resetting the
controller. NVMe bounds no command's time, and a Flush of a large volatile
cache is the slowest thing a healthy drive does: a drive still flushing at ten
`COMMAND_SILENCE`, ten seconds (`system/nvme/src/nvme.rs`), by resetting the
controller. The same ten seconds bound each admin command of the reset, and
with the reset they set the block protocol's silence bound
(`toyos_blockring::SILENCE_BOUND_NS`), which a longer one must raise. NVMe
bounds no command's time, and a Flush of a large volatile cache is the
slowest thing a healthy drive does: a drive still flushing at ten
seconds is reset, every command it held is answered `Device`, and a flush
retried past `toyos_blockring::client::MAX_ATTEMPTS` reaches its caller as
`Device` — a loss reported for a drive that lost nothing.
Expand All @@ -22,3 +25,15 @@ size and write rate, or a measured worst Flush on the T14's NVMe and on QEMU),
or a silent command is first aborted (NVMe Abort, §5.1) before the controller
is reset; and a guest test holds a Flush past the old bound without the
session hearing `Device`.

Every admin command, at bring-up and in a reset, is held to the same ten
seconds since #1235, where it was `CAP.TO`. The T14 brought its controller up
under it at #1235's head (a `metalcase` boot: up in 2 ms, APST on, DATA served
off the NVMe).

- **Owed on metal:** the AMD bench, the `microcode` and `machine_reboot` rows
(the `metalcase` image) in one boot of main after #1235. Main must pass both
rows, and its log must show the nvme driver's `up:` line with no `NOT
SERVING` and no `was not answered in`, `fileserver: Data serving` off the
NVMe, and no session's server that broke the protocol, answered nothing in
the bound, or would not take the session back.
Loading