Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
a4b6a1e
osbench: persist.write_4k_fsync_beside_1m
Japabu Nov 1, 2026
31206cb
paged: a holder's read is atomic against its pager's write (XSH 2.9.7)
Japabu Nov 1, 2026
b3897d5
fileserver: an fsync commits its own file, and what the commit carries
Japabu Nov 1, 2026
6ae35a4
review r1 of #1141: a holder's WouldBlock is paged_read's, and the cl…
Japabu Nov 1, 2026
029ca22
fileserver: an fsync that leaves nothing owed answers the write-back
Japabu Nov 1, 2026
ea23ee4
paged: a holder's fstat is its object's, without a request (FS stage 3a)
Japabu Nov 1, 2026
ca6340e
issues: FS stage 3b waits for memory stage 7b's mappings
Japabu Nov 1, 2026
6469897
Merge existing paged-read and metadata work into storage consolidation
Japabu Nov 2, 2026
4f2efdb
WIP: move volatile files to anonymous userland objects
Japabu Nov 2, 2026
73e3d96
Merge published paged atomicity and stat work into storage consolidation
Japabu Nov 2, 2026
9ceeb34
Route filesystem paths through process capabilities
Japabu Nov 2, 2026
d5e69ee
Retain memory files in anonymous objects and remove kernel filesystems
Japabu Nov 2, 2026
1046a1f
Replace provisional DATA storage with bcachefs adapter
Japabu Nov 2, 2026
d137499
Integrate userland memory files and remove kernel filesystem paths
Japabu Nov 2, 2026
a4bcb2e
Integrate capability-only filesystem routing and userland cwd
Japabu Nov 2, 2026
add7a00
Add bcachefs copy-on-write file data and capacity reservations
Japabu Nov 2, 2026
0b81b56
Preserve live DATA objects and defer unwritten inode sizes
Japabu Nov 2, 2026
17934a8
Export shared filesystem descriptions to spawned children
Japabu Nov 2, 2026
265de0e
Finish partial truncation without allocating file data
Japabu Nov 2, 2026
70d982f
std: remove obsolete ToyOS filesystem routing export
Japabu Nov 2, 2026
4d73d0a
Simplify file-server allocation and keep lint checks clean
Japabu Nov 2, 2026
538da31
Guard persistent open-unlinked file lifetime through paged holders
Japabu Nov 2, 2026
f916f41
Move filesystem consumers to served descriptors
Japabu Nov 2, 2026
483562e
Remove unused paged filesystem metadata
Japabu Nov 2, 2026
09ede52
Keep directory descriptions correct across replacement
Japabu Nov 2, 2026
08ac5ed
Remove the copy command workaround for kernel file-cache pressure
Japabu Nov 2, 2026
24560b3
Remove tests of the retired private disk format
Japabu Nov 2, 2026
d679ad3
Coalesce underfull btree siblings and collapse redundant roots
Japabu Nov 2, 2026
e3169f8
Track live metadata records and geometry-bounded allocation indexes
Japabu Nov 2, 2026
c839976
Reserve metadata growth separately from shared rewrite capacity
Japabu Nov 2, 2026
3a47539
Align filesystem guest probes with served namespace contracts
Japabu Nov 2, 2026
c432f64
Keep buffered write reservations until their bytes are copied and wri…
Japabu Nov 2, 2026
9ff9d05
Exercise dirty supply prefix without a capacity limit masking the gap
Japabu Nov 2, 2026
336ad30
Start the userland temporary filesystem in console and test images
Japabu Nov 2, 2026
f9a9729
Keep merged btrees at their journaled depth and size a reclaim batch …
Japabu Nov 2, 2026
23097b9
Account retained root spines and exercise data capacity reclamation
Japabu Nov 2, 2026
8b73ea0
Check fixture validation at the writer admission boundary
Japabu Nov 2, 2026
8655a2f
Keep storage bounds and tests clean under workspace lint checks
Japabu Nov 2, 2026
a4e94cf
Exercise both bucket sizes with the reclaim journal included in ROOT …
Japabu Nov 2, 2026
7f52fb7
Fix exported file spawning and exclusive symlink creation
Japabu Nov 2, 2026
30be18a
Test cancelled writes without assuming metadata reclaim leaves space …
Japabu Nov 2, 2026
3b5b1e0
Evacuate sparse data buckets without changing logical extent ranges
Japabu Nov 2, 2026
40724bf
Refuse unreadable data checksum layouts before reserving a page
Japabu Nov 2, 2026
51b18b8
Schedule a sole-child parent when its underfull child cannot merge
Japabu Nov 2, 2026
63783da
Check native and retired filesystem signatures before designation
Japabu Nov 2, 2026
a2ce051
Keep evacuated small-file checksums within one filesystem block
Japabu Nov 2, 2026
f8cf985
Add explicit native DATA formatting and reboot persistence evidence
Japabu Nov 2, 2026
8bcffd2
Count all data blocks read after sparse-file evacuation
Japabu Nov 2, 2026
605c7c3
Own replay keys in fallibly prepared sorted runs
Japabu Nov 2, 2026
0507cbe
Keep post-admission spawn failure coverage after removing kernel paths
Japabu Nov 2, 2026
a879ad4
Record the engine's preexisting in-flight cancellation gap
Japabu Nov 2, 2026
11632e8
Stage runtime btree changes in journal-owned pending runs
Japabu Nov 2, 2026
4e829b2
Keep ordinary writers out of journal reclaim's reserve
Japabu Nov 2, 2026
25f19a0
Run the process-depth chain from an endowed immutable image
Japabu Nov 3, 2026
48316fc
Hold served stdio, holes and served /tmp in their tests
Japabu Nov 3, 2026
f32a988
Close the storage records the consolidation resolved
Japabu Nov 3, 2026
db6cab9
Merge main into the storage consolidation
Japabu Nov 3, 2026
3860848
Refuse a spawn whose working directory a child cannot read back
Japabu Nov 3, 2026
ebead35
Mount DATA only on the partition it was made for, and rewrite the pro…
Japabu Nov 3, 2026
7987307
Keep a swapped binary in the supervisor's memory, not on /tmp
Japabu Nov 3, 2026
e7f2a85
Merge main (#934) into the storage consolidation
Japabu Nov 3, 2026
8e117f1
bcachefs: a clean unmount ends in one self-describing publication
Japabu Nov 3, 2026
f14f79e
bcachefs: refuse NoSpace once what preceded the request is written
Japabu Nov 3, 2026
1cc4996
issues: the bcachefs track names the terminal publication and its unc…
Japabu Nov 3, 2026
2acadfb
issues: a-swapped-binary-lives-where-any-process-can-rewrite-it close…
Japabu Nov 4, 2026
248db3d
bcachefs: a publication member left nothing to append is not written
Japabu Nov 4, 2026
c8d5772
bcachefs: review r1 of #1152: comments, records and further tests
Japabu Nov 4, 2026
d5158bd
bcachefs: write nodes down to reclaim's split bound, 16 KiB
Japabu Nov 4, 2026
c12721a
bcachefs: a mount judges the superblock it runs on
Japabu Nov 4, 2026
938f1e7
issues: DATA mounted a volume another format had written over
Japabu Nov 4, 2026
719e0f0
bcachefs: a superblock place another writer took refuses the volume
Japabu Nov 4, 2026
adb3b56
bcachefs: refuse a superblock, node or extent that declares no checksum
Japabu Nov 4, 2026
39df624
bcachefs: mkfs leaves no other format's bytes where it reserves or pa…
Japabu Nov 4, 2026
164f391
fileserver: osbench's DATA sequence over the benches' DATA states, re…
Japabu Nov 4, 2026
94fdd7f
bcachefs: an ordinary transaction's keys go into the nodes it holds
Japabu Nov 4, 2026
f450182
Merge origin/main (K4, #1175) into storage consolidation
Japabu Nov 4, 2026
f1b2f3e
tests: the DATA images start the UART driver and logkeeper holds the …
Japabu Nov 4, 2026
f56cc63
storage: a write is one extent for each run of contiguous blocks
Japabu Nov 4, 2026
8f52abe
tests: nvme_disk_boots expects DATA's own line
Japabu Nov 4, 2026
45283fb
fileserver: a DATA path is resolved once a call, from the names it knows
Japabu Nov 4, 2026
c836688
bcachefs: a node takes a position's pending deltas in one pass, once …
Japabu Nov 4, 2026
af6ae82
bcachefs: a reserved write's extent points at the block holding its o…
Japabu Nov 4, 2026
dfa6e21
bcachefs: a whole superblock copy of another volume refuses the volume
Japabu Nov 4, 2026
537af08
issues: the two limits #1152 brings, and the replayed-leaf refusal it…
Japabu Nov 4, 2026
c74862d
fileserver: remembered DATA paths are bounded by bytes and forgotten …
Japabu Nov 4, 2026
a0098f4
bootlog: a boot whose job list ran past its bound is no pass
Japabu Nov 4, 2026
a043c5e
issues: a bcachefs replay holds its whole journal window in memory
Japabu Nov 4, 2026
aecd87e
Merge origin/main (1049225ef) into the storage consolidation
Japabu Nov 4, 2026
f433570
review r4 of #1152: the name cache counts each entry's cost, and the …
Japabu Nov 4, 2026
bdf062c
bcachefs: the journal's live entries are held to 32 MiB, and a scan k…
Japabu Nov 4, 2026
62e24c9
Merge origin/main (e740f01ac) into the storage consolidation
Japabu Nov 4, 2026
4410a56
fileserver: a block's place in the eviction order is the version it w…
Japabu Nov 4, 2026
6ff184a
bcachefs: mkfs's journal ring is the replay bound, and the writer's c…
Japabu Nov 4, 2026
cd639c3
review r6 of #1152: a journal larger than mkfs lays out is not written
Japabu Nov 4, 2026
49fe479
storage: a write that continues the last extends its reservation, and…
Japabu Nov 5, 2026
a1f0762
Merge origin/main (116dde5de) into the storage consolidation
Japabu Nov 5, 2026
92dcd86
issues: the two rows #1152 lands behind main
Japabu Nov 5, 2026
8cee28a
issues: the smaller DATA rows that trail main
Japabu Nov 5, 2026
79cd415
review r7 of #1152: the ring check after the shapes, extensions whole…
Japabu Nov 5, 2026
c4d1db6
Merge origin/main (eae2c7eb7) into the storage consolidation
Japabu Nov 5, 2026
9fe8f77
review r8 of #1152: a reservation's take is granted its whole credit
Japabu Nov 5, 2026
6fd7151
tests: two testcases members state what the served filesystem does
Japabu Nov 5, 2026
275a9e6
tests: readdir_bound rides the shared boot as a member again
Japabu Nov 5, 2026
57dedf1
review r10 of #1152: the refusal case takes the SDK's wake pair
Japabu Nov 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 28 additions & 3 deletions build/bootlog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ pub const REBOOTING: &str = "Rebooting.";
pub const STOPPING: &str = toyos_logstream::STOPPING;

/// What `tests/guest/test-runner` says when its job list runs past
/// `toyos_tco::JOB_BOUND_MS`, with the job it was inside as the next word.
/// **Console only**: a userland write reaches the serial backend and never a
/// log record, so no stick carries it.
/// `toyos_tco::JOB_BOUND_MS`, with the job it was inside as the next word,
/// before it kills that job and reboots: a boot that says it is no pass.
pub const JOB_DEADLINE_SAID: &str =
"test-runner: the job list ran past its bound, and the job it was inside is";

Expand Down Expand Up @@ -337,6 +336,9 @@ pub enum Unfit {
/// The loader pass after the foreign-identity arm's reset read no
/// [`FOREIGN_DONE`]: the stop it staged never finished.
NoForeignDone,
/// The test runner's word that its job list ran past its bound: the
/// line, naming the job it killed.
JobsPastBound(String),
}

impl fmt::Display for Unfit {
Expand All @@ -354,6 +356,10 @@ impl fmt::Display for Unfit {
"the loader's pass after the reset carries no {HANDED_BACK:?} with {REBOOTING:?} \
under {LOG_TAIL:?}: the stop this boot asked for never reached the reset"
),
Self::JobsPastBound(said) => write!(
f,
"the test runner killed a job and rebooted at its bound: {said:?}"
),
Self::NoForeignDone => write!(
f,
"the loader's pass after the reset carries no {FOREIGN_DONE:?}: the stop this \
Expand Down Expand Up @@ -541,6 +547,12 @@ pub fn boot_millis(log: &str) -> Option<u64> {
/// pass's ([`HANDED_BACK`], and [`REBOOTING`] under [`LOG_TAIL`]).
pub fn verdict(log: &str) -> Result<u64, Unfit> {
let boot_ms = boot_millis(log).ok_or(Unfit::NoBootRecord)?;
let past_bound = |line: &&str| {
toyos_logstream::program_line(line).is_some_and(|said| said.tag == "test-runner" && said.text.starts_with(JOB_DEADLINE_SAID))
};
if let Some(line) = log.lines().find(past_bound) {
return Err(Unfit::JobsPastBound(line.trim().to_string()));
}
if stopping_line(log).is_none() {
let last = log.lines().rev().find(|line| !line.trim().is_empty()).unwrap_or_default();
return Err(Unfit::Unfinished(last.trim().to_string()));
Expand Down Expand Up @@ -578,6 +590,19 @@ mod tests {
assert_eq!(verdict(""), Err(Unfit::NoBootRecord));
}

/// A boot the test runner ended at its job bound stopped as a passing
/// one does, and is none.
#[test]
fn a_boot_whose_job_list_ran_past_its_bound_is_not_a_pass() {
let past = format!("[2026-11-04 17:07:50 535.506 test-runner tid=1] {JOB_DEADLINE_SAID} osbench_persist (525000 ms)\n");
let log = format!(
"[ 1.151 cpu0 kernel] Boot: complete (1151ms)\n{past}[ 535.507 supervisor] {STOPPING} (Reboot)\n"
);
assert_eq!(verdict(&log), Err(Unfit::JobsPastBound(past.trim().to_string())));
let quoted = log.replace("test-runner tid=1] ", "osbench tid=4] said ");
assert_eq!(verdict(&quoted), Ok(1151));
}

/// Only the supervisor's own stop line, naming a shutdown, is a power-off.
#[test]
fn a_power_off_is_the_supervisors_stop_naming_a_shutdown() {
Expand Down
22 changes: 13 additions & 9 deletions build/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2091,7 +2091,7 @@ fn shipped_parts(root: &Path, boot: &Boot, plan: &Plan) -> (Vec<u8>, Vec<u8>, Us
/// a real one.
///
/// Designates the result, because every caller here is making a scratch disk
/// for a guest that expects a working `/apps` and `/home`, and the kernel will
/// for a guest that expects a working `/apps` and `/home`, and the file server will
/// not format an undesignated one. Leaving it to the call sites would mean two
/// places to forget; forgetting is not silent (the boot says so and both paths
/// are volatile) but it is not worth the chance.
Expand All @@ -2104,12 +2104,12 @@ pub fn create_sparse(path: &Path, len: u64) {
}

/// Partition this image as a DATA disk and stamp the designation on that
/// partition, so the kernel is allowed to format it.
/// partition, so the file server is allowed to format it.
///
/// The kernel never formats a volume that does not carry the stamp, which is
/// The file server never formats a volume that does not carry the stamp, which is
/// what stops it taking the disk of any machine it is booted on. So a throwaway
/// image has to say so, and this is the whole of the test harness's opt-in:
/// **data on a scratch file, not a build flag.** The kernel binary and the
/// **data on a scratch file, not a build flag.** The file-server binary and the
/// code path are identical either way — `probe` runs the same match
/// on metal as it does here — so the configuration under test is the
/// configuration that ships, which a `#[cfg]` could not have given us.
Expand Down Expand Up @@ -3087,12 +3087,14 @@ mod tests {
#[test]
fn a_disk_changing_authorization_goes_to_no_row_a_login_can_start() {
let gate = |text: &str| held_by_their_holders_alone(&toml::from_str(text).unwrap());
assert!(gate("[programs.runner]\nauthorize = [\"designate:data\"]\n").is_ok());
for word in ["designate:data", "format:data"] {
assert!(gate(&format!("[programs.runner]\nauthorize = [\"{word}\"]\n")).is_ok());
let launched = gate(&format!("[programs.disks]\nauthorize = [\"{word}\"]\n[programs.shell]\nstarts = [\"disks\"]\n"));
assert!(launched.unwrap_err().contains("may start it"));
let login = gate(&format!("[boot]\nstart = [\"sshserver\"]\n[programs.sshserver]\nlogin = true\nauthorize = [\"{word}\"]\n"));
assert!(login.unwrap_err().contains("is a login authority"));
}
assert!(gate("[programs.disks]\nauthorize = [\"tables\"]\n[programs.shell]\nstarts = [\"disks\"]\n").is_ok());
let launched = gate("[programs.disks]\nauthorize = [\"designate:data\"]\n[programs.shell]\nstarts = [\"disks\"]\n");
assert!(launched.unwrap_err().contains("may start it"));
let login = gate("[boot]\nstart = [\"sshserver\"]\n[programs.sshserver]\nlogin = true\nauthorize = [\"designate:data\"]\n");
assert!(login.unwrap_err().contains("is a login authority"));
assert!(gate("[programs.sshserver]\nlogin = true\nauthorize = [\"tables\"]\n").is_ok(), "reading the tables is no change");
assert!(gate("[programs.disks]\nauthorize = [\"format:everything\"]\n").unwrap_err().contains("no disk authorization"));
}
Expand Down Expand Up @@ -3403,6 +3405,8 @@ mod tests {
"tests/images/benchcase.toml",
"tests/images/clipboardcase.toml",
"tests/images/designatecase.toml",
"tests/images/formatcase.toml",
"tests/images/datacase.toml",
"tests/images/drivescase.toml",
"tests/images/hostedclangcase.toml",
"tests/images/hotplugcase.toml",
Expand Down
4 changes: 3 additions & 1 deletion build/image.rs
Original file line number Diff line number Diff line change
Expand Up @@ -999,7 +999,9 @@ mod tests {
#[test]
fn file_sets_across_a_bucket_size_edge_fit_and_read_back() {
let mut sides = std::collections::BTreeSet::new();
for kib in (58880..59904).step_by(256) {
// Leave room for format bookkeeping on either side; the journal's
// reclaim reserve is part of the image, not of these file bytes.
for kib in [48 << 10, 56 << 10, 60 << 10, 64 << 10] {
let big: Vec<u8> = (0..kib << 10).map(|i| (i / 4096 + kib) as u8).collect();
let files: Vec<(String, Vec<u8>)> = std::iter::once(("bin/big".to_string(), big))
.chain((0..200u32).map(|i| (format!("share/s{i:03}"), vec![i as u8; 3000 + i as usize])))
Expand Down
2 changes: 1 addition & 1 deletion images/console.toml
Original file line number Diff line number Diff line change
Expand Up @@ -132,5 +132,5 @@ receives = ["disk", "input"]
[programs.fileserver]
restart = true
pager = true
roles = ["data", "log", "boot"]
roles = ["data", "log", "boot", "memory"]
receives = ["block"]
4 changes: 2 additions & 2 deletions images/system.toml
Original file line number Diff line number Diff line change
Expand Up @@ -386,12 +386,12 @@ receives = ["disk"]
drives = ["pci:0c:03:30"]
receives = ["disk", "input"]

# The file servers: DATA, the log and the boot volume, one process
# The file servers: DATA, temporary files, the log and boot volumes, one process
# each, serving every program the directories of its role. Started again when
# one ends, on the same ports. A pager: its readers wait on the pages it
# supplies, which only a row of this file may make a program do.
[programs.fileserver]
restart = true
pager = true
roles = ["data", "log", "boot"]
roles = ["data", "log", "boot", "memory"]
receives = ["block"]
30 changes: 30 additions & 0 deletions issues/a-4-kib-overwrite-on-data-costs-about-2-4-times-mains.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
status: open
kind: defect
opened: 2026-11-05
---

# A 4 KiB overwrite on DATA costs about 2.4 times main's

**Owner.** The storage track, which owns `lib/bcachefs` and the file
server's DATA volume.

osbench's `fs.disk.write_4k_at`, 4 KiB writes at scattered pages of a
16 MiB file, at #1152's a1f07627e: 23.9 us on the T14 and 30.8 us on the
AMD, p50, one boot each. Main dc1a8642e, on its private format, took
10.2 and 12.2 us (nightly 2026-10-31-2316); Linux ext4 0.7 and 2.3 us.

Each overwrite is a copy-on-write replace: the write lands in new sectors
and one transaction removes or trims the old extent, inserts the new one,
and updates its backpointer, bucket accounting and the inode. On the host
model of the T14's DATA (CPU only, no device latency), taking the sectors
costs 0.8 us and that transaction 30.6 us an overwrite. A macOS `sample` of
it puts about 11% of the busy samples in `Cached::put`'s moves of a node's
keys (`memmove` under `mount::put` in `apply`); the rest is spread over the
pending overlay, allocation and the block writes, with no single cost
(#1152's evidence, `storage-integration/r9-small-writes`, `perf.rs`'s
`scattered_overwrites` and `scatter-sample.txt`).

**Exit.** `write_4k_at` within 1.5 times main's on both benches, measured
by osbench against the same nightly's main, with the host model's
transaction cost and a sample of it published.
32 changes: 0 additions & 32 deletions issues/a-bcachefs-hole-costs-a-block-write-per-page.md

This file was deleted.

29 changes: 29 additions & 0 deletions issues/a-data-directory-listing-reads-every-entrys-inode-alone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
status: open
kind: defect
opened: 2026-11-05
---

# A DATA directory listing reads every entry's inode alone

**Owner.** The storage track, which owns the file server's DATA volume and
`lib/bcachefs`.

osbench's `fs.disk.readdir.1000`, listing a directory of 1 000 files, at
#1152's a1f07627e: 2.86 ms on the T14 and 1.99 ms on the AMD, p50, one boot
each; main dc1a8642e 0.91 and 0.69 ms (nightly 2026-10-31-2316), about 3x.

**Cause.** A listing entry carries its file's size, so `DataVolume::list`
(`system/fileserver/src/data.rs`) reads each entry's inode with a lookup of
its own, from the root of the inodes btree, after one readdir of the
dirents; and every READDIR request lists the whole directory again before
taking its batch from the cursor. On the host model of the T14's DATA (CPU
only), one whole listing of 1 000 entries costs 534 us: the dirents 39 us,
the 1 000 inode lookups 416 us, sorting and encoding the rest (#1152's
evidence, `storage-integration/r10-known-gaps`, `perf.rs`'s
`listing_a_thousand`). A client asks in 64 KiB batches, so 1 000 short
names are one or two requests.

**Exit.** The inodes of a listing read in one ordered pass over the inodes
btree, or the listing kept across a directory's READDIR batches, with
`readdir.1000` within 1.5 times main's on both benches.
20 changes: 0 additions & 20 deletions issues/a-fat-files-mtime-through-its-handle-is-its-last-level.md

This file was deleted.

This file was deleted.

20 changes: 0 additions & 20 deletions issues/a-kernel-files-fstat-answers-its-handles-mtime.md

This file was deleted.

21 changes: 0 additions & 21 deletions issues/a-link-on-a-kernel-mount-to-a-served-path-leads-nowhere.md

This file was deleted.

Loading