From fd5d2628fa7159030aa4a372e85e61abd9b4552d Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 17:45:24 +0200 Subject: [PATCH 1/7] spawn_cost: what a spawn costs the machine, beside how long it takes A spawn's log lines cost the `/log` file server, its driver and logkeeper CPU on other CPUs, off the spawn's own path, so its latency cannot show them. The row now prints the machine's busy CPU over its repetitions, a spawn's share, from the sysinfo header every process may read: the figure an A/B of the log's write path, or of anything else a spawn reaches, is read from. Co-Authored-By: Claude Opus 5.5 --- tests/guest/rust-tests/src/bin/spawn_cost.rs | 23 +++++++++++++++++++- tests/toyos.rs | 6 ++--- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/tests/guest/rust-tests/src/bin/spawn_cost.rs b/tests/guest/rust-tests/src/bin/spawn_cost.rs index 7a1a81789e6..f733a0661ca 100644 --- a/tests/guest/rust-tests/src/bin/spawn_cost.rs +++ b/tests/guest/rust-tests/src/bin/spawn_cost.rs @@ -5,6 +5,11 @@ //! runs on, `tests/images/testcases.toml`, and carries the most `RELATIVE` entries of what //! that ROOT does. //! +//! **And what it costs the machine**: the CPU every CPU was busy for over the +//! repetitions, a spawn's share, which counts what the spawn costs the +//! processes it reaches off its own path, the log among them. Work a spawn +//! leaves for later than the last repetition is not in it. +//! //! **No threshold, and the host reads the numbers instead**, as //! `syscall_cost`'s: a duration measured under TCG is no duration, and one //! measured on metal is for a same-session A/B against another build of this @@ -14,6 +19,7 @@ use std::process::{Command, Stdio}; use toyos_abi::clock::nanos_since_boot as clock_nanos; +use toyos_abi::syscall::{SysinfoHeader, SYSINFO_HEADER_SIZE}; const PROGRAM: &str = "/system/bin/toybox"; const REPS: usize = 50; @@ -32,9 +38,24 @@ fn spawn_ns() -> u64 { end - start } +/// The CPU time every CPU has been busy for since the boot. +fn busy_ns() -> u64 { + let mut header = [0u8; SYSINFO_HEADER_SIZE]; + let n = toyos::system::sysinfo(&mut header); + assert!(n >= SYSINFO_HEADER_SIZE, "sysinfo returned {n} bytes"); + SysinfoHeader::decode(&header).total_cpu_ns +} + fn main() { spawn_ns(); + let busy = busy_ns(); let mut ns: Vec = (0..REPS).map(|_| spawn_ns()).collect(); + let machine = (busy_ns() - busy) / REPS as u64; ns.sort_unstable(); - println!("spawn_cost: {PROGRAM} min {} us median {} us over {REPS}", ns[0] / 1000, ns[REPS / 2] / 1000); + println!( + "spawn_cost: {PROGRAM} min {} us median {} us over {REPS}, the machine busy {} us a spawn", + ns[0] / 1000, + ns[REPS / 2] / 1000, + machine / 1000 + ); } diff --git a/tests/toyos.rs b/tests/toyos.rs index f930b853955..2ef2ad96309 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -1072,9 +1072,9 @@ const METAL: &[(&str, metal::Metal)] = &[ }, ), ( - // How long a spawn of `toybox` takes to its exit, printed by the job and - // read off the log for an A/B against another build: that it ran, and - // what it said, are the verdict. + // How long a spawn of `toybox` takes to its exit, and the machine's + // busy CPU a spawn, printed by the job and read off the log for an A/B + // against another build: that it ran, and what it said, are the verdict. "spawn_cost", metal::Metal { arms: TESTCASES, From 9f9e7e7c15673a0309c5877150dfa742bd252d4c Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 16:42:31 +0200 Subject: [PATCH 2/7] logkeeper: the file is a consumer of the replay, written in batches on its own thread Measured on both benches (agents/spawnperf), each spawn's two log lines cost 370-475 us of CPU in /log's file server, nvme and logkeeper together on the T14 and 1.14-1.23 ms on the AMD bench, one spawn at a time: a fixed cost per write, because every round was written and synced. And the loop made every /log call itself, so a /log server that took a call and never answered left every console channel and reader silent. - The file is a thread with an offset into the replay, as the readers and carriers are (`file.rs`). It opens the volume (the opener thread goes), writes, syncs, rotates and asks a refusing volume again. The loop appends a round to the replay and says so; it makes no call on /log. - A batch is written and synced once 64 KiB are owed, once its oldest line has waited 1 s, at once for a line at Error or above (a program's panic report is its stderr, a kernel fault an Alert), and at the stop's flush. Both bounds are justified at their site. A machine that dies without a stop loses at most that second of lines below Error from the file; the console carried them. - The stop's flush and mark are asks the loop waits on within STOP_FILE, three quarters of the supervisor's bound, and gives up on with a line. From the flush until a refused stop the file thread makes no call but the mark, which it writes only over a file the flush reached and only before its deadline, so the mark stays the last write before the drivers stop. - A caught-up reader waits on a bell the replay's growth rings beside its pipe's other end leaving, so a reader that leaves while the log is quiet gives back its thread and slot at once. - `policy::Step` goes with the per-round write it named. - The manual's diagnostics page says what the file holds and what a crash costs it; the ABI's severity note says Error rather than Alert. Closes issues/the-console-waits-on-logkeepers-file-writes.md and issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md. Co-Authored-By: Claude Opus 5.5 --- docs/src/architecture/diagnostics.md | 10 +- ...s-a-reader-leave-only-at-its-next-write.md | 19 - ...console-waits-on-logkeepers-file-writes.md | 42 -- sdk/abi/src/log.rs | 4 +- system/logkeeper/src/file.rs | 441 ++++++++++++++++++ system/logkeeper/src/inspect.rs | 31 +- system/logkeeper/src/main.rs | 368 +++------------ system/logkeeper/src/policy.rs | 37 +- system/logkeeper/src/serve.rs | 144 ++++-- system/logkeeper/src/store.rs | 5 +- 10 files changed, 645 insertions(+), 456 deletions(-) delete mode 100644 issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md delete mode 100644 issues/the-console-waits-on-logkeepers-file-writes.md create mode 100644 system/logkeeper/src/file.rs diff --git a/docs/src/architecture/diagnostics.md b/docs/src/architecture/diagnostics.md index 2d6351a8ac5..f24903c2f22 100644 --- a/docs/src/architecture/diagnostics.md +++ b/docs/src/architecture/diagnostics.md @@ -20,9 +20,13 @@ exports them. - The kernel's logging responsibility is its own record ring and its console. Userland output never enters the kernel ring. Reading the machine's log needs a right on the system capability. -- The log keeper owns the log files. It holds each program to an allowance, - keeps every boot's beginning, and syncs on an alert or an interval. Severity - is an ordered ladder. +- The log keeper owns the log files. It holds each program to an allowance + and keeps every boot's beginning. Severity is an ordered ladder. +- The file is written in batches: at once for an error, an alert or the + machine's stop, and otherwise within a second. A machine that dies without + stopping loses at most that second from its file, never from its console. + No file call holds up the console or a reader, and a stop waits on the file + within a bound. - A slow sink drops and counts; it never blocks a producer and fails alone. - The kernel waits on the log keeper neither at a panic nor at a stop. diff --git a/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md b/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md deleted file mode 100644 index 1fd3804b6bf..00000000000 --- a/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-10-08 ---- - -# logkeeper hears a reader leave only at its next write - -A log reader's thread (`system/logkeeper/src/serve.rs`, `feed`) waits on -`shared.grew` once it has caught up, and learns its reader is gone only when a -write to the sink fails. A reader that leaves while the log is quiet keeps its -thread and its place in `MAX_READERS` until the next record lands. - -Read from the code, not measured. - -**Exit condition**: a reader's end is what wakes its thread, or the header of -`serve.rs` says why the next record is soon enough. - -**Owner**: whoever holds `issues/redesign-the-log-subsystem.md`. diff --git a/issues/the-console-waits-on-logkeepers-file-writes.md b/issues/the-console-waits-on-logkeepers-file-writes.md deleted file mode 100644 index 71f20a5c4e5..00000000000 --- a/issues/the-console-waits-on-logkeepers-file-writes.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-11-05 -severity: [hang] ---- - -# The console waits on logkeeper's file writes - -`logkeeper`'s loop is the only thing that reads the rings and grows the -replay the console's carriers and readers are fed from, and it makes every -call on the LOG file server itself, each one blocking (`system/logkeeper/src`): - -- a round's write and sync (`Log::to_volume`: `Volume::write`, `Volume::sync`); -- the catch-up of a file behind the replay (`Log::caught_up`, from a round and - from `Log::adopt`: `Volume::catch_up`'s reopen, writes, sync and rotation); -- a rotation's sync, retention sweep and create (`Log::rotate_if_full`); -- the stop's catch-up and sync (`Volume::finish`), whose time bounds how many - calls it starts and not how long one takes; -- the stop mark and its removal (`store::mark_stopped`, `unmark_stopped`). - -`LOG_WRITE_BUDGET` judges a call only once it returns. So a LOG server that -takes a call and never answers, alive but wedged, leaves every console channel -silent for as long as it does, with no failure bound, and the supervisor -restarts only a server that ended. The stop's wait on the volume's opener is -the one wait here with a bound (`STOP_CATCH_UP`). - -Read from the code, not reproduced. The volume's open waited the same way -before it moved to a thread of its own, which is how the console stood silent -for the supervisor's 30 s declaration bound on `acpi_press_is_one_sci`'s boot. - -**Exit**: no console channel waits on a `/log` call; a guest boot whose LOG -server stops answering mid-boot still carries its later lines. The shape: the -file is a consumer of the replay with its own offset, written on its own -thread as the readers and carriers are, and the stop asks it to reach the -replay's end and sync within a bound. The per-file offset and the stop's -bounded catch-up are in the tree (`Volume::through`, `catch_up`, `finish`). -What is left is the move: the loop still owns the `Volume` and makes the calls -above, so the volume goes to a thread of its own that reads the replay, and -the stop's bound becomes a wait on that thread, which the loop can give up on. - -Owner: `system/logkeeper`; unheld. diff --git a/sdk/abi/src/log.rs b/sdk/abi/src/log.rs index dc2075b8cb0..83dd7073a29 100644 --- a/sdk/abi/src/log.rs +++ b/sdk/abi/src/log.rs @@ -64,8 +64,8 @@ pub const RECORD_BYTES: usize = 1024; /// (`log!`) and `Alert` (`alert!`); a program's stdout is `Info` and its stderr /// `Error`, and its own lines choose. Every rendered line names one above /// `Info` by its [`word`](Self::word), a screen colours the line by it, and -/// `/system/bin/logkeeper` makes the volume durable at `Alert` rather than on -/// its interval. +/// `/system/bin/logkeeper` makes the volume durable at `Error` and above rather +/// than with its batch. #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] #[repr(u8)] pub enum Severity { diff --git a/system/logkeeper/src/file.rs b/system/logkeeper/src/file.rs new file mode 100644 index 00000000000..38808292fcc --- /dev/null +++ b/system/logkeeper/src/file.rs @@ -0,0 +1,441 @@ +//! This boot's file on `/log`: a consumer of the replay with an offset of its +//! own, on a thread of its own, as the readers and the console's carriers are. +//! +//! **The loop makes no call on `/log`.** It appends a round to the replay and +//! says so ([`File::grew`]). This thread opens the volume, writes, syncs, +//! rotates and asks a refusing volume again, so a `/log` server that takes a +//! call and never answers holds this thread and nothing else. The stop's +//! flush and mark are asks the loop waits on within a bound and can give up on. +//! +//! **The file is written in batches**: once [`BATCH_BYTES`] are owed, once +//! the oldest owed line has waited [`BATCH_AGE`], and at once for a round +//! holding a line at `Error` or above (a program's panic report is its +//! stderr, a fault the kernel reports an `Alert`) and at the stop's flush. +//! Each batch is written and synced whole. +//! +//! **What a crash costs the file** is the batch not yet written: at most +//! [`BATCH_AGE`] of lines below `Error`, which the console carried as they +//! were read. A stop writes the batch before the stop mark, and after the +//! mark this thread makes no call until the stop is refused. + +use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender, SyncSender}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use toyos::say; +use toyos_logstream::{Severity, LOG_OPENED}; + +use crate::inspect::{Published, State}; +use crate::policy::{Retry, LOG_WRITE_BUDGET}; +use crate::serve::Hub; +use crate::store::{self, Volume, DIR, MAX_LOG_BYTES, MAX_LOG_PARTS}; + +/// How long a line below `Error` may wait for the file. Policy, against two +/// costs: each write and sync costs the volume's server, its driver and this +/// program a fixed amount whatever its size, and a spawn says two lines, so a +/// sync per round is that cost per spawn; a machine that dies without +/// stopping loses this much of its file. A second keeps the file a second +/// behind the console at worst and makes one write of everything a busy +/// second says. +pub const BATCH_AGE: Duration = Duration::from_secs(1); + +/// How much owed text is written at once, whatever its age: a flood is +/// written in pieces this size rather than held for [`BATCH_AGE`], so no one +/// write is a burst the stick's other users wait behind. +pub const BATCH_BYTES: u64 = 64 * 1024; + +/// Whether a batch is written at `now`: `owed` bytes, the oldest owed since +/// `since`, of which a line at `Error` or above makes the batch `urgent`. +fn batch_due(owed: u64, since: Option, urgent: bool, now: Instant) -> bool { + owed > 0 && (urgent || owed >= BATCH_BYTES || since.is_some_and(|since| now >= since + BATCH_AGE)) +} + +/// Whether a round's line at `severity` makes its batch written at once. +pub fn urgent(severity: Severity) -> bool { + severity >= Severity::Error +} + +enum Ask { + /// The replay grew; `urgent` where by a line written at once. + Grew { urgent: bool }, + /// The stop: reach the replay's end and sync, stopping the catch-up at + /// `by`, and answer. + Finish { by: Instant, answer: SyncSender<()> }, + /// Write the stop mark, unless `by` has passed, and answer whether it is. + Mark { by: Instant, answer: SyncSender }, + /// The stop was refused: take back the mark where it was written, and + /// write what the replay was given since at once. + Resume { marked: bool }, +} + +/// The loop's handle on the file's thread. +pub struct File { + asks: Sender, +} + +impl File { + /// Start the file's thread, which opens this boot's file under `stem` + /// (an undated boot's where `None`) and says so with `dated`. + pub fn start(stem: Option, dated: String, hub: Arc, published: Arc) -> Self { + let (asks, asked) = mpsc::channel(); + std::thread::Builder::new() + .name("log-file".into()) + .spawn(move || { + published.publish(None, State::Opening); + let volume = open(stem, &dated); + let retry = volume.as_ref().is_some_and(|v| !v.is_open()).then(|| Retry::after(Instant::now())); + let writer = Writer { + volume, + hub, + published, + retry, + degraded: false, + owed_since: None, + urgent: false, + finished: false, + stopped: false, + }; + writer.run(asked); + }) + .expect("logkeeper: the file's thread could not be started"); + Self { asks } + } + + /// The replay grew by a round. + pub fn grew(&self, urgent: bool) { + self.ask(Ask::Grew { urgent }); + } + + /// The stop's flush, its catch-up ended at `by`: whether the file + /// answered by `until`. + pub fn finish(&self, by: Instant, until: Instant) -> bool { + let (answer, answered) = mpsc::sync_channel(1); + self.ask(Ask::Finish { by, answer }); + answered.recv_timeout(until.saturating_duration_since(Instant::now())).is_ok() + } + + /// The stop mark, written by `by`: whether it is, or `None` where the + /// file did not answer by then. + pub fn mark(&self, by: Instant) -> Option { + let (answer, answered) = mpsc::sync_channel(1); + self.ask(Ask::Mark { by, answer }); + answered.recv_timeout(by.saturating_duration_since(Instant::now())).ok() + } + + /// The stop was refused, after the mark was written where `marked`. + pub fn resume(&self, marked: bool) { + self.ask(Ask::Resume { marked }); + } + + fn ask(&self, ask: Ask) { + self.asks.send(ask).expect("logkeeper: the file's thread ended"); + } +} + +/// This boot's file, opened, and this program's half of the startup report: +/// the kernel says whether it has a console, this whether it has a volume +/// and what its name was decided by. +fn open(stem: Option, dated: &str) -> Option { + let volume = Volume::open(DIR, stem, MAX_LOG_BYTES, |line| say!("{line}")); + match &volume { + Some(v) if v.is_open() => say!("{LOG_OPENED}{} ({dated})", v.path()), + Some(v) => say!( + "logkeeper: {} is asked again - this boot's lines are kept until {DIR} takes them ({dated})", + v.path() + ), + // `Volume::open` has said why. + None => say!("logkeeper: this boot's kernel log is on the console only ({dated})"), + } + volume +} + +struct Writer { + volume: Option, + hub: Arc, + published: Arc, + /// When a part the volume refused is next opened again. + retry: Option, + /// Whether the volume answers, slower than [`LOG_WRITE_BUDGET`] a batch. + degraded: bool, + /// When the replay first grew past the file's last batch. + owed_since: Option, + urgent: bool, + /// The stop's flush reached the file, which the mark is written over. + finished: bool, + /// From the stop's flush until it is refused: no call on `/log` but the mark. + stopped: bool, +} + +impl Writer { + fn run(mut self, asked: Receiver) { + loop { + self.publish(); + let ask = match self.next_due() { + Some(at) => asked.recv_timeout(at.saturating_duration_since(Instant::now())), + None => asked.recv().map_err(|_| RecvTimeoutError::Disconnected), + }; + match ask { + Ok(Ask::Grew { urgent }) => { + self.owed_since.get_or_insert_with(Instant::now); + self.urgent |= urgent; + } + Ok(Ask::Finish { by, answer }) => { + self.finish(by); + // A loop that gave up on the answer has said so. + let _ = answer.send(()); + } + Ok(Ask::Mark { by, answer }) => { + let _ = answer.send(self.mark(by)); + } + Ok(Ask::Resume { marked }) => self.resume(marked), + Err(RecvTimeoutError::Timeout) => {} + // The loop never lets go of its handle while the program runs. + Err(RecvTimeoutError::Disconnected) => return, + } + if self.due(Instant::now()) { + self.write(); + } + } + } + + fn publish(&self) { + let state = match &self.volume { + None => State::ConsoleOnly, + Some(v) if !v.is_open() || self.degraded => State::Degraded, + Some(_) => State::Writing, + }; + self.published.publish(self.volume.as_ref(), state); + } + + /// When a batch next falls due with no ask: the oldest owed line's age, + /// or a refused part's retry. `None` while nothing can be written. + fn next_due(&self) -> Option { + let v = self.volume.as_ref().filter(|_| !self.stopped)?; + if !v.is_open() { + return self.retry.map(|retry| retry.at()); + } + self.owed_since.map(|since| since + BATCH_AGE) + } + + fn due(&self, now: Instant) -> bool { + let Some(v) = self.volume.as_ref().filter(|_| !self.stopped) else { return false }; + if !v.is_open() { + return self.retry.is_some_and(|retry| retry.due(now)); + } + batch_due(self.hub.end() - v.through(), self.owed_since, self.urgent, now) + } + + /// Give the file what the replay owes it, a part's worth, and make it + /// durable once it has all of it; a part refused is opened again here. + fn write(&mut self) { + let began = Instant::now(); + let Some(v) = self.volume.as_mut() else { return }; + let was_open = v.is_open(); + let hub = &self.hub; + let caught = v.catch_up( + |at, max| hub.owed(at, max), + |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), + MAX_LOG_BYTES, + |line| say!("{line}"), + ); + match caught { + Ok(caught) => { + if !was_open { + self.retry = None; + say!("logkeeper: {DIR} answers again - {} holds this boot's lines", v.path()); + } + if !v.is_open() { + // A continuation's create was refused. + self.retry = Some(Retry::after(Instant::now())); + } else if caught { + (self.owed_since, self.urgent) = (None, false); + } + let took = began.elapsed(); + if took > LOG_WRITE_BUDGET && !self.degraded { + self.degraded = true; + toyos::warn!( + "logkeeper: {DIR} answers but slowly (the write took {took:?}) - degraded, nothing lost, \ + {} is still this boot's log", + v.path() + ); + } else if took <= LOG_WRITE_BUDGET && self.degraded { + self.degraded = false; + say!("logkeeper: {DIR} answers at pace again - {}", v.path()); + } + } + Err(e) if v.part() >= MAX_LOG_PARTS => { + toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); + self.volume = None; + } + Err(e) => { + v.lost(); + match self.retry.as_mut() { + Some(retry) if !was_open => retry.again(Instant::now()), + _ => { + toyos::error!( + "logkeeper: {DIR} has not answered ({e}) - {} is asked again, and this boot's lines \ + are kept until it answers", + v.path() + ); + self.retry = Some(Retry::after(Instant::now())); + } + } + } + } + } + + /// The stop: the file given the replay to its end, catching up until + /// `by`, and made durable with what it ends short by said in it. + fn finish(&mut self, by: Instant) { + self.stopped = true; + self.finished = false; + (self.owed_since, self.urgent) = (None, false); + if Instant::now() >= by { + toyos::error!("logkeeper: {DIR} took the stop's flush after its time - the file is as its last batch left it"); + return; + } + let Some(v) = self.volume.as_mut() else { return }; + let hub = &self.hub; + let finished = v.finish( + |at, max| hub.owed(at, max), + |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), + hub.end(), + MAX_LOG_BYTES, + || Instant::now() < by, + |short| { + let text = format!("logkeeper: this file ends {short} bytes short of the boot: the stop came before {DIR} took them"); + hub.own_line(Severity::Error, &text) + }, + |line| say!("{line}"), + ); + let short = hub.end() - v.through(); + match finished { + Ok(_) if short == 0 => {} + Ok(_) => toyos::error!("logkeeper: {} ends {short} bytes short of the boot at the stop", v.path()), + Err(e) => toyos::error!( + "logkeeper: {DIR} refused the stop's flush ({e}) - {} ends {short} bytes short of the boot", + v.path() + ), + } + self.finished = v.is_open(); + } + + /// The stop mark, over a file the stop's flush made durable, and before + /// `by`. Withheld, and no call made, unless the flush reached the file: + /// `/log`'s server may never have answered, and a part the volume refused + /// is not this boot's log, so the mark would call a stop clean whose log + /// is not on the volume. + fn mark(&mut self, by: Instant) -> bool { + if !self.finished { + toyos::warn!("logkeeper: {DIR} does not hold this boot's log, so the stop is not marked - the next boot will not count this one as clean"); + return false; + } + if Instant::now() >= by { + toyos::error!("logkeeper: the stop mark was asked of {DIR} after its time - the next boot will not count this one as clean"); + return false; + } + store::mark_stopped() + .inspect_err(|e| { + toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean") + }) + .is_ok() + } + + fn resume(&mut self, marked: bool) { + if marked && let Err(e) = store::unmark_stopped() { + toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); + } + (self.stopped, self.finished) = (false, false); + self.owed_since.get_or_insert_with(Instant::now); + self.urgent = true; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// **A batch is written at its size, at its age, or at once for a line + /// that matters**, and never for nothing. + #[test] + fn a_batch_falls_due_at_its_size_its_age_or_an_urgent_line() { + let since = Instant::now(); + let young = since + BATCH_AGE - Duration::from_nanos(1); + assert!(!batch_due(BATCH_BYTES - 1, Some(since), false, young)); + assert!(batch_due(BATCH_BYTES, Some(since), false, young), "a full batch waits for no age"); + assert!(batch_due(1, Some(since), false, since + BATCH_AGE), "an old line waits no longer"); + assert!(batch_due(1, Some(since), true, since), "an urgent line waits for nothing"); + assert!(!batch_due(0, Some(since), true, since + BATCH_AGE), "nothing owed is no write"); + assert!(urgent(Severity::Error) && urgent(Severity::Alert) && !urgent(Severity::Warn)); + } + + fn writer(dir: &str) -> Writer { + let volume = Volume::open(dir, Some("2026-11-11-120000".into()), MAX_LOG_BYTES, |_| {}); + Writer { + volume, + hub: Arc::new(Hub::start(1 << 20, None)), + published: Arc::new(Published::new()), + retry: None, + degraded: false, + owed_since: None, + urgent: false, + finished: false, + stopped: false, + } + } + + fn grew(w: &mut Writer, text: &[u8], urgent: bool, at: Instant) { + w.hub.append(text, None); + w.owed_since.get_or_insert(at); + w.urgent |= urgent; + } + + fn on_disk(w: &Writer) -> Vec { + std::fs::read(w.volume.as_ref().unwrap().path()).unwrap() + } + + /// **The replay reaches the file a batch at a time**, all of it at the + /// batch's age or at once for a line that matters. + #[test] + fn the_file_takes_the_replay_a_batch_at_a_time() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let t = Instant::now(); + grew(&mut w, b"one\n", false, t); + grew(&mut w, b"two\n", false, t + BATCH_AGE / 2); + assert!(!w.due(t + BATCH_AGE / 2)); + assert!(w.due(t + BATCH_AGE), "the first line's age is the batch's"); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\n"); + assert!(w.owed_since.is_none() && w.next_due().is_none(), "a file with everything waits on nothing"); + grew(&mut w, b"panicked\n", true, t + BATCH_AGE); + assert!(w.due(t + BATCH_AGE)); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\npanicked\n"); + } + + /// **From the stop's flush until it is refused the file takes no call**, + /// so the stop mark is the last write; and the mark is never asked of a + /// file the flush did not reach. A refused stop's lines are written at once. + #[test] + fn nothing_reaches_the_file_between_the_stops_flush_and_its_refusal() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let t = Instant::now(); + assert!(!w.mark(t + BATCH_AGE), "no flush, no mark"); + grew(&mut w, b"before\n", false, t); + w.finish(Instant::now() + BATCH_AGE); + assert!(w.finished); + assert_eq!(on_disk(&w), b"before\n"); + grew(&mut w, b"after\n", true, t); + w.volume.as_mut().unwrap().lost(); + w.retry = Some(Retry::after(t)); + assert!(!w.due(t + Duration::from_secs(3600)) && w.next_due().is_none(), "a stopped file is written nothing"); + w.resume(false); + let retry = w.retry.unwrap().at(); + assert_eq!(w.next_due(), Some(retry), "a refused part is asked again at its retry"); + assert!(w.due(retry)); + w.write(); + assert_eq!(on_disk(&w), b"before\nafter\n"); + } +} diff --git a/system/logkeeper/src/inspect.rs b/system/logkeeper/src/inspect.rs index 3f185a41fae..f907cc862c3 100644 --- a/system/logkeeper/src/inspect.rs +++ b/system/logkeeper/src/inspect.rs @@ -1,9 +1,9 @@ //! The `log` port: `inspect`'s answer about where this boot's log is going, //! and a reader on this machine handed the log itself ([`READ`]). //! -//! **On a thread of its own, so no reader can reach the loop that writes the -//! file.** The loop publishes what it knows once a round — a few relaxed -//! stores, and a lock only when the file it writes changes — and this thread +//! **On a thread of its own, so no reader can reach the loop or the file.** +//! They publish what they know — a few relaxed stores, and a lock only when +//! the file changes — and this thread //! owns the acceptor, reads each reader's request without ever blocking on it, //! answers in one non-blocking write and closes. A reader that connects and //! says nothing costs a slot and [`HANDSHAKE_TIMEOUT`], never the log. @@ -40,11 +40,11 @@ const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(2); /// The poll token of the acceptor, clear of every handle's own value. const ACCEPT: u64 = u64::MAX; -/// Where the file side of the loop stands. +/// Where the file stands. #[derive(Clone, Copy, PartialEq, Eq, Debug)] #[repr(u8)] pub enum State { - /// Every round written, flushed and published. + /// Every batch written and synced as it falls due. Writing = 0, /// The volume answers slower than a log is worth, or refused the part /// and is asked again: the replay keeps what the file is owed. @@ -67,11 +67,11 @@ impl State { } } -/// The loop's side of the answer. One writer, the loop; one reader, this -/// module's thread. +/// The answer's facts: the file's, published by the file's thread, and the +/// records lost, by the loop; one reader, this module's thread. pub struct Published { state: AtomicU8, - /// The part the loop last published, or 0 for none: the loop's own record + /// The part last published, or 0 for none: the file thread's own record /// of when `file` changes, and read by nothing else. part: AtomicU32, /// The part being written and its path, one pair under one lock, so no @@ -92,24 +92,27 @@ impl Published { } } - /// What the loop knows at the top of a round. `lost` is the records the - /// kernel overwrote before this reader reached them. - pub fn publish(&self, volume: Option<&Volume>, state: State, lost: u64) { + /// Where the file stands, from the file's thread. + pub fn publish(&self, volume: Option<&Volume>, state: State) { self.state.store(state as u8, Ordering::Relaxed); - self.lost.store(lost, Ordering::Relaxed); let part = volume.map_or(0, Volume::part); self.bytes.store(volume.map_or(0, Volume::bytes), Ordering::Relaxed); - // The lock is taken when the file changes and at no other round. + // The lock is taken when the file changes and at no other batch. if self.part.swap(part, Ordering::Relaxed) != part { *self.file.lock().expect("the inspect thread does not panic holding this") = volume.map(|v| (v.part(), v.path())); } } + /// The records the kernel overwrote before the loop reached them. + pub fn publish_lost(&self, lost: u64) { + self.lost.store(lost, Ordering::Relaxed); + } + fn snapshot(&self) -> Vec { let mut snap = Snapshot::new(toyos_inspect::LOG); snap.put("volume.state", State::word(self.state.load(Ordering::Relaxed))); - let file = self.file.lock().expect("the loop does not panic holding this").clone(); + let file = self.file.lock().expect("the file's thread does not panic holding this").clone(); if let Some((part, path)) = file { snap.put("volume.path", path); snap.put("volume.part", part); diff --git a/system/logkeeper/src/main.rs b/system/logkeeper/src/main.rs index 72197c9a77f..6d495a1ab44 100644 --- a/system/logkeeper/src/main.rs +++ b/system/logkeeper/src/main.rs @@ -40,19 +40,17 @@ //! //! # Three sinks, and only one of them is the sink of record //! -//! The file is. [`serve`]'s readers are the second: the same lines, in the same -//! order, to whoever asks, from the boot's first line however late they ask. -//! The console's carriers are the third ([`carrier`]): the console UART's -//! driver, and a VM's console channel, which the UART goes quiet for while it -//! carries the log. Neither of the two can slow the file: each is a thread -//! with an offset into the replay. The replay follows the file while the file -//! has all of it, and leads it while the part is refused or catching up. +//! The file is ([`file`]). [`serve`]'s readers are the second: the same +//! lines, in the same order, to whoever asks, from the boot's first line +//! however late they ask. The console's carriers are the third ([`carrier`]): +//! the console UART's driver, and a VM's console channel, which the UART goes +//! quiet for while it carries the log. Each of the three is a thread with an +//! offset into the replay the loop appends each round to, so none can slow +//! the loop or another. //! -//! **The volume is opened on a thread of its own**: /log's file server may -//! start late. Only the stop's flush waits for it, within [`STOP_CATCH_UP`]. -//! Until it opens, the readers and the console have each round as it is read; -//! once it opens, the file takes the boot so far from the replay, a part's -//! worth a round. +//! **The file is opened and written on its thread**: /log's file server may +//! start late, and may take a call and never answer. Only the stop waits for +//! the file, within [`STOP_FILE`]. //! //! # Its whole authority //! @@ -69,31 +67,31 @@ //! //! # Durability, which is a contract and not a hope //! -//! **A round's lines are written and the volume made durable before the next -//! round**, and again when the supervisor asks before the machine stops +//! **The file is written and made durable in batches** ([`file`]): within +//! [`file::BATCH_AGE`] of a line, at once for a line at `Error` or above, and +//! when the supervisor asks before the machine stops //! ([`toyos_logstream::FLUSH`]); once the stop has synced its files it asks -//! for the stop mark ([`toyos_logstream::MARK`]). The kernel waits on none of it, so -//! what it says after the flush, at a panic or in the stop, reaches only its -//! console and panel. +//! for the stop mark ([`toyos_logstream::MARK`]). The kernel waits on none of +//! it, so what it says after the flush, at a panic or in the stop, reaches +//! only its console and panel. //! **A program's line after that flush reaches the UART and is held back from -//! the file**, to a bound ([`STOPPING_BYTES`]): the stop syncs no file still open, -//! so the file ends where the flush made it durable, and a reader of the -//! served log is handed what the file holds and no more. A stop the kernel -//! refused is the supervisor's [`toyos_logstream::RESUME`], and what was held is written -//! then. +//! the replay**, to a bound ([`STOPPING_BYTES`]): the stop syncs no file still +//! open, so the file ends where the flush made it durable and the mark is its +//! last write. A stop the kernel refused is the supervisor's +//! [`toyos_logstream::RESUME`], and what was held is written then. //! //! **A volume that refuses, or whose server ends, is asked again** //! ([`policy::Retry`], and at the stop's flush whatever the wait), and its //! file given back what it missed ([`store::Volume`]). mod carrier; +mod file; mod inspect; mod origin; mod policy; mod serve; mod store; -use std::sync::mpsc::{self, RecvTimeoutError, TryRecvError}; use std::sync::Arc; use std::time::{Duration, Instant}; @@ -106,7 +104,7 @@ use toyos::say; use toyos::syscap::SysCap; use toyos::wake::Bell; use toyos_logstream::{ - ProgramLine, Registration, Severity, Tag, CARRIER, FLUSH, FLUSHED, LOG_OPENED, MARK, MARKED, MAX_TAG, ORIGINS, REGISTER, RESUME, + ProgramLine, Registration, Tag, CARRIER, FLUSH, FLUSHED, MARK, MARKED, MAX_TAG, ORIGINS, REGISTER, RESUME, SERVICE, WIRE, }; use toyos_wallclock::Civil; @@ -114,8 +112,8 @@ use toyos_wallclock::Civil; use carrier::Carriers; use serve::Positions; use origin::{Origin, Said}; -use policy::{Retry, Step, LOG_WRITE_BUDGET}; -use store::{Volume, DIR, MAX_LOG_BYTES, MAX_LOG_FILES, MAX_LOG_PARTS}; +use file::File; +use store::{DIR, MAX_LOG_BYTES, MAX_LOG_FILES}; /// Records asked of `SYS_LOG_READ` at once: large enough that an ordinary /// boot's burst is a handful of syscalls rather than one per line. @@ -157,31 +155,11 @@ fn main() { // this program nothing new. let (stem, boot_secs, dated) = boot_stamp(); - // Opened off the loop: a request on /log waits for its file server, which - // the supervisor starts only with storage. - let (opened, opening) = mpsc::sync_channel(1); - std::thread::Builder::new() - .name("log-volume".into()) - .spawn(move || { - let volume = Volume::open(DIR, stem, MAX_LOG_BYTES, |line| say!("{line}")); - match &volume { - // This program's half of the startup report, in one line: the - // kernel says whether it has a console, this program whether it - // has a volume and what the name it chose was decided by. - Some(v) if v.is_open() => say!("{LOG_OPENED}{} ({dated})", v.path()), - Some(v) => say!( - "logkeeper: {} is asked again - this boot's lines are kept until {DIR} takes them ({dated})", - v.path() - ), - // `Volume::open` has said why. - None => say!("logkeeper: this boot's kernel log is on the console only ({dated})"), - } - opened.send(volume).expect("logkeeper: the loop holds the volume's receiver for good"); - }) - .expect("logkeeper: the volume's thread could not be started"); - let hub = Arc::new(serve::Hub::start(REPLAY_BYTES, boot_secs)); let published = Arc::new(inspect::Published::new()); + // A request on /log waits for its file server, which the supervisor + // starts only with storage: the file is opened and written off the loop. + let file = File::start(stem, dated, Arc::clone(&hub), Arc::clone(&published)); if let Some(acceptor) = endow::acceptor(SERVICE) { inspect::serve(acceptor, Arc::clone(&published), Arc::clone(&hub)); } @@ -210,10 +188,7 @@ fn main() { records: vec![Record::EMPTY; BATCH], lost: 0, waiting: Vec::new(), - volume: None, - opening: Some(opening), - retry: None, - degraded: false, + file, boot_secs, hub, stopping: None, @@ -236,13 +211,7 @@ struct Log { /// Lines stamped after the round they were read in began, which the next /// round merges. waiting: Vec, - volume: Option, - /// The volume's opener, until it answers. - opening: Option>>, - /// When a part the volume refused is next opened again. - retry: Option, - /// Whether the volume answers, slower than `LOG_WRITE_BUDGET` a round. - degraded: bool, + file: File, boot_secs: Option, hub: Arc, /// The supervisor's flush was answered and the machine stops: the file's text held @@ -297,7 +266,12 @@ struct Stopping { /// it does not reach is said, in the file and on the console. const STOP_CATCH_UP: Duration = Duration::from_millis(toyos_quiesce::FLUSH_MS / 2); -const OPENER_ENDED: &str = "logkeeper: the volume's thread ended without answering"; +/// How long the loop waits for the file to answer the stop's flush or its +/// mark, each of which the supervisor bounds by [`toyos_quiesce::FLUSH_MS`]: +/// past [`STOP_CATCH_UP`] by the final sync's time, and short of the bound by +/// the answer's. A file that has not answered by then is given up on and +/// said, and the stop goes on. +const STOP_FILE: Duration = Duration::from_millis(toyos_quiesce::FLUSH_MS * 3 / 4); /// Text a stop may hold back from the file: a refused stop writes it, and /// what is past it is counted and said then. @@ -323,15 +297,7 @@ impl Log { // Programs whose end a watch reported, until a round has swept them. let mut ended: Vec = Vec::new(); loop { - self.adopt(); - let state = match (&self.volume, self.degraded) { - (None, _) if self.opening.is_some() => inspect::State::Opening, - (None, _) => inspect::State::ConsoleOnly, - (Some(v), _) if !v.is_open() => inspect::State::Degraded, - (Some(_), true) => inspect::State::Degraded, - (Some(_), false) => inspect::State::Writing, - }; - published.publish(self.volume.as_ref(), state, self.tail.lost()); + published.publish_lost(self.tail.lost()); // **Armed before anything is read**, in the shape every reader of // an edge needs: what arrives after a read and before the park has @@ -525,7 +491,7 @@ impl Log { lines.into_iter().partition(|line| all || line.at_ns <= cut); let through = written_through(self.tail.positions(), &later); self.waiting = later; - self.write(now, all, Arc::clone(&through)); + self.write(now, Arc::clone(&through)); self.start_wire(&through); oldest } @@ -565,11 +531,11 @@ impl Log { out } - /// The lines, in stamp order, into the file, the readers and the console's - /// carriers, which are the kernel's log through the read positions - /// `through`; while the machine stops, held back from the file, and each - /// program's line to the UART's carrier. - fn write(&mut self, mut lines: Vec, flushing: bool, through: Positions) { + /// The lines, in stamp order, into the replay the file, the readers and + /// the console's carriers take them from, which are the kernel's log + /// through the read positions `through`; while the machine stops, held + /// back from the replay, and each program's line to the UART's carrier. + fn write(&mut self, mut lines: Vec, through: Positions) { if lines.is_empty() { return; } @@ -579,9 +545,11 @@ impl Log { // are yet to be read past. let mut wire = String::new(); let mut programs = String::new(); + let mut urgent = false; for line in &lines { match &line.kind { Kind::Kernel(record) => { + urgent |= record.severity().is_some_and(file::urgent); let text = format!("{}\n", record.dated(&stamp(self.boot_secs, record.at_ns()))); if !self.handover.as_ref().is_some_and(|handover| on_the_uart(record, &handover.spoken)) { wire.push_str(&text); @@ -589,6 +557,7 @@ impl Log { file.push_str(&text); } Kind::Program { tag, owner, said } => { + urgent |= file::urgent(said.severity); let tag = Tag::new(tag).expect("an origin's name is a tag"); let pid = (said.pid != *owner).then_some(said.pid); let at = stamp(self.boot_secs, Some(said.at_ns)); @@ -621,11 +590,8 @@ impl Log { } return; } - // The file first: while it has all of the replay, a reader is served - // only what /log already holds, whenever the machine stops between the - // two. A part refused or catching up takes this text from the replay. - self.to_volume(file.as_bytes(), !flushing); self.hub.append(file.as_bytes(), Some(through)); + self.file.grew(urgent); if let Some(handover) = &mut self.handover { handover.owed.push_str(&wire); } @@ -648,211 +614,16 @@ impl Log { self.hub.set_wire_from(self.hub.end(), handover.owed.into_bytes()); } - /// Write a round to the volume, and make it durable unless it is one of a - /// flush's rounds, which `flushed` makes durable together: a stop's flush - /// is bounded by the supervisor, and a sync per round of it is what outruns that. - fn to_volume(&mut self, text: &[u8], sync: bool) { - if !self.caught_up() { - return; - } - let Some(v) = self.volume.as_mut() else { return }; - let began = Instant::now(); - let mut refused = v.write(text).err().map(|e| (Step::Append, e.to_string())); - if refused.is_none() && sync { - refused = self.sync().err(); - } - // A volume that answered, and took longer than a log is worth doing it. - if refused.is_none() && began.elapsed() > LOG_WRITE_BUDGET { - refused = Some((Step::TooSlow, format!("it took {:?}", began.elapsed()))); - } - self.answered(refused); - } - - /// The volume, once its opener answers outside a stop. Its file stands - /// at the replay's start and is given the boot so far, a part's worth - /// here and a part's worth each round after. - fn adopt(&mut self) { - if self.stopping.is_some() { - return; - } - let Some(opening) = &self.opening else { return }; - match opening.try_recv() { - Ok(volume) => { - self.took(volume); - self.caught_up(); - } - Err(TryRecvError::Empty) => {} - Err(TryRecvError::Disconnected) => panic!("{OPENER_ENDED}"), - } - } - - /// The opener's answer. - fn took(&mut self, volume: Option) { - self.opening = None; - self.volume = volume; - } - - /// Whether the file has every byte the replay has, so a round's text - /// goes after them. A part the volume refused is opened again once the - /// retry is due, and the file is given what it is owed, a part's worth a - /// round. - fn caught_up(&mut self) -> bool { - let now = Instant::now(); - let Some(v) = self.volume.as_mut() else { return false }; - let was_open = v.is_open(); - if was_open && v.through() == self.hub.end() { - return true; - } - if !was_open && !self.retry.get_or_insert(Retry::after(now)).due(now) { - return false; - } - let hub = &self.hub; - let caught = v.catch_up( - |at, max| hub.owed(at, max), - |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), - MAX_LOG_BYTES, - |line| say!("{line}"), - ); - match caught { - Ok(caught) => { - if !was_open { - self.retry = None; - say!("logkeeper: {DIR} answers again - {} holds this boot's lines", v.path()); - } - if !v.is_open() { - // A continuation's create was refused. - self.retry = Some(Retry::after(now)); - } - caught && v.is_open() - } - Err(e) if v.part() >= MAX_LOG_PARTS => { - toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); - self.volume = None; - false - } - Err(e) => { - v.lost(); - match self.retry.as_mut() { - Some(retry) if !was_open => retry.again(now), - _ => { - toyos::error!("logkeeper: {DIR} has not answered ({e}) - {} is asked again", v.path()); - self.retry = Some(Retry::after(now)); - } - } - false - } - } - } - - /// Make the volume durable now. - fn sync(&mut self) -> Result<(), (Step, String)> { - let Some(v) = self.volume.as_mut() else { return Ok(()) }; - v.sync().map_err(|e| (Step::Flush, e.to_string())) - } - - /// What a round's write came to: rotation after a clean one, and the - /// give-up policy after a refused one. - fn answered(&mut self, refused: Option<(Step, String)>) { - let Some(path) = self.volume.as_ref().map(Volume::path) else { return }; - let Some((step, why)) = refused else { - if self.degraded { - self.degraded = false; - say!("logkeeper: {DIR} answers at pace again - {path}"); - } - self.rotate_if_full(); - return; - }; - match step { - // Say so once, keep the lines in the replay, and ask again. - Step::Append | Step::Flush => { - toyos::error!( - "logkeeper: {DIR} has not answered ({}: {why}) - {path} is asked again, and this \ - boot's lines are kept until it answers", - step.as_str() - ); - self.lost(); - } - // Every call answered, slowly: the round is durable. - Step::TooSlow => { - if !self.degraded { - self.degraded = true; - toyos::warn!( - "logkeeper: {DIR} answers but slowly ({}: {why}) - degraded, nothing lost, \ - {path} is still this boot's log", - step.as_str() - ); - } - self.rotate_if_full(); - } - } - } - - fn rotate_if_full(&mut self) { - let Some(v) = self.volume.as_mut() else { return }; - if !v.full() { - return; - } - match v.rotate(|line| say!("{line}")) { - Ok(()) if v.is_open() => {} - Ok(()) => self.lost(), - Err(e) if v.part() >= MAX_LOG_PARTS => { - toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); - self.volume = None; - } - Err(e) => { - toyos::error!("logkeeper: {DIR} has not answered ({e}) - {} is asked again", v.path()); - self.lost(); - } - } - } - - /// The volume refused the part: it is asked again once the retry is due. - fn lost(&mut self) { - if let Some(v) = self.volume.as_mut() { - v.lost(); - } - self.retry = Some(Retry::after(Instant::now())); - } - - /// The supervisor asked for the log whole before the machine stops: every line is - /// written by now, so the volume is made durable, and the supervisor is told. + /// The supervisor asked for the log whole before the machine stops: every + /// line is in the replay by now, so the file is given the rest and made + /// durable, within [`STOP_FILE`], and the supervisor is told. fn flushed(&mut self) { - let deadline = Instant::now() + STOP_CATCH_UP; - // An opener that has answered since the loop last asked, or answers - // in the stop's own time: the file takes the boot in `finish` below, - // inside the deadline. - if let Some(opening) = &self.opening { - match opening.recv_timeout(deadline.saturating_duration_since(Instant::now())) { - Ok(volume) => self.took(volume), - Err(RecvTimeoutError::Timeout) => { - toyos::error!("logkeeper: {DIR} had not opened by the stop - this boot's log is on the console only") - } - Err(RecvTimeoutError::Disconnected) => panic!("{OPENER_ENDED}"), - } - } - if let Some(v) = self.volume.as_mut() { - let hub = &self.hub; - let finished = v.finish( - |at, max| hub.owed(at, max), - |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), - hub.end(), - MAX_LOG_BYTES, - || Instant::now() < deadline, - |short| { - let text = format!("logkeeper: this file ends {short} bytes short of the boot: the stop came before {DIR} took them"); - hub.own_line(Severity::Error, &text) - }, - |line| say!("{line}"), + let began = Instant::now(); + if !self.file.finish(began + STOP_CATCH_UP, began + STOP_FILE) { + toyos::error!( + "logkeeper: {DIR} had not answered the stop's flush in {} ms - this boot's end may be on the console only", + STOP_FILE.as_millis() ); - let short = hub.end() - v.through(); - match finished { - Ok(_) if short == 0 => {} - Ok(_) => toyos::error!("logkeeper: {} ends {short} bytes short of the boot at the stop", v.path()), - Err(e) => toyos::error!( - "logkeeper: {DIR} refused the stop's flush ({e}) - {} ends {short} bytes short of the boot", - v.path() - ), - } } // The console's carriers take the log up to here, and the UART what // the machine says from here as it stops. @@ -866,25 +637,19 @@ impl Log { } /// The supervisor's stop has synced its files and is about to stop its - /// drivers: write the stop mark over a durable log, and say so. The mark - /// is withheld, and no call made, unless the part is open: `/log`'s - /// server may never have answered, and a part the volume refused is not - /// this boot's log, so the mark would call a stop clean whose log is not - /// on the volume. + /// drivers: the file writes the stop mark over a durable log, within + /// [`STOP_FILE`], and the supervisor is told. fn mark(&mut self) { let Some(stopping) = self.stopping.as_mut() else { panic!("logkeeper: the supervisor asked for the stop mark, and no stop was flushed for"); }; - let open = self.volume.as_ref().is_some_and(Volume::is_open); - if !open { - toyos::warn!("logkeeper: {DIR} does not hold this boot's log, so the stop is not marked - the next boot will not count this one as clean"); - } - stopping.marked = open - && store::mark_stopped() - .inspect_err(|e| { - toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean") - }) - .is_ok(); + stopping.marked = self.file.mark(Instant::now() + STOP_FILE).unwrap_or_else(|| { + toyos::error!( + "logkeeper: {DIR} had not answered the stop mark in {} ms - the next boot will not count this one as clean", + STOP_FILE.as_millis() + ); + false + }); if let Err(e) = self.from_supervisor.signal(MARKED) { panic!("logkeeper: the supervisor could not be told the stop is marked: {e:?}"); } @@ -897,11 +662,6 @@ impl Log { panic!("logkeeper: the supervisor said the machine runs on, and no stop was flushed for"); }; toyos::warn!("logkeeper: the stop was refused, so {DIR} takes this boot's lines again"); - if held.marked - && let Err(e) = store::unmark_stopped() - { - toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); - } if held.unwritten > 0 { toyos::warn!( "logkeeper: {} line(s) said while the machine stopped were past {STOPPING_BYTES} bytes \ @@ -909,8 +669,8 @@ impl Log { held.unwritten ); } - self.to_volume(held.text.as_bytes(), true); self.hub.resume(held.text.as_bytes()); + self.file.resume(held.marked); if let Some(carriers) = &self.carriers { carriers.resume(); } diff --git a/system/logkeeper/src/policy.rs b/system/logkeeper/src/policy.rs index 4f495f08ac5..4a31ee723d2 100644 --- a/system/logkeeper/src/policy.rs +++ b/system/logkeeper/src/policy.rs @@ -1,6 +1,6 @@ use std::time::{Duration, Instant}; -/// The round time past which a volume that answered every call is announced +/// The batch time past which a volume that answered every call is announced /// slow. Nothing in this program ends a volume for slowness: only the device's /// own refusal does. /// @@ -34,6 +34,10 @@ impl Retry { now >= self.at } + pub fn at(&self) -> Instant { + self.at + } + /// Asked again at `now` and refused: twice the wait, to [`RETRY_MAX`]. pub fn again(&mut self, now: Instant) { self.wait = (self.wait * 2).min(RETRY_MAX); @@ -41,29 +45,6 @@ impl Retry { } } -/// Which call refused. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum Step { - /// One of the batch's lines did not reach the file. - Append, - /// The bytes are in the file and did not reach the device. - Flush, - /// Everything answered, and the round took longer than - /// [`LOG_WRITE_BUDGET`]. - TooSlow, -} - -impl Step { - /// The word this program's console line uses for it. - pub fn as_str(self) -> &'static str { - match self { - Self::Append => "the append", - Self::Flush => "the sync", - Self::TooSlow => "the write", - } - } -} - #[cfg(test)] mod tests { use super::*; @@ -83,12 +64,4 @@ mod tests { } assert_eq!(waits, [1, 2, 4, 8, 16, 32, 64, 64, 64]); } - - /// The console words, which the table in `main` is written against. - #[test] - fn every_step_names_itself() { - assert_eq!(Step::Append.as_str(), "the append"); - assert_eq!(Step::Flush.as_str(), "the sync"); - assert_eq!(Step::TooSlow.as_str(), "the write"); - } } diff --git a/system/logkeeper/src/serve.rs b/system/logkeeper/src/serve.rs index 2be0559ecf4..dc56f277169 100644 --- a/system/logkeeper/src/serve.rs +++ b/system/logkeeper/src/serve.rs @@ -3,17 +3,17 @@ //! //! **Every reader gets the boot from its first line**, however late it //! connects: the main loop appends each round's lines to one -//! [`toyos_logstream::Replay`], after the file has them while the file has -//! all of the replay; a part not open yet, refused or catching up takes them -//! from the replay later. A reader is a thread with an offset into it. So the -//! same text reaches `/log` and every reader, in the same order, and nothing -//! a reader does can reach the file. +//! [`toyos_logstream::Replay`], which the file (`crate::file`) takes its +//! batches from. A reader is a thread with an offset into it. So the same +//! text reaches `/log` and every reader, in the same order, and nothing a +//! reader does can reach the file. //! //! A reader that takes none for [`STALLED`] while bytes are owed to it is let //! go, and that is a line in the log. //! -//! **A reader that is caught up waits on the replay growing**, and nothing -//! else wakes it: there is no poll and no timer. +//! **A reader that is caught up waits on the replay growing or on its reader +//! leaving**, whichever comes first: a reader that leaves while the log is +//! quiet gives back its thread and its slot then. There is no timer. //! //! The console's carriers read the same replay (`crate::carrier`): the hub //! keeps the kernel's read positions each round is written through @@ -28,7 +28,8 @@ use std::sync::{Arc, Condvar, Mutex}; use std::time::Duration; use toyos::ipc::Connection; -use toyos::poller::{Poller, WRITABLE}; +use toyos::poller::{Poller, OTHER_END_GONE, READABLE, WRITABLE}; +use toyos::wake::{self, Bell, Waker}; use toyos::{AsHandle, Pipe}; use toyos_abi::syscall::SyscallError; use toyos::say; @@ -79,6 +80,9 @@ struct Shared { replay: Mutex, grew: Condvar, readers: AtomicUsize, + /// A wake per reader, rung as the replay grows: a reader waits on its + /// own end beside it, which a condition variable cannot. + reader_wakes: Mutex>, /// The wall clock the boot started at, for a line a reader is owed. boot_secs: Option, } @@ -95,6 +99,7 @@ impl Hub { }), grew: Condvar::new(), readers: AtomicUsize::new(0), + reader_wakes: Mutex::new(Vec::new()), boot_secs, }); Self { shared } @@ -104,7 +109,7 @@ impl Hub { /// of a pipe of its own, which the log is written into. pub fn read(&self, conn: &Connection) { let Some(pipe) = self.hand_over(conn) else { return }; - admit(&self.shared, format!("reader {}", conn.as_handle().raw().0), PipeSink::new(pipe)); + admit(&self.shared, format!("reader {}", conn.as_handle().raw().0), pipe); } /// The lines the file has just taken, for every reader, and the kernel's @@ -121,6 +126,7 @@ impl Hub { } drop(held); self.shared.grew.notify_all(); + self.shared.reader_wakes.lock().expect("logkeeper: a reader's wake is poisoned").retain(|w| w.ring().is_ok()); } /// The positions of the last round wholly before `at`: what a channel @@ -264,16 +270,27 @@ enum Left { } /// A reader thread, where its count allows one. -fn admit(shared: &Arc, who: String, sink: impl Write + Send + 'static) { +fn admit(shared: &Arc, who: String, pipe: Pipe) { let count = &shared.readers; if count.fetch_add(1, Ordering::SeqCst) >= MAX_READERS { count.fetch_sub(1, Ordering::SeqCst); say!("logkeeper: refusing {who}: {MAX_READERS} readers on this machine are already served"); return; } + let (waker, bell) = match wake::pair() { + Ok(ends) => ends, + Err(e) => { + count.fetch_sub(1, Ordering::SeqCst); + say!("logkeeper: no wake for {who}: {e:?}"); + return; + } + }; + // Before the thread reads the replay, so no growth after its first look + // goes unrung. + shared.reader_wakes.lock().expect("logkeeper: a reader's wake is poisoned").push(waker); let theirs = Arc::clone(shared); let spawned = std::thread::Builder::new().name("log-reader".into()).spawn(move || { - let (sent, left) = feed(&theirs, sink); + let (sent, left) = feed(&theirs, PipeSink::new(pipe, bell)); theirs.readers.fetch_sub(1, Ordering::SeqCst); match left { Left::Stalled => say!( @@ -291,11 +308,16 @@ fn admit(shared: &Arc, who: String, sink: impl Write + Send + 'static) { /// Write the boot to `sink` from its first byte, then each round as it lands, /// until the reader goes or stalls. Answers the bytes it sent and why it ended. -fn feed(shared: &Shared, mut sink: impl Write) -> (u64, Left) { +fn feed(shared: &Shared, mut sink: PipeSink) -> (u64, Left) { let mut at = 0u64; let mut sent = 0u64; loop { - let chunk = next(shared, &mut at); + let Some(chunk) = next(shared, &mut at) else { + if !sink.await_more() { + return (sent, Left::Gone); + } + continue; + }; match sink.write_all(&chunk) { Ok(()) => sent += chunk.len() as u64, Err(e) if e.kind() == std::io::ErrorKind::TimedOut => return (sent, Left::Stalled), @@ -304,23 +326,20 @@ fn feed(shared: &Shared, mut sink: impl Write) -> (u64, Left) { } } -/// The bytes after `at`, at most [`CHUNK`] of them, once there are any; `at` -/// moves past them. A reader the replay no longer holds the bytes of is -/// handed the line that says so instead. -fn next(shared: &Shared, at: &mut u64) -> Vec { - let mut held = shared.replay.lock().expect("logkeeper: the replay is poisoned"); - loop { - match held.replay.next(*at, CHUNK) { - Next::Bytes(bytes) => { - *at += bytes.len() as u64; - return bytes.to_vec(); - } - Next::Evicted { lost, at: resume } => { - *at = resume; - return evicted(shared.boot_secs, lost); - } - Next::CaughtUp => held = shared.grew.wait(held).expect("logkeeper: the replay is poisoned"), +/// The bytes after `at`, at most [`CHUNK`] of them, or `None` where there are +/// none yet; `at` moves past them. A reader the replay no longer holds the +/// bytes of is handed the line that says so instead. +fn next(shared: &Shared, at: &mut u64) -> Option> { + match shared.replay.lock().expect("logkeeper: the replay is poisoned").replay.next(*at, CHUNK) { + Next::Bytes(bytes) => { + *at += bytes.len() as u64; + Some(bytes.to_vec()) + } + Next::Evicted { lost, at: resume } => { + *at = resume; + Some(evicted(shared.boot_secs, lost)) } + Next::CaughtUp => None, } } @@ -339,33 +358,84 @@ fn own_line(boot_secs: Option, severity: Severity, text: &str) -> Vec { format!("{line}\n").into_bytes() } -/// A pipe as a byte sink: one `write` takes what fits, and a full pipe is -/// waited on for [`STALLED`] and no longer. +/// The tokens of a reader's watches: room in its pipe, the replay grown, and +/// its reader gone. +const ROOM: u64 = 0; +const MORE: u64 = 1; +const GONE: u64 = 2; + +/// A reader's pipe as a byte sink: one `write` takes what fits, and a full +/// pipe is waited on for [`STALLED`] and no longer. Its watches are one-shot, +/// so each is armed again only once it has answered. struct PipeSink { pipe: Pipe, + /// Rung as the replay grows. + more: Bell, poller: Poller, + armed: [bool; 3], + gone: bool, } impl PipeSink { - fn new(pipe: Pipe) -> Self { - Self { pipe, poller: Poller::new(1) } + fn new(pipe: Pipe, more: Bell) -> Self { + Self { pipe, more, poller: Poller::new(3), armed: [false; 3], gone: false } + } + + fn arm(&mut self, token: u64) { + if !std::mem::replace(&mut self.armed[token as usize], true) { + match token { + ROOM => self.poller.watch(&self.pipe, WRITABLE, ROOM), + MORE => self.poller.watch(&self.more, READABLE, MORE), + _ => self.poller.watch(&self.pipe, OTHER_END_GONE, GONE), + } + } + } + + /// Wait at most `timeout_nanos` for any armed watch; the tokens answered. + fn wait(&mut self, timeout_nanos: u64) -> Vec { + self.arm(GONE); + let mut answered = Vec::new(); + self.poller.wait(1, timeout_nanos, |token| answered.push(token)); + for &token in &answered { + self.armed[token as usize] = false; + self.gone |= token == GONE; + } + answered + } + + /// Wait until the replay may have grown: `false` once the reader is gone. + fn await_more(&mut self) -> bool { + loop { + if self.gone { + return false; + } + if self.more.take() { + return true; + } + self.arm(MORE); + self.wait(u64::MAX); + } } } impl Write for PipeSink { fn write(&mut self, buf: &[u8]) -> std::io::Result { + let until = std::time::Instant::now() + STALLED; loop { match self.pipe.write_nonblock(buf) { Ok(n) => return Ok(n), Err(SyscallError::WouldBlock) => {} Err(e) => return Err(std::io::Error::other(format!("{e:?}"))), } - self.poller.watch(&self.pipe, WRITABLE, 0); - let mut room = false; - self.poller.wait(1, STALLED.as_nanos() as u64, |_| room = true); - if !room { + let left = until.saturating_duration_since(std::time::Instant::now()); + if self.gone { + return Err(std::io::ErrorKind::BrokenPipe.into()); + } + if left.is_zero() { return Err(std::io::ErrorKind::TimedOut.into()); } + self.arm(ROOM); + self.wait(left.as_nanos() as u64); } } diff --git a/system/logkeeper/src/store.rs b/system/logkeeper/src/store.rs index 38bd7b0911d..5a3417f09cb 100644 --- a/system/logkeeper/src/store.rs +++ b/system/logkeeper/src/store.rs @@ -460,9 +460,8 @@ mod tests { format!("[{lost} bytes let go]\n").into_bytes() } - /// One of the loop's rounds: the file given what it is owed, then the - /// round's text written and synced where it is caught up, then the - /// replay's. + /// A batch: the file given what it is owed, then the text written and + /// synced where it is caught up, then the replay's. fn round(v: &mut Volume, hub: &Hub, text: &[u8]) { let caught = (v.is_open() && v.through() == hub.end()) || v.catch_up(|at, max| hub.owed(at, max), note, MAX_LOG_BYTES, |_| {}).unwrap_or(false); From 58f62292d860dd6188c3be35eb4a7a2dcceceb77 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 16:45:21 +0200 Subject: [PATCH 3/7] logkeeper: what a reader is served and what the flush answers, as batching left them A reader of the served log is handed each round as logkeeper reads it, before the file has it, and the stop's flush answers within logkeeper's own bound even where the volume does not. Co-Authored-By: Claude Opus 5.5 --- lib/logstream/src/lib.rs | 9 +++++---- tests/guest/rust-tests/src/served_log.rs | 3 +-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/logstream/src/lib.rs b/lib/logstream/src/lib.rs index d5198e92440..b3b9020bde5 100644 --- a/lib/logstream/src/lib.rs +++ b/lib/logstream/src/lib.rs @@ -122,10 +122,11 @@ pub const REGISTER: u32 = 1; pub const SERVED: u32 = 2; /// The supervisor → `logkeeper`: the machine is about to be stopped. `logkeeper` reads every ring -/// and the kernel's records, writes them, makes the volume durable — or says, -/// in the file and on the console, what of the boot it could not — and -/// answers [`FLUSHED`]; the supervisor asks for the stop once it has, or once its bound -/// is spent. +/// and the kernel's records, writes them, makes the volume durable — or says +/// what of the boot it could not, in the file where the volume answers and on +/// the console, and gives up on a volume that does not answer within its own +/// bound — and answers [`FLUSHED`]; the supervisor asks for the stop once it +/// has, or once its bound is spent. pub const FLUSH: u32 = 5; pub const FLUSHED: u32 = 6; diff --git a/tests/guest/rust-tests/src/served_log.rs b/tests/guest/rust-tests/src/served_log.rs index 716be5f9885..5ac95f23b07 100644 --- a/tests/guest/rust-tests/src/served_log.rs +++ b/tests/guest/rust-tests/src/served_log.rs @@ -1,6 +1,5 @@ //! This boot's log as logkeeper serves it on test-runner's `log` port, from -//! its first line. A reader is handed each round only after it is on the -//! stick. +//! its first line, each round as logkeeper reads it. use std::time::{Duration, Instant}; From ab336ebb3e043a08e127b9c61a6fdf4b5a6a6060 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 17:39:34 +0200 Subject: [PATCH 4/7] logkeeper: rounds the file has not taken are one ask, and what a crash costs is stated as far as it holds The loop told the file's thread of every round with a message on an unbounded channel, so a `/log` call that never returns left a queue growing by a message a round behind it. Growth is now one atomic beside the channel: the loop sends a `Grew` only when none is unread, and the thread takes the growth, urgency included, whole. The manual and the file's header said the console never loses what a crash costs the file; the console is a sink that drops under load like any other. They now say the console is not batched, and the header bounds the loss for a volume that answers. `BATCH_BYTES` states its policy. The batch test takes its instant by a checked subtraction (clippy's adopted `unchecked_time_subtraction`). Co-Authored-By: Claude Opus 5.5 --- docs/src/architecture/diagnostics.md | 2 +- system/logkeeper/src/file.rs | 78 ++++++++++++++++++++++------ 2 files changed, 62 insertions(+), 18 deletions(-) diff --git a/docs/src/architecture/diagnostics.md b/docs/src/architecture/diagnostics.md index f24903c2f22..246ef74f5a4 100644 --- a/docs/src/architecture/diagnostics.md +++ b/docs/src/architecture/diagnostics.md @@ -24,7 +24,7 @@ exports them. and keeps every boot's beginning. Severity is an ordered ladder. - The file is written in batches: at once for an error, an alert or the machine's stop, and otherwise within a second. A machine that dies without - stopping loses at most that second from its file, never from its console. + stopping loses at most that second of its file; the console is not batched. No file call holds up the console or a reader, and a stop waits on the file within a bound. - A slow sink drops and counts; it never blocks a producer and fails alone. diff --git a/system/logkeeper/src/file.rs b/system/logkeeper/src/file.rs index 38808292fcc..4b31167d2c0 100644 --- a/system/logkeeper/src/file.rs +++ b/system/logkeeper/src/file.rs @@ -13,11 +13,13 @@ //! stderr, a fault the kernel reports an `Alert`) and at the stop's flush. //! Each batch is written and synced whole. //! -//! **What a crash costs the file** is the batch not yet written: at most -//! [`BATCH_AGE`] of lines below `Error`, which the console carried as they -//! were read. A stop writes the batch before the stop mark, and after the -//! mark this thread makes no call until the stop is refused. +//! **What a crash costs the file** is the batch not yet written: on a volume +//! that answers, at most [`BATCH_AGE`] of lines below `Error`, which the +//! console was handed as they were read. From the stop's flush until the stop +//! is refused this thread makes no call but the stop mark, so the mark is the +//! volume's last write from this program. +use std::sync::atomic::{AtomicU8, Ordering}; use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender, SyncSender}; use std::sync::Arc; use std::time::{Duration, Instant}; @@ -39,9 +41,10 @@ use crate::store::{self, Volume, DIR, MAX_LOG_BYTES, MAX_LOG_PARTS}; /// second says. pub const BATCH_AGE: Duration = Duration::from_secs(1); -/// How much owed text is written at once, whatever its age: a flood is -/// written in pieces this size rather than held for [`BATCH_AGE`], so no one -/// write is a burst the stick's other users wait behind. +/// How much owed text is written whatever its age. Policy: a flood reaches +/// the file in writes of about this size rather than in one large one a +/// [`BATCH_AGE`], and at this size the fixed cost of a write is small beside +/// what it carries. pub const BATCH_BYTES: u64 = 64 * 1024; /// Whether a batch is written at `now`: `owed` bytes, the oldest owed since @@ -55,9 +58,17 @@ pub fn urgent(severity: Severity) -> bool { severity >= Severity::Error } +/// The replay's growth the file's thread has not taken yet, one atomic shared +/// by the two: [`TOLD`] while an [`Ask::Grew`] is in the channel, so any +/// number of rounds is one ask and a file whose `/log` call never returns has +/// no more than that queued; [`URGENT`] for a line among them at `Error` or +/// above. +const TOLD: u8 = 1; +const URGENT: u8 = 2; + enum Ask { - /// The replay grew; `urgent` where by a line written at once. - Grew { urgent: bool }, + /// The replay grew: what by is in the shared growth. + Grew, /// The stop: reach the replay's end and sync, stopping the catch-up at /// `by`, and answer. Finish { by: Instant, answer: SyncSender<()> }, @@ -71,6 +82,7 @@ enum Ask { /// The loop's handle on the file's thread. pub struct File { asks: Sender, + growth: Arc, } impl File { @@ -78,6 +90,8 @@ impl File { /// (an undated boot's where `None`) and says so with `dated`. pub fn start(stem: Option, dated: String, hub: Arc, published: Arc) -> Self { let (asks, asked) = mpsc::channel(); + let growth = Arc::new(AtomicU8::new(0)); + let theirs = Arc::clone(&growth); std::thread::Builder::new() .name("log-file".into()) .spawn(move || { @@ -88,6 +102,7 @@ impl File { volume, hub, published, + growth: theirs, retry, degraded: false, owed_since: None, @@ -98,12 +113,16 @@ impl File { writer.run(asked); }) .expect("logkeeper: the file's thread could not be started"); - Self { asks } + Self { asks, growth } } - /// The replay grew by a round. + /// The replay grew by a round, holding a line written at once where + /// `urgent`. pub fn grew(&self, urgent: bool) { - self.ask(Ask::Grew { urgent }); + let bits = if urgent { TOLD | URGENT } else { TOLD }; + if self.growth.fetch_or(bits, Ordering::AcqRel) & TOLD == 0 { + self.ask(Ask::Grew); + } } /// The stop's flush, its catch-up ended at `by`: whether the file @@ -153,6 +172,7 @@ struct Writer { volume: Option, hub: Arc, published: Arc, + growth: Arc, /// When a part the volume refused is next opened again. retry: Option, /// Whether the volume answers, slower than [`LOG_WRITE_BUDGET`] a batch. @@ -175,10 +195,7 @@ impl Writer { None => asked.recv().map_err(|_| RecvTimeoutError::Disconnected), }; match ask { - Ok(Ask::Grew { urgent }) => { - self.owed_since.get_or_insert_with(Instant::now); - self.urgent |= urgent; - } + Ok(Ask::Grew) => self.took_growth(), Ok(Ask::Finish { by, answer }) => { self.finish(by); // A loop that gave up on the answer has said so. @@ -198,6 +215,13 @@ impl Writer { } } + /// The replay's growth since the last [`Ask::Grew`], taken whole: a round + /// after this one asks again. + fn took_growth(&mut self) { + self.urgent |= self.growth.swap(0, Ordering::AcqRel) & URGENT != 0; + self.owed_since.get_or_insert_with(Instant::now); + } + fn publish(&self) { let state = match &self.volume { None => State::ConsoleOnly, @@ -360,7 +384,7 @@ mod tests { #[test] fn a_batch_falls_due_at_its_size_its_age_or_an_urgent_line() { let since = Instant::now(); - let young = since + BATCH_AGE - Duration::from_nanos(1); + let young = (since + BATCH_AGE).checked_sub(Duration::from_nanos(1)).expect("a nanosecond after `since`"); assert!(!batch_due(BATCH_BYTES - 1, Some(since), false, young)); assert!(batch_due(BATCH_BYTES, Some(since), false, young), "a full batch waits for no age"); assert!(batch_due(1, Some(since), false, since + BATCH_AGE), "an old line waits no longer"); @@ -375,6 +399,7 @@ mod tests { volume, hub: Arc::new(Hub::start(1 << 20, None)), published: Arc::new(Published::new()), + growth: Arc::new(AtomicU8::new(0)), retry: None, degraded: false, owed_since: None, @@ -414,6 +439,25 @@ mod tests { assert_eq!(on_disk(&w), b"one\ntwo\npanicked\n"); } + /// **Any number of rounds is one ask in the channel** until the file's + /// thread takes it, with an urgent line among them kept: a file whose + /// `/log` call never returns has no growing queue behind it. + #[test] + fn rounds_the_file_has_not_taken_are_one_ask() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let (asks, asked) = mpsc::channel(); + let file = File { asks, growth: Arc::clone(&w.growth) }; + for round in 0..100 { + file.grew(round == 50); + } + assert_eq!(asked.try_iter().count(), 1); + w.took_growth(); + assert!(w.urgent && w.owed_since.is_some(), "the urgent round among them is kept"); + file.grew(false); + assert_eq!(asked.try_iter().count(), 1, "a round after the take asks again"); + } + /// **From the stop's flush until it is refused the file takes no call**, /// so the stop mark is the last write; and the mark is never asked of a /// file the flush did not reach. A refused stop's lines are written at once. From 9108a464a92803096f9accd02d090aff48ca5a41 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:17:05 +0200 Subject: [PATCH 5/7] logkeeper: the file thread owns the stop mark's record, and a growth already written starts no wait A round appended while a write caught up was written by it, and the ask that round had sent, taken after the write, started the batch's age with nothing owed: `next_due` then answered a time that passed while `due` stayed false, and the file thread woke at once, over and over, until the next line. A refused stop that held nothing did the same. The file is now owed a batch only where the replay holds bytes past it. The loop recorded whether the mark was written from the file thread's answer, which is no answer once the loop stops waiting, while the mark's create and sync could still land after it. A refused stop then left the mark on the volume, and a later death of the boot would read as clean and raise the anti-rollback floor. The file thread now records any mark it asked the volume for and takes it back at a refused stop; the loop keeps no copy and no longer says, when it stops waiting, that the boot will not count as clean. The mark is the volume's own: it goes in the directory of the file it vouches for. `Hub::append`'s header says what the replay is for now. The manual bounds a crash's loss for a volume that keeps pace, the cadence's header gives the file's latency to its batch, the write's text no longer shares the file's name, and the ABI's severity note leaves logkeeper's policy to logkeeper. Co-Authored-By: Claude Opus 5.5 --- docs/src/architecture/diagnostics.md | 3 +- sdk/abi/src/log.rs | 4 +- system/logkeeper/src/file.rs | 134 ++++++++++++++++++++------- system/logkeeper/src/main.rs | 35 ++++--- system/logkeeper/src/serve.rs | 5 +- system/logkeeper/src/store.rs | 37 ++++---- 6 files changed, 141 insertions(+), 77 deletions(-) diff --git a/docs/src/architecture/diagnostics.md b/docs/src/architecture/diagnostics.md index 246ef74f5a4..f058cddf46e 100644 --- a/docs/src/architecture/diagnostics.md +++ b/docs/src/architecture/diagnostics.md @@ -24,7 +24,8 @@ exports them. and keeps every boot's beginning. Severity is an ordered ladder. - The file is written in batches: at once for an error, an alert or the machine's stop, and otherwise within a second. A machine that dies without - stopping loses at most that second of its file; the console is not batched. + stopping loses at most that second of its file while the volume keeps pace; + the console is not batched. No file call holds up the console or a reader, and a stop waits on the file within a bound. - A slow sink drops and counts; it never blocks a producer and fails alone. diff --git a/sdk/abi/src/log.rs b/sdk/abi/src/log.rs index 83dd7073a29..d14fd807fa9 100644 --- a/sdk/abi/src/log.rs +++ b/sdk/abi/src/log.rs @@ -63,9 +63,7 @@ pub const RECORD_BYTES: usize = 1024; /// **Four because four have writers and readers.** The kernel writes `Info` /// (`log!`) and `Alert` (`alert!`); a program's stdout is `Info` and its stderr /// `Error`, and its own lines choose. Every rendered line names one above -/// `Info` by its [`word`](Self::word), a screen colours the line by it, and -/// `/system/bin/logkeeper` makes the volume durable at `Error` and above rather -/// than with its batch. +/// `Info` by its [`word`](Self::word), and a screen colours the line by it. #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] #[repr(u8)] pub enum Severity { diff --git a/system/logkeeper/src/file.rs b/system/logkeeper/src/file.rs index 4b31167d2c0..cce0c544709 100644 --- a/system/logkeeper/src/file.rs +++ b/system/logkeeper/src/file.rs @@ -30,7 +30,7 @@ use toyos_logstream::{Severity, LOG_OPENED}; use crate::inspect::{Published, State}; use crate::policy::{Retry, LOG_WRITE_BUDGET}; use crate::serve::Hub; -use crate::store::{self, Volume, DIR, MAX_LOG_BYTES, MAX_LOG_PARTS}; +use crate::store::{Volume, DIR, MAX_LOG_BYTES, MAX_LOG_PARTS}; /// How long a line below `Error` may wait for the file. Policy, against two /// costs: each write and sync costs the volume's server, its driver and this @@ -72,11 +72,11 @@ enum Ask { /// The stop: reach the replay's end and sync, stopping the catch-up at /// `by`, and answer. Finish { by: Instant, answer: SyncSender<()> }, - /// Write the stop mark, unless `by` has passed, and answer whether it is. - Mark { by: Instant, answer: SyncSender }, - /// The stop was refused: take back the mark where it was written, and - /// write what the replay was given since at once. - Resume { marked: bool }, + /// Write the stop mark, unless `by` has passed, and answer. + Mark { by: Instant, answer: SyncSender<()> }, + /// The stop was refused: take back any mark it asked for, and write what + /// the replay was given since at once. + Resume, } /// The loop's handle on the file's thread. @@ -109,6 +109,7 @@ impl File { urgent: false, finished: false, stopped: false, + marked: false, }; writer.run(asked); }) @@ -133,17 +134,17 @@ impl File { answered.recv_timeout(until.saturating_duration_since(Instant::now())).is_ok() } - /// The stop mark, written by `by`: whether it is, or `None` where the - /// file did not answer by then. - pub fn mark(&self, by: Instant) -> Option { + /// The stop mark, written by `by`: whether the file answered by then. + /// Whether the mark is on the volume is the file thread's record. + pub fn mark(&self, by: Instant) -> bool { let (answer, answered) = mpsc::sync_channel(1); self.ask(Ask::Mark { by, answer }); - answered.recv_timeout(by.saturating_duration_since(Instant::now())).ok() + answered.recv_timeout(by.saturating_duration_since(Instant::now())).is_ok() } - /// The stop was refused, after the mark was written where `marked`. - pub fn resume(&self, marked: bool) { - self.ask(Ask::Resume { marked }); + /// The stop was refused. + pub fn resume(&self) { + self.ask(Ask::Resume); } fn ask(&self, ask: Ask) { @@ -184,6 +185,9 @@ struct Writer { finished: bool, /// From the stop's flush until it is refused: no call on `/log` but the mark. stopped: bool, + /// The stop mark may be on the volume: this stop asked for it, whatever + /// the volume answered, and whether or not the loop still waited. + marked: bool, } impl Writer { @@ -202,9 +206,10 @@ impl Writer { let _ = answer.send(()); } Ok(Ask::Mark { by, answer }) => { - let _ = answer.send(self.mark(by)); + self.mark(by); + let _ = answer.send(()); } - Ok(Ask::Resume { marked }) => self.resume(marked), + Ok(Ask::Resume) => self.resume(), Err(RecvTimeoutError::Timeout) => {} // The loop never lets go of its handle while the program runs. Err(RecvTimeoutError::Disconnected) => return, @@ -218,7 +223,19 @@ impl Writer { /// The replay's growth since the last [`Ask::Grew`], taken whole: a round /// after this one asks again. fn took_growth(&mut self) { - self.urgent |= self.growth.swap(0, Ordering::AcqRel) & URGENT != 0; + let urgent = self.growth.swap(0, Ordering::AcqRel) & URGENT != 0; + self.owe(urgent); + } + + /// The file is owed what the replay holds past it, at once where + /// `urgent`: the batch's age counts from now. A growth a write has taken + /// already, with the round before it, is owed nothing, so no wait starts + /// that no batch would end. + fn owe(&mut self, urgent: bool) { + if self.volume.as_ref().is_none_or(|v| self.hub.end() == v.through()) { + return; + } + self.urgent |= urgent; self.owed_since.get_or_insert_with(Instant::now); } @@ -348,30 +365,34 @@ impl Writer { /// `by`. Withheld, and no call made, unless the flush reached the file: /// `/log`'s server may never have answered, and a part the volume refused /// is not this boot's log, so the mark would call a stop clean whose log - /// is not on the volume. - fn mark(&mut self, by: Instant) -> bool { - if !self.finished { + /// is not on the volume. Once asked for, the mark is taken to be there, + /// whatever the volume answered, until a refused stop takes it back. + fn mark(&mut self, by: Instant) { + let Some(v) = self.volume.as_ref().filter(|_| self.finished) else { toyos::warn!("logkeeper: {DIR} does not hold this boot's log, so the stop is not marked - the next boot will not count this one as clean"); - return false; - } + return; + }; if Instant::now() >= by { toyos::error!("logkeeper: the stop mark was asked of {DIR} after its time - the next boot will not count this one as clean"); - return false; + return; + } + self.marked = true; + if let Err(e) = v.mark_stopped() { + toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean"); } - store::mark_stopped() - .inspect_err(|e| { - toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean") - }) - .is_ok() } - fn resume(&mut self, marked: bool) { - if marked && let Err(e) = store::unmark_stopped() { - toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); + /// The stop was refused: any mark it asked for is taken back, and what + /// the replay was given since is written at once. + fn resume(&mut self) { + if std::mem::take(&mut self.marked) { + let v = self.volume.as_ref().expect("a stop's volume stays until the stop is refused"); + if let Err(e) = v.unmark_stopped() { + toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); + } } (self.stopped, self.finished) = (false, false); - self.owed_since.get_or_insert_with(Instant::now); - self.urgent = true; + self.owe(true); } } @@ -406,6 +427,7 @@ mod tests { urgent: false, finished: false, stopped: false, + marked: false, } } @@ -448,6 +470,7 @@ mod tests { let mut w = writer(tmp.path().to_str().unwrap()); let (asks, asked) = mpsc::channel(); let file = File { asks, growth: Arc::clone(&w.growth) }; + w.hub.append(b"line\n", None); for round in 0..100 { file.grew(round == 50); } @@ -466,7 +489,8 @@ mod tests { let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); let mut w = writer(tmp.path().to_str().unwrap()); let t = Instant::now(); - assert!(!w.mark(t + BATCH_AGE), "no flush, no mark"); + w.mark(t + BATCH_AGE); + assert!(!w.marked, "no flush, no mark"); grew(&mut w, b"before\n", false, t); w.finish(Instant::now() + BATCH_AGE); assert!(w.finished); @@ -475,11 +499,53 @@ mod tests { w.volume.as_mut().unwrap().lost(); w.retry = Some(Retry::after(t)); assert!(!w.due(t + Duration::from_secs(3600)) && w.next_due().is_none(), "a stopped file is written nothing"); - w.resume(false); + w.resume(); let retry = w.retry.unwrap().at(); assert_eq!(w.next_due(), Some(retry), "a refused part is asked again at its retry"); assert!(w.due(retry)); w.write(); assert_eq!(on_disk(&w), b"before\nafter\n"); } + + /// **A growth a write has taken already starts no wait**: a round + /// appended while a write catches up is written by it, and the ask that + /// round sent, taken after, owes nothing, so the thread parks rather than + /// waking at once until the next line. A refused stop that held nothing + /// owes nothing either. + #[test] + fn a_growth_already_written_starts_no_wait() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let (asks, _asked) = mpsc::channel(); + let file = File { asks, growth: Arc::clone(&w.growth) }; + w.hub.append(b"one\n", None); + file.grew(false); + w.took_growth(); + w.hub.append(b"two\n", None); + file.grew(true); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\n"); + w.took_growth(); + assert!(w.next_due().is_none() && !w.urgent, "a file with everything waits on nothing"); + w.finish(Instant::now() + BATCH_AGE); + w.resume(); + assert!(w.next_due().is_none(), "a refused stop that held nothing owes nothing"); + } + + /// **A refused stop takes back the mark it asked for, whatever the loop + /// heard of it**: the file's thread keeps the record, so a mark that lands + /// after the loop stopped waiting is not left for a later death to read as + /// a clean stop. + #[test] + fn a_refused_stop_takes_back_the_mark_it_asked_for() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let mark = tmp.join(toyos_logstream::STOPPED); + grew(&mut w, b"before\n", false, Instant::now()); + w.finish(Instant::now() + BATCH_AGE); + w.mark(Instant::now() + BATCH_AGE); + assert!(mark.exists(), "the mark goes over the flushed file"); + w.resume(); + assert!(!mark.exists(), "a refused stop leaves no mark"); + } } diff --git a/system/logkeeper/src/main.rs b/system/logkeeper/src/main.rs index 6d495a1ab44..b0c6769c2bd 100644 --- a/system/logkeeper/src/main.rs +++ b/system/logkeeper/src/main.rs @@ -131,8 +131,8 @@ const MAX_ORIGINS: usize = Poller::MAX_HANDLES as usize - ORIGIN_BASE as usize; /// How soon the rings are read again after a round that read any, and the /// longest the cadence backs off to while none has anything: the latency of a -/// program's line to the file and the console, against wakes on an idle -/// machine. +/// program's line to the replay, and so to the console and the readers, +/// against wakes on an idle machine. The file's is its batch's. const POLL_QUICK: Duration = Duration::from_millis(10); const POLL_SLOW: Duration = Duration::from_millis(250); @@ -252,12 +252,10 @@ impl Handover { } } -/// The file's text held back after a flush, the lines past the bound, and -/// whether the flush wrote the stop mark. +/// The file's text held back after a flush, and the lines past the bound. struct Stopping { text: String, unwritten: u64, - marked: bool, } /// How long the stop's flush may spend giving the file what it is owed: half @@ -540,7 +538,7 @@ impl Log { return; } lines.sort_by_key(|line| line.at_ns); - let mut file = String::new(); + let mut rendered = String::new(); // What the UART is owed of these lines, while the kernel's own on it // are yet to be read past. let mut wire = String::new(); @@ -554,7 +552,7 @@ impl Log { if !self.handover.as_ref().is_some_and(|handover| on_the_uart(record, &handover.spoken)) { wire.push_str(&text); } - file.push_str(&text); + rendered.push_str(&text); } Kind::Program { tag, owner, said } => { urgent |= file::urgent(said.severity); @@ -571,7 +569,7 @@ impl Log { text: &said.text, }; let line = format!("{line}\n"); - file.push_str(&line); + rendered.push_str(&line); wire.push_str(&line); if self.stopping.is_some() { programs.push_str(&line); @@ -583,14 +581,14 @@ impl Log { // The kernel's records from the flush on are the stop's to put on // the UART; a program's reach it only through its carrier. self.hub.append_stopping(programs.as_bytes()); - if held.text.len() + file.len() > STOPPING_BYTES { + if held.text.len() + rendered.len() > STOPPING_BYTES { held.unwritten += lines.len() as u64; } else { - held.text.push_str(&file); + held.text.push_str(&rendered); } return; } - self.hub.append(file.as_bytes(), Some(through)); + self.hub.append(rendered.as_bytes(), Some(through)); self.file.grew(urgent); if let Some(handover) = &mut self.handover { handover.owed.push_str(&wire); @@ -630,7 +628,7 @@ impl Log { if let Some(carriers) = &self.carriers { carriers.stop(self.hub.end()); } - self.stopping = Some(Stopping { text: String::new(), unwritten: 0, marked: false }); + self.stopping = Some(Stopping { text: String::new(), unwritten: 0 }); if let Err(e) = self.from_supervisor.signal(FLUSHED) { panic!("logkeeper: the supervisor could not be told the flush is done: {e:?}"); } @@ -640,16 +638,15 @@ impl Log { /// drivers: the file writes the stop mark over a durable log, within /// [`STOP_FILE`], and the supervisor is told. fn mark(&mut self) { - let Some(stopping) = self.stopping.as_mut() else { + if self.stopping.is_none() { panic!("logkeeper: the supervisor asked for the stop mark, and no stop was flushed for"); - }; - stopping.marked = self.file.mark(Instant::now() + STOP_FILE).unwrap_or_else(|| { + } + if !self.file.mark(Instant::now() + STOP_FILE) { toyos::error!( - "logkeeper: {DIR} had not answered the stop mark in {} ms - the next boot will not count this one as clean", + "logkeeper: {DIR} had not answered the stop mark in {} ms - the stop goes on without knowing whether it is written", STOP_FILE.as_millis() ); - false - }); + } if let Err(e) = self.from_supervisor.signal(MARKED) { panic!("logkeeper: the supervisor could not be told the stop is marked: {e:?}"); } @@ -670,7 +667,7 @@ impl Log { ); } self.hub.resume(held.text.as_bytes()); - self.file.resume(held.marked); + self.file.resume(); if let Some(carriers) = &self.carriers { carriers.resume(); } diff --git a/system/logkeeper/src/serve.rs b/system/logkeeper/src/serve.rs index dc56f277169..3ebbca107f9 100644 --- a/system/logkeeper/src/serve.rs +++ b/system/logkeeper/src/serve.rs @@ -112,8 +112,9 @@ impl Hub { admit(&self.shared, format!("reader {}", conn.as_handle().raw().0), pipe); } - /// The lines the file has just taken, for every reader, and the kernel's - /// read positions they are written through where they are a round's. + /// Lines for every reader and the console's carriers, and for the file, + /// which takes them from here in its next batch; with the kernel's read + /// positions they are written through where they are a round's. pub fn append(&self, lines: &[u8], through: Option) { let mut held = self.lock(); held.replay.append(lines); diff --git a/system/logkeeper/src/store.rs b/system/logkeeper/src/store.rs index 5a3417f09cb..2a0687c9ac2 100644 --- a/system/logkeeper/src/store.rs +++ b/system/logkeeper/src/store.rs @@ -54,24 +54,6 @@ pub const MAX_LOG_PARTS: u32 = 9999; /// what `/system/bin/console` reads off USB before it paints anything. pub const MAX_LOG_BYTES: u64 = 1024 * 1024; -/// Say on the volume that this boot stopped cleanly -/// ([`toyos_logstream::STOPPED`]), durably. -pub fn mark_stopped() -> std::io::Result<()> { - File::create(stopped_path())?.sync_all() -} - -/// Take back [`mark_stopped`]: the stop it was written for was refused. -pub fn unmark_stopped() -> std::io::Result<()> { - match fs::remove_file(stopped_path()) { - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), - done => done, - } -} - -fn stopped_path() -> String { - format!("{DIR}/{}", toyos_logstream::STOPPED) -} - /// Where one file of this boot's sequence is: `toyos_wallclock::Part` is its /// name. pub fn path(dir: &str, stem: &str, part: u32) -> String { @@ -264,6 +246,25 @@ impl Volume { path(&self.dir, &self.stem, self.part) } + /// Say on this volume that the boot stopped cleanly + /// ([`toyos_logstream::STOPPED`]), durably. + pub fn mark_stopped(&self) -> std::io::Result<()> { + File::create(self.stopped_path())?.sync_all() + } + + /// Take back [`Self::mark_stopped`]: the stop it was written for was + /// refused. A mark that is not there is taken back already. + pub fn unmark_stopped(&self) -> std::io::Result<()> { + match fs::remove_file(self.stopped_path()) { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + done => done, + } + } + + fn stopped_path(&self) -> String { + format!("{}/{}", self.dir, toyos_logstream::STOPPED) + } + /// Whether the part is open: false once the volume refused it, until /// [`Self::catch_up`] opens it again. pub fn is_open(&self) -> bool { From 9356faaadcf2ec348747ad4f3367b08913f29f58 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:39:27 +0200 Subject: [PATCH 6/7] logkeeper: a catch-up that spends its budget on the replay's last line is caught up and synced `Volume::catch_up` returned at its budget before asking the replay for more, so a budget the replay's last line used up left the file unsynced and reported not caught up with nothing owed. The file thread then kept a batch's age that no write would end and woke at once, over and over, until the next line. The replay is now asked first, and a file with every byte is synced whatever the budget left. Co-Authored-By: Claude Opus 5.5 --- system/logkeeper/src/store.rs | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/system/logkeeper/src/store.rs b/system/logkeeper/src/store.rs index 2a0687c9ac2..f62017cfd35 100644 --- a/system/logkeeper/src/store.rs +++ b/system/logkeeper/src/store.rs @@ -330,7 +330,8 @@ impl Volume { /// `budget` bytes of the boot this call, through part-sized writes and /// their continuations, each part synced as it fills; `note(lost)` is the /// line in place of a gap. Whether the file has every byte the replay - /// has; an `Err` is the volume refusing again. + /// has, which a file that does is synced to, whatever the budget left; + /// an `Err` is the volume refusing again. pub fn catch_up( &mut self, mut owed: impl FnMut(u64, usize) -> Owed, @@ -349,11 +350,13 @@ impl Volume { return Ok(false); } } - if written >= budget { - return Ok(false); - } - let room = (self.rotate_at - self.size).min(budget - written); + let room = (self.rotate_at - self.size).min(budget.saturating_sub(written)); match owed(self.through, room as usize) { + Owed::Nothing => { + self.sync()?; + return Ok(true); + } + _ if written >= budget => return Ok(false), Owed::Text(text) => { written += text.len() as u64; self.write(&text)?; @@ -362,10 +365,6 @@ impl Volume { self.put(¬e(lost))?; self.through = at; } - Owed::Nothing => { - self.sync()?; - return Ok(true); - } } } } @@ -543,6 +542,23 @@ mod tests { } } + /// **A catch-up whose budget runs out on the replay's last line is caught + /// up and synced**: nothing is left owed, so no later batch would come + /// for that tail, and the file thread would have no due write to wait for. + #[test] + fn a_budget_spent_on_the_replays_last_line_is_caught_up_and_synced() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-store"); + let dir = tmp.path().to_str().unwrap(); + let hub = Hub::start(1 << 16, None); + let mut v = Volume::open(dir, Some(STEM.into()), 1 << 16, |_| {}).unwrap(); + for n in 0..4 { + hub.append(&line(n), None); + } + let caught = v.catch_up(|at, max| hub.owed(at, max), note, hub.end(), |_| {}).unwrap(); + assert!(caught && v.through() == hub.end(), "the file has every byte"); + assert_eq!((v.durable, v.durable_through), (v.size, v.through), "and they are synced"); + } + /// A stop whose reopen is refused after an unsynced write: the file is /// what its last sync left, and the bytes it lacks are counted from /// there, the unsynced write's included. From 79b15d36a428d1bd997d78c0299dffcfaa281184 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 21:59:01 +0200 Subject: [PATCH 7/7] issues: what the batched log file owes the T14, now that the benches are down The T14's `machine_reboot` boot and its `spawn_cost` A/B were staged and never ran. They are owed on metal against main, in the log subsystem's record, each with what main must show; the AMD bench's A/B and `machine_reboot` passed. Co-Authored-By: Claude Opus 5.5 --- issues/redesign-the-log-subsystem.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/issues/redesign-the-log-subsystem.md b/issues/redesign-the-log-subsystem.md index e480730e633..cb3b00acee5 100644 --- a/issues/redesign-the-log-subsystem.md +++ b/issues/redesign-the-log-subsystem.md @@ -109,3 +109,19 @@ sources, and the review's target was one `gates` directory under `kernel/src/` so that what test machinery exists is auditable in one listing. `input_merge_test` was the tell that pure logic was trapped in the kernel: the merge is `toyos-input`'s now, host-tested with synthetic multi-source streams. + +**The file written in batches** (#1232) rests on host tests and guest rows. +The benches went down before its T14 half ran; on the AMD bench its +`spawn_cost` A/B (one boot per arm) and `machine_reboot` passed. + +- **Owed on metal:** `machine_reboot` on the T14, at main. Main must pass the + boot verdict with the stop mark and the supervisor's stop line in the file, + `machine_reboot`'s reset-register line, and `test_rs_ps2server_hold` exit 0 + (a served-log reader is handed its line through the reader's wake), with no + "had not answered the stop's flush", "stop mark was asked … after its time", + "answers but slowly" or "refusing reader" line. +- **Owed on metal:** `spawn_cost`'s A/B on the T14, main before #1232 against + main after it, back to back. Main after must show "the machine busy … a + spawn" lower than before, by about one write and sync per logkeeper round the + loop spans (on the AMD bench, 430 µs against 360 µs a spawn), min and median + no worse than the two runs' spread, and both boots passing with the stop mark.