Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,33 @@ sequential reads and 1 MiB writes ahead, and the 4 KiB and 8-byte reads
unresolved against Linux's HPET-widened spread. `seek`, which the client
answers itself, is 248 ns against Linux's 116 ns on the T14.

#1246 takes the process table and the process's data lock off every
handle-taking call: a served file's read (`SYS_PAGED_READ`) looks its holder
up under the handle table's own lock, read off the calling thread, where
pipeperf had measured the lookup at 60 ns on the T14 and 33 ns on the AMD
bench. Its speed-up is not yet measured on metal.

- **Owed on metal:** the T14's whole `testcases`, `testcases-watchdog` and
`shared-debug` boots of main after #1246. Main must pass every member,
among them `handle_lifetime`, `abuse_handle_table`, `abuse_thread_table`,
`messages_are_whole`, `stream_handles`, `spawn_lands_claimed`,
`spawn_child_ends_first`, `spawn_returns_the_census`, `endowment_denied`,
`a_served_file_reads_through_its_paged_object`,
`a_pager_s_end_answers_its_waiting_holders_gone`, `std_threads`,
`inbox_cancel_wakes` and `port_badge`.
- **Owed on metal:** an A/B of main before and after #1246, on the T14 and
the AMD bench: `osbench files`, `ipc`, `time` and `sched`, four boots a
side in rotated order, each boot's p50, compared by an exact two-sided
Mann-Whitney test; and Linux's `osbench files` and `ipc` on each bench.
The claimed rows are `fs.mem.pread_8`, `fs.mem.pread_4k`,
`fs.mem.direct.pread_4k`, `fs.mem.read_4k`, `fs.mem.read_4k_at`,
`fs.mem.direct.read_4k_at`, `ipc.reqrep.8` and `ipc.reqrep_handle.8`:
main after reads each lower or not separated, with how far each moved set
against the 60 ns (T14) and 33 ns (AMD) lookup, and no other row 5% or
more worse unless the code #1246 changed explains it. A missing gain does
not reject the change; it decides whether a lock-free handle read is worth
building.

**Exit**: every `fs.mem` row of the second table within Linux's on both benches in a labelled
current-state comparison, or each remaining difference measured to its parts
and justified.
Expand Down
2 changes: 1 addition & 1 deletion kernel/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The module header at the site owns its subsystem — read it before changing a m
- **The idle loop may not take a global lock unconditionally** — the crash report reads the process table through a `try_lock` it must never block on. `sched/reap_gate.rs` is the pattern: a relaxed-load gate in front of the lock.
- **Every x86 Ring 0 entry clears the direction flag, and every interrupt or trap gate `AC`** — no hardware gate does either; `memmove` sets `DF` across `rep` operations with interrupts on, and Ring 3 sets `AC`, under which SMAP lets Ring 0 reach user pages. `arch::entry::ring3_naked_asm` is where they live; `SYSCALL` clears `AC` through `IA32_FMASK`.
- **`BackendGuard` masks interrupts for its whole life**, so anything written under it is an interrupt latency; a new holder bounds itself as the console drain does.
- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_process_data` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern.
- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_handles` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern.
- **A decision the process table makes lives in `kernel/proclife/`, never in `process.rs`** — its defects are interleavings and its host tests are the only machine that can enumerate one.
- **A task holds at most one watch registration** — a standing registration across a loop must not call anything that registers again. A double registration panics only at attempt ≥ 2, so the contention depth is the coverage.
- **`ops::close` cancels a poll only for a source its object really ends** — `Watch::cancel_polls` answers every ring's poll on that watch; `ops::close_ends_polls` is where a new object kind answers.
Expand Down
14 changes: 7 additions & 7 deletions kernel/src/inbox/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -531,9 +531,9 @@ fn process_watch(inbox: &Arc<Inbox>, submission: &Submission) {
}

/// The object `handle` names in this process's table, not the thread's: a ring is process-wide.
/// Cloned out, so what follows holds no process lock.
/// Cloned out, so what follows holds no handle-table lock.
fn resolve(handle: RawHandle) -> Result<KObjectRef, HandleError> {
process::with_process_data(|data| data.handles.get_ref(handle, Rights::WAIT).cloned())
process::with_handles(|handles| handles.get_ref(handle, Rights::WAIT).cloned())
}

/// Keep `poll` as its handle's one poll, or in the place of `looked_at`, the
Expand Down Expand Up @@ -663,13 +663,13 @@ impl Submitter<Arc<Inbox>> for Arc<Inbox> {
fn process_accept(inbox: &Arc<Inbox>, submission: &Submission) {
let user_data = submission.token;

let acceptor = process::with_process_data(|data| {
data.handles.get::<crate::object::port::Acceptor>(submission.handle, Rights::READ)
let acceptor = process::with_handles(|handles| {
handles.get::<crate::object::port::Acceptor>(submission.handle, Rights::READ)
});

let acceptor = match acceptor {
Ok(a) => a,
// Nothing held: `with_process_data` has given the guard up.
// Nothing held: `with_handles` has given the guard up.
Err(e) => {
let refusal = e.refuse_as_error();
polls::complete(inbox, user_data, -(refusal as i32));
Expand All @@ -687,8 +687,8 @@ fn process_accept(inbox: &Arc<Inbox>, submission: &Submission) {
};
match acceptor.pop() {
Some(conn) => {
let installed = process::with_process_data(|data| {
ops::install(&mut data.handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn)))
let installed = process::with_handles(|handles| {
ops::install(handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn)))
});
match installed {
Ok(h) => polls::complete(inbox, user_data, h.0 as i32),
Expand Down
4 changes: 2 additions & 2 deletions kernel/src/inventory.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ fn claims() -> Vec<Claim> {
};
let mut out = Vec::new();
for (pid, name, data) in processes {
let data = data.lock();
for (_, entry) in data.handles.iter() {
let handles = Arc::clone(&data.lock().handles);
for (_, entry) in handles.lock().iter() {
let KObjectRef::Device(claim) = entry.object() else { continue };
let on = if claim.class() == toyos_abi::syscall::DeviceType::PciFunction {
// A function the last handle has let go names nothing.
Expand Down
7 changes: 4 additions & 3 deletions kernel/src/loader/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ pub fn spawn<C: Commit>(
parent: Parent,
) -> Result<(Pid, C::Held), crate::object::Refusal> {
let job = Arc::clone(commit.job());
let pager = if commit.pager() { crate::process::PagerRole::May } else { crate::process::PagerRole::None };
let pager = commit.pager();
// Before anything is built; dropped on every way out below but the insert.
let admission = Admission::ask(parent)?;
let charge = ProcessData::charge(&job, path).map_err(SyscallError::from)?;
Expand Down Expand Up @@ -189,15 +189,15 @@ pub fn spawn<C: Commit>(
// The point of no return: every failure above answers the caller with its
// table untouched.
let (handles, endowments, held) = commit.commit(start::Own::new(&object, &notice))?;
let handles = crate::process::Handles::new(handles, pager);
let proc_data = Arc::new(Lock::new(ProcessData {
handles,
handles: Arc::clone(&handles),
space: Arc::clone(&child_pt),
spawn_ns: crate::clock::nanos_since_boot(),
accounting: ProcessAccounting::default(),
endowments,
job: job.clone(),
_charge: charge,
pager,
}));

let thread_data = Arc::new(Lock::new(ThreadData {
Expand Down Expand Up @@ -231,6 +231,7 @@ pub fn spawn<C: Commit>(
thread_pointer: 0,
job,
image,
handles,
// The spawning thread's, as a new thread's is.
mask: crate::sched::driver::current_mask(),
near: None,
Expand Down
31 changes: 15 additions & 16 deletions kernel/src/loader/start.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use crate::object::notice::NoticeObject;
use crate::object::process::ProcessObject;
use crate::object::{ops, HandleEntry, HandleTable, KObjectRef, Refusal};
use crate::process::{
process_data, Endowments, KERNEL_STACK_SIZE,
handles, process_data, with_handles, Endowments, KERNEL_STACK_SIZE,
};
use crate::user_ptr::UserBytes;
use toyos_abi::handle::{RawHandle, Rights};
Expand Down Expand Up @@ -114,7 +114,7 @@ pub trait Commit {
/// The job the child runs in, which everything built for it is counted to.
fn job(&self) -> &Arc<Job>;

/// Whether the child starts as a pager (`process::PagerRole`).
/// Whether the child starts as a pager (`process::Handles::is_pager`).
fn pager(&self) -> bool;

/// The child's table around `own`, what the kernel puts beside its
Expand All @@ -138,8 +138,9 @@ impl Commit for PendingHandles {
/// last answer. The handle under [`JOB_LABEL`] still names [`job`](Self::job), or the spawn is refused.
fn commit(self, own: Own) -> Result<(HandleTable, Endowments, RawHandle), Refusal> {
let Self { mut table, endow, mut labels, job, .. } = self;
let data_arc = process_data();
let mut data = data_arc.lock();
let own_job = process_data().lock().job.clone();
let caller = handles();
let mut handles = caller.lock();

let mut moving: Vec<(EndowEntry, RawHandle, Rights)> = Vec::with_capacity(endow.len());
let mut endowed_job: Option<Arc<Job>> = None;
Expand All @@ -156,10 +157,10 @@ impl Commit for PendingHandles {
if endowed_job.is_some() {
return Err(SyscallError::InvalidArgument.into());
}
endowed_job = Some(data.handles.get_sent::<JobObject>(handle, Rights::WRITE)?.job().clone());
endowed_job = Some(handles.get_sent::<JobObject>(handle, Rights::WRITE)?.job().clone());
}
// Checked before any removal, so a missing `TRANSFER` refuses the spawn instead of leaving a hole.
let rights = data.handles.rights_to_move(handle, want)?;
let rights = handles.rights_to_move(handle, want)?;
// A duplicate is refused here: unremoved handles let it pass twice, then the second removal's
// `expect` below panics.
if moving.iter().any(|(_, seen, _)| *seen == handle) {
Expand All @@ -169,20 +170,19 @@ impl Commit for PendingHandles {
}
// Before any removal: the table changed under the spawn if the job
// everything was built for is no longer the one it names.
if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&data.job)) {
if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&own_job)) {
return Err(SyscallError::InvalidArgument.into());
}
// Checked before any removal, so a failed install can't strand a handle out of a table that never spawned.
if !table.make_room(moving.len() + Own::LABELS.len()) || !data.handles.make_room(1) {
if !table.make_room(moving.len() + Own::LABELS.len()) || !handles.make_room(1) {
return Err(SyscallError::ResourceExhausted.into());
}
let mut entries = Vec::with_capacity(moving.len());
// What each device claim moving into the child handed out, taken off
// it under the caller's lock and revoked once that lock is let go.
let mut handed = Vec::new();
for (mut entry, parent_handle, rights) in moving {
let moved = data
.handles
let moved = handles
.remove(parent_handle)
.expect("an endowed handle verified under this lock stopped resolving")
.arriving_with(rights);
Expand All @@ -195,9 +195,9 @@ impl Commit for PendingHandles {
entries.push(entry);
}
// Minted while `own` is held, and `own` goes into a table no thread reaches until the child lands: another thread of the caller closing this handle never closes the object's last.
let held = ops::install(&mut data.handles, own.process.object().clone())
let held = ops::install(&mut handles, own.process.object().clone())
.expect("a caller's table with verified room refused its child");
drop(data);
drop(handles);
for handed in handed {
handed.revoke();
}
Expand Down Expand Up @@ -268,12 +268,11 @@ fn label_of(labels: &[u8], off: u32, len: u32) -> Option<&[u8]> {
/// The job a child runs in: the one endowed under [`JOB_LABEL`], or else the
/// caller's. Only looked up: [`PendingHandles::commit`] refuses the rest.
fn child_job(endow: &[SpawnEndow], labels: &[u8]) -> Result<Arc<Job>, Refusal> {
let data_arc = process_data();
let data = data_arc.lock();
for entry in endow {
if label_of(labels, entry.label_off, entry.label_len) == Some(JOB_LABEL.as_bytes()) {
return Ok(data.handles.get_sent::<JobObject>(entry.moved.handle, Rights::WRITE)?.job().clone());
let named = with_handles(|handles| handles.get_sent::<JobObject>(entry.moved.handle, Rights::WRITE))?;
return Ok(named.job().clone());
}
}
Ok(data.job.clone())
Ok(process_data().lock().job.clone())
}
4 changes: 2 additions & 2 deletions kernel/src/message.rs
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ impl QueueRef {
}

/// Queue `src`, already copied in, and `batch`, its handles, whole in
/// this hold, called under the sender's process lock as its handles move;
/// this hold, called under the sender's handle-table lock as its handles move;
/// a refusal hands the batch back and queues nothing. No copy waits on it.
pub fn send_now(&self, src: &[u8], mut batch: Vec<HandleEntry>) -> Result<(), (Vec<HandleEntry>, Unsent)> {
let mut state = self.0.state.lock();
Expand All @@ -280,7 +280,7 @@ impl QueueRef {

/// Take the front message whole into `into` and install its handles in
/// `table`, in this hold, where a reader holding `buf` bytes and `cap`
/// handles can take it, called under the receiver's process lock. A table
/// handles can take it, called under the receiver's handle-table lock. A table
/// that must grow first answers [`Unclaimed::Room`], taking nothing.
pub fn recv_now(&self, table: &mut HandleTable, buf: usize, cap: usize, into: &mut MessageBounce) -> Result<Received, Unclaimed> {
let mut state = self.0.state.lock();
Expand Down
6 changes: 3 additions & 3 deletions kernel/src/object/handle.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! One process's handle table.
//!
//! Lives behind `ProcessData`'s own lock; every accessor returns an owned
//! value, so no borrow into the table outlives the guard.
//! Lives behind its own lock (`process::Handles`); every accessor returns an
//! owned value, so no borrow into the table outlives the guard.
//!
//! Not a file-descriptor table — `fd` names only `sdk/libc`'s POSIX
//! surface (owner ruling) — and a slot at its last generation retires
Expand All @@ -22,7 +22,7 @@ use crate::mm::charge::{self, Charge};

use super::{KObjectRef, KObjectVariant};

/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the process's own lock, where [`HandleError::refuse`] must not be called.
/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the table's own lock, where [`HandleError::refuse`] must not be called.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct TableFull;

Expand Down
4 changes: 2 additions & 2 deletions kernel/src/object/notice.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ use super::{KObjectVariant, ObjectCore};

pub struct NoticeObject {
pub(super) core: ObjectCore,
/// A leaf: a quit takes it under `PROCESS_TABLE`, a read under the
/// reader's `ProcessData`, and nothing is taken under it.
/// A leaf: a quit takes it under `PROCESS_TABLE`, and nothing is taken
/// under it.
state: Lock<Notice>,
/// Posted when a reason is put in it.
watch: Arc<Watch>,
Expand Down
4 changes: 2 additions & 2 deletions kernel/src/pipe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -330,7 +330,7 @@ impl PipeRef {
}

/// Write `src`, already copied in, whole in one hold of the pipe's lock,
/// for a caller holding its process's lock: `None` where that cannot answer
/// for a caller holding its handle-table lock: `None` where that cannot answer
/// it without a wait or frames taken, which [`Self::write`] does.
pub fn put_now(&self, src: &[u8]) -> Option<PipeWrite> {
let mut pipe = self.0.state.lock();
Expand All @@ -347,7 +347,7 @@ impl PipeRef {
}

/// Take at most `max` bytes, up to [`BOUNCE`], into `into` whole in one
/// hold of the pipe's lock, for a caller holding its process's lock:
/// hold of the pipe's lock, for a caller holding its handle-table lock:
/// `None` where the pipe holds nothing or more than a bounce would take,
/// or another read holds its claim, which [`Self::read`] answers.
pub fn take_now(&self, max: usize, into: &mut PipeBounce) -> Option<Taken> {
Expand Down
Loading