From 6d9c8671cae1723636f705f8c4215c49ebbcc0ec Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 17:51:13 +0200 Subject: [PATCH 1/2] A thread carries its process's handles, and a lookup takes no lock but the table's Every handle-taking syscall went through `process::with_process_data`: the machine-wide `PROCESS_TABLE` lock, a walk of the process table, an `Arc` clone of the process's data, and that data's lock, which also guards its accounting, space, endowments and job. Every thread of every process serialized on the one table lock for each lookup (pipeperf: 60 ns on the T14 and 33 ns on the AMD bench per `pread` before the read itself). The handle table now sits behind a lock of its own in `process::Handles`, which each thread's scheduler record carries beside its address space and image, set at spawn and at thread creation. `process::with_handles` reads it off the running task and takes only the table's lock. `ProcessData` keeps an `Arc` to it for readers outside the process: teardown's `close_all`, which still empties it in place when the last thread leaves, and the device-claim inventory. The pager role moves with it. A served file's read (`SYS_PAGED_READ`) refuses a pager's process, so the role has to be readable in the same place as the holder's handle, or the read would take the process table again: whether the process was started as a pager is a plain field, and whether it made its pager object an atomic claim, which replaces `PagerRole`. `sys_job_own` and a spawn's commit read the job off `ProcessData` before they take the table's lock rather than under one lock with it; the job never changes. Comments that named the process's lock as the one a lookup holds now name the table's, and `kernel/CLAUDE.md`'s park rule names `with_handles`. Co-Authored-By: Claude Opus 5.5 --- kernel/CLAUDE.md | 2 +- kernel/src/inbox/mod.rs | 14 ++-- kernel/src/inventory.rs | 4 +- kernel/src/loader/mod.rs | 7 +- kernel/src/loader/start.rs | 31 ++++----- kernel/src/message.rs | 4 +- kernel/src/object/handle.rs | 6 +- kernel/src/object/notice.rs | 4 +- kernel/src/pipe.rs | 4 +- kernel/src/process.rs | 91 +++++++++++++++++------- kernel/src/sched/driver.rs | 8 +++ kernel/src/sched/payload.rs | 4 +- kernel/src/scheduler.rs | 2 +- kernel/src/syscall/device.rs | 28 ++++---- kernel/src/syscall/dispatch.rs | 6 +- kernel/src/syscall/event.rs | 6 +- kernel/src/syscall/handles.rs | 22 +++--- kernel/src/syscall/io.rs | 24 +++---- kernel/src/syscall/ipc.rs | 124 ++++++++++++++++----------------- kernel/src/syscall/machine.rs | 8 +-- kernel/src/syscall/paged.rs | 57 ++++++++------- kernel/src/syscall/proc.rs | 39 ++++++----- 22 files changed, 273 insertions(+), 222 deletions(-) diff --git a/kernel/CLAUDE.md b/kernel/CLAUDE.md index c4d084ea941..d18d11af32a 100644 --- a/kernel/CLAUDE.md +++ b/kernel/CLAUDE.md @@ -9,7 +9,7 @@ The module header at the site owns its subsystem — read it before changing a m - **The idle loop may not take a global lock unconditionally** — the crash report reads the process table through a `try_lock` it must never block on. `sched/reap_gate.rs` is the pattern: a relaxed-load gate in front of the lock. - **Every x86 Ring 0 entry clears the direction flag, and every interrupt or trap gate `AC`** — no hardware gate does either; `memmove` sets `DF` across `rep` operations with interrupts on, and Ring 3 sets `AC`, under which SMAP lets Ring 0 reach user pages. `arch::entry::ring3_naked_asm` is where they live; `SYSCALL` clears `AC` through `IA32_FMASK`. - **`BackendGuard` masks interrupts for its whole life**, so anything written under it is an interrupt latency; a new holder bounds itself as the console drain does. -- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_process_data` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern. +- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_handles` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern. - **A decision the process table makes lives in `kernel/proclife/`, never in `process.rs`** — its defects are interleavings and its host tests are the only machine that can enumerate one. - **A task holds at most one watch registration** — a standing registration across a loop must not call anything that registers again. A double registration panics only at attempt ≥ 2, so the contention depth is the coverage. - **`ops::close` cancels a poll only for a source its object really ends** — `Watch::cancel_polls` answers every ring's poll on that watch; `ops::close_ends_polls` is where a new object kind answers. diff --git a/kernel/src/inbox/mod.rs b/kernel/src/inbox/mod.rs index 6b7e1474b52..d8cb31a3edd 100644 --- a/kernel/src/inbox/mod.rs +++ b/kernel/src/inbox/mod.rs @@ -531,9 +531,9 @@ fn process_watch(inbox: &Arc, submission: &Submission) { } /// The object `handle` names in this process's table, not the thread's: a ring is process-wide. -/// Cloned out, so what follows holds no process lock. +/// Cloned out, so what follows holds no handle-table lock. fn resolve(handle: RawHandle) -> Result { - process::with_process_data(|data| data.handles.get_ref(handle, Rights::WAIT).cloned()) + process::with_handles(|handles| handles.get_ref(handle, Rights::WAIT).cloned()) } /// Keep `poll` as its handle's one poll, or in the place of `looked_at`, the @@ -663,13 +663,13 @@ impl Submitter> for Arc { fn process_accept(inbox: &Arc, submission: &Submission) { let user_data = submission.token; - let acceptor = process::with_process_data(|data| { - data.handles.get::(submission.handle, Rights::READ) + let acceptor = process::with_handles(|handles| { + handles.get::(submission.handle, Rights::READ) }); let acceptor = match acceptor { Ok(a) => a, - // Nothing held: `with_process_data` has given the guard up. + // Nothing held: `with_handles` has given the guard up. Err(e) => { let refusal = e.refuse_as_error(); polls::complete(inbox, user_data, -(refusal as i32)); @@ -687,8 +687,8 @@ fn process_accept(inbox: &Arc, submission: &Submission) { }; match acceptor.pop() { Some(conn) => { - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn))) + let installed = process::with_handles(|handles| { + ops::install(handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn))) }); match installed { Ok(h) => polls::complete(inbox, user_data, h.0 as i32), diff --git a/kernel/src/inventory.rs b/kernel/src/inventory.rs index ec2e0a3878e..50aca43c139 100644 --- a/kernel/src/inventory.rs +++ b/kernel/src/inventory.rs @@ -35,8 +35,8 @@ fn claims() -> Vec { }; let mut out = Vec::new(); for (pid, name, data) in processes { - let data = data.lock(); - for (_, entry) in data.handles.iter() { + let handles = Arc::clone(&data.lock().handles); + for (_, entry) in handles.lock().iter() { let KObjectRef::Device(claim) = entry.object() else { continue }; let on = if claim.class() == toyos_abi::syscall::DeviceType::PciFunction { // A function the last handle has let go names nothing. diff --git a/kernel/src/loader/mod.rs b/kernel/src/loader/mod.rs index 2b0d60c186e..0a1edf8d15a 100644 --- a/kernel/src/loader/mod.rs +++ b/kernel/src/loader/mod.rs @@ -135,7 +135,7 @@ pub fn spawn( parent: Parent, ) -> Result<(Pid, C::Held), crate::object::Refusal> { let job = Arc::clone(commit.job()); - let pager = if commit.pager() { crate::process::PagerRole::May } else { crate::process::PagerRole::None }; + let pager = commit.pager(); // Before anything is built; dropped on every way out below but the insert. let admission = Admission::ask(parent)?; let charge = ProcessData::charge(&job, path).map_err(SyscallError::from)?; @@ -189,15 +189,15 @@ pub fn spawn( // The point of no return: every failure above answers the caller with its // table untouched. let (handles, endowments, held) = commit.commit(start::Own::new(&object, ¬ice))?; + let handles = crate::process::Handles::new(handles, pager); let proc_data = Arc::new(Lock::new(ProcessData { - handles, + handles: Arc::clone(&handles), space: Arc::clone(&child_pt), spawn_ns: crate::clock::nanos_since_boot(), accounting: ProcessAccounting::default(), endowments, job: job.clone(), _charge: charge, - pager, })); let thread_data = Arc::new(Lock::new(ThreadData { @@ -231,6 +231,7 @@ pub fn spawn( thread_pointer: 0, job, image, + handles, // The spawning thread's, as a new thread's is. mask: crate::sched::driver::current_mask(), near: None, diff --git a/kernel/src/loader/start.rs b/kernel/src/loader/start.rs index 9faf8530c00..a9114523dff 100644 --- a/kernel/src/loader/start.rs +++ b/kernel/src/loader/start.rs @@ -14,7 +14,7 @@ use crate::object::notice::NoticeObject; use crate::object::process::ProcessObject; use crate::object::{ops, HandleEntry, HandleTable, KObjectRef, Refusal}; use crate::process::{ - process_data, Endowments, KERNEL_STACK_SIZE, + handles, process_data, with_handles, Endowments, KERNEL_STACK_SIZE, }; use crate::user_ptr::UserBytes; use toyos_abi::handle::{RawHandle, Rights}; @@ -114,7 +114,7 @@ pub trait Commit { /// The job the child runs in, which everything built for it is counted to. fn job(&self) -> &Arc; - /// Whether the child starts as a pager (`process::PagerRole`). + /// Whether the child starts as a pager (`process::Handles::is_pager`). fn pager(&self) -> bool; /// The child's table around `own`, what the kernel puts beside its @@ -138,8 +138,9 @@ impl Commit for PendingHandles { /// last answer. The handle under [`JOB_LABEL`] still names [`job`](Self::job), or the spawn is refused. fn commit(self, own: Own) -> Result<(HandleTable, Endowments, RawHandle), Refusal> { let Self { mut table, endow, mut labels, job, .. } = self; - let data_arc = process_data(); - let mut data = data_arc.lock(); + let own_job = process_data().lock().job.clone(); + let caller = handles(); + let mut handles = caller.lock(); let mut moving: Vec<(EndowEntry, RawHandle, Rights)> = Vec::with_capacity(endow.len()); let mut endowed_job: Option> = None; @@ -156,10 +157,10 @@ impl Commit for PendingHandles { if endowed_job.is_some() { return Err(SyscallError::InvalidArgument.into()); } - endowed_job = Some(data.handles.get_sent::(handle, Rights::WRITE)?.job().clone()); + endowed_job = Some(handles.get_sent::(handle, Rights::WRITE)?.job().clone()); } // Checked before any removal, so a missing `TRANSFER` refuses the spawn instead of leaving a hole. - let rights = data.handles.rights_to_move(handle, want)?; + let rights = handles.rights_to_move(handle, want)?; // A duplicate is refused here: unremoved handles let it pass twice, then the second removal's // `expect` below panics. if moving.iter().any(|(_, seen, _)| *seen == handle) { @@ -169,11 +170,11 @@ impl Commit for PendingHandles { } // Before any removal: the table changed under the spawn if the job // everything was built for is no longer the one it names. - if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&data.job)) { + if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&own_job)) { return Err(SyscallError::InvalidArgument.into()); } // Checked before any removal, so a failed install can't strand a handle out of a table that never spawned. - if !table.make_room(moving.len() + Own::LABELS.len()) || !data.handles.make_room(1) { + if !table.make_room(moving.len() + Own::LABELS.len()) || !handles.make_room(1) { return Err(SyscallError::ResourceExhausted.into()); } let mut entries = Vec::with_capacity(moving.len()); @@ -181,8 +182,7 @@ impl Commit for PendingHandles { // it under the caller's lock and revoked once that lock is let go. let mut handed = Vec::new(); for (mut entry, parent_handle, rights) in moving { - let moved = data - .handles + let moved = handles .remove(parent_handle) .expect("an endowed handle verified under this lock stopped resolving") .arriving_with(rights); @@ -195,9 +195,9 @@ impl Commit for PendingHandles { entries.push(entry); } // Minted while `own` is held, and `own` goes into a table no thread reaches until the child lands: another thread of the caller closing this handle never closes the object's last. - let held = ops::install(&mut data.handles, own.process.object().clone()) + let held = ops::install(&mut handles, own.process.object().clone()) .expect("a caller's table with verified room refused its child"); - drop(data); + drop(handles); for handed in handed { handed.revoke(); } @@ -268,12 +268,11 @@ fn label_of(labels: &[u8], off: u32, len: u32) -> Option<&[u8]> { /// The job a child runs in: the one endowed under [`JOB_LABEL`], or else the /// caller's. Only looked up: [`PendingHandles::commit`] refuses the rest. fn child_job(endow: &[SpawnEndow], labels: &[u8]) -> Result, Refusal> { - let data_arc = process_data(); - let data = data_arc.lock(); for entry in endow { if label_of(labels, entry.label_off, entry.label_len) == Some(JOB_LABEL.as_bytes()) { - return Ok(data.handles.get_sent::(entry.moved.handle, Rights::WRITE)?.job().clone()); + let named = with_handles(|handles| handles.get_sent::(entry.moved.handle, Rights::WRITE))?; + return Ok(named.job().clone()); } } - Ok(data.job.clone()) + Ok(process_data().lock().job.clone()) } diff --git a/kernel/src/message.rs b/kernel/src/message.rs index 3764e6439b0..406a53ad28b 100644 --- a/kernel/src/message.rs +++ b/kernel/src/message.rs @@ -255,7 +255,7 @@ impl QueueRef { } /// Queue `src`, already copied in, and `batch`, its handles, whole in - /// this hold, called under the sender's process lock as its handles move; + /// this hold, called under the sender's handle-table lock as its handles move; /// a refusal hands the batch back and queues nothing. No copy waits on it. pub fn send_now(&self, src: &[u8], mut batch: Vec) -> Result<(), (Vec, Unsent)> { let mut state = self.0.state.lock(); @@ -280,7 +280,7 @@ impl QueueRef { /// Take the front message whole into `into` and install its handles in /// `table`, in this hold, where a reader holding `buf` bytes and `cap` - /// handles can take it, called under the receiver's process lock. A table + /// handles can take it, called under the receiver's handle-table lock. A table /// that must grow first answers [`Unclaimed::Room`], taking nothing. pub fn recv_now(&self, table: &mut HandleTable, buf: usize, cap: usize, into: &mut MessageBounce) -> Result { let mut state = self.0.state.lock(); diff --git a/kernel/src/object/handle.rs b/kernel/src/object/handle.rs index 651a39b64be..f598a6958ea 100644 --- a/kernel/src/object/handle.rs +++ b/kernel/src/object/handle.rs @@ -1,7 +1,7 @@ //! One process's handle table. //! -//! Lives behind `ProcessData`'s own lock; every accessor returns an owned -//! value, so no borrow into the table outlives the guard. +//! Lives behind its own lock (`process::Handles`); every accessor returns an +//! owned value, so no borrow into the table outlives the guard. //! //! Not a file-descriptor table — `fd` names only `sdk/libc`'s POSIX //! surface (owner ruling) — and a slot at its last generation retires @@ -22,7 +22,7 @@ use crate::mm::charge::{self, Charge}; use super::{KObjectRef, KObjectVariant}; -/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the process's own lock, where [`HandleError::refuse`] must not be called. +/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the table's own lock, where [`HandleError::refuse`] must not be called. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub struct TableFull; diff --git a/kernel/src/object/notice.rs b/kernel/src/object/notice.rs index 27cd5a5425e..1506411fce4 100644 --- a/kernel/src/object/notice.rs +++ b/kernel/src/object/notice.rs @@ -19,8 +19,8 @@ use super::{KObjectVariant, ObjectCore}; pub struct NoticeObject { pub(super) core: ObjectCore, - /// A leaf: a quit takes it under `PROCESS_TABLE`, a read under the - /// reader's `ProcessData`, and nothing is taken under it. + /// A leaf: a quit takes it under `PROCESS_TABLE`, and nothing is taken + /// under it. state: Lock, /// Posted when a reason is put in it. watch: Arc, diff --git a/kernel/src/pipe.rs b/kernel/src/pipe.rs index 7c407f625f8..9581228fa6d 100644 --- a/kernel/src/pipe.rs +++ b/kernel/src/pipe.rs @@ -330,7 +330,7 @@ impl PipeRef { } /// Write `src`, already copied in, whole in one hold of the pipe's lock, - /// for a caller holding its process's lock: `None` where that cannot answer + /// for a caller holding its handle-table lock: `None` where that cannot answer /// it without a wait or frames taken, which [`Self::write`] does. pub fn put_now(&self, src: &[u8]) -> Option { let mut pipe = self.0.state.lock(); @@ -347,7 +347,7 @@ impl PipeRef { } /// Take at most `max` bytes, up to [`BOUNCE`], into `into` whole in one - /// hold of the pipe's lock, for a caller holding its process's lock: + /// hold of the pipe's lock, for a caller holding its handle-table lock: /// `None` where the pipe holds nothing or more than a bounce would take, /// or another read holds its claim, which [`Self::read`] answers. pub fn take_now(&self, max: usize, into: &mut PipeBounce) -> Option { diff --git a/kernel/src/process.rs b/kernel/src/process.rs index 01b96fef2db..91ea761899f 100644 --- a/kernel/src/process.rs +++ b/kernel/src/process.rs @@ -29,10 +29,11 @@ use alloc::collections::BTreeMap; use alloc::string::String; use alloc::sync::Arc; use alloc::vec::Vec; +use core::sync::atomic::{AtomicBool, Ordering}; use crate::arch::percpu; use crate::mm::charge::{self, Charge, Exhausted}; use crate::object::{ops, HandleTable}; -use crate::sync::Lock; +use crate::sync::{Lock, LockGuard}; use crate::sched::driver; use crate::sched::payload::ThreadSched; use kernel::sched::cpu::{homes, Near}; @@ -60,6 +61,51 @@ pub use crate::loader::{build_child_handles, spawn, spawn_supervisor}; /// A process's space, which every thread of it runs in. pub use crate::mm::space::PageTables; +/// What a process names kernel objects by. Each of its threads carries it, so +/// a lookup ([`with_handles`]) takes neither the process table nor +/// [`ProcessData`]'s lock. +pub struct Handles { + /// Every kernel object it can name; stdio is handles `0`, `1`, `2`. + table: Lock, + /// Started as a pager, which no call changes (`object::paged`): it may + /// make one pager object, and is refused a holder's read, which would wait + /// on itself or on another pager. + pager: bool, + /// It made that pager object. + made_pager: AtomicBool, +} + +impl Handles { + pub fn new(table: HandleTable, pager: bool) -> Arc { + Arc::new(Self { table: Lock::new(table), pager, made_pager: AtomicBool::new(false) }) + } + + pub fn lock(&self) -> LockGuard<'_, HandleTable> { + self.table.lock() + } + + pub fn is_pager(&self) -> bool { + self.pager + } + + /// Claim the one pager object a pager may make: `PermissionDenied` for a + /// process started as none, `AlreadyExists` once claimed. + pub fn claim_pager(&self) -> Result<(), SyscallError> { + if !self.pager { + return Err(SyscallError::PermissionDenied); + } + if self.made_pager.swap(true, Ordering::Relaxed) { + return Err(SyscallError::AlreadyExists); + } + Ok(()) + } + + /// Give the claim back: its pager object was never installed. + pub fn unclaim_pager(&self) { + self.made_pager.store(false, Ordering::Relaxed); + } +} + /// An object the kernel mapped into a process for it and takes back itself: an /// inbox's rings. The kernel writes what it owes the process into the object. pub struct KernelMapping { @@ -229,8 +275,9 @@ impl Processes for ProcessTable { /// Process-level data shared across all threads via `Arc>`. pub struct ProcessData { - /// Every kernel object this process can name; stdio is handles `0`, `1`, `2`. - pub handles: HandleTable, + /// What its threads carry, under a lock of its own; here for a reader that + /// is not one of them. + pub handles: Arc, /// The space every thread of it runs in. pub space: PageTables, @@ -243,25 +290,6 @@ pub struct ProcessData { pub job: Arc, /// What [`ProcessData::charge`] charged. pub _charge: Charge, - /// Whether it was started as a pager, which no call changes, and whether - /// it made its one pager object. - pub pager: PagerRole, -} - -/// A process's part in paged objects (`object::paged`): set at its spawn. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum PagerRole { - None, - /// Started as a pager; it has not made its pager object. - May, - /// It made it. - Made, -} - -impl PagerRole { - pub fn is_pager(self) -> bool { - self != PagerRole::None - } } impl ProcessData { @@ -275,6 +303,7 @@ impl ProcessData { // A thread table of one thread is a whole leaf. let threads = charge::heap(Layout::array::<(Tid, ThreadEntry)>(11).expect("process: a leaf's layout")); let records = charge::arc::>() + + charge::arc::() + charge::arc::() + charge::arc::() + charge::arc::() @@ -475,6 +504,17 @@ pub fn with_process_data(f: impl FnOnce(&mut ProcessData) -> R) -> R { f(&mut guard) } +/// The running thread's process's handles, off the thread itself. +pub fn handles() -> Arc { + scheduler::current_handles().expect("handles: no thread is running") +} + +/// `f` over the running thread's process's handle table, under the table's +/// own lock: the lookup every handle-taking call makes. +pub fn with_handles(f: impl FnOnce(&mut HandleTable) -> R) -> R { + f(&mut handles().lock()) +} + /// Whom a new process is placed under. #[derive(Clone, Copy)] pub enum Parent { @@ -573,6 +613,7 @@ pub fn spawn_thread( (addr_space, Arc::clone(&proc.process_data)) }; let job = process_data_arc.lock().job.clone(); + let handles = handles(); // Phase 2: the kernel stack, outside any lock. let (ks_alloc, ks_sp) = alloc_kernel_stack(&job, Start::Thread { entry, sp: stack_ptr, arg })?; @@ -609,6 +650,7 @@ pub fn spawn_thread( thread_pointer: 0, job, image, + handles, mask, near: Some(Near { spawner: driver::current_cpu(), homes }), }); @@ -664,11 +706,12 @@ fn teardown_resources( #[cfg(feature = "mask-windows")] crate::windows::report(); - let mut data = process_data_arc.lock(); - ops::close_all(&mut data.handles); + let handles = Arc::clone(&process_data_arc.lock().handles); + ops::close_all(&mut handles.lock()); // Every thread has left, and none returns to Ring 3 to use them. crate::isa::process_ends(pid); + let data = process_data_arc.lock(); let usage = data.space.usage(); Consumed { peak_memory: usage.peak_bytes(), diff --git a/kernel/src/sched/driver.rs b/kernel/src/sched/driver.rs index 1b3ce5d8324..98af68cb65a 100644 --- a/kernel/src/sched/driver.rs +++ b/kernel/src/sched/driver.rs @@ -397,6 +397,8 @@ pub struct NewTask { pub job: Arc, /// The process's image. pub image: Arc, + /// The process's handles. + pub handles: Arc, /// The CPUs it may run on. pub mask: CpuMask, /// For a thread spawned into its own process, its spawner and its @@ -430,6 +432,7 @@ pub fn spawn(new: NewTask) -> (ThreadSched, CpuId) { address_space: new.address_space, handle: handle.clone(), image: new.image, + handles: new.handles, }, rt: RtState::default(), mask: new.mask, @@ -781,6 +784,11 @@ pub fn current_address_space() -> Option { try_with_cpu(|cpu| cpu.running().map(|t| t.ext().address_space.clone())).flatten() } +/// The running task's process's handles. `None` off-task. +pub fn current_handles() -> Option> { + try_with_cpu(|cpu| cpu.running().map(|t| t.ext().handles.clone())).flatten() +} + pub fn with_current_acct( f: impl FnOnce(&kernel::sched::task::TaskAccounting) -> R, ) -> Option { diff --git a/kernel/src/sched/payload.rs b/kernel/src/sched/payload.rs index e1ef23909f7..8676856bae7 100644 --- a/kernel/src/sched/payload.rs +++ b/kernel/src/sched/payload.rs @@ -15,7 +15,7 @@ use kernel::sched::park::WaitTicket; use crate::watch::Watch; use crate::mm::paging::Root; use crate::loader::KernelStack; -use crate::process::{PageTables, ProcessAccounting, TaskId, UserImage}; +use crate::process::{Handles, PageTables, ProcessAccounting, TaskId, UserImage}; use crate::sync::Lock; /// The environment's leaf lock; holding it raises the preempt count, making a wake path a legal mailbox producer. @@ -82,6 +82,8 @@ pub struct KernelPayload { pub handle: Arc, /// This task's process's image, as a crash report records a frame in it; kept here, not looked up via the process table, so a crash report never takes that lock. pub image: Arc, + /// This task's process's handles, kept here for the same reason: a lookup takes no lock but the table's. + pub handles: Arc, } impl SchedPayload for KernelPayload { diff --git a/kernel/src/scheduler.rs b/kernel/src/scheduler.rs index 351d718fbe7..bc5765dbb1a 100644 --- a/kernel/src/scheduler.rs +++ b/kernel/src/scheduler.rs @@ -20,7 +20,7 @@ use crate::time::Deadline; use crate::DirectMap; pub use crate::sched::driver::{ - current_address_space, enter_idle_loop, in_pass as in_schedule_self, total_cpu_ns, + current_address_space, current_handles, enter_idle_loop, in_pass as in_schedule_self, total_cpu_ns, NewTask, Ticket, }; #[cfg(feature = "heap-tripwire")] diff --git a/kernel/src/syscall/device.rs b/kernel/src/syscall/device.rs index e7df3fe6948..a40ade251f3 100644 --- a/kernel/src/syscall/device.rs +++ b/kernel/src/syscall/device.rs @@ -27,8 +27,8 @@ pub(super) fn sys_device_reg_read(handle: RawHandle, offset: u64, width: u64) -> let Some(width) = toyos_abi::syscall::RegWidth::from_raw(width) else { return SyscallError::InvalidArgument.to_u64(); }; - let claim = process::with_process_data(|data| { - data.handles.get::(handle, Rights::NONE).map(|claim| { + let claim = process::with_handles(|handles| { + handles.get::(handle, Rights::NONE).map(|claim| { (claim.class() == device::DeviceType::PciFunction).then_some(claim) }) }); @@ -59,7 +59,7 @@ pub(super) fn sys_device_reg_read(handle: RawHandle, offset: u64, width: u64) -> /// `PermissionDenied`, as its ports answer nothing there. pub(super) fn sys_acpi(ctx: &SyscallContext, handle: RawHandle, op: u64, at: u64) -> u64 { use toyos_abi::acpi::op as ops; - let claim = match process::with_process_data(|data| data.handles.get::(handle, Rights::WRITE)) { + let claim = match process::with_handles(|handles| handles.get::(handle, Rights::WRITE)) { Ok(claim) if claim.class() == device::DeviceType::Acpi => claim, Ok(claim) => { let held = claim.class().class_name(); @@ -171,8 +171,8 @@ pub(super) fn sys_device_claim(syscap: RawHandle, class: u64, selector: [u64; 2] // Not yet, rather than never: the device set says when. Err(device::ClaimError::Undeclared) => return SyscallError::WouldBlock.to_u64(), }; - process::with_process_data(|data| { - handle_result(ops::install(&mut data.handles, KObjectRef::Device(claim))) + process::with_handles(|handles| { + handle_result(ops::install(handles, KObjectRef::Device(claim))) }) } @@ -186,8 +186,8 @@ pub(super) fn sys_device_claim(syscap: RawHandle, class: u64, selector: [u64; 2] /// **The claim, and nothing read out of it**: a call reaches its function only /// through [`on_function`], so what it acts on is this claim's or nothing. fn pci_claim(handle: RawHandle) -> Result, crate::object::Refusal> { - let claim = process::with_process_data(|data| { - data.handles.get::(handle, Rights::WRITE) + let claim = process::with_handles(|handles| { + handles.get::(handle, Rights::WRITE) })?; // A claim of another class is the wrong kind of thing here, not a missing // device: it names a device this call has no meaning for. @@ -236,12 +236,12 @@ fn install_handed( claim: &Arc, entry: crate::object::HandleEntry, ) -> Result { - process::with_process_data(|data| { - let held = data.handles.get::(handle, Rights::NONE).is_ok_and(|held| Arc::ptr_eq(&held, claim)); + process::with_handles(|handles| { + let held = handles.get::(handle, Rights::NONE).is_ok_and(|held| Arc::ptr_eq(&held, claim)); if !held { return Err(SyscallError::Gone); } - data.handles.install(entry).map_err(|_| SyscallError::ResourceExhausted) + handles.install(entry).map_err(|_| SyscallError::ResourceExhausted) }) } @@ -312,8 +312,8 @@ pub(super) fn sys_device_dma_alloc( Ok(grant) => grant, Err(e) => return e.to_u64(), }; - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, KObjectRef::SharedMem(alloc::sync::Arc::clone(&memory))) + let installed = process::with_handles(|handles| { + ops::install(handles, KObjectRef::SharedMem(alloc::sync::Arc::clone(&memory))) }); let shm = match installed { Ok(handle) => handle, @@ -348,8 +348,8 @@ pub(super) fn sys_device_dma_map( Ok(claim) => claim, Err(e) => return e.refuse(), }; - let memory = match process::with_process_data(|data| { - data.handles.get::(region, Rights::MAP.union(Rights::WRITE)) + let memory = match process::with_handles(|handles| { + handles.get::(region, Rights::MAP.union(Rights::WRITE)) }) { Ok(memory) => memory, Err(e) => { diff --git a/kernel/src/syscall/dispatch.rs b/kernel/src/syscall/dispatch.rs index dc95fea4245..cb6eeecdbf2 100644 --- a/kernel/src/syscall/dispatch.rs +++ b/kernel/src/syscall/dispatch.rs @@ -158,8 +158,8 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> } SYS_PROCESS_WAIT => sys_process_wait(RawHandle(a1 as u32), a2), SYS_PROCESS_KILL => { - match process::with_process_data(|data| { - data.handles + match process::with_handles(|handles| { + handles .get::(RawHandle(a1 as u32), Rights::MANAGE) }) { Ok(object) => process::kill_process(&object), @@ -441,7 +441,7 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> let Ok(slot) = u16::try_from(a2) else { return SyscallError::InvalidArgument.to_u64(); }; - match process::with_process_data(|d| d.handles.stage_last_generation(slot)) { + match process::with_handles(|handles| handles.stage_last_generation(slot)) { Some(h) => u64::from(h.0), None => SyscallError::InvalidArgument.to_u64(), } diff --git a/kernel/src/syscall/event.rs b/kernel/src/syscall/event.rs index 8dae451a94a..abba27528ad 100644 --- a/kernel/src/syscall/event.rs +++ b/kernel/src/syscall/event.rs @@ -25,7 +25,7 @@ pub(super) fn sys_event_create() -> u64 { } pub(super) fn sys_event_signal(h: RawHandle) -> u64 { - match process::with_process_data(|data| data.handles.get::(h, Rights::WRITE)) { + match process::with_handles(|handles| handles.get::(h, Rights::WRITE)) { Ok(event) if event.signal() => 0, Ok(_) => SyscallError::Gone.to_u64(), Err(e) => e.refuse(), @@ -35,7 +35,7 @@ pub(super) fn sys_event_signal(h: RawHandle) -> u64 { /// `1` for a signal taken, `WouldBlock` for none pending, `Gone` for none /// pending and no signaler left. Never blocks: a wait is `SYS_HANDLE_WAIT`. pub(super) fn sys_event_take(h: RawHandle) -> u64 { - match process::with_process_data(|data| data.handles.get::(h, Rights::READ)) { + match process::with_handles(|handles| handles.get::(h, Rights::READ)) { Ok(event) => match event.take() { Taken::Signal => 1, Taken::Empty => SyscallError::WouldBlock.to_u64(), @@ -55,7 +55,7 @@ pub(super) fn sys_handle_wait(h: RawHandle, interest: u64, timeout_nanos: u64) - Ok(bit @ (READABLE | WRITABLE | OTHER_END_GONE)) => bit, _ => return SyscallError::InvalidArgument.to_u64(), }; - let object = match process::with_process_data(|data| data.handles.get_ref(h, Rights::WAIT).cloned()) { + let object = match process::with_handles(|handles| handles.get_ref(h, Rights::WAIT).cloned()) { Ok(object) => object, Err(e) => return e.refuse(), }; diff --git a/kernel/src/syscall/handles.rs b/kernel/src/syscall/handles.rs index 8a78ebe74e8..822d2c7129e 100644 --- a/kernel/src/syscall/handles.rs +++ b/kernel/src/syscall/handles.rs @@ -11,19 +11,19 @@ use toyos_abi::syscall::*; /// Requires that `syscap` resolves to a `SysCap` holding `need`, and answers /// every right it holds, read under the same guard. pub(super) fn demand_syscap(syscap: RawHandle, need: Rights) -> Result { - process::with_process_data(|data| { - data.handles.get::(syscap, need)?; - data.handles.rights_of(syscap) + process::with_handles(|handles| { + handles.get::(syscap, need)?; + handles.rights_of(syscap) }) } -/// The object `h` names, cloned out of the process's lock: what a call that +/// The object `h` names, cloned out of the handle table's lock: what a call that /// copies to or from its caller, or may park, works on (`user_ptr`). pub(super) fn object(h: RawHandle, need: Rights) -> Result { - process::with_process_data(|data| data.handles.get_ref(h, need).cloned()) + process::with_handles(|handles| handles.get_ref(h, need).cloned()) } -/// `f` over the object `h` names, outside the process's lock, for the calls +/// `f` over the object `h` names, outside the handle table's lock, for the calls /// whose answer is already a raw syscall word. pub(super) fn with_object(h: RawHandle, need: Rights, f: impl FnOnce(&KObjectRef) -> u64) -> u64 { match object(h, need) { @@ -41,8 +41,8 @@ pub(super) fn handle_result(r: Result) -> u64 { /// Wakes nobody: `pipe::close_write`'s own zero-refcount path does that. pub(super) fn sys_close(h: RawHandle) -> u64 { - let result = process::with_process_data(|data| { - ops::close(&mut data.handles, h) + let result = process::with_handles(|handles| { + ops::close(handles, h) }); match result { Ok(()) => 0, @@ -56,15 +56,15 @@ pub(super) fn sys_close(h: RawHandle) -> u64 { /// /// `want` is the wire form of `Option` (`RIGHTS_UNCHANGED` = the source's own set), decoded here and nowhere else. pub(super) fn sys_handle_dup(h: RawHandle, want: u64) -> u64 { - let duplicated = process::with_process_data(|data| { - let held = data.handles.rights_of(h)?; + let duplicated = process::with_handles(|handles| { + let held = handles.rights_of(h)?; let rights = if want == RIGHTS_UNCHANGED { held } else { let bits = u32::try_from(want).map_err(|_| SyscallError::InvalidArgument)?; Rights::from_bits(bits).ok_or(SyscallError::InvalidArgument)? }; - Ok::<_, crate::object::Refusal>(data.handles.duplicate(h, rights)?) + Ok::<_, crate::object::Refusal>(handles.duplicate(h, rights)?) }); match duplicated { Ok(new_h) => new_h.0 as u64, diff --git a/kernel/src/syscall/io.rs b/kernel/src/syscall/io.rs index e7d3773066b..e4600eab541 100644 --- a/kernel/src/syscall/io.rs +++ b/kernel/src/syscall/io.rs @@ -1,6 +1,6 @@ //! Blocking `read`/`write` register on the object's watch before re-checking the //! condition — closing the check-then-block window. Each works on its object -//! cloned out of the process's lock, which no copy and no park runs under. +//! cloned out of the handle table's lock, which no copy and no park runs under. //! //! Buffers arrive already bounded: `super::dispatch` converts the caller's //! pointer and length into a [`UserBytes`]/[`UserBytesMut`] window before @@ -39,7 +39,7 @@ enum ReadBlock { } /// A write of at most `BOUNCE` bytes to a pipe, made whole in the hold of the -/// process's lock that looks its handle up, its bytes copied in before it, as +/// handle table's lock that looks its handle up, its bytes copied in before it, as /// a write was before any copy left the locks: `None` for anything else, which /// the general path answers (copying a small write to another object again). fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { @@ -49,8 +49,8 @@ fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { let mut bounce = PipeBounce::new(); // A buffer that faults is the general path's to answer, after its handle. let src = bounce.read(buf).ok()?.bytes(); - let (word, pipe) = process::with_process_data(|data| { - let KObjectRef::PipeWrite(w) = data.handles.get_ref(h, Rights::WRITE).ok()? else { return None }; + let (word, pipe) = process::with_handles(|handles| { + let KObjectRef::PipeWrite(w) = handles.get_ref(h, Rights::WRITE).ok()? else { return None }; let word = ops::pipe_written(w.pipe().put_now(src)?)?; Some((word, w.pipe().clone())) })?; @@ -59,12 +59,12 @@ fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { } /// A read of a pipe holding at most `BOUNCE` bytes for it, taken in the hold -/// of the process's lock that looks its handle up and copied out after it: +/// of the handle table's lock that looks its handle up and copied out after it: /// `None` for anything else, which the general path answers. fn pipe_read_now(h: RawHandle, buf: &mut UserBytesMut) -> Option { let mut bounce = PipeBounce::new(); - let (taken, pipe) = process::with_process_data(|data| { - let KObjectRef::PipeRead(r) = data.handles.get_ref(h, Rights::READ).ok()? else { return None }; + let (taken, pipe) = process::with_handles(|handles| { + let KObjectRef::PipeRead(r) = handles.get_ref(h, Rights::READ).ok()? else { return None }; Some((r.pipe().take_now(buf.len(), &mut bounce)?, r.pipe().clone())) })?; let n = taken.n; @@ -121,7 +121,7 @@ fn read_block(object: &KObjectRef) -> ReadBlock { } } -/// What a read's handle names, out of the process's lock: a device claim is +/// What a read's handle names, out of the handle table's lock: a device claim is /// read under it, since describing one installs handles, and its answer copied /// out after. enum Readable { @@ -130,11 +130,11 @@ enum Readable { } fn readable(h: RawHandle, room: usize) -> Result { - process::with_process_data(|data| { - let object = data.handles.get_ref(h, Rights::READ)?; + process::with_handles(|handles| { + let object = handles.get_ref(h, Rights::READ)?; let KObjectRef::Device(claim) = object else { return Ok(Readable::Object(object.clone())) }; let claim = alloc::sync::Arc::clone(claim); - Ok(Readable::Device(ops::read_device(&claim, &mut data.handles, room))) + Ok(Readable::Device(ops::read_device(&claim, handles, room))) }) } @@ -228,7 +228,7 @@ pub(super) fn sys_null(want: u64) -> u64 { Err(e) => return SyscallError::from(e).to_u64(), }; let entry = crate::object::HandleEntry::new(object, rights); - match process::with_process_data(|data| data.handles.install(entry)) { + match process::with_handles(|handles| handles.install(entry)) { Ok(h) => h.0 as u64, Err(_) => SyscallError::ResourceExhausted.to_u64(), } diff --git a/kernel/src/syscall/ipc.rs b/kernel/src/syscall/ipc.rs index 7ba0cb86cf6..09914f3a039 100644 --- a/kernel/src/syscall/ipc.rs +++ b/kernel/src/syscall/ipc.rs @@ -12,7 +12,7 @@ use alloc::vec::Vec; use crate::watch; use crate::object::{ops, port, KObjectRef}; use crate::message::{MessageBounce, QueueRef, SendRefused, Unclaimed, Unsent}; -use crate::object::handle::HandleEntry; +use crate::object::handle::{HandleEntry, HandleTable}; use crate::object::service::ConnectionEnd; use crate::time::Deadline; use crate::user_ptr::SyscallContext; @@ -40,13 +40,13 @@ pub(super) fn sys_pipe() -> u64 { /// Install two objects or neither; the two handles come back packed into one word, which cannot be read as an error. pub(super) fn install_pair(first: KObjectRef, second: KObjectRef) -> u64 { - let installed = process::with_process_data(|data| { + let installed = process::with_handles(|handles| { // Room for both before either: a refused pair installs nothing, so nothing is taken back. - if !data.handles.make_room(2) { + if !handles.make_room(2) { return Err((first, second)); } let mut install = - |object| ops::install(&mut data.handles, object).expect("room was asked for first"); + |object| ops::install(handles, object).expect("room was asked for first"); let first = install(first); let second = install(second); Ok(((first.0 as u64) << 32) | second.0 as u64) @@ -97,8 +97,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - Vec::new(); if keep_all { - let base = match process::with_process_data(|data| { - data.handles.get::(args.base, Rights::READ) + let base = match process::with_handles(|handles| { + handles.get::(args.base, Rights::READ) }) { Ok(base) => base, Err(e) => return e.refuse(), @@ -115,8 +115,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - else { return SyscallError::BadAddress.to_u64(); }; - let base = match process::with_process_data(|data| { - data.handles.get::(args.base, Rights::READ) + let base = match process::with_handles(|handles| { + handles.get::(args.base, Rights::READ) }) { Ok(base) => base, Err(e) => return e.refuse(), @@ -156,8 +156,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - let Some(name) = name_at(off, len) else { return SyscallError::InvalidArgument.to_u64(); }; - let connector = match process::with_process_data(|data| { - data.handles.get_sent::(handle, Rights::TRANSFER) + let connector = match process::with_handles(|handles| { + handles.get_sent::(handle, Rights::TRANSFER) }) { Ok(c) => c, Err(e) => return e.refuse(), @@ -178,17 +178,15 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - return SyscallError::ResourceExhausted.to_u64() } }; - process::with_process_data(|data| { - handle_result(ops::install(&mut data.handles, KObjectRef::Namespace(namespace))) + process::with_handles(|handles| { + handle_result(ops::install(handles, KObjectRef::Namespace(namespace))) }) } /// Open a connection to `name` in the namespace `ns_h` holds; `NotFound` means the namespace lacks the name, `Gone` means its port has closed. pub(super) fn sys_namespace_open(ns_h: RawHandle, name: &str) -> u64 { - let connector = match process::with_process_data(|data| { - let ns = data - .handles - .get::(ns_h, Rights::READ)?; + let connector = match process::with_handles(|handles| { + let ns = handles.get::(ns_h, Rights::READ)?; Ok::<_, crate::object::HandleError>(ns.lookup(name).cloned()) }) { Ok(Some(c)) => c, @@ -200,10 +198,10 @@ pub(super) fn sys_namespace_open(ns_h: RawHandle, name: &str) -> u64 { /// Inspect only the names this namespace already holds. The Arc keeps the /// immutable entries alive after the handle-table lock is released; copying -/// into user memory therefore holds no object or process lock. +/// into user memory therefore holds no object or handle-table lock. pub(super) fn sys_namespace_names(ns_h: RawHandle, out: &mut crate::user_ptr::UserBytesMut) -> u64 { - let namespace = match process::with_process_data(|data| { - data.handles.get::(ns_h, Rights::READ) + let namespace = match process::with_handles(|handles| { + handles.get::(ns_h, Rights::READ) }) { Ok(namespace) => namespace, Err(e) => return e.refuse(), @@ -240,12 +238,12 @@ fn connect_through(connector: &port::Connector) -> u64 { Err(e) => return SyscallError::from(e).to_u64(), }; // One hold from the room to the install: the port can still refuse, and the client's end is in the table only once it has not. - let installed = process::with_process_data(|data| { - if !data.handles.make_room(1) { + let installed = process::with_handles(|handles| { + if !handles.make_room(1) { return Err((SyscallError::ResourceExhausted, object, pending)); } match connector.push(pending) { - Ok(()) => Ok(ops::install(&mut data.handles, object).expect("room was asked for first")), + Ok(()) => Ok(ops::install(handles, object).expect("room was asked for first")), Err((pending, port::PushError::Closed)) => Err((SyscallError::Gone, object, pending)), Err((pending, port::PushError::QueueFull)) => { Err((SyscallError::ResourceExhausted, object, pending)) @@ -268,8 +266,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { ACCEPT_NONBLOCK => false, _ => return SyscallError::InvalidArgument.to_u64(), }; - let acceptor = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let acceptor = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(a) => a, Err(e) => return e.refuse(), @@ -284,8 +282,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { if let Some(conn) = acceptor.pop() { // The ends move from the port's queue into the connection: ownership transfers, no count changes. let object = KObjectRef::Connection(ConnectionEnd::accepted(core, conn)); - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, object) + let installed = process::with_handles(|handles| { + ops::install(handles, object) }); return handle_result(installed); } @@ -314,8 +312,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { /// A connector to the caller's own port stamping every connection with `badge`, which the dispatch has copied in and bounded. `READ` on the acceptor, the right accepting takes, is the authority: whoever answers every connection on a port grants nothing by labelling them. pub(super) fn sys_port_mint(h: RawHandle, badge: &[u8]) -> u64 { - let acceptor = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let acceptor = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(a) => a, Err(e) => return e.refuse(), @@ -324,7 +322,7 @@ pub(super) fn sys_port_mint(h: RawHandle, badge: &[u8]) -> u64 { Ok(connector) => KObjectRef::Connector(connector), Err(e) => return SyscallError::from(e).to_u64(), }; - process::with_process_data(|data| handle_result(ops::install(&mut data.handles, connector))) + process::with_handles(|handles| handle_result(ops::install(handles, connector))) } /// The badge a connection accepted from `acceptor_h`'s port was stamped with, written to `out`; answers its length. @@ -335,9 +333,9 @@ pub(super) fn sys_port_badge( conn_h: RawHandle, out: &mut crate::user_ptr::UserBytesMut, ) -> u64 { - let found = process::with_process_data(|data| { - let acceptor = data.handles.get::(acceptor_h, Rights::READ)?; - let conn = data.handles.get::(conn_h, Rights::READ)?; + let found = process::with_handles(|handles| { + let acceptor = handles.get::(acceptor_h, Rights::READ)?; + let conn = handles.get::(conn_h, Rights::READ)?; Ok::<_, crate::object::HandleError>((acceptor.port_id(), conn.stamp().cloned())) }); let (port, stamp) = match found { @@ -359,7 +357,7 @@ pub(super) fn sys_port_badge( pub(super) fn sys_shm_create(size: u64, flags: u64, job: RawHandle) -> u64 { let charged = match flags & SHM_FOR_JOB { 0 => crate::mm::charge::current(), - _ => match process::with_process_data(|data| data.handles.get::(job, Rights::WRITE)) { + _ => match process::with_handles(|handles| handles.get::(job, Rights::WRITE)) { Ok(object) => alloc::sync::Arc::clone(object.job()), Err(e) => return e.refuse(), }, @@ -373,15 +371,15 @@ pub(super) fn sys_shm_create(size: u64, flags: u64, job: RawHandle) -> u64 { Ok(shm) => KObjectRef::SharedMem(shm), Err(e) => return e.to_u64(), }; - process::with_process_data(|data| handle_result(ops::install(&mut data.handles, object))) + process::with_handles(|handles| handle_result(ops::install(handles, object))) } /// `toyos_abi::syscall::shm_sub`: the new object's handle carries `h`'s rights. pub(super) fn sys_shm_sub(h: RawHandle, offset: u64, len: u64) -> u64 { // A refusal is made with nothing held: a bad handle's ends the process. - let held = process::with_process_data(|data| { - let whole = data.handles.get::(h, Rights::MAP)?; - Ok::<_, crate::object::handle::HandleError>((whole, data.handles.rights_of(h)?)) + let held = process::with_handles(|handles| { + let whole = handles.get::(h, Rights::MAP)?; + Ok::<_, crate::object::handle::HandleError>((whole, handles.rights_of(h)?)) }); let (whole, rights) = match held { Ok(held) => held, @@ -392,8 +390,8 @@ pub(super) fn sys_shm_sub(h: RawHandle, offset: u64, len: u64) -> u64 { Err(e) => return e.to_u64(), }; let entry = crate::object::handle::HandleEntry::new(KObjectRef::SharedMem(sub), rights); - process::with_process_data(|data| { - handle_result(data.handles.install(entry).map_err(|_| SyscallError::ResourceExhausted)) + process::with_handles(|handles| { + handle_result(handles.install(entry).map_err(|_| SyscallError::ResourceExhausted)) }) } @@ -404,8 +402,8 @@ pub(super) fn sys_shm_map(h: RawHandle, prot: u64) -> u64 { Some(VmProt::ReadWrite) => (Prot::ReadWrite, Rights::MAP.union(Rights::WRITE)), Some(VmProt::None | VmProt::ReadExec) | None => return SyscallError::InvalidArgument.to_u64(), }; - let shm = match process::with_process_data(|data| { - data.handles.get::(h, need) + let shm = match process::with_handles(|handles| { + handles.get::(h, need) }) { Ok(shm) => shm, Err(e) => return e.refuse(), @@ -474,14 +472,14 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 let need = if out.count == 0 { Rights::WRITE } else { Rights::WRITE.union(Rights::TRANSFER) }; // Every handle is checked before anything is queued or moved, and again // under the hold that moves them: the answer is what each arrives with. - let checked = |data: &process::ProcessData| { - let conn = data.handles.get::(conn_h, need)?; + let checked = |handles: &HandleTable| { + let conn = handles.get::(conn_h, need)?; let mut moving = [(RawHandle(0), Rights::NONE); MAX_TRANSFER_HANDLES]; for (i, (h, want)) in wanted.iter().enumerate() { if *h == conn_h || wanted[..i].iter().any(|(seen, _)| seen == h) { return Err(SyscallError::InvalidArgument.into()); } - moving[i] = (*h, data.handles.rights_to_move(*h, *want)?); + moving[i] = (*h, handles.rights_to_move(*h, *want)?); } Ok::<_, crate::object::Refusal>((conn.tx().clone(), moving)) }; @@ -491,9 +489,9 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 let src = bounce.read(&out.bytes).map_err(|faulted| SyscallError::from(faulted).to_u64())?.bytes(); let parkable = crate::scheduler::Parkable::at_entry(); loop { - let moved = process::with_process_data(|data| { - let (tx, moving) = checked(data)?; - let (moved, handed) = move_handles(data, &moving[..out.count], |batch| tx.send_now(src, batch)); + let moved = process::with_handles(|handles| { + let (tx, moving) = checked(handles)?; + let (moved, handed) = move_handles(handles, &moving[..out.count], |batch| tx.send_now(src, batch)); Ok::<_, crate::object::Refusal>((tx, moved, handed)) }); let (tx, refused) = match moved { @@ -506,9 +504,9 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 Ok((tx, Err(Unsent::Claim), _)) => { let refused = match tx.claim_send(&parkable, src, out.count) { Ok(mut claim) => { - let moved = process::with_process_data(|data| { - let (_, moving) = checked(data)?; - Ok::<_, crate::object::Refusal>(move_handles(data, &moving[..out.count], |batch| claim.commit(batch))) + let moved = process::with_handles(|handles| { + let (_, moving) = checked(handles)?; + Ok::<_, crate::object::Refusal>(move_handles(handles, &moving[..out.count], |batch| claim.commit(batch))) }); drop(claim); match moved { @@ -551,16 +549,16 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 /// go. A refusal restores every handle at its own number, and every mapping, /// so it is invisible. fn move_handles( - data: &mut process::ProcessData, + handles: &mut HandleTable, wanted: &[(RawHandle, Rights)], sink: impl FnOnce(Vec) -> Result<(), (Vec, E)>, ) -> (Result<(), E>, Vec) { let claims: Vec<_> = wanted .iter() - .filter_map(|(h, _)| data.handles.get::(*h, Rights::TRANSFER).ok()) + .filter_map(|(h, _)| handles.get::(*h, Rights::TRANSFER).ok()) .filter_map(|claim| claim.take_mappings().map(|handed| (claim, handed))) .collect(); - let moved = data.handles.transfer(wanted, sink); + let moved = handles.transfer(wanted, sink); let handed = match moved { Ok(()) => claims.into_iter().map(|(_, handed)| handed).collect(), Err(_) => { @@ -589,12 +587,12 @@ fn recv(conn_h: RawHandle, into: &mut Incoming<'_>, nonblock: bool) -> u64 { loop { // Taken, and its handles installed, in the hold that looks the // connection up; its bytes are copied out after. - let looked = process::with_process_data(|data| { - let rx = data.handles.get::(conn_h, Rights::READ)?.rx().clone(); + let looked = process::with_handles(|handles| { + let rx = handles.get::(conn_h, Rights::READ)?.rx().clone(); let taken = loop { - match rx.recv_now(&mut data.handles, buf, cap, &mut bounce) { + match rx.recv_now(handles, buf, cap, &mut bounce) { // Grown with the queue's lock let go, then asked again. - Err(Unclaimed::Room(n)) if data.handles.make_room(n) => {} + Err(Unclaimed::Room(n)) if handles.make_room(n) => {} Err(Unclaimed::Room(_)) => break Err(Unclaimed::Untaken(kernel::message::Untaken::NoRoom)), taken => break taken, } @@ -643,12 +641,12 @@ fn delivered(rx: &QueueRef, into: &mut Incoming<'_>, head: kernel::message::Head /// A message taken whose reader's buffers faulted: its bytes are lost, and /// its handles, installed at numbers the reader was never told, are closed. fn undelivered(rx: &QueueRef, installed: &[RawHandle], faulted: crate::user_ptr::Faulted) -> u64 { - process::with_process_data(|data| { + process::with_handles(|handles| { for &h in installed { // A sibling may have closed the number since, or reused it for a // handle of its own, which this then closes: the race is the // misbehaving process's own, and stays inside it. - let _ = ops::close(&mut data.handles, h); + let _ = ops::close(handles, h); } }); rx.write_watch().post(); @@ -666,14 +664,14 @@ pub(super) fn sys_inbox_setup(depth: u32, out: &mut crate::user_ptr::UserBytesMu Ok(object) => KObjectRef::Inbox(object), Err(e) => return SyscallError::from(e).to_u64(), }; - let installed = process::with_process_data(|data| ops::install(&mut data.handles, object)); + let installed = process::with_handles(|handles| ops::install(handles, object)); let handle = match installed { Ok(h) => h, Err(e) => return e.to_u64(), }; // A caller that cannot be told its handle does not keep it. if let Err(faulted) = out.write_at(0, toyos_abi::usersafe::bytes(&InboxSetup { handle, _pad: 0, vaddr })) { - let _ = process::with_process_data(|data| ops::close(&mut data.handles, handle)); + let _ = process::with_handles(|handles| ops::close(handles, handle)); return SyscallError::from(faulted).to_u64(); } 0 @@ -686,8 +684,8 @@ pub(super) fn sys_inbox_submit( timeout_nanos: u64, ) -> u64 { // NotFound and PermissionDenied are the table's own words for gone/wrong-type, not collapsed into InvalidArgument. - let object = process::with_process_data(|data| { - data.handles.get::( + let object = process::with_handles(|handles| { + handles.get::( inbox_h, Rights::READ.union(Rights::WRITE), ) diff --git a/kernel/src/syscall/machine.rs b/kernel/src/syscall/machine.rs index 8f9f540676b..c08b586fa7b 100644 --- a/kernel/src/syscall/machine.rs +++ b/kernel/src/syscall/machine.rs @@ -373,8 +373,8 @@ pub(super) fn sys_device_set(syscap: RawHandle) -> u64 { Ok(set) => set, Err(e) => return SyscallError::from(e).to_u64(), }; - process::with_process_data(|data| { - super::handles::handle_result(crate::object::ops::install(&mut data.handles, crate::object::KObjectRef::DeviceSet(set))) + process::with_handles(|handles| { + super::handles::handle_result(crate::object::ops::install(handles, crate::object::KObjectRef::DeviceSet(set))) }) } @@ -384,11 +384,11 @@ pub(super) fn sys_device_set(syscap: RawHandle) -> u64 { /// seen its generation. /// /// **What a slot is owed is acted on here, on the reader's thread** -/// (`pcidev::snapshot`), outside every process lock. +/// (`pcidev::snapshot`), outside every handle-table lock. pub(super) fn sys_device_set_read(set: RawHandle, out: &mut UserBytesMut) -> u64 { use toyos_abi::devset::{self, Header}; use toyos_abi::inventory::RECORD_BYTES; - let set = match process::with_process_data(|data| data.handles.get::(set, Rights::READ)) { + let set = match process::with_handles(|handles| handles.get::(set, Rights::READ)) { Ok(set) => set, Err(e) => return e.refuse(), }; diff --git a/kernel/src/syscall/paged.rs b/kernel/src/syscall/paged.rs index bf05a078fe9..2c459417e0a 100644 --- a/kernel/src/syscall/paged.rs +++ b/kernel/src/syscall/paged.rs @@ -11,41 +11,32 @@ use toyos_abi::handle::{RawHandle, Rights}; use toyos_abi::syscall::{PagedArgs, PagedOp, SyscallError, PAGED_DIRTY, PAGED_RANGE_PAGES, PAGED_WRITTEN}; use crate::object::paged::{PagedHolderObject, PagerObject, Unmade}; -use crate::object::{ops, KObjectRef}; -use crate::process::{self, PagerRole}; +use crate::object::{ops, HandleError, HandleTable, KObjectRef}; +use crate::process; use crate::user_ptr::{SyscallContext, UserBytesMut}; use crate::UserAddr; pub(super) fn sys_pager_create(capacity: u64, dirty: u64, objects: u64, report_ns: u64) -> u64 { - let claimed = process::with_process_data(|data| match data.pager { - PagerRole::May => { - data.pager = PagerRole::Made; - Ok(()) - } - PagerRole::Made => Err(SyscallError::AlreadyExists), - PagerRole::None => Err(SyscallError::PermissionDenied), - }); - if let Err(e) = claimed { + let handles = process::handles(); + if let Err(e) = handles.claim_pager() { return e.to_u64(); } let made = PagerObject::create(capacity, dirty, objects, report_ns).map_err(|why| match why { Unmade::Invalid => SyscallError::InvalidArgument, Unmade::Exhausted => SyscallError::ResourceExhausted, }); - let installed = made.and_then(|pager| { - process::with_process_data(|data| ops::install(&mut data.handles, KObjectRef::Pager(pager))) - }); + let installed = made.and_then(|pager| ops::install(&mut handles.lock(), KObjectRef::Pager(pager))); match installed { Ok(h) => u64::from(h.0), Err(e) => { - process::with_process_data(|data| data.pager = PagerRole::May); + handles.unclaim_pager(); e.to_u64() } } } fn pager(h: RawHandle, rights: Rights) -> Result, u64> { - process::with_process_data(|data| data.handles.get::(h, rights)).map_err(|e| e.refuse()) + process::with_handles(|handles| handles.get::(h, rights)).map_err(|e| e.refuse()) } pub(super) fn sys_paged_create(h: RawHandle, size: u64, anonymous: bool) -> u64 { @@ -65,8 +56,8 @@ pub(super) fn sys_paged_holder(h: RawHandle, object: u64) -> u64 { Err(e) => return e.to_u64(), }; // Counted under the same hold as its install: no close can come between. - let installed = process::with_process_data(|data| { - let h = ops::install(&mut data.handles, KObjectRef::Paged(Arc::clone(&holder)))?; + let installed = process::with_handles(|handles| { + let h = ops::install(handles, KObjectRef::Paged(Arc::clone(&holder)))?; holder.held(); Ok::<_, SyscallError>(h) }); @@ -87,21 +78,29 @@ pub(super) fn sys_paged_read(h: RawHandle, out: &mut UserBytesMut, offset: u64) } fn holder(h: RawHandle) -> Result, u64> { - let resolved = process::with_process_data(|data| { - let KObjectRef::Paged(holder) = data.handles.get_ref(h, Rights::NONE)? else { - return Ok(Err(SyscallError::InvalidArgument)); - }; - let holder = Arc::clone(holder); - if !data.handles.rights_of(h)?.contains(Rights::READ) { - return Ok(Err(SyscallError::PermissionDenied)); - } - Ok(if data.pager.is_pager() { Err(SyscallError::PermissionDenied) } else { Ok(holder) }) - }); + let handles = process::handles(); + let resolved = readable_holder(&handles.lock(), h, handles.is_pager()); match resolved { Ok(Ok(holder)) => Ok(holder), Ok(Err(e)) => Err(e.to_u64()), - Err(e) => Err(crate::object::HandleError::refuse(e)), + Err(e) => Err(HandleError::refuse(e)), + } +} + +/// The holder `h` names in `table`, if it carries `READ` and its process is +/// no `pager`. +fn readable_holder( + table: &HandleTable, + h: RawHandle, + pager: bool, +) -> Result, SyscallError>, HandleError> { + let KObjectRef::Paged(holder) = table.get_ref(h, Rights::NONE)? else { + return Ok(Err(SyscallError::InvalidArgument)); + }; + if !table.rights_of(h)?.contains(Rights::READ) || pager { + return Ok(Err(SyscallError::PermissionDenied)); } + Ok(Ok(Arc::clone(holder))) } pub(super) fn sys_paged_op(ctx: &SyscallContext<'_>, h: RawHandle, op: u64, args: UserAddr) -> u64 { diff --git a/kernel/src/syscall/proc.rs b/kernel/src/syscall/proc.rs index ede3437188e..f9084def278 100644 --- a/kernel/src/syscall/proc.rs +++ b/kernel/src/syscall/proc.rs @@ -59,8 +59,8 @@ pub(super) fn spawn_place(place: u64) -> Result { let Ok(raw) = u32::try_from(place) else { return Err(SyscallError::InvalidArgument.to_u64()); }; - process::with_process_data(|data| { - data.handles.get_sent::(RawHandle(raw), Rights::WRITE) + process::with_handles(|handles| { + handles.get_sent::(RawHandle(raw), Rights::WRITE) }) .map(|object| process::Parent::Under(object.pid())) .map_err(|refused| refused.refuse()) @@ -84,8 +84,8 @@ pub(super) fn spawn_program(ctx: &crate::user_ptr::SyscallContext<'_>, args: &Sp handle if handle == u64::from(toyos_abi::HANDLE_INVALID.0) => None, handle => { let raw_handle = u32::try_from(handle).map_err(|_| invalid())?; - let object = process::with_process_data(|data| { - data.handles.get::(RawHandle(raw_handle), Rights::MAP) + let object = process::with_handles(|handles| { + handles.get::(RawHandle(raw_handle), Rights::MAP) }); Some(object.map_err(|refused| refused.refuse())?) } @@ -134,8 +134,8 @@ pub(super) fn sys_process_wait(h: RawHandle, flags: u64) -> u64 { if flags & !PROCESS_WAIT_FLAGS_KNOWN != 0 { return SyscallError::InvalidArgument.to_u64(); } - let object = match process::with_process_data(|data| { - data.handles.get::(h, Rights::WAIT) + let object = match process::with_handles(|handles| { + handles.get::(h, Rights::WAIT) }) { Ok(object) => object, Err(e) => return e.refuse(), @@ -169,8 +169,8 @@ pub(super) fn sys_process_quit(h: RawHandle, reason: u64) -> u64 { let Some(reason) = u8::try_from(reason).ok().and_then(toyos_abi::quit::Reason::from_byte) else { return SyscallError::InvalidArgument.to_u64(); }; - match process::with_process_data(|data| { - data.handles.get::(h, Rights::MANAGE) + match process::with_handles(|handles| { + handles.get::(h, Rights::MANAGE) }) { Ok(object) => { process::quit_process(&object, reason); @@ -203,7 +203,7 @@ pub(super) fn sys_wake_latency_hold(syscap: RawHandle, bound_ns: u64) -> u64 { Ok(object) => crate::object::KObjectRef::WakeLatency(object), Err(e) => return SyscallError::from(e).to_u64(), }; - match process::with_process_data(|data| crate::object::ops::install(&mut data.handles, object)) { + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -218,7 +218,7 @@ pub(super) fn sys_job_create(syscap: RawHandle) -> u64 { Ok(object) => crate::object::KObjectRef::Job(object), Err(e) => return SyscallError::from(e).to_u64(), }; - match process::with_process_data(|data| crate::object::ops::install(&mut data.handles, object)) { + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -226,11 +226,12 @@ pub(super) fn sys_job_create(syscap: RawHandle) -> u64 { /// Install a handle to the job the caller runs in. pub(super) fn sys_job_own() -> u64 { - let installed = process::with_process_data(|data| { - let object = crate::object::KObjectRef::Job(crate::object::job::JobObject::new(data.job.clone())?); - crate::object::ops::install(&mut data.handles, object) - }); - match installed { + let job = process::with_process_data(|data| data.job.clone()); + let object = match crate::object::job::JobObject::new(job) { + Ok(object) => crate::object::KObjectRef::Job(object), + Err(e) => return SyscallError::from(e).to_u64(), + }; + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -372,8 +373,8 @@ pub(super) fn sys_process_stats( h: RawHandle, out: UserAddr, ) -> u64 { - let object = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let object = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(object) => object, Err(e) => return e.refuse(), @@ -393,7 +394,7 @@ pub(super) fn sys_job_floor(syscap: RawHandle, job: RawHandle, bytes: u64) -> u6 if let Err(e) = demand_syscap(syscap, Rights::FLOOR) { return e.refuse(); } - let object = match process::with_process_data(|data| data.handles.get::(job, Rights::WRITE)) { + let object = match process::with_handles(|handles| handles.get::(job, Rights::WRITE)) { Ok(object) => object, Err(e) => return e.refuse(), }; @@ -405,7 +406,7 @@ pub(super) fn sys_job_floor(syscap: RawHandle, job: RawHandle, bytes: u64) -> u6 /// Copy out what the account of the job a handle names holds, gated on its `READ`. pub(super) fn sys_account_info(ctx: &crate::user_ptr::SyscallContext, job: RawHandle, out: UserAddr) -> u64 { - let object = match process::with_process_data(|data| data.handles.get::(job, Rights::READ)) { + let object = match process::with_handles(|handles| handles.get::(job, Rights::READ)) { Ok(object) => object, Err(e) => return e.refuse(), }; From 9619ad214bcdefb5d7f01d932d2d72878bf6357d Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 21:59:16 +0200 Subject: [PATCH 2/2] issues: the handle table's whole-boot rows and A/B are owed on metal Both benches are down. #1246 lands on its review, host checks and guest rows, and `issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md`, the record of the served-file reads it targets, records what only metal can establish: the T14's whole `testcases`, `testcases-watchdog` and `shared-debug` boots of main after it, and the A/B of main before and after it on both benches with the Linux comparison. Its speed-up is not yet measured. Co-Authored-By: Claude Opus 5.5 --- ...-call-costs-toyos-up-to-35-times-linuxs.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md b/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md index e1f825c669c..8fcd562b8cb 100644 --- a/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md +++ b/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md @@ -78,6 +78,33 @@ sequential reads and 1 MiB writes ahead, and the 4 KiB and 8-byte reads unresolved against Linux's HPET-widened spread. `seek`, which the client answers itself, is 248 ns against Linux's 116 ns on the T14. +#1246 takes the process table and the process's data lock off every +handle-taking call: a served file's read (`SYS_PAGED_READ`) looks its holder +up under the handle table's own lock, read off the calling thread, where +pipeperf had measured the lookup at 60 ns on the T14 and 33 ns on the AMD +bench. Its speed-up is not yet measured on metal. + +- **Owed on metal:** the T14's whole `testcases`, `testcases-watchdog` and + `shared-debug` boots of main after #1246. Main must pass every member, + among them `handle_lifetime`, `abuse_handle_table`, `abuse_thread_table`, + `messages_are_whole`, `stream_handles`, `spawn_lands_claimed`, + `spawn_child_ends_first`, `spawn_returns_the_census`, `endowment_denied`, + `a_served_file_reads_through_its_paged_object`, + `a_pager_s_end_answers_its_waiting_holders_gone`, `std_threads`, + `inbox_cancel_wakes` and `port_badge`. +- **Owed on metal:** an A/B of main before and after #1246, on the T14 and + the AMD bench: `osbench files`, `ipc`, `time` and `sched`, four boots a + side in rotated order, each boot's p50, compared by an exact two-sided + Mann-Whitney test; and Linux's `osbench files` and `ipc` on each bench. + The claimed rows are `fs.mem.pread_8`, `fs.mem.pread_4k`, + `fs.mem.direct.pread_4k`, `fs.mem.read_4k`, `fs.mem.read_4k_at`, + `fs.mem.direct.read_4k_at`, `ipc.reqrep.8` and `ipc.reqrep_handle.8`: + main after reads each lower or not separated, with how far each moved set + against the 60 ns (T14) and 33 ns (AMD) lookup, and no other row 5% or + more worse unless the code #1246 changed explains it. A missing gain does + not reject the change; it decides whether a lock-free handle read is worth + building. + **Exit**: every `fs.mem` row of the second table within Linux's on both benches in a labelled current-state comparison, or each remaining difference measured to its parts and justified.