diff --git a/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md b/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md index e1f825c669c..8fcd562b8cb 100644 --- a/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md +++ b/issues/a-memory-filesystem-call-costs-toyos-up-to-35-times-linuxs.md @@ -78,6 +78,33 @@ sequential reads and 1 MiB writes ahead, and the 4 KiB and 8-byte reads unresolved against Linux's HPET-widened spread. `seek`, which the client answers itself, is 248 ns against Linux's 116 ns on the T14. +#1246 takes the process table and the process's data lock off every +handle-taking call: a served file's read (`SYS_PAGED_READ`) looks its holder +up under the handle table's own lock, read off the calling thread, where +pipeperf had measured the lookup at 60 ns on the T14 and 33 ns on the AMD +bench. Its speed-up is not yet measured on metal. + +- **Owed on metal:** the T14's whole `testcases`, `testcases-watchdog` and + `shared-debug` boots of main after #1246. Main must pass every member, + among them `handle_lifetime`, `abuse_handle_table`, `abuse_thread_table`, + `messages_are_whole`, `stream_handles`, `spawn_lands_claimed`, + `spawn_child_ends_first`, `spawn_returns_the_census`, `endowment_denied`, + `a_served_file_reads_through_its_paged_object`, + `a_pager_s_end_answers_its_waiting_holders_gone`, `std_threads`, + `inbox_cancel_wakes` and `port_badge`. +- **Owed on metal:** an A/B of main before and after #1246, on the T14 and + the AMD bench: `osbench files`, `ipc`, `time` and `sched`, four boots a + side in rotated order, each boot's p50, compared by an exact two-sided + Mann-Whitney test; and Linux's `osbench files` and `ipc` on each bench. + The claimed rows are `fs.mem.pread_8`, `fs.mem.pread_4k`, + `fs.mem.direct.pread_4k`, `fs.mem.read_4k`, `fs.mem.read_4k_at`, + `fs.mem.direct.read_4k_at`, `ipc.reqrep.8` and `ipc.reqrep_handle.8`: + main after reads each lower or not separated, with how far each moved set + against the 60 ns (T14) and 33 ns (AMD) lookup, and no other row 5% or + more worse unless the code #1246 changed explains it. A missing gain does + not reject the change; it decides whether a lock-free handle read is worth + building. + **Exit**: every `fs.mem` row of the second table within Linux's on both benches in a labelled current-state comparison, or each remaining difference measured to its parts and justified. diff --git a/kernel/CLAUDE.md b/kernel/CLAUDE.md index c4d084ea941..d18d11af32a 100644 --- a/kernel/CLAUDE.md +++ b/kernel/CLAUDE.md @@ -9,7 +9,7 @@ The module header at the site owns its subsystem — read it before changing a m - **The idle loop may not take a global lock unconditionally** — the crash report reads the process table through a `try_lock` it must never block on. `sched/reap_gate.rs` is the pattern: a relaxed-load gate in front of the lock. - **Every x86 Ring 0 entry clears the direction flag, and every interrupt or trap gate `AC`** — no hardware gate does either; `memmove` sets `DF` across `rep` operations with interrupts on, and Ring 3 sets `AC`, under which SMAP lets Ring 0 reach user pages. `arch::entry::ring3_naked_asm` is where they live; `SYSCALL` clears `AC` through `IA32_FMASK`. - **`BackendGuard` masks interrupts for its whole life**, so anything written under it is an interrupt latency; a new holder bounds itself as the console drain does. -- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_process_data` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern. +- **A syscall that can block resolves its handle and clones the object out before it blocks** — a `with_object`/`with_handles` guard held across a park is a runtime panic no compile check catches; the paged-object reads in `syscall/paged.rs` are the pattern. - **A decision the process table makes lives in `kernel/proclife/`, never in `process.rs`** — its defects are interleavings and its host tests are the only machine that can enumerate one. - **A task holds at most one watch registration** — a standing registration across a loop must not call anything that registers again. A double registration panics only at attempt ≥ 2, so the contention depth is the coverage. - **`ops::close` cancels a poll only for a source its object really ends** — `Watch::cancel_polls` answers every ring's poll on that watch; `ops::close_ends_polls` is where a new object kind answers. diff --git a/kernel/src/inbox/mod.rs b/kernel/src/inbox/mod.rs index 6b7e1474b52..d8cb31a3edd 100644 --- a/kernel/src/inbox/mod.rs +++ b/kernel/src/inbox/mod.rs @@ -531,9 +531,9 @@ fn process_watch(inbox: &Arc, submission: &Submission) { } /// The object `handle` names in this process's table, not the thread's: a ring is process-wide. -/// Cloned out, so what follows holds no process lock. +/// Cloned out, so what follows holds no handle-table lock. fn resolve(handle: RawHandle) -> Result { - process::with_process_data(|data| data.handles.get_ref(handle, Rights::WAIT).cloned()) + process::with_handles(|handles| handles.get_ref(handle, Rights::WAIT).cloned()) } /// Keep `poll` as its handle's one poll, or in the place of `looked_at`, the @@ -663,13 +663,13 @@ impl Submitter> for Arc { fn process_accept(inbox: &Arc, submission: &Submission) { let user_data = submission.token; - let acceptor = process::with_process_data(|data| { - data.handles.get::(submission.handle, Rights::READ) + let acceptor = process::with_handles(|handles| { + handles.get::(submission.handle, Rights::READ) }); let acceptor = match acceptor { Ok(a) => a, - // Nothing held: `with_process_data` has given the guard up. + // Nothing held: `with_handles` has given the guard up. Err(e) => { let refusal = e.refuse_as_error(); polls::complete(inbox, user_data, -(refusal as i32)); @@ -687,8 +687,8 @@ fn process_accept(inbox: &Arc, submission: &Submission) { }; match acceptor.pop() { Some(conn) => { - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn))) + let installed = process::with_handles(|handles| { + ops::install(handles, KObjectRef::Connection(crate::object::service::ConnectionEnd::accepted(core, conn))) }); match installed { Ok(h) => polls::complete(inbox, user_data, h.0 as i32), diff --git a/kernel/src/inventory.rs b/kernel/src/inventory.rs index ec2e0a3878e..50aca43c139 100644 --- a/kernel/src/inventory.rs +++ b/kernel/src/inventory.rs @@ -35,8 +35,8 @@ fn claims() -> Vec { }; let mut out = Vec::new(); for (pid, name, data) in processes { - let data = data.lock(); - for (_, entry) in data.handles.iter() { + let handles = Arc::clone(&data.lock().handles); + for (_, entry) in handles.lock().iter() { let KObjectRef::Device(claim) = entry.object() else { continue }; let on = if claim.class() == toyos_abi::syscall::DeviceType::PciFunction { // A function the last handle has let go names nothing. diff --git a/kernel/src/loader/mod.rs b/kernel/src/loader/mod.rs index 2b0d60c186e..0a1edf8d15a 100644 --- a/kernel/src/loader/mod.rs +++ b/kernel/src/loader/mod.rs @@ -135,7 +135,7 @@ pub fn spawn( parent: Parent, ) -> Result<(Pid, C::Held), crate::object::Refusal> { let job = Arc::clone(commit.job()); - let pager = if commit.pager() { crate::process::PagerRole::May } else { crate::process::PagerRole::None }; + let pager = commit.pager(); // Before anything is built; dropped on every way out below but the insert. let admission = Admission::ask(parent)?; let charge = ProcessData::charge(&job, path).map_err(SyscallError::from)?; @@ -189,15 +189,15 @@ pub fn spawn( // The point of no return: every failure above answers the caller with its // table untouched. let (handles, endowments, held) = commit.commit(start::Own::new(&object, ¬ice))?; + let handles = crate::process::Handles::new(handles, pager); let proc_data = Arc::new(Lock::new(ProcessData { - handles, + handles: Arc::clone(&handles), space: Arc::clone(&child_pt), spawn_ns: crate::clock::nanos_since_boot(), accounting: ProcessAccounting::default(), endowments, job: job.clone(), _charge: charge, - pager, })); let thread_data = Arc::new(Lock::new(ThreadData { @@ -231,6 +231,7 @@ pub fn spawn( thread_pointer: 0, job, image, + handles, // The spawning thread's, as a new thread's is. mask: crate::sched::driver::current_mask(), near: None, diff --git a/kernel/src/loader/start.rs b/kernel/src/loader/start.rs index 9faf8530c00..a9114523dff 100644 --- a/kernel/src/loader/start.rs +++ b/kernel/src/loader/start.rs @@ -14,7 +14,7 @@ use crate::object::notice::NoticeObject; use crate::object::process::ProcessObject; use crate::object::{ops, HandleEntry, HandleTable, KObjectRef, Refusal}; use crate::process::{ - process_data, Endowments, KERNEL_STACK_SIZE, + handles, process_data, with_handles, Endowments, KERNEL_STACK_SIZE, }; use crate::user_ptr::UserBytes; use toyos_abi::handle::{RawHandle, Rights}; @@ -114,7 +114,7 @@ pub trait Commit { /// The job the child runs in, which everything built for it is counted to. fn job(&self) -> &Arc; - /// Whether the child starts as a pager (`process::PagerRole`). + /// Whether the child starts as a pager (`process::Handles::is_pager`). fn pager(&self) -> bool; /// The child's table around `own`, what the kernel puts beside its @@ -138,8 +138,9 @@ impl Commit for PendingHandles { /// last answer. The handle under [`JOB_LABEL`] still names [`job`](Self::job), or the spawn is refused. fn commit(self, own: Own) -> Result<(HandleTable, Endowments, RawHandle), Refusal> { let Self { mut table, endow, mut labels, job, .. } = self; - let data_arc = process_data(); - let mut data = data_arc.lock(); + let own_job = process_data().lock().job.clone(); + let caller = handles(); + let mut handles = caller.lock(); let mut moving: Vec<(EndowEntry, RawHandle, Rights)> = Vec::with_capacity(endow.len()); let mut endowed_job: Option> = None; @@ -156,10 +157,10 @@ impl Commit for PendingHandles { if endowed_job.is_some() { return Err(SyscallError::InvalidArgument.into()); } - endowed_job = Some(data.handles.get_sent::(handle, Rights::WRITE)?.job().clone()); + endowed_job = Some(handles.get_sent::(handle, Rights::WRITE)?.job().clone()); } // Checked before any removal, so a missing `TRANSFER` refuses the spawn instead of leaving a hole. - let rights = data.handles.rights_to_move(handle, want)?; + let rights = handles.rights_to_move(handle, want)?; // A duplicate is refused here: unremoved handles let it pass twice, then the second removal's // `expect` below panics. if moving.iter().any(|(_, seen, _)| *seen == handle) { @@ -169,11 +170,11 @@ impl Commit for PendingHandles { } // Before any removal: the table changed under the spawn if the job // everything was built for is no longer the one it names. - if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&data.job)) { + if !Arc::ptr_eq(&job, endowed_job.as_ref().unwrap_or(&own_job)) { return Err(SyscallError::InvalidArgument.into()); } // Checked before any removal, so a failed install can't strand a handle out of a table that never spawned. - if !table.make_room(moving.len() + Own::LABELS.len()) || !data.handles.make_room(1) { + if !table.make_room(moving.len() + Own::LABELS.len()) || !handles.make_room(1) { return Err(SyscallError::ResourceExhausted.into()); } let mut entries = Vec::with_capacity(moving.len()); @@ -181,8 +182,7 @@ impl Commit for PendingHandles { // it under the caller's lock and revoked once that lock is let go. let mut handed = Vec::new(); for (mut entry, parent_handle, rights) in moving { - let moved = data - .handles + let moved = handles .remove(parent_handle) .expect("an endowed handle verified under this lock stopped resolving") .arriving_with(rights); @@ -195,9 +195,9 @@ impl Commit for PendingHandles { entries.push(entry); } // Minted while `own` is held, and `own` goes into a table no thread reaches until the child lands: another thread of the caller closing this handle never closes the object's last. - let held = ops::install(&mut data.handles, own.process.object().clone()) + let held = ops::install(&mut handles, own.process.object().clone()) .expect("a caller's table with verified room refused its child"); - drop(data); + drop(handles); for handed in handed { handed.revoke(); } @@ -268,12 +268,11 @@ fn label_of(labels: &[u8], off: u32, len: u32) -> Option<&[u8]> { /// The job a child runs in: the one endowed under [`JOB_LABEL`], or else the /// caller's. Only looked up: [`PendingHandles::commit`] refuses the rest. fn child_job(endow: &[SpawnEndow], labels: &[u8]) -> Result, Refusal> { - let data_arc = process_data(); - let data = data_arc.lock(); for entry in endow { if label_of(labels, entry.label_off, entry.label_len) == Some(JOB_LABEL.as_bytes()) { - return Ok(data.handles.get_sent::(entry.moved.handle, Rights::WRITE)?.job().clone()); + let named = with_handles(|handles| handles.get_sent::(entry.moved.handle, Rights::WRITE))?; + return Ok(named.job().clone()); } } - Ok(data.job.clone()) + Ok(process_data().lock().job.clone()) } diff --git a/kernel/src/message.rs b/kernel/src/message.rs index 3764e6439b0..406a53ad28b 100644 --- a/kernel/src/message.rs +++ b/kernel/src/message.rs @@ -255,7 +255,7 @@ impl QueueRef { } /// Queue `src`, already copied in, and `batch`, its handles, whole in - /// this hold, called under the sender's process lock as its handles move; + /// this hold, called under the sender's handle-table lock as its handles move; /// a refusal hands the batch back and queues nothing. No copy waits on it. pub fn send_now(&self, src: &[u8], mut batch: Vec) -> Result<(), (Vec, Unsent)> { let mut state = self.0.state.lock(); @@ -280,7 +280,7 @@ impl QueueRef { /// Take the front message whole into `into` and install its handles in /// `table`, in this hold, where a reader holding `buf` bytes and `cap` - /// handles can take it, called under the receiver's process lock. A table + /// handles can take it, called under the receiver's handle-table lock. A table /// that must grow first answers [`Unclaimed::Room`], taking nothing. pub fn recv_now(&self, table: &mut HandleTable, buf: usize, cap: usize, into: &mut MessageBounce) -> Result { let mut state = self.0.state.lock(); diff --git a/kernel/src/object/handle.rs b/kernel/src/object/handle.rs index 651a39b64be..f598a6958ea 100644 --- a/kernel/src/object/handle.rs +++ b/kernel/src/object/handle.rs @@ -1,7 +1,7 @@ //! One process's handle table. //! -//! Lives behind `ProcessData`'s own lock; every accessor returns an owned -//! value, so no borrow into the table outlives the guard. +//! Lives behind its own lock (`process::Handles`); every accessor returns an +//! owned value, so no borrow into the table outlives the guard. //! //! Not a file-descriptor table — `fd` names only `sdk/libc`'s POSIX //! surface (owner ruling) — and a slot at its last generation retires @@ -22,7 +22,7 @@ use crate::mm::charge::{self, Charge}; use super::{KObjectRef, KObjectVariant}; -/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the process's own lock, where [`HandleError::refuse`] must not be called. +/// The table has no slot left, or no memory to grow by one — its own type since `install` runs under the table's own lock, where [`HandleError::refuse`] must not be called. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub struct TableFull; diff --git a/kernel/src/object/notice.rs b/kernel/src/object/notice.rs index 27cd5a5425e..1506411fce4 100644 --- a/kernel/src/object/notice.rs +++ b/kernel/src/object/notice.rs @@ -19,8 +19,8 @@ use super::{KObjectVariant, ObjectCore}; pub struct NoticeObject { pub(super) core: ObjectCore, - /// A leaf: a quit takes it under `PROCESS_TABLE`, a read under the - /// reader's `ProcessData`, and nothing is taken under it. + /// A leaf: a quit takes it under `PROCESS_TABLE`, and nothing is taken + /// under it. state: Lock, /// Posted when a reason is put in it. watch: Arc, diff --git a/kernel/src/pipe.rs b/kernel/src/pipe.rs index 7c407f625f8..9581228fa6d 100644 --- a/kernel/src/pipe.rs +++ b/kernel/src/pipe.rs @@ -330,7 +330,7 @@ impl PipeRef { } /// Write `src`, already copied in, whole in one hold of the pipe's lock, - /// for a caller holding its process's lock: `None` where that cannot answer + /// for a caller holding its handle-table lock: `None` where that cannot answer /// it without a wait or frames taken, which [`Self::write`] does. pub fn put_now(&self, src: &[u8]) -> Option { let mut pipe = self.0.state.lock(); @@ -347,7 +347,7 @@ impl PipeRef { } /// Take at most `max` bytes, up to [`BOUNCE`], into `into` whole in one - /// hold of the pipe's lock, for a caller holding its process's lock: + /// hold of the pipe's lock, for a caller holding its handle-table lock: /// `None` where the pipe holds nothing or more than a bounce would take, /// or another read holds its claim, which [`Self::read`] answers. pub fn take_now(&self, max: usize, into: &mut PipeBounce) -> Option { diff --git a/kernel/src/process.rs b/kernel/src/process.rs index 01b96fef2db..91ea761899f 100644 --- a/kernel/src/process.rs +++ b/kernel/src/process.rs @@ -29,10 +29,11 @@ use alloc::collections::BTreeMap; use alloc::string::String; use alloc::sync::Arc; use alloc::vec::Vec; +use core::sync::atomic::{AtomicBool, Ordering}; use crate::arch::percpu; use crate::mm::charge::{self, Charge, Exhausted}; use crate::object::{ops, HandleTable}; -use crate::sync::Lock; +use crate::sync::{Lock, LockGuard}; use crate::sched::driver; use crate::sched::payload::ThreadSched; use kernel::sched::cpu::{homes, Near}; @@ -60,6 +61,51 @@ pub use crate::loader::{build_child_handles, spawn, spawn_supervisor}; /// A process's space, which every thread of it runs in. pub use crate::mm::space::PageTables; +/// What a process names kernel objects by. Each of its threads carries it, so +/// a lookup ([`with_handles`]) takes neither the process table nor +/// [`ProcessData`]'s lock. +pub struct Handles { + /// Every kernel object it can name; stdio is handles `0`, `1`, `2`. + table: Lock, + /// Started as a pager, which no call changes (`object::paged`): it may + /// make one pager object, and is refused a holder's read, which would wait + /// on itself or on another pager. + pager: bool, + /// It made that pager object. + made_pager: AtomicBool, +} + +impl Handles { + pub fn new(table: HandleTable, pager: bool) -> Arc { + Arc::new(Self { table: Lock::new(table), pager, made_pager: AtomicBool::new(false) }) + } + + pub fn lock(&self) -> LockGuard<'_, HandleTable> { + self.table.lock() + } + + pub fn is_pager(&self) -> bool { + self.pager + } + + /// Claim the one pager object a pager may make: `PermissionDenied` for a + /// process started as none, `AlreadyExists` once claimed. + pub fn claim_pager(&self) -> Result<(), SyscallError> { + if !self.pager { + return Err(SyscallError::PermissionDenied); + } + if self.made_pager.swap(true, Ordering::Relaxed) { + return Err(SyscallError::AlreadyExists); + } + Ok(()) + } + + /// Give the claim back: its pager object was never installed. + pub fn unclaim_pager(&self) { + self.made_pager.store(false, Ordering::Relaxed); + } +} + /// An object the kernel mapped into a process for it and takes back itself: an /// inbox's rings. The kernel writes what it owes the process into the object. pub struct KernelMapping { @@ -229,8 +275,9 @@ impl Processes for ProcessTable { /// Process-level data shared across all threads via `Arc>`. pub struct ProcessData { - /// Every kernel object this process can name; stdio is handles `0`, `1`, `2`. - pub handles: HandleTable, + /// What its threads carry, under a lock of its own; here for a reader that + /// is not one of them. + pub handles: Arc, /// The space every thread of it runs in. pub space: PageTables, @@ -243,25 +290,6 @@ pub struct ProcessData { pub job: Arc, /// What [`ProcessData::charge`] charged. pub _charge: Charge, - /// Whether it was started as a pager, which no call changes, and whether - /// it made its one pager object. - pub pager: PagerRole, -} - -/// A process's part in paged objects (`object::paged`): set at its spawn. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum PagerRole { - None, - /// Started as a pager; it has not made its pager object. - May, - /// It made it. - Made, -} - -impl PagerRole { - pub fn is_pager(self) -> bool { - self != PagerRole::None - } } impl ProcessData { @@ -275,6 +303,7 @@ impl ProcessData { // A thread table of one thread is a whole leaf. let threads = charge::heap(Layout::array::<(Tid, ThreadEntry)>(11).expect("process: a leaf's layout")); let records = charge::arc::>() + + charge::arc::() + charge::arc::() + charge::arc::() + charge::arc::() @@ -475,6 +504,17 @@ pub fn with_process_data(f: impl FnOnce(&mut ProcessData) -> R) -> R { f(&mut guard) } +/// The running thread's process's handles, off the thread itself. +pub fn handles() -> Arc { + scheduler::current_handles().expect("handles: no thread is running") +} + +/// `f` over the running thread's process's handle table, under the table's +/// own lock: the lookup every handle-taking call makes. +pub fn with_handles(f: impl FnOnce(&mut HandleTable) -> R) -> R { + f(&mut handles().lock()) +} + /// Whom a new process is placed under. #[derive(Clone, Copy)] pub enum Parent { @@ -573,6 +613,7 @@ pub fn spawn_thread( (addr_space, Arc::clone(&proc.process_data)) }; let job = process_data_arc.lock().job.clone(); + let handles = handles(); // Phase 2: the kernel stack, outside any lock. let (ks_alloc, ks_sp) = alloc_kernel_stack(&job, Start::Thread { entry, sp: stack_ptr, arg })?; @@ -609,6 +650,7 @@ pub fn spawn_thread( thread_pointer: 0, job, image, + handles, mask, near: Some(Near { spawner: driver::current_cpu(), homes }), }); @@ -664,11 +706,12 @@ fn teardown_resources( #[cfg(feature = "mask-windows")] crate::windows::report(); - let mut data = process_data_arc.lock(); - ops::close_all(&mut data.handles); + let handles = Arc::clone(&process_data_arc.lock().handles); + ops::close_all(&mut handles.lock()); // Every thread has left, and none returns to Ring 3 to use them. crate::isa::process_ends(pid); + let data = process_data_arc.lock(); let usage = data.space.usage(); Consumed { peak_memory: usage.peak_bytes(), diff --git a/kernel/src/sched/driver.rs b/kernel/src/sched/driver.rs index 1b3ce5d8324..98af68cb65a 100644 --- a/kernel/src/sched/driver.rs +++ b/kernel/src/sched/driver.rs @@ -397,6 +397,8 @@ pub struct NewTask { pub job: Arc, /// The process's image. pub image: Arc, + /// The process's handles. + pub handles: Arc, /// The CPUs it may run on. pub mask: CpuMask, /// For a thread spawned into its own process, its spawner and its @@ -430,6 +432,7 @@ pub fn spawn(new: NewTask) -> (ThreadSched, CpuId) { address_space: new.address_space, handle: handle.clone(), image: new.image, + handles: new.handles, }, rt: RtState::default(), mask: new.mask, @@ -781,6 +784,11 @@ pub fn current_address_space() -> Option { try_with_cpu(|cpu| cpu.running().map(|t| t.ext().address_space.clone())).flatten() } +/// The running task's process's handles. `None` off-task. +pub fn current_handles() -> Option> { + try_with_cpu(|cpu| cpu.running().map(|t| t.ext().handles.clone())).flatten() +} + pub fn with_current_acct( f: impl FnOnce(&kernel::sched::task::TaskAccounting) -> R, ) -> Option { diff --git a/kernel/src/sched/payload.rs b/kernel/src/sched/payload.rs index e1ef23909f7..8676856bae7 100644 --- a/kernel/src/sched/payload.rs +++ b/kernel/src/sched/payload.rs @@ -15,7 +15,7 @@ use kernel::sched::park::WaitTicket; use crate::watch::Watch; use crate::mm::paging::Root; use crate::loader::KernelStack; -use crate::process::{PageTables, ProcessAccounting, TaskId, UserImage}; +use crate::process::{Handles, PageTables, ProcessAccounting, TaskId, UserImage}; use crate::sync::Lock; /// The environment's leaf lock; holding it raises the preempt count, making a wake path a legal mailbox producer. @@ -82,6 +82,8 @@ pub struct KernelPayload { pub handle: Arc, /// This task's process's image, as a crash report records a frame in it; kept here, not looked up via the process table, so a crash report never takes that lock. pub image: Arc, + /// This task's process's handles, kept here for the same reason: a lookup takes no lock but the table's. + pub handles: Arc, } impl SchedPayload for KernelPayload { diff --git a/kernel/src/scheduler.rs b/kernel/src/scheduler.rs index 351d718fbe7..bc5765dbb1a 100644 --- a/kernel/src/scheduler.rs +++ b/kernel/src/scheduler.rs @@ -20,7 +20,7 @@ use crate::time::Deadline; use crate::DirectMap; pub use crate::sched::driver::{ - current_address_space, enter_idle_loop, in_pass as in_schedule_self, total_cpu_ns, + current_address_space, current_handles, enter_idle_loop, in_pass as in_schedule_self, total_cpu_ns, NewTask, Ticket, }; #[cfg(feature = "heap-tripwire")] diff --git a/kernel/src/syscall/device.rs b/kernel/src/syscall/device.rs index e7df3fe6948..a40ade251f3 100644 --- a/kernel/src/syscall/device.rs +++ b/kernel/src/syscall/device.rs @@ -27,8 +27,8 @@ pub(super) fn sys_device_reg_read(handle: RawHandle, offset: u64, width: u64) -> let Some(width) = toyos_abi::syscall::RegWidth::from_raw(width) else { return SyscallError::InvalidArgument.to_u64(); }; - let claim = process::with_process_data(|data| { - data.handles.get::(handle, Rights::NONE).map(|claim| { + let claim = process::with_handles(|handles| { + handles.get::(handle, Rights::NONE).map(|claim| { (claim.class() == device::DeviceType::PciFunction).then_some(claim) }) }); @@ -59,7 +59,7 @@ pub(super) fn sys_device_reg_read(handle: RawHandle, offset: u64, width: u64) -> /// `PermissionDenied`, as its ports answer nothing there. pub(super) fn sys_acpi(ctx: &SyscallContext, handle: RawHandle, op: u64, at: u64) -> u64 { use toyos_abi::acpi::op as ops; - let claim = match process::with_process_data(|data| data.handles.get::(handle, Rights::WRITE)) { + let claim = match process::with_handles(|handles| handles.get::(handle, Rights::WRITE)) { Ok(claim) if claim.class() == device::DeviceType::Acpi => claim, Ok(claim) => { let held = claim.class().class_name(); @@ -171,8 +171,8 @@ pub(super) fn sys_device_claim(syscap: RawHandle, class: u64, selector: [u64; 2] // Not yet, rather than never: the device set says when. Err(device::ClaimError::Undeclared) => return SyscallError::WouldBlock.to_u64(), }; - process::with_process_data(|data| { - handle_result(ops::install(&mut data.handles, KObjectRef::Device(claim))) + process::with_handles(|handles| { + handle_result(ops::install(handles, KObjectRef::Device(claim))) }) } @@ -186,8 +186,8 @@ pub(super) fn sys_device_claim(syscap: RawHandle, class: u64, selector: [u64; 2] /// **The claim, and nothing read out of it**: a call reaches its function only /// through [`on_function`], so what it acts on is this claim's or nothing. fn pci_claim(handle: RawHandle) -> Result, crate::object::Refusal> { - let claim = process::with_process_data(|data| { - data.handles.get::(handle, Rights::WRITE) + let claim = process::with_handles(|handles| { + handles.get::(handle, Rights::WRITE) })?; // A claim of another class is the wrong kind of thing here, not a missing // device: it names a device this call has no meaning for. @@ -236,12 +236,12 @@ fn install_handed( claim: &Arc, entry: crate::object::HandleEntry, ) -> Result { - process::with_process_data(|data| { - let held = data.handles.get::(handle, Rights::NONE).is_ok_and(|held| Arc::ptr_eq(&held, claim)); + process::with_handles(|handles| { + let held = handles.get::(handle, Rights::NONE).is_ok_and(|held| Arc::ptr_eq(&held, claim)); if !held { return Err(SyscallError::Gone); } - data.handles.install(entry).map_err(|_| SyscallError::ResourceExhausted) + handles.install(entry).map_err(|_| SyscallError::ResourceExhausted) }) } @@ -312,8 +312,8 @@ pub(super) fn sys_device_dma_alloc( Ok(grant) => grant, Err(e) => return e.to_u64(), }; - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, KObjectRef::SharedMem(alloc::sync::Arc::clone(&memory))) + let installed = process::with_handles(|handles| { + ops::install(handles, KObjectRef::SharedMem(alloc::sync::Arc::clone(&memory))) }); let shm = match installed { Ok(handle) => handle, @@ -348,8 +348,8 @@ pub(super) fn sys_device_dma_map( Ok(claim) => claim, Err(e) => return e.refuse(), }; - let memory = match process::with_process_data(|data| { - data.handles.get::(region, Rights::MAP.union(Rights::WRITE)) + let memory = match process::with_handles(|handles| { + handles.get::(region, Rights::MAP.union(Rights::WRITE)) }) { Ok(memory) => memory, Err(e) => { diff --git a/kernel/src/syscall/dispatch.rs b/kernel/src/syscall/dispatch.rs index dc95fea4245..cb6eeecdbf2 100644 --- a/kernel/src/syscall/dispatch.rs +++ b/kernel/src/syscall/dispatch.rs @@ -158,8 +158,8 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> } SYS_PROCESS_WAIT => sys_process_wait(RawHandle(a1 as u32), a2), SYS_PROCESS_KILL => { - match process::with_process_data(|data| { - data.handles + match process::with_handles(|handles| { + handles .get::(RawHandle(a1 as u32), Rights::MANAGE) }) { Ok(object) => process::kill_process(&object), @@ -441,7 +441,7 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> let Ok(slot) = u16::try_from(a2) else { return SyscallError::InvalidArgument.to_u64(); }; - match process::with_process_data(|d| d.handles.stage_last_generation(slot)) { + match process::with_handles(|handles| handles.stage_last_generation(slot)) { Some(h) => u64::from(h.0), None => SyscallError::InvalidArgument.to_u64(), } diff --git a/kernel/src/syscall/event.rs b/kernel/src/syscall/event.rs index 8dae451a94a..abba27528ad 100644 --- a/kernel/src/syscall/event.rs +++ b/kernel/src/syscall/event.rs @@ -25,7 +25,7 @@ pub(super) fn sys_event_create() -> u64 { } pub(super) fn sys_event_signal(h: RawHandle) -> u64 { - match process::with_process_data(|data| data.handles.get::(h, Rights::WRITE)) { + match process::with_handles(|handles| handles.get::(h, Rights::WRITE)) { Ok(event) if event.signal() => 0, Ok(_) => SyscallError::Gone.to_u64(), Err(e) => e.refuse(), @@ -35,7 +35,7 @@ pub(super) fn sys_event_signal(h: RawHandle) -> u64 { /// `1` for a signal taken, `WouldBlock` for none pending, `Gone` for none /// pending and no signaler left. Never blocks: a wait is `SYS_HANDLE_WAIT`. pub(super) fn sys_event_take(h: RawHandle) -> u64 { - match process::with_process_data(|data| data.handles.get::(h, Rights::READ)) { + match process::with_handles(|handles| handles.get::(h, Rights::READ)) { Ok(event) => match event.take() { Taken::Signal => 1, Taken::Empty => SyscallError::WouldBlock.to_u64(), @@ -55,7 +55,7 @@ pub(super) fn sys_handle_wait(h: RawHandle, interest: u64, timeout_nanos: u64) - Ok(bit @ (READABLE | WRITABLE | OTHER_END_GONE)) => bit, _ => return SyscallError::InvalidArgument.to_u64(), }; - let object = match process::with_process_data(|data| data.handles.get_ref(h, Rights::WAIT).cloned()) { + let object = match process::with_handles(|handles| handles.get_ref(h, Rights::WAIT).cloned()) { Ok(object) => object, Err(e) => return e.refuse(), }; diff --git a/kernel/src/syscall/handles.rs b/kernel/src/syscall/handles.rs index 8a78ebe74e8..822d2c7129e 100644 --- a/kernel/src/syscall/handles.rs +++ b/kernel/src/syscall/handles.rs @@ -11,19 +11,19 @@ use toyos_abi::syscall::*; /// Requires that `syscap` resolves to a `SysCap` holding `need`, and answers /// every right it holds, read under the same guard. pub(super) fn demand_syscap(syscap: RawHandle, need: Rights) -> Result { - process::with_process_data(|data| { - data.handles.get::(syscap, need)?; - data.handles.rights_of(syscap) + process::with_handles(|handles| { + handles.get::(syscap, need)?; + handles.rights_of(syscap) }) } -/// The object `h` names, cloned out of the process's lock: what a call that +/// The object `h` names, cloned out of the handle table's lock: what a call that /// copies to or from its caller, or may park, works on (`user_ptr`). pub(super) fn object(h: RawHandle, need: Rights) -> Result { - process::with_process_data(|data| data.handles.get_ref(h, need).cloned()) + process::with_handles(|handles| handles.get_ref(h, need).cloned()) } -/// `f` over the object `h` names, outside the process's lock, for the calls +/// `f` over the object `h` names, outside the handle table's lock, for the calls /// whose answer is already a raw syscall word. pub(super) fn with_object(h: RawHandle, need: Rights, f: impl FnOnce(&KObjectRef) -> u64) -> u64 { match object(h, need) { @@ -41,8 +41,8 @@ pub(super) fn handle_result(r: Result) -> u64 { /// Wakes nobody: `pipe::close_write`'s own zero-refcount path does that. pub(super) fn sys_close(h: RawHandle) -> u64 { - let result = process::with_process_data(|data| { - ops::close(&mut data.handles, h) + let result = process::with_handles(|handles| { + ops::close(handles, h) }); match result { Ok(()) => 0, @@ -56,15 +56,15 @@ pub(super) fn sys_close(h: RawHandle) -> u64 { /// /// `want` is the wire form of `Option` (`RIGHTS_UNCHANGED` = the source's own set), decoded here and nowhere else. pub(super) fn sys_handle_dup(h: RawHandle, want: u64) -> u64 { - let duplicated = process::with_process_data(|data| { - let held = data.handles.rights_of(h)?; + let duplicated = process::with_handles(|handles| { + let held = handles.rights_of(h)?; let rights = if want == RIGHTS_UNCHANGED { held } else { let bits = u32::try_from(want).map_err(|_| SyscallError::InvalidArgument)?; Rights::from_bits(bits).ok_or(SyscallError::InvalidArgument)? }; - Ok::<_, crate::object::Refusal>(data.handles.duplicate(h, rights)?) + Ok::<_, crate::object::Refusal>(handles.duplicate(h, rights)?) }); match duplicated { Ok(new_h) => new_h.0 as u64, diff --git a/kernel/src/syscall/io.rs b/kernel/src/syscall/io.rs index e7d3773066b..e4600eab541 100644 --- a/kernel/src/syscall/io.rs +++ b/kernel/src/syscall/io.rs @@ -1,6 +1,6 @@ //! Blocking `read`/`write` register on the object's watch before re-checking the //! condition — closing the check-then-block window. Each works on its object -//! cloned out of the process's lock, which no copy and no park runs under. +//! cloned out of the handle table's lock, which no copy and no park runs under. //! //! Buffers arrive already bounded: `super::dispatch` converts the caller's //! pointer and length into a [`UserBytes`]/[`UserBytesMut`] window before @@ -39,7 +39,7 @@ enum ReadBlock { } /// A write of at most `BOUNCE` bytes to a pipe, made whole in the hold of the -/// process's lock that looks its handle up, its bytes copied in before it, as +/// handle table's lock that looks its handle up, its bytes copied in before it, as /// a write was before any copy left the locks: `None` for anything else, which /// the general path answers (copying a small write to another object again). fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { @@ -49,8 +49,8 @@ fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { let mut bounce = PipeBounce::new(); // A buffer that faults is the general path's to answer, after its handle. let src = bounce.read(buf).ok()?.bytes(); - let (word, pipe) = process::with_process_data(|data| { - let KObjectRef::PipeWrite(w) = data.handles.get_ref(h, Rights::WRITE).ok()? else { return None }; + let (word, pipe) = process::with_handles(|handles| { + let KObjectRef::PipeWrite(w) = handles.get_ref(h, Rights::WRITE).ok()? else { return None }; let word = ops::pipe_written(w.pipe().put_now(src)?)?; Some((word, w.pipe().clone())) })?; @@ -59,12 +59,12 @@ fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { } /// A read of a pipe holding at most `BOUNCE` bytes for it, taken in the hold -/// of the process's lock that looks its handle up and copied out after it: +/// of the handle table's lock that looks its handle up and copied out after it: /// `None` for anything else, which the general path answers. fn pipe_read_now(h: RawHandle, buf: &mut UserBytesMut) -> Option { let mut bounce = PipeBounce::new(); - let (taken, pipe) = process::with_process_data(|data| { - let KObjectRef::PipeRead(r) = data.handles.get_ref(h, Rights::READ).ok()? else { return None }; + let (taken, pipe) = process::with_handles(|handles| { + let KObjectRef::PipeRead(r) = handles.get_ref(h, Rights::READ).ok()? else { return None }; Some((r.pipe().take_now(buf.len(), &mut bounce)?, r.pipe().clone())) })?; let n = taken.n; @@ -121,7 +121,7 @@ fn read_block(object: &KObjectRef) -> ReadBlock { } } -/// What a read's handle names, out of the process's lock: a device claim is +/// What a read's handle names, out of the handle table's lock: a device claim is /// read under it, since describing one installs handles, and its answer copied /// out after. enum Readable { @@ -130,11 +130,11 @@ enum Readable { } fn readable(h: RawHandle, room: usize) -> Result { - process::with_process_data(|data| { - let object = data.handles.get_ref(h, Rights::READ)?; + process::with_handles(|handles| { + let object = handles.get_ref(h, Rights::READ)?; let KObjectRef::Device(claim) = object else { return Ok(Readable::Object(object.clone())) }; let claim = alloc::sync::Arc::clone(claim); - Ok(Readable::Device(ops::read_device(&claim, &mut data.handles, room))) + Ok(Readable::Device(ops::read_device(&claim, handles, room))) }) } @@ -228,7 +228,7 @@ pub(super) fn sys_null(want: u64) -> u64 { Err(e) => return SyscallError::from(e).to_u64(), }; let entry = crate::object::HandleEntry::new(object, rights); - match process::with_process_data(|data| data.handles.install(entry)) { + match process::with_handles(|handles| handles.install(entry)) { Ok(h) => h.0 as u64, Err(_) => SyscallError::ResourceExhausted.to_u64(), } diff --git a/kernel/src/syscall/ipc.rs b/kernel/src/syscall/ipc.rs index 7ba0cb86cf6..09914f3a039 100644 --- a/kernel/src/syscall/ipc.rs +++ b/kernel/src/syscall/ipc.rs @@ -12,7 +12,7 @@ use alloc::vec::Vec; use crate::watch; use crate::object::{ops, port, KObjectRef}; use crate::message::{MessageBounce, QueueRef, SendRefused, Unclaimed, Unsent}; -use crate::object::handle::HandleEntry; +use crate::object::handle::{HandleEntry, HandleTable}; use crate::object::service::ConnectionEnd; use crate::time::Deadline; use crate::user_ptr::SyscallContext; @@ -40,13 +40,13 @@ pub(super) fn sys_pipe() -> u64 { /// Install two objects or neither; the two handles come back packed into one word, which cannot be read as an error. pub(super) fn install_pair(first: KObjectRef, second: KObjectRef) -> u64 { - let installed = process::with_process_data(|data| { + let installed = process::with_handles(|handles| { // Room for both before either: a refused pair installs nothing, so nothing is taken back. - if !data.handles.make_room(2) { + if !handles.make_room(2) { return Err((first, second)); } let mut install = - |object| ops::install(&mut data.handles, object).expect("room was asked for first"); + |object| ops::install(handles, object).expect("room was asked for first"); let first = install(first); let second = install(second); Ok(((first.0 as u64) << 32) | second.0 as u64) @@ -97,8 +97,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - Vec::new(); if keep_all { - let base = match process::with_process_data(|data| { - data.handles.get::(args.base, Rights::READ) + let base = match process::with_handles(|handles| { + handles.get::(args.base, Rights::READ) }) { Ok(base) => base, Err(e) => return e.refuse(), @@ -115,8 +115,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - else { return SyscallError::BadAddress.to_u64(); }; - let base = match process::with_process_data(|data| { - data.handles.get::(args.base, Rights::READ) + let base = match process::with_handles(|handles| { + handles.get::(args.base, Rights::READ) }) { Ok(base) => base, Err(e) => return e.refuse(), @@ -156,8 +156,8 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - let Some(name) = name_at(off, len) else { return SyscallError::InvalidArgument.to_u64(); }; - let connector = match process::with_process_data(|data| { - data.handles.get_sent::(handle, Rights::TRANSFER) + let connector = match process::with_handles(|handles| { + handles.get_sent::(handle, Rights::TRANSFER) }) { Ok(c) => c, Err(e) => return e.refuse(), @@ -178,17 +178,15 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - return SyscallError::ResourceExhausted.to_u64() } }; - process::with_process_data(|data| { - handle_result(ops::install(&mut data.handles, KObjectRef::Namespace(namespace))) + process::with_handles(|handles| { + handle_result(ops::install(handles, KObjectRef::Namespace(namespace))) }) } /// Open a connection to `name` in the namespace `ns_h` holds; `NotFound` means the namespace lacks the name, `Gone` means its port has closed. pub(super) fn sys_namespace_open(ns_h: RawHandle, name: &str) -> u64 { - let connector = match process::with_process_data(|data| { - let ns = data - .handles - .get::(ns_h, Rights::READ)?; + let connector = match process::with_handles(|handles| { + let ns = handles.get::(ns_h, Rights::READ)?; Ok::<_, crate::object::HandleError>(ns.lookup(name).cloned()) }) { Ok(Some(c)) => c, @@ -200,10 +198,10 @@ pub(super) fn sys_namespace_open(ns_h: RawHandle, name: &str) -> u64 { /// Inspect only the names this namespace already holds. The Arc keeps the /// immutable entries alive after the handle-table lock is released; copying -/// into user memory therefore holds no object or process lock. +/// into user memory therefore holds no object or handle-table lock. pub(super) fn sys_namespace_names(ns_h: RawHandle, out: &mut crate::user_ptr::UserBytesMut) -> u64 { - let namespace = match process::with_process_data(|data| { - data.handles.get::(ns_h, Rights::READ) + let namespace = match process::with_handles(|handles| { + handles.get::(ns_h, Rights::READ) }) { Ok(namespace) => namespace, Err(e) => return e.refuse(), @@ -240,12 +238,12 @@ fn connect_through(connector: &port::Connector) -> u64 { Err(e) => return SyscallError::from(e).to_u64(), }; // One hold from the room to the install: the port can still refuse, and the client's end is in the table only once it has not. - let installed = process::with_process_data(|data| { - if !data.handles.make_room(1) { + let installed = process::with_handles(|handles| { + if !handles.make_room(1) { return Err((SyscallError::ResourceExhausted, object, pending)); } match connector.push(pending) { - Ok(()) => Ok(ops::install(&mut data.handles, object).expect("room was asked for first")), + Ok(()) => Ok(ops::install(handles, object).expect("room was asked for first")), Err((pending, port::PushError::Closed)) => Err((SyscallError::Gone, object, pending)), Err((pending, port::PushError::QueueFull)) => { Err((SyscallError::ResourceExhausted, object, pending)) @@ -268,8 +266,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { ACCEPT_NONBLOCK => false, _ => return SyscallError::InvalidArgument.to_u64(), }; - let acceptor = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let acceptor = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(a) => a, Err(e) => return e.refuse(), @@ -284,8 +282,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { if let Some(conn) = acceptor.pop() { // The ends move from the port's queue into the connection: ownership transfers, no count changes. let object = KObjectRef::Connection(ConnectionEnd::accepted(core, conn)); - let installed = process::with_process_data(|data| { - ops::install(&mut data.handles, object) + let installed = process::with_handles(|handles| { + ops::install(handles, object) }); return handle_result(installed); } @@ -314,8 +312,8 @@ pub(super) fn sys_accept(h: RawHandle, flags: u64) -> u64 { /// A connector to the caller's own port stamping every connection with `badge`, which the dispatch has copied in and bounded. `READ` on the acceptor, the right accepting takes, is the authority: whoever answers every connection on a port grants nothing by labelling them. pub(super) fn sys_port_mint(h: RawHandle, badge: &[u8]) -> u64 { - let acceptor = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let acceptor = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(a) => a, Err(e) => return e.refuse(), @@ -324,7 +322,7 @@ pub(super) fn sys_port_mint(h: RawHandle, badge: &[u8]) -> u64 { Ok(connector) => KObjectRef::Connector(connector), Err(e) => return SyscallError::from(e).to_u64(), }; - process::with_process_data(|data| handle_result(ops::install(&mut data.handles, connector))) + process::with_handles(|handles| handle_result(ops::install(handles, connector))) } /// The badge a connection accepted from `acceptor_h`'s port was stamped with, written to `out`; answers its length. @@ -335,9 +333,9 @@ pub(super) fn sys_port_badge( conn_h: RawHandle, out: &mut crate::user_ptr::UserBytesMut, ) -> u64 { - let found = process::with_process_data(|data| { - let acceptor = data.handles.get::(acceptor_h, Rights::READ)?; - let conn = data.handles.get::(conn_h, Rights::READ)?; + let found = process::with_handles(|handles| { + let acceptor = handles.get::(acceptor_h, Rights::READ)?; + let conn = handles.get::(conn_h, Rights::READ)?; Ok::<_, crate::object::HandleError>((acceptor.port_id(), conn.stamp().cloned())) }); let (port, stamp) = match found { @@ -359,7 +357,7 @@ pub(super) fn sys_port_badge( pub(super) fn sys_shm_create(size: u64, flags: u64, job: RawHandle) -> u64 { let charged = match flags & SHM_FOR_JOB { 0 => crate::mm::charge::current(), - _ => match process::with_process_data(|data| data.handles.get::(job, Rights::WRITE)) { + _ => match process::with_handles(|handles| handles.get::(job, Rights::WRITE)) { Ok(object) => alloc::sync::Arc::clone(object.job()), Err(e) => return e.refuse(), }, @@ -373,15 +371,15 @@ pub(super) fn sys_shm_create(size: u64, flags: u64, job: RawHandle) -> u64 { Ok(shm) => KObjectRef::SharedMem(shm), Err(e) => return e.to_u64(), }; - process::with_process_data(|data| handle_result(ops::install(&mut data.handles, object))) + process::with_handles(|handles| handle_result(ops::install(handles, object))) } /// `toyos_abi::syscall::shm_sub`: the new object's handle carries `h`'s rights. pub(super) fn sys_shm_sub(h: RawHandle, offset: u64, len: u64) -> u64 { // A refusal is made with nothing held: a bad handle's ends the process. - let held = process::with_process_data(|data| { - let whole = data.handles.get::(h, Rights::MAP)?; - Ok::<_, crate::object::handle::HandleError>((whole, data.handles.rights_of(h)?)) + let held = process::with_handles(|handles| { + let whole = handles.get::(h, Rights::MAP)?; + Ok::<_, crate::object::handle::HandleError>((whole, handles.rights_of(h)?)) }); let (whole, rights) = match held { Ok(held) => held, @@ -392,8 +390,8 @@ pub(super) fn sys_shm_sub(h: RawHandle, offset: u64, len: u64) -> u64 { Err(e) => return e.to_u64(), }; let entry = crate::object::handle::HandleEntry::new(KObjectRef::SharedMem(sub), rights); - process::with_process_data(|data| { - handle_result(data.handles.install(entry).map_err(|_| SyscallError::ResourceExhausted)) + process::with_handles(|handles| { + handle_result(handles.install(entry).map_err(|_| SyscallError::ResourceExhausted)) }) } @@ -404,8 +402,8 @@ pub(super) fn sys_shm_map(h: RawHandle, prot: u64) -> u64 { Some(VmProt::ReadWrite) => (Prot::ReadWrite, Rights::MAP.union(Rights::WRITE)), Some(VmProt::None | VmProt::ReadExec) | None => return SyscallError::InvalidArgument.to_u64(), }; - let shm = match process::with_process_data(|data| { - data.handles.get::(h, need) + let shm = match process::with_handles(|handles| { + handles.get::(h, need) }) { Ok(shm) => shm, Err(e) => return e.refuse(), @@ -474,14 +472,14 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 let need = if out.count == 0 { Rights::WRITE } else { Rights::WRITE.union(Rights::TRANSFER) }; // Every handle is checked before anything is queued or moved, and again // under the hold that moves them: the answer is what each arrives with. - let checked = |data: &process::ProcessData| { - let conn = data.handles.get::(conn_h, need)?; + let checked = |handles: &HandleTable| { + let conn = handles.get::(conn_h, need)?; let mut moving = [(RawHandle(0), Rights::NONE); MAX_TRANSFER_HANDLES]; for (i, (h, want)) in wanted.iter().enumerate() { if *h == conn_h || wanted[..i].iter().any(|(seen, _)| seen == h) { return Err(SyscallError::InvalidArgument.into()); } - moving[i] = (*h, data.handles.rights_to_move(*h, *want)?); + moving[i] = (*h, handles.rights_to_move(*h, *want)?); } Ok::<_, crate::object::Refusal>((conn.tx().clone(), moving)) }; @@ -491,9 +489,9 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 let src = bounce.read(&out.bytes).map_err(|faulted| SyscallError::from(faulted).to_u64())?.bytes(); let parkable = crate::scheduler::Parkable::at_entry(); loop { - let moved = process::with_process_data(|data| { - let (tx, moving) = checked(data)?; - let (moved, handed) = move_handles(data, &moving[..out.count], |batch| tx.send_now(src, batch)); + let moved = process::with_handles(|handles| { + let (tx, moving) = checked(handles)?; + let (moved, handed) = move_handles(handles, &moving[..out.count], |batch| tx.send_now(src, batch)); Ok::<_, crate::object::Refusal>((tx, moved, handed)) }); let (tx, refused) = match moved { @@ -506,9 +504,9 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 Ok((tx, Err(Unsent::Claim), _)) => { let refused = match tx.claim_send(&parkable, src, out.count) { Ok(mut claim) => { - let moved = process::with_process_data(|data| { - let (_, moving) = checked(data)?; - Ok::<_, crate::object::Refusal>(move_handles(data, &moving[..out.count], |batch| claim.commit(batch))) + let moved = process::with_handles(|handles| { + let (_, moving) = checked(handles)?; + Ok::<_, crate::object::Refusal>(move_handles(handles, &moving[..out.count], |batch| claim.commit(batch))) }); drop(claim); match moved { @@ -551,16 +549,16 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 /// go. A refusal restores every handle at its own number, and every mapping, /// so it is invisible. fn move_handles( - data: &mut process::ProcessData, + handles: &mut HandleTable, wanted: &[(RawHandle, Rights)], sink: impl FnOnce(Vec) -> Result<(), (Vec, E)>, ) -> (Result<(), E>, Vec) { let claims: Vec<_> = wanted .iter() - .filter_map(|(h, _)| data.handles.get::(*h, Rights::TRANSFER).ok()) + .filter_map(|(h, _)| handles.get::(*h, Rights::TRANSFER).ok()) .filter_map(|claim| claim.take_mappings().map(|handed| (claim, handed))) .collect(); - let moved = data.handles.transfer(wanted, sink); + let moved = handles.transfer(wanted, sink); let handed = match moved { Ok(()) => claims.into_iter().map(|(_, handed)| handed).collect(), Err(_) => { @@ -589,12 +587,12 @@ fn recv(conn_h: RawHandle, into: &mut Incoming<'_>, nonblock: bool) -> u64 { loop { // Taken, and its handles installed, in the hold that looks the // connection up; its bytes are copied out after. - let looked = process::with_process_data(|data| { - let rx = data.handles.get::(conn_h, Rights::READ)?.rx().clone(); + let looked = process::with_handles(|handles| { + let rx = handles.get::(conn_h, Rights::READ)?.rx().clone(); let taken = loop { - match rx.recv_now(&mut data.handles, buf, cap, &mut bounce) { + match rx.recv_now(handles, buf, cap, &mut bounce) { // Grown with the queue's lock let go, then asked again. - Err(Unclaimed::Room(n)) if data.handles.make_room(n) => {} + Err(Unclaimed::Room(n)) if handles.make_room(n) => {} Err(Unclaimed::Room(_)) => break Err(Unclaimed::Untaken(kernel::message::Untaken::NoRoom)), taken => break taken, } @@ -643,12 +641,12 @@ fn delivered(rx: &QueueRef, into: &mut Incoming<'_>, head: kernel::message::Head /// A message taken whose reader's buffers faulted: its bytes are lost, and /// its handles, installed at numbers the reader was never told, are closed. fn undelivered(rx: &QueueRef, installed: &[RawHandle], faulted: crate::user_ptr::Faulted) -> u64 { - process::with_process_data(|data| { + process::with_handles(|handles| { for &h in installed { // A sibling may have closed the number since, or reused it for a // handle of its own, which this then closes: the race is the // misbehaving process's own, and stays inside it. - let _ = ops::close(&mut data.handles, h); + let _ = ops::close(handles, h); } }); rx.write_watch().post(); @@ -666,14 +664,14 @@ pub(super) fn sys_inbox_setup(depth: u32, out: &mut crate::user_ptr::UserBytesMu Ok(object) => KObjectRef::Inbox(object), Err(e) => return SyscallError::from(e).to_u64(), }; - let installed = process::with_process_data(|data| ops::install(&mut data.handles, object)); + let installed = process::with_handles(|handles| ops::install(handles, object)); let handle = match installed { Ok(h) => h, Err(e) => return e.to_u64(), }; // A caller that cannot be told its handle does not keep it. if let Err(faulted) = out.write_at(0, toyos_abi::usersafe::bytes(&InboxSetup { handle, _pad: 0, vaddr })) { - let _ = process::with_process_data(|data| ops::close(&mut data.handles, handle)); + let _ = process::with_handles(|handles| ops::close(handles, handle)); return SyscallError::from(faulted).to_u64(); } 0 @@ -686,8 +684,8 @@ pub(super) fn sys_inbox_submit( timeout_nanos: u64, ) -> u64 { // NotFound and PermissionDenied are the table's own words for gone/wrong-type, not collapsed into InvalidArgument. - let object = process::with_process_data(|data| { - data.handles.get::( + let object = process::with_handles(|handles| { + handles.get::( inbox_h, Rights::READ.union(Rights::WRITE), ) diff --git a/kernel/src/syscall/machine.rs b/kernel/src/syscall/machine.rs index 8f9f540676b..c08b586fa7b 100644 --- a/kernel/src/syscall/machine.rs +++ b/kernel/src/syscall/machine.rs @@ -373,8 +373,8 @@ pub(super) fn sys_device_set(syscap: RawHandle) -> u64 { Ok(set) => set, Err(e) => return SyscallError::from(e).to_u64(), }; - process::with_process_data(|data| { - super::handles::handle_result(crate::object::ops::install(&mut data.handles, crate::object::KObjectRef::DeviceSet(set))) + process::with_handles(|handles| { + super::handles::handle_result(crate::object::ops::install(handles, crate::object::KObjectRef::DeviceSet(set))) }) } @@ -384,11 +384,11 @@ pub(super) fn sys_device_set(syscap: RawHandle) -> u64 { /// seen its generation. /// /// **What a slot is owed is acted on here, on the reader's thread** -/// (`pcidev::snapshot`), outside every process lock. +/// (`pcidev::snapshot`), outside every handle-table lock. pub(super) fn sys_device_set_read(set: RawHandle, out: &mut UserBytesMut) -> u64 { use toyos_abi::devset::{self, Header}; use toyos_abi::inventory::RECORD_BYTES; - let set = match process::with_process_data(|data| data.handles.get::(set, Rights::READ)) { + let set = match process::with_handles(|handles| handles.get::(set, Rights::READ)) { Ok(set) => set, Err(e) => return e.refuse(), }; diff --git a/kernel/src/syscall/paged.rs b/kernel/src/syscall/paged.rs index bf05a078fe9..2c459417e0a 100644 --- a/kernel/src/syscall/paged.rs +++ b/kernel/src/syscall/paged.rs @@ -11,41 +11,32 @@ use toyos_abi::handle::{RawHandle, Rights}; use toyos_abi::syscall::{PagedArgs, PagedOp, SyscallError, PAGED_DIRTY, PAGED_RANGE_PAGES, PAGED_WRITTEN}; use crate::object::paged::{PagedHolderObject, PagerObject, Unmade}; -use crate::object::{ops, KObjectRef}; -use crate::process::{self, PagerRole}; +use crate::object::{ops, HandleError, HandleTable, KObjectRef}; +use crate::process; use crate::user_ptr::{SyscallContext, UserBytesMut}; use crate::UserAddr; pub(super) fn sys_pager_create(capacity: u64, dirty: u64, objects: u64, report_ns: u64) -> u64 { - let claimed = process::with_process_data(|data| match data.pager { - PagerRole::May => { - data.pager = PagerRole::Made; - Ok(()) - } - PagerRole::Made => Err(SyscallError::AlreadyExists), - PagerRole::None => Err(SyscallError::PermissionDenied), - }); - if let Err(e) = claimed { + let handles = process::handles(); + if let Err(e) = handles.claim_pager() { return e.to_u64(); } let made = PagerObject::create(capacity, dirty, objects, report_ns).map_err(|why| match why { Unmade::Invalid => SyscallError::InvalidArgument, Unmade::Exhausted => SyscallError::ResourceExhausted, }); - let installed = made.and_then(|pager| { - process::with_process_data(|data| ops::install(&mut data.handles, KObjectRef::Pager(pager))) - }); + let installed = made.and_then(|pager| ops::install(&mut handles.lock(), KObjectRef::Pager(pager))); match installed { Ok(h) => u64::from(h.0), Err(e) => { - process::with_process_data(|data| data.pager = PagerRole::May); + handles.unclaim_pager(); e.to_u64() } } } fn pager(h: RawHandle, rights: Rights) -> Result, u64> { - process::with_process_data(|data| data.handles.get::(h, rights)).map_err(|e| e.refuse()) + process::with_handles(|handles| handles.get::(h, rights)).map_err(|e| e.refuse()) } pub(super) fn sys_paged_create(h: RawHandle, size: u64, anonymous: bool) -> u64 { @@ -65,8 +56,8 @@ pub(super) fn sys_paged_holder(h: RawHandle, object: u64) -> u64 { Err(e) => return e.to_u64(), }; // Counted under the same hold as its install: no close can come between. - let installed = process::with_process_data(|data| { - let h = ops::install(&mut data.handles, KObjectRef::Paged(Arc::clone(&holder)))?; + let installed = process::with_handles(|handles| { + let h = ops::install(handles, KObjectRef::Paged(Arc::clone(&holder)))?; holder.held(); Ok::<_, SyscallError>(h) }); @@ -87,21 +78,29 @@ pub(super) fn sys_paged_read(h: RawHandle, out: &mut UserBytesMut, offset: u64) } fn holder(h: RawHandle) -> Result, u64> { - let resolved = process::with_process_data(|data| { - let KObjectRef::Paged(holder) = data.handles.get_ref(h, Rights::NONE)? else { - return Ok(Err(SyscallError::InvalidArgument)); - }; - let holder = Arc::clone(holder); - if !data.handles.rights_of(h)?.contains(Rights::READ) { - return Ok(Err(SyscallError::PermissionDenied)); - } - Ok(if data.pager.is_pager() { Err(SyscallError::PermissionDenied) } else { Ok(holder) }) - }); + let handles = process::handles(); + let resolved = readable_holder(&handles.lock(), h, handles.is_pager()); match resolved { Ok(Ok(holder)) => Ok(holder), Ok(Err(e)) => Err(e.to_u64()), - Err(e) => Err(crate::object::HandleError::refuse(e)), + Err(e) => Err(HandleError::refuse(e)), + } +} + +/// The holder `h` names in `table`, if it carries `READ` and its process is +/// no `pager`. +fn readable_holder( + table: &HandleTable, + h: RawHandle, + pager: bool, +) -> Result, SyscallError>, HandleError> { + let KObjectRef::Paged(holder) = table.get_ref(h, Rights::NONE)? else { + return Ok(Err(SyscallError::InvalidArgument)); + }; + if !table.rights_of(h)?.contains(Rights::READ) || pager { + return Ok(Err(SyscallError::PermissionDenied)); } + Ok(Ok(Arc::clone(holder))) } pub(super) fn sys_paged_op(ctx: &SyscallContext<'_>, h: RawHandle, op: u64, args: UserAddr) -> u64 { diff --git a/kernel/src/syscall/proc.rs b/kernel/src/syscall/proc.rs index ede3437188e..f9084def278 100644 --- a/kernel/src/syscall/proc.rs +++ b/kernel/src/syscall/proc.rs @@ -59,8 +59,8 @@ pub(super) fn spawn_place(place: u64) -> Result { let Ok(raw) = u32::try_from(place) else { return Err(SyscallError::InvalidArgument.to_u64()); }; - process::with_process_data(|data| { - data.handles.get_sent::(RawHandle(raw), Rights::WRITE) + process::with_handles(|handles| { + handles.get_sent::(RawHandle(raw), Rights::WRITE) }) .map(|object| process::Parent::Under(object.pid())) .map_err(|refused| refused.refuse()) @@ -84,8 +84,8 @@ pub(super) fn spawn_program(ctx: &crate::user_ptr::SyscallContext<'_>, args: &Sp handle if handle == u64::from(toyos_abi::HANDLE_INVALID.0) => None, handle => { let raw_handle = u32::try_from(handle).map_err(|_| invalid())?; - let object = process::with_process_data(|data| { - data.handles.get::(RawHandle(raw_handle), Rights::MAP) + let object = process::with_handles(|handles| { + handles.get::(RawHandle(raw_handle), Rights::MAP) }); Some(object.map_err(|refused| refused.refuse())?) } @@ -134,8 +134,8 @@ pub(super) fn sys_process_wait(h: RawHandle, flags: u64) -> u64 { if flags & !PROCESS_WAIT_FLAGS_KNOWN != 0 { return SyscallError::InvalidArgument.to_u64(); } - let object = match process::with_process_data(|data| { - data.handles.get::(h, Rights::WAIT) + let object = match process::with_handles(|handles| { + handles.get::(h, Rights::WAIT) }) { Ok(object) => object, Err(e) => return e.refuse(), @@ -169,8 +169,8 @@ pub(super) fn sys_process_quit(h: RawHandle, reason: u64) -> u64 { let Some(reason) = u8::try_from(reason).ok().and_then(toyos_abi::quit::Reason::from_byte) else { return SyscallError::InvalidArgument.to_u64(); }; - match process::with_process_data(|data| { - data.handles.get::(h, Rights::MANAGE) + match process::with_handles(|handles| { + handles.get::(h, Rights::MANAGE) }) { Ok(object) => { process::quit_process(&object, reason); @@ -203,7 +203,7 @@ pub(super) fn sys_wake_latency_hold(syscap: RawHandle, bound_ns: u64) -> u64 { Ok(object) => crate::object::KObjectRef::WakeLatency(object), Err(e) => return SyscallError::from(e).to_u64(), }; - match process::with_process_data(|data| crate::object::ops::install(&mut data.handles, object)) { + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -218,7 +218,7 @@ pub(super) fn sys_job_create(syscap: RawHandle) -> u64 { Ok(object) => crate::object::KObjectRef::Job(object), Err(e) => return SyscallError::from(e).to_u64(), }; - match process::with_process_data(|data| crate::object::ops::install(&mut data.handles, object)) { + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -226,11 +226,12 @@ pub(super) fn sys_job_create(syscap: RawHandle) -> u64 { /// Install a handle to the job the caller runs in. pub(super) fn sys_job_own() -> u64 { - let installed = process::with_process_data(|data| { - let object = crate::object::KObjectRef::Job(crate::object::job::JobObject::new(data.job.clone())?); - crate::object::ops::install(&mut data.handles, object) - }); - match installed { + let job = process::with_process_data(|data| data.job.clone()); + let object = match crate::object::job::JobObject::new(job) { + Ok(object) => crate::object::KObjectRef::Job(object), + Err(e) => return SyscallError::from(e).to_u64(), + }; + match process::with_handles(|handles| crate::object::ops::install(handles, object)) { Ok(h) => u64::from(h.0), Err(e) => e.to_u64(), } @@ -372,8 +373,8 @@ pub(super) fn sys_process_stats( h: RawHandle, out: UserAddr, ) -> u64 { - let object = match process::with_process_data(|data| { - data.handles.get::(h, Rights::READ) + let object = match process::with_handles(|handles| { + handles.get::(h, Rights::READ) }) { Ok(object) => object, Err(e) => return e.refuse(), @@ -393,7 +394,7 @@ pub(super) fn sys_job_floor(syscap: RawHandle, job: RawHandle, bytes: u64) -> u6 if let Err(e) = demand_syscap(syscap, Rights::FLOOR) { return e.refuse(); } - let object = match process::with_process_data(|data| data.handles.get::(job, Rights::WRITE)) { + let object = match process::with_handles(|handles| handles.get::(job, Rights::WRITE)) { Ok(object) => object, Err(e) => return e.refuse(), }; @@ -405,7 +406,7 @@ pub(super) fn sys_job_floor(syscap: RawHandle, job: RawHandle, bytes: u64) -> u6 /// Copy out what the account of the job a handle names holds, gated on its `READ`. pub(super) fn sys_account_info(ctx: &crate::user_ptr::SyscallContext, job: RawHandle, out: UserAddr) -> u64 { - let object = match process::with_process_data(|data| data.handles.get::(job, Rights::READ)) { + let object = match process::with_handles(|handles| handles.get::(job, Rights::READ)) { Ok(object) => object, Err(e) => return e.refuse(), };