diff --git a/issues/a-function-refused-under-translation-faults-read-permission-after-the-hand-over.md b/issues/a-function-refused-under-translation-faults-read-permission-after-the-hand-over.md deleted file mode 100644 index 3069f536bae..00000000000 --- a/issues/a-function-refused-under-translation-faults-read-permission-after-the-hand-over.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-04 ---- - -# A function refused under translation faults read permission after the hand-over - -On the T14, when the integrated GPU (`00:02.0`, unit0) has its scanout reads -refused for about 2 ms (0.153 s to 0.155 s) by a unit left translating, and that unit is then -handed over and programmed, the GPU's next reads fault `0x06` (read -permission) through this kernel's identity domain, which grants them, and the -kernel halts on `DMA FAULT owner=kernel`. `vtd::enable` and `Unit::hand_over` -(`kernel/src/arch/x86_64/vtd/mod.rs`) do what VT-d Rev. 4.1 asks before `TE` -goes back on: `SRTP` to a fresh root, then a global context-cache and a global -IOTLB invalidation with both drains, acknowledged (§6.6). A unit reporting -`CAP.CM=0`, as unit0 does, caches no faulted translation (§6.2.4.1), so a -refusal before the hand-over leaves nothing in the unit the spec lets fault -after it. What carries it across, the GPU or the unit, is unknown. - -Firmware does not hand the T14 over this way: its four units read -`gsts=0x40000000`. The state is the one the `iommu-firmware-left` actuator left -at `9be0a8ac8` of pull request #700, which pointed every unit at an empty root -while unit0's identity domain was built. Today's actuator translates through -the identity domain and refuses nothing. - -Evidence, pull request #700, T14 boots: - -- `9be0a8ac8` plus the `FSTS` probe: unit0's one record, before `fault::arm` - clears it, is `00:02.0` read `0x9cb80000` reason `0x01` with `FSTS=0x3` - (overflowed), from the empty root. Once `TE` is confirmed, `FSTS=0` and the - record is clear. Next, `unit0 fault recording overflowed`, then - `00:02.0` read `0x9cc39000` reason `0x06`. Every address is inside - `rmrr0` (`0x9c000000..0xa07fffff`, scope `00:02.0`). -- `55a5ccb42`, the same empty-root actuator but handed over before the build, - so the refusal and the hand-over are both logged at 0.153 s and `TE` stayed - off until 0.155 s: no fault. -- `25ab31bd5` and `5e8cbb06e`, no refusal, and the hand-over and `translating` - logged in the same millisecond: no fault. - -- `9be0a8ac8` plus a walk probe: unit0's record before `fault::arm` is again - `00:02.0` read `0x9ccc0000` reason `0x01` with `FSTS=0x3`. At the fault, - `00:02.0` read `0x9cdb7000` reason `0x06`, raw high word - `0xc000000600000010`: `AT=0`, `PP=0`, `EXE=0`, `PRIV=0`, `T2=0`, an - untranslated read with no PASID. The unit's walk, from `RTADDR` `0xa18000`, - each entry read from DRAM after `clflush`: root `0xa19001`, context - `0xa04001:0x102` (`P=1`, `TT=00`, `AW=2`, `DID=1`), then `0xa05003`, - `0xa08003` and the 2 MiB leaf `0x9cc00083`, present with read and write over - `0x9cdb7000`. The GPU's PCI `STATUS` reads `0x0010`, no master or target - abort, before the actuator, before the hand-over, at `TE` and at the fault. - The unit refused a read the tables in memory grant. -- the walk probe plus 3 ms with `TE` off after each hand-over: the same - empty-root refusals (`FSTS=0x3`, record `0x01`) before `fault::arm`, then no - `DMA FAULT` and `Boot: complete`. - -So the tables are not what refuses, and neither is any cached state VT-d Rev. -4.1 lets the unit keep. The `0x06` needs both a period of refused reads and -`TE` back on within microseconds of the hand-over: three boots of that -actuator with no hold faulted (`9be0a8ac8` alone, with the `FSTS` probe and -with the walk probe), one with a 3 ms hold did not, and `55a5ccb42`, -`TE` off about 2 ms after its refusals, did not. - -**Exit**: on the T14, the reproduction (the selftests boot at `9be0a8ac8`) -reaches `Boot: complete` with no `DMA FAULT` line, through a hand-over that -waits on an event the unit or the function reports. A timed hold with `TE` -off does not close this: it is a flat wait no hardware document mandates, it -widens the untranslated window -`issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md` -records, and it rests on one green boot against three reds. diff --git a/issues/a-unit-in-scalable-or-abort-mode-loses-translation-for-its-root-table-switch.md b/issues/a-unit-in-scalable-or-abort-mode-loses-translation-for-its-root-table-switch.md deleted file mode 100644 index b5cf73c5fb2..00000000000 --- a/issues/a-unit-in-scalable-or-abort-mode-loses-translation-for-its-root-table-switch.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-04 ---- - -# A unit in scalable or abort mode loses translation for its root-table switch - -A unit firmware leaves translating with a root table in scalable or abort-DMA -mode (`RTADDR_REG.TTM` not `00`) and `CAP.ESRTPS` clear may not have its root -table pointer set while translation is on (VT-d Rev. 4.1 §6.6). Handing such a -unit over to this kernel's root table therefore turns translation off for that -one switch, and for that moment every function behind the unit reaches all of -memory. `issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md` -removes the gap where the switch may stay translating; on these units it may -not. - -Evidence: the specification alone. No machine here has such a unit: every unit of the -T14 reports `CAP` bit 63 (`ESRTPS`), `ECAP` bit 43 (`SMTS`) and bit 52 -(`ADMS`) clear. - -**Ruled** (owner, 2026-10-04, "Accept and file"): "Allow the one-moment gap on -those machines, log it, and file it as a known weakness with a fix (older -protection registers) as its exit." - -Owner: the orchestrator, under `issues/the-iommu-refuses-nothing-yet.md`. - -**Exit**: on such a unit that reports `CAP.PLMR` (bit 5) and `CAP.PHMR` -(bit 6), `PMEN`'s protected memory regions (§11.4.8.1) cover all of memory -before translation goes off for the switch and are released only once it is -back on; a host test reads that order, and is red with the `PMEN` step -removed. It is a host test because no machine reaches the path: QEMU's unit -always reports `ESRTPS` and implements `PMEN_REG` as read-only zero, and the -T14's units report `ESRTPS` clear but support neither mode. - -A unit that reports either bit clear has no fix under this exit: with either -clear, that region's base and limit are read-only (§11.4.8.1), so `PMEN` -cannot cover all of memory; the same section plans the protected memory -registers for deprecation, pointing new software at abort-DMA mode. Such a -unit keeps the gap, logged, and this file stays open for it once the `PMEN` -exit is met. - -**Exit for a unit with `PLMR` or `PHMR` clear**: on such a unit no function -behind it reaches memory while translation is off for the switch; a host test -reads that, and is red with the fix removed. diff --git a/issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md b/issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md deleted file mode 100644 index bff8201b065..00000000000 --- a/issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-04 ---- - -# A unit left translating passes DMA untranslated while it is programmed - -`vtd::enable` (`kernel/src/arch/x86_64/vtd/mod.rs`) switches off the -translation firmware left on (`Unit::hand_over`) once the unit's tables are -built, and turns it back on through this kernel's root table only after the -queue, fault reporting, both table pointers and a global invalidation are -programmed. In between, every function behind that unit reaches all of memory. - -Keeping `TE` on instead is what VT-d Rev. 4.1 §6.6 allows only onto tables that -remap exactly as the ones being walked, which takes firmware's tables read and -copied first, as Linux's kdump path does (`copy_translation_tables`, -`drivers/iommu/intel/iommu.c`); Linux outside kdump switches `TE` off as this -kernel does. Any copy has to read them before the pmm hands their memory out: -it takes `EfiBootServicesData` as free (`lib/bootmap/src/lib.rs`, -`is_usable_type`). - -Evidence: no machine here is handed a unit translating; the T14's four units -read `gsts=0x40000000` (pull request #700's selftests boot). Staged by the -`iommu-firmware-left` actuator on QEMU 11.1.1, unit0 logs `was handed over -with translation on` at 0.083 s, just before translation goes off, and -`translating` at 0.090 s, just after it is back; two of the hand-over's own log -lines fall between. - -One case keeps the window whatever this kernel does: a unit handed over with -`TTM` not `00` and `CAP.ESRTPS` clear, where §6.6 forbids switching tables -under translation, so `TE` has to go off. **Owner ruling, 2026-10-04: accept -it and file it.** The fix that closes this issue logs that case and files it -as its own defect, whose exit covers the window with protected memory regions -(`PMEN` over every pmm frame) or has the kernel speak scalable mode. -No machine here reaches it. - -**Exit**: under `iommu-firmware-left`, on QEMU and on the T14, no unit logs -`was handed over with translation on; it goes off first`, and every unit's -`translating` line reads `tes=y`. diff --git a/issues/a-unit-passes-dma-untranslated-outside-its-protected-memory-regions-while-it-is-handed-over.md b/issues/a-unit-passes-dma-untranslated-outside-its-protected-memory-regions-while-it-is-handed-over.md new file mode 100644 index 00000000000..bc198747115 --- /dev/null +++ b/issues/a-unit-passes-dma-untranslated-outside-its-protected-memory-regions-while-it-is-handed-over.md @@ -0,0 +1,74 @@ +--- +status: open +kind: defect +opened: 2026-10-04 +--- + +# A unit passes DMA untranslated outside its protected memory regions while it is handed over + +`vtd::init` (`kernel/src/arch/x86_64/vtd/mod.rs`) hands every unit over with +translation off. That covers a unit firmware left translating, whatever its +table mode. VT-d Rev. 4.1 gives no condition under which system software may +switch firmware's tables for its own under translation. §6.6 requires +identical remapping results, and nothing documented lets this kernel know +firmware's mappings. `DMA_CTRL_PLATFORM_OPT_IN` (§8.1) says only where +platform DMA goes. + +The hand-over is guarded where every described unit is programmed, has both +protected memory regions (`CAP.PLMR`, `CAP.PHMR`) and takes register-based +invalidation. On such units the regions block DMA to what they cover whether +or not the unit translates (§11.4.8.1). They are programmed on every unit +after any firmware left on is switched off, and released once the last unit +translates (`vtd/pmr.rs`). The low region and the high region are each the +largest stretch of the OS's RAM (usable or ACPI-reclaimable, never NVS) that +avoids every RMRR and holds every 2 MiB page the hand-over's tables were +carved from, and this kernel's image. The boot logs the regions, what they +cover and the functions mastering the bus as they go on. The T14's four units +are guarded. The queue and interrupt remapping are programmed after the +release. + +What passes untranslated during the hand-over: + +- **A machine whose hand-over is unguarded:** all of memory. Its log says why: + - a unit without both regions (QEMU's); + - a unit that refuses register-based invalidation (major version 6 or later, + §11.4.6.1), or whose IOTLB registers lie outside the window this kernel + maps; + - fewer units programmed than described; + - tables or kernel outside every region. + + Such a machine invalidates through the queue, enabled before translation + comes on. +- **A guarded machine:** + - RAM outside the two chosen stretches; + - every RMRR; + - peer MMIO. +- **Before the hand-over, on any machine:** + - Where firmware leaves translation off, as the T14's does, every function + reaches all of memory from ExitBootServices until the regions go on. + - Where firmware leaves it on, `mm::init` has already taken + `EfiBootServicesData` as free (`lib/bootmap/src/lib.rs`, + `is_usable_type`). A unit may walk firmware's tables in memory the pmm has + handed out. + +Evidence: +- No machine here is handed a unit translating. +- `iommu-firmware-left` stages one: + - QEMU 11.1.1 runs the unguarded hand-over (`vtd_firmware_left`); + - the T14 runs the guarded one (`iommu_firmware_left`'s row, which reads + the regions go on before translation first goes off and come off before + the first unit is programmed). + +Owner: the IOMMU track (`issues/the-iommu-refuses-nothing-yet.md`). + +**Exit**: during the hand-over no function reaches memory outside its RMRRs. +Each of these cases is closed by a documented mechanism and read on a +machine or a host test: + +1. a unit without protected memory regions. §11.4.8.1 marks the regions + "planned for deprecation", so new units will be this case. Abort-DMA mode + (§3.4.4) is one mechanism; it needs `ECAP.ADMS` and is ruled out while a + device an RMRR names masters the bus; +2. a unit that refuses register-based invalidation; +3. RAM a guarded hand-over leaves outside its regions; +4. peer MMIO. diff --git a/issues/a-unit-this-kernel-does-not-program-keeps-what-firmware-left-on.md b/issues/a-unit-this-kernel-does-not-program-keeps-what-firmware-left-on.md index f95ecaf94d1..c4bc8d37375 100644 --- a/issues/a-unit-this-kernel-does-not-program-keeps-what-firmware-left-on.md +++ b/issues/a-unit-this-kernel-does-not-program-keeps-what-firmware-left-on.md @@ -7,11 +7,12 @@ opened: 2026-10-04 # A unit this kernel does not program keeps what firmware left on `vtd::init` (`kernel/src/arch/x86_64/vtd/mod.rs`) switches off what firmware -left on (`Unit::hand_over`) only inside `enable`, which a unit reaches only if -`plan` accepts it and it is among the first `MAX_UNITS` described. A unit +left on (`Unit::hand_over`) only in a unit's hand-over (`translate`), which a +unit reaches only if `plan` accepts it and it is among the first `MAX_UNITS` described. A unit `plan` refuses, and every unit past `MAX_UNITS`, keeps whatever firmware left on while the kernel treats it as absent: translation through firmware's tables, -and interrupt remapping, which blocks every compatibility-format message this +protected memory regions (`vtd/pmr.rs` switches them off on programmed units +only), and interrupt remapping, which blocks every compatibility-format message this kernel then sends through the unit unless firmware also left `CFI` on with `EIME` off (VT-d Rev. 4.1 §5.1.4). The module header's "leaves the unit switched off" is false of such a unit. diff --git a/issues/the-iommu-refuses-nothing-yet.md b/issues/the-iommu-refuses-nothing-yet.md index 131c9a09b23..f4d2558dbc8 100644 --- a/issues/the-iommu-refuses-nothing-yet.md +++ b/issues/the-iommu-refuses-nothing-yet.md @@ -52,8 +52,9 @@ a unit firmware left translating: scalable or abort-DMA mode, with `CAP.ESRTPS` clear, may not have its root table switched under translation (VT-d Rev. 4.1 §6.6), so translation goes off for that one switch. The older protection registers are `PMEN`'s - protected memory regions (§11.4.8.1). The weakness is - `issues/a-unit-in-scalable-or-abort-mode-loses-translation-for-its-root-table-switch.md`. + protected memory regions (§11.4.8.1). Every hand-over now takes + translation off inside them where every unit has them; what remains is + `issues/a-unit-passes-dma-untranslated-outside-its-protected-memory-regions-while-it-is-handed-over.md`. - **"Apply it at hand-over"**, on reserved memory: "From the hand-over on, only display and USB controllers keep access to their reserved region; any other device's access is refused. A device that firmware was still using @@ -67,6 +68,21 @@ a unit firmware left translating: boot, he answered: "You can do with the t14 what you want." The grant goes beyond the question asked. +**Proposed, for the owner** (pull request #1086's review): a fault from an +enumerated function that no driver runs halts the machine only when the +address is one its domain maps. A mapped address means the unit contradicted +tables this kernel wrote, which the shared identity domain does not confine +to one unit. An unmapped address is firmware's DMA straying outside its +grant: the function is stopped, the fault logged, and the machine goes on, as +"Apply it at hand-over" already rules for reserved regions. An unclaimed +function sits in the identity domain, which maps all of `[0, top)` with +nothing carved out, so "unmapped" is `address >= top` and the handler stays +bounded and lock-free. The change is to the representation: `Owner::Kernel` +today merges functions this kernel drives with functions nobody drives, and +the split gives the second its own arm. One caveat: the iGPU is not wholly +undriven. Its scanout reads the firmware framebuffer the kernel's panic +console draws into, so stopping it blanks that console. + Owner: the orchestrator. **Exit**: a test or a T14 row reads each ruled refusal, each with its diff --git a/issues/the-t14s-igpu-faults-read-permission-after-translation-goes-off-and-on-under-its-scanout.md b/issues/the-t14s-igpu-faults-read-permission-after-translation-goes-off-and-on-under-its-scanout.md new file mode 100644 index 00000000000..13c34473d0f --- /dev/null +++ b/issues/the-t14s-igpu-faults-read-permission-after-translation-goes-off-and-on-under-its-scanout.md @@ -0,0 +1,92 @@ +--- +status: open +kind: defect +opened: 2026-10-04 +--- + +# The T14's iGPU faults read permission after translation goes off and on under its scanout + +On the T14, when unit0, the integrated GPU's (`00:02.0`), has translation +turned off for microseconds and back on while the GPU's scanout reads through +it, the GPU's +reads sometimes fault `0x06` (read permission) through this kernel's identity +domain, which grants them. The kernel then halts on `DMA FAULT owner=kernel`. +Before `TE` goes back on, the hand-over does what VT-d Rev. 4.1 asks: `SRTP` +to a fresh root, then a global context-cache and a global IOTLB invalidation +with both drains, acknowledged (§6.6). A unit reporting `CAP.CM=0`, as unit0 +does, caches no faulted translation (§6.2.4.1). What carries the fault across, +the GPU or the unit, is unknown. + +Firmware does not hand the T14 over this way: its four units read +`gsts=0x40000000`. The state below is the one the `iommu-firmware-left` +actuator left at `9be0a8ac8` of pull request #700, which pointed every unit at +an empty root while unit0's identity domain was built. + +Evidence, pull request #700, T14 boots: + +- `9be0a8ac8` plus the `FSTS` probe: unit0's one record, before `fault::arm` + clears it, is `00:02.0` read `0x9cb80000` reason `0x01` with `FSTS=0x3` + (overflowed), from the empty root. Once `TE` is confirmed, `FSTS=0` and the + record is clear. Next, `unit0 fault recording overflowed`, then + `00:02.0` read `0x9cc39000` reason `0x06`. Every address is inside + `rmrr0` (`0x9c000000..0xa07fffff`, scope `00:02.0`). +- `55a5ccb42`, the same empty-root actuator but handed over before the build, + so the refusal and the hand-over are both logged at 0.153 s and `TE` stayed + off until 0.155 s: no fault. +- `25ab31bd5` and `5e8cbb06e`, no refusal, and the hand-over and `translating` + logged in the same millisecond: no fault. + +- `9be0a8ac8` plus a walk probe: unit0's record before `fault::arm` is again + `00:02.0` read `0x9ccc0000` reason `0x01` with `FSTS=0x3`. At the fault, + `00:02.0` read `0x9cdb7000` reason `0x06`, raw high word + `0xc000000600000010`: `AT=0`, `PP=0`, `EXE=0`, `PRIV=0`, `T2=0`, an + untranslated read with no PASID. The unit's walk, from `RTADDR` `0xa18000`, + each entry read from DRAM after `clflush`: root `0xa19001`, context + `0xa04001:0x102` (`P=1`, `TT=00`, `AW=2`, `DID=1`), then `0xa05003`, + `0xa08003` and the 2 MiB leaf `0x9cc00083`, present with read and write over + `0x9cdb7000`. The GPU's PCI `STATUS` reads `0x0010`, no master or target + abort, before the actuator, before the hand-over, at `TE` and at the fault. + The unit refused a read the tables in memory grant. +- the walk probe plus 3 ms with `TE` off after each hand-over: the same + empty-root refusals (`FSTS=0x3`, record `0x01`) before `fault::arm`, then no + `DMA FAULT` and `Boot: complete`. + +Pull request #1086, T14: + +- One shared-debug boot (7ef0a48c8, today's actuator, which refuses nothing) + halted on `00:02.0` read `0x9cea7000` reason `0x06`. It was the one fault in + 46 such boots, against 0 in 787 boots whose unit0 only turns translation on. +- `iommu-handover-cycles` (measurement branch `igpu-fault-measure`, e3b1562e7, + never landed) moved every unit between two identical identity roots, three + boots, with fault events held. 200 root switches under translation ran + first, then 200 cycles of translation off, root, queued invalidation and + translation on. Totals: + - **switch: 0 of 600 cycles faulted;** + - **off-on: 22 of 600 cycles faulted**, with 94 records. +- Every off-on record is `00:02.0`, a read, `0x06`, at an address inside the + firmware framebuffer in data stolen memory (`BDSM=0x9c800000`, + `BGSM=0x9c000000`, `TOLUD=0xa0800000`). The first comes 546 to 598 µs after + translation is back on. +- Each printed walk from memory reads a present root and context entry and a + read-write 2 MiB leaf. + +So no period of refused reads is needed. Translation going off for +microseconds and back on under live scanout is enough, in about one cycle in +27, and the unit then refuses reads its tables grant. Turning translation on +for the first time since firmware is not: the 787 boots did only that. The 3 ms hold's one green boot is no evidence at that rate. +Neither the tables nor any cached state VT-d Rev. 4.1 lets the unit keep is +what refuses. Intel's 11th Generation Core specification update (631123-034, +errata TGL001 to TGL075) documents no erratum for VT-d or the iGPU's DMA. + +Every hand-over still takes translation off, now inside protected memory +regions, so `iommu-firmware-left` still reaches this on the T14. Real T14 +firmware leaves translation off, so no other boot does. How often a boot of +today's sequence halts is not measured. The estimate is between the cycles' +one in 27, a different sequence with no regions, and the pre-#1086 boots' 1 +in 46. The row that arms the actuator rides a boot of its own, so a halt costs +only that boot. + +**Exit**: this kernel's hand-over sequence is fixed so that the measurement's +off-on cycles fault no more than its switch cycles, or a documented erratum +for it is cited and its workaround applied. Either is judged on the T14 by +the measurement and by `iommu_firmware_left`'s row. diff --git a/kernel/pure/dmaguard.rs b/kernel/pure/dmaguard.rs new file mode 100644 index 00000000000..77a531a7b44 --- /dev/null +++ b/kernel/pure/dmaguard.rs @@ -0,0 +1,148 @@ +//! Where a remapping unit's protected memory regions go while it is handed over. + +#![forbid(unsafe_code)] + +use alloc::vec::Vec; + +pub const FOUR_GIB: u64 = 1 << 32; + +/// `[base, end)`, never empty. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub struct Span { + pub base: u64, + pub end: u64, +} + +impl Span { + pub fn holds(self, (start, end): (u64, u64)) -> bool { + self.base <= start && end <= self.end + } +} + +/// The largest part of `ram` inside `[lo, hi)` that touches no `avoid` range +/// and spans no gap between `ram`'s ranges, with its base rounded up and its +/// end rounded down to `align` (a power of two), among the parts holding every +/// `within` range that starts inside `[lo, hi)`. `None` if no part does. +pub fn largest( + ram: &[(u64, u64)], + lo: u64, + hi: u64, + avoid: &[(u64, u64)], + align: u64, + within: &[(u64, u64)], +) -> Option { + assert!(align.is_power_of_two(), "a protected region's alignment {align:#x} is not a power of two"); + let mut ram: Vec<(u64, u64)> = ram.iter().copied().filter(|&(s, e)| s < e).collect(); + ram.sort_unstable(); + // Adjacent ranges are one stretch; a gap ends it. + let mut runs: Vec<(u64, u64)> = Vec::new(); + for (start, end) in ram { + match runs.last_mut() { + Some(last) if start <= last.1 => last.1 = last.1.max(end), + _ => runs.push((start, end)), + } + } + let mut avoid: Vec<(u64, u64)> = avoid.iter().copied().filter(|&(s, e)| s < e).collect(); + avoid.sort_unstable(); + let needed: Vec<(u64, u64)> = within.iter().copied().filter(|&(s, _)| lo <= s && s < hi).collect(); + let mut best: Option = None; + for (start, end) in runs { + let mut piece = start.max(lo); + let stop = end.min(hi); + for &(a, b) in avoid.iter().chain(core::iter::once(&(stop, u64::MAX))) { + if b <= piece { + continue; + } + let cut = a.min(stop); + if cut > piece { + let base = piece.div_ceil(align).saturating_mul(align); + let end = cut / align * align; + let span = Span { base, end }; + if base < end + && needed.iter().all(|&r| span.holds(r)) + && best.is_none_or(|b| end - base > b.end - b.base) + { + best = Some(span); + } + } + if b >= stop { + break; + } + piece = piece.max(b); + } + } + best +} + +#[cfg(test)] +mod tests { + use super::*; + use std::vec; + + const MIB: u64 = 1 << 20; + + /// RAM up to a graphics region an RMRR names, and RAM again above 4 GiB. + #[test] + fn a_region_stops_at_the_reserved_region_and_at_4_gib() { + let ram = [(0x10_0000, 0x9c00_0000), (0x1_0000_0000, 0x4_5f80_0000)]; + let avoid = [(0x9c00_0000, 0xa080_0000)]; + assert_eq!( + largest(&ram, 0, FOUR_GIB, &avoid, MIB, &[]), + Some(Span { base: 0x10_0000, end: 0x9c00_0000 }) + ); + assert_eq!( + largest(&ram, FOUR_GIB, u64::MAX, &avoid, MIB, &[]), + Some(Span { base: FOUR_GIB, end: 0x4_5f80_0000 }) + ); + } + + #[test] + fn a_gap_in_the_map_ends_a_stretch_and_adjacent_ranges_join() { + let ram = [(0x0100_0000, 0x2000_0000), (0x2000_0000, 0x3000_0000), (0x3010_0000, 0x3800_0000)]; + assert_eq!(largest(&ram, 0, FOUR_GIB, &[], MIB, &[]), Some(Span { base: 0x0100_0000, end: 0x3000_0000 })); + } + + #[test] + fn a_reserved_region_inside_a_stretch_leaves_its_larger_side() { + let ram = [(0x0100_0000, 0x8000_0000)]; + let avoid = [(0x2000_0000, 0x2010_0000)]; + assert_eq!(largest(&ram, 0, FOUR_GIB, &avoid, MIB, &[]), Some(Span { base: 0x2010_0000, end: 0x8000_0000 })); + } + + #[test] + fn a_region_is_the_part_that_holds_what_it_must_cover_even_when_smaller() { + let ram = [(0x0100_0000, 0x2000_0000), (0x2010_0000, 0x8000_0000)]; + let tables = [(0x0160_0000, 0x0180_0000)]; + assert_eq!(largest(&ram, 0, FOUR_GIB, &[], MIB, &tables), Some(Span { base: 0x0100_0000, end: 0x2000_0000 })); + // Ranges that start in the other window bind nothing here. + assert_eq!( + largest(&ram, 0, FOUR_GIB, &[], MIB, &[(0x1_0000_0000, 0x1_0020_0000)]), + Some(Span { base: 0x2010_0000, end: 0x8000_0000 }) + ); + } + + #[test] + fn what_no_one_part_holds_leaves_no_region() { + let ram = [(0x0100_0000, 0x2000_0000), (0x2010_0000, 0x8000_0000)]; + let split = [(0x0160_0000, 0x0180_0000), (0x5000_0000, 0x5020_0000)]; + assert!(largest(&ram, 0, FOUR_GIB, &[], MIB, &split).is_none()); + let under_an_rmrr = [(0x3000_0000, 0x3020_0000)]; + assert!(largest(&ram, 0, FOUR_GIB, &[(0x3000_0000, 0x3100_0000)], MIB, &under_an_rmrr).is_none()); + } + + #[test] + fn alignment_only_ever_shrinks_a_region() { + let ram = [(0x0123_4000, 0x0987_6000)]; + let span = largest(&ram, 0, FOUR_GIB, &[], 16 * MIB, &[]).unwrap(); + assert_eq!(span, Span { base: 0x0200_0000, end: 0x0900_0000 }); + assert!(largest(&ram, 0, FOUR_GIB, &[], 1 << 30, &[]).is_none()); + } + + #[test] + fn nothing_is_protected_where_nothing_is_ram() { + assert!(largest(&[], 0, FOUR_GIB, &[], MIB, &[]).is_none()); + let ram = vec![(0x1000_0000, 0x2000_0000)]; + assert!(largest(&ram, 0, FOUR_GIB, &[(0, u64::MAX)], MIB, &[]).is_none()); + assert!(largest(&ram, FOUR_GIB, u64::MAX, &[], MIB, &[]).is_none()); + } +} diff --git a/kernel/pure/lib.rs b/kernel/pure/lib.rs index a78b30e8250..6bc8a883667 100644 --- a/kernel/pure/lib.rs +++ b/kernel/pure/lib.rs @@ -13,6 +13,7 @@ pub mod bootwrite; pub mod clock; pub mod cpu; pub mod dirnodes; +pub mod dmaguard; pub mod frames; pub mod iova; pub mod message; diff --git a/kernel/src/arch/aarch64/smmu/mod.rs b/kernel/src/arch/aarch64/smmu/mod.rs index d4e6c89a37d..e62c981607c 100644 --- a/kernel/src/arch/aarch64/smmu/mod.rs +++ b/kernel/src/arch/aarch64/smmu/mod.rs @@ -210,7 +210,13 @@ pub fn drivers_settled() {} /// The SMMUv3, armed; a refusal leaves every unit the IORT names aborting /// what it is given, or says why one could not be told to. -pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[toyos_abi::boot::RootBridgeWindow]>) { +/// `_kernel` is the VT-d hand-over's to guard; this backend has no use for it. +pub fn init( + rsdp_addr: u64, + devices: &[PciDevice], + windows: Option<&[toyos_abi::boot::RootBridgeWindow]>, + _kernel: (u64, u64), +) { let iort = match toyos_acpi::iort(direct_phys(), rsdp_addr) { Ok(iort) => iort, Err(IortRefused::Table(TableError::Absent)) => { diff --git a/kernel/src/arch/x86_64/iommu_unit.rs b/kernel/src/arch/x86_64/iommu_unit.rs index b34cbc47ba7..0eab569ca86 100644 --- a/kernel/src/arch/x86_64/iommu_unit.rs +++ b/kernel/src/arch/x86_64/iommu_unit.rs @@ -35,8 +35,8 @@ fn backend() -> Backend { const MAX_PHYS: u64 = 1 << 52; /// The units of whichever table firmware published. -pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeWindow]>) { - let found = if super::vtd::init(rsdp_addr, devices, windows) { +pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeWindow]>, kernel: (u64, u64)) { + let found = if super::vtd::init(rsdp_addr, devices, windows, kernel) { Backend::Vtd } else if super::amdvi::init(rsdp_addr, devices, windows) { Backend::AmdVi diff --git a/kernel/src/arch/x86_64/vtd/dmar.rs b/kernel/src/arch/x86_64/vtd/dmar.rs index 38ef40e19fd..e803ab85c74 100644 --- a/kernel/src/arch/x86_64/vtd/dmar.rs +++ b/kernel/src/arch/x86_64/vtd/dmar.rs @@ -22,7 +22,8 @@ const FLAGS: usize = 37; pub const FLAG_INTR_REMAP: u8 = 1 << 0; /// Flags bit 1: firmware asks the OS not to enable x2APIC mode. pub const FLAG_X2APIC_OPT_OUT: u8 = 1 << 1; -/// Flags bit 2: DMA blocked until the OS programs the unit; honoured for free by this kernel's ordering. +/// Flags bit 2, `DMA_CTRL_PLATFORM_OPT_IN_FLAG` (§8.1): firmware reports that +/// any DMA the platform initiates at hand-over is restricted to its RMRRs. pub const FLAG_DMA_CTRL_OPT_IN: u8 = 1 << 2; /// A list element whose declared size the list cannot hold; the last item a walk yields. diff --git a/kernel/src/arch/x86_64/vtd/fault.rs b/kernel/src/arch/x86_64/vtd/fault.rs index e5ea287014d..00584db8c3e 100644 --- a/kernel/src/arch/x86_64/vtd/fault.rs +++ b/kernel/src/arch/x86_64/vtd/fault.rs @@ -134,23 +134,7 @@ fn drain(index: usize, regs: Mmio, records: u64, count: u32) -> usize { let mut seen = 0usize; for i in 0..count { let record = records + i as u64 * 16; - let high = regs.read_u64(record + 8); - if high & (1u64 << 63) == 0 { - continue; - } - let stream = StreamId::pci( - (high >> 8) as u8, - ((high >> 3) & 0x1F) as u8, - (high & 0x7) as u8, - ); - let reason = ((high >> 32) & 0xFF) as u8; - let fault = Fault { - who: Who::Function(stream), - address: regs.read_u64(record) & !0xFFF, - access: if high & (1u64 << 62) != 0 { Access::Read } else { Access::Write }, - reason, - name: reason_name(reason), - }; + let Some(fault) = read_record(regs, record) else { continue }; let halts = policy::report(index, &UNITS[index].faults, fault); clear_record(regs, record); if halts { @@ -161,6 +145,23 @@ fn drain(index: usize, regs: Mmio, records: u64, count: u32) -> usize { seen } +/// The fault recording register at `record`, if its `F` bit is set. +fn read_record(regs: Mmio, record: u64) -> Option { + let high = regs.read_u64(record + 8); + if high & (1u64 << 63) == 0 { + return None; + } + let stream = StreamId::pci((high >> 8) as u8, ((high >> 3) & 0x1F) as u8, (high & 0x7) as u8); + let reason = ((high >> 32) & 0xFF) as u8; + Some(Fault { + who: Who::Function(stream), + address: regs.read_u64(record) & !0xFFF, + access: if high & (1u64 << 62) != 0 { Access::Read } else { Access::Write }, + reason, + name: reason_name(reason), + }) +} + fn clear_record(regs: Mmio, record: u64) { regs.write_u32(record + 12, RECORD_FAULT); } diff --git a/kernel/src/arch/x86_64/vtd/mod.rs b/kernel/src/arch/x86_64/vtd/mod.rs index 8acbe253d5d..bbe4cefaaf7 100644 --- a/kernel/src/arch/x86_64/vtd/mod.rs +++ b/kernel/src/arch/x86_64/vtd/mod.rs @@ -7,11 +7,15 @@ //! points every unit at one interrupt remapping table. A capability the kernel //! cannot use leaves the unit switched off rather than halting, logged by //! register value rather than a bare "unsupported". +//! +//! Every unit is handed over with translation off, and where every unit has +//! protected memory regions, inside them (`pmr`). pub mod dmar; pub mod domain; pub mod fault; pub mod interrupt; +mod pmr; mod queue; mod table; @@ -48,6 +52,9 @@ const FEUADDR_REG: u64 = 0x44; const IQH_REG: u64 = 0x80; const IQT_REG: u64 = 0x88; const IQA_REG: u64 = 0x90; +/// Register-based invalidation (§6.5.1); `IOTLB_REG` sits `ECAP.IRO` past the base. +const CCMD_REG: u64 = 0x28; +const IOTLB_REG_AFTER_IRO: u64 = 0x08; /// `GCMD` bits; `GSTS` confirms each at the same bit position. const TRANSLATION_ENABLE: u32 = 1 << 31; @@ -86,8 +93,8 @@ static TABLES: Lock = Lock::new(Tables::new()); /// queue: a queue has one tail, so a second holder would submit behind its back. static UNITS: Lock> = Lock::new(Vec::new()); -/// A unit past `enable`: its window, its queue, and the root table its context -/// entries live in. +/// A unit past its hand-over: its window, its queue, and the root table its +/// context entries live in. pub(super) struct Live { regs: Mmio, queue: Queue, @@ -119,7 +126,7 @@ pub(super) fn invalidate_interrupt_entries() { /// `windows` and every region firmware reserved are what no domain's addresses /// reach. `false` where firmware published no DMAR. -pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeWindow]>) -> bool { +pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeWindow]>, kernel: (u64, u64)) -> bool { let dmar = match Dmar::open(rsdp_addr) { Ok(dmar) => dmar, Err(TableError::Absent) => return false, @@ -151,6 +158,8 @@ pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeW // Described and planned before any unit is armed: whether sources may move // to the remappable format is one decision, taken before the first `IRE`. let mut ready: Vec<(Unit, Plan)> = Vec::new(); + // Every RMRR, which no protected region may cover during the hand-over. + let mut rmrrs: Vec<(u64, u64)> = Vec::new(); // What no domain's addresses may reach, RMRRs added below as the walk // meets them; nothing where no declaration is held. let mut reserved: Option> = windows @@ -204,6 +213,7 @@ pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeW if let Some(reserved) = reserved.as_mut() { reserved.push((base, limit.saturating_add(1))); } + rmrrs.push((base, limit.saturating_add(1))); describe_scopes("rmrr", regions, rmrr.scopes()); regions += 1; held_regions += usize::from(held); @@ -237,12 +247,93 @@ pub fn init(rsdp_addr: u64, devices: &[PciDevice], windows: Option<&[RootBridgeW // One identity-domain table set per address width: units may disagree on // `CAP.SAGAW`, and a shared set would be programmed at the wrong depth for one. let mut domains: [Option; 2] = [None, None]; - for (unit, plan) in ready { - enable(unit, plan, devices, &mut domains, remap); + // Before any table is built: what a domain of a driver's own can be is the + // narrowest thing every translating unit on this machine agrees to. + for (unit, plan) in &ready { + domain::unit_agrees(plan.width, unit.caps.domains(), unit.caps.mgaw()); + } + #[cfg(feature = "boot-actuators")] + if crate::actuator::iommu_firmware_left() { + for (unit, plan) in &ready { + let domain = identity_domain(&mut TABLES.lock(), &mut domains, plan.width); + unit.leave_on(devices, domain, plan.width); + } + } + for (unit, _) in &ready { + pmr::hand_over(unit.index, unit.regs); + } + // Built before the regions are chosen, so they can be chosen to hold them. + let ready: Vec<(Unit, Plan, Table)> = ready + .into_iter() + .map(|(unit, plan)| { + let mut tables = TABLES.lock(); + let domain = identity_domain(&mut tables, &mut domains, plan.width); + let root = identity_root(&mut tables, devices, domain, plan.width); + (unit, plan, root) + }) + .collect(); + let guard = { + let candidates: Vec = ready + .iter() + .map(|(unit, _, _)| pmr::Candidate { + index: unit.index, + regs: unit.regs, + regions: unit.caps.protected_memory(), + registers: unit.invalidates_through_registers(), + }) + .collect(); + // What the OS owns once firmware is done with it: the pmm's memory and + // ACPI's reclaimable tables, never firmware's NVS. + let ram: Vec<(u64, u64)> = crate::mm::firmware_map() + .iter() + .filter(|e| toyos_bootmap::is_usable_type(e.uefi_type) || e.uefi_type == toyos_bootmap::EFI_ACPI_RECLAIM_MEMORY) + .map(|e| (e.start, e.end)) + .collect(); + let cover: Vec<(u64, u64)> = TABLES.lock().pages().chain([kernel]).collect(); + if candidates.is_empty() { + None + } else { + pmr::program(&candidates, units, &ram, &rmrrs, &cover, crate::mm::pmm::top()) + } + }; + if let Some(guard) = &guard { + log_bus_masters(devices); + guard.arm(); + } + let via = if guard.is_some() { Invalidation::Registers } else { Invalidation::Queue }; + let translating: Vec = + ready.into_iter().map(|(unit, plan, root)| translate(unit, plan, root, via)).collect(); + if let Some(guard) = guard { + guard.release(); + } + for translating in translating { + finish(translating, devices, remap); } true } +/// Every function mastering the bus as the regions go on: what they block, +/// they block without a record (§11.4.8.1). +fn log_bus_masters(devices: &[PciDevice]) { + const BUS_MASTER: u16 = 1 << 2; + let masters: Vec = devices + .iter() + .filter(|d| d.read_config_u16(0x04) & BUS_MASTER != 0) + .map(|d| alloc::format!("{}", StreamId::pci(d.bus, d.dev, d.func))) + .collect(); + log!("iommu: functions mastering the bus as protected memory goes on: [{}]", masters.join(" ")); +} + +/// The identity domain at `width`, built the first time it is asked for. +fn identity_domain(tables: &mut Tables, domains: &mut [Option
; 2], width: AddressWidth) -> Table { + *domains[domain_slot(width)].get_or_insert_with(|| { + let top = crate::mm::pmm::top(); + let (domain, frames) = table::identity_domain(tables, width, top); + log!("iommu: identity domain aw={} covers 0x0..{top:#x} in {frames} 2 MiB leaves", width.bits()); + domain + }) +} + /// Whether every interrupt source may be moved to the remappable format, and /// with `EIME` if so; `None` leaves the machine exactly as it boots with no unit. /// @@ -301,6 +392,8 @@ struct Unit { base: u64, regs: Mmio, caps: Capabilities, + /// `VER_REG`'s major version. + major: u32, } impl Unit { @@ -357,6 +450,36 @@ impl Unit { } } + /// Whether this unit takes register-based invalidation: a unit of major + /// version 6 or later refuses every request (§11.4.6.1), and the IOTLB + /// registers must lie inside the window this kernel maps. + fn invalidates_through_registers(&self) -> bool { + self.major < 6 && self.caps.iotlb_registers() + 16 <= REGISTER_WINDOW + } + + /// A global context-cache invalidation, then a global IOTLB one draining + /// reads and writes, through the unit's registers (§6.5.1); valid only with + /// queued invalidation off, whatever `TES` says. A unit that reports either + /// done at no granularity refused it. + fn invalidate_through_registers(&self) { + const START: u64 = 1 << 63; + const GLOBAL_CONTEXT: u64 = 0b01 << 61; + const CONTEXT_DONE: u64 = 0b11 << 59; + const GLOBAL_IOTLB: u64 = 0b01 << 60; + const DRAIN: u64 = (1 << 49) | (1 << 48); + const IOTLB_DONE: u64 = 0b11 << 57; + let iotlb = self.caps.iotlb_registers() + IOTLB_REG_AFTER_IRO; + for (at, request, done, what) in [ + (CCMD_REG, START | GLOBAL_CONTEXT, CONTEXT_DONE, "the context-cache invalidation"), + (iotlb, START | GLOBAL_IOTLB | DRAIN, IOTLB_DONE, "the IOTLB invalidation"), + ] { + self.regs.write_u64(at, request); + self.wait(what, " done", || self.regs.read_u64(at) & START == 0); + let read = self.regs.read_u64(at); + assert!(read & done != 0, "iommu: unit{} refused {what}: {read:#018x}", self.index); + } + } + /// Every field of [`FIRMWARE_LEFT`] that is on, off. fn hand_over(&self) { for (bit, what) in FIRMWARE_LEFT { @@ -400,7 +523,7 @@ fn describe_unit(index: usize, drhd: &dmar::Drhd) -> Option { log!( "iommu: unit{index} @{base:#x} seg={} pci_all={} ver={}.{} cap={:#018x} ecap={:#018x} \ gsts={:#010x} aw={} sagaw={:#04x} mgaw={} nd={} sps2m={} cm={} psi={} nfr={} fro={:#x} \ - qi={} ir={} eim={} pt={} coherent={} sc={}", + qi={} ir={} eim={} pt={} coherent={} sc={} pmr={}", drhd.segment(), yn(drhd.include_pci_all()), (version >> 4) & 0xF, @@ -427,10 +550,11 @@ fn describe_unit(index: usize, drhd: &dmar::Drhd) -> Option { yn(caps.passthrough()), yn(caps.coherent()), yn(caps.snoop_control()), + yn(caps.protected_memory()), ); describe_scopes("unit", index, drhd.scopes()); - Some(Unit { index, base, regs, caps }) + Some(Unit { index, base, regs, caps, major: (version >> 4) & 0xF }) } /// What a unit's capabilities let this kernel program, or `None` with the @@ -488,59 +612,70 @@ fn plan(unit: &Unit) -> Option { Some(Plan { width, records }) } -/// Programs the unit in its own order — queue, root pointer, interrupt remap -/// pointer, global invalidation, then `TE` — each confirmed in `GSTS` before -/// the next. -fn enable( +/// How a hand-over invalidates after pointing the unit at this kernel's root. +#[derive(Clone, Copy)] +enum Invalidation { + /// Through the unit's registers, inside protected memory: the queue's + /// memory is no remapping structure (Appendix A, `ECAP.C`), so the regions + /// may cover it. + Registers, + /// Through the queue, enabled first. + Queue, +} + +/// A unit translating through this kernel's root, its interrupt remapping not +/// yet programmed. +struct Translating { unit: Unit, - plan: Plan, - devices: &[PciDevice], - domains: &mut [Option
; 2], - remap: Option, -) { - let index = unit.index; - let Plan { width, records } = plan; - // Before any table is built: what a domain of a driver's own can be is the - // narrowest thing every translating unit on this machine agrees to. - domain::unit_agrees(width, unit.caps.domains(), unit.caps.mgaw()); + width: AddressWidth, + root: Table, + /// Enabled already where the hand-over invalidated through it. + queue: Option, +} - let domain = *domains[domain_slot(width)].get_or_insert_with(|| { - let top = crate::mm::pmm::top(); - let (domain, frames) = table::identity_domain(&mut TABLES.lock(), width, top); - log!("iommu: identity domain aw={} covers 0x0..{top:#x} in {frames} 2 MiB leaves", width.bits()); - domain - }); - #[cfg(feature = "boot-actuators")] - if crate::actuator::iommu_firmware_left() { - unit.leave_on(devices, domain, width); +/// Whatever firmware left on, off; then `root` pointed, the global +/// invalidation §6.6 owes it, and translation on. +fn translate(unit: Unit, plan: Plan, root: Table, via: Invalidation) -> Translating { + let Plan { width, records } = plan; + // After the tables are built, so a unit firmware left translating passes + // DMA untranslated only while it is programmed, and before the queue is + // pointed, which §6.5.2 does only with it off. + unit.hand_over(); + // Before `TE`: the first blocked transaction must be reportable, not merely counted. + fault::arm(unit.index, unit.regs, records, crate::arch::idt::DMA_FAULT_VECTOR); + let mut queue = match via { + Invalidation::Registers => None, + Invalidation::Queue => Some(enable_queue(&unit)), + }; + unit.regs.write_u64(RTADDR_REG, root.phys()); + unit.command(SET_ROOT_TABLE_POINTER, true, "the root table pointer"); + match queue.as_mut() { + Some(queue) => queue.invalidate_all(unit.regs), + None => unit.invalidate_through_registers(), } - let root = identity_root(&mut TABLES.lock(), devices, domain, width); + unit.command(TRANSLATION_ENABLE, true, "translation"); + Translating { unit, width, root, queue } +} - // After the tables are built, so a unit firmware left translating passes DMA - // untranslated only while it is programmed - // (`issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md`), - // and before the queue is pointed, which §6.5.2 does only with it off. - unit.hand_over(); - let mut queue = Queue::new(&mut TABLES.lock(), unit.regs); +fn enable_queue(unit: &Unit) -> Queue { + let queue = Queue::new(&mut TABLES.lock(), unit.regs); + unit.command(QUEUED_INVALIDATION_ENABLE, true, "queued invalidation"); + queue +} + +/// The queue where it is not on yet, then interrupt remapping, each confirmed +/// in `GSTS` before the next. +fn finish(translating: Translating, devices: &[PciDevice], remap: Option) { + let Translating { unit, width, root, queue } = translating; + let index = unit.index; + let mut queue = queue.unwrap_or_else(|| enable_queue(&unit)); // Outside the `TABLES` lock: `interrupt::arm` takes its own lock and // then that one, and the order this subsystem holds is the reverse. let irta = remap.map(interrupt::arm); - - // Before `TE`: the first blocked transaction must be reportable, not merely counted. - fault::arm(index, unit.regs, records, crate::arch::idt::DMA_FAULT_VECTOR); - - unit.command(QUEUED_INVALIDATION_ENABLE, true, "queued invalidation"); - unit.regs.write_u64(RTADDR_REG, root.phys()); - unit.command(SET_ROOT_TABLE_POINTER, true, "the root table pointer"); if let Some(irta) = irta { unit.regs.write_u64(interrupt::IRTA_REG, irta); unit.command(interrupt::SET_TABLE_POINTER, true, "the interrupt remap table pointer"); queue.invalidate_interrupts(unit.regs); - } - queue.invalidate_all(unit.regs); - - unit.command(TRANSLATION_ENABLE, true, "translation"); - if remap.is_some() { unit.command(interrupt::INTERRUPT_REMAPPING_ENABLE, true, "interrupt remapping"); } // After `IRE`, so nothing is invalidated on a unit that is not yet reading @@ -679,6 +814,16 @@ impl Capabilities { self.cap & (1 << 39) != 0 } + /// `CAP.PLMR` and `CAP.PHMR` (bits 5 and 6): both protected memory regions. + fn protected_memory(&self) -> bool { + self.cap & (0b11 << 5) == 0b11 << 5 + } + + /// `ECAP.IRO`, in 16-byte units: where the IOTLB registers start. + fn iotlb_registers(&self) -> u64 { + ((self.ecap >> 8) & 0x3FF) * 16 + } + /// `CAP.NFR`, encoded one less than it is. fn fault_recording_registers(&self) -> u32 { (((self.cap >> 40) & 0xFF) + 1) as u32 @@ -694,7 +839,7 @@ impl Capabilities { self.ecap & (1 << 0) != 0 } - /// `ECAP.QI`; absent falls back to `CCMD_REG`/`IOTLB_REG`, correct but slower. + /// `ECAP.QI`; a unit without it is left unprogrammed (`plan`). fn queued_invalidation(&self) -> bool { self.ecap & (1 << 1) != 0 } diff --git a/kernel/src/arch/x86_64/vtd/pmr.rs b/kernel/src/arch/x86_64/vtd/pmr.rs new file mode 100644 index 00000000000..d78d241943f --- /dev/null +++ b/kernel/src/arch/x86_64/vtd/pmr.rs @@ -0,0 +1,182 @@ +//! Protected memory regions (VT-d §11.4.8). On a unit of major version 2 or +//! later, and on every one since Skylake, they block DMA to the memory they +//! cover whether or not the unit translates, while the unit's own reads of its +//! remapping structures are exempt (§11.4.8.1, `EPM`). So holding them from +//! before the first unit's translation goes off until the last unit translates +//! keeps the hand-over's tables, and this kernel, out of every device's reach. +//! +//! The same regions go on every unit, as §11.4.8.1 requires for them to +//! protect anything. They avoid every RMRR, so firmware's DMA to its reserved +//! memory runs on through the hand-over. + +use alloc::vec::Vec; + +use crate::log; +use crate::mm::Mmio; +use crate::time::{Duration, Tripwire}; +use kernel::dmaguard::{self, Span}; + +const PMEN_REG: u64 = 0x64; +const PLMBASE_REG: u64 = 0x68; +const PLMLIMIT_REG: u64 = 0x6C; +const PHMBASE_REG: u64 = 0x70; +const PHMLIMIT_REG: u64 = 0x78; + +/// `PMEN.EPM`, and `PMEN.PRS` reporting it in force. +const ENABLE: u32 = 1 << 31; +const STATUS: u32 = 1 << 0; + +const STATUS_TIMEOUT: Tripwire = Tripwire::absurd( + Duration::from_secs(1), + "a unit whose protected regions never report their state is one whose reach nothing can state", +); + +/// Protected memory firmware left on, off: its bases and limits may not be +/// rewritten while it is (§11.4.8.2-§11.4.8.5), and left on it would block +/// DMA after the hand-over. +pub fn hand_over(index: usize, regs: Mmio) { + if regs.read_u32(PMEN_REG) & STATUS != 0 { + log!("iommu: unit{index} was handed over with protected memory on; it goes off first"); + set(index, regs, false); + } +} + +/// One unit the hand-over would guard. +pub struct Candidate { + pub index: usize, + pub regs: Mmio, + /// `CAP.PLMR` and `CAP.PHMR` both. + pub regions: bool, + /// It invalidates through its registers, which the window needs. + pub registers: bool, +} + +/// Every unit's protected regions, programmed and not yet on. +pub struct Guard { + units: Vec<(usize, Mmio)>, +} + +/// The same regions programmed on every unit, covering every `cover` range, +/// or `None` with the reason logged: the hand-over is then unguarded. `ram` is +/// what firmware's map says the OS owns, `avoid` every RMRR, `top` the end of +/// what the identity domain maps. +pub fn program( + units: &[Candidate], + described: usize, + ram: &[(u64, u64)], + avoid: &[(u64, u64)], + cover: &[(u64, u64)], + top: u64, +) -> Option { + fn unguarded(why: core::fmt::Arguments) -> Option { + log!("iommu: the hand-over is unguarded: {why}, so every function reaches all of memory while the units are programmed"); + None + } + if units.len() < described { + return unguarded(format_args!( + "{} of {described} units are programmed, and the regions protect only where every unit holds them", + units.len() + )); + } + if let Some(unit) = units.iter().find(|u| !u.regions) { + return unguarded(format_args!("unit{} has no protected memory regions (CAP.PLMR or CAP.PHMR clear)", unit.index)); + } + if let Some(unit) = units.iter().find(|u| !u.registers) { + return unguarded(format_args!( + "unit{} cannot invalidate through its registers, and its queue's memory may lie under the regions", + unit.index + )); + } + // The coarsest alignment any base or limit decodes (§11.4.8.2-§11.4.8.5), + // so one pair of regions suits every unit. + let mut low_align = 1u64; + let mut high_align = 1u64; + for unit in units { + let regs = unit.regs; + regs.write_u32(PLMBASE_REG, u32::MAX); + regs.write_u32(PLMLIMIT_REG, u32::MAX); + regs.write_u64(PHMBASE_REG, u64::MAX); + regs.write_u64(PHMLIMIT_REG, u64::MAX); + let low = [regs.read_u32(PLMBASE_REG), regs.read_u32(PLMLIMIT_REG)]; + let high = [regs.read_u64(PHMBASE_REG), regs.read_u64(PHMLIMIT_REG)]; + if low.contains(&0) || high.contains(&0) { + return unguarded(format_args!( + "unit{} decodes no protected region (PLMBASE/LIMIT={low:x?} PHMBASE/LIMIT={high:x?})", + unit.index + )); + } + for bits in low { + low_align = low_align.max(1 << bits.trailing_zeros()); + } + for bits in high { + high_align = high_align.max(1 << bits.trailing_zeros()); + } + } + let low = dmaguard::largest(ram, 0, dmaguard::FOUR_GIB, avoid, low_align, cover); + let high = dmaguard::largest(ram, dmaguard::FOUR_GIB, top, avoid, high_align, cover); + log!( + "iommu: protected memory for the hand-over on {} unit(s): low {} high {}, aligned to {low_align:#x} and {high_align:#x}", + units.len(), + Shown(low), + Shown(high), + ); + for &range in cover { + if !low.is_some_and(|s| s.holds(range)) && !high.is_some_and(|s| s.holds(range)) { + return unguarded(format_args!("{:#x}..{:#x}, which it must protect, lies outside both regions", range.0, range.1)); + } + } + log!("iommu: the regions cover the hand-over's tables and this kernel: {:x?}", cover); + for unit in units { + let regs = unit.regs; + // A limit below its base is a region switched off (§11.4.8.3, §11.4.8.5). + let (base, limit) = low.map_or((low_align, 0), |s| (s.base, s.end - 1)); + regs.write_u32(PLMBASE_REG, base as u32); + regs.write_u32(PLMLIMIT_REG, limit as u32); + let (base, limit) = high.map_or((high_align, 0), |s| (s.base, s.end - 1)); + regs.write_u64(PHMBASE_REG, base); + regs.write_u64(PHMLIMIT_REG, limit); + } + Some(Guard { units: units.iter().map(|u| (u.index, u.regs)).collect() }) +} + +impl Guard { + pub fn arm(&self) { + for &(index, regs) in &self.units { + set(index, regs, true); + } + log!("iommu: protected memory on for {} unit(s)", self.units.len()); + } + + pub fn release(self) { + for &(index, regs) in &self.units { + set(index, regs, false); + } + log!("iommu: protected memory released on {} unit(s)", self.units.len()); + } +} + +/// §11.4.8.1: `EPM` written, then `PRS` waited for before it changes again. +fn set(index: usize, regs: Mmio, on: bool) { + regs.write_u32(PMEN_REG, if on { ENABLE } else { 0 }); + let deadline = crate::clock::nanos_since_boot() + STATUS_TIMEOUT.nanos(); + while (regs.read_u32(PMEN_REG) & STATUS != 0) != on { + assert!( + crate::clock::nanos_since_boot() < deadline, + "iommu: unit{index}'s protected regions never reported {}, PMEN={:#010x}", + if on { "on" } else { "off" }, + regs.read_u32(PMEN_REG) + ); + core::hint::spin_loop(); + } +} + +struct Shown(Option); + +impl core::fmt::Display for Shown { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self.0 { + Some(s) => write!(f, "{:#x}..{:#x}", s.base, s.end), + None => f.write_str("none"), + } + } +} diff --git a/kernel/src/arch/x86_64/vtd/queue.rs b/kernel/src/arch/x86_64/vtd/queue.rs index 1c4c106ade4..22230e98582 100644 --- a/kernel/src/arch/x86_64/vtd/queue.rs +++ b/kernel/src/arch/x86_64/vtd/queue.rs @@ -1,9 +1,9 @@ //! Queued invalidation: submits a descriptor and an Invalidation Wait //! descriptor, then polls the wait's status write to completion. //! Acknowledgement timeout is a panic — the kernel cannot say what a device -//! can reach past it. Only the queued-invalidation path is implemented: every -//! unit reachable here has it, so a `CCMD_REG`/`IOTLB_REG` fallback would be -//! dead code; a unit without `ECAP.QI` is left unprogrammed. +//! can reach past it. A hand-over inside protected memory invalidates through +//! the unit's registers instead, before the queue is enabled. A unit without +//! `ECAP.QI` is left unprogrammed. use crate::mm::Mmio; use crate::time::{Duration, Tripwire}; diff --git a/kernel/src/arch/x86_64/vtd/table.rs b/kernel/src/arch/x86_64/vtd/table.rs index 115414e579f..ed913d9689e 100644 --- a/kernel/src/arch/x86_64/vtd/table.rs +++ b/kernel/src/arch/x86_64/vtd/table.rs @@ -46,6 +46,14 @@ impl Tables { Self { pages: Vec::new(), used: PAGE_2M as usize } } + /// Every 2 MiB page tables have been carved from, `[start, end)` physical. + pub fn pages(&self) -> impl Iterator + '_ { + self.pages.iter().map(|page| { + let start = page.direct_map().phys(); + (start, start + PAGE_2M) + }) + } + /// Returns one zeroed 4 KiB table, usable as a root, context, second-level, or invalidation-queue table. pub fn alloc(&mut self) -> Table { if self.used + TABLE_BYTES > PAGE_2M as usize { diff --git a/kernel/src/iommu/mod.rs b/kernel/src/iommu/mod.rs index 96397140197..4a5ef200fc1 100644 --- a/kernel/src/iommu/mod.rs +++ b/kernel/src/iommu/mod.rs @@ -278,12 +278,15 @@ impl core::fmt::Display for StreamId { /// /// No domain's addresses reach into one of `windows`, the root bridges' /// declared memory, or `None` where no declaration is held ([`window`]). +/// `kernel` is this kernel's image, `[start, end)` physical: what a unit's +/// hand-over keeps out of every device's reach where it can. pub fn init( rsdp_addr: u64, devices: &[crate::drivers::pci::PciDevice], windows: Option<&toyos_pci::aperture::Declaration>>, + kernel: (u64, u64), ) { - unit::init(rsdp_addr, devices, windows.map(|declared| declared.windows())); + unit::init(rsdp_addr, devices, windows.map(|declared| declared.windows()), kernel); } /// Once, after every kernel driver has taken its functions from firmware and diff --git a/kernel/src/main.rs b/kernel/src/main.rs index ce9eda9a7b5..f4e21e76433 100644 --- a/kernel/src/main.rs +++ b/kernel/src/main.rs @@ -432,7 +432,12 @@ pub(crate) unsafe extern "C" fn kernel_main(loader_args: &mut KernelArgs) -> ! { } // After ACPI is readable and PCI is enumerable, before any driver `init`: each enumerated device needs a context entry before it can DMA. // Refuses nothing — a machine with no usable IOMMU boots exactly as one without it. - iommu::init(kernel_args.rsdp_addr, &pci_devices, pcidev::declared().as_ref()); + iommu::init( + kernel_args.rsdp_addr, + &pci_devices, + pcidev::declared().as_ref(), + (kernel_args.kernel_memory_addr, kernel_args.kernel_memory_addr + kernel_args.kernel_memory_size), + ); // Before storage and everything under it: what it covers is the rest of this // boot, and a wedge down there is the reason to have one. arch::watchdog::init(&pci_devices); diff --git a/lib/bootmap/src/lib.rs b/lib/bootmap/src/lib.rs index 689c5ee6c02..c9884816552 100644 --- a/lib/bootmap/src/lib.rs +++ b/lib/bootmap/src/lib.rs @@ -105,7 +105,7 @@ pub const EFI_LOADER_DATA: u32 = 2; const EFI_BOOT_SERVICES_CODE: u32 = 3; const EFI_BOOT_SERVICES_DATA: u32 = 4; const EFI_CONVENTIONAL_MEMORY: u32 = 7; -const EFI_ACPI_RECLAIM_MEMORY: u32 = 9; +pub const EFI_ACPI_RECLAIM_MEMORY: u32 = 9; const EFI_ACPI_MEMORY_NVS: u32 = 10; /// Whether a UEFI memory type becomes free RAM the pmm hands out. diff --git a/tests/common/iommu.rs b/tests/common/iommu.rs index 3d05f046a5f..20a5cafd8e7 100644 --- a/tests/common/iommu.rs +++ b/tests/common/iommu.rs @@ -4,6 +4,8 @@ //! over behind the unit and refused where no unit is. //! `amdvi_firmware_left`: whether an AMD-Vi unit firmware left blocking is //! switched off before the boot stick is driven. +//! `vtd_firmware_left`: whether a VT-d unit firmware left translating, with no +//! protected memory regions, goes off before it is programmed. use std::collections::{BTreeMap, BTreeSet}; use std::path::Path; @@ -345,6 +347,31 @@ pub(crate) fn amdvi_left_on(unit: &str) -> [String; 3] { .map(|field| format!("iommu: amdvi {unit} was handed over with {field} on; it goes off")) } +/// **A VT-d unit left translating, on a unit with no protected memory +/// regions, goes off first and is programmed.** QEMU's unit reports +/// `CAP.PLMR` and `CAP.PHMR` clear, so this is the hand-over without the guard +/// the T14 takes (`iommu_firmware_left`'s row), logged as such. Under +/// `iommu-firmware-left` the unit is left queueing, translating and remapping; +/// each goes off by its own write, the root is set and invalidated through +/// the queue, and the unit ends translating with the boot stick's DMA behind it. +pub fn vtd_firmware_left(test_config: &Path) -> Result<(), String> { + let options = BootOptions { profile: Profile::Metal, kernel_params: &["iommu-firmware-left"], ..Default::default() }; + let qemu = QemuInstance::boot_with_options(test_config, &[], &[], options); + let log = Serial::boot(&qemu); + log.must_be_clean()?; + let unguarded = log.must_say("iommu: the hand-over is unguarded: unit0 has no protected memory regions")?; + let off = log.must_say("iommu: unit0 was handed over with translation on; it goes off first")?; + log.must_say("iommu: unit0 was handed over with queued invalidation on; it goes off first")?; + log.must_not_say("iommu: protected memory on for")?; + let translating = log.must_say(" translating gsts=")?; + if !translating.contains(" tes=y ") { + return Err(format!("unit0 was programmed and is not translating: {translating:?}")); + } + log.must_say("Boot: complete")?; + eprintln!(" [vtd-firmware-left] {}; {}; {}", unguarded.trim(), off.trim(), translating.trim()); + Ok(()) +} + /// **A unit an AMD machine's firmware left on is switched off and /// programmed.** QEMU's `amd-iommu` is the only AMD-Vi unit the nightly guest /// suite runs: the T14 is an Intel machine, and the hand-over's register diff --git a/tests/toyos.rs b/tests/toyos.rs index 56c71fdc049..2e045eb87ab 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -482,6 +482,9 @@ const MACHINE_TESTS: &[&str] = &[ // DMA it lets through are no host test's, and the AMD bench is staged by // hand rather than by CI. "amdvi_firmware_left", + // A VT-d unit left translating, with no protected memory regions to guard + // its hand-over: the T14's units have them, so only QEMU's takes this one. + "vtd_firmware_left", // A translating AMD-Vi unit's refusal, recorded and read: a kernel-owned // record halts the machine, so no metal row reaches it, and only QEMU's // `iommu-testdev` aims a DMA where no domain maps. @@ -1300,7 +1303,7 @@ const METAL: &[(&str, metal::Metal)] = &[ // log running, on real silicon: each field goes off by its own write // before the unit is programmed or left off, and the boot goes on. "iommu_firmware_left", - metal::Metal { arms: SELFTESTS, judge: |b| iommu_firmware_left(b[0].kernel().text()) }, + metal::Metal { arms: FIRMWARE_LEFT, judge: |b| iommu_firmware_left(b[0].kernel().text()) }, ), // ---- a claimed function at the unit, on tests/images/testcases.toml ---- ( @@ -1803,7 +1806,6 @@ const SELFTEST_PARAMS: &[&str] = &[ "lapic-spurious-selftest", "unclaimed-vector-selftest", "sysret-ss-probe", - "iommu-firmware-left", ]; const SELFTESTS: &[metal::Arm] = &[metal::Arm { @@ -1811,6 +1813,15 @@ const SELFTESTS: &[metal::Arm] = &[metal::Arm { ..metal::once("shared-debug", "tests/images/testcases.toml", SELFTEST_PARAMS, &[]) }]; +/// Its own boot: on the T14 the hand-over it stages can halt the machine on +/// the iGPU's fault +/// (`issues/the-t14s-igpu-faults-read-permission-after-translation-goes-off-and-on-under-its-scanout.md`), +/// which would take every row batched with it. +const FIRMWARE_LEFT: &[metal::Arm] = &[metal::Arm { + features: ACTUATOR_KERNEL, + ..metal::once("iommu-firmware-left", "tests/images/testcases.toml", &["iommu-firmware-left"], &[]) +}]; + /// The two boots every discovered member rides on the T14, each behind its /// rows' jobs: `testcases`, the shipping kernel's, carries every discovered /// Rust binary, then the C corpus, then [`LAST_MEMBERS`]; `shared-debug` @@ -4425,6 +4436,7 @@ fn run_machine_test(name: &str, test_config: &Path) -> Result<(), String> { common::iommu::the_kernel_tells_a_bypassing_console_physical_addresses() } "amdvi_firmware_left" => common::iommu::amdvi_firmware_left(test_config), + "vtd_firmware_left" => common::iommu::vtd_firmware_left(test_config), "amdvi_selftest" => common::iommu::amdvi_selftest(test_config), "nested_nmi_is_loud" => faults::nested_nmi_is_loud(test_config), "kernel_wx_leaf_is_refused" => faults::kernel_wx_leaf_is_refused(test_config, qemu::Profile::Metal), @@ -5977,11 +5989,16 @@ fn sysret_ss(log: &str) -> Result<(), String> { } /// Every unit this kernel went on to program, and every AMD-Vi unit it -/// switched off, said it was handed over with what the actuator left on. +/// switched off, said it was handed over with what the actuator left on; and +/// where every VT-d unit has protected memory regions, as the T14's do, they +/// went on before the hand-over and off after it. /// /// An AMD-Vi unit's lines are on the boot stick only because the switch-off /// let the stick's DMA through the unit the actuator left blocking it. fn iommu_firmware_left(log: &str) -> Result<(), String> { + // VT-d unit lines alone: an AMD-Vi unit's read `iommu: amdvi unit`. + let vtd: Vec<&str> = log.lines().filter(|l| l.contains("iommu: unit") && l.contains(" cap=")).collect(); + let guarded = !vtd.is_empty() && vtd.iter().all(|l| l.contains(" pmr=y")); let mut units = 0; for line in log.lines().filter(|l| l.contains(" translating gsts=")) { let unit = line @@ -6030,10 +6047,25 @@ fn iommu_firmware_left(log: &str) -> Result<(), String> { if units == 0 { return Err(format!("no unit was programmed or switched off, so nothing was handed over:\n{log}")); } + let at = |needle: &str| log.lines().position(|l| l.contains(needle)); + let armed = at("iommu: protected memory on for"); + let first_off = at("was handed over with translation on; it goes off first"); + let released = at("iommu: protected memory released on"); + let first_translating = at(" translating gsts="); + if guarded + && !matches!((armed, first_off, released, first_translating), + (Some(a), Some(o), Some(r), Some(t)) if a < o && r < t) + { + return Err(format!( + "every unit has protected memory regions, and they did not go on before translation \ + first went off and come off before the first unit was programmed:\n{log}" + )); + } let passed = log.matches("handed over with compatibility-format pass-through on").count(); eprintln!( - " [iommu-firmware-left] {units} unit(s) switched off what they were handed over with; \ - {passed} of them reported compatibility format passed" + " [iommu-firmware-left] {units} unit(s) switched off what they were handed over with, \ + {} protected memory; {passed} of them reported compatibility format passed", + if guarded { "inside" } else { "without" } ); Ok(()) }