[Japabu]

Stacked on #1277: the diff is this PR's commit alone, and it lands after #1277.

Problem

kernel::mitigation::decide names SSBD and STIBP for a program that asks, and an IBPB between two programs one of which asks. That is ToyOS's policy for protections a vendor prescribes for every program (#1243 and the manual's Trust page). Nothing applied them: no program could ask, and no switch read or wrote a speculation control (issues/no-program-runs-with-speculative-store-bypass-disabled.md, issues/no-entry-or-switch-clears-the-bhb-fills-the-rsb-or-issues-an-ibpb.md).

What changes

• How a program asks: isolated-speculation = true on a system.toml row.
  • The manifest carries it, and the supervisor starts the row with CommandExt::isolated_speculation. SpawnArgs::isolated_speculation carries it to the kernel: 1 asks, 0 asks nothing of its own, any other value is InvalidArgument.
  • A child asks as the program that spawned it does, or because its own spawn asked (Speculation::of_child).
  • A launch runs as its own row says, so the SDK refuses a launch that names the ask.
  • Any spawner may ask for its child, as with Linux's prctl: the ask is a protection, not a privilege.
• What the CPU applies: Decision::on_request names SSBD and STIBP in IA32_SPEC_CTRL, AMD's VIRT_SPEC_CTRL SSBD for a guest that enumerates only that, an IBPB, and the fill after it. Each comes only where the CPU enumerates it. install(), called beside report(), gives each CPU its set and the IA32_SPEC_CTRL its bring-up read back, once SMT is known and before any program runs.
• At a switch: arch::x86_64::mitigation::switch_to runs beside the port bitmap's.
  • It sets the asked bits while a thread of a program that asked runs, and clears them for any other thread and for idle, as Intel asks of STIBP and SSBD before a sleep state.
  • It writes a register only when the ask changes and the CPU's decision names a bit.
  • Cost where nothing is asked: on a CPU whose decision names nothing on request, a switch pays the per-CPU lookup, one load and one predictable branch, and returns. On a CPU that names something but where no program asks, it pays the lookup, two loads and about five predictable branches, with no store and no MSR access.
• IBPB: issued on a switch into a program's address space when the space last run on this CPU is a different one and either side asked (LastAsked).
  • A pid identifies the space. The process table's allocator (toyos_proclife::pids) issues no pid twice for the machine's life, and a process keeps one space; the IBPB site cites that.
  • The host test holds two IBPBs over B, C, idle, B, A, idle, A, B, where only A asked.
• The return predictor's fill: AMD-SB-1040 says an IBPB on an AMD CPU without IBPB_RET (APM Vol. 3 Fn8000_0008_EBX[30]) can leave the return address predictor's entries, and AMD asks for 70586's V2-3 fill after it. The decision names "return address predictor filled after IBPB" there, enumerated together with the IBPB it follows, and the switch follows each IBPB with 32 calls. Intel's IBPB already covers the RSB, so for the CPUs the decision covers no other vendor prescribes a fill on a switch. The record's exit is restated to match, as approved by the orchestrator.
• The boot lines say SSBD, STIBP, IBPB and the fill are mitigated wherever the CPU enumerates them. No register can show them before a program asks.
• Manual: the Trust page now names SSBD among what a program that asks gets, and says how a program asks: through its row, and any program may ask for a child it starts.
• AArch64 applies nothing on request; that is recorded in issues/no-aarch64-program-runs-with-speculative-store-bypass-disabled.md.

Tests

• speculation_on_request, a member of the shared-debug boot (it needs SYS_DEBUG, so it runs on the test kernel only). It reads IA32_SPEC_CTRL through three new test-only debug actions:
  • the asked bits are clear for itself;
  • they are set for speculationprobe, a test row that asks, and for the child the probe starts without asking;
  • beside the probe on one CPU, that CPU issues at least 2 IBPBs, and back on it afterwards this job runs with the bits clear again;
  • beside a child of its own that does not ask, the CPU issues none.
  • It needs a guest because the oracle is the CPU's own register, read in Ring 0 while the program runs.
  • Targets: CPUs that enumerate IBPB and SSBD, namely both benches and the KVM guest runner. Where CPUID enumerates no IBPB, as under TCG's qemu64, it says so and holds nothing; the registration states this.
• Host: an_ibpb_goes_between_two_spaces_one_of_which_asked, a_cpu_applies_on_request_what_its_decision_names and a_child_asks_as_its_spawner_or_its_start. The manifest round trip now carries the ask, and the SDK's shape tests refuse a launch that asks. The spawn abuse test now refuses an ask of 2.

Checks

Head 1ab7a9b0b, on #1277's 98c605c46, rebased onto main 302fb3698. #1252 rewrote the spawn path: build_child_handles takes the job and no labels, and the ask still rides PendingHandles beside the pager. Round 2 answers review r1's B1 to B4 and suggestions S1 to S7.

• Host: the kernel's mitigation tests, the harness's mitigation checks and clippy (26 invocations) are clean at this head. The full cargo run -- --ci host ran at the top of the stack, Every entry from Ring 3 clears branch history where the CPU's decision names it #1279's c07ac2ef3, which carries this branch: exit 1, 122 of 123 steps green; the red is the build system, the recorded macOS llvm::tests failure (issues/the-llvm-environment-test-breaks-git-on-macos.md). Round 1's run at 50c9da201 exited 1 with 122 of 123 green; the red is the recorded macOS llvm::tests failure.
• Images: the shipping image and the testcases image on the test kernel both build. Every guest binary whose spawn arguments changed, including the members A child's handles are a list its parent names: labels and namespaces leave the kernel #1252 adapted, builds for x86_64-unknown-toyos.
• Negative controls:
  • Deciding the IBPB by the previous space alone reds an_ibpb_goes_between_two_spaces_one_of_which_asked with 1 IBPB instead of 2.
  • Guest control run 43168182421, on branch mitigate-ask-control-inherit at this head plus of_child ignoring the spawner's ask, must red speculation_on_request. It is queued.
• Guest: run 43168163441 is queued: mitigations_in_a_guest, shared_debug_members_exit_0 and testcases_members_exit_0 on x86-64, and on AArch64 virt_user_mode, virt_first_entry, virt_jobs_at_el2 and virt_smp, since the loader and the switch are shared code. Runs at earlier heads (43156124553, 43156167715, 43165353231, 43165372000) were cancelled. The heads moved once more after the host round: mitigations_in_a_guest now prints every line (masked as before), so a green run shows the guest's verdicts too; only tests/toyos.rs differs from the heads the host checks ran at.

Evidence: round 1 (host), https://github.com/ToyOSOrg/evidence/tree/4aefff57e0efe9d5cd99ff41c89859988d7673d0/r1. Round 2 (host), https://github.com/ToyOSOrg/evidence/tree/11840d907c119513a48c5971ad125936c8692bff/r2. The guest round follows.

Owed on metal (benches down)

Recorded in the two records above:

• speculation_on_request, a member of main's shared-debug boot, on the T14 and the AMD bench. Main must show it exit 0, naming 0x4 asked on the T14 and 0x6 (SSBD and STIBP) on the AMD bench, with at least 2 IBPBs beside the probe and none beside the child that did not ask;
• the same boot with of_child ignoring the spawner's ask, which must red it.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1278 at 50c9da2

Architecture. The ask is a kernel fact and an ABI field. The kernel validates it once in SYS_SPAWN and turns it into a Speculation there. It binds every child the program spawns itself, and it rides each thread's switch context, as KernelCtx requires for anything Hw::switch loads. The manifest, supervisor and SDK only carry it, and the SDK refuses it on a launch, so a launch is decided by its own row. The pure library owns both the per-CPU set applied on request (on_request) and the IBPB state machine (LastAsked), which is host-tested. The arch module only applies them. This matches the approved plan: SSBD and STIBP when the ask changes, an IBPB on a change of address space when either side asked, and the AMD-SB-1040 fill where IBPB_RET is clear.

Correctness, traced:

• Every entry into a user thread goes through Hw::switch, and so through switch_to. That includes a first run, a preemption, a migration and an exit from idle. loaded tracks this CPU's register exactly. Nothing else writes 0x48 after bring-up: there is no suspend, no late microcode load, and the IBRS-on-entry work is not built. install runs in report(), before spawn_supervisor, while every AP is still idle. So a thread that asked cannot run with SSBD clear on a CPU whose decision names SSBD.
• LastAsked gives the expected barriers for B, C, idle, B, A, idle, A, B and for A, D, A, idle, A, A. The pid invariant holds. Pids::take never reissues a pid. A given-back pid was never held by an entry, so it never ran. Pid::MAX is never issued. There is no exec.
• The fill matches the standard sequence. It is 32 calls, each followed by int3, then add rsp, 256. Every entry it plants points at an int3, and it runs only after an IBPB on a CPU that names the fill.
• The switch-path cost is small. With nothing asked, it is the cpu_local::here() lookup, two more loads (last_asked, loaded) and about five predictable branches, with no stores and no MSR access. That is comparable to Linux's per-switch cond_mitigation bookkeeping, but it is not what the PR body claims (B5).

Size: about +400 lines of production code (kernel about +330, ABI/SDK/manifest/supervisor about +70) and about +235 of tests. The records and manual grow by about +90 lines.

Evidence: host CI fails only on the same recorded llvm::tests red. Clippy's 26 invocations are clean, including aarch64 with test-actuators. The previous-space-only mutant reds the IBPB test with the right counts. The r1 logs are masked. Guest run 43156124553 and control 43156167715 are still queued, and judging them is the orchestrator's. Words that #1243's guest run logged show that the guest runner exposes SSBD and IBPB, so the member can hold its claims there.

Blocking

B1 This head does not merge onto main 302fb36. #1252 conflicts in kernel/src/loader/mod.rs, kernel/src/loader/start.rs, kernel/src/syscall/dispatch.rs, sdk/toyos/src/spawn.rs and tests/guest/rust-tests/src/bin/served_file_offset.rs. On main, build_child_handles(endow, job, pager) takes no labels, so the ask has to be threaded again through the spawn path that speculation_on_request and the no-inheritance control exercise. Rebase. The guest run and the control then have to come from the rebased head: the resolution rewrites the path they test.

B2 docs/src/architecture/trust.md:49-57: the manual does not describe the design this PR builds.

• The first bullet lists only the barrier between programs and isolation from the core's other thread. SSBD, the protection this ask turns on everywhere it is enumerated, is missing. On the T14, SSBD (0x4) is the only IA32_SPEC_CTRL bit the ask sets.
• The new bullet says "A program asks through its row in the system manifest", but the shipped interface lets any spawner start a helper child that asks (SpawnArgs::isolated_speculation, CommandExt::isolated_speculation). The PR body states that as a design choice ("Any spawner may ask for its child…").

The Trust page should state both.

B3 kernel/pure/mitigation.rs:487: in_force's contract still says "[ALWAYS] and each control a register shows on". It now also returns every ON_REQUEST mitigation the CPU enumerates, so the doc is false. Correct it.

B4 kernel/src/arch/x86_64/mitigation.rs:244,282: Local::has_spec_ctrl is stored on every kernel, but only the test-actuators read debug_spec_ctrl (:360) uses it. The switch decides from asked_spec_ctrl. This is production state that exists only for a test. Gate it under test-actuators, or derive the debug answer without it.

B5 The PR description (the orchestrator's to correct):

• "A CPU that names nothing, or a machine where no program asks, pays a load and two compares per switch" is false, as traced above. It is the evidence offered for the hot-path constraint. State the real cost, or make the code match the claim (S1).
• "as you approved" should name who approved, so that an approval by the orchestrator is not read as the owner's.

Suggestions

S1 kernel/src/arch/x86_64/mitigation.rs:302: where a CPU's OnRequest is empty, last_asked and loaded serve nothing. Where IBPB is not named, LastAsked tracking serves nothing either. A per-CPU early return, set at install, would make the unselected path one load and one branch, which is the brief's constraint taken literally.

S2 kernel/src/arch/x86_64/mitigation.rs:168,179: report() now also installs the per-CPU switch state that SSBD and IBPB depend on. The call site in smp.rs reads like logging only. A name that says it settles the switch state, or a separate call, keeps someone from moving or gating it as a log.

S3 kernel/pure/mitigation.rs:582: LastAsked<S> is generic only so the tests can use char. The library already uses toyos_abi, so Option<Pid> would be simpler.

S4 tests/guest/speculationprobe/src/main.rs:13: the probe takes SYS_DEBUG's answer without checking for an error. An error answer is u64::MAX minus a small code, so now & asked == asked can pass on one. Refuse an error answer, as the job's answered does.

S5 tests/toyos.rs:71: the member needs a CPU that enumerates SSBD to the guest: the benches and the KVM guest runner. Under TCG's qemu64 it refuses by design, so shared_debug_members_exit_0 goes red there, although the shared boots declare TCG where a host has no KVM. State the member's targets at its registration.

S6 tests/guest/rust-tests/src/bin/speculation_on_request.rs:74: after the probe phase, the job is pinned to the CPU where the probe ran with the asked bits set. Reading SPEC_CTRL there and asserting the bits are clear would exercise the clearing transition. The job's only read now comes before any program that asked has run on that CPU.

S7 kernel/src/arch/x86_64/mitigation.rs:179: the base value is computed twice: by apply without SMT, and by report with it. The two agree today, because nothing spec_ctrl() depends on varies with SMT. Installing the value bring-up actually wrote would remove the latent split.

SEND BACK

[Japabu]

Review r2 of #1278 at 1ab7a9b

Architecture. The architecture is unchanged since r1. The round-2 commit (+106/−46 over 7 files) splits install from report, adds the per-CPU early return and makes LastAsked hold an Option<Pid>. The rest is the Trust page, the docs and the probe's checks.

Round 1

• B1 resolved. Checked with git diff 302fb3698 1ab7a9b0b over loader/, dispatch.rs, process.rs and sched/.
  • A child's handles are a list its parent names: labels and namespaces leave the kernel #1252's endowment semantics are kept. The ask rides beside job and pager and nowhere else: PendingHandles::speculation, Commit::speculation, build_child_handles(endow, job, pager, speculation) and NewTask::speculation. A child's handles are a list its parent names: labels and namespaces leave the kernel #1252's commit, Own::install, the TRANSFER_ENTRY_LEN reads and Inherit are untouched.
  • The ask is inherited as designed.
    • SYS_SPAWN takes current_speculation().of_child(asked == 1) before building anything (dispatch.rs:131), and any value other than 0 or 1 is InvalidArgument.
    • A new thread takes its process's ask (process.rs:611). spawn_thread only creates threads in the caller's own process.
    • The kernel's supervisor table spawns with Shared, and the supervisor passes each row's ask (supervisor/src/main.rs:3188).
• B2 resolved. The Trust page names SSBD, says that any program may ask for a child it starts, and says how a launch asks.
• B3 resolved at this head. Every entry from Ring 3 clears branch history where the CPU's decision names it #1279 rewrites that doc again.
• B4 resolved. has_spec_ctrl and its store exist only under test-actuators.
• B5: the second bullet is resolved. The first bullet is B2 below.
• S1, the orchestrator's question: still correct for a CPU that later clears SSBD for idle.
  • applies is written once, by install (mitigation.rs:303), from the same OnRequest that supplies the asked bits. That happens at the end of boot_aps (smp.rs:253), which has no early return, and before any program runs.
  • On a CPU where applies is false there are no asked bits, so loaded never becomes true and idle has nothing to clear.
  • A CPU whose decision names SSBD or STIBP has applies true and runs the whole path. A switch to idle (next = None) makes asked false and clears the bits.
  • Nothing else writes 0x48 after bring-up: there is no hotplug, no late microcode load and no resume, and IBRS on entry is not built. So applies cannot go stale.
• S5 taken. The registration states the member's targets, and the member gates on CPUID's IBPB. A CPU with IBPB but without SSBD reds on asked != 0, but no such CPU is a declared target.
• S2–S4, S6 and S7 taken, and each reads correctly. For S7, install takes the bring-up read-back, read.registers.spec_ctrl.

Findings

B1 This head does not merge onto main 29c5f2dfa.

• x86-64: FSGSBASE is taken, and the switch carries each thread's FS and GS bases #1264 (FSGSBASE) conflicts in two files:
  • kernel/src/sched/payload.rs:54, where KernelCtx gains user_bases beside speculation;
  • kernel/src/sched/driver.rs:378,430, the two contexts built there.
• The resolution is to keep both fields.
• git merge-tree places the switch hook beside pio::switch_to, ahead of x86-64: FSGSBASE is taken, and the switch carries each thread's FS and GS bases #1264's base writes, which is correct.
• After the rebase, compile and run the host checks. If the resolution only keeps both fields, the guest evidence at this head stays valid. The rebased Every entry from Ring 3 clears branch history where the CPU's decision names it #1279's guest run carries this member anyway.

B2 PR body (the orchestrator's to correct). "On a CPU that names something but where no program asks, it pays the lookup, two loads and about five predictable branches" should say three loads:

• applies (mitigation.rs:328);
• last_asked (:331);
• loaded (:348).

ibpb is loaded only when a barrier is owed. "No store and no MSR access" holds. Round 1 asked for the real cost, so the number should be right.

Evidence

• Host round 2: run at Every entry from Ring 3 clears branch history where the CPU's decision names it #1279's c07ac2ef3. It differs from this head only in tests/toyos.rs's guest print. Exit 1, 122 of 123 steps green; the only red is the recorded macOS llvm::tests failure.
• Clippy: 26 invocations clean.
• Builds: the images and the changed guest binaries build (a2r2-builds.log).
• Guest: run 43168163441 and control 43168182421 are queued and are the orchestrator's to judge. The control's diff is exactly of_child returning Shared.

SEND BACK