[Japabu]

Head 4e06cf1: code-identical to 69de741, at which every check below ran (later commits add issue records only). It lands on guest evidence: both metal benches are down, and the speed-up this PR exists for is not yet measured on metal.

Problem

On the AMD bench (Ryzen 7 4800U, Zen 2) pipe.throughput sits near 3 GB/s at every write size from 4 KiB, against 15-20 GB/s when the reader's buffer starts on a page. The 64 KiB reader buffer the allocator hands osbench starts 16 bytes into a page, so the kernel's copy from the pipe's frames into it is a forward rep movsb whose destination is 16 bytes above its source modulo 4 KiB: the store-forwarding stall #1139 took out of the kernel's own memcpy (AMD 56305 rev. 3.02, §2.12) but not out of its user copies. Cached reads stall the same way (a 1 MiB pread 97 µs into a buffer at a page's start, 392 µs at 16 bytes in).

Diagnosis, two ToyOS boots and two Linux runs per bench with stage timers in a scratch kernel: tables and logs.

AMD, MB/s, ToyOS / Linux	reader at 0	at 16	at 32	allocator's own (16)
4096-byte writes	6830 / 5738	3025 / 4414	6414 / 4335	3025 / 5740
16384	15281 / 3716	3037 / 3216	15885 / 3234	3037 / 3896
65536	19987 / 2107	3032 / 2461	22151 / 2641	3043 / 1982

The T14 shows no such stall (64 KiB at 16 costs 9-12%), as #1139's rule for Intel says.

Change

• x86-64's user copies ask copy::Rule exactly as the kernel's memcpy does and, where it sends a copy down, run the same descending copy: one instruction body (copy_down_asm!) that copy_down and the two user routines expand, integer registers only. Its fixup covers the body's range: a faulting mov is rerun once the fault is served, and one nothing serves goes to a shared exit that closes SMAP's door and answers failure. A fault on a kernel address inside that range still ends the kernel: the page-fault handler takes a fixup only for a user address.
• uaccess::{copy_from_user, copy_to_user, clear_user} answer Result<(), Faulted> on both architectures instead of a count of bytes left. No caller used the count but to test for zero, and a descending copy has no meaningful one.
• Test-only actuator copies-go-down (window 4096) sends every copy of 64 bytes or more at a nonzero page distance down, so the guest runner's CPU, ruled Intel or other, takes the descending copy and its fault paths. user_copy_faults gains a pipe round 16 bytes off the pipe's frames (down by the AMD rule on that bench), and the new row a_user_copy_highest_first_faults_or_ends_as_bad_address runs the job on the actuator kernel: one boot of QEMU's x86-64 machine. On metal the AMD bench's shared testcases boot takes the descending path by its own rule.
• Records: what this change owes on metal, in the kernel copies' record; and the diagnosis's two findings outside this change, each with its exit: a pipe's frames going back to the allocator as its reader empties them and coming out again for the next write (35% of 4 KiB writes on the T14), and the AMD bench reading a freshly written 8 MiB file at about 70% of a 16 MiB file's rate (the T14 the other way).

Not changed: userland's memcpy (agents/memperf's, #1198's record). The predicate this PR uses, offered for libc to share: a copy goes highest first when n >= 64 and 0 < (dest - src) mod 4096 < window, window 32 on AMD and 0 elsewhere.

Checks

Targeted, per the owner's testing rules of 2026-11-11: the change is the x86-64 and AArch64 kernels' user-copy routines, their one caller (user_ptr), an actuator and one guest job.

• Kernel clippy, warnings denied, the kernel shapes of build/clippy.rs this change reaches (x86-64: default, boot-actuators, boot-actuators,test-actuators; AArch64: default, boot-actuators,test-actuators) at 69de741: exit 0 on all five (log).
• kernel-copies (the descending copy and every rule against copy_within, every page distance), cargo test -p kernel-copies --target x86_64-apple-darwin under Rosetta on the tree committed as 69de741: exit 0, 3 passed (log). It tests copy.rs, whose body the user routines now expand.
• Image build, cargo run -- --build-only on the same tree: exit 0 (log).
• Guest rows at 69de741, run 43037821945: success, 3 of 3 PASS: the new a_user_copy_highest_first_faults_or_ends_as_bad_address (x86-64, every copy down), a_user_copy_faults_in_mid_copy_or_ends_as_bad_address (x86-64, by the runner CPU's own rule) and virt_user_copy_faults (AArch64) (log).
• Negative control: 125dca5 on pipeperf-usercopy-mut is this change with the descending copy's fixup disabled (patch); run 43037875955: failure, as required. The new row FAILs on KERNEL PANIC: read unmapped address (log). So the row reaches the descending copy's fault path, which nothing else on the guest runner does.
• Not run: the full host CI (no build-system or harness logic changed beyond one row's registration, which the guest run exercised), the kernel's host library (kernel/pure, unchanged).

Owed on metal

The speed-up is a prediction from the diagnosis, not a measurement of this change: on the AMD bench the pipe with osbench's own buffers is expected to go from 3.0 GB/s to about 7-20 GB/s by write size. Recorded in issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md as Owed on metal, for when a bench returns:

• A/B of main from before and after this change, both benches, four boots a side, running osbench pipeab (the measurement-only commit on pipeperf-ab-base, run beside A paged read takes two passes of its pager's lock and populates nothing ahead #1245's arm: 32 boots with the shared boots below). A row changes where all four paired boots move one way and their median ratio is beyond 1 ± 3%. Main after must show on the AMD bench ab.pipe.natural.4096, .16384 and .65536 up from about 3.0 GB/s to about 7, 15 and 20 GB/s, and each cached read 16 bytes into a page at most 1.25 times the same read at a page's start (1.9 to 4 times before); on the T14, none of those rows changed. libc: a forward copy a short way above its source modulo 4 KiB copies down #1133's copyat.place rows should not move.
• testcases and shared-debug boots of main, both benches; on the AMD bench user_copy_faults there copies down by its own rule.

Risks

• On Zen 2 a descending copy moves 8 bytes a store where rep movsb streams; at the distances it now takes (1-31 bytes) the forward copy stalls far worse, but a copy that sits there and is long enough to reach DRAM is measured only by the A/B's 1 MiB and 16 MiB rows.
• copies-go-down keeps an actuator arm in boot.rs's rule choice; the shipping kernel folds it to false.

🤖 Generated with Claude Code

[Japabu]

Review of #1244 at c6e7bee (code at 69de741)

Architecture

arch::x86_64::copy owns both the rule (one WINDOW static, set at boot) and the descending copy (one instruction body, copy_down_asm!). The user copies call copy::rule().goes_down and expand that body inside their own exception-table ranges. The rule and the instruction sequence each have one definition. The forward rep movsb in uaccess stays separate because the exception table needs its exact instruction address. The body uses only rax, rcx and r8–r10 and no stack. Everything stays in x86-64's architecture modules, and AArch64 changes only its signature.

Changing the return type to Result makes the interface say what its only caller already relied on. On main, user_ptr's View::{read_raw, write_raw, fill_zero} were the only callers, and each turned the count into a Result through whole. Nothing read the number of bytes left.

Size: kernel +190/−90. That covers two exported descending routines, their fixup ranges, the macro conversion, the actuator arm, and whole moving into each architecture's module. Tests +36/−17. Records +56.

Correctness, verified by reading

• Fixup. [*_down_access, *_down_end) covers every instruction of the expansion, and the end label comes before clac.
  • When the fault is served, user_copy_fault leaves rip alone, so the faulting mov runs again with the registers the trap frame restored. In the four-load, four-store group, only the faulting instruction reruns.
  • When nothing serves the fault, execution resumes at toyos_copy_down_fault: clac, eax = 1, ret. This is sound because the body pushes nothing, so rsp at the fault equals rsp at entry. Every register it changes is caller-saved under SysV.
• Kernel addresses. exceptions.rs:471-472 takes the fixup only when CR2 is a user address. A kernel-side fault inside the range therefore still ends the kernel. The AC assert holds because stac comes before the access label. windows_entered (idt/mod.rs:366) gives the range the same treatment.
• Spectre. Both descending routines keep the cmovae that clears the user pointer.
• Partial progress. On Err, a descending copy_to_user may already have written the top of the user buffer. The new contract, "some or none of them were copied", says exactly that.
  • No caller depends on a written prefix. The paged read counts done in whole pieces, and the pipe's frame_bytes::{fill, drain} fail as a whole.
  • A user source that changes during a copy_from_user leaves a mix of bytes the user held at some point. The kernel validates its own copy afterwards, so this creates no double fetch.
• Rule. goes_down requires n >= 64, which covers the body's minimum of 8. The distance is kernel VA minus user VA modulo 4 KiB, which is the aliasing that matters.
• Device memory. On a user BAR mapping, the overlapping stores and loads touch some words twice. These are still only accesses the process could make through that mapping (uaccess.rs:20-21).

Test and control

Under copies-go-down (window 4096), every copy of at least 64 B with a nonzero distance goes down. The row therefore checks:

• served faults in both directions, with the data compared;
• the path where nothing serves the fault. namespace_build's 600-byte blob sits at user offset 3796 mod 4096, which is not 16-aligned, so no heap Vec can be at distance 0. The copy goes down every time, and the test asserts BadAddress and that the kernel keeps answering.

I confirmed that 125dca5 is 69de741 plus the one-line mutation. Its run (43037875955) panics on user address 0x100b1c124 during syscall 0, which shows the row reaches the descending copy's fault path. A fixup that answered success would fail the BadAddress asserts.

The actuator cannot be silently inert: kernel_params selects the test kernel, and the kernel refuses a token it does not know.

Evidence checked

• Clippy: five shapes, exit 0, no warnings.
• kernel-copies: 3 passed. This exercises the macro body through copy_down.
• Guest run 43037821945: checks out 69de741, 3/3 PASS, virt_user_copy_faults included. Every filter word names a declared row.
• c6e7bee adds only issues/, so the evidence gathered at 69de741 still applies.
• A/B arms:
  • U's measurement hunks match base's line for line.
  • 34cb4c9..b566931 changes only issues/, so U's base and B's base are the same code.
• The branch merges cleanly with main 67713cb and with A paged read takes two passes of its pager's lock and populates nothing ahead #1245.

Records (c6e7bee)

• The pipe-frames finding matches PipeRef::put_now (None when held.put needs frames) and diagnosis/t14.md: 35,552 of 102,720 writes bounced at 616 ns, against 475 ns for a put.
• The 8 MiB finding matches diagnosis/amd.md: 7542 MB/s (7257–7828) against Linux's 9164 (9004–9324).
• Both records follow issues/README.md and state an exit.
• The edit to the track record is accurate.

Required

None.

Suggestions

• tests/toyos.rs:677-678: "the AMD bench's shared boot takes it by its rule" claims more coverage than the boot gives. On AMD, only the round 16 bytes off the pipe's frames goes down. namespace_build's copy goes down only if its heap Vec happens to sit 1–31 bytes above 3796 mod 4096. The AMD boot therefore covers the descending copy's served faults, not its failure exit. Suggested wording: "takes its served faults by its rule".

What metal must show

• Shared boots. testcases and shared-debug at this head pass on both benches. On AMD, user_copy_faults prints "a fault served mid-copy resumes 16 bytes from the pipe's frames: 12888 bytes" and "user copies: all checks passed". That is the descending copy's served faults on real Zen 2.
• A/B, AMD.
  • Under the rule, U must count a gain on ab.pipe.*.r16.*, ab.pipe.natural.{4096..} and ab.fs.memory.pread.*.d16.
  • No regression may count on the r0/d0 rows, ab.pipe.thread.*, ab.pipe.child.*, copyat.place.* or proc.*.
  • The 1 MiB d16 pread and fresh8m are the DRAM-reach check the PR's risk names.
  • If no gain counts on AMD, the change has not earned its code.
• A/B, T14. If the U boots run, no regression counts. If they are dropped, inspection is enough: under Rule::NEVER the only extra work is one relaxed load and one compare per copy.

The decision rule

The rule is sound for the predicted, directional effects. The direction was fixed in advance, so the sign test alone passes by chance with probability 1/16, before the 3% filter.

• For rows with no prediction, all four blocks land on the same side by chance with probability 1/8. Across dozens of rows, several will pass that test by luck. A counted change on a row nobody predicted should be confirmed with more blocks before it blocks or credits either PR.
• P's price rows have no acceptance bound yet; see A paged read takes two passes of its pager's lock and populates nothing ahead #1245's review.
• Repeating block 1's order in block 4 is harmless.

Verdict: LAND, pending the metal above.

[Japabu]

Orchestrator verification, landing under the benches-down rule (metal is unavailable; landing on review + host + guest, metal owed):

• Review verdict LAND (pending metal). Head 4e06cf1 differs from guest-tested 69de741 only in issues/ (3 records, +70 lines) — verified with git diff --stat.
• Guest run 43037821945 at 69de741: success (3/3; negative control red as required).
• Owed on metal recorded in issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md; the speed-up is not yet measured on metal, as the body says.