[Japabu]

Head ced7b1b: code-identical to 77f15f6, at which every check below ran (ced7b1b adds issue records only). It lands on guest evidence: both metal benches are down, and the speed-up is not yet measured on metal.

Problem

pipeperf's diagnosis of cached reads (tables and logs, stage timers in a scratch kernel, two boots per bench) found two costs in PagerObject::read that buy nothing:

• A third pass of the pager's lock. A read pinned its chunk under the lock, copied, took the lock again to check that the pager had not ended and no write had met the read, then looped and took it a third time only to find the read done. A pread of 8 bytes on the T14: 516 ns against Linux's 340, of which the lock and pin passes were 2 x 49 ns (2 x 44 on the AMD bench), the check 31-34.
• A populate pass over present pages. Before copying a chunk of more than one frame, the read faulted its destination in (UserBytesMut::fault_in): one hold of the space's lock, then an inventory lookup and a table walk for every page, present or not. Reading 1 MiB into a buffer it had read into before, that cost 547-1,185 ns per 16 pages on the T14 and 685-689 on the AMD bench (34-74 and 43 ns a page). The row: 132.7 µs against Linux's 103.5 on the T14.

fault_in was added (cfc30bd) because a range of a memory object, such as the shared memory a spawn reads its image into, took one copy fault per page: a 32 MiB spawn took 8,193. A private range does not, since a touch beside a populated page fills its aligned 64 KiB run (kernel::vm::around, a75abd3); object ranges were left out of that rule, though the space's module contract already says every page is populated at first touch or with that run.

Change

• The check after a chunk's copy answers the read where the chunk reached the read's end (Step::Copy { count, last }). A read now takes the pager's lock twice per chunk: once to pin, once to check.
• fault_in goes, with UserBytesMut::fault_in and process::fault_in.
• AddressSpace::populate applies the run rule to object ranges too: a touch beside a mapped page maps the object's pages of its aligned 64 KiB run, allocating nothing, since an object's frames are its own from creation. A copy into memory nothing has touched now takes the faults a touch would, one per run, private or object; a copy into present pages takes none.
• Records: user-memory-hot-paths-are-slower-than-2-mib-pages.md (faults, copies) and a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md (the populate pass its figures were read with) say what the kernel now does, and what this change owes on metal.

The cost moved: a first touch through a kernel copy pays an exception's delivery and return per 64 KiB run where fault_in paid one lock round trip per chunk. The A/B's fresh-memory rows (ab.fs.memory.fresh.* anonymous, ab.fs.memory.shm.* shared memory as a spawn's image is read into, up to 16 MiB) and proc.spawn_wait.large measure it.

Checks

Targeted, per the owner's testing rules of 2026-11-11: the change is the paged read, the address space's fault service (shared by both architectures) and two records.

• Kernel clippy, warnings denied, the kernel shapes of build/clippy.rs (x86-64: default, boot-actuators, boot-actuators,test-actuators; AArch64: default, boot-actuators,test-actuators) at 77f15f6: exit 0 on all five (log).
• No host test reaches either change: kernel::vm::around, the run rule, is unchanged and tested in kernel/pure; populate and the paged read live in the kernel binary.
• Guest rows at 77f15f6, run 43061625403: success, 9 of 9 PASS (log). x86-64: a_spawn_from_an_image_frees_it_when_the_child_ends (a spawn reads its image through a paged read into fresh shared memory, so the copy faults on an object range and maps its runs), bar_map_again and bar_map_behind_a_bridge (a claimed BAR's aperture, an object range whose faults now map runs), a_user_copy_faults_in_mid_copy_or_ends_as_bad_address. AArch64, since the fault service is shared code: virt_user_copy_faults, virt_unmap_touch, virt_readonly_copyout, virt_user_mode, virt_claim_lpi. An earlier, wider run of mine (43051951823) was cancelled before it ran.
• The A/B job's guest smoke, osbench_pipeab_smoke at fa3e55b (this change + the measurement commit), run 43051958389: PASS, every row of the job measured, the fresh-memory and shared-memory reads among them (log). Correctness only: QEMU's timings are no evidence of speed.

Owed on metal

The speed-up is a prediction from the diagnosis, not a measurement of this change. Recorded as Owed on metal, for when a bench returns:

• A/B of main from before and after this change, both benches, four boots a side, running osbench pipeab (the measurement-only commit on pipeperf-ab-base, run beside x86-64 user copies go by memcpy's rule, and a failed copy is a failure #1244's arm: 32 boots with the shared boots below), with the rule and the rows in issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md. Main after must show ab.fs.memory.pread.8.d0 and ab.fs.memory.pread.1048576.d0 lower (predicted about 45 ns of 470-520, and 5-9%), and ab.fs.memory.fresh.* and ab.fs.memory.shm.*, the price, unchanged or their loss recorded against the gain.
• The price-row bound, in issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md: proc.spawn_wait.large and ab.fs.memory.shm.16777216 must not regress on either bench: no rise in all four paired boots with a median ratio above 1.03.
• testcases and shared-debug boots of main, both benches.

Risks

• A first touch through a kernel copy now takes an exception per 64 KiB run instead of a lock round trip per 16 pages; if the fresh-memory rows lose more than the warm rows gain, the fix is in the fault path, not a pre-fault pass.
• An object range touched beside a mapped page now maps up to 15 more of the object's pages, a claimed BAR's among them. Mapping touches no device register, and every one of those pages was the process's to touch.

🤖 Generated with Claude Code

[Japabu]

Review of #1245 at 77f15f6

Architecture

The PR deletes a compensating mechanism: fault_in, the populate pass that one caller, the paged read, ran over each chunk. In its place, the address space's single population rule now applies to every kind of range, which mm/space.rs's module contract already promised.

• Populating pages is the fault path's job, so no caller needs to pre-fault.
• map7b's approved design (rev 4, step 1) adds the same 64 KiB run for object ranges. The orchestrator's ruling that this is an implementation choice therefore matches an approved design, and no owner decision changes.
• The manual says only "populated lazily", so it needs no edit.

Kernel: +33/−55.

The two lock passes

Under the old code, the third pass always answered done when done == want: Gone with done > 0 returned done, and done == want returned Done. The new last returns that same answer right after the post-copy check.

When a read reaches EOF short, last answers with the size seen in pass 1, after the post-copy check found nothing ended and nothing written. That is a valid linearization. A holder's read that meets a write is still caught by every chunk's post-copy check.

Concurrent withdraw and revoke:

• Pins are unchanged: taken in pass 1 and released before pass 2.
• If a frame is withdrawn during the copy, Held::give_back puts it in draining, so it is not freed until the copy unpins it.
• If the pager ends during the copy, pass 2 answers Io as before. If it ends after pass 2, the old third pass answered done, and so does the new code.

The Waiting::request lost wake (#1233's R1)

That strand is in the absent path: a request made again after a withdraw already has its range marked reported. This PR does not touch pin_chunk's Absent branch, Waiting::request or the wait. last returns only after a copy, and no wait follows. Chunks that are not last loop as before.

So the PR neither fixes nor widens that defect. It and #1233 edit disjoint hunks of paged.rs.

For map7b:

• Whichever of the two lands second takes this PR's shape of populate, with around hoisted.
• The copy already faults while it holds its chunk's pins, as the old code did on every page fault_in left. No such fault waits today: private frames are promised, and an object's frames are its own. "No wait holds a pin" therefore holds now. Step 2 owns the change once a fault can park.

Fault-around over apertures: the claim holds

• Only SharedMemObject ranges are Kind::Object. Paged objects are never mapped.
• The fill writes only Empty leaves inside the entry's [start, end), with the entry's protection and object.cache(), exactly as touching each page would. object.page(offset + (at - start)) stays inside the object because map checks offset + len <= size (space.rs:410).
• Claimed BAR. It is Uncacheable (pcidev/mod.rs:1690).
  • x86-64: this selects the PAT UC entry, "UC under every MTRR type" (x86_64/paging.rs:89-92). The SDM's strong UC makes no speculative memory accesses, so a mapped register page nobody touched sees no access.
  • AArch64: Device memory with PXN|UXN (aarch64/paging.rs:67). The architecture permits no speculative data access to Device memory, and XN rules out fetches.
• Write-combining. Only the firmware scanout uses it (device.rs:218). Speculative reads are allowed there, but a framebuffer read has no side effect.
• OpRegion window. It is ordinary RAM.
• IOMMU. Untouched. Only CPU leaves change, and DMA grants are domain mappings of Run objects.
• Revocation. It works by range: sys_shm_map records the whole range, and the last handle or Handed::revoke releases all of it, run-mapped leaves included.
• shm_map refuses ReadExec, so AArch64's make_executable never walks aperture frames.

Required

1. The queued guest run will run nothing. Run 43051951823 (shm_ std_mmap mmap_ spawn_image_object spawn_from_tmp a_served_file a_pager_s_end user_copy demand_paging claim_bar_fault virt_unmap_touch virt_readonly_copyout virt_user_mode) is still queued, and it will be refused before booting any guest.
   • tests/toyos.rs:9022-9025 exits 1 on the first filter word that matches no declared row. declared() (tests/toyos.rs:8903) covers only MACHINE_TESTS and SCREEN_TESTS.
   • 9 of the 13 words match neither: shm_, std_mmap, mmap_, spawn_image_object, spawn_from_tmp, a_served_file, a_pager_s_end, demand_paging and claim_bar_fault. These are shared-boot members, not QEMU rows. The same refusal ended 43048940831 with No test matches filter "a_pager_s_end_answers_its_waiting_holders_gone".
   • The PR's Checks section lists them as guest rows. Correct that list. The shared members are covered by the testcases boots already staged for both benches.
   • Cheapest valid dispatch: cancel 43051951823 and dispatch user_copy bar_map virt_unmap_touch virt_readonly_copyout virt_user_mode virt_claim_lpi a_spawn_from_an_image. Every word matches a row.
     • bar_map_again and bar_map_behind_a_bridge map a claimed BAR on x86-64.
     • virt_claim_lpi runs claim_bar_fault on AArch64.
     • a_spawn_from_an_image_frees_it_when_the_child_ends reads images from DATA and /tmp through paged objects.
     • The virt_* rows cover the shared fault code on AArch64.

Suggestions

• issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md:33: "The separate populate pass touches each page's leaf a second time" is in the present tense, but this PR deletes that pass. Put it in the past tense to match the edited intro.
• issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md, the Copies bullet: "one per 64 KiB run" holds once a run's neighbour is mapped. A copy into memory nothing has touched takes two faults for its first run, because a lone touch maps one page, as the Faults bullet says.

Evidence checked

• Clippy: five shapes at 77f15f6, exit 0, no warnings.
• No host test reaches populate or the read loop. Inspection settles the equivalence above.
• The P arm (fa3e55b) is this head plus a measurement commit identical to base's.
• The branch merges cleanly with main 67713cb and with x86-64 user copies go by memcpy's rule, and a failed copy is a failure #1244.

What landing needs

• A valid guest run, green: Required 1.
• Shared boots. testcases and shared-debug at 77f15f6 pass on both benches. They cover the shm, mmap, spawn and served-file members the guest list named.
• A/B. Under the rule, P counts a gain on ab.fs.memory.pread.* at d0, at least at 8 B, on both benches.
  • Price rows: ab.fs.memory.fresh.*, ab.fs.memory.shm.*, fresh8m and proc.spawn_wait.large. The request gives no acceptance bound for them. Set one before reading the readbacks.
  • My proposal: proc.spawn_wait.large must not regress under the rule, since it is the defect fault_in was added to fix and object fault-around is meant to replace it.
  • A counted regression on a fresh or shm row is investigated, and its cost is recorded in user-memory-hot-paths-are-slower-than-2-mib-pages.md before landing. It does not reject the design by itself alone.
• The rule itself is sound for the predicted directions. The caveats are in x86-64 user copies go by memcpy's rule, and a failed copy is a failure #1244's review: confirm any change counted on a row nobody predicted with more blocks.

Verdict: SEND BACK. The blocker is evidence only: the queued guest dispatch is invalid. No code change is asked for. After a valid green run and the metal above, this is LAND.

[Japabu]

Orchestrator verification, landing under the benches-down rule (metal is unavailable; landing on review + host + guest, metal owed):

• The review's send-back was evidence-only; no code changed since. Head ced7b1b differs from guest-tested 77f15f6 only in issues/ (2 records) — verified with git diff --stat.
• Guest run 43061625403 at 77f15f6: success (9/9). That is the evidence the review asked for.
• Owed on metal (A/B with the cached-read rows; price row proc.spawn_wait.large and the 16 MiB shared read) recorded in the two records; the speed-up is not yet measured on metal, as the body says.