[Japabu]

Problem

osbench time's time.now (std's Instant::now, answered from the clock page) read 21.0 ns on the T14
against Linux's 14.3 ns (issues/a-clock-read-costs-toyos-half-again-linuxs-on-the-t14.md). The read
fenced the counter on both sides (LFENCE; RDTSC; LFENCE), and std's Instant held a Duration, so every
now divided by 10^9.

Resulting behavior

• x86-64 reads the counter with RDTSCP and nothing after it. RDTSCP waits for every earlier
  instruction to execute and every earlier load to be globally visible (SDM Vol. 2B, RDTSCP); AMD's APM
  Vol. 3: it "forces all older instructions to retire before reading the time-stamp counter". That is the
  ordering the clock needs before the read. The old leading LFENCE gave it on AMD only where LFENCE is
  dispatch serializing (DE_CFG bit 1, or CPUID Fn8000_0021 EAX bit 2), which the kernel neither sets nor
  checks (see below). The kernel's clock already refuses a CPU without RDTSCP.
• AArch64 is unchanged (ISB; MRS; ISB): the Arm ARM lets a counter read happen out of order against
  other instructions, and nothing architectural orders it before a later store that does not depend on it,
  so the x86 argument below does not carry over; and no AArch64 bench could measure it (review r1).
• std's Instant is the page's nanoseconds as a u64, converted to Duration only by subtraction.
• nanos_since_boot and its read are #[inline]; the page-without-a-clock panic is a cold function.

Why dropping the trailing fence is correct

For a process's read, it held later instructions back so the latch's recheck and the page's demoted word
could not be answered before the counter. Neither needs it:

• A line replaced in that window. The kernel starts a new line at a counter reading at taken before it
  publishes, and the new line starts where the old one stands (clock::retune). Every reader between at
  and the publication already pairs the old line with a counter past at; the out-of-order window
  lengthens that interval, it adds no new case. The clock bends there and never steps.
• A demotion. The handover floors each CPU's answers in a kick handler after the mark. An interrupt is
  taken at an instruction boundary: either the reader's whole read, counter included, retired before it
  (so its counter is before the floor), or its early load of the demoted word is discarded and executed
  again after it, and sees the mark. The handover's stated contract is unchanged.

The kernel reads the same page through its own cpu::counter_ordered (RDTSCP; LFENCE), which this
change leaves as it is. So the latch now says only that whether its recheck may be answered before
inside's read is inside's to decide, and the clock page's header says which reader may pair a line
with a counter read just after it was replaced.

AMD's LFENCE (asked of this branch)

The kernel neither sets MSR C001_1029[1] (AMD 70586, G-2; families 10h, 12h and 14h to 17h) nor reads
CPUID Fn8000_0021 EAX[2] LFenceAlwaysSerializing (APM Vol. 3, LFENCE). Only cpu::rdtsc_ordered
(RDTSCP; LFENCE) relies on LFENCE holding later instructions back: the kernel's own clock read
(cpu::counter_ordered inside the clock page's latch) and, in arch/x86_64/clocksource.rs, the demotion's
two readings, floor_here, stamp_now, the AP start's trip, check, and the brackets around each HPET and
PM timer sample. cpu::wrmsr_fence (MFENCE; LFENCE before an x2APIC WRMSR) is ordered by its MFENCE,
which is serializing on AMD (APM Vol. 3, MFENCE), and Intel's LFENCE always serializes. A process's clock
read relies on none of it.

This PR records it as issues/the-kernel-relies-on-an-amd-lfence-it-never-makes-dispatch-serializing.md,
assigned to harden, with #1243 as its exit. If #1243 lands first, this branch drops the record before it
lands; otherwise #1243 deletes it.

Checks

Targeted, per the testing policy: only what this change can affect. Host checks rerun for 3d06ecc70 (logs
r4); the std type-check ran at 03f88418e, and the review commits change nothing std compiles but an
AArch64 counter they restore to main's. 9c1911df5 (review r2) changes the issue record and doc comments
only; cargo doc -p toyos-abi --no-deps exit 0, with no warning from its links.

check	why	result
cargo test -p toyos-abi clock	the clock page's line, latch and boot-relative reading	exit 0, 6 passed
cargo clippy -p toyos-abi --all-targets --keep-going --target {aarch64,x86_64}-apple-darwin --target-dir target/clippy-abi -- -W clippy::undocumented_unsafe_blocks -D warnings	build/clippy.rs's toyos-abi shape, once per architecture so both counter arms are linted	exit 0 both
std type-check, build/CLAUDE.md's scratch-crate method, -Z build-std=std,panic_abort for x86_64-unknown-toyos and aarch64-unknown-toyos	sdk/std/sys/time.rs has no host test	exit 0 both

Not run: kernel/loom's clock_latch model (the latch's code is unchanged; only its doc comment changed);
the full --ci host (no build-system or harness change).

Guest rows: run 43050350414 at 03f88418e, green: a_stepped_tsc_is_slewed_then_demoted (x86-64, q35:
processes' readings after a demotion go to the kernel), udp_receive_times_out (x86-64: std Instant
arithmetic in a timeout), virt_wall_clock_utc (AArch64). The head differs from 03f88418e in comments, two
issue records, and one code line: AArch64's counter regains its trailing ISB, which is main's code again;
the x86-64 code is identical. QEMU shows neither the fence's cost nor reordering; these rows establish that
the read still answers and the demotion still reaches processes.

Owed on metal

Both benches are down, so this lands on its review, host checks and guest rows. The speed-up is not yet
measured on metal. issues/a-clock-read-costs-toyos-half-again-linuxs-on-the-t14.md records what is owed:

• an A/B of main before and after this PR, on the T14 and the AMD bench: osbench time's time.now, four
  boots a side in rotated order, each boot's p50, exact two-sided Mann-Whitney, with the rest of osbench time, files, ipc and sched as controls; and Linux's TSC-timed osbench time on each bench;
• main after must read time.now lower than main before in every T14 boot, set against Linux's 14.3 ns, with
  no control row 5% or more worse unless this change explains it; on the AMD bench a rise is recorded as
  undetermined between ordering's price and a regression, unless a ToyOS boot logs MSR C001_1029 bit 1 before
  writing it (A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's boot line).

Risks

• An Instant reaches 584 years past boot; Instant + d past that panics where it used to succeed, and
  checked_add answers None, as std's other platforms do past their own range.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1240 at 03f8841

Architecture. The read stays where it belongs: the ABI's counter at the kernel/program boundary and std's ToyOS backend. Holding the page's nanoseconds as a u64 removes a conversion and adds no mechanism; the cold no_clock split is sound. +51/-32 in 5 files, no test change; most of the production growth is doc comment.

x86-64 ordering is sound.

• Same thread: RDTSCP waits until every earlier instruction has executed (SDM Vol. 2B, RDTSCP), an earlier RDTSCP included, so one thread's readings never go backwards.
• Across threads and CPUs: RDTSCP also waits until earlier loads are globally visible (the reader's half). A store after the writer's RDTSCP becomes visible only after that RDTSCP retires, because x86 never makes a store visible speculatively and RDTSCP has its value at retirement (the writer's half). With synchronized TSCs, which demotion polices, a reading ordered after another is no older. The kernel refuses a CPU without RDTSCP (kernel/src/arch/x86_64/clocksource.rs:377).
• Later loads: the latch recheck and the demoted word may now be answered before the counter. I checked both of the PR's arguments against the kernel:
  • clock::publish starts the new line at old.stamp_of(at), with at read before RETUNING is taken (clocksource.rs:749). Base already pairs the old line with counters past at. The out-of-order window, which an interrupt discards, extends that interval by a bounded amount, and floor_at reads both lines.
  • floor_here runs in the kick handler after the mark. A precise interrupt either finds the reader's counter retired or discards its early load of the demoted word. Both arguments hold.
• AMD: RDTSCP gives the leading order without depending on a dispatch-serializing LFENCE. On a bench where LFENCE does not serialize, this also fixes a read that was unordered.

u64 and precision: no finding. The page answers integer nanoseconds, so Instant(u64) is exact, and so is Duration::from_nanos on subtraction. checked_add_duration and checked_sub_duration refuse a Duration of 2^64 ns or more through try_from. The limit is 584 years after boot: past it, checked_add answers None and Instant + d panics. That range is narrower than Linux's i64-second Timespec but beyond any uptime. Far-future idioms that use checked_add (std's recv_timeout) fall back to an unbounded wait, as they do elsewhere. Accepted.

Blockers

1. sdk/abi/src/arch/aarch64.rs:1-11: dropping the trailing ISB is unjustified, and it is AArch64-only work that no bench can measure.

   • The PR says AArch64 drops it "for the same reason" as x86. That reason does not carry over.
   • The latch never needed the ISB on AArch64. Only the x86 clocksource calls clock::retune and clock::demote (clocksource.rs:550,555,749,756), so the AArch64 line never changes after set_counter.
   • What the ISB did provide was to order the counter read before later instructions. The Arm ARM (K.a D12.2.2, which kernel/src/arch/aarch64/cpu.rs already cites) says counter reads may occur speculatively and out of order relative to other instructions on the PE. DMB and release order memory effects, not this read.
   • Without the ISB, nothing architectural orders the read before a later store that does not depend on its value. The doc's "as the x86-64 one is" therefore imports the writer half of the x86 contract ("never older than one whose writer's record this thread has loaded"), which AArch64 does not hold. x86 gets that half from never making a store visible speculatively; AArch64 has no equivalent rule.
   • Root CLAUDE.md: "Until an AArch64 metal bench exists, there is no AArch64-only work."
   • Fix: revert this hunk, keeping isb; mrs; isb and "before any later one". If the hunk stays, cite Arm ARM text that orders the read before later stores, and narrow the doc to what then holds.

2. sdk/abi/src/clock/latch.rs:62-67 and the PR's LFENCE section: the tree now holds two contracts for one read section, and the record handed to harden is false.

   • The kernel reads the same latch through cpu::counter_ordered (kernel/src/clock.rs:147). That is RDTSCP; LFENCE on x86 (kernel/src/arch/x86_64/cpu.rs:53-65) and isb; mrs; isb on AArch64 (kernel/src/arch/aarch64/cpu.rs:16-24), documented as "before every later one: the read the clock takes inside its line's read section".
   • The new latch sentence "Nothing orders the recheck after inside's read but an interrupt" is therefore false for the kernel's reader.
   • The PR body says that after this change no ToyOS code relies on LFENCE being dispatch serializing except cpu::wrmsr_fence. That is false: rdtsc_ordered's trailing LFENCE relies on it. It sits under the kernel's clock read, floor_here (clocksource.rs:608) and stamp_now (:642,650).
   • Fix, implementer's choice:
     • apply this PR's own argument to the kernel's reader: drop the trailing fence and its "before every later one" contract, after checking stamp_now's "counter read before the mark" and the demotion's first/after; or
     • keep the kernel's reader as it is and reword the latch sentence so it says only that the latch orders nothing there.
   • Either way, the statement handed to harden must name rdtsc_ordered. Correcting the PR body is the orchestrator's.

3. The AMD LFENCE finding has no issues/ record. The kernel neither sets MSR C001_1029[1] (G-2) nor reads CPUID Fn8000_0021 EAX[2], and on AMD wrmsr_fence and rdtsc_ordered depend on it. Root's "No silent debt" puts that record in issues/, with its evidence, owner (harden) and exit; a PR body is not that record. Either this PR adds the record, or the orchestrator shows harden's before this lands.

Evidence

• The host checks match the head (logs/r3, rev 03f8841): cargo test -p toyos-abi clock (6 passed), the toyos-abi clippy shape for both architectures, and the std type-check for both ToyOS targets, each exit 0. That covers what a host can.
• Guest run 43050350414 is still queued, and its three names are valid rows (two MACHINE_TESTS, one SCREEN_TESTS). It must come back green. It establishes that the read answers and that a demotion still reaches processes, not ordering or cost.
• The A/B's time image rebuilt the sysroot from this head ("external deps changed" in ab/s3-time/stage.log), so it measures the new std.

What metal must show

• T14: time.now lower at the head than at base in all 4 vs 4 boots (p 0.029). Record the result in issues/a-clock-read-costs-toyos-half-again-linuxs-on-the-t14.md against Linux's 14.3 ns, and close it if within its exit.
• AMD: time.now may rise. If that bench's LFENCE is not dispatch serializing, base's read was unordered and RDTSCP is the price of ordering it. Record which is the case.
• Controls: no other row separates by 5% or more.

On the A/B rule (request.txt)

Sound in design:

• It was fixed before any boot, and each image takes each position once per bench.
• Every head is compared with one shared base, by an exact two-sided Mann-Whitney test over per-boot p50s. At 4 vs 4 only complete separation reaches p < 0.05, which suits boot-to-boot variance.
• The images are hashed.

To state before the boots:

• (a) Control rows that separate. "Under 5% is noise" says nothing about a control row that separates by 5% or more; it is a regression to investigate. With roughly 30 control rows a head, about one per head per bench will separate by chance (2/70 each).
• (b) AMD time.now. Say how a rise there is judged, as above.
• (c) Other failed boots. A boot that fails for any reason but a network drop is a finding for its head, not a rerun.
• (d) The judging script. abjudge.py keeps the last value when a boot logs a row twice, and silently shrinks n when a boot lacks a row. It should print n per side.

SEND BACK. Blocker 1 is a one-hunk revert. Blocker 2 is a doc sentence or a kernel alignment, plus the record. Blocker 3 is a record. After those it can land once the guest run is green and the A/B results above are recorded.

[Japabu]

Response to review r1, head 3d06ecc.

1. AArch64's counter is ISB; MRS; ISB again, and its doc no longer says it reads as the x86-64 one does.
2. The latch's read_with now says only that whether its recheck may be answered before inside's read is inside's to decide; the clock page's header says that a reading whose counter holds nothing after it (arch::counter on x86-64) may pair a line with a counter read just after it was replaced. The kernel's rdtsc_ordered is unchanged. The PR body's LFENCE section now names rdtsc_ordered with every user, and wrmsr_fence.
3. issues/the-kernel-relies-on-an-amd-lfence-it-never-makes-dispatch-serializing.md records it: assigned to harden, exit A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243.

Rechecked at 3d06ecc: cargo test -p toyos-abi clock (6 passed) and the toyos-abi clippy shape for both architectures, exit 0. The A/B image built at 03f8841 is the x86-64 code this head lands. The A/B rule in the request gained the review's (a) to (d) before any boot, and abjudge.py now refuses duplicate and missing rows and prints n per side.

🤖 Generated with Claude Code

[Japabu]

Review r2 of #1240 at 3d06ecc (follow-up to r1 at 03f8841)

Scope. This round checks the r1 blockers, the code changed in 03f8841..3d06ecc, and the A/B rule's revision. Everything else is settled.

Architecture. Unchanged since r1. The whole branch is now +82/-27 in 6 files, including a 37-line issue record. The AArch64 counter's code and safety comment match main's again; only the parity sentence is gone.

r1 blockers

1. AArch64 counter: resolved. It is isb; mrs; isb again (sdk/abi/src/arch/aarch64.rs:10), and "as the x86-64 one is" is gone. The only remaining difference from main is that deleted sentence, which would now be false.
2. The latch's contract: resolved. read_with (sdk/abi/src/clock/latch.rs:62-65) now leaves the recheck's order to inside. That holds for both readers: the kernel's cpu::counter_ordered and the process's arch::counter. The clock header (sdk/abi/src/clock.rs:23-27) limits the case where an old line is paired with a later counter to a counter that holds nothing after it, and names x86-64's arch::counter. AArch64's trailing ISB does hold later instructions back.
   • The corrected fact, checked against the tree. git grep -i lfence outside rust/, issues/ and docs/ finds only two kernel LFENCE sequences, cpu::wrmsr_fence (MFENCE; LFENCE, kernel/src/arch/x86_64/cpu.rs:39) and cpu::rdtsc_ordered (RDTSCP; LFENCE, :63).
     • rdtsc_ordered is used by counter_ordered (the kernel's latch read, kernel/src/clock.rs:147). In clocksource.rs it also serves:
       • the HPET and PM-timer brackets (:102,104,169,171);
       • the AP trip (:418,429) and check (:501);
       • the demotion's first and after (:546,553), floor_here (:608) and stamp_now (:642,650).
     • Outside the kernel, the only users are syscall_cost and wake_storm_cost (LFENCE; RDTSC).
     • A process's read is RDTSCP alone.
     • The fact as given to A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243 is correct.
3. The record: filed, but it states one false fact. See Blocker 1.

Evidence

• The A/B image built at 03f8841 still measures this head on x86-64. git diff 03f88418e 3d06ecc70 changes these files:

  • in sdk/abi, only /// and //! lines and AArch64's counter;
  • nothing in sdk/std;
  • one issue file.

  So no x86-64 code differs. ab/s3-time/rev.txt reads 03f8841.

• The host checks cover this commit. logs/r4 (toyos-abi clock tests, 6 passed; the toyos-abi clippy shape for both architectures, exit 0) ran from the working tree a minute before the commit; it records no revision. That is enough for one restored asm line and doc comments.

• Guest run 43050350414 is still queued (at 03f8841, created 15:38Z, queued at 17:19Z). It remains valid for this head. The only code change since is AArch64's trailing ISB, now as on main, and an emulated virt_wall_clock_utc cannot tell the two apart.

• Main has moved to be76e6f. The branch merges cleanly, and nothing since its base touches sdk/abi, sdk/std, kernel/src/clock.rs or the x86 cpu.rs/clocksource.rs.

A/B rule r2 (request.txt against request-r1.txt)

• r1's (a) to (d) are present:
  • (a): a control row that separates by 5% or more is a regression to investigate;
  • (c): the AMD rise;
  • (d): a failed boot is a finding;
  • (e): the judge.
• The claimed rows are named exactly.
• tools/abjudge.py refuses a row logged twice in one boot, refuses a row missing from any boot on either side, refuses mixed units, and prints n per side.
• One clause is wrong; see Blocker 2.

Blockers

1. issues/the-kernel-relies-on-an-amd-lfence-it-never-makes-dispatch-serializing.md:18,25-26,29: the record says cpu::wrmsr_fence relies on LFENCE holding later instructions back. On AMD it does not.
   • Why it is false. APM Vol. 3 rev 3.39, MFENCE: "The MFENCE instruction is a serializing instruction" (cited in A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's review; I did not reread the APM). A WRMSR after MFENCE therefore cannot start until every earlier load and store has completed, whether or not the LFENCE serializes. On Intel, LFENCE always serializes.
   • Consequence. A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's review requires that same claim deleted from A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's record, code comment and commit message (its Required 4). This record would bring the claim back to main and contradict A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243.
   • Fix. Make it one sequence: rdtsc_ordered. Either drop the wrmsr_fence bullet, or say that it uses LFENCE without relying on it because AMD's MFENCE serializes. Reword "relies on neither" (:29) to match. Correcting the PR body's LFENCE section, which repeats the claim, is the orchestrator's.
2. request.txt rule (c): "Part 3 asks the bench's family and CPUID leaf 0x80000021, which together say which case it is" is false for the AMD bench.
   • What Part 3 shows. The bench is family 17h (signature 0x860f01; A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's review expects cpuid.80000021.eax to be 0). By this branch's own record, LFENCE on 17h serializes only once C001_1029[1] is set. Family and CPUID therefore say only that the MSR decides.
   • Why no reading settles it yet.
     • Firmware may or may not set the bit (A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's review).
     • Part 3 runs under Linux, which sets it itself, so nothing read there shows the bit as ToyOS boots find it.
     • No ToyOS boot reads it yet.
   • Consequence. An AMD rise would be recorded as "ordering's price" on evidence that does not establish it. r1 asked for the case to be recorded.
   • Fix (text only; the boots need not wait). (c) should say the case turns on C001_1029[1] as firmware leaves it, which no boot reads. The record then gives an AMD time.now rise as undetermined between ordering's price and a regression, unless a ToyOS boot logs that bit before writing it. A pure function decides a CPU's speculation mitigations, and each boot says its verdict #1243's suggested before-write field would do that.

Suggestion

• sdk/abi/src/arch/x86_64.rs:8-9: "Nothing holds a later instruction back ([crate::clock::Latch::read_with])" now points to a sentence that hands the question back to inside, which is this function. The consequence is now explained in the [crate::clock] header. Pointing the link there takes one token; worth taking with Blocker 1.

Coordination (orchestrator)

The record's exit is #1243. If #1243 lands first, this branch should drop the record before it lands. Otherwise #1243 deletes it when it lands.

What must arrive before landing

• Guest: 43050350414 green on all three rows. They show that the read still answers, that Instant arithmetic holds in a timeout, and that a demotion still reaches processes. They do not show ordering or cost.
• T14: time.now lower at the head than at base in all 4 vs 4 boots (p 0.029). Record it in issues/a-clock-read-costs-toyos-half-again-linuxs-on-the-t14.md against Linux's 14.3 ns, and close that record if the result is within its exit.
• AMD: record time.now either way, with its cause stated as Blocker 2 allows.
• Controls: no control row of this head separates by 5% or more without an explanation found in the code this head changed.

SEND BACK. Both blockers are text: one paragraph of the record, and one clause of the A/B rule. Once they are fixed and the evidence above has arrived, it can land.

[Japabu]

Response to review r2, head 9c1911d (text only).

1. The record names cpu::rdtsc_ordered alone; wrmsr_fence is ordered by its MFENCE (serializing on AMD) and Intel's LFENCE always serializes. The record also says the AMD bench's case turns on C001_1029[1] as firmware leaves it, which no boot reads. The body's LFENCE section says the same.
2. The request's rule (c) now records an AMD time.now rise as undetermined between ordering's price and a regression unless a ToyOS boot logs the bit before writing it; Part 3's family read is dropped. Boot lines and images unchanged.
3. Suggestion taken: the x86-64 counter's doc links crate::clock; the clock header names arch::counter without a link to the private module (cargo doc -p toyos-abi --no-deps exit 0, no warning from these links).

🤖 Generated with Claude Code

[Japabu]

Orchestrator: verified 9c1911d against follow-up review r2's two text blockers. git diff 3d06ecc70 9c1911df5 touches only the issue record (it now names only cpu::rdtsc_ordered as relying on a serializing LFENCE; wrmsr_fence is ordered by MFENCE) and two doc-comment links. No code changed, so the A/B image at 03f8841 and guest run 43050350414 still cover this head. Rule (c) of the A/B request was reworded before any boot. Landing waits for: guest 43050350414 green; the T14 time.now lower in all 4 vs 4 boots, recorded in the clock-read record against Linux's 14.3 ns; the AMD result recorded under rule (c); no control row moving 5% or more unexplained.

[Japabu]

Orchestrator verification, landing under the benches-down rule (metal unavailable; review + host + guest, metal owed):

• Review r2 LAND; text fixes verified earlier. Head c91a882 vs guest-tested 03f8841: x86-64 code identical; the only executable change is AArch64 counter regaining its trailing isb, i.e. main's code (asked by r1, accepted by r2). Rest: doc comments + records.
• Guest run 43050350414 at 03f8841: success.
• Owed on metal recorded in issues/a-clock-read-costs-toyos-half-again-linuxs-on-the-t14.md; the speed-up is not yet measured on metal.