diff --git a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md index ef047e352f4..0688bf01967 100644 --- a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md +++ b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md @@ -6,12 +6,13 @@ opened: 2026-10-24 # [tcp] opens a connection's window slower than Linux in its first 300 ms -On the T14's wired path to the AMD bench's Wi-Fi peer, a ToyOS TCP sender -delivers 2.1–3.3 MB in a new connection's first 100 ms and 24–25 MB by -300 ms, where the bench's Linux delivers 6.4–6.5 MB and 29.7 MB. From then on -both run at line rate. The upload row reads 112 MB/s against Linux's 113 in -the same session (#1102's A/B, at the end of this record). Received TCP is at -parity on that path. +On the T14's wired path to the AMD bench's Wi-Fi peer, a paced ToyOS TCP +sender (#1114) delivers 6.07 MB in a new connection's first 100 ms and +29.2 MB by 300 ms. In the same session the bench's Linux delivered +6.10–6.38 MB and 28.6–29.6 MB. The upload row reads 114 MB/s against +Linux's 113. Only the 100 ms mark is short, by 30 KB in one boot (the +parity boot, at the end of this record). Before #1114, ToyOS delivered +2.0–3.3 MB and 23.8–25.6 MB. Received TCP is at parity on that path. This record began as the upload at a third of Linux's (41 MB/s against 113). What follows is that history, the causes found and removed, and the residual. @@ -243,8 +244,76 @@ readings, `mhz` just after it and `mhz_settled` before it, were about 1.5 GHz (settled 1.5 or 4.2 GHz), and Linux still moved 113 MB/s. At most, the clock matters through what ToyOS spends per byte sent. +**#1112's session, 2026-10-29** +([evidence](https://github.com/ToyOSOrg/evidence/tree/96c693b58d242d8ec236e0805b34b714306ec4c2/r1/t14)). +B was #1102's head, C per-segment RTT samples (#1112), D C with L = ∞ and +no pacing, a diagnostic arm that does not land. Three of each, B C D B C D +B C D, with Linux at both ends: + +| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | +|---|---|---|---|---| +| Linux (2 runs) | 113, 114 | 5.9–6.0 MB | 27.2–29.0 MB | 108–110 MB | +| B (3 runs) | 112, 112, 112 | 2.1–2.4 MB | 24.0–24.7 MB | 105.5–105.8 MB | +| C (3 runs) | 112, 112, 112 | 2.2–3.2 MB | 24.3–25.6 MB | 105.2–106.6 MB | +| D (3 runs) | 114, 113, 114 | 5.2–6.4 MB | 28.3–29.7 MB | 109.9–111.2 MB | + +- L = 8 is what holds the ramp. Without it D reached Linux's marks, and + nothing was resent. +- CSS entries were 2.0–2.7 times the rounds that entered CSS, in D as in B + and C. Most of them are flaps within a round (s_cc_023), and removing them + was not needed for parity. +- Finer RTT samples (C) did not change the ramp. + +RFC 9406 §4.3 gives L = ∞ only to a paced sender. Pull request #1114 +paces [tcp] (RFC 9002 §7.7) and removes the cap. + +**#1114's A/B, 2026-10-31** +([evidence](https://github.com/ToyOSOrg/evidence/tree/58dcdecab592116996c0e3cc84e4e3c8b7aa50c7/r4/t14)). +C was #1112 after its first review, P #1114's pacing with N = 2 in slow +start, Q P with N = 1.25 throughout. Three of each, C P Q C P Q C P Q, with +Linux at both ends, on trees from before netstack's drivers became their +own processes (#1106): + +| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | +|---|---|---|---|---| +| Linux (2 runs) | 112, 112 | 5.7 MB | 26.1–26.2 MB | 105.7–106.4 MB | +| C (3 runs) | 112, 112, 112 | 2.0–3.1 MB | 23.8–25.4 MB | 105.4–106.8 MB | +| P (3 runs) | 114, 114, 114 | 6.3–6.6 MB | 29.3–29.9 MB | 110.9–111.1 MB | +| Q (3 runs) | 114, 114, 114 | 5.9–6.3 MB | 29.1–29.5 MB | 110.2–111.0 MB | + +- P beat C at 300 ms in every pair, and resent nothing. +- P beat Q at 100 ms in every pair, by 0.1–0.5 MB, so N = 2 in slow start + stays. +- In C every RTT sample but two came from a segment's departure. +- That session's bucket could hold up to a millisecond at the rate, which on + a short round trip is more than the window. #1114's pacer, as of + 3a0744d34, holds at most half the window. The parity read waits on a boot + of that head, on current main. + +**The parity boot, 2026-11-01** +([evidence](https://github.com/ToyOSOrg/evidence/tree/8f08a8e9399817fc669e395dc717ee236a18ed35/r7/t14)). +One boot of #1114 at 3a0744d34, on main after #1106, with Linux before and +after it: + +| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | 2 s | +|---|---|---|---|---|---| +| Linux (2 runs) | 113, 113 | 6.10, 6.38 MB | 28.63, 29.56 MB | 109.1, 110.5 MB | 224.6, 226.1 MB | +| ToyOS | 114 | 6.07 MB | 29.19 MB | 110.7 MB | 227.0 MB | + +- The rule set before the run asked for the 100 ms and 300 ms marks at or + above the lower Linux run's, p50 within 1 MB/s, and nothing resent. The + 300 ms mark, p50 and loss passed. The 100 ms mark missed by 30 KB, 0.5 %, + so the rule is not met and this record stays open. +- That gap is smaller than the spread of the marks across runs: Linux's two + runs differ by 280 KB, and the A/B's three P runs by 260 KB. One boot does + not separate it from noise. +- Nothing was resent, and no RTO, recovery or loss probe occurred. Every RTT + sample came from a departure (33,319, none from the echo). CSS entered in + 134 rounds, 420 times, and never exited. +- The pacer held 73 ms over the row, against 33–41 ms with the A/B's + bucket. The half-window bound holds more on the short round trip, as + expected. + **Exit**: `osbench_lan` on the T14 reads `lan.wired.tcp.upload` at Linux's in the same session, with the first 300 ms read at the receiver no slower, or -the cause is named and owned by its own issue. The next leads are the CSS -oscillation on this path (RFC 9406 §4.3 lets an implementation tune -`MIN_RTT_THRESH` and `N_RTT_SAMPLE`) and pacing with L = ∞ (§4.3). +the cause is named and owned by its own issue. diff --git a/system/netstack/shard/tcp/src/cc.rs b/system/netstack/shard/tcp/src/cc.rs index 2012b795580..89f0e4d2c6a 100644 --- a/system/netstack/shard/tcp/src/cc.rs +++ b/system/netstack/shard/tcp/src/cc.rs @@ -19,8 +19,7 @@ const CUBE_DENOMINATOR: i128 = 5_000_000_000_000_000_000; /// The farthest from `K` the cubic is evaluated, so the cube stays inside `i128`. const CUBE_SPAN_US: i128 = 1 << 34; -/// RFC 9406 §4.3's constants. L is the non-paced value: this sender does not pace. -const L: u32 = 8; +/// RFC 9406 §4.3's constants; L is infinity, as for a paced sender (`pace`). const MIN_RTT_THRESH: Duration = Duration::from_millis(4); const MAX_RTT_THRESH: Duration = Duration::from_millis(16); const MIN_RTT_DIVISOR: u32 = 8; @@ -103,9 +102,8 @@ fn rtt_thresh(last_round_min: Duration) -> Duration { impl HyStart { /// What the ACK grows cwnd by when the window was in use, of `bytes` newly acknowledged. - fn growth(&self, bytes: u32, smss: u32) -> u32 { - let capped = bytes.min(smss.saturating_mul(L)); - if self.css.is_some() { capped / CSS_GROWTH_DIVISOR } else { capped } + fn growth(&self, bytes: u32) -> u32 { + if self.css.is_some() { bytes / CSS_GROWTH_DIVISOR } else { bytes } } /// The ACK's RTT sample and round, after its growth. @@ -180,8 +178,9 @@ pub struct Cc { pub ssthresh: u32, pub smss: u32, /// The window is in use: cwnd stopped the sender's last transmit opportunity with data - /// waiting, or the caller's credit ended it with at least half of cwnd in flight (`hand_off`). - /// Every ACK until the next opportunity grows the window, however many one pass brings. + /// waiting, or the caller's credit or the pacer ended it with at least half of cwnd in flight + /// (`hand_off`). Every ACK until the next opportunity grows the window, however many one pass + /// brings. pub limited: bool, w_max: Option, /// RFC 9438 §4.3's cwnd_prior: cwnd when ssthresh was last set, and before that the initial @@ -217,7 +216,7 @@ impl Cc { let since_last = self.last_ack.replace(now).map_or(Duration::ZERO, |last| now.since(last)); if self.ssthresh == SSTHRESH_INITIAL { if grow { - self.cwnd = self.cwnd.saturating_add(self.hystart.growth(acked, self.smss)); + self.cwnd = self.cwnd.saturating_add(self.hystart.growth(acked)); } let change = self.hystart.observe(ack); if change == Some(SlowStart::Ended) { @@ -569,14 +568,15 @@ mod tests { assert_eq!(r.cc.ssthresh, SSTHRESH_INITIAL); } - /// RFC 9406 runs only in the initial slow start: after a loss, slow start is RFC 5681 §3.1's, - /// one SMSS per ACK at most. + /// RFC 9406 runs only in the initial slow start, where a paced sender grows by all an ACK + /// acknowledged (L = infinity, §4.3): after a loss, slow start is RFC 5681 §3.1's, one SMSS per + /// ACK at most. #[test] fn s_cc_022_l_applies_only_to_the_initial_slow_start() { let mut cc = Cc::new(1460, 0); cc.limited = true; cc.on_ack(&ack(at(0), 20_000, None)); - assert_eq!(cc.cwnd, 14_600 + 8 * 1460); + assert_eq!(cc.cwnd, 14_600 + 20_000); cc.on_timeout(cc.cwnd, false); cc.limited = true; cc.on_ack(&ack(at(1), 20_000, None)); diff --git a/system/netstack/shard/tcp/src/conn.rs b/system/netstack/shard/tcp/src/conn.rs index 14012209fce..cd76d9b2705 100644 --- a/system/netstack/shard/tcp/src/conn.rs +++ b/system/netstack/shard/tcp/src/conn.rs @@ -11,6 +11,7 @@ use toyos_net_wire::tcp::{SackBlock, SynOptions, TcpFlags, TcpOptions, TcpSegmen use crate::cc::{self, Cc, SlowStart}; use crate::counters::{Counter, Log}; +use crate::pace::Pacer; use crate::ring::{Lent, Ring}; use crate::rtt::{Rtt, RTO_AFTER_HANDSHAKE_LOSS, RTO_MAX}; use crate::rx::{Placed, Rx}; @@ -351,13 +352,13 @@ pub struct Sync { timeout_rtx: bool, /// When the new data in flight left, while none of it has been resent since (RFC 6298 §3, /// Karn): the samples an ACK gives at the clock's own resolution (RFC 8985 §6.1). Every send - /// is timed. A send at the same instant as the last joins its run, and so does any send after - /// a run shorter than SMSS until an ACK lands in the run. Only the oldest run can be both short - /// and closed, so the record holds at most a run for each SMSS in flight and two more. An ACK - /// of a whole run is timed from its last send, and one partway into a run that left at one - /// instant from that instant. In a run of several instants, the first ACK partway is timed - /// from its first send, which overstates it by less than the joins took before that ACK ended - /// them; a later partway ACK is not timed. + /// is timed. A send at the same instant as the last joins its run, and so does one shorter than + /// SMSS after a run shorter than SMSS until an ACK lands in the run, so a burst after a short + /// header starts its own run and its ACKs are each timed. The record holds at most two runs + /// for each SMSS in flight and two more. An ACK of a whole run is timed from its last send, and + /// one partway into a run that left at one instant from that instant. In a run of several + /// instants, the first ACK partway is timed from its first send, which overstates it by less + /// than the joins took before that ACK ended them; a later partway ACK is not timed. departures: VecDeque, /// RFC 8985 §7: when the loss probe is due, whether it is, the probe in flight (SND.NXT once /// it left and whether it was a retransmission), and whether an RTT sample came since the last. @@ -368,6 +369,9 @@ pub struct Sync { persist: Option, sws: Option, sws_fired: bool, + pacer: Pacer, + /// When the pacer releases the new data it last held. + paced: Option, last_data_sent: Option, progress_since: Option, acked_since: Option, @@ -389,8 +393,9 @@ pub struct Sync { retransmission_held: Option, } -/// What a sender's new data waits on: RFC 4898 §4's `tcpEStatsPerfSndLimTime{Snd,Rwin,Cwnd}`. -/// Unlike the RFC, which divides all of a connection's time among the three, a span here runs from +/// What a sender's new data waits on: RFC 4898 §4's `tcpEStatsPerfSndLimTime{Snd,Rwin,Cwnd}`, and +/// the pacer. +/// Unlike the RFC, which divides all of a connection's time among its three, a span here runs from /// a transmit opportunity that left new data unsent to the next one that sends some: time spent /// sending, or waiting for the caller's next opportunity, is charged to none. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -401,6 +406,8 @@ enum SendLimit { Window, /// The congestion window, or loss recovery's pipe. Congestion, + /// The pacer, with room in both windows. + Pacing, } impl SendLimit { @@ -409,6 +416,7 @@ impl SendLimit { Self::User => Counter::SendLimitedByUserNanos, Self::Window => Counter::SendLimitedByWindowNanos, Self::Congestion => Counter::SendLimitedByCongestionNanos, + Self::Pacing => Counter::SendLimitedByPacingNanos, } } } @@ -477,6 +485,8 @@ impl Sync { persist: None, sws: None, sws_fired: false, + pacer: Pacer::default(), + paced: None, last_data_sent: None, progress_since: None, acked_since: None, @@ -1127,6 +1137,7 @@ impl Sync { self.persist.filter(|p| !p.due).map(|p| p.at), self.keepalive_at(ctx), self.sws.filter(|_| !self.sws_fired), + self.paced, self.rx.delayed, self.give_up_at(ctx), self.orphan_since.map(|at| at.after(limits::ORPHAN_IDLE)), @@ -1169,6 +1180,9 @@ impl Sync { self.sws = None; self.sws_fired = true; } + if self.paced.is_some_and(|at| at <= now) { + self.paced = None; + } self.rx.delayed_due(now); Tick::Keep } @@ -1233,7 +1247,8 @@ impl Sync { } else { Some(self.declined) }; - if let Some(limit) = held { + // A pacing hold leaves the window as the last send found it (`hand_off`). + if let Some(limit) = held.filter(|&limit| limit != SendLimit::Pacing) { self.cc.limited = limit == SendLimit::Congestion; } let was = self.held_by.map(|(limit, _)| limit); @@ -1331,6 +1346,7 @@ impl Sync { self.tx.nxt = end; } if len > 0 { + self.pacer.spend(now, len, &self.cc, self.rtt.srtt()); self.last_data_sent = Some(now); self.sws = None; self.sws_fired = false; @@ -1355,7 +1371,7 @@ impl Sync { } else if end.after(old) { let smss = self.smss(); match self.departures.back_mut() { - Some(run) if run.end == start && (run.last == now || (!run.sampled && run.end.since(run.start) < smss)) => { + Some(run) if run.end == start && (run.last == now || (!run.sampled && run.end.since(run.start) < smss && len < smss)) => { (run.end, run.last) = (end, now); } _ => self.departures.push_back(Departure { start, end, first: now, last: now, sampled: false }), @@ -1485,6 +1501,7 @@ impl Sync { /// New data and the FIN (RFC 9293 §3.7.4 Nagle, §3.8.6.2.1 sender silly-window avoidance), /// within SND.UNA + min(cwnd, SND.WND) (RFC 5681 §2). fn new_data(&mut self, ctx: &mut Ctx<'_>, exit: &mut dyn Exit) -> Result { + self.paced = None; let unsent = self.tx.unsent(); if unsent == 0 && !self.tx.fin_unsent() { self.declined = SendLimit::User; @@ -1546,6 +1563,11 @@ impl Sync { return Ok(false); } let len = u32::try_from(usable).unwrap_or(0); + if let Some(at) = self.pacer.hold(now, len, &self.cc, self.rtt.srtt()) { + self.declined = SendLimit::Pacing; + self.paced = Some(at); + return Ok(false); + } let beyond_cwnd = limited_transmit && signed(len) > cwnd.saturating_sub(signed(self.tx.flight())); let fin = self.fin_fits(start.add(len)) && signed(len) < w_rcv; self.emit(ctx, exit, start, len, fin, blocks)?; diff --git a/system/netstack/shard/tcp/src/counters.rs b/system/netstack/shard/tcp/src/counters.rs index c8a19191b58..5211c3faf39 100644 --- a/system/netstack/shard/tcp/src/counters.rs +++ b/system/netstack/shard/tcp/src/counters.rs @@ -75,6 +75,7 @@ toyos_net_wire::counters! { SendLimitedByUserNanos = "tcp.send-limited.user.ns"; SendLimitedByWindowNanos = "tcp.send-limited.window.ns"; SendLimitedByCongestionNanos = "tcp.send-limited.congestion.ns"; + SendLimitedByPacingNanos = "tcp.send-limited.pacing.ns"; CongestionStops = "tcp.congestion-stops"; CongestionStopsCwndBytes = "tcp.congestion-stops.cwnd.bytes"; CongestionStopsSrttMicros = "tcp.congestion-stops.srtt.us"; diff --git a/system/netstack/shard/tcp/src/lib.rs b/system/netstack/shard/tcp/src/lib.rs index 7472ed7aa85..05a27c8cb9e 100644 --- a/system/netstack/shard/tcp/src/lib.rs +++ b/system/netstack/shard/tcp/src/lib.rs @@ -1,9 +1,10 @@ //! ToyOS's TCP (RFC 9293) with RFC 5961's blind-attack checks, MSS, window scaling, timestamps -//! with PAWS (RFC 7323), SACK and D-SACK (RFC 2018, 2883), RFC 6298's timer, CUBIC (RFC 9438), -//! NewReno (RFC 6582) and SACK-based recovery (RFC 6675), keyed initial sequence numbers and ports -//! (RFC 6528, 6056), and TIME-WAIT hardened against assassination (RFC 1337). Pure: the caller hands -//! in the time, the secrets, parsed segments, classified ICMP errors, user calls and transmit -//! credit; nothing here reads a clock, draws randomness or does I/O. +//! with PAWS (RFC 7323), SACK and D-SACK (RFC 2018, 2883), RFC 6298's timer, CUBIC (RFC 9438) +//! after HyStart++ (RFC 9406), pacing (RFC 9002 §7.7), NewReno (RFC 6582) and SACK-based recovery +//! (RFC 6675), keyed initial sequence numbers and ports (RFC 6528, 6056), and TIME-WAIT hardened +//! against assassination (RFC 1337). Pure: the caller hands in the time, the secrets, parsed +//! segments, classified ICMP errors, user calls and transmit credit; nothing here reads a clock, +//! draws randomness or does I/O. //! //! **Pull egress.** A segment exists only while [`Tcp::transmit_owed`] or [`Tcp::serve`] hands it //! to the caller's sink, built from the state of that moment, once the caller has answered that its @@ -38,6 +39,7 @@ mod cc; mod conn; mod counters; mod open; +mod pace; #[cfg(test)] mod props; mod ring; diff --git a/system/netstack/shard/tcp/src/pace.rs b/system/netstack/shard/tcp/src/pace.rs new file mode 100644 index 00000000000..d34c8ad7bb3 --- /dev/null +++ b/system/netstack/shard/tcp/src/pace.rs @@ -0,0 +1,203 @@ +//! Pacing (RFC 9002 §7.7): new data leaves at `N · cwnd / srtt` through a token bucket, and every +//! data segment sent spends from it. +//! +//! The bucket holds what the rate brings in [`GRANULARITY`], at most half of cwnd, and never less +//! than the initial window. §7.7 bounds a burst at the initial window unless the path is known to +//! absorb more. The millisecond departs from that so that a fast sender's bursts, and its wakes, +//! are sized by time rather than by the initial window. The half-window bound keeps the bucket +//! below cwnd at any srtt, so a window past the initial one always leaves in more than one burst. +//! That is what lets HyStart++ take L = infinity (RFC 9406 §4.3) whatever the path's round trip. +//! +//! Some data spends from the bucket but is never held: +//! - retransmissions: recovery's pipe, and the window after a timeout from one segment, bound +//! them; +//! - new data in SACK recovery: RFC 6675's pipe and the ACK clock bound it, as they do +//! retransmissions; +//! - a loss probe of new data: RFC 8985 §7.3's single segment at the PTO, which a hold would only +//! delay; +//! - a persist probe of new data: one segment into a closed window (RFC 9293 §3.8.6.1). +//! +//! A held sender is offered its data again once the bucket holds half its capacity: it wakes once +//! per half the bucket's refill time, and a wake up to that much late loses no rate. That is half +//! a [`GRANULARITY`] while the millisecond sets the bucket, less where half the window sets it, +//! and more where the initial window does. In a small window the wait is half the initial +//! window's time at the rate, even when one segment would do. Without a smoothed RTT nothing is +//! paced; cwnd is at most the initial window until the first sample. + +use core::num::{NonZeroU128, NonZeroU64}; +use core::time::Duration; + +use crate::cc::{initial_window, Cc}; +use crate::Instant; + +/// What a full bucket holds at the pacing rate, while that is under half of cwnd and over the +/// initial window: the span that sizes a paced sender's bursts and its wakes. +const GRANULARITY: Duration = Duration::from_millis(1); + +/// RFC 9002 §7.7's N, in quarters, where cwnd holds across the round trip. +const N_AVOIDANCE: u64 = 5; +/// Slow start doubles cwnd across the round trip, so the round's data is twice the window it +/// began with: a smaller N would hold the doubling back. +const N_SLOW_START: u64 = 8; + +/// `bytes` every `per_ns`. +#[derive(Clone, Copy, Debug)] +struct Rate { + bytes: NonZeroU64, + per_ns: NonZeroU64, +} + +impl Rate { + fn of(cc: &Cc, srtt: Option) -> Option { + let per_ns = NonZeroU64::new(u64::try_from(srtt?.as_nanos()).unwrap_or(u64::MAX))?; + let quarters = if cc.cwnd < cc.ssthresh { N_SLOW_START } else { N_AVOIDANCE }; + let bytes = NonZeroU64::new(u64::from(cc.cwnd).saturating_mul(quarters) / 4)?; + Some(Self { bytes, per_ns }) + } + + fn over(self, d: Duration) -> u64 { + let n = u128::from(self.bytes.get()).saturating_mul(d.as_nanos()) / NonZeroU128::from(self.per_ns); + u64::try_from(n).unwrap_or(u64::MAX) + } + + /// How long the rate takes to bring `bytes`, rounded up. + fn time_for(self, bytes: u64) -> Duration { + let ns = u128::from(bytes).saturating_mul(u128::from(self.per_ns.get())).div_ceil(u128::from(self.bytes.get())); + Duration::from_nanos(u64::try_from(ns).unwrap_or(u64::MAX)) + } + + fn capacity(self, cc: &Cc) -> u64 { + self.over(GRANULARITY).min(u64::from(cc.cwnd / 2)).max(u64::from(initial_window(cc.smss))) + } +} + +/// A connection's bucket: full until it first spends. +#[derive(Clone, Copy, Debug, Default)] +pub struct Pacer { + tokens: u64, + at: Option, +} + +impl Pacer { + fn refill(&mut self, now: Instant, rate: Rate, capacity: u64) { + self.tokens = match self.at { + Some(at) => self.tokens.saturating_add(rate.over(now.since(at))).min(capacity), + None => capacity, + }; + self.at = Some(now); + } + + /// `None` if a segment of `len` bytes may leave now; else when to offer the held data again. + pub fn hold(&mut self, now: Instant, len: u32, cc: &Cc, srtt: Option) -> Option { + let rate = Rate::of(cc, srtt)?; + let capacity = rate.capacity(cc); + self.refill(now, rate, capacity); + let len = u64::from(len); + if self.tokens >= len { + return None; + } + let release = (capacity / 2).max(len); + Some(now.after(rate.time_for(release.saturating_sub(self.tokens)))) + } + + /// `len` bytes of data left, new or resent. + pub fn spend(&mut self, now: Instant, len: u32, cc: &Cc, srtt: Option) { + if let Some(rate) = Rate::of(cc, srtt) { + self.refill(now, rate, rate.capacity(cc)); + self.tokens = self.tokens.saturating_sub(u64::from(len)); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const MS: Duration = Duration::from_millis(1); + + fn at(us: u64) -> Instant { + Instant::from_nanos(us * 1_000) + } + + /// cwnd, in slow start or in avoidance, with SMSS 1,460 and an initial window of 14,600. + fn cc(cwnd: u32, slow_start: bool) -> Cc { + let mut cc = Cc::new(1460, 0); + cc.cwnd = cwnd; + if !slow_start { + cc.ssthresh = cwnd; + } + cc + } + + /// Segments of 1,460 a pacer lets leave at `now` before it holds one, and when it releases it. + fn burst(p: &mut Pacer, now: Instant, cc: &Cc, srtt: Duration) -> (u32, Option) { + let mut n = 0; + loop { + if let Some(release) = p.hold(now, 1460, cc, Some(srtt)) { + return (n, Some(release)); + } + p.spend(now, 1460, cc, Some(srtt)); + n += 1; + assert!(n < 100_000, "never held"); + } + } + + #[test] + fn without_a_smoothed_rtt_nothing_is_paced() { + let mut p = Pacer::default(); + let cc = cc(14_600, true); + for _ in 0..100 { + assert_eq!(p.hold(at(0), 1460, &cc, None), None); + p.spend(at(0), 1460, &cc, None); + } + assert_eq!(p.hold(at(0), 1460, &cc, Some(Duration::ZERO)), None); + } + + /// RFC 9002 §7.7: a burst is the initial window while the rate brings less in a millisecond. In + /// slow start the rate is 2 · 29,200 / 10 ms, 5,840 bytes a millisecond, so the held segment + /// leaves once half the bucket, 7,300 bytes, has come: 1.25 ms. + #[test] + fn a_slow_bucket_holds_the_initial_window_and_releases_at_half() { + let mut p = Pacer::default(); + let cc = cc(29_200, true); + assert_eq!(burst(&mut p, at(0), &cc, 10 * MS), (10, Some(at(1_250)))); + assert_eq!(burst(&mut p, at(1_250), &cc, 10 * MS), (5, Some(at(2_500)))); + } + + /// RFC 9002 §7.7's N = 1.25 once cwnd holds across the round trip: the same half bucket takes + /// 7,300 / (1.25 · 29,200 / 10 ms), 2 ms. + #[test] + fn avoidance_paces_at_a_quarter_over_the_window() { + let mut p = Pacer::default(); + let cc = cc(29_200, false); + assert_eq!(burst(&mut p, at(0), &cc, 10 * MS), (10, Some(at(2_000)))); + } + + /// Past the initial window the bucket holds a millisecond at the rate: 2 · 1,460,000 / 8 ms is + /// 365,000 bytes a millisecond, 250 segments, under half the window's thousand. + #[test] + fn a_fast_bucket_holds_a_millisecond() { + let mut p = Pacer::default(); + let cc = cc(1_460_000, true); + assert_eq!(burst(&mut p, at(0), &cc, 8 * MS), (250, Some(at(500)))); + } + + /// On a short round trip a millisecond at the rate is more than the window: at 2 · 146,000 / + /// 1 ms the bucket holds half the window, 50 of its 100 segments, so the window still leaves in + /// bursts, the next half bucket 125 µs later. + #[test] + fn a_bucket_holds_at_most_half_the_window() { + let mut p = Pacer::default(); + let cc = cc(146_000, true); + assert_eq!(burst(&mut p, at(0), &cc, MS), (50, Some(at(125)))); + } + + /// However long the sender was idle, the bucket holds no more than its capacity. + #[test] + fn an_idle_bucket_fills_only_to_its_capacity() { + let mut p = Pacer::default(); + let cc = cc(29_200, true); + burst(&mut p, at(0), &cc, 10 * MS); + assert_eq!(burst(&mut p, at(60_000_000), &cc, 10 * MS).0, 10); + } +} diff --git a/system/netstack/shard/tcp/tests/congestion.rs b/system/netstack/shard/tcp/tests/congestion.rs index c381f4def6b..243190a9699 100644 --- a/system/netstack/shard/tcp/tests/congestion.rs +++ b/system/netstack/shard/tcp/tests/congestion.rs @@ -111,27 +111,28 @@ fn s_cc_016_acks_in_one_pass_each_grow_slow_start() { /// half, it is application limited (§5.8), and grows nothing. #[test] fn s_cc_017_a_sender_the_card_holds_grows_a_window_half_in_flight() { - for (credit, grown) in [(9, 1460), (8, 0)] { + for (credit, grown) in [(10, 1460), (9, 0)] { let mut h = fixture_e(); assert_eq!(h.send(0, 60_000).len(), 10, "a stop at the initial window"); h.credit = Some(credit); - assert_eq!(h.deliver(seg(5001).ack(15_601)).len(), credit); + assert_eq!(h.input(10, seg(5001).ack(15_601)).len(), credit); let cwnd = h.info().cwnd; - assert_eq!(cwnd, 26_280, "the ACK after cwnd's stop grows it"); + assert_eq!(cwnd, 29_200, "the ACK after cwnd's stop grows it"); h.credit = Some(0); h.deliver(seg(5001).ack(17_061)); - assert_eq!(h.info().cwnd - cwnd, grown, "{credit} of the half's 9 segments in flight"); + assert_eq!(h.info().cwnd - cwnd, grown, "{credit} of the half's 10 segments in flight"); } } -/// RFC 9406 §4.2 with §4.3's L = 8 for a sender that does not pace: an ACK of the whole flight -/// grows the initial slow start by eight segments, where RFC 5681 §3.1 grows it by one. +/// RFC 9406 §4.2 with §4.3's L = infinity for a paced sender: an ACK of the whole flight grows the +/// initial slow start by the whole flight, where RFC 5681 §3.1 grows it by one segment. #[test] -fn s_cc_018_a_stretch_ack_grows_the_initial_slow_start_by_eight_segments() { +fn s_cc_018_a_stretch_ack_grows_the_initial_slow_start_by_all_it_acknowledged() { let mut h = fixture_e(); assert_eq!(h.send(0, 60_000).len(), 10); - assert_eq!(h.input(10, seg(5001).ack(15_601)).len(), 18); - assert_eq!(h.info().cwnd, 26_280); - h.input(20, seg(5001).ack(41_881)); - assert_eq!(h.info().cwnd, 37_960); + h.input(10, seg(5001).ack(15_601)); + assert_eq!(h.info().cwnd, 29_200); + assert_eq!(h.at(19).len(), 10, "the pacer releases the rest of the window"); + h.input(20, seg(5001).ack(44_801)); + assert_eq!(h.info().cwnd, 58_400); } diff --git a/system/netstack/shard/tcp/tests/net.rs b/system/netstack/shard/tcp/tests/net.rs index eab404b0a34..e36af955f90 100644 --- a/system/netstack/shard/tcp/tests/net.rs +++ b/system/netstack/shard/tcp/tests/net.rs @@ -481,9 +481,13 @@ fn syn_offering_1392(from: usize, o: &O) -> Option> { }) } +/// What a reader that keeps up on a gigabit path with a 15 ms round trip grows its window to at +/// least: twice what it reads per round trip. +const TWICE_THE_PATH: u32 = 2 * 125_000 * 15; + /// A download shaped like the T14's from a CDN: gigabit, a 15 ms round trip, timestamps and SACK, /// and a peer whose segments are shorter than the downloader's own send MSS. Full-sized is what -/// the peer sends, so its echoes are sampled and the window grows to the whole buffer; measured +/// the peer sends, so its echoes are sampled and the window grows to twice the path; measured /// against the send MSS, none ever was, and the window stayed at 65,535. #[test] fn a_peer_sending_segments_shorter_than_our_send_mss_still_grows_the_window() { @@ -497,12 +501,12 @@ fn a_peer_sending_segments_shorter_than_our_send_mss_still_grows_the_window() { net.assert_exact(); let longest = net.wire.iter().filter(|(from, _)| *from == 1).map(|(_, o)| o.payload.len()).max(); assert_eq!((longest, peaks[0].shifts), (Some(1380), Some((7, 7)))); - assert_eq!(peaks[0].window, BUFFER, "{:?}", peaks[0]); + assert!(peaks[0].window >= TWICE_THE_PATH, "{:?}", peaks[0]); } /// The same download into a lent receive ring, as a stream's region is: the reader takes text out /// of the ring itself and gives it back with `consume`, which grows the window as a read does, so -/// the window passes 65,535 and reaches the whole buffer. +/// the window passes 65,535 and reaches twice the path. #[test] fn a_lent_receive_ring_grows_the_window_as_a_read_does() { let mut net = Net::buffered(15, BUFFER); @@ -514,8 +518,7 @@ fn a_lent_receive_ring_grows_the_window_as_a_read_does() { let peaks = watch(&mut net, 600_000); assert!(net.finished(), "the transfer did not finish:\n{}", net.dump()); net.assert_exact(); - assert!(peaks[0].window > 65_535, "{:?}", peaks[0]); - assert_eq!(peaks[0].window, BUFFER, "{:?}", peaks[0]); + assert!(peaks[0].window >= TWICE_THE_PATH, "{:?}", peaks[0]); } /// RFC 8985 §7.3: no probe leaves unless an RTT sample came since the last. Without timestamps a diff --git a/system/netstack/shard/tcp/tests/pacing.rs b/system/netstack/shard/tcp/tests/pacing.rs new file mode 100644 index 00000000000..7ac9bb4ae91 --- /dev/null +++ b/system/netstack/shard/tcp/tests/pacing.rs @@ -0,0 +1,45 @@ +//! Pacing (RFC 9002 §7.7) as a connection meets it. The bucket's arithmetic lives beside it in +//! `src/pace.rs`. + +mod common; + +use common::*; +use toyos_net_tcp::Counter; + +/// Fixture E's window after the ACK of its initial ten segments at 10 ms: cwnd 29,200 in slow +/// start, paced at 2 · 29,200 / 10 ms. The bucket held the initial window, so ten leave with the +/// ACK and the rest once half the bucket has come back, 7,300 bytes in 1.25 ms each time. +fn released() -> H { + let mut h = fixture_e(); + assert_eq!(h.send(0, 60_000).len(), 10); + assert_eq!(h.input(10, seg(5001).ack(15_601)).len(), 10, "the bucket holds the initial window"); + assert_eq!(h.info().cwnd, 29_200); + h +} + +#[test] +fn a_window_past_the_bucket_leaves_at_the_pacing_rate() { + let mut h = released(); + assert_eq!(h.at(11).len(), 0, "nothing before 11.25 ms"); + assert_eq!(h.at(12).len(), 5); + assert_eq!(h.at(13).len(), 5, "and cwnd is full"); + assert_eq!(h.at(20).len(), 0); + assert_eq!(h.count(Counter::SendLimitedByPacingNanos), 2_500_000); +} + +/// A pacing hold leaves the window in use as the send before it found it: the ACK that arrives +/// during it grows cwnd. +#[test] +fn an_ack_during_a_pacing_hold_grows_the_window() { + let mut h = released(); + h.input_past(10, 500, seg(5001).ack(17_061)); + assert_eq!(h.info().cwnd, 30_660); +} + +/// RFC 9002 §7.7: a segment carrying only an acknowledgment is not paced. +#[test] +fn an_acknowledgment_leaves_through_a_pacing_hold() { + let mut h = released(); + h.arrive(seg(5001).ack(15_601).len(1460)); + expect(&h.deliver(seg(6461).ack(15_601).len(1460)), &["ACK=7921 LEN=0"]); +} diff --git a/system/netstack/shard/tcp/tests/rtt.rs b/system/netstack/shard/tcp/tests/rtt.rs index fcd7246c9fc..db4b13f8faa 100644 --- a/system/netstack/shard/tcp/tests/rtt.rs +++ b/system/netstack/shard/tcp/tests/rtt.rs @@ -265,3 +265,18 @@ fn s_rt_024_a_later_ack_partway_into_a_run_of_writes_is_not_timed() { h.input(28, seg(5001).ack(1301)); assert_eq!(h.info().srtt, Some(std::time::Duration::from_micros(12_343)), "a 20 ms sample"); } + +/// HTTP's shape: a short header, then a window of body a millisecond later. The body is its own +/// run, not joined to the header's, so every ACK into it is a sample: four ACKs, four samples. +#[test] +fn s_rt_025_a_body_after_a_header_is_timed_by_each_ack() { + let mut h = fixture_e(); + let (_, outs) = h.call(0, |tcp, now, id| tcp.set_options(now, id, toyos_net_tcp::Options { nodelay: true, ..Default::default() })); + nothing(&outs); + assert_eq!(h.send(0, 200).len(), 1); + assert_eq!(h.send(1, 4 * 1460).len(), 4); + for (t, ack) in [(20, 1201), (21, 1201 + 1460), (22, 1201 + 3 * 1460), (23, 1201 + 4 * 1460)] { + h.input(t, seg(5001).ack(ack)); + } + assert_eq!(h.count(Counter::RttFromDeparture), 4); +} diff --git a/tests/guest/osbench/src/sys/toyos.rs b/tests/guest/osbench/src/sys/toyos.rs index d03a6fe200b..c15d6a9a55c 100644 --- a/tests/guest/osbench/src/sys/toyos.rs +++ b/tests/guest/osbench/src/sys/toyos.rs @@ -477,7 +477,7 @@ pub fn persistent_dir() -> Option<(String, String, String)> { /// What a transfer row reports of the stack beneath it, as netstack's `inspect` answers it: [tcp]'s /// and the card's counters. Those a card does not report are left out. pub fn net_counters() -> Result, String> { - const READ: [&str; 32] = [ + const READ: [&str; 33] = [ "net.tcp.sent.bytes", "net.tcp.retransmit.bytes", "net.tcp.rto", @@ -491,6 +491,7 @@ pub fn net_counters() -> Result, String> { "net.tcp.send-limited.user.ns", "net.tcp.send-limited.window.ns", "net.tcp.send-limited.congestion.ns", + "net.tcp.send-limited.pacing.ns", "net.tcp.congestion-stops", "net.tcp.congestion-stops.cwnd.bytes", "net.tcp.congestion-stops.srtt.us",