From d31b0696823495bb40234cb862f87f192b158868 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 20:18:00 +0100 Subject: [PATCH 1/3] net: a stream's local address is netstack's, not QEMU's 10.0.2.15 std's TcpStream::local_addr and libc's getsockname answered QEMU's user network address with the stream's port, whatever the machine leased. Since #1122 osbench names a connection's link by that address, so on the T14 osbench_lan refused its group: "no netstack link holds 10.0.2.15". netstack's connect and accept answers and its ADOPT now carry the address the stream left from with its port: the node's StreamEvent::Connected, Accepted and export give the connection's local Endpoint, the tuple [tcp] holds. The SDK takes it only as a stream's end can be, a unicast address and a port; a connect or accept answer naming another is refused as a protocol error, and an ADOPT naming one as malformed. stream::endpoints returns it, std's local_addr and libc's getsockname answer it, and getsockname answers a listener's or datagram socket's bound address rather than the literal. Connected and Accepted lose their unread local_port. The guest jobs that play netstack (stream_handles, libc_recv_beside_close) name a TEST-NET-1 local address in their answers. The node's host tests assert the lease's address on connect, accept and export. link_failover opens a stream to the host on each of its two QEMU user networks and asserts each one's local address is its link's lease. The two leases differ, so one is not 10.0.2.15 and the value is netstack's: with std's literal restored the job fails, whichever order the links came up in. issues/getsockname-answers-an-address-nobody-asked-netstack-for.md is closed, and its references go. Co-Authored-By: Claude Opus 5.5 --- ...rs-an-address-nobody-asked-netstack-for.md | 39 ---------- ...w-slower-than-linux-in-its-first-300-ms.md | 4 +- ...oyos-runs-in-any-open-source-hypervisor.md | 2 - sdk/libc/src/socket.rs | 12 +-- sdk/std/sys/net/connection.rs | 4 +- sdk/toyos/src/net.rs | 20 +++-- sdk/toyos/src/net/stream.rs | 77 +++++++++++++------ system/netstack/node/src/lease/tcp.rs | 10 +-- system/netstack/node/src/listeners.rs | 4 +- system/netstack/node/src/streams.rs | 12 +-- system/netstack/node/tests/listeners.rs | 4 +- system/netstack/node/tests/streams.rs | 8 +- system/netstack/src/serve.rs | 8 +- .../src/bin/libc_recv_beside_close.rs | 2 +- .../guest/rust-tests/src/bin/link_failover.rs | 31 ++++++-- .../rust-tests/src/bin/stream_handles.rs | 4 +- tests/toyos.rs | 6 ++ 17 files changed, 134 insertions(+), 113 deletions(-) delete mode 100644 issues/getsockname-answers-an-address-nobody-asked-netstack-for.md diff --git a/issues/getsockname-answers-an-address-nobody-asked-netstack-for.md b/issues/getsockname-answers-an-address-nobody-asked-netstack-for.md deleted file mode 100644 index c5855f4c709..00000000000 --- a/issues/getsockname-answers-an-address-nobody-asked-netstack-for.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-09-08 ---- - -# `getsockname` answers an address nobody asked netd for - -`sdk/libc/src/socket.rs`'s `getsockname()` fills the caller's `sockaddr` -with `[10, 0, 2, 15]` and the socket's local port. The address is a literal in -that function; netd is not asked, and the SDK has nothing to ask it with — -`toyos::net` carries a stream's connect, a listen, an accept, the datagram -calls and `dns_lookup`, and no call that answers "what is this machine's -address": a socket's `toyos::net::socket::local` is the address its bind -named. - -It was true by coincidence until now: netd carried the same literal, so the -shim and the stack agreed. netd takes its address from DHCP as of the change -that filed this, so the two agree only on a machine whose server happens to -lease `10.0.2.15` — QEMU's user-mode backend does, and the bench's router does -not. Every C program that asks what address it is bound to is told the wrong -one there. - -What it costs to fix is a message type on netd's protocol and one plumbed -answer; what it costs to leave is that the one caller of `getsockname` in a -POSIX program is the one that then advertises an address nothing can reach. -Nothing in the tree reads it today, which is why this is a finding and not a -defect. - -**Now a defect, 2026-11-01.** std carries the same literal: -`sdk/std/sys/net/connection.rs`'s `TcpStream::socket_addr` answers -`10.0.2.15` with the stream's local port. Since #1122, osbench's -`sys::link_of` names a connection's link by its local address, so on any -network that leases another address `osbench_lan` refuses the group. The -T14 session for the ramp issue read `bench=lan refused=no_netstack_link_holds_10.0.2.15` -on a lease from the bench's router. QEMU's guest test passes only because -slirp leases that address. The fix gives a stream's local address with -its port: netstack's connect and accept answers carry it, and the SDK's -`stream::endpoints`, std and libc return it. diff --git a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md index a37e0ac54c9..3e1f96e5561 100644 --- a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md +++ b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md @@ -355,8 +355,8 @@ exiting 0. ToyOS's `osbench_lan` found the peer and then refused the group: `no netstack link holds 10.0.2.15`. std's `TcpStream::local_addr` answers QEMU's user-network address whatever the lease is, and since #1122 osbench names a connection's link by that address. The T14's lease is not it. So the -session does not decide, and the next one waits on -`issues/getsockname-answers-an-address-nobody-asked-netstack-for.md`. +session does not decide. The next one runs on a head where netstack's +answers carry each stream's local address and std, libc and the SDK return it. Its Linux runs do give the per-connection spread the rule needs: diff --git a/issues/toyos-runs-in-any-open-source-hypervisor.md b/issues/toyos-runs-in-any-open-source-hypervisor.md index 665c899b084..d4ed6e01437 100644 --- a/issues/toyos-runs-in-any-open-source-hypervisor.md +++ b/issues/toyos-runs-in-any-open-source-hypervisor.md @@ -39,8 +39,6 @@ and, filed before this track: - `issues/a-machine-without-an-iommu-refuses-every-claim.md` — bhyve; crosvm for every device it emulates; and VirtualBox, QEMU and Cloud Hypervisor by default (`issues/no-claim-gets-an-address-space-under-virtio-iommu.md`). -- `issues/getsockname-answers-an-address-nobody-asked-netstack-for.md` — - QEMU's user-mode network address in `sdk/libc`. - `issues/pci-is-enumerated-on-segment-group-0-alone.md` — Cloud Hypervisor's `num_pci_segments`. diff --git a/sdk/libc/src/socket.rs b/sdk/libc/src/socket.rs index c791c8ed2a6..062881fe63f 100644 --- a/sdk/libc/src/socket.rs +++ b/sdk/libc/src/socket.rs @@ -825,23 +825,23 @@ pub unsafe extern "C" fn getsockname( addr: *mut Sockaddr, addrlen: *mut SocklenT, ) -> i32 { - let local_port = match sock_from_fd(fd) { + let (ip, port) = match sock_from_fd(fd) { Some(Sock::Stream(held)) => match stream::endpoints(held.handle()) { - Ok((_, local_port)) => local_port, + Ok((_, local)) => local, Err(refused) => { set_errno(crate::errcode::of_stream(refused)); return -1; } }, Some(Sock::Listener(held) | Sock::Datagram(held)) => match socket::local(held.handle()) { - Ok((_, port)) => port, + Ok(local) => local, Err(e) => { set_errno(net_err_to_errno(e)); return -1; } }, + // Neither bound nor connected: the unspecified address, and no port. Some(Sock::Entry(mut socket)) => match *socket.slot() { - Some(_) => 0, + Some(_) => ([0; 4], 0), None => { set_errno(EBADF); return -1; } }, None => { set_errno(EBADF); return -1; } }; - // Local address: 10.0.2.15 (QEMU default) - fill_sockaddr(addr, addrlen, [10, 0, 2, 15], local_port); + fill_sockaddr(addr, addrlen, ip, port); 0 } diff --git a/sdk/std/sys/net/connection.rs b/sdk/std/sys/net/connection.rs index 3bef6d9d3de..a8859e1a509 100644 --- a/sdk/std/sys/net/connection.rs +++ b/sdk/std/sys/net/connection.rs @@ -215,8 +215,8 @@ impl TcpStream { } pub fn socket_addr(&self) -> io::Result { - let (_, local_port) = stream::endpoints(self.handle()).map_err(stream_refused)?; - Ok(SocketAddr::V4(SocketAddrV4::new(Ipv4Addr::new(10, 0, 2, 15), local_port))) + let (_, (addr, port)) = stream::endpoints(self.handle()).map_err(stream_refused)?; + Ok(SocketAddr::V4(SocketAddrV4::new(Ipv4Addr::from(addr), port))) } pub fn shutdown(&self, how: Shutdown) -> io::Result<()> { diff --git a/sdk/toyos/src/net.rs b/sdk/toyos/src/net.rs index c77a679c7a6..85f0b6869d0 100644 --- a/sdk/toyos/src/net.rs +++ b/sdk/toyos/src/net.rs @@ -185,8 +185,9 @@ ipc_payload! { } /// The answer to [`MsgType::TcpConnect`], with the stream's bells - /// (`net::stream::BELLS`). + /// (`net::stream::BELLS`): the address and port the stream left from. pub struct TcpConnectResponse { + pub local_addr: [u8; 4], pub local_port: u16, pub _pad: u16, } @@ -221,6 +222,7 @@ ipc_payload! { pub remote_addr: [u8; 4], pub remote_port: u16, pub local_port: u16, + pub local_addr: [u8; 4], pub nodelay: u32, } @@ -539,19 +541,23 @@ mod tests { )); } - /// The accept's answer as a client decodes it: the peer, the port, and - /// `nodelay` last, any non-zero word on, and an answer without it refused. + /// The accept's answer as a client decodes it: the peer, the local port + /// and address, and `nodelay` last, any non-zero word on, and an answer + /// without it refused. #[test] fn an_accepts_answer_ends_in_its_streams_nodelay() { - let mut wire = [192, 0, 2, 7, 0x39, 0x30, 0x16, 0, 0, 0, 0, 0]; + let mut wire = [192, 0, 2, 7, 0x39, 0x30, 0x16, 0, 198, 51, 100, 9, 0, 0, 0, 0]; let answer: TcpAcceptResponse = crate::ipc::decode_payload(&wire).unwrap(); - assert_eq!((answer.remote_addr, answer.remote_port, answer.local_port), ([192, 0, 2, 7], 12345, 22)); + assert_eq!( + (answer.remote_addr, answer.remote_port, answer.local_port, answer.local_addr), + ([192, 0, 2, 7], 12345, 22, [198, 51, 100, 9]) + ); assert_eq!(answer.nodelay, 0); - wire[8..].copy_from_slice(&[0, 0, 0, 0x80]); + wire[12..].copy_from_slice(&[0, 0, 0, 0x80]); let answer: TcpAcceptResponse = crate::ipc::decode_payload(&wire).unwrap(); assert_ne!(answer.nodelay, 0); assert!(matches!( - crate::ipc::decode_payload::(&wire[..8]), + crate::ipc::decode_payload::(&wire[..12]), Err(IpcError::Malformed) )); } diff --git a/sdk/toyos/src/net/stream.rs b/sdk/toyos/src/net/stream.rs index ca6e068f75f..fd989eab56e 100644 --- a/sdk/toyos/src/net/stream.rs +++ b/sdk/toyos/src/net/stream.rs @@ -145,6 +145,7 @@ crate::ipc_payload! { pub holder: u16, pub peer_port: u16, pub peer_addr: [u8; 4], + pub local_addr: [u8; 4], pub local_port: u16, pub _pad: u16, } @@ -160,6 +161,9 @@ const HELD_REGION: Rights = Rights::DUP.union(Rights::MAP); /// table has. An estimate: no measurement sets it. pub const MAX_STREAMS: usize = 128; +/// An IPv4 address and a port: one end of a stream. +pub type EndAddr = ([u8; 4], u16); + type Rx = Receiving; type Tx = Sending; @@ -196,7 +200,7 @@ struct End { /// hangs nothing up. connection: BorrowedHandle<'static>, peer: ([u8; 4], u16), - local_port: u16, + local: ([u8; 4], u16), } /// An end's region, its handle and mapping, the bells netstack rings for it @@ -569,10 +573,17 @@ pub fn failure(handle: BorrowedHandle<'_>) -> Result, Refused> { }) } -/// The peer's address and port, and this end's port. -pub fn endpoints(handle: BorrowedHandle<'_>) -> Result<(([u8; 4], u16), u16), Refused> { +/// The peer's address and port, and this end's: the address netstack sent the +/// stream from. +pub fn endpoints(handle: BorrowedHandle<'_>) -> Result<(EndAddr, EndAddr), Refused> { let end = TABLE.lock().get(handle.raw()).ok_or(Refused::NotThisProcess(handle.raw()))?; - Ok((end.peer, end.local_port)) + Ok((end.peer, end.local)) +} + +/// A stream's own end as netstack names it, where it can be one: a unicast +/// address, neither unspecified, multicast nor broadcast, and a port. +fn local_end(addr: [u8; 4], port: u16) -> Option<([u8; 4], u16)> { + (addr != [0; 4] && addr[0] < 224 && port != 0).then_some((addr, port)) } /// The handle a watch of `way` is made on, for reading: the holder's receive @@ -723,7 +734,6 @@ pub fn let_go(handle: OwnedHandle) { pub struct Connected { /// Its connection, which reaches its end. pub handle: OwnedHandle, - pub local_port: u16, } /// An active open to `addr`:`port`, answered once the handshake ends or @@ -735,7 +745,7 @@ pub fn connect(addr: [u8; 4], port: u16, timeout_ms: u32, nodelay: bool) -> Resu let opened = open(nodelay, (addr, port), |region| { NetstackConn::connect()?.request_with_handles([region], MsgType::TcpConnect, &request)?.answer_with_handles::() })?; - Ok(Connected { handle: opened.handle, local_port: opened.answer.local_port }) + Ok(Connected { handle: opened.handle }) } /// A stream a listener's accept was handed. @@ -744,7 +754,6 @@ pub struct Accepted { pub handle: OwnedHandle, pub remote_addr: [u8; 4], pub remote_port: u16, - pub local_port: u16, /// Whether Nagle's algorithm is off for it, as its listener had it. pub nodelay: bool, } @@ -768,7 +777,6 @@ pub(super) fn accept(listener: BorrowedHandle<'_>, drain: super::socket::Drain<' handle: opened.handle, remote_addr: answer.remote_addr, remote_port: answer.remote_port, - local_port: answer.local_port, nodelay: answer.nodelay != 0, }) } @@ -781,14 +789,18 @@ struct Opened { /// What an answer says of the stream's endpoints. trait Answered { - fn local_port(&self) -> u16; + /// The message the answer is to, which a malformed one is refused as. + const ASKED: u32; + /// This end, as the answer spells it. + fn local(&self) -> ([u8; 4], u16); /// The peer, where the answer names it. fn peer(&self) -> Option<([u8; 4], u16)>; } impl Answered for TcpConnectResponse { - fn local_port(&self) -> u16 { - self.local_port + const ASKED: u32 = MsgType::TcpConnect as u32; + fn local(&self) -> ([u8; 4], u16) { + (self.local_addr, self.local_port) } fn peer(&self) -> Option<([u8; 4], u16)> { None @@ -796,8 +808,9 @@ impl Answered for TcpConnectResponse { } impl Answered for TcpAcceptResponse { - fn local_port(&self) -> u16 { - self.local_port + const ASKED: u32 = super::socket::ACCEPT; + fn local(&self) -> ([u8; 4], u16) { + (self.local_addr, self.local_port) } fn peer(&self) -> Option<([u8; 4], u16)> { Some((self.remote_addr, self.remote_port)) @@ -825,13 +838,15 @@ fn open( let (rx, tx) = session::client(words, LAYOUT); let lent = region.share().map_err(|_| NetError::Io)?; let (answer, bells, connection) = ask(lent)?; + let (addr, port) = answer.local(); + let local = local_end(addr, port).ok_or(NetError::Protocol(A::ASKED))?; let (region, base) = region.into_raw(); // SAFETY: `into_raw` gave this the region's handle, which nothing else answers for. let held = take_up(connection.into_owned(), bells, unsafe { OwnedHandle::from_raw(region) }, base)?; *HALVES[slot].rx.lock() = Kept::Held(rx); *HALVES[slot].tx.lock() = Kept::Held(tx); let peer = answer.peer().unwrap_or(peer); - let end = held.end(base as usize, peer, answer.local_port()); + let end = held.end(base as usize, peer, local); Ok((held.descriptor, end, answer)) })(); let mut table = TABLE.lock(); @@ -860,12 +875,12 @@ struct Held { impl Held { /// The end, its region mapped at `base`. - fn end(&self, base: usize, peer: ([u8; 4], u16), local_port: u16) -> End { + fn end(&self, base: usize, peer: ([u8; 4], u16), local: ([u8; 4], u16)) -> End { // SAFETY: the handles `take_up` narrowed for this end, which keeps them until its `free`. let (receive_bell, send_bell, rung, connection) = unsafe { (kept(self.receive_bell), kept(self.send_bell), kept(self.rung), kept(self.connection)) }; let region = Mapped { handle: self.region, base, receive_bell, send_bell, rung }; - End { region: Some(region), connection, peer, local_port } + End { region: Some(region), connection, peer, local } } } @@ -974,24 +989,24 @@ pub(super) fn ask_export(connection: BorrowedHandle<'_>, rung: Option) -> Option { let slot = TABLE.lock().reserve()?; - let taken = first.and_then(adoption).and_then(|(holder, adopt, [region, receive, send, rung])| { + let taken = first.and_then(adoption).and_then(|(holder, adopt, local, [region, receive, send, rung])| { let region = SharedMemory::lent(region, REGION_BYTES).ok()?; let (region, base) = region.into_raw(); // SAFETY: `into_raw` gave this the region's handle, which nothing else answers for. let region = unsafe { OwnedHandle::from_raw(region) }; - Some((holder, adopt, base, [receive, send, rung], region)) + Some((holder, adopt, local, base, [receive, send, rung], region)) }); let made = match taken { - Some((holder, adopt, base, bells, region)) => take_up(connection.into_owned(), bells, region, base).map(|held| { + Some((holder, adopt, local, base, bells, region)) => take_up(connection.into_owned(), bells, region, base).map(|held| { let (rx, tx) = session::holder(LAYOUT, holder); *HALVES[slot].rx.lock() = Kept::Held(rx); *HALVES[slot].tx.lock() = Kept::Held(tx); - let end = held.end(base as usize, (adopt.peer_addr, adopt.peer_port), adopt.local_port); + let end = held.end(base as usize, (adopt.peer_addr, adopt.peer_port), local); (held.descriptor, end) }), None => hold_connection(connection.into_owned()).map(|(descriptor, connection)| { // SAFETY: the end's own connection, which it keeps until its `free`. - (descriptor, End { region: None, connection: unsafe { kept(connection) }, peer: ([0; 4], 0), local_port: 0 }) + (descriptor, End { region: None, connection: unsafe { kept(connection) }, peer: ([0; 4], 0), local: ([0; 4], 0) }) }), }; let mut table = TABLE.lock(); @@ -1007,17 +1022,31 @@ pub(super) fn adopted(connection: Connection, first: Option<(&[u8], ipc::Batch)> } } -/// An [`ADOPT`] frame's holder, payload and handles. -fn adoption((payload, handles): (&[u8], ipc::Batch)) -> Option<(Holder, Adopt, [OwnedHandle; BELLS + 1])> { +/// An [`ADOPT`] frame's holder, payload, this end and handles. +fn adoption((payload, handles): (&[u8], ipc::Batch)) -> Option<(Holder, Adopt, EndAddr, [OwnedHandle; BELLS + 1])> { let adopt: Adopt = ipc::decode_payload(payload).ok()?; let handles = handles.exact::<{ BELLS + 1 }>()?; - Some((Holder::new(adopt.holder)?, adopt, handles)) + let local = local_end(adopt.local_addr, adopt.local_port)?; + Some((Holder::new(adopt.holder)?, adopt, local, handles)) } #[cfg(test)] mod tests { use super::*; + /// A stream's own end is a unicast address and a port: an answer naming + /// the unspecified address, a multicast or the broadcast one, or no port, + /// is no end a stream can have. + #[test] + fn a_streams_own_end_is_a_unicast_address_and_a_port() { + assert_eq!(local_end([192, 0, 2, 1], 49152), Some(([192, 0, 2, 1], 49152))); + assert_eq!(local_end([223, 255, 255, 254], 1), Some(([223, 255, 255, 254], 1))); + for addr in [[0; 4], [224, 0, 0, 1], [239, 255, 255, 250], [255; 4]] { + assert_eq!(local_end(addr, 49152), None, "{addr:?}"); + } + assert_eq!(local_end([192, 0, 2, 1], 0), None); + } + /// A wait for a gate that lasts is reported each stall interval and goes /// on until rung; one with a deadline ends at the deadline, reported only /// where it reached a stall interval first. diff --git a/system/netstack/node/src/lease/tcp.rs b/system/netstack/node/src/lease/tcp.rs index 099c6c513de..db193d5e819 100644 --- a/system/netstack/node/src/lease/tcp.rs +++ b/system/netstack/node/src/lease/tcp.rs @@ -29,14 +29,14 @@ impl Stack { held(self.shard.status(id)) } - pub(crate) fn tcp_local_port(&mut self, id: ConnId) -> Port { - held(self.shard.tuple(id)).local.port + pub(crate) fn tcp_local(&mut self, id: ConnId) -> Endpoint { + held(self.shard.tuple(id)).local } - /// The connection's peer and its local port. - pub(crate) fn tcp_endpoints(&mut self, id: ConnId) -> (Endpoint, Port) { + /// The connection's peer and its own end. + pub(crate) fn tcp_endpoints(&mut self, id: ConnId) -> (Endpoint, Endpoint) { let tuple = held(self.shard.tuple(id)); - (tuple.remote, tuple.local.port) + (tuple.remote, tuple.local) } /// [`toyos_net_tcp::Tcp::lend`]. diff --git a/system/netstack/node/src/listeners.rs b/system/netstack/node/src/listeners.rs index 1a4be4c7f78..e2026c7b369 100644 --- a/system/netstack/node/src/listeners.rs +++ b/system/netstack/node/src/listeners.rs @@ -62,7 +62,7 @@ pub enum ListenRefused { pub struct Accepted { pub id: StreamId, pub remote: Endpoint, - pub local: Port, + pub local: Endpoint, /// Whether Nagle's algorithm is off for the stream as it is handed over: what /// [`Node::nodelay`] answers for as long as the node holds the stream, which may be no /// longer than this accept's own pass. @@ -202,7 +202,7 @@ impl Node { let session = session().ok_or(AcceptRefused::NoRegion)?; let Some((conn, tuple, options)) = self.stack.tcp_accept(bound) else { unreachable!("[tcp] counted a connection it would not hand over") }; let id = self.streams.accepted(&mut self.stack, conn, options, session); - Ok(Accepted { id, remote: tuple.remote, local: tuple.local.port, nodelay: options.nodelay }) + Ok(Accepted { id, remote: tuple.remote, local: tuple.local, nodelay: options.nodelay }) } fn end_listener(&mut self, now: Instant, id: ListenerId) { diff --git a/system/netstack/node/src/streams.rs b/system/netstack/node/src/streams.rs index f7e3e88d4d2..57c316ef634 100644 --- a/system/netstack/node/src/streams.rs +++ b/system/netstack/node/src/streams.rs @@ -32,7 +32,7 @@ use core::time::Duration; use toyos::net::stream::{Server, Words, ADDRESS_GONE, LAYOUT, NODELAY, RECEIVE_BYTES, RESET, TIMED_OUT, UNREACHABLE}; use toyos_net_shard::ConnectError; use toyos_net_tcp::{ConnId, Endpoint, Error, Failure, Lent, Options, State, Status}; -use toyos_net_wire::{Instant, Port}; +use toyos_net_wire::Instant; use toyos::transport::session::{Holder, Rings}; use toyos::transport::Violation; @@ -74,8 +74,8 @@ pub struct StreamId(u64); #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum StreamEvent { - /// The answer to a connect: established, from this port. - Connected { id: StreamId, local: Port }, + /// The answer to a connect: established, from this address and port. + Connected { id: StreamId, local: Endpoint }, /// The answer to a connect: the peer refused it or [tcp] gave it up. Failed { id: StreamId, failure: Failure }, /// The answer to a connect: its deadline passed first. @@ -150,7 +150,7 @@ impl Stream { (_, Some(failure)) => StreamEvent::Failed { id, failure }, (State::SynSent | State::SynReceived, None) if self.deadline.is_some_and(|at| at <= now) => StreamEvent::TimedOut { id }, (State::SynSent | State::SynReceived, None) => return true, - (_, None) => StreamEvent::Connected { id, local: stack.tcp_local_port(conn) }, + (_, None) => StreamEvent::Connected { id, local: stack.tcp_local(conn) }, }; events.push_back(event); if !matches!(event, StreamEvent::Connected { .. }) { @@ -325,8 +325,8 @@ impl Node { } /// A holder more for established stream `id`, which takes a place: the stream's peer and - /// its local port, for the holder to be told. - pub fn export(&mut self, id: StreamId) -> Result<(Endpoint, Port), ExportRefused> { + /// its own end, for the holder to be told. + pub fn export(&mut self, id: StreamId) -> Result<(Endpoint, Endpoint), ExportRefused> { if self.room() == 0 { return Err(ExportRefused::Full); } diff --git a/system/netstack/node/tests/listeners.rs b/system/netstack/node/tests/listeners.rs index 672fd2d69e2..6d7a749606d 100644 --- a/system/netstack/node/tests/listeners.rs +++ b/system/netstack/node/tests/listeners.rs @@ -354,7 +354,7 @@ impl Net { /// `holder`'s accept that takes `peer`'s connection. fn accepts_as(&mut self, id: ListenerId, holder: HolderId, peer: Peer) -> (StreamId, Rc) { let (accepted, client) = self.accept_as(id, holder).expect("a connection and a place"); - assert_eq!((accepted.remote, accepted.local), (peer.endpoint(), Port::new(SSH).unwrap())); + assert_eq!((accepted.remote, accepted.local), (peer.endpoint(), Endpoint { addr: A, port: Port::new(SSH).unwrap() })); (accepted.id, client) } @@ -418,7 +418,7 @@ fn a_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_ends() { let (client, session) = Client::new(); let accepted = net.node.accept(net.now, id, HolderId::FIRST, || Some(session)).unwrap(); - assert_eq!((accepted.remote, accepted.local), (P1.endpoint(), Port::new(SSH).unwrap())); + assert_eq!((accepted.remote, accepted.local), (P1.endpoint(), Endpoint { addr: A, port: Port::new(SSH).unwrap() })); assert_eq!(inbox(&client), b"hello", "what arrived before the accept moves in it"); assert_eq!((net.node.streams(), net.node.listeners(), net.node.held()), (1, 1, 2)); assert_eq!(client.write(b"welcome"), Ok(7)); diff --git a/system/netstack/node/tests/streams.rs b/system/netstack/node/tests/streams.rs index cad6c41ade1..7497f3f3c56 100644 --- a/system/netstack/node/tests/streams.rs +++ b/system/netstack/node/tests/streams.rs @@ -417,7 +417,8 @@ fn established() -> (Net, StreamId, Rc) { fn established_within(timeout: Option) -> (Net, StreamId, Rc) { let mut net = Net::new(); let (id, client) = net.connect(timeout); - let local = Port::new(net.far.port).expect("the SYN's source port"); + // The address the stream left from is the node's lease, as the SYN carried it. + let local = Endpoint { addr: A, port: Port::new(net.far.port).expect("the SYN's source port") }; assert_eq!(net.events(), [StreamEvent::Connected { id, local }]); (net, id, client) } @@ -427,7 +428,7 @@ fn established_within(timeout: Option) -> (Net, StreamId, Rc) // RFC 9293 §3.5, figure 6: a SYN that acknowledges nothing, and after the peer's SYN-ACK an ACK // of its sequence number plus one from ours plus one. #[test] -fn a_connect_is_a_handshake_and_its_answer_names_the_port() { +fn a_connect_is_a_handshake_and_its_answer_names_the_address_and_port() { let (net, _, client) = established(); let [syn, ack] = &net.far.segments[..] else { panic!("a SYN and an ACK, not {:?}", net.far.segments) }; assert!(syn.syn && syn.ack.is_none() && syn.text.is_empty() && !syn.fin && !syn.rst, "{syn:?}"); @@ -1073,8 +1074,9 @@ fn a_holder_past_the_first_takes_a_place_and_none_is_made_past_the_places() { let (mut net, id, _client) = established(); net.node.set_places(2); assert_eq!(net.node.held(), 1); - let (remote, _) = net.node.export(id).expect("a place for the second holder"); + let (remote, local) = net.node.export(id).expect("a place for the second holder"); assert_eq!(remote, peer(), "the holder is told the stream's peer"); + assert_eq!(local.addr, A, "and the address the stream left from"); assert_eq!(net.node.held(), 2); assert_eq!(net.node.export(id), Err(ExportRefused::Full)); assert_eq!(net.node.held(), 2, "a refused export took nothing"); diff --git a/system/netstack/src/serve.rs b/system/netstack/src/serve.rs index d237a5119a6..16158805e5e 100644 --- a/system/netstack/src/serve.rs +++ b/system/netstack/src/serve.rs @@ -533,7 +533,7 @@ impl Sockets { let answer_to = self.by_stream.get(&id).and_then(|key| self.streams.get_mut(key)?.answer.take()); let Some((bells, holders)) = answer_to else { unreachable!("the node answered a connect nobody waits for") }; match answer { - Ok(local) => hand_over(&holders, bells, &TcpConnectResponse { local_port: local.get(), _pad: 0 }), + Ok(local) => hand_over(&holders, bells, &TcpConnectResponse { local_addr: local.addr.octets(), local_port: local.port.get(), _pad: 0 }), Err(code) => answer_refused(&holders, code), } } @@ -716,7 +716,8 @@ impl Sockets { holder: number.number(), peer_port: peer.port.get(), peer_addr: peer.addr.octets(), - local_port: local.get(), + local_addr: local.addr.octets(), + local_port: local.port.get(), _pad: 0, }; let handles = [region, receive_bell, send_bell, rung_waker]; @@ -833,7 +834,8 @@ impl Sockets { let answer = TcpAcceptResponse { remote_addr: accepted.remote.addr.octets(), remote_port: accepted.remote.port.get(), - local_port: accepted.local.get(), + local_port: accepted.local.port.get(), + local_addr: accepted.local.addr.octets(), nodelay: u32::from(accepted.nodelay), }; self.hold_stream(accepted.id, &holders, None); diff --git a/tests/guest/rust-tests/src/bin/libc_recv_beside_close.rs b/tests/guest/rust-tests/src/bin/libc_recv_beside_close.rs index a1f72929169..075d974a3a0 100644 --- a/tests/guest/rust-tests/src/bin/libc_recv_beside_close.rs +++ b/tests/guest/rust-tests/src/bin/libc_recv_beside_close.rs @@ -140,7 +140,7 @@ fn test(role: &str) { let (_send, send_bell) = wake::pair().expect("the send bell"); let (rung_waker, rung) = wake::pair().expect("the rung bell"); let bells = [OwnedHandle::from(receive_bell), send_bell.into(), rung_waker.into()]; - conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_port: 49_152, _pad: 0 }, bells) + conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_addr: [192, 0, 2, 1], local_port: 49_152, _pad: 0 }, bells) .expect("answer the connect with the bells"); assert_eq!(line(&mut out), "connected"); diff --git a/tests/guest/rust-tests/src/bin/link_failover.rs b/tests/guest/rust-tests/src/bin/link_failover.rs index da1b47d7165..a12f8dc37f1 100644 --- a/tests/guest/rust-tests/src/bin/link_failover.rs +++ b/tests/guest/rust-tests/src/bin/link_failover.rs @@ -8,6 +8,9 @@ //! names. It asks the harness, on a stream over the other link, to set the //! first link's NIC down; the next datagrams leave by the other link and none //! by the first. Asked to set it up again, the datagrams return to it. +//! +//! A stream to each network's host first reports that link's lease as its +//! local address: netstack's, since the two leases differ. use std::collections::BTreeMap; use std::io::{BufRead, BufReader, Write}; @@ -85,22 +88,36 @@ fn main() { let snap = until("both links' leases", |snap| { snap.get("net.links") == Some(&Value::U64(2)) && (0..2).all(|n| snap.get(&format!("net.link.{n}.lease.held")) == Some(&Value::Bool(true))) }); - // Each link's network, by its lease's address: QEMU's host is `.2` of it. - let network = |n: usize| { + // Each link's lease, and its network: QEMU's host is `.2` of it. + let lease = |n: usize| -> Ipv4Addr { let address = text(&snap, &format!("net.link.{n}.lease.address")).unwrap_or_else(|| panic!("link_failover: link {n} holds no lease")); - let address: Ipv4Addr = address.split('/').next().and_then(|a| a.parse().ok()).expect("link_failover: an address"); - let [a, b, c, _] = address.octets(); + address.split('/').next().and_then(|a| a.parse().ok()).expect("link_failover: an address") + }; + let leases = [lease(0), lease(1)]; + let network = |n: usize| { + let [a, b, c, _] = leases[n].octets(); Ipv4Addr::new(a, b, c, 0) }; let networks = [network(0), network(1)]; + assert_ne!(networks[0], networks[1], "link_failover: both links lease on one network"); let socket = UdpSocket::bind("0.0.0.0:0").expect("link_failover: a datagram socket"); let (first, second) = carried(round(&socket)); println!(" PASS: a round left by link {first}, on {}", networks[first]); - // The harness, on a stream over the second link: its network's own host. - let [a, b, c, _] = networks[second].octets(); - let mut control = TcpStream::connect(SocketAddr::from((Ipv4Addr::new(a, b, c, 2), port))).expect("link_failover: a stream to the harness over the second link"); + // The harness on each network's own host: a stream over each link, the + // first's let go and the second's the control. + let harness = |n: usize| { + let [a, b, c, _] = networks[n].octets(); + let stream = TcpStream::connect(SocketAddr::from((Ipv4Addr::new(a, b, c, 2), port))).expect("link_failover: a stream to the harness"); + let local = stream.local_addr().expect("link_failover: the stream's local address"); + assert_eq!(local.ip(), leases[n], "link_failover: the stream over link {n} says it left from {local}"); + assert_ne!(local.port(), 0, "link_failover: the stream's local port"); + println!(" PASS: the stream over link {n} left from its lease, {local}"); + stream + }; + drop(harness(first)); + let mut control = harness(second); control.set_read_timeout(Some(BOUND)).expect("link_failover: a read timeout"); let mut answers = BufReader::new(control.try_clone().expect("link_failover: the stream's reader")); let mut ask = |word: &str| { diff --git a/tests/guest/rust-tests/src/bin/stream_handles.rs b/tests/guest/rust-tests/src/bin/stream_handles.rs index fb12c593014..606822bab0d 100644 --- a/tests/guest/rust-tests/src/bin/stream_handles.rs +++ b/tests/guest/rust-tests/src/bin/stream_handles.rs @@ -451,7 +451,7 @@ impl Netstack { let theirs = self.exports.1.open("export").expect("a connection of the private port"); let ours = self.exports.0.accept().expect("the private port's connection"); let ([receive_bell, send_bell, rung_waker], (receive, send, rung)) = bells(); - let adopt = Adopt { holder: number.number(), peer_port: 80, peer_addr: [192, 0, 2, 2], local_port: 49_152, _pad: 0 }; + let adopt = Adopt { holder: number.number(), peer_port: 80, peer_addr: [192, 0, 2, 2], local_addr: [192, 0, 2, 1], local_port: 49_152, _pad: 0 }; let region = self.region.share().expect("a handle of the region"); let handles = [region, receive_bell, send_bell, rung_waker]; ours.send_with(ADOPT, &adopt, handles).expect("queue the holder's adoption"); @@ -514,7 +514,7 @@ fn run(role: &str, peer: &[u8]) -> (Child, Netstack) { region.bytes().write(RECEIVE_RING, peer); let _ = server.publish(words_of(®ion), peer.len() as u32).expect("the client keeps the protocol"); let (handed, (receive, send, rung)) = bells(); - conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_port: 49_152, _pad: 0 }, handed) + conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_addr: [192, 0, 2, 1], local_port: 49_152, _pad: 0 }, handed) .expect("answer the connect with the bells"); let (exports, export_connector) = port::create().expect("the private port"); let names = namespace::build().add("export", &export_connector).finish().expect("the private port's namespace"); diff --git a/tests/toyos.rs b/tests/toyos.rs index 5ab2ecebfe8..d6f8d5524cc 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -699,6 +699,10 @@ const MACHINE_TESTS: &[&str] = &[ // when it is up. Two drivers' links, a NIC's carrier and netstack's // routes cross three processes and two devices, which the node's and the // shard's host tests of rank, failover and return hold below the drivers. + // std's `local_addr` of a stream over each link is that link's lease, and + // the two differ, so one is not QEMU's 10.0.2.15: what netstack answered + // through the SDK, which the node's host tests hold only as far as the + // node. // QEMU x86-64 (`qemu::SUITE_ARCH`), on two user networks; the datagrams go // to RFC 5737's TEST-NET-1, which answers nothing, and the harness is the // peer on the host's loopback, so it depends on no public network. The networks are not @@ -5940,6 +5944,8 @@ fn traffic_leaves_by_the_second_link_while_the_first_is_down() -> Result<(), Str let port = peer.local_addr().map_err(|e| format!("the listener's port: {e}"))?.port(); let socket = qemu.qmp_socket().to_path_buf(); let heard = std::thread::spawn(move || -> Result<(), String> { + // The job's stream over its first link only reports its local address. + drop(peer.accept().map_err(|e| format!("the guest's first connect: {e}"))?); let (conn, _) = peer.accept().map_err(|e| format!("the guest's connect: {e}"))?; conn.set_read_timeout(Some(CEILING)).map_err(|e| e.to_string())?; let mut reader = BufReader::new(conn.try_clone().map_err(|e| e.to_string())?); From 997843da0bb242287d928a110fefe73ac18f5734 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 23:25:59 +0100 Subject: [PATCH 2/3] review r1 of #1147: a connected datagram socket's local address is netstack's too A connected datagram socket answered its bind's address, 0.0.0.0 for a wildcard, where Linux answers the address its connect's route sends from. [udp] already fixes that address at connect (Binding::Connected's local). The shard's udp_peer now gives it as UdpPeer, and netstack's PEER answer (socket::Peer) carries it beside the peer. The SDK's socket::local asks PEER for a datagram socket, so every holder sees the address whichever holder connected; an answer naming no host address is refused. std's UdpSocket::local_addr and libc's getsockname return it. The PEER answer carries it rather than CONNECT's: a socket's connection is every holder's, and only the connecting holder reads a CONNECT answer. - The stream end's check and the datagram one share net::unicast, which says what its doc says through Ipv4Addr's own predicates. - stream::Endpoint names an address and port throughout the stream end. - stream_handles answers a connect as from 192.0.2.1 and asserts std's local_addr reads it, and answers one with the unspecified address and asserts the connect is refused, so the check at open is held. - link_failover also connects a datagram socket over each link and asserts its local address is that link's lease. The stream's port-0 assert is gone: the SDK refuses port 0 before an end exists. - The ramp issue's account of std is in the past tense, and a node test comment that claimed what nothing checks is gone. Co-Authored-By: Claude Opus 5.5 --- ...w-slower-than-linux-in-its-first-300-ms.md | 4 +- sdk/toyos/src/net.rs | 7 +++ sdk/toyos/src/net/socket.rs | 46 ++++++++++++++++--- sdk/toyos/src/net/stream.rs | 32 ++++++------- system/netstack/node/src/datagram.rs | 6 +-- system/netstack/node/src/lease.rs | 4 +- system/netstack/node/src/lib.rs | 2 +- system/netstack/node/tests/datagram.rs | 5 +- system/netstack/node/tests/streams.rs | 1 - system/netstack/shard/src/lib.rs | 15 ++++-- system/netstack/src/serve.rs | 9 +++- .../guest/rust-tests/src/bin/link_failover.rs | 14 ++++-- .../rust-tests/src/bin/stream_handles.rs | 41 ++++++++++++++++- tests/toyos.rs | 9 ++-- 14 files changed, 148 insertions(+), 47 deletions(-) diff --git a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md index 3e1f96e5561..68e4746ec79 100644 --- a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md +++ b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md @@ -352,8 +352,8 @@ run and the run's own spread. **The exit's first session read no ToyOS row, 2026-11-01** ([evidence](https://github.com/ToyOSOrg/evidence/tree/35f09268e09714a8547496bdd766bcd3ca998325/r1)). This was Linux, ToyOS at `4f52c3056`, Linux, all exiting 0. ToyOS's `osbench_lan` found the peer and then refused the group: -`no netstack link holds 10.0.2.15`. std's `TcpStream::local_addr` answers -QEMU's user-network address whatever the lease is, and since #1122 osbench +`no netstack link holds 10.0.2.15`. std's `TcpStream::local_addr` answered +QEMU's user-network address whatever the lease was, and since #1122 osbench names a connection's link by that address. The T14's lease is not it. So the session does not decide. The next one runs on a head where netstack's answers carry each stream's local address and std, libc and the SDK return it. diff --git a/sdk/toyos/src/net.rs b/sdk/toyos/src/net.rs index 85f0b6869d0..c49ab8e9e9a 100644 --- a/sdk/toyos/src/net.rs +++ b/sdk/toyos/src/net.rs @@ -155,6 +155,13 @@ pub enum NetError { Io, } +/// Whether `addr` can be a host's own: neither unspecified, multicast nor broadcast, the +/// check an answer naming this machine's end of a stream or a connected socket passes. +fn unicast(addr: [u8; 4]) -> bool { + let addr = core::net::Ipv4Addr::from(addr); + !addr.is_unspecified() && !addr.is_multicast() && !addr.is_broadcast() +} + impl NetError { pub fn from_error_code(code: u32) -> Self { match code { diff --git a/sdk/toyos/src/net/socket.rs b/sdk/toyos/src/net/socket.rs index eae1046b436..065028e54d4 100644 --- a/sdk/toyos/src/net/socket.rs +++ b/sdk/toyos/src/net/socket.rs @@ -92,6 +92,16 @@ crate::ipc_payload! { pub value: u32, } + /// [`PEER`]'s answer: the peer a connect named, and the address the socket sends there from, + /// which the connect fixed. + #[derive(Debug, PartialEq, Eq)] + pub struct Peer { + pub addr: [u8; 4], + pub port: u16, + pub _pad: u16, + pub local_addr: [u8; 4], + } + /// An exported end's [`ADOPT`]: what kind of socket it holds, and the /// address and port it is at. pub struct Adopt { @@ -371,10 +381,16 @@ pub fn is_socket(handle: BorrowedHandle<'_>) -> bool { kind(handle).is_some() } -/// The address the socket `handle` reaches is at, as its listen or bind named -/// it, and the port it holds. +/// The address the socket `handle` reaches is at and the port it holds: a +/// connected datagram socket's is the one its connect fixed, as netstack answers +/// it, and any other's is the one its listen or bind named. pub fn local(handle: BorrowedHandle<'_>) -> Result<([u8; 4], u16), NetError> { - TABLE.lock().get(handle.raw()).map(|end| (end.addr, end.port)).ok_or(NetError::NotASocket) + let end = TABLE.lock().get(handle.raw()).ok_or(NetError::NotASocket)?; + let connected = match end.kind { + Kind::Datagram => connected(handle)?.map(|peer| peer.local_addr), + Kind::Listener => None, + }; + Ok((connected.unwrap_or(end.addr), end.port)) } /// `handle`, which reaches one of this process's sockets, as a descriptor's object: let go by @@ -459,15 +475,21 @@ pub fn connect(handle: BorrowedHandle<'_>, addr: [u8; 4], port: u16) -> Result<( /// The peer the socket `handle` reaches is connected to, by any of its holders. pub fn peer(handle: BorrowedHandle<'_>) -> Result, NetError> { + Ok(connected(handle)?.map(|peer| (peer.addr, peer.port))) +} + +/// netstack's [`PEER`] answer for the datagram socket `handle` reaches, `None` while no +/// holder connected it. One naming a local address no socket sends from is refused. +fn connected(handle: BorrowedHandle<'_>) -> Result, NetError> { call_on(handle, Kind::Datagram, |slot, end| { let _turn = TURNS[slot].lock(); ipc::signal(end.connection, PEER).map_err(hangup)?; let mut rx = FrameRx::::new(); match answer(&end.connection, &mut rx) { - Ok(frame) => { - let peer: Datagram = ipc::decode_payload(rx.payload(frame.payload_len)).map_err(|_| NetError::Protocol(PEER))?; - Ok(Some((peer.addr, peer.port))) - } + Ok(frame) => match ipc::decode_payload::(rx.payload(frame.payload_len)) { + Ok(peer) if super::unicast(peer.local_addr) => Ok(Some(peer)), + _ => Err(NetError::Protocol(PEER)), + }, Err(NetError::NotConnected) => Ok(None), Err(e) => Err(e), } @@ -579,6 +601,16 @@ pub(super) fn adopted(connection: Connection, payload: &[u8], handles: ipc::Batc mod tests { use super::*; + /// [`PEER`]'s answer as a holder decodes it: the peer, its port, then the address the + /// socket sends from; an answer without that address is refused. + #[test] + fn a_peers_answer_ends_in_the_address_it_is_sent_from() { + let wire = [192, 0, 2, 7, 53, 0, 0, 0, 198, 51, 100, 9]; + let peer: Peer = ipc::decode_payload(&wire).unwrap(); + assert_eq!(peer, Peer { addr: [192, 0, 2, 7], port: 53, _pad: 0, local_addr: [198, 51, 100, 9] }); + assert!(ipc::decode_payload::(&wire[..8]).is_err()); + } + /// A datagram is one frame's payload: its far end, then exactly its bytes, read back as it /// was written, whatever bytes it carries, an empty one too. #[test] diff --git a/sdk/toyos/src/net/stream.rs b/sdk/toyos/src/net/stream.rs index fd989eab56e..7a8880aa05d 100644 --- a/sdk/toyos/src/net/stream.rs +++ b/sdk/toyos/src/net/stream.rs @@ -162,7 +162,7 @@ const HELD_REGION: Rights = Rights::DUP.union(Rights::MAP); pub const MAX_STREAMS: usize = 128; /// An IPv4 address and a port: one end of a stream. -pub type EndAddr = ([u8; 4], u16); +pub type Endpoint = ([u8; 4], u16); type Rx = Receiving; type Tx = Sending; @@ -199,8 +199,8 @@ struct End { /// waits through and only [`free`] closes: a descriptor closed under a call /// hangs nothing up. connection: BorrowedHandle<'static>, - peer: ([u8; 4], u16), - local: ([u8; 4], u16), + peer: Endpoint, + local: Endpoint, } /// An end's region, its handle and mapping, the bells netstack rings for it @@ -575,15 +575,15 @@ pub fn failure(handle: BorrowedHandle<'_>) -> Result, Refused> { /// The peer's address and port, and this end's: the address netstack sent the /// stream from. -pub fn endpoints(handle: BorrowedHandle<'_>) -> Result<(EndAddr, EndAddr), Refused> { +pub fn endpoints(handle: BorrowedHandle<'_>) -> Result<(Endpoint, Endpoint), Refused> { let end = TABLE.lock().get(handle.raw()).ok_or(Refused::NotThisProcess(handle.raw()))?; Ok((end.peer, end.local)) } /// A stream's own end as netstack names it, where it can be one: a unicast -/// address, neither unspecified, multicast nor broadcast, and a port. -fn local_end(addr: [u8; 4], port: u16) -> Option<([u8; 4], u16)> { - (addr != [0; 4] && addr[0] < 224 && port != 0).then_some((addr, port)) +/// address ([`super::unicast`]) and a port. +fn local_end(addr: [u8; 4], port: u16) -> Option { + (super::unicast(addr) && port != 0).then_some((addr, port)) } /// The handle a watch of `way` is made on, for reading: the holder's receive @@ -792,27 +792,27 @@ trait Answered { /// The message the answer is to, which a malformed one is refused as. const ASKED: u32; /// This end, as the answer spells it. - fn local(&self) -> ([u8; 4], u16); + fn local(&self) -> Endpoint; /// The peer, where the answer names it. - fn peer(&self) -> Option<([u8; 4], u16)>; + fn peer(&self) -> Option; } impl Answered for TcpConnectResponse { const ASKED: u32 = MsgType::TcpConnect as u32; - fn local(&self) -> ([u8; 4], u16) { + fn local(&self) -> Endpoint { (self.local_addr, self.local_port) } - fn peer(&self) -> Option<([u8; 4], u16)> { + fn peer(&self) -> Option { None } } impl Answered for TcpAcceptResponse { const ASKED: u32 = super::socket::ACCEPT; - fn local(&self) -> ([u8; 4], u16) { + fn local(&self) -> Endpoint { (self.local_addr, self.local_port) } - fn peer(&self) -> Option<([u8; 4], u16)> { + fn peer(&self) -> Option { Some((self.remote_addr, self.remote_port)) } } @@ -822,7 +822,7 @@ impl Answered for TcpAcceptResponse { /// stream's first holder. Nothing made outlives a refusal. fn open( nodelay: bool, - peer: ([u8; 4], u16), + peer: Endpoint, ask: impl FnOnce(OwnedHandle) -> Result<(A, [OwnedHandle; BELLS], Connection), NetError>, ) -> Result, NetError> { let slot = TABLE.lock().reserve().ok_or(NetError::ResourceExhausted)?; @@ -875,7 +875,7 @@ struct Held { impl Held { /// The end, its region mapped at `base`. - fn end(&self, base: usize, peer: ([u8; 4], u16), local: ([u8; 4], u16)) -> End { + fn end(&self, base: usize, peer: Endpoint, local: Endpoint) -> End { // SAFETY: the handles `take_up` narrowed for this end, which keeps them until its `free`. let (receive_bell, send_bell, rung, connection) = unsafe { (kept(self.receive_bell), kept(self.send_bell), kept(self.rung), kept(self.connection)) }; @@ -1023,7 +1023,7 @@ pub(super) fn adopted(connection: Connection, first: Option<(&[u8], ipc::Batch)> } /// An [`ADOPT`] frame's holder, payload, this end and handles. -fn adoption((payload, handles): (&[u8], ipc::Batch)) -> Option<(Holder, Adopt, EndAddr, [OwnedHandle; BELLS + 1])> { +fn adoption((payload, handles): (&[u8], ipc::Batch)) -> Option<(Holder, Adopt, Endpoint, [OwnedHandle; BELLS + 1])> { let adopt: Adopt = ipc::decode_payload(payload).ok()?; let handles = handles.exact::<{ BELLS + 1 }>()?; let local = local_end(adopt.local_addr, adopt.local_port)?; diff --git a/system/netstack/node/src/datagram.rs b/system/netstack/node/src/datagram.rs index e6987a531d2..5758030d4a1 100644 --- a/system/netstack/node/src/datagram.rs +++ b/system/netstack/node/src/datagram.rs @@ -43,7 +43,7 @@ use toyos_net_wire::{Instant, Port}; use crate::holders::{Bell, HolderId, Holders}; use crate::streams::ExportRefused; -use crate::Node; +use crate::{Node, UdpPeer}; /// One client socket. An id outlives its socket only as a refusal: [udp] never hands a closed /// socket's id to another. @@ -179,8 +179,8 @@ impl Node { connected } - /// The peer [`Self::udp_connect`] named, if any. - pub fn udp_peer(&mut self, id: DatagramId) -> Result, Refused> { + /// The peer [`Self::udp_connect`] named, if any, and the address the socket sends there from. + pub fn udp_peer(&mut self, id: DatagramId) -> Result, Refused> { self.stack.peer(id.0).map_err(refused) } diff --git a/system/netstack/node/src/lease.rs b/system/netstack/node/src/lease.rs index 64d3420028b..7636db9bb48 100644 --- a/system/netstack/node/src/lease.rs +++ b/system/netstack/node/src/lease.rs @@ -26,7 +26,7 @@ use core::net::Ipv4Addr; use toyos_dhcp::{Lease, Transmission}; use toyos_dns::Cache; -use toyos_net_shard::{Config, Credit, Event, IfIndex, Origin, Rank, Refusal, Shard}; +use toyos_net_shard::{Config, Credit, Event, IfIndex, Origin, Rank, Refusal, Shard, UdpPeer}; use toyos_net_udp::{Received, SocketId}; use toyos_net_wire::ethernet::{IndividualMac, MacAddr}; use toyos_net_wire::ipv4::{MulticastAddr, Ttl}; @@ -264,7 +264,7 @@ impl Stack { self.shard.udp_readable(id) } - pub(crate) fn peer(&mut self, id: SocketId) -> Result, toyos_net_udp::Error> { + pub(crate) fn peer(&mut self, id: SocketId) -> Result, toyos_net_udp::Error> { self.shard.udp_peer(id) } diff --git a/system/netstack/node/src/lib.rs b/system/netstack/node/src/lib.rs index 7ca9ccc85fa..95a73008b9e 100644 --- a/system/netstack/node/src/lib.rs +++ b/system/netstack/node/src/lib.rs @@ -55,7 +55,7 @@ use alloc::vec; use alloc::vec::Vec; use toyos_dhcp::{AddressRequest, Client, HostName, Lease, Output}; -pub use toyos_net_shard::{Credit, IfIndex, Rank}; +pub use toyos_net_shard::{Credit, IfIndex, Rank, UdpPeer}; use toyos_net_shard::{Config, Shard}; use toyos_net_wire::ethernet::IndividualMac; use toyos_net_wire::Instant; diff --git a/system/netstack/node/tests/datagram.rs b/system/netstack/node/tests/datagram.rs index ae37485013b..8cc10c342ff 100644 --- a/system/netstack/node/tests/datagram.rs +++ b/system/netstack/node/tests/datagram.rs @@ -13,7 +13,7 @@ use std::time::Duration; use common::{from_server, message_of, terms, xid, A, ACK, BROADCAST, DNS, MAC, MAC_B, MAC_R, R}; use lan::{udp, unreachable, Lan, Seen, Udp, B, OFF_LINK}; -use toyos_net_node::{Datagram, Event, ExportRefused, HolderId, Refused}; +use toyos_net_node::{Datagram, Event, ExportRefused, HolderId, Refused, UdpPeer}; use toyos_net_shard::Refusal; use toyos_net_udp::Counter; use toyos_net_wire::Port; @@ -276,7 +276,8 @@ fn a_connected_sockets_peer_is_every_holders() { assert_eq!((lan.node.udp_peer(id), lan.node.udp_send(lan.now, id, b"x")), (Ok(None), Err(Refused::NotConnected))); assert_eq!(lan.node.udp_connect(lan.now, id, B, 0), Err(Refused::InvalidInput), "port 0"); lan.node.udp_connect(lan.now, id, B, 7).expect("one host of the link"); - assert_eq!(lan.node.udp_peer(id), Ok(Some((B, Port::new(7).unwrap())))); + // A is the lease, which the route to B leaves from. + assert_eq!(lan.node.udp_peer(id), Ok(Some(UdpPeer { addr: B, port: Port::new(7).unwrap(), local: A }))); lan.node.udp_send(lan.now, id, b"to the peer").expect("a connected socket's send"); lan.pump(); let last = lan.datagrams().last().map(|(_, udp)| (udp.destination, udp.port, udp.payload.clone())); diff --git a/system/netstack/node/tests/streams.rs b/system/netstack/node/tests/streams.rs index 7497f3f3c56..ad11de887b8 100644 --- a/system/netstack/node/tests/streams.rs +++ b/system/netstack/node/tests/streams.rs @@ -417,7 +417,6 @@ fn established() -> (Net, StreamId, Rc) { fn established_within(timeout: Option) -> (Net, StreamId, Rc) { let mut net = Net::new(); let (id, client) = net.connect(timeout); - // The address the stream left from is the node's lease, as the SYN carried it. let local = Endpoint { addr: A, port: Port::new(net.far.port).expect("the SYN's source port") }; assert_eq!(net.events(), [StreamEvent::Connected { id, local }]); (net, id, client) diff --git a/system/netstack/shard/src/lib.rs b/system/netstack/shard/src/lib.rs index 4b4bf9be3fa..61e608ad21f 100644 --- a/system/netstack/shard/src/lib.rs +++ b/system/netstack/shard/src/lib.rs @@ -63,6 +63,14 @@ use toyos_net_wire::{Instant, Port}; pub use toyos_net_ip::{Credit, IfIndex, Rank}; pub use toyos_net_udp::Origin; +/// A connected datagram socket's peer, and the address it sends there from. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct UdpPeer { + pub addr: Ipv4Addr, + pub port: Port, + pub local: Ipv4Addr, +} + /// Independent 128-bit keys from the system's randomness: one leaked value predicts no other. #[derive(Clone, Debug)] pub struct Secrets { @@ -674,10 +682,11 @@ impl Shard { self.udp.readable(id) } - /// The peer a connected socket's datagrams go to and come from. - pub fn udp_peer(&mut self, id: SocketId) -> Result, toyos_net_udp::Error> { + /// The peer a connected socket's datagrams go to and come from, and the address it sends + /// from, which its connect fixed. + pub fn udp_peer(&mut self, id: SocketId) -> Result, toyos_net_udp::Error> { self.udp.binding(id).map(|(binding, _)| match binding { - toyos_net_udp::Binding::Connected { peer, peer_port, .. } => Some((peer, peer_port)), + toyos_net_udp::Binding::Connected { local, peer, peer_port } => Some(UdpPeer { addr: peer, port: peer_port, local }), toyos_net_udp::Binding::Any | toyos_net_udp::Binding::Specific(_) => None, }) } diff --git a/system/netstack/src/serve.rs b/system/netstack/src/serve.rs index 16158805e5e..46a395b4c01 100644 --- a/system/netstack/src/serve.rs +++ b/system/netstack/src/serve.rs @@ -1019,7 +1019,7 @@ fn asked(kind: Kind, msg_type: u32, payload: &[u8], handles: usize) -> Option), } @@ -1042,7 +1042,12 @@ fn answer_socket(node: &mut Node, now: Instant, socket: Held, holder: HolderId, node.udp_connect(now, id, Ipv4Addr::from(to), port).map(|()| Reply::Bare).map_err(refused) } (Held::Datagram(id), Asked::Peer) => match node.udp_peer(id).map_err(refused)? { - Some((addr, port)) => Ok(Reply::Peer(socket::Datagram { addr: addr.octets(), port: port.get(), _pad: 0 })), + Some(peer) => Ok(Reply::Peer(socket::Peer { + addr: peer.addr.octets(), + port: peer.port.get(), + _pad: 0, + local_addr: peer.local.octets(), + })), None => Err(ERR_NOT_CONNECTED), }, (Held::Datagram(id), Asked::SendPeer(data)) => node.udp_send(now, id, &data).map(|()| Reply::Bare).map_err(refused), diff --git a/tests/guest/rust-tests/src/bin/link_failover.rs b/tests/guest/rust-tests/src/bin/link_failover.rs index a12f8dc37f1..288660e28b9 100644 --- a/tests/guest/rust-tests/src/bin/link_failover.rs +++ b/tests/guest/rust-tests/src/bin/link_failover.rs @@ -9,8 +9,9 @@ //! first link's NIC down; the next datagrams leave by the other link and none //! by the first. Asked to set it up again, the datagrams return to it. //! -//! A stream to each network's host first reports that link's lease as its -//! local address: netstack's, since the two leases differ. +//! A stream and a connected datagram socket to each network's host first +//! report that link's lease as their local address: netstack's, since the two +//! leases differ. use std::collections::BTreeMap; use std::io::{BufRead, BufReader, Write}; @@ -112,10 +113,17 @@ fn main() { let stream = TcpStream::connect(SocketAddr::from((Ipv4Addr::new(a, b, c, 2), port))).expect("link_failover: a stream to the harness"); let local = stream.local_addr().expect("link_failover: the stream's local address"); assert_eq!(local.ip(), leases[n], "link_failover: the stream over link {n} says it left from {local}"); - assert_ne!(local.port(), 0, "link_failover: the stream's local port"); println!(" PASS: the stream over link {n} left from its lease, {local}"); stream }; + for (n, network) in networks.iter().enumerate() { + let [a, b, c, _] = network.octets(); + let connected = UdpSocket::bind("0.0.0.0:0").expect("link_failover: a datagram socket to connect"); + connected.connect(SocketAddr::from((Ipv4Addr::new(a, b, c, 2), port))).expect("link_failover: a datagram socket's connect"); + let local = connected.local_addr().expect("link_failover: the datagram socket's local address"); + assert_eq!(local.ip(), leases[n], "link_failover: the datagram socket connected over link {n} says it sends from {local}"); + println!(" PASS: the datagram socket connected over link {n} sends from its lease, {local}"); + } drop(harness(first)); let mut control = harness(second); control.set_read_timeout(Some(BOUND)).expect("link_failover: a read timeout"); diff --git a/tests/guest/rust-tests/src/bin/stream_handles.rs b/tests/guest/rust-tests/src/bin/stream_handles.rs index 606822bab0d..a1b2e46754b 100644 --- a/tests/guest/rust-tests/src/bin/stream_handles.rs +++ b/tests/guest/rust-tests/src/bin/stream_handles.rs @@ -25,6 +25,9 @@ //! two read its stdout and stderr, and another waits for its exit; a child //! that holds its pipes open is killed from a thread other than the one //! reading them, and that read ends. `sshserver` runs every program so. +//! - **A stream's local address is netstack's answer**: std's `local_addr` +//! reads the address and port the connect's answer named, and a connect +//! whose answer names the unspecified address is refused. //! //! **This test is netstack for its children**: a port of its own served to //! each as `netstack`, which answers a connect with the region judged and its @@ -93,6 +96,8 @@ fn main() { Some("two-threads") => two_threads(), Some("echo") => echo(), Some("hold") => hold(), + Some("local") => local(), + Some("unaddressed") => unaddressed(), Some("child") => std::process::exit(0), Some(other) => panic!("unknown role {other:?}"), None => test(), @@ -108,6 +113,7 @@ fn test() { refuses_a_stream_as_a_file(); reads_and_writes_on_two_threads(); pipes_across_threads(); + takes_its_local_address_from_the_answer(); println!("a stream's holders: one dead inside the gate leaves the other working; netstack's exit ends both"); } @@ -243,6 +249,18 @@ fn refuses_a_stream_as_a_file() { println!(" a stream as a file: refused by name"); } +/// std's `local_addr` is what the connect's answer named, and an answer naming +/// the unspecified address refuses the connect. +fn takes_its_local_address_from_the_answer() { + let (child, _netstack) = run("local", &[]); + let said = child.wait_with_output().expect("wait the child"); + assert_eq!(String::from_utf8_lossy(&said.stdout).trim(), "192.0.2.1:49152", "the stream's local address"); + let (child, _netstack) = run_answering("unaddressed", &[], [0; 4]); + let said = child.wait_with_output().expect("wait the child"); + assert_eq!(String::from_utf8_lossy(&said.stdout).trim(), "refused", "a connect answered with no local address"); + println!(" a stream's local address: the answer's, and none refused"); +} + // ---- this test as netstack ---- /// What the client writes while its other thread reads. @@ -490,6 +508,11 @@ impl Netstack { /// Spawns `role` with this test as its netstack, and serves its connect, the /// stream's receive ring holding `peer`. fn run(role: &str, peer: &[u8]) -> (Child, Netstack) { + run_answering(role, peer, [192, 0, 2, 1]) +} + +/// [`run`], the connect answered as left from `local`, port 49152. +fn run_answering(role: &str, peer: &[u8], local: [u8; 4]) -> (Child, Netstack) { let (acceptor, connector) = port::create().expect("a port"); // This test's own names, so a child reaches the file server its own // spawns read images through, and this test as its `netstack`. @@ -514,7 +537,7 @@ fn run(role: &str, peer: &[u8]) -> (Child, Netstack) { region.bytes().write(RECEIVE_RING, peer); let _ = server.publish(words_of(®ion), peer.len() as u32).expect("the client keeps the protocol"); let (handed, (receive, send, rung)) = bells(); - conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_addr: [192, 0, 2, 1], local_port: 49_152, _pad: 0 }, handed) + conn.send_with(RespType::Result as u32, &TcpConnectResponse { local_addr: local, local_port: 49_152, _pad: 0 }, handed) .expect("answer the connect with the bells"); let (exports, export_connector) = port::create().expect("the private port"); let names = namespace::build().add("export", &export_connector).finish().expect("the private port's namespace"); @@ -703,6 +726,22 @@ fn foreign() -> ! { std::process::exit(0) } +/// A client that says its stream's local address. +fn local() -> ! { + let stream = connect(); + say(&stream.local_addr().expect("the stream's local address").to_string()); + std::process::exit(0) +} + +/// A client whose connect's answer names no local address. +fn unaddressed() -> ! { + match TcpStream::connect(ADDR.parse::().expect("an address")) { + Ok(stream) => say(&format!("connected from {:?}", stream.local_addr())), + Err(_) => say("refused"), + } + std::process::exit(0) +} + /// A client that takes its stream's handle as a file. fn as_file() -> ! { let stream = connect(); diff --git a/tests/toyos.rs b/tests/toyos.rs index d6f8d5524cc..01dfbf7bef3 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -699,10 +699,10 @@ const MACHINE_TESTS: &[&str] = &[ // when it is up. Two drivers' links, a NIC's carrier and netstack's // routes cross three processes and two devices, which the node's and the // shard's host tests of rank, failover and return hold below the drivers. - // std's `local_addr` of a stream over each link is that link's lease, and - // the two differ, so one is not QEMU's 10.0.2.15: what netstack answered - // through the SDK, which the node's host tests hold only as far as the - // node. + // std's `local_addr` of a stream and of a connected datagram socket over + // each link is that link's lease, and the two differ, so one is not QEMU's + // 10.0.2.15: what netstack answered through the SDK, which the node's host + // tests hold only as far as the node. // QEMU x86-64 (`qemu::SUITE_ARCH`), on two user networks; the datagrams go // to RFC 5737's TEST-NET-1, which answers nothing, and the harness is the // peer on the host's loopback, so it depends on no public network. The networks are not @@ -6096,6 +6096,7 @@ fn stream_holders(test_config: &Path) -> Result<(), String> { "given out and taken back: one end; a duplicate of its handle endowed or sent: refused", "one end, two threads: the write completes beside an unanswered read, which then takes the peer's next bytes", "piped children across threads: fed, read and waited for apart, and one killed beside its reader", + "a stream's local address: the answer's, and none refused", ] { if !result.stdout.contains(claim) { return Err(format!("the job never said {claim:?}:\n{}", result.stdout)); From a7c1b31f93b90dfbd57406e0c0c15998a0b71480 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 23:48:23 +0100 Subject: [PATCH 3/3] issues: the T14 upload's first 300 ms read at Linux's; the ramp issue closes The session the issue's exit names ran on 9070b6111 (this PR's local address fix, which let osbench_lan measure on the T14's lease): Linux, ToyOS, Linux, all rc 0. judge.py applied the rule set before the run: - p50: ToyOS 114 MB/s, Linux 113 and 113. - Nothing resent: no RTO, recovery or loss probe. - 100 ms: ToyOS's five rounds 6.31-9.93 MB (mean 7.11) against Linux's ten 6.24-6.68 MB; the one-sided rank test that ToyOS ranks low gives p = 0.22. - 300 ms: 29.3-33.2 MB against 28.3-30.0 MB; p = 0.78. - Linux's two runs agree: p = 0.55 and 1.00, so the rounds compare. Verdict: parity. ToyOS's first round read 9.93 MB by 100 ms, far above every other round on either system; its cause is not known. Without it the 100 ms test gives p = 0.053, still a pass, so the verdict does not rest on it. Evidence: https://github.com/ToyOSOrg/evidence/tree/1c9f4c83c362a1b30746f1e096ad412a2d5f25f9/r2/session Co-Authored-By: Claude Opus 5.5 --- ...ompares-toyos-with-linux-on-one-machine.md | 5 +- ...w-slower-than-linux-in-its-first-300-ms.md | 383 ------------------ 2 files changed, 3 insertions(+), 385 deletions(-) delete mode 100644 issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md diff --git a/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md b/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md index b710a7fd1c1..3b1d8bae098 100644 --- a/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md +++ b/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md @@ -49,8 +49,9 @@ test depends on the public internet. The first paired session (`ToyOSOrg/evidence` branch `pr-1065`, `r3/`) put ToyOS's T14 LAN download and 64 B round trip at Linux's and its upload at -a third of it (`issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md`, -where what remains of that is now recorded). +a third of it. That gap was closed by #1069, #1102, #1112 and #1114, and +the T14 parity session of #1147 read the upload and its first 300 ms at +Linux's ([evidence](https://github.com/ToyOSOrg/evidence/tree/1c9f4c83c362a1b30746f1e096ad412a2d5f25f9/r2/session)). The cryptographic rows are not osbench's. AES-GCM, ChaCha20-Poly1305, SHA-256 and the rest were measured under Linux on both benches against ring, diff --git a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md b/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md deleted file mode 100644 index 68e4746ec79..00000000000 --- a/issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md +++ /dev/null @@ -1,383 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-24 ---- - -# [tcp] opens a connection's window slower than Linux in its first 300 ms - -On the T14's wired path to the AMD bench's Wi-Fi peer, a paced ToyOS TCP -sender (#1114) delivers 6.07 MB in a new connection's first 100 ms and -29.2 MB by 300 ms. In the same session the bench's Linux delivered -6.10–6.38 MB and 28.6–29.6 MB. The upload row reads 114 MB/s against -Linux's 113. Only the 100 ms mark is short, by 30 KB in one boot (the -parity boot, at the end of this record). Before #1114, ToyOS delivered -2.0–3.3 MB and 23.8–25.6 MB. Received TCP is at parity on that path. - -This record began as the upload at a third of Linux's (41 MB/s against 113). -What follows is that history, the causes found and removed, and the residual. - -**Measured** on 2026-10-24 with `osbench lan` at `d99ef5a35`. The path is -the T14's I219 at 1000 Mb/s, through the access point, to the AMD bench's -Linux `osbench-peer` on its Wi-Fi. Each cell is one run's p50 of five 3 s -rounds: - -| | ToyOS (3 runs) | Linux (4 runs) | -|---|---|---| -| `lan.wired.tcp.upload`, MB/s | 41, 48, 31 | 113, 113, 114, 113 | -| `lan.wired.tcp.download`, MB/s | 112, 112, 109 | 112, 110, 109, 112 | - -- No ToyOS round reached a Linux one: ToyOS's rounds ran 31 to 59 MB/s and - Linux's 112 to 114. The exact rank test over the runs gives p = 1/35. -- The same runs' HTTPS upload to the public internet took 1.40 times - Linux's time. - -Evidence: `ToyOSOrg/evidence` branch `pr-1065`, `r3/`. - -The paired session at `aeba59fb0` read 62, 63, 65 and 64 MB/s against -Linux's 113, 110, 111 and 112 (p = 0.029): [evidence `aeba59fb0/`](https://github.com/ToyOSOrg/evidence/tree/37288c84c175a647145fe83fd51d945806cdb4ff/r1/aeba59fb0). - -**Not the limit on the 2026-10-27 and -28 heads: the transmit ring's -depth.** At 16 descriptors the ring holds -15 frames, 184 µs of a gigabit wire, and 64 MB/s is exactly 15 segments -every 339 µs, so a ring drained before the pass that refills it was the -candidate. Pull request #1069 read it at 256 descriptors on 2026-10-27, -four ToyOS runs interleaved with four Linux ones: ToyOS 36, 36, 39 and -44 MB/s, Linux 112 in all four. The rounds inside each ToyOS run ranged from -27–35 to 46–67 MB/s, and downloads stayed at parity (110–113). On that -head the deeper ring moved nothing; whether depth limits a sender whose -window grows as it should is not read by it -([evidence `af239c8bb/`](https://github.com/ToyOSOrg/evidence/tree/37288c84c175a647145fe83fd51d945806cdb4ff/r1/af239c8bb)). - -**Ruled out: loss.** Pull request #1069's counters read on 2026-10-28, four -runs on the 16-descriptor ring and two on 256, against Linux's 111–112: no -retransmission timeout and no recovery in any run, one 1448-byte segment -resent across all six, no frame missed by the card. Linux resent none. The -256-descriptor ring's two runs read the same 36 MB/s with a ninth to a half -of the 16-descriptor ring's 60,665–66,044 full-ring waits -([evidence `132d5ef75/`](https://github.com/ToyOSOrg/evidence/tree/f06c0e92fa4c7172fe5c0946c540c34b434179dc/r2/132d5ef75)). - -**Found on the way: slow start grew once per pass.** [tcp]'s cwnd grew by -one segment for the first ACK after a stop at cwnd, and not for the rest: -netstack hands the stack every ACK one pass received before the pass's -transmit opportunity, so a pass's ACKs grew the window as one. #1069 makes -each ACK count (RFC 5681 §3.1), and the window grow only after an -opportunity cwnd ended, not one the card's room ended (RFC 9438 §5.8). -That is a correction the RFCs require, held by [tcp]'s host tests -`s_cc_016` and `s_cc_017`. Its effect on the T14's upload is read by one -session that alternated it with a control, below; the other readings come -from separate sessions and credit it nothing. - -**Read at `afe0c7dac`, 2026-10-28**, a head that also grew the window while -the card held the sender, so neither its rate nor its stop samples are the -per-ACK fix's alone; the corrected head, `d184223b0`, is read below. Upload 58, 59, 58 and 59 MB/s against -Linux's 110–112 ([evidence `afe0c7dac/`](https://github.com/ToyOSOrg/evidence/tree/695218c8513e785bf0465e2f13e0575f5723a2e3/r3/afe0c7dac)). Each ToyOS row's 15 s was -held by cwnd for 1.5–2.1 s, at stops whose mean cwnd was 192–228 KB and -mean smoothed round trip 4.5–5.9 ms; by the peer's window for 87–189 ms; -by the sender itself for about 1 ms. Every resent segment was a loss -probe, and nearly every one came back D-SACKed. Linux ended its rounds at a -mean cwnd of 1,706–2,208 segments and a round trip of 9.7–11.7 ms: its -window is never the limit, and its own queue fills the path. - -**Two hypotheses, which the `d184223b0` session tested**: - -- The card's refill limits a sender whose window no longer does: at - `afe0c7dac` the card was full 100,053–106,424 times a row, though that - head's cwnd times are not creditable. If so, the 256-descriptor ring - reads faster than 16 at `d184223b0` and its full-ring waits fall. -- The deeper ring read no better on 2026-10-27 and -28 because slow start, - growing once per pass, held those rows at cwnd. Nothing measured that: - `af239c8bb` had no counters, and `132d5ef75` charged Nagle holds to - congestion. - -**Read at `d184223b0`, 2026-10-28** -([evidence `d184223b0/`](https://github.com/ToyOSOrg/evidence/tree/bc4188c1070d283e38f7a96019437b99697fe4c5/r4/d184223b0)), four runs on 16 descriptors and two on 256, -against Linux's 112–113: - -- On 16 descriptors ToyOS read 58, 61, 59 and 57 MB/s. -- The ring's depth is not the limit. 256 descriptors read 60 and 57 MB/s. - The card was still full 77,036–79,567 times a row (100,246–106,959 on 16). - cwnd held the sender 4.6–4.9 s of a row (2.1–2.4 s on 16), at a mean of - 488–506 KB and 8.0 ms (310–354 KB and 4.1–5.6 ms on 16): on the deeper - ring cwnd was larger and the round trip longer, and the rate was the - same. That the deeper ring's own queue lengthened the round trip is an - inference from those three readings, not a measurement. -- No loss: one segment resent, D-SACKed, across the six runs, and no - probe that ended a slow start. - -So the first hypothesis did not hold: with the corrected slow start the -deeper ring still reads no better. The second is not settled by this -session, which has no unfixed control. What remains open -is how fast the card drains what it is given: even at 256 descriptors it -is full about 5,000 times a second. That lead is the I219's transmit -configuration (interrupt moderation, descriptor write-back, how often -netstack learns of completions) and is taken up apart from this record's -pull request. - -**The per-ACK fix, measured: 2026-10-28, interleaved with its control** -([evidence `3599f5bb3/`](https://github.com/ToyOSOrg/evidence/tree/d03d7394ed2d34deb1f0c9244bd85f78d7551537/r5/3599f5bb3)). A was `3599f5bb3` with only the per-ACK -fix reverted, every counter kept; B was `3599f5bb3`; four of each, A B A B, -with Linux at the session's start and end (109 and 110 MB/s). - -- B read 59, 56, 57 and 57 MB/s; A read 52, 31, 31 and 40. B was faster in - every pair, and over the eight runs the exact rank test's one-sided p is - 1/70. -- cwnd held each 15 s row 2.2–2.5 s under B, 5.5–8.9 s under A, at a mean - cwnd of 335–354 KB against 266–301 KB. -- Pooled across the four runs of each, A's rounds spanned 30–60 MB/s and - B's 55–66. - -So in one session the fix raises the rate and narrows the spread of its -rounds, pooled across runs, from 30 MB/s to 11. Heads without it read 31–65 MB/s across -earlier sessions while Linux held 110–114. The fix is consistent with most -of that, but not with `aeba59fb0`'s 62–65, above B's reading here. That -part of the spread across sessions stays unexplained. - -**The card sends slowly, read 2026-10-28** ([evidence `c54355277/`](https://github.com/ToyOSOrg/evidence/tree/5db2f7f5ed4e54c12fd74c8aa2368de4b4a6590a/r1/c54355277)). -Pull request #1075 counted, at each full ring, the frames `GPTC` had on the -wire that netstack had not seen written back: none, in either of two runs, -while the ring sat full 10.2–10.4 s of each 15 s row. The part sent what it -held at about 42,000 frames a second, 23.8 µs a frame against 12.3 µs of -wire. - -**Measured 2026-10-28** ([evidence `f9563de4f/`](https://github.com/ToyOSOrg/evidence/tree/d6fbb4b149683056857dd8554c2c77cfbdd465ef/r2/f9563de4f)): -`TCTL`'s `MULR` and `RRTHRESH` set, against the bring-up's whole write -clearing both, raises the upload. One session alternated the two, four of -each, with Linux at both ends (111 and 110 MB/s): - -- The bring-up left `TCTL` at `0x3003f0fa` with both fields and `0x0003f0fa` - without. -- Upload p50 87, 86, 82 and 85 MB/s with them, against 58, 56, 55 and 56 - without: faster in every pair, and over the eight runs the exact rank - test's one-sided p is 1/70. -- The wire carried 870,000–912,000 frames a row with them, against - 618,000–626,000 without. No completion was unreported at any full answer. -- The ring still sat full 8.3–8.7 s of each 15 s row (10.3–10.6 s without), - and cwnd held the sender 4.1–4.5 s (2.3–2.6 s without). - -**Derived from the datasheet, not separated by the measurement**: which of -the two fields the gain is. 82574 §10.2.6.1 has `MULR` set the number of -read requests for transmit data: "When set to 0b, the 82574 submits only -one request at a time, When set to 1b, the 82574 might submit up to four -concurrent requests". It gives `MULR` an initial value of `1b` and -`RRTHRESH` `01b`. - -**The T14's reset does not give §10.2.6.1's initial values.** §10.2.6.1 -says `MULR` "is loaded from the NVM word 0x24/0x14", and §4.2 that a -software reset "re-reads the per-function NVM fields" and resets the bits -normally read from the NVM to "their default hardware values". On the T14, -`TCTL` read `0x3003f0f8` both before and after the bring-up's reset -([evidence `19fbf239c/`](https://github.com/ToyOSOrg/evidence/tree/4cbc624ffec885040aa268ad47743ed5df94e51e/r4/19fbf239c)), `CT` and `COLD` included, where those -values would read about `0x30000008`. Either the reset kept the register or -this MAC's own reset value is that one: no reading separates the two, and -the 82574's table does not describe the PCH's MAC. Either way the value the -bring-up met was not one the driver can count on, so it now writes every -`TCTL` field itself, `MULR` and `RRTHRESH` at §10.2.6.1's initial values. - -**The rest of the transmit setup the datasheet requires, read 2026-10-28** -([evidence `800e35c69/`](https://github.com/ToyOSOrg/evidence/tree/a911b388eac3990bde439b95091b41a00335748c/r1/800e35c69)): no measured rate effect. Pull request #1078 sets -`TXDCTL` bit 22, which 82574 §10.2.6.11 says "Must be set to 1b for proper -operation", leaves `TIDV` alone, sets `TARC0`'s `ENABLE` and writes the -receive filter registers. One session alternated main's driver (A), that -change (B) and that change with transmit prefetch (C, `PTHRESH` 16, -`HTHRESH` 1), three of each, with Linux at both ends (110 and 111 MB/s): - -- Upload p50 A 86, 86 and 85 MB/s, B 85, 86 and 86, C 88, 87 and 87. B reads - no different from A, and lands as the datasheet's requirement, not a gain. -- C read higher than B in all three rounds, by about 1.5 %, which three runs - a side cannot resolve: the exact rank test's least p is 1/20 one-sided and - 1/10 two-sided. The prefetch tuning did not land. -- The ring sat full 7.8–8.1 s of each 15 s row in all three arms, and no - completion was unreported at a full answer. - -What is left, about 1.3 times short of Linux with the ring full about half -of each row whatever the descriptor fetch does, is the next lead: what the -part waits on for each frame once it holds it. - -**The part keeps line rate while it holds frames; the gap is each -connection's first second, read 2026-10-28** ([evidence `d1ea41856/`](https://github.com/ToyOSOrg/evidence/tree/9f309c28654d89c37e8fb0defc9fd2b1c61cc91b/r1/d1ea41856)). -On the full ring the I219 sent 83,600 frames a second, line rate. One session -read what the peer had received after its first byte, a mean over each row's -five connections, at `d1ea41856` (A) and A with a card-held stop counted as -cwnd's (B), two of each, with Linux at both ends: - -| received by | 100 ms | 300 ms | 1 s | 2 s | upload p50 | -|---|---|---|---|---|---| -| Linux (2 runs) | 5.9–6.4 MB | 27.9–29.0 MB | 109–110 MB | 224–225 MB | 113, 113 | -| A (2 runs) | 0.32 MB | 2.0–2.1 MB | 38–41 MB | 149–153 MB | 88, 89 | -| B (2 runs) | 0.32–0.35 MB | 2.1–2.3 MB | 46–47 MB | 162–163 MB | 93, 92 | - -- From 1 s to 2 s ToyOS carried 112–117 MB, Linux 115. Nothing was lost. -- The peer acknowledged about 50 KB, 34 segments, an ACK. Each ACK that grew - cwnd grew it by one SMSS (RFC 5681 §3.1's min(N, SMSS)), so slow start grew - a few percent a round trip instead of doubling. -- Under A, 75–76 % of ACKs grew nothing: the card's room had ended the - opportunity before them with more than a segment of cwnd left. - -Pull request #1102 replaces the initial slow start with HyStart++ -(RFC 9406, which RFC 9438 §4.10 recommends for CUBIC), whose L = 8 for a -sender that does not pace caps the growth an ACK gives at eight segments, and -counts a card-held window as in use when at least half of it is in flight -(RFC 7661 §4.3's validated window, which RFC 9438 §4.6 admits). - -**#1102's A/B, 2026-10-28** ([evidence](https://github.com/ToyOSOrg/evidence/tree/8f70147d28011645de1fd458f3091297287b57b9/r2/ab)). A was the fix -reverted, B the fix; three of each, A B A B A B, with Linux at both ends: - -| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | 2 s | -|---|---|---|---|---|---| -| Linux (2 runs) | 113, 113 | 6.4–6.5 MB | 29.7 MB | 110–111 MB | 226–227 MB | -| A (3 runs) | 87, 87, 87 | 0.33–0.34 MB | 2.0–2.1 MB | 34–38 MB | 144–149 MB | -| B (3 runs) | 112, 112, 112 | 2.1–3.3 MB | 23.9–25.4 MB | 106–107 MB | 222–223 MB | - -- B was faster in every pair: one-sided p = 1/20. No segment was resent in - any B run, and no loss probe was sent. -- B's rounds ran 111–115 MB/s. Linux's 113 in this session is above them, - inside Linux's 109–114 across this record's sessions. -- What is left is the first 300 ms. In every B row HyStart++ entered - Conservative Slow Start 590–702 times, about 120–140 a connection, and - resumed every time, never exiting. CSS grows by a quarter while it lasts. - That this oscillation is the residual is an inference: no run separates it. - -The clock does not separate the two systems. Each upload row's own -readings, `mhz` just after it and `mhz_settled` before it, were about -1.6 GHz under ToyOS (settled 1.6, 1.8 or 3.8 GHz). Under Linux they were -1.5 GHz (settled 1.5 or 4.2 GHz), and Linux still moved 113 MB/s. At most, -the clock matters through what ToyOS spends per byte sent. - -**#1112's session, 2026-10-29** -([evidence](https://github.com/ToyOSOrg/evidence/tree/96c693b58d242d8ec236e0805b34b714306ec4c2/r1/t14)). -B was #1102's head, C per-segment RTT samples (#1112), D C with L = ∞ and -no pacing, a diagnostic arm that does not land. Three of each, B C D B C D -B C D, with Linux at both ends: - -| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | -|---|---|---|---|---| -| Linux (2 runs) | 113, 114 | 5.9–6.0 MB | 27.2–29.0 MB | 108–110 MB | -| B (3 runs) | 112, 112, 112 | 2.1–2.4 MB | 24.0–24.7 MB | 105.5–105.8 MB | -| C (3 runs) | 112, 112, 112 | 2.2–3.2 MB | 24.3–25.6 MB | 105.2–106.6 MB | -| D (3 runs) | 114, 113, 114 | 5.2–6.4 MB | 28.3–29.7 MB | 109.9–111.2 MB | - -- L = 8 is what holds the ramp. Without it D reached Linux's marks, and - nothing was resent. -- CSS entries were 2.0–2.7 times the rounds that entered CSS, in D as in B - and C. Most of them are flaps within a round (s_cc_023), and removing them - was not needed for parity. -- Finer RTT samples (C) did not change the ramp. - -RFC 9406 §4.3 gives L = ∞ only to a paced sender. Pull request #1114 -paces [tcp] (RFC 9002 §7.7) and removes the cap. - -**#1114's A/B, 2026-10-31** -([evidence](https://github.com/ToyOSOrg/evidence/tree/58dcdecab592116996c0e3cc84e4e3c8b7aa50c7/r4/t14)). -C was #1112 after its first review, P #1114's pacing with N = 2 in slow -start, Q P with N = 1.25 throughout. Three of each, C P Q C P Q C P Q, with -Linux at both ends, on trees from before netstack's drivers became their -own processes (#1106): - -| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | -|---|---|---|---|---| -| Linux (2 runs) | 112, 112 | 5.7 MB | 26.1–26.2 MB | 105.7–106.4 MB | -| C (3 runs) | 112, 112, 112 | 2.0–3.1 MB | 23.8–25.4 MB | 105.4–106.8 MB | -| P (3 runs) | 114, 114, 114 | 6.3–6.6 MB | 29.3–29.9 MB | 110.9–111.1 MB | -| Q (3 runs) | 114, 114, 114 | 5.9–6.3 MB | 29.1–29.5 MB | 110.2–111.0 MB | - -- P beat C at 300 ms in every pair, and resent nothing. -- P beat Q at 100 ms in every pair, by 0.1–0.5 MB, so N = 2 in slow start - stays. -- In C every RTT sample but two came from a segment's departure. -- That session's bucket could hold up to a millisecond at the rate, which on - a short round trip is more than the window. #1114's pacer, as of - 3a0744d34, holds at most half the window. The parity read waits on a boot - of that head, on current main. - -**The parity boot, 2026-11-01** -([evidence](https://github.com/ToyOSOrg/evidence/tree/8f08a8e9399817fc669e395dc717ee236a18ed35/r7/t14)). -One boot of #1114 at 3a0744d34, on main after #1106, with Linux before and -after it: - -| | upload p50, MB/s | received by 100 ms | 300 ms | 1 s | 2 s | -|---|---|---|---|---|---| -| Linux (2 runs) | 113, 113 | 6.10, 6.38 MB | 28.63, 29.56 MB | 109.1, 110.5 MB | 224.6, 226.1 MB | -| ToyOS | 114 | 6.07 MB | 29.19 MB | 110.7 MB | 227.0 MB | - -- The rule set before the run asked for the 100 ms and 300 ms marks at or - above the lower Linux run's, p50 within 1 MB/s, and nothing resent. The - 300 ms mark, p50 and loss passed. The 100 ms mark missed by 30 KB, 0.5 %, - so the rule is not met and this record stays open. -- That gap is smaller than the spread of the marks across runs: Linux's two - runs differ by 280 KB, and the A/B's three P runs by 260 KB. One boot does - not separate it from noise. -- Nothing was resent, and no RTO, recovery or loss probe occurred. Every RTT - sample came from a departure (33,319, none from the echo). CSS entered in - 134 rounds, 420 times, and never exited. -- The pacer held 73 ms over the row, against 33–41 ms with the A/B's - bucket. The half-window bound holds more on the short round trip, as - expected. - -**The 100 ms miss is not resolved by the bench, 2026-11-01.** Three readings -say the parity boot's 30 KB is below what the session form can tell apart: - -- The rule cannot pass reliably at parity. With one ToyOS run between two Linux - runs and all three drawn alike, ToyOS is the lowest of the three one time in - three. A third of sessions would miss at exact parity. -- The instrument's step is a read. The peer counts at a mark what it had - before the read that passed it, and it reads up to 64 KiB at a time - (`osbench-lan`'s `RampCount`). So each round's mark can sit up to 64 KiB - low, and how far depends on how the sender's segments meet the peer's - reads. -- The marks spread more than the miss, and where they spread is the session. - Linux's two runs differed by 280 KB at 100 ms in the parity boot. Across - this record's sessions Linux read 5.7–6.5 MB. The paced sender read 6.3–6.6 - MB in its A/B, above both Linux runs (5.7 MB) in all three pairs, and 6.07 MB - in the parity boot. - -A model of the ramp gives no candidate either. On [tcp]'s host network, a -1 Gb/s sender into a card at 78 frames a millisecond (113 MB/s) at round trips -of 2, 4 and 8 ms delivers 10.26, 8.27 and 3.49 MB by 100 ms. - -- Two changes move those by under 0.1 MB: the 1 ms bucket without the - half-window bound, and a window stop that grows cwnd at half of cwnd in - flight, as a card stop does (RFC 9438 §4.6 admits RFC 7661 there). -- Removing pacing costs 0.07, 0.48 and 1.1 MB. - -That model's receiver is [tcp]'s, not the peer's Linux. On the bench, both -senders are held by the peer's window first: ToyOS for 66 ms a connection in -the parity boot, Linux for 92–148 ms. So the first 100 ms reads mostly the -receiver's window growth, which both senders share. - -`osbench` now reports each round's marks beside their mean -(`peer_received_by_ms_bytes`), so a session gives five connections a -run and the run's own spread. - -**The exit's first session read no ToyOS row, 2026-11-01** -([evidence](https://github.com/ToyOSOrg/evidence/tree/35f09268e09714a8547496bdd766bcd3ca998325/r1)). This was Linux, ToyOS at `4f52c3056`, Linux, all -exiting 0. ToyOS's `osbench_lan` found the peer and then refused the group: -`no netstack link holds 10.0.2.15`. std's `TcpStream::local_addr` answered -QEMU's user-network address whatever the lease was, and since #1122 osbench -names a connection's link by that address. The T14's lease is not it. So the -session does not decide. The next one runs on a head where netstack's -answers carry each stream's local address and std, libc and the SDK return it. - -Its Linux runs do give the per-connection spread the rule needs: - -- p50 113 and 113 MB/s. -- At 100 ms the rounds ran 5.82–6.63 MB and 5.13–6.38 MB, a standard - deviation of 478 KB over the ten. The parity boot's 30 KB miss is a - sixteenth of that. -- At 300 ms they ran 28.8–30.0 MB and 27.7–29.7 MB. -- The two runs do not differ: the two-sided rank test gives p = 0.69 at - 100 ms and 0.42 at 300 ms. So the rule's exchangeability check passes. - -**Exit**: in one T14 session, Linux, ToyOS, Linux, `osbench_lan`'s -`lan.wired.tcp.upload` reads: - -- p50 at or above the lower Linux run's, less 1 MB/s; -- nothing resent beyond a loss probe; -- at 100 ms and at 300 ms, ToyOS's five rounds not ranked below Linux's ten: - the exact one-sided rank test gives p > 0.05, so parity fails one session - in twenty. - -If Linux's two runs differ at either mark at p ≤ 0.05 by the exact two-sided -rank test, the connections are not exchangeable across runs. Then the session -does not decide, and an interleaving of Linux, ToyOS, Linux, ToyOS, Linux -compares runs instead. Or the cause is named and owned by its own issue.