Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
status: open
kind: defect
opened: 2026-10-08
---

# A listening socket's TCP_NODELAY is refused, and an accepted stream's is answered from a guess

A host takes `TCP_NODELAY` on a listening socket and gives it to the
connections that begin afterwards. Read on macOS 27 on arm64, by libc's
`setsockopt` and `getsockopt` over loopback: set before the connection began,
the accepted socket has the option; set after the connection was established
and before the accept, it does not. `userland/netstack/node/tests/host.rs`
asks the same of every host the suite runs on, and sets the option in the
second case only once the listener is readable, which is the host saying the
connection waits to be accepted: a connect returns before the listener's end
is established, so the answer rests on that event and not on the order
loopback delivers in. A red there prints the host's name and both answers.

ToyOS as it ships does neither half:

- netstack answers `TcpSetOption` and `TcpGetOption` `ERR_NOT_CONNECTED` for
any id that is not a stream (`handle_tcp_set_option`,
`userland/netstack/src/main.rs`), and libc's `setsockopt` sends a listening
socket's id there (`userland/libc/src/socket.rs`), so a C program that sets
the option on its listener is refused.
- std builds every `TcpStream`, an accepted one included, with
`nodelay: false` and answers `nodelay()` from that field without asking
netstack (`sdk/std/sys/net/connection.rs`). Nothing can make that false
today, since no accepted connection has the option. Once a listener's
option reaches the connections it accepts, it is false for each of them.

The node (`userland/netstack/node/src/listeners.rs`) has the calls,
`Node::set_listener_nodelay` and `Node::listener_nodelay`, and gives an
accepted stream what its connection took from its listener; nothing ships it.

Not read: Linux. No Linux host was at hand when this was filed. The `host`
check runs `tests/host.rs` on Linux, and its first run on a pull request that
carries the file is that reading.

**Exit condition**: `tests/host.rs` is green in a `host` check's log, which
is the Linux reading; and on ToyOS a program that sets `TCP_NODELAY` on a
listening socket and accepts one connection begun before and one begun after
reads the option on each as that test has a host answer, by libc's
`getsockopt` and by std's `nodelay()`.

**Owner**: whoever holds `issues/toyos-has-its-own-network-stack.md`; the
Linux reading is the orchestrator's, at the first `host` run of the pull
request that adds `tests/host.rs`.
16 changes: 16 additions & 0 deletions issues/netstack-datagram-sockets-and-listeners-have-no-bound.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,22 @@ the two that exist.

Read from the code, not measured.

On the node (`userland/netstack/node/src/places.rs`) a datagram socket and a
listener each hold a place, and a bind or a listen with none left is refused
with nothing made: `a_datagram_socket_holds_a_place_and_a_bind_without_one_makes_nothing`
and `a_listener_holds_a_place_and_a_listen_without_one_makes_nothing`
(`userland/netstack/node/tests/listeners.rs`). The sockets of the node's own,
the responder's for the machine's name and each query's of a lookup, hold no
place: a lookup is bounded by `toyos_dns::MAX_LOOKUPS` and its rounds, so
clients at the bound refuse no lookup
(`a_query_with_no_port_to_leave_from_ends_its_lookup_by_name`,
`userland/netstack/node/tests/resolve.rs`). What is left: netstack as it
ships is the code above until it runs on the node; the node's places are one
number for every client, so its tests bind past the bound and see the refusal
but have no second client whose bind is answered, which the track records
with its own exit; and the word the pipe ABI answers a refused listen in is
the move's to map.

**Exit condition**: a bind past a stated bound is refused
`ERR_RESOURCE_EXHAUSTED` with nothing made, and a test binds past it and sees
the refusal and another client's bind answered.
Expand Down
11 changes: 8 additions & 3 deletions issues/toyos-has-its-own-network-stack.md

Large diffs are not rendered by default.

50 changes: 46 additions & 4 deletions toyos-net-shard/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,15 @@ pub enum ConnectError {
Tcp(toyos_net_tcp::Error),
}

/// Why a passive open was refused.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ListenError {
/// The address named is none the interface may be bound to: the rule [udp]'s bind refuses
/// by (`udp.bind-address-not-local`).
NotLocal,
Tcp(toyos_net_tcp::Error),
}

/// [ip]'s MTU, as TCP's configuration takes it.
const TCP_MTU: u16 = {
let [low, high, rest @ ..] = toyos_net_ip::MTU.to_le_bytes();
Expand Down Expand Up @@ -467,16 +476,44 @@ impl Shard {
connected
}

/// `addr` is the local address to listen on, UNSPECIFIED for any; port 0 takes `random`'s
/// draws.
pub fn listen(&mut self, addr: Ipv4Addr, port: Option<Port>, random: impl FnMut() -> u16) -> Result<ListenerId, toyos_net_tcp::Error> {
self.tcp.listen(addr, port, random)
/// `addr` is the local address to listen on, UNSPECIFIED for any, and otherwise one [ip]
/// holds assigned or announcing, as a datagram socket's is; port 0 takes `random`'s draws.
pub fn listen(&mut self, addr: Ipv4Addr, port: Option<Port>, random: impl FnMut() -> u16) -> Result<ListenerId, ListenError> {
if !addr.is_unspecified() && !self.ip.is_assigned(addr) {
return Err(ListenError::NotLocal);
}
self.tcp.listen(addr, port, random).map_err(ListenError::Tcp)
}

pub fn accept(&mut self, id: ListenerId) -> Result<Option<ConnId>, toyos_net_tcp::Error> {
self.tcp.accept(id)
}

pub fn listener_port(&mut self, id: ListenerId) -> Result<Port, toyos_net_tcp::Error> {
self.tcp.listener_port(id)
}

/// [`Tcp::set_listener_options`]: what a connection whose SYN arrives from here on starts
/// with.
pub fn set_listener_options(&mut self, id: ListenerId, options: toyos_net_tcp::Options) -> Result<(), toyos_net_tcp::Error> {
self.tcp.set_listener_options(id, options)
}

/// [`Tcp::ready`].
pub fn ready(&mut self, id: ListenerId) -> Result<usize, toyos_net_tcp::Error> {
self.tcp.ready(id)
}

/// [`Tcp::close_listener`]. The resets it owes leave outside the round.
pub fn close_listener(&mut self, now: Instant, id: ListenerId) -> Result<(), toyos_net_tcp::Error> {
self.tcp.close_listener(now, id)
}

/// [`Tcp::orphans`].
pub fn orphans(&self) -> usize {
self.tcp.orphans()
}

pub fn send(&mut self, now: Instant, id: ConnId, data: &[u8]) -> Result<usize, toyos_net_tcp::Error> {
let done = self.tcp.send(now, id, data);
self.settle(now);
Expand All @@ -502,6 +539,11 @@ impl Shard {
done
}

/// [`Tcp::options`].
pub fn options(&mut self, id: ConnId) -> Result<toyos_net_tcp::Options, toyos_net_tcp::Error> {
self.tcp.options(id)
}

pub fn shutdown_write(&mut self, now: Instant, id: ConnId) -> Result<(), toyos_net_tcp::Error> {
let done = self.tcp.shutdown_write(now, id);
self.settle(now);
Expand Down
24 changes: 24 additions & 0 deletions toyos-net-shard/tcp/src/stack.rs
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,8 @@ pub struct Tcp {
demux: BTreeMap<Tuple, Entry>,
time_waits: BTreeSet<(Instant, Tuple)>,
deadlines: BTreeSet<(Instant, u32)>,
/// Connections whose user let go and that have not ended.
orphans: usize,
/// Connections that became eligible for the caller's round since it last drained them.
eligible: Vec<ConnId>,
/// Connections freed while offered to the caller's round, since it last drained them.
Expand Down Expand Up @@ -375,6 +377,7 @@ impl Tcp {
demux: BTreeMap::new(),
time_waits: BTreeSet::new(),
deadlines: BTreeSet::new(),
orphans: 0,
eligible: Vec::new(),
gone: Vec::new(),
stubs: VecDeque::new(),
Expand All @@ -401,6 +404,11 @@ impl Tcp {
self.time_waits.len()
}

/// Connections [`Self::close`] left to finish alone, each until it ends.
pub fn orphans(&self) -> usize {
self.orphans
}

fn local(&self, tuple: &Tuple) -> Local {
let mtu = self.config.mtu;
Local {
Expand Down Expand Up @@ -441,6 +449,9 @@ impl Tcp {
fn free(&mut self, index: u32) {
let Some(generation) = self.conns.get(usize::try_from(index).unwrap_or(usize::MAX)).map(|s| s.generation) else { return };
let Some(conn) = release(&mut self.conns, &mut self.free_conns, index) else { return };
if conn.user == User::Orphan {
self.orphans = self.orphans.saturating_sub(1);
}
let remote = conn.tuple.remote.addr;
let parked = self.parked.get(&remote).is_some_and(|p| p.conns.contains(&index));
// Its index may name another connection next.
Expand Down Expand Up @@ -572,6 +583,11 @@ impl Tcp {
slot(&mut self.listeners, id.index, id.generation).map(|l| l.options = options).ok_or(Error::NoSuchSocket)
}

/// How many children completed their handshake and wait for [`Self::accept`].
pub fn ready(&mut self, id: ListenerId) -> Result<usize, Error> {
slot(&mut self.listeners, id.index, id.generation).map(|l| l.ready.len()).ok_or(Error::NoSuchSocket)
}

/// The oldest child that completed its handshake.
pub fn accept(&mut self, id: ListenerId) -> Result<Option<ConnId>, Error> {
let listener = slot(&mut self.listeners, id.index, id.generation).ok_or(Error::NoSuchSocket)?;
Expand Down Expand Up @@ -958,6 +974,12 @@ impl Tcp {

// ---- user calls ----

/// The options the connection has: the ones its listener had when its SYN arrived, or the
/// defaults of an active open, until [`Self::set_options`] writes others.
pub fn options(&mut self, id: ConnId) -> Result<Options, Error> {
Ok(self.conn(id)?.options)
}

pub fn set_options(&mut self, now: Instant, id: ConnId, options: Options) -> Result<(), Error> {
self.conn(id)?.options = options;
self.settle(id.index, now);
Expand Down Expand Up @@ -1056,6 +1078,7 @@ impl Tcp {
Tcb::SynRcvd(rcvd) => {
rcvd.shutdown_write();
conn.user = User::Orphan;
self.orphans = self.orphans.saturating_add(1);
self.settle(id.index, now);
}
Tcb::Sync(sync) if sync.rx.unread() > 0 => {
Expand All @@ -1068,6 +1091,7 @@ impl Tcp {
sync.shutdown_write(now);
sync.orphan(now);
conn.user = User::Orphan;
self.orphans = self.orphans.saturating_add(1);
self.settle(id.index, now);
}
}
Expand Down
83 changes: 83 additions & 0 deletions toyos-net-shard/tcp/tests/held.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
//! What [tcp] says it holds, for a caller that bounds it or keeps a copy: `orphans`, `ready` and
//! `options`. No scenario ids: the specifications hold none of the calls; the connections
//! counted are `close.rs`'s and `listen.rs`'s, by their scenarios' own segments.

mod common;

use common::*;
use toyos_net_tcp::{limits, Error, Options, State};

#[test]
fn an_orphan_is_counted_until_both_fins() {
let mut h = fixture_e();
assert_eq!(h.tcp.orphans(), 0, "a held connection is its user's");
h.close(0);
assert_eq!(h.tcp.orphans(), 1);
h.input(10, seg(5001).ack(1002).fin());
assert_eq!((h.tcp.orphans(), h.tcp.time_wait_count()), (0, 1), "TIME-WAIT is no connection");
}

#[test]
fn an_orphan_is_counted_until_its_idle_reset() {
let mut h = fixture_e();
h.close(0);
h.input(10, seg(5001).ack(1002));
h.at(60_009);
assert_eq!(h.tcp.orphans(), 1);
h.at(60_010);
assert_eq!(h.tcp.orphans(), 0);
}

#[test]
fn a_close_before_the_handshake_ends_leaves_an_orphan_until_it_gives_up() {
let mut h = H::new(65_535);
h.start(0);
h.conn = Some(h.tcp.connect(h.now(), A, Some(port(49152)), ep(B, 80)).unwrap());
h.transmit();
h.input(5, seg(5000).syn().mss(1460));
assert_eq!(h.status().state, State::SynReceived);
h.close(6);
assert_eq!(h.tcp.orphans(), 1);
h.at(2 * i64::try_from(limits::SYN_GIVE_UP.as_millis()).unwrap());
assert_eq!(h.tcp.orphans(), 0);
}

#[test]
fn a_close_that_resets_leaves_no_orphan() {
let mut h = fixture_e();
h.input(0, seg(5001).ack(1001).len(100));
h.close(1);
assert_eq!(h.tcp.orphans(), 0, "unread text makes the close a reset");
}

#[test]
fn ready_counts_the_connections_accept_has_yet_to_return() {
let mut h = listening();
let listener = h.listener.unwrap();
h.input(0, seg(5000).syn().mss(1460));
assert_eq!(h.tcp.ready(listener), Ok(0), "a handshake in progress is not ready");
h.input(1, seg(5001).ack(1001));
assert_eq!(h.tcp.ready(listener), Ok(1));
assert!(h.tcp.accept(listener).unwrap().is_some());
assert_eq!(h.tcp.ready(listener), Ok(0));
let now = h.now();
h.tcp.close_listener(now, listener).unwrap();
assert_eq!(h.tcp.ready(listener), Err(toyos_net_tcp::Error::NoSuchSocket));
}

#[test]
fn options_are_the_ones_the_connection_has() {
let mut h = listening();
let listener = h.listener.unwrap();
let nodelay = Options { nodelay: true, ..Options::default() };
h.tcp.set_listener_options(listener, nodelay).unwrap();
h.input(0, seg(5000).syn().mss(1460));
h.input(1, seg(5001).ack(1001));
let conn = h.tcp.accept(listener).unwrap().unwrap();
assert_eq!(h.tcp.options(conn), Ok(nodelay), "its listener's, as its SYN found them");
let now = h.now();
h.tcp.set_options(now, conn, Options::default()).unwrap();
assert_eq!(h.tcp.options(conn), Ok(Options::default()));
h.tcp.abort(now, conn).unwrap();
assert_eq!(h.tcp.options(conn), Err(Error::NoSuchSocket));
}
8 changes: 7 additions & 1 deletion userland/netstack/node/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "toyos-net-node"
description = "Netstack's decisions, pure: one interface of ToyOS's own stack, its DHCP lease, its clients' datagram sockets and streams, its mDNS name and its resolver, frames and time in, frames out against the device's credit."
description = "Netstack's decisions, pure: one interface of ToyOS's own stack, its DHCP lease, its clients' datagram sockets, streams and listeners and the one bound on all three, its mDNS name and its resolver, frames and time in, frames out against the device's credit."
version = "0.1.0"
edition = "2021"
license = "MIT OR Apache-2.0"
Expand All @@ -23,3 +23,9 @@ toyos-net-wire = { path = "../../../toyos-net-wire" }
# The outside reader of every frame the node emits: a published Ethernet, ARP,
# IPv4, UDP and ICMP parser, used as a black box.
etherparse = "0.21.0"
# The host's own TCP as an oracle, on Linux, macOS and Windows alike, with no
# `unsafe`: `socket2` sets and reads a socket option std has no call for on a
# listener, and `mio` waits until a listener has a connection to accept, which
# std can learn only by accepting it.
mio = { version = "1", default-features = false, features = ["os-poll", "net"] }
socket2 = "0.6"
18 changes: 15 additions & 3 deletions userland/netstack/node/src/datagram.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
//! nothing. A destination, a port or a length is the client's number and is refused, never
//! trusted; a datagram off the wire reaches a socket only through [udp]'s demultiplexer.
//!
//! **A socket holds a place** (`places`) from its bind to its close: what it makes the node hold
//! is its two queues, `limits::RX_DATAGRAMS` received and `limits::TX_DATAGRAMS` accepted. A bind
//! with no place left is answered [`Refused::ResourceExhausted`] with nothing made.
//!
//! A refusal [udp] logs comes out of [`Node::drain_events`] after the node's next `receive`,
//! `fire` or `link`.

Expand All @@ -32,7 +36,8 @@ pub enum Refused {
/// The call names what no datagram of this socket may carry: an address that is not this
/// machine's to bind, a destination nothing is sent to, port 0, more than one frame holds.
InvalidInput,
/// The socket holds all it may until some of it has left: the same call succeeds later.
/// The socket holds all it may until some of it has left, or the node holds all it has
/// places for: the same call succeeds later.
ResourceExhausted,
}

Expand Down Expand Up @@ -98,7 +103,11 @@ impl Node {
/// or, with none named, an ephemeral one, which spends the one draw. Returns the socket and
/// the port it holds.
pub fn udp_bind(&mut self, addr: Ipv4Addr, port: Option<Port>, draw: impl FnOnce() -> u32) -> Result<(DatagramId, Port), Refused> {
if self.room() == 0 {
return Err(Refused::ResourceExhausted);
}
let (id, port) = self.stack.bind(addr, port, draw).map_err(refused)?;
self.sockets = self.sockets.saturating_add(1);
Ok((DatagramId(id), port))
}

Expand All @@ -115,8 +124,11 @@ impl Node {
}

/// Closes the socket: its port is free at once, what it received is gone, and what it had
/// accepted still leaves, to [udp]'s bound.
/// accepted still leaves, to [udp]'s bound. Its place is back.
pub fn udp_close(&mut self, now: Instant, id: DatagramId) -> Result<(), Refused> {
self.stack.close(now, id.0).map_err(refused)
self.stack.close(now, id.0).map_err(refused)?;
self.sockets = self.sockets.saturating_sub(1);
self.wake_owners(now);
Ok(())
}
}
Loading
Loading