diff --git a/quest/m0/README.md b/quest/m0/README.md index 8263867ee71..5ec1c79c845 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -47,6 +47,7 @@ Published API or wire breaks still land on dev; each quest's Plan says so. - [quest check everywhere](/quest/m0/quest-check-everywhere.md) - `quest check` guards `main`, `dev`, and the line branches on push and PR, not only PRs into `main` - [noq reassembly cap](/quest/m0/noq-reassembly-cap.md) - noq carries quinn's stream reassembly cap and the connection receive window is finite by default - [qmux reset race](/quest/m0/qmux-reset-race.md) - qmux handles RESET_STREAM under one lock instead of panicking +- [Path hunting](/quest/m0/path-hunting.md) - a withdrawn path is retracted about once per relay instead of hunting stale alternatives, with a hold-down on route updates and no wire change - [Shared fronts](/quest/m0/shared-fronts.md) - viewer sessions share a front, so fronts scale with peers, not viewers - [Frame alloc budget](/quest/m0/frame-alloc-budget.md) - frame buffers pre-allocate within a per-session budget and otherwise grow with bytes received - [Handshake deadline](/quest/m0/handshake-deadline.md) - an unfinished handshake or slow HTTP header times out diff --git a/quest/m0/path-hunting.md b/quest/m0/path-hunting.md new file mode 100644 index 00000000000..7ca416b2467 --- /dev/null +++ b/quest/m0/path-hunting.md @@ -0,0 +1,49 @@ +# [M] A withdrawn path stops hunting + +## Goal + +When a broadcast ends, each relay in a lite-06 mesh retracts it about once, +instead of re-announcing stale alternatives. Today one withdrawal on live's +34-relay graph costs hundreds to over a thousand extra announce starts, and +on Kick dev an ended name drew 575 to 57,805 mesh announce starts (median +about 13,000) while live names drew one flood (moq-dev/moq.pro#2116). New +and removed routes still propagate at once, and no wire changes. + +## Plan + +Promoted from the cluster routing line to m0 on 2026-09-30: it is today's +partial-mesh hunting that #4399 left open, and it is hurting production. + +Decided (2026-09-30), lite-06, no wire change: + +- A route *update*, where a relay's advertised route for a path switches to a + different route, waits a short hold-down before it is re-advertised. A new + path (nothing advertised yet) and a removal (nothing left) go out at once. +- So when the best route is withdrawn and an alternative remains, the relay + retracts at once and re-announces the alternative only if it survives the + hold-down. Stale alternatives are withdrawn inside that window and never + spread. A genuine failover, where a link dies but the origin still + publishes, is delayed by the hold-down; pick the value from the repro, and + report the failover cost it adds. +- Check the interaction with #4399's withdrawal hiding and with + subscriptions already pinned to a route (fronts keep serving from a live + source; only advertisements are held). + +Test first: land moq.pro's in-process repro upstream (real lite-06 sessions +over the mock transport on live's 34-relay graph, mocked time), asserting +about one retraction per relay for a single withdrawal from a non-hub +publisher, plus a failover test where a link dies and the path comes back +through another route after the hold-down. It must fail on `main` today. + +Then moq.pro moves its pin and confirms about one retraction wave per ended +name on Kick dev after a deploy the user approves (moq.pro's own quest). + +If no hold-down closes it, the fallback is a per-origin seqno on cluster +links in the wip lite version; report before building it. + +Public API: none expected. Wire: none. + +## Related + +- [Cluster routing](/quest/m1/cluster-routing/README.md) - tiers shrink the core graph this hunts on +- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - closed broadcasts announced for minutes, the same symptom diff --git a/quest/m1/README.md b/quest/m1/README.md index ff2fe1e7033..c7c305e9aac 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -23,7 +23,10 @@ QUIC studies there on that rule. ## Required - [Cluster idle timeout](/quest/m1/cluster-idle-timeout.md) - a relay notices a silent peer relay within seconds, not after the shared 30 s QUIC idle timeout -- [Cluster routing](/quest/m1/cluster-routing/README.md) - a relay learns the relay graph once and each announcement once, not once per neighbour, and redundant publishers share an epoch instead of `--hop` +- [moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md) - no moq-transport session negotiates the cluster extension, so none can splice a pool's members on a Hop ID label +- [Cluster routing](/quest/m1/cluster-routing/README.md) - edge and core tiers carry each broadcast into a region once over path vector, with the backbone hidden from end users +- [Remove `--hop`](/quest/m1/hop-removal.md) - on `dev`, redundant publishers share an explicit `@`, and `--hop` and the publisher's Hop ID are gone +- [One route cost](/quest/m1/route-cost.md) - on `dev`, Warm and Cold collapse to one static route cost - [Track tail interop](/quest/m1/track-tail-interop.md) - a Rust publisher ending a track with a group in flight is read to its end by the JS subscriber, and the reverse, in `just test interop` - [Merge queue](/quest/m1/merge-queue.md) - the required checks run on `merge_group`, so a stale green check can no longer break main - [Binding audio delay](/quest/m1/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 16121aff8dd..386f8655e93 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -2,128 +2,66 @@ ## Goal -Cut cluster gossip. A relay learns how to reach each other relay once, from -a topology shared across the cluster rather than repeated in every route, -and learns each announcement once rather than once per neighbour. Every relay -still holds the ledger of live announcements that ANNOUNCE_REQUEST and a cold -SUBSCRIBE need, and a broadcast under overlapping prefixes routes to one -origin deterministically. Redundancy an operator configures may deliver more -than one copy. - -Non-goals: warm re-origination (a warm relay would be one more origin with a -cost, so leave room for it), and a permanently mixed-version cluster. +A cluster carries each broadcast into a region once and never exposes its +backbone to end users. Relays take an explicit edge or core role: edges +serve end users and spread paths over their region's cores, cores link to +the cores of other regions over configured links with static costs, and +routing between them stays path vector. A broadcast under overlapping +prefixes routes to one origin deterministically. Redundancy an operator +configures may deliver more than one copy. + +Non-goals: edges acting as each other's intermediates (it makes edges a +bigger DDoS target), cache-aware route switching, and a permanently +mixed-version cluster. ## Plan ### Decisions -Settled in the 2026-09-30 `/quest-plan`: - -- The goal stays wide. The design below is the current candidate, not a - decision; [Propagation](/quest/m1/cluster-routing/propagation.md) settles - how announcements move and writes the implementation children that follow. -- Topology is a cluster message kept apart from routes, built from configured - links only (`--cluster-connect`, the connect API, LAN mDNS). Gossip - discovery goes first, but only the topology child waits on it. -- Every relay holds a record for every live announcement, since - ANNOUNCE_REQUEST and a SUBSCRIBE with no prior announce both need one. - Scoping that knowledge by demand needs registries, which Propagation may - defer to m2. -- `--hop` goes. An epoch-qualified concrete path (`foo/@`) is the - identity of a source: origins that announce the same one are - interchangeable, which is how a redundant pair is expressed. It is strictly - better than a Hop ID, which is per session, so a connection could not - publish several broadcasts with different identities. A path a claim - produces keeps Wildcard's per-origin identity, even once announced - concretely, since each worker's output is its own. -- The line lands on `dev`: deleting `--hop` and the publisher's Hop setup - parameter breaks a published CLI and wire. Wire changes go in the current - wip version (`moq-lite-07-wip` today, dropping `Hop Base` and `Hop Keep` - before they publish). If lite-07 is finalized first for the Wildcard - rollout, the remaining children move to the next wip version; finalizing - never waits on this line. -- A short idle timeout for cluster sessions is its own m1 quest, - [Cluster idle timeout](/quest/m1/cluster-idle-timeout.md), since it helps - today's path vector too. -- Reduced flooding ([RFC 9667](https://www.rfc-editor.org/rfc/rfc9667)) and - registries are m2 unless Propagation pulls registries in; Propagation writes - whichever it defers. +Settled in the 2026-09-30 wildcard audit (cache tiers), replacing the +link-state and existence-split design planned earlier that day: + +- Two roles, edge and core. Where a region has one edge, that edge is also + its core. A region may have several cores, which do not link to each other. + An edge dials every core in its region over qmux on TLS (intra-region links + are not congested) and picks one per path by rendezvous hashing, so a + broadcast crosses into a region once. An edge never re-advertises one + core's routes to another, so it is never transit. Cores dial the cores of + the regions they link to, only those with lower names, so each pair has one + connection. Cores are hidden from end users, which is the DDoS shield. See + [Edge and core](/quest/m1/cluster-routing/roles.md). +- Core links are configured and static, and may skip PoPs when the RTT is + short: Seattle pulls from San Jose directly rather than via Oregon. Nothing + switches routes on cache state, so Warm and Cold collapse to one route cost + ([One route cost](/quest/m1/route-cost.md)), and the wildcard line's route + upgrade quest is deleted. +- No link-state topology. With split-horizon edges and a sparse core graph, + path vector with hop lists stays loop-free, path hunting is confined to the + cores, liveness flooding (the simulator's dominant link-state cost) is never + paid, and failure detection sets the outage window either way. The parked + implementation is #4631. If stale paths remain on the tiered layout, the + fix is [Path hunting](/quest/m0/path-hunting.md), promoted to m0: a hold-down + on route updates, with no wire change. +- The line lands on `main`: its children are additive. The two breaking + changes left it for `dev` on their own: + [Remove `--hop`](/quest/m1/hop-removal.md) and + [One route cost](/quest/m1/route-cost.md). +- Cluster links are moq-lite only; moq-transport peers are plain clients + ([moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md)). +- An epoch-qualified concrete path (`foo/@`) is a source's identity: + origins that announce the same one are interchangeable, which is how a + redundant pair is expressed. A path a claim produces keeps Wildcard's + per-origin identity. See [Selection](/quest/m1/cluster-routing/selection.md). - Between clusters, announcements stay path vector with cluster ids as hops, in the last child. - -### Why not path vector or Babel - -Today every relay advertises its best route to every peer not already in the -hop chain, and pulls every peer's full table with an empty-prefix -ANNOUNCE_REQUEST. One publish costs about R·(d-1) announces for R relays of -mesh degree d, every relay learns every broadcast (`.stats` and `.internal` -included), and a link change rewrites every route crossing it. On moq.pro's -live fleet (26 PoPs, average degree about 5) each relay receives every event -about five times. Babel ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966)) -shrinks the message and skips equal-cost reroutes, but it is still distance -vector: the same fan-out, the same global knowledge, and no loop freedom -when several sources claim a prefix (section 2.7), which pools and wildcards -make MoQ's common case. - -moq.pro's routing simulator (`just rs sim`, -[moq.pro#2020](https://github.com/moq-dev/moq.pro/pull/2020)) measured worse -than that on live's relay graph (34 relays, mean degree 8.4): a publish costs -about 750 announces, and ending a path's only publisher explores stale -alternatives for about a second and 39k to 80k announces, with subscribes -looping meanwhile. Coalescing each writer for 10 ms cuts that by 7x and leaves -the loops. Hiding the routes through a peer that withdrew the path -([#4399](https://github.com/moq-dev/moq/pull/4399)) reduces that path hunting -but does not end it; see the findings below. - -### Candidate design - -The children own their parts: topology in -[Topology](/quest/m1/cluster-routing/topology.md), existence and registries -in [Propagation](/quest/m1/cluster-routing/propagation.md), origin choice in -[Selection](/quest/m1/cluster-routing/selection.md). - -- Existence is split from reachability. An announcement carries the path, its - origin relay, and the origin's cost, and nothing about the path to it. -- An existence event carries the origin's seqno, scoped to its incarnation. A - relay applies an event only when it is newer than the last it applied for - that path and origin, and keeps an ended path's seqno, so a start delayed on - a stale tree or a failed-over registry cannot revive it. Babel keeps - feasibility past withdrawal for the same reason - ([RFC 8966 section 3.7.3](https://www.rfc-editor.org/rfc/rfc8966#section-3.7.3)). -- A new incarnation is learned from existence, never inferred from liveness: - it announces a reset (its incarnation at seqno 0) that ends everything the - old one published, and a relay applies a reset and the records after it as - one batch. Incarnations are ordered, so a delayed reset from an older one is - dropped. Purging on a liveness report instead drops a restarted origin's - broadcasts until their re-announce lands, which the simulator showed as live - broadcasts reported offline. -- Relays flood per-link liveness among themselves with a per-link seqno scoped - to the relay's incarnation, batched for a short hold-down, with a digest - exchange on session up. -- A relay picks the origin with the lowest shortest-path distance plus origin - cost, ties broken by rendezvous hashing (HRW) of the requested path and the - origin id, and forwards along its shortest path. The longest covering - prefix still ranks first, per [Wildcard](/quest/m0/wildcard/README.md). The - first relay's choice rides the SUBSCRIBE, and transit relays forward toward - that origin by topology alone. SUBSCRIBE and FETCH carry a visited-relay - list as a loop safety net. -- Announcements to clients are on demand: a relay forwards only the union of - its clients' ANNOUNCE_REQUEST prefixes. `.stats` becomes ordinary demand. -- Registries are an optional, configured tier: moq-relay in a registry mode, - one or more per region. Registries form a small full mesh and flood - existence among themselves, so an event crosses an ocean once per remote - registry. A relay fails over to the next-nearest registry and reconciles; - with none reachable it freezes its view and alerts rather than falling back - to flooding. -- Without registries, existence floods along the shortest-path tree, one copy - per relay. A relay forwards an event only when it changes its view, and a - relay that gains a child in its tree pushes that origin's reset and records - to it. -- The cluster switches versions as a whole; older lite and IETF sessions stay - at its edges. +- Anything specific to moq.pro's deployment (generating peer lists and roles + from its inventory, its simulator) is planned in moq.pro, not here. ### Simulator findings +Measured on the earlier flat mesh; they motivated the link-state design the +tiers decision replaced, and still bound what path vector costs. + The report is moq.pro's `just rs sim` (every scenario on live's graph) and `just rs sim sweep` (synthetic regional graphs of 34, 340, and 1020 relays). It carries messages and bytes by kind, per relay and cross-region, @@ -147,8 +85,8 @@ registry. relay loss is noticed after the 30 s QUIC idle timeout in every candidate, and subscribes through it go nowhere until then. - The simulator saw no loop while views agreed, and HRW split an equal-cost - pool 63/49 where today's hash of the announced prefix sends all of it to one - sibling. On live's graph no disagreement looped across 20 seeds of link, + pool 63/49 where a hash of the announced prefix sent all of it to one + sibling (Wildcard has since keyed the tie on the requested path). On live's graph no disagreement looped across 20 seeds of link, cost, and relay churn. - One registry per region and two cost about the same; two halves the busiest registry's load. A registration sent on a dead registry session @@ -169,26 +107,25 @@ registry. Once every child has landed: -- An end-to-end test: a multi-relay cluster in `rs/moq-relay/tests` where a - publish, an end, a relay restart, and a redundant-pair failover each reach - a subscriber on a far relay, with time mocked. -- Rewrite `doc/bin/relay/cluster.md` into the operator's view (topology, - link costs, idle timeout, redundant pairs), and add a routing page under - `doc/concept`. +- An end-to-end test: a two-region cluster in `rs/moq-relay/tests` where a + publish, an end, a core loss, and a redundant-pair failover each reach a + subscriber on a far edge, with time mocked. +- Rewrite `doc/bin/relay/cluster.md` into the operator's view (edge and core + layout, dial rules, TLS edge links, link costs, idle timeout, redundant + pairs), and add a routing page under `doc/concept`. ## Required -- [Topology](/quest/m1/cluster-routing/topology.md) - relays learn the relay graph once from a cluster message, apart from routes -- [Propagation](/quest/m1/cluster-routing/propagation.md) - decides how each announcement reaches every relay once, and writes the implementation children +- [Edge and core](/quest/m1/cluster-routing/roles.md) - relays take an explicit edge or core role; edges spread paths over their region's cores and are never transit - [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-epoch origins are one source -- [Remove `--hop`](/quest/m1/cluster-routing/hop-removal.md) - redundant publishers share an explicit `@`, and `--hop` and the publisher's Hop ID are gone - [Between clusters](/quest/m1/cluster-routing/inter-cluster.md) - announcements crossing a cluster boundary stay path vector with cluster ids as hops ## Related - [Wildcard](/quest/m0/wildcard/README.md) - the longest-prefix rule, pool spread, and reply identity Selection builds on -- [Cluster idle timeout](/quest/m1/cluster-idle-timeout.md) - failure detection sets every outage window, today and after this line +- [Remove `--hop`](/quest/m1/hop-removal.md) - on `dev`: redundant publishers share an explicit `@` +- [One route cost](/quest/m1/route-cost.md) - on `dev`: Warm and Cold collapse to one static cost - [Same-hop importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair; `--hop` removal re-keys it to a shared epoch - [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report also shows closed broadcasts announced for up to 229 s and flapping between Retracted and Announced across nodes, evidence for per-incarnation seqnos +- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s, evidence for per-origin seqnos - [Routing cost domains](/quest/m3/routing-cost-domains.md) - cost across the cluster boundaries this keeps path vector diff --git a/quest/m1/cluster-routing/inter-cluster.md b/quest/m1/cluster-routing/inter-cluster.md index 43386d65eff..7bc07c072fa 100644 --- a/quest/m1/cluster-routing/inter-cluster.md +++ b/quest/m1/cluster-routing/inter-cluster.md @@ -5,18 +5,18 @@ Announcements crossing a cluster boundary stay path vector with cluster ids as the hops, like BGP between autonomous systems. A customer's on-prem cluster is one hop, and an announcement naming the receiving cluster is -dropped. Inside a cluster nothing carries a list of relay hops. +dropped. Inside a cluster, hop lists name relays; across a boundary they name +clusters. ## Plan - A boundary is a configured link to a relay of another cluster; decide how a relay knows its own cluster id and its peer's. -- What crosses a boundary is what the chosen - [Propagation](/quest/m1/cluster-routing/propagation.md) design holds, plus - its cluster-id list. An imported record keeps that list as provenance, so +- What crosses a boundary is a cluster's announcement, plus its cluster-id + list. An imported record keeps that list as provenance, so whichever boundary relay exports it appends its own cluster id to the full list; stripping it on import would let A → B → C → A re-enter A. -- The remote origin is outside the receiving cluster's topology, so the +- The remote origin is outside the receiving cluster, so the importing boundary relay announces the record inside its cluster as the origin, with the boundary link's cost added. The serving origin's identity still rides the reply, so re-originating does not merge two sources. @@ -31,10 +31,6 @@ draft updated in the same PR. Tests: two clusters with two boundary links and an origin that is not a boundary relay, and three clusters in a cycle with different ingress and egress relays, where an announcement neither loops nor re-enters its origin cluster. -## Required - -- [Propagation](/quest/m1/cluster-routing/propagation.md) - the record shape a boundary exports - ## Related - [Routing cost domains](/quest/m3/routing-cost-domains.md) - designs policy on this path vector diff --git a/quest/m1/cluster-routing/propagation.md b/quest/m1/cluster-routing/propagation.md deleted file mode 100644 index 283b0d20c78..00000000000 --- a/quest/m1/cluster-routing/propagation.md +++ /dev/null @@ -1,67 +0,0 @@ -# [M] Plan announcement propagation - -## Goal - -Decide how an announcement reaches every relay in a cluster once, rather than -once per neighbour, and how every relay keeps the ledger of live announcements -that ANNOUNCE_REQUEST and a cold SUBSCRIBE need. The output is a decision -recorded in the [questline README](/quest/m1/cluster-routing/README.md), -worked counterexamples, and rewritten implementation quests, not production -code. - -## Plan - -Decided: every relay holds a record for every live announcement, and an -announcement names its origin relay and cost, not a path to it -([Topology](/quest/m1/cluster-routing/topology.md) supplies reachability). - -Candidates to choose between or combine: - -- **Tree flooding with per-origin seqnos.** The README's candidate design: - existence floods along the origin's shortest-path tree, a relay forwards an - event only when it changes its view, and per-origin seqnos with - incarnation resets stop stale revivals. This is the same as "skip a - neighbour already closer to the origin": a relay forwards only to its - children in that origin's tree. -- **A central node owns gossip.** Registries: an optional tier, one or more - per region, full-meshed among themselves. An ingest relay registers with - its nearest, an edge sends its ANNOUNCE_REQUEST there, and a relay fails - over to the next-nearest and reconciles (its full live set at its current - seqno, ending whatever it leaves out; the registry's snapshot carries each - origin's last reconcile seqno). With none reachable a relay freezes its - view and alerts. The simulator found registries buy cross-ocean bytes and - one-round-trip announce latency, not message count, at live's 34 relays. -- **A compressed ledger.** The records are a replicated ledger of every live - announcement; sync it with digests (anti-entropy) and compress it, such as - prefix-shared paths, instead of per-event messages. - -Measure the choice with the simulator's scenarios (messages and bytes per -event, convergence, stale revival, restart, failover). - -Open questions the chosen design must answer: - -- How long a relay keeps an ended path's seqno. A new origin incarnation - clears it; within one, it must outlive every delayed copy of the start. -- How a relay that just became another's tree child catches up on what it - missed within the same incarnation. Its reset only marks a new incarnation, - so the push may need a watermark like the registry reconcile's: it ends only - what the child holds at or below it, and a delayed push cannot erase newer - records. -- How long a relay keeps the records of an origin that never comes back. They - stop being reported once it is unreachable, but nothing removes them. -- Whether registries belong in this line. If deferred, write an m2 quest for - them with sharding (HRW over a prefix key) and a mixed-version bridge as its - open questions, and one for reduced flooding - ([RFC 9667](https://www.rfc-editor.org/rfc/rfc9667)). - -The implementation quests it writes replace the empty-prefix ANNOUNCE_REQUEST -between peers and per-hop path-vector announcements, drop `Hop Base` and -`Hop Keep` from the current wip lite version. Add each one to the Required of -[Selection](/quest/m1/cluster-routing/selection.md) and -[Routing between clusters](/quest/m1/cluster-routing/inter-cluster.md), so -neither starts on a record shape that is not implemented yet. - -## Related - -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its report of closed broadcasts announced for minutes is evidence for per-origin seqnos -- [Announcement shapes](/quest/m2/announce-shapes.md) - announcements between relays must keep their shape diff --git a/quest/m1/cluster-routing/roles.md b/quest/m1/cluster-routing/roles.md new file mode 100644 index 00000000000..c6e2f9628e9 --- /dev/null +++ b/quest/m1/cluster-routing/roles.md @@ -0,0 +1,54 @@ +# [M] Edge and core relays + +## Goal + +A relay runs as an edge or a core, and a cluster built from them carries +each broadcast into a region once. An edge serves end users, dials every core +in its region, spreads paths over those cores by rendezvous hashing, and is +never transit between cores. A core serves only cluster peers, dials the +cores of the regions it links to, and never dials an edge. + +## Plan + +Decided in the 2026-09-30 wildcard audit (cache tiers): + +- The role is the node's own `cluster.role`, explicit, from config or + alongside the peer list the connect API returns. A relay whose links + contradict its role (an edge linked to another edge, a core dialing an + edge) fails at startup. Where a region has one edge, that edge is its core. +- A relay with no role is a legacy peer and links any peer, as before roles. + A core accepts and dials a legacy peer as a core-grade link, and an edge + never links one. This is what lets an operator migrate one region at a + time: a migrated region's cores bridge to the unmigrated mesh. A + permanently mixed cluster stays a non-goal. +- A core refuses a session that is not a cluster peer, loudly, so a + misrouted client fails instead of silently exposing the shield. +- Cores dial only cores with lower names, so each pair has one connection; + cores in one region do not link to each other. This is configuration: the + relay follows its peer list, and whatever generates the list applies the + rule. +- Edge links run qmux over TLS on TCP, not WebSocket and not QUIC: + intra-region links are not congested. qmux over TCP exists today only in + plaintext (`tcp://`); add `tls://`, wired to qmux's existing `tls` module. + (Scheme name decided 2026-09-30.) +- An edge never re-advertises a route learned from one core to another core: + its cluster dials publish only what entered locally + (`origin::Consumer::local()`), so no core routes through an edge. +- An edge picks a core per path with the path-keyed rendezvous hash the + wildcard line gave `route_order`, so every edge in a region sends a path to + the same core, and failover moves only the paths the lost core served. The + per-path spread across equally ranked cores is + [Selection](/quest/m1/cluster-routing/selection.md)'s. + +Test: a two-region cluster (two edges and two cores in one region, one core +in the other) in `rs/moq-relay/tests`, where subscribers on both edges of a +region cause one cross-region subscription per broadcast, a route from one +core never reaches another through an edge, a client dialing a core is +refused, and losing a core moves only its paths. + +Public API: a relay config field for the role and a TLS qmux URL scheme. +Wire: none expected. + +## Related + +- [moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md) - cluster links are lite-only diff --git a/quest/m1/cluster-routing/selection.md b/quest/m1/cluster-routing/selection.md index c75f2b5b81a..8cb3b0cbd50 100644 --- a/quest/m1/cluster-routing/selection.md +++ b/quest/m1/cluster-routing/selection.md @@ -3,7 +3,8 @@ ## Goal A broadcast under overlapping announcements routes to one origin chosen -deterministically, and every relay forwards toward that origin by topology. +deterministically, and an edge spreads paths over its region's cores so +every edge sends a given path the same way. Origins that announce the same epoch-qualified concrete path are one source, so a subscriber moves between them at a group boundary when the incumbent ends or becomes unreachable. @@ -26,45 +27,34 @@ Decided: announce the same epoch-qualified path and must not pool. This extends Wildcard's resume rule to concrete same-epoch origins. -Candidate mechanics: +Candidate mechanics (re-scoped to path vector by the 2026-09-30 tiers +decision; there is no topology to forward by): -- A relay picks the origin with the lowest shortest-path distance plus origin - cost, ties broken by rendezvous hashing (HRW) of the requested path and the - origin id, and forwards along its shortest path. That is a shortest path to - a virtual node linked to every origin, so it is loop-free whenever relays - agree on the topology. -- The first relay's choice rides the SUBSCRIBE and FETCH, and transit relays - forward toward that origin by topology alone, never re-selecting. - Re-selection against another existence view loops: a relay that lost a - specific claim falls back to a broader one through a relay still routing to - the specific one - ([RFC 8966 section 3.5.4](https://www.rfc-editor.org/rfc/rfc8966#section-3.5.4)). - A refusal follows Wildcard's refusal rule: only a capacity refusal lets the - first relay select once more within the same longest-prefix tier, excluding - the refusing origin, and any other refusal is terminal. -- SUBSCRIBE and FETCH carry a visited-relay list end to end. It catches loops - while liveness views disagree and names the path for stats. The serving - origin's identity rides the reply, per Wildcard's first-hop resume rule. +- Route selection stays `route_order` over the routes a relay holds: longest + prefix, then cost, then a rendezvous hash (HRW) of the requested path and + the route's origin. At an edge that spreads paths over the region's cores + and makes every edge pick the same core for a path; failover moves only the + paths the lost core won. +- A refusal follows Wildcard's refusal rule: every refusal is terminal, and + an origin sheds load by withdrawing or re-pricing its route instead. +- The serving origin's identity rides the Origin field of the reply, per + Wildcard's resume rule. Open: -- Whether equal-cost next hops should spread by a hash of the path. A fixed - tie-break sends every path through the same neighbour, and its failure - takes them all. - How a relay tells claim output from a redundant pair. One candidate: a concrete path an origin announces under its own claim keeps per-origin identity, since a redundant publisher claims nothing. -Wire: SUBSCRIBE and FETCH fields in the current wip lite version, with the -draft updated in the same PR. Tests cover an HRW split across an equal-cost +Wire: none expected; if one is needed it goes in the current wip lite +version with the draft. Tests cover an HRW split across an equal-cost pool, refusal and reselection, a same-epoch pair failing over mid-track with no timestamp rewind, and a concrete double claim whose loser's subscribers end and resubscribe rather than splice. ## Required -- [Topology](/quest/m1/cluster-routing/topology.md) - distance to each origin -- [Propagation](/quest/m1/cluster-routing/propagation.md) - the record shape that names each origin +- [Edge and core](/quest/m1/cluster-routing/roles.md) - the edges and cores this spreads over - [Wildcard](/quest/m0/wildcard/README.md) - the longest-prefix rule, pool spread, and reply identity this builds on ## Related diff --git a/quest/m1/cluster-routing/topology.md b/quest/m1/cluster-routing/topology.md deleted file mode 100644 index d10cc59f9c2..00000000000 --- a/quest/m1/cluster-routing/topology.md +++ /dev/null @@ -1,56 +0,0 @@ -# [L] Cluster topology - -## Goal - -Relays learn the relay graph once, from a cluster message kept apart from -routes, instead of every route repeating how to reach its origin. Each relay -knows every configured link, whether it is up, and its cost, and computes its -shortest path to every other relay. No announcement is needed to learn that -relay K exists or how to reach it. - -## Plan - -Decided: the topology is configured. `--cluster-connect` or the connect API -gives the relay graph and link costs, and LAN mDNS dials peers that then count -as configured links. Gossip discovery is already gone. - -Candidate mechanics, from the simulator (see the questline README's -findings): - -- Relays flood per-link liveness among themselves with a per-link seqno. The - seqno is scoped to the relay's incarnation, so a restarted relay's links - supersede its stale ones instead of looking older. -- A relay batches the liveness reports it sends, its own and those it - forwards, for a short hold-down (50 ms in the simulator), and recomputes its - shortest paths after a matching delay, as OSPF's SPF delay does. Unbatched, - one relay restart at 340 relays sent half a million messages; batched, 27k. -- On session up, relays exchange a digest (each reporter's incarnation and - its seqno per link) and send only what the other lacks. A reporter's - reports flood separately, so its newest seqno alone would hide a missing - older one. The digest already counts the fresh report of the link that just - came up; sending the database first replays the relay's own report from - when the link went down, and the peer drops the link it is using. -- Distance compares cost, then hop count, so every hop strictly shortens it - even across `?cost=0` links. - -Reduced flooding ([RFC 9667](https://www.rfc-editor.org/rfc/rfc9667)) is -out of scope; [Propagation](/quest/m1/cluster-routing/propagation.md) writes -it as an m2 quest if it stays deferred. - -Open: moq-transport cluster peers. Cluster links accept moq-transport 17+ -today via the cluster extension (`drafts/draft-lcurley-moq-cluster.md`), -which these messages do not reach. -Either extend that draft, or refuse a cluster session that does not negotiate -the wip lite version, per the README's "the cluster switches versions as a -whole". Never accept a peer that cannot carry the topology. - -Wire: new cluster-session messages in the current wip lite version, with the -draft updated in the same PR. Tests drive topologies in process with mocked -time, including restart, a link flapping, and a digest racing a link that -just came up. Expose the computed graph where operators already look -(`/nodes` in `rs/moq-relay/src/internal.rs`). - -## Related - -- [Cluster idle timeout](/quest/m1/cluster-idle-timeout.md) - liveness only reports what the session layer detects -- [Drain](/quest/m1/drain/README.md) - a second relay per PoP joins this topology diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 250076067bd..56ec73b95bc 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -15,7 +15,7 @@ exits (#4354). Decided: same-hop publishers MUST publish the same broadcasts and tracks. Keep `--hop` for now: the users' bugs are on one relay today. Cluster routing replaces it with a shared explicit epoch on `dev` (decided -2026-09-30), and its [`--hop` removal](/quest/m1/cluster-routing/hop-removal.md) +2026-09-30), and its [`--hop` removal](/quest/m1/hop-removal.md) re-keys this quest's docs and tests. The importer work holds under either key. Make every container importer (ts, fmp4, flv, mkv, and the SRT, RTMP, and HLS gateways that reuse them) meet the contract when fed one @@ -44,5 +44,5 @@ survives the standby joining and the incumbent stopping. ## Related -- [Remove `--hop`](/quest/m1/cluster-routing/hop-removal.md) - on dev, a redundant pair shares an explicit epoch instead, and this quest's docs and tests are re-keyed to it +- [Remove `--hop`](/quest/m1/hop-removal.md) - on dev, a redundant pair shares an explicit epoch instead, and this quest's docs and tests are re-keyed to it - [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - a redundant pair shares one epoch diff --git a/quest/m1/cluster-routing/hop-removal.md b/quest/m1/hop-removal.md similarity index 96% rename from quest/m1/cluster-routing/hop-removal.md rename to quest/m1/hop-removal.md index 2dcdf8221f1..7000837b894 100644 --- a/quest/m1/cluster-routing/hop-removal.md +++ b/quest/m1/hop-removal.md @@ -32,7 +32,7 @@ idle timeout (a failover service that retracts it, or active-active delivery to the relay). Neither is required to land this. Public API and wire: removes a CLI flag and the publisher's Hop setup -parameter; lands with the line on `dev`. +parameter; lands on `dev`, outside the cluster routing line (which lands on `main`). ## Required diff --git a/quest/m1/ietf-cluster-off.md b/quest/m1/ietf-cluster-off.md new file mode 100644 index 00000000000..4c8bd4be470 --- /dev/null +++ b/quest/m1/ietf-cluster-off.md @@ -0,0 +1,40 @@ +# [S] moq-transport peers are plain clients + +## Goal + +No moq-transport session negotiates the MoQ Cluster extension +(`draft-lcurley-moq-cluster`). A moq-transport peer is always served as a +plain client: it never receives Hop IDs or relay costs and never acts as a +cluster link. Cluster links are moq-lite only. moq-transport end users stay +fully supported. + +## Plan + +Why now: the Wildcard line spreads a pool's paths behind one route, and names +the serving member only in lite-07's reply Origin. The cluster extension has +no such field, so a moq-transport downstream relay pins failover to the +advertisement's first Hop ID, which labels the pool rather than the member, +and can splice one member's objects onto another's. A peer that never learns +Hop IDs cannot make that mistake. Decided in the 2026-09-30 wildcard audit, +matching the topology decision that cluster links are lite-only. + +- Stop offering and accepting the cluster Setup Options in Rust + (`rs/moq-net/src/ietf/cluster.rs`, `parameters.rs`, `publish_namespace.rs`) + and JS (`js/net/src/ietf/cluster.ts` and its callers). A peer that offers + them is served as if it had not. +- Delete what becomes dead, in both languages. The draft stays: + [moq-transport cluster peers](/quest/m2/ietf-cluster-peers.md) extends it + and brings the code back. +- Update `doc/bin/relay/cluster.md` and `doc/concept` wherever they say + moq-transport relays can join a cluster. +- Test: a moq-transport 17+ peer that offers the extension gets a session + with no cluster parameters, and an announcement reaches it with no Hop IDs. + +Lands on `main`: extension negotiation is optional, so a peer that offered it +still gets a working session. Public API: none expected. Wire: we stop +negotiating an optional extension. + +## Related + +- [Edge and core](/quest/m1/cluster-routing/roles.md) - cluster links are lite-only +- [Wildcard](/quest/m0/wildcard/README.md) - the pool spread that makes a Hop ID label unsafe to splice on diff --git a/quest/m1/route-cost.md b/quest/m1/route-cost.md new file mode 100644 index 00000000000..e058957d870 --- /dev/null +++ b/quest/m1/route-cost.md @@ -0,0 +1,29 @@ +# [M] One route cost + +## Goal + +A route carries one static cost. Warm and Cold Route Cost, and the +cache-state accounting behind Warm, are gone from the next wip lite version +and from moq-net's model. lite-06 peers keep working. + +## Plan + +Decided in the 2026-09-30 wildcard audit (cache tiers): with configured edge +and core tiers and static core links, nothing switches routes on cache +state, so Warm has no job. + +- The next wip lite version (`moq-lite-07-wip` while it is unpublished) + carries a single route cost in ANNOUNCE_START and ANNOUNCE_UPDATE. Update + `drafts/draft-lcurley-moq-lite.md` in the same PR. +- lite-06 sessions keep parsing both fields: read Warm as the cost and + ignore Cold, and write the one cost as Warm with Cold at the saturation + ceiling. +- The single cost still prices a standby claim above a live one, per the + wildcard line's standby floor. Update that text in + `quest/m0/wildcard/README.md` and anything else naming `Cost { warm, cold }`. +- Replace `Cost { warm, cold }` in `rs/moq-net/src/model/origin.rs` and its + JS mirror, and delete what computes Warm from cache state. +- The moq-transport cluster extension already carries one cost. + +Public API: `Cost` changes shape (a break, so it lands on `dev`). Wire: the +wip version drops a field; lite-06 is unchanged. diff --git a/quest/m2/README.md b/quest/m2/README.md index 4de4234a256..e9c6f6e9f11 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -112,5 +112,6 @@ waiting on an upstream release waits in [m4](/quest/m4/README.md). - [SEI separation study](/quest/m2/sei.md) - measure whether separating SEI saves enough, or has a metadata-only consumer, to justify a split - [Compressed tracks](/quest/m2/flate.md) - moq-ffi and every wrapper expose flate tracks through a `flate` namespace like `json` - [Announcement shapes](/quest/m2/announce-shapes.md) - moq-lite announcements and interests carry prefix, exact, suffix, or prefix+suffix shapes that survive relay hops, benchmarked over the announce table +- [moq-transport cluster peers](/quest/m2/ietf-cluster-peers.md) - an extended cluster draft lets moq-transport relays peer again, stitching failover on the reply's origin - [MSFTS convergence](/quest/m2/msfts-convergence.md) - the demultiplexed TS lane converges on MSFTS where the two still differ: program tables and the ES payload unit - [VAAPI encode and decode](/quest/m2/video-vaapi.md) - H.265 encode and decode, pre-generated bindings, and pooled resize surfaces, including the moq-dev/vaapi release that carries them diff --git a/quest/m2/ietf-cluster-peers.md b/quest/m2/ietf-cluster-peers.md new file mode 100644 index 00000000000..24c9d708499 --- /dev/null +++ b/quest/m2/ietf-cluster-peers.md @@ -0,0 +1,32 @@ +# [L] moq-transport cluster peers + +## Goal + +A moq-transport relay can join a cluster as a peer again, through an +extended `draft-lcurley-moq-cluster`, and never splices one pool member's +content onto another's. + +## Plan + +Replaces the wildcard line's "Cluster origin reply" quest. Until this lands, +moq-transport peers are plain clients +([moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md)). + +The extension must carry what a lite cluster link carries by then: + +- The serving origin in the reply, mirroring lite-07's SUBSCRIBE_OK and + FETCH_OK Origin, so a downstream stitches failover on the member serving the + path rather than the advertisement's first Hop ID. The draft's "Several + Publishers of One Namespace" section changes with it. +- Whatever the [cluster routing line](/quest/m1/cluster-routing/README.md) + settles for topology and selection, since cluster links are lite-only there. + +Test: two workers behind a pool relay behind a moq-transport downstream relay; +killing the serving worker ends the downstream subscription rather than +splicing the survivor's objects. + +Wire: extends `draft-lcurley-moq-cluster`, updated in the same PR. + +## Required + +- [Cluster routing](/quest/m1/cluster-routing/README.md) - settles what a cluster link carries