From 4d22184d2cd93541e359e52321312a03b3d96da6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 20 Oct 2026 12:20:00 -0700 Subject: [PATCH 1/2] chore: claim redundant ingest guide quest Co-Authored-By: GPT-6.1 From 244a308479e91a569c5489a12b332a09b1777cd0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 20 Oct 2026 12:23:08 -0700 Subject: [PATCH 2/2] docs: explain redundant ingest and route failover Co-Authored-By: GPT-6.1 --- doc/.vitepress/config.ts | 1 + doc/bin/cli.md | 10 +-- doc/bin/relay/cluster.md | 2 + doc/setup/redundancy.md | 128 +++++++++++++++++++++++++++++ quest/m1/README.md | 1 - quest/m1/redundant-ingest-guide.md | 37 --------- 6 files changed, 134 insertions(+), 45 deletions(-) create mode 100644 doc/setup/redundancy.md delete mode 100644 quest/m1/redundant-ingest-guide.md diff --git a/doc/.vitepress/config.ts b/doc/.vitepress/config.ts index 42b1fc54d4e..fa8aefcaf4f 100644 --- a/doc/.vitepress/config.ts +++ b/doc/.vitepress/config.ts @@ -77,6 +77,7 @@ export default defineConfig({ { text: "Install", link: "/setup/install" }, { text: "Development", link: "/setup/dev" }, { text: "Production", link: "/setup/prod" }, + { text: "Redundant ingest", link: "/setup/redundancy" }, { text: "Upgrade", link: "/setup/upgrade" }, { text: "Coding agents", link: "/setup/agent" }, ], diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 455ebac8acb..4ee33a97b19 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -347,13 +347,9 @@ Other versions carry no epoch, so a restart from another session is the newest route at an equal cost, or a better one. Each independent importer publishes its own epoch, even when two importers read -one stream and publish one path. On a version that carries epochs, the newest -importer wins new requests. If it leaves, a still-live older importer wins again -as a restart. Players following announcements and `moq export ts --stitch` -follow those restarts; other consumers end. `--stitch` also switches as soon as a newer -importer starts, even while the current importer remains healthy. Each switch -marks a program discontinuity, so redundant ingest does not promise hitless -output. +one stream and publish one path. See [Redundant ingest](/setup/redundancy) for +same-path switching, subscriber-controlled selection on separate paths, and +several routes to one publisher. An importer announces once its catalog names a servable track. Input that ends before then, such as an empty input or a header that configures no track, diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index bc5e58d6b39..d9ba66e9715 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -48,6 +48,8 @@ worker. Failover routes must carry copies of one publisher's broadcast. Independent importers use distinct epochs and switch as a restart, even when fed one stream. +See [Redundant ingest](/setup/redundancy) to choose between publisher switching, +subscriber selection, and several routes to one publisher. A source whose track differs in timescale, retention, publisher priority, or group order is refused for that track. diff --git a/doc/setup/redundancy.md b/doc/setup/redundancy.md new file mode 100644 index 00000000000..6e3c5961e8f --- /dev/null +++ b/doc/setup/redundancy.md @@ -0,0 +1,128 @@ +--- +title: Redundant Ingest +description: Protect publishers and delivery routes without confusing broadcast identity +--- + +# Redundant ingest + +Choose what must survive: a publisher failing, a delivery route failing, or +both. Two independent publishers can announce one path for automatic switching, +or use separate paths so the subscriber chooses. Several routes carrying one +publisher's broadcast can resume the same stream. + +| Setup | What a failure does | +| --- | --- | +| Independent publishers, one path | Switches to another broadcast at that path. | +| Independent publishers, separate paths | Leaves selection to the subscriber. | +| One publisher, several routes | Resumes the same broadcast on another route. | + +A [path and publisher epoch](/concept/moq-lite#publisher-epochs) identify one +broadcast. Every independent publisher mints its own epoch, even if both read +the same source. Sharing an epoch would claim that their group and frame +numbers name the same content, which independently running importers cannot +promise. + +## Independent publishers on one path + +1. Run the publishers on separate hosts and ingress routes. Connect their + relays into a [cluster](/bin/relay/cluster) so the viewer's relay can reach + both. Give both publishers permission to announce the same full path. +2. Publish the same broadcast name under the same auth path. Each publisher + keeps its own epoch; `moq` creates one automatically. +3. Use epoch-capable connections on every hop: moq-lite 07, which is opt-in, + or moq-transport draft 17 or later with the [epoch extension](/draft/moq-epoch) + enabled. Keep the publisher hosts' clocks synchronized, since epochs order + by their UUIDv7 creation time. + +For example, two HLS importers can publish `event.hang` through different +relays. Run each command on its publisher host, with its own source URL: + +```bash +# Publisher A +moq --connect https://ingest-a.example.com/anon \ + --connect-version moq-transport-17 --broadcast event.hang \ + import hls https://source-a.example.com/live/index.m3u8 + +# Publisher B +moq --connect https://ingest-b.example.com/anon \ + --connect-version moq-transport-17 --broadcast event.hang \ + import hls https://source-b.example.com/live/index.m3u8 +``` + +The relays must forward these announcements under the same path; publishing to +two disconnected relays does not make one a fallback for the other. The same +setup applies to other publishers, including browser capture and ingest +gateways. See [Production](/setup/prod) for TLS and authentication. + +At the same path, the newest epoch wins new requests, ahead of route cost. +Starting B after A therefore switches followers to B even while A is healthy. +If B's route goes away and A remains live and reachable, A wins again as a new +broadcast. This is not a fixed primary/standby priority: restarting either +publisher can make it the newest. + +The switch is announced as a `Restart`. Players that follow announcements +request the path again and begin the replacement broadcast. Existing +subscriptions stay on the old instance while it remains available, unless the +consumer chooses to follow the restart. + +For MPEG-TS output, opt into following replacements: + +```bash +moq --connect https://edge.example.com/anon \ + --connect-version moq-transport-17 --broadcast event.hang \ + export ts --stitch --linger 10s > event.ts +``` + +`--stitch` follows a replacement as a program switch: its catalog supplies a +new PMT and every PID flags the discontinuity. Starting a backup can cause +this switch too. `--linger 10s` bounds waiting after an end and waiting for a +replacement's first catalog; choose that window for your application. It does +not make the switch hitless. Without `--stitch`, the TS export stays on the +old instance while it is up, then fails on a replacement. Other stdout export +formats end with their broadcast; they do not follow a replacement. TS +passthrough also cannot stitch. See [CLI retention and latency](/bin/cli#retention-and-latency). + +A connection without epochs cannot prove continuity or apply the newest-epoch +rule: routing chooses among its epochless routes instead, and another route +winning is a restart. Do not rely on same-path continuity through such a hop. + +## Independent publishers on separate paths + +Publish each stream under its own name, such as `event/a.hang` and +`event/b.hang`. The end subscriber can subscribe to both and decide whether to +switch between them, combine them, or keep them separate. This is useful when +selection should depend on the media or operator policy rather than which +publisher started last. + +Each publisher produces its own stream with its own epoch. The relay routes +the two paths independently; it does not align their frames or stitch their +output. The subscriber owns any transition between them. + +## One publisher through several routes + +Keep a single publisher instance and distribute copies of its broadcast +through multiple relays or CDNs. All copies retain its path, epoch, group +numbers, and frame content. + +Connect the publisher's relay to multiple peers, then make those routes +reachable from the subscriber's relay. An application can also hold sessions +to [multiple CDNs](/concept/moq-lite#multiple-cdns) feeding one shared origin +and rank its links. Do not start a second importer to create the second copy. + +When the serving route fails or another route with the same epoch wins, the +subscription resumes from the first frame it lacks. This needs an explicit +matching epoch across every hop and a surviving route that can still serve +those frames within the broadcast's retention. It preserves broadcast +continuity, but a late recovery can still miss playback deadlines. Losing the +only publisher cannot be repaired by adding routes to it. + +## Hitless protection at the boundaries + +For ST 2022-7 protection downstream, form both RTP legs from one broadcast and +supply matching RTP packetization in the downstream system. Independent +publishers switching at one path are different broadcasts, so their output is +not a pair of identical protection legs. + +To protect the input to an importer hitlessly, merge duplicate legs before a +single importer, using upstream protection equipment. That importer then +publishes one broadcast, which can travel over several MoQ routes. diff --git a/quest/m1/README.md b/quest/m1/README.md index 0520ca02390..acdcff2a78e 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -81,7 +81,6 @@ lands before the cut. - [Announcement takeover](/quest/m1/announce-takeover.md) - a new independent route explicitly continues instances served through its predecessor, with shared Rust/JS semantics and lite-07 wire - [IETF takeover](/quest/m1/ietf-takeover.md) - carry the shared takeover contract through the negotiated IETF epoch extension - [Identity and recovery guide](/quest/m1/identity-guide.md) - explain pinned catalogs, lazy jobs, takeover, restart and version compatibility -- [Redundant ingest guide](/quest/m1/redundant-ingest-guide.md) - a setup page on redundant ingest over multiple routes, for any publisher - [Generated C bindings](/quest/m1/c/README.md) - C generated from moq-ffi ships as `moq-c` 0.8.0 and replaces the hand-written libmoq - [moq-ffi reports why a first dial fails](/quest/m1/ffi-first-dial-failure.md) - OBS names the reason while the first dial retries, as moq-c already can - [Scope track priority](/quest/m1/track-priority-scope.md) - priority orders one owner's streams, and a shared cluster session is fair across tenants diff --git a/quest/m1/redundant-ingest-guide.md b/quest/m1/redundant-ingest-guide.md deleted file mode 100644 index 19ea2cbaa8e..00000000000 --- a/quest/m1/redundant-ingest-guide.md +++ /dev/null @@ -1,37 +0,0 @@ -# [S] Redundant ingest guide - -## Goal - -A setup page, `doc/setup/redundancy.md` next to Production in the Setup -sidebar, tells an operator how to make ingest redundant with multiple -routes, for any publisher rather than TS alone. Nothing says today that two -publishers announcing one path is the supported way, so operators reach for -a shared epoch instead. - -It covers: - -- Two publishers of one path, each minting its own epoch: the relay serves - the newest, and losing it falls back to a still-live older one as a new - broadcast. -- What a switch looks like: players follow the announce `Restart`, - `moq export ts --stitch` follows as a program switch, and other exports - end. Starting a backup is a switch too. -- Each publisher on its own path instead, so the end subscriber subscribes - to both and decides whether to switch between, combine, or keep them - separate (maintainer: each publisher produces its own stream, and the - subscriber chooses). -- One publisher announced through several relays or CDNs under its own - epoch, which resumes without a seam when a path fails. -- Where hitless importer protection lives instead: 2022-7 legs of one - broadcast downstream, or merging legs before a single importer. - -## Plan - -Decided 2026-10-19 by the maintainer: a general page, not a TS section, -since multiple routes are the non-obvious part. Link it from the relay -cluster page and the `moq` publisher-runs section of `doc/bin/cli.md`. - -## Related - -- [Identity and recovery guide](/quest/m1/identity-guide.md) - restart and takeover semantics the page links to -- [Routing guide](/quest/m1/routing-guide.md) - route ranking and federation