[kixelated]

Problem

An archive, or anything else that wants every track, has to read a catalog and open a SUBSCRIBE per track. It also misses tracks the catalog does not name. Quest: SUBSCRIBE_ALL, which gates the lite-07 cut.

Approach

Model.

• origin::Consumer::broadcasts(prefix) yields every broadcast under a prefix as it appears.
• broadcast::Consumer::tracks() yields every track its producer creates.
• A broadcast published on the origin is its own answer.
• A route that a handler serves is asked, through the handler's Lister, for a Listing of the prefix. Every route one handler announces shares that Lister.
• A handler that never answers lists nothing, so a track served only on request (broadcast::Dynamic) is never created.

Wire (lite-07-wip only).

• Bidi stream 0x8 SUBSCRIBE_ALL: {Subscribe ID, Prefix, Priority, Order, Max Age}, then SUBSCRIBE_ALL_UPDATE.
• Typed replies:
  • BROADCAST_START (suffix, epoch) and TRACK_START (broadcast, name, TRACK_INFO) each assign the next ID implicitly.
  • BROADCAST_END means no more tracks; listed tracks end on their own.
  • TRACK_OK, TRACK_END and TRACK_RESET stand in for SUBSCRIBE_OK, SUBSCRIBE_END and the stream reset.
• Uni stream 0x2 TRACK_GROUP: {Subscribe ID, Track ID, Sequence, Frame Start}.
• A TRACK_GROUP stream may beat its TRACK_START, so the subscriber waits for the TRACK_START. The draft says so.
• No datagrams, and one subscription-cap slot per request.
• Auth checks subscribe permission on the prefix. A broadcast outside the grant is skipped, and a refused track gets TRACK_RESET(Unauthorized).
• Older versions reset the stream type, with no emulation.

Publisher.

• Each listed broadcast and track is its own kio::Tasks child, so a wake costs only the touched track.
• Each track runs the same TrackRun as a SUBSCRIBE, behind a Respond trait that maps its responses onto TRACK_*.
• Tracks share the request's priority through a new Priority lane.
• Once a listed track's start resolves, its demand is from that start on. A relay serving it therefore still sees another reader asking below it, instead of folding that floor into "oldest".
• SUBSCRIBE_START and TRACK_OK name past the groups the cursor gave up before resolving. A warm relay used to name its cache's start even when its budget skipped those groups, so it answered differently from a cold relay.

Subscriber / relay.

• One ListServe per (session, prefix) fills every listing of that prefix from a single upstream request.
• One copy per track. Each session keeps the copies it holds by path and epoch. A request and a listing of the same broadcast instance resolve through that copy, so a relay never holds two copies of a track that is both listed and subscribed.
  • A track that was listed first serves later requests from the listing's feed.
  • A track that was requested first gets the listing's feed attached. Its SUBSCRIBE is bounded at the feed's start, then cancelled once the groups below are in.
• Start floor. A listed track is served by the same serve loop as a requested one, with the SUBSCRIBE_ALL feed as its live cover.
  • A reader asking below TRACK_OK's group opens a SUBSCRIBE bounded at that group, cancelled once those groups are in.
  • At a relay, that SUBSCRIBE is just another reader of its copy, so the relay does the same upstream if its cache lacks the groups. A warm relay answers from cache.
• Preferences. Each listed track's demand is counted into the request: highest priority, widest max delay, and ascending order only when every track asks for it. SUBSCRIBE_ALL_UPDATE goes out when that aggregate changes.

Settled decisions

Settled by the maintainer for this iterate:

• Naming: keep broadcasts(prefix) and tracks(). They name what is yielded, and the SUBSCRIBE_TRACKS mapping reuses them.
• One copy: resolve a listed broadcast through the copy the session already holds, minting only when there is none.
• Floor: a floor below the listed feed opens a bounded SUBSCRIBE, as an ordinary joiner's would.
• Preferences: aggregate them the way ordinary subscriptions do, and send SUBSCRIBE_ALL_UPDATE when the aggregate changes.

Open decisions

The recommended option is implemented for each.

1. Epochless broadcasts get their own copy per request.
   • Recommended (implemented): share a copy only for a broadcast announced with an epoch, since without one nothing says two answers are the same instance (never stitch instances).
   • Alternative: share epochless copies by path too, which risks splicing a restarted publisher into the old copy.
2. The upstream aggregate also counts ordinary readers of an attached track.
   • Recommended (implemented): count them. They read the same copy that the feed fills.
   • Alternative: count only readers that came through a listing.
3. TRACK_START racing a requested track's TRACK_INFO. If the listing reaches a track whose request has not finished its TRACK_INFO yet, the feed is dropped, and that track keeps its own SUBSCRIBE (one duplicate upstream).
   • Recommended (implemented): accept that for now.
   • Alternative: queue the feed until the request's serve loop starts.
4. Overlapping prefixes from different SUBSCRIBE_ALLs. A track already fed by one request ignores a second request's feed, so the second request's groups for it go upstream too.
   • Recommended: a follow-up that dedupes at the relay by answering a narrower prefix from the wider request's feed.

Benchmark

benches/subscribe-all.rs sweeps listers (1, 8, 32) against broadcasts (1, 16, 64) through a relay, each against a plain-SUBSCRIBE control. The broadcasts now carry epochs, so the shared-copy registry is on the measured path.

Medians from one local run after merging main (the round benchmark is on par with its control, as before):

subscribe_all_new_broadcast	broadcasts=1	16	64
listers=1	90 µs	91 µs	98 µs
listers=8	405 µs	425 µs	455 µs
listers=32	1.63 ms	1.68 ms	2.00 ms

The control (subscribe_new_broadcast) grows by a similar 13 to 15% from 1 to 64 broadcasts at 8 and 32 listers, so there is no per-route slope. Before this PR's second commit, 64 broadcasts cost 813 µs at 1 lister.

subscribe_all_round stays ahead of subscribe_round at every point (for example, 143 ms against 190 ms at 32 listers and 64 broadcasts).

Impact

• Rust public API (additive):
  • origin::Consumer::broadcasts, with origin::Broadcasts (poll_next / next);
  • broadcast::Consumer::tracks, with broadcast::Tracks (poll_next / next).
  • Nothing breaks, so there is no MIGRATE entry.
• Wire: moq-lite-07-wip gains bidi stream 0x8 and uni stream 0x2, specified in draft-lcurley-moq-lite with a changelog bullet.
  • The draft now says a relay carries its readers' highest priority and max age with SUBSCRIBE_ALL_UPDATE.
  • It also says a subscriber wanting groups below TRACK_OK's group SUBSCRIBEs for them.
  • SUBSCRIBE_START's group now matches its definition ("where delivery starts") when a warm cache skips stale groups. That changes behavior, not encoding, on every lite version with a track stream.
  • Published versions keep their encoding.
• JS: none. The JS half is split into SUBSCRIBE_ALL in js/net, which also gates the lite-07 cut.

Alternatives

• Per-route listings with session-side scoping: rejected after benchmarking, since a cursor still reads one listing per route.
• A track-listing stream plus a SUBSCRIBE per track, a name filter, and emulation on older versions: rejected in the quest plan.
• Starting listed tracks at the live edge, so a relay's aggregate keeps an absolute floor visible: rejected. The draft serves a listed track as a SUBSCRIBE with no bounds, which resolves through max age.

Follow-ups

• SUBSCRIBE_ALL in js/net, split out here.
• An ordinary relay has the same hidden floor this fixes for listed tracks. A reader asking for an absolute floor below a copy that an "oldest" reader already resolved is never fetched upstream, because the aggregate folds the floor into "oldest". The fix is to count a resolved subscription's demand from its start, as listed tracks now do, but that changes every relay's upstream SUBSCRIBE_UPDATE, so it gets its own PR.
• Open decisions 3 and 4.
• A grant that grows mid-request does not re-list skipped broadcasts. A revoked listed track only ends through its gate.
• broadcast::Tracks skips a track that closed before the cursor reached it.

Checks

just check, just drafts check, and just test interop --all pass locally after merging main. Each new test was checked to fail with its fix reverted.

🤖 Generated with Claude Code

[kixelated]

Quest outcome: draft, not complete. The Rust half of SUBSCRIBE_ALL works end to end and is tested on mock time, directly and through a relay. Covered: tracks now and later, a broadcast appearing and ending, a new epoch at one path, dynamic-only tracks never created, two downstreams sharing one upstream request, and lite-06 listing nothing. just check, just drafts check, and just test interop --all pass.

Left open, each with a recommendation in the description:

• Carrying a resumed track's start floor upstream through a relay (untested, not implemented).
• Sharing listed tracks with ordinary SUBSCRIBE sources on a relay.
• Propagating downstream preferences upstream.
• Naming: broadcasts / tracks.

The benchmark caught an O(broadcasts) cost per new broadcast (813 µs at 64 broadcasts), now fixed by one lister per session (93 µs). The JS half is split into quest/m1/archive/subscribe-all-js.md.

(Written by Claude Opus 5.5)

[kixelated]

Iterate pushed (da26dca40). It fixes both gaps the previous draft left open.

One copy. A session now keeps its copies by path and epoch. A request and a listing of the same broadcast instance share one broadcast::Producer:

• A track listed first serves later requests from the feed.
• A track requested first gets the listing's feed attached. Its SUBSCRIBE is bounded at the feed's start and cancelled once filled.

Tests: a_listed_track_serves_requests_too (no SUBSCRIBE on either hop) and a_subscribed_track_hands_over_to_its_listing (group 2 goes upstream once).

Floor through a relay. A reader below TRACK_OK's group opens a SUBSCRIBE bounded at that group, at every hop. Two root causes surfaced while getting the relay case to pass, and both are fixed in TrackRun:

• A listed track's demand stayed "oldest" after its start resolved, so a relay folded a downstream absolute floor into it and never asked upstream. Its demand is now from its resolved start on.
• SUBSCRIBE_START and TRACK_OK named the cache's start even when the cursor skipped those groups as stale. A warm relay therefore claimed groups it never sent, and answered differently from a cold one. The start now names past the groups the cursor gave up (the_start_names_past_the_stale_groups_it_skips).

Tests: resumes_a_listed_track_mid_group directly, through a cold relay (one upstream SUBSCRIBE), and through a warm relay (none).

Preferences. Listed tracks' demand is aggregated: highest priority, widest max delay, and ascending only if every track asks for it. SUBSCRIBE_ALL_UPDATE carries the aggregate (carries_reader_preferences_*, direct and through a relay).

The draft gained two sentences on the relay aggregate and on SUBSCRIBEing below TRACK_OK. The bench now announces with epochs, and subscribe_all_new_broadcast stays flat in broadcasts. Each new test fails with its fix reverted. just check, just drafts check and just test interop --all pass.

Open decisions are in the description, each with the recommendation implemented: epochless broadcasts are not shared, ordinary readers count toward the aggregate, the TRACK_INFO race drops the feed, and overlapping prefixes are left for a follow-up.

One follow-up is worth its own PR. An ordinary relay has the same hidden floor: an absolute floor below a copy that an "oldest" reader already resolved is never fetched upstream. I reproduced it with a throwaway test.

(Written by Claude Opus 5.5)

[kixelated]

Closing unmerged per maintainer decision. Listing every track moves from a push model (SUBSCRIBE_ALL) to a pull model: a per-broadcast lite TRACKS listing, then a SUBSCRIBE per track. So relays never merge pushed feeds into pulled copies. Replanned in #5985.

Pieces to port by hand from this branch:

• broadcast::Consumer::tracks() (ba2db36, f074d09): the starting point for TRACKS.
• The stale-start fix and the_start_names_past_the_stale_groups_it_skips (f074d09): quest/m1/subscribe-ok-start.md in quest(m1): replace SUBSCRIBE_ALL with a TRACKS listing #5985.
• listed_demand, generalized: quest/m1/resolved-demand.md in quest(m1): plan the hidden floor below a resolved relay copy #5986.
• Possibly the fan-out benchmark harness.

(Written by Claude Opus 5.5)