diff --git a/quest/m1/README.md b/quest/m1/README.md index 3c6a0aa732..9156e54ef8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -104,6 +104,8 @@ lands before the cut. - [Timed-out requests clean up](/quest/m1/legacy-request-timeout-cleanup.md) - on drafts 14-16, Rust and JS send PUBLISH_NAMESPACE_DONE or UNSUBSCRIBE_NAMESPACE for a timed-out request before its local reset - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails +- [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - a warm relay serves a reader with a wider max age the cached groups a cold relay would, with no wire change +- [Drop set](/quest/m1/drop-set.md) - each cursor judges missing groups against its own budget, so a narrow reader drops promptly and a widening re-offers groups already reported dropped on lite-07 - [SUBSCRIBE_OK names where delivery starts](/quest/m1/subscribe-ok-start.md) - a warm relay names the same start group as a cold one instead of groups it skipped as stale - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track keeps its declared end across copies, and its `closed` and `largest()` settle on a clean end diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md new file mode 100644 index 0000000000..5c14778d89 --- /dev/null +++ b/quest/m1/drop-set.md @@ -0,0 +1,225 @@ +# [L] Drop set + +## Goal + +Every cursor judges a missing group against its own budget. Today a dropped +group is final for every reader of a track: a narrow reader can wait +indefinitely for an upstream OLD naming a group the relay never cached, +and a wider reader never gets a group an upstream already reported stale. +The track state instead keeps the dropped and expired runs with their +reason and budget, so a narrow reader gets a prompt drop, a wide reader +waits, and a widening re-offers what its new budget covers. On lite-07 this +restores "growing means I want it": a group reported dropped under a +narrower budget is served after a widening, as a cold relay would. + +## Plan + +Decided by the maintainer on 2026-10-19, splitting this from +[A widened max age re-offers cached groups](/quest/m1/resolved-demand.md), +which keeps drops final for now. + +- **One run-length set per track.** The track state keeps the dropped and + expired group runs within the subscription range. Each run carries its + reason (stale, cut, never produced, or evicted or expired where no copy + remains: received as such from the upstream, or at an origin with no + upstream) and, for a stale run, the + budget it was judged under. A group that arrives removes its sequence; a + drop never removes a group that already arrived. A relay's own eviction + is never a run: the group is unknown again, and any cursor that wants it, + new, widened, or already owed it, fills it from upstream by range, the + existing miss fill. That needs no new state, since the cursor's record + already holds the span as unknown, a run at a time. +- **Each cursor judges for itself, a run at a time.** A cursor walks the + set's runs in its range, never their sequences, and settles each whole run + in O(1): + - A run whose reason does not depend on the budget (a cut, a group never + produced, an upstream or origin eviction or expiry) is dropped for every reader + as one range, as today. + - A received stale run is dropped promptly as one range for a reader + whose budget is no wider than the one it was judged under and whose + range covers this relay's aggregate range for the track, which the + upstream judged under. A verdict depends on the range as well as the + budget, since `drifted` caps the edge to the reader's range + (`Edge::cap`), so a narrower reader treats the run as unknown and fills + it by range, where the upstream judges it under that narrower range. + The cost is an extra range request, only for narrower readers, when a + stale drop arrives. + - A missing run this relay judges itself is dropped for a reader under + whose own budget and range `TrackState::drifted` convicts it. `drifted` + judges a missing sequence by its successor here, and every sequence in + a missing run shares one successor, so one call settles the run. + - A wider reader skips the run and waits for a group, a re-offer, or its + own verdict, each of which splits the run at most once. + - An unknown span (a gap between runs, or an evicted run) is judged the + same way, as one unit bounded by its neighbours: by one `drifted` call + against its shared successor, or by one range miss fill upstream. + + A hostile upstream's huge run costs one Charge and O(1) per cursor, and + splitting it costs per-run Charges the pool bounds, so CPU tracks pool + bytes, never span. + + This fixes the gap Codex found on + [moq-dev/moq#5986](https://github.com/moq-dev/moq/pull/5986): today + `TrackRun::poll_forward_drop` emits only recorded drops, cuts, or stale + cached groups. A sequence the relay never cached is not judged until the + track finishes. +- **It extends resolved-demand's per-cursor disposition record,** which + stays next to the per-track set: a group that arrives leaves the shared + set, so only the cursor's record remembers that it reported cached G + stale. A reported run now carries its reason and, for a stale one, the + budget it was judged under. On lite-07 a reported stale run is no longer + final: like a skipped one, it is re-offered when a widening covers it. +- **A widening re-offers the dropped runs now inside its budget,** through + the re-offer path the resolved-demand quest adds, in the subscription's + order. Upstream, the relay's own widening asks for them again. +- **Only a widening reverses a stale verdict,** as resolved-demand + decides, local or received alike: no run tracks the group that convicted + it, and an abort or eviction of that group leaves the run standing (the + open question in resolved-demand). A received run is also reversed by + the upstream re-offering its group after a widening, since a later group + wins over its drop. Cuts and never-produced runs are unaffected. +- **A stale drop names its budget.** A SUBSCRIBE_UPDATE has no + acknowledgement, so after a widening from B1 to B2, an in-flight B1 drop + and the B2 re-drop look identical, and no guess settles a B2 reader. On + lite-07 SUBSCRIBE_DROP carries `Max Age`: the budget a staleness verdict + was judged under, encoded like `Publisher Max Age` (milliseconds plus one), + with `0` for a drop that does not depend on the budget: a cut, a group + never produced, or a group the sender evicted or expired with no copy it + could fetch (an origin, or a relay forwarding its upstream's eviction + drop), none of which depends on the reader's budget. A relay never sends + an eviction drop for its own eviction; it fills the group by range + instead. A widening never reverses a `0` drop. The relay records each run with that + exact budget. A B1 + drop then settles only readers no wider than B1, and the B2 re-drop + settles a B2 reader. The cost is one varint per drop, a few bytes on an + event that is already per group, which is worth an unambiguous term. + Rejected: an implicit generation counted by both sides per + SUBSCRIBE_UPDATE. It needs a generation-to-budget history at the relay + and agreement on every update counted, while the budget itself is + self-describing, order-free, and unaffected by updates that change + nothing. +- **lite-07 only for re-offering reported groups.** Released lite-05 and + lite-06 peers treat a flushed SUBSCRIBE_DROP as final, so a drop received + over those versions is final for every reader, and a lite-07 hop never + sends a group onto a lite-05 or lite-06 hop after reporting it. On + lite-07, a widening obliges the publisher to serve the newly covered groups + it holds and never delivered on this subscription, including ones it + reported dropped, and to drop again, under the new terms, the ones it + cannot serve. SUBSCRIBE_UPDATE stays incremental and never resets, since a + reset races in-flight, out-of-order data. A shrink sends no drops. +- **Draft wording.** `drafts/draft-lcurley-moq-lite.md` states, for lite-07's + restored SUBSCRIBE_DROP, that a drop holds only under the terms in effect + when it was sent, that a widening re-offers held groups and re-drops the + rest, and that a later group, when sent, wins over its drop. Whichever of this and + [SUBSCRIBE_DROP accounts for every group](/quest/m1/subscribe-drop.md) + lands second writes it, including the `Max Age` field above, which is + free to add because lite-07 is unpublished. + +Code it replaces, in Rust and `js/net` alike: + +- `TrackState::drop_groups`, the `drops` range set and `DropCursor` in + `rs/moq-net/src/model/track.rs`, which record a whole range without its + reason or budget, even over groups already cached. +- In `js/net/src/track.ts`: `#drop`, `#undrop`, the `drops` signal, and + the wire cursor's `#pending` and `#dropsSeen`, whose `#tryRecvGroup` + closes any group already reported. + +Untrusted input: a hostile upstream can fragment the set by alternating +drops and arrivals, and an open live range is no bound, since its groups are +evicted while their runs would stay. Bill each run to the cache's existing +byte accounting like a group: the Rust `Pool` through a `Charge` at a fixed +per-run cost. Under pressure the pool evicts the oldest runs as it evicts +groups, and a run older than the cache's retention window is forgotten, so +the set never outgrows what the pool would hold, with no new constant. An +evicted run's sequences become unknown again. `js/net` has no byte pool and +gets no new cap: its shared and per-cursor runs expire with the same idle +window as the groups they sit between, and expiry applies the same +fold-or-`TooFarBehind` rule as a Rust eviction. A run sits between cached +sequences, so there are at most about one per cached sequence (alternating +drop and arrival), and JS run state grows no faster than the JS cache +itself. The JS byte bound for both is +[JS unbounded groups](/quest/m2/unbounded-groups/js.md). + +The per-cursor record keeps resolved-demand's pool bound and floor rule, +with "final" narrowed: only a delivered run or a budget-independent drop (a +cut, a group never produced, an upstream or origin eviction or expiry, or any drop +reported over lite-05 or lite-06) +folds into the floor. A stale run reported on lite-07 within reach of the +max age bound is budget-qualified, so it is treated like an unknown hole: +the floor never rises over it, and if the pool must evict it or a run above +it, the cursor fails with `TooFarBehind`. Folding it would forgo the +re-offer this plan promises after a widening. A later group wins over its +drop only while the cursor's record still holds that drop: a drop folded +into the floor under pool pressure is final for that cursor, so its group +arriving late is not delivered there. + +A cursor judges an unknown missing sequence (never a stale run its record +still holds as reported, which only a widening settles) as it would a +group never received: locally against its own budget (`TrackState::drifted`), +and otherwise as a cache miss, which the relay fills upstream by range +([Subscribe ranges](/quest/m1/subscribe-ranges/README.md)), so no reader +waits indefinitely on forgotten state. Extend `track_subscriber_churn` in +`rs/moq-net/benches/track.rs` with a fragmented set up to pool capacity, +swept over subscriber count, shared runs, runs per cursor record, and run +span, so enumeration cost tracks the touched runs, never the set, the +record, or a run's span. + +Test on mock time through a relay: +- On a live track, a reader no wider than the upstream budget gets a prompt + drop for an upstream OLD naming a group the relay never cached. +- On lite-07, a widened reader gets that group re-offered. A drop already + flushed before the widening SUBSCRIBE_UPDATE was received, so the reader + then gets the re-offer and no further final drop; a drop still in flight + does not settle the widened reader. Check what each reader receives, not + only tail accounting. +- On lite-07, after a widening from B1 to B2 for a group the upstream no + longer holds, an in-flight drop tagged B1 settles only the B1 reader, and + the re-drop tagged B2 settles the B2 reader. +- A group arriving after its drop, while the cursor's record still holds + the drop, is served, and a drop arriving after its group leaves it + served. +- Through a lite-06 hop, a reported group stays dropped for every reader, + with no session error. +- A shrink sends no drops. +- Rust and `js/net` each encode and decode an origin's eviction drop and + retention-expiry drop with `Max Age` `0`, across languages; a relay + forwards them unchanged, and a widening re-offers neither. +- A group a relay evicted while the upstream still holds it is never + reported dropped: a new reader, a widened reader, and a reader already + owed it each fill it by range and receive it. +- A reader whose range is narrower than the relay's aggregate range is not + settled by a received stale drop: it fills G by range and gets G or a + drop judged under its own range, while a reader covering the aggregate + range is settled at once. +- On lite-07, a cursor that delivered F, reported its neighbour G stale + while G stayed cached, and delivered H, gets only G after a widening that + covers all three: G is re-offered and F and H are never resent. +- Under pool pressure, a cursor's delivered runs and budget-independent + drops fold into the floor, while evicting a lite-07 stale-reported run + within reach, or a run above it, ends the cursor with `TooFarBehind` + instead of losing the re-offer. Through a lite-06 hop the reported run + folds. +- A flood of fragmented runs stays within the pool in Rust, evicting the + oldest shared runs first, and a sequence whose shared run was evicted is + judged locally or filled by range, never waited on indefinitely. In + `js/net` the same flood holds at most one run more than its cached + sequences, and the runs expire with the idle window. +- A single run spanning billions of sequences is settled by each cursor in + one step, whether dropped, skipped by a wider reader, or evicted and + judged again. + +Mirror each case in `js/net`. + +Public API: none. Wire: lite-07 SUBSCRIBE_DROP (unpublished) gains `Max Age` +and the widening and ordering rules above; published versions are +unchanged. + +## Required + +- [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - the re-offer path this extends to reported groups +- [SUBSCRIBE_DROP accounts for every group](/quest/m1/subscribe-drop.md) - restores the lite-07 drop whose wire and behavior this extends + +## Related + +- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - fills the cache miss an evicted run leaves +- [JS unbounded groups](/quest/m2/unbounded-groups/js.md) - the JS byte bound for groups and the runs between them; not required, since JS runs never outgrow the JS cache diff --git a/quest/m1/resolved-demand.md b/quest/m1/resolved-demand.md new file mode 100644 index 0000000000..a00ee625d3 --- /dev/null +++ b/quest/m1/resolved-demand.md @@ -0,0 +1,179 @@ +# [M] A widened max age re-offers cached groups + +## Goal + +A warm relay serves a new reader the same groups a cold one would. Today it +does not: once a reader asking for the oldest group (`Range::oldest`) holds +a copy, its upstream subscription resolved where that reader's max age put +it, say group 10. A newcomer whose wider max age covers groups 5 through 9, +with a floor at 5 or with no floor at all, gets nothing below 10. The relay +does send the wider max age upstream in SUBSCRIBE_UPDATE, but the upstream +cursor never re-offers the cached groups it already skipped as stale. The +reader silently starts later than it asked, and only through a warm relay. +Groups the upstream already reported dropped stay out of scope here (see +[Drop set](/quest/m1/drop-set.md)). + +An archive resuming at its last stored frame through a relay that a viewer +already warmed hits this, as does any reader with a wider max age than the +copy's first one. + +## Plan + +Found while building SUBSCRIBE_ALL +([moq-dev/moq#5931](https://github.com/moq-dev/moq/pull/5931)), which +patched it for listed tracks only by sending the resolved start as an +absolute floor (`listed_demand` in `f074d09e7`). + +Decided by the maintainer on 2026-10-19: re-offer when the max age widens, +with no wire change. + +- **The aggregate is the union of subscriber ranges**, as + [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) already decided + ("Merging subscribers is the union of their ranges"). On a version with a + single floor, the union of a no-floor reader and a floor F is no floor, so + the relay sends no floor upstream and filters F locally, which the Rust and + JS aggregates already do. A floor on the wire + is a filter (the publisher "MUST NOT deliver an older one"), so sending one + would drop a late group the no-floor reader is still owed. +- **Why max age, not the floor.** An "oldest" reader resolves at S only + because the groups below S were stale under its max age, or missing. Age is + measured against the newest group, so it only grows. A newcomer is owed a + group below S only if its max age is wider, and a wider max age already + changes the aggregate (the max of every reader's) and goes upstream in + SUBSCRIBE_UPDATE. The missing piece is the publisher acting on it. +- **Today a widened max age does not re-offer.** Confirmed in + `rs/moq-net/src/model/track.rs`: the ordered cursor (`poll_next_group`) + holds a stale conviction in `seek_pending` and walks it back if the budget + widens, but only until it delivers a later group, which commits the skip + and moves `next_sequence` past it for good. The arrival-order path drops + the gaps past budget from `Delivered` (`forget_past_budget`) and never + revisits them. Only an uncommitted conviction can come back. +- **The fix.** When a subscriber's max age widens, re-offer the cached groups + at or above its floor that its cursor gave up as stale and that the new + budget covers, in the subscription's order: newest first when descending + (the default, so congestion sheds backlog rather than the live edge), and + lowest first only when ascending. This extends the re-offer + [moq-dev/moq#5929](https://github.com/moq-dev/moq/pull/5929) adds for a + widened range (a group outside the ranges is parked and re-offered when they + widen); land after it and reuse that path rather than a second one. Only + groups this cursor never delivered come back, keeping #5929's rule that + nothing already served goes out twice; a group already delivered and since + evicted downstream is a cache miss, which the relay fills upstream by range + per [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) (a miss below + the cache becomes a range request), not a re-offer. +- **Only a widening reverses a stale verdict,** a bigger budget or a wider + range, decided by the maintainer on 2026-10-19 ("Narrow, then merge"). + Nothing at the same terms re-offers a group the cursor gave up, so no + cursor tracks what convicted it. +- **Open question: warm/cold equivalence after an edge loss is not + promised.** When the newer group that made G stale is aborted or evicted, + a cold cursor may serve G while a warm one stays past it. Deferred + because watching for it needs more than this plan carries: a verdict + depends on two groups (`drifted` uses G's successor for `reach` and a + possibly later presentation edge in `live_above`, and losing either can + free G), a range may end mid-group so an edge needs a frame position, and + a run skipped at the max age bound would no longer be final. Likewise, + a stale drop in flight across an aggregate range change (lite-07, see + [Drop set](/quest/m1/drop-set.md)) may settle a reader whose own range + would not convict, since SUBSCRIBE_UPDATE has no acknowledgement and the + drop names only its `Max Age`; deferred for the same reason. A group + arriving after its drop was folded into a cursor's floor under pool + pressure is not delivered to that cursor ([Drop set](/quest/m1/drop-set.md)). +- **Each cursor keeps a disposition record,** as the maintainer first framed + it ("keep what was dropped within each cursor and within the track + state"). It is what makes a re-offer safe: the cursor must tell a group it + gave up as stale (re-offer it) from one it delivered (never resend it). + The record is one run-length map per cursor, each run marked delivered, + skipped as stale (with the budget and range it was judged under), or + reported in SUBSCRIBE_DROP (final for now, below). It replaces today's `Delivered` + (`rs/moq-net/src/model/delivered.rs`), whose floor (`forget_below`) folds + stale gaps in with delivered groups, and absorbs `TrackRun::served` in + `rs/moq-net/src/lite/publisher.rs` and the JS wire cursor's `#reported` in + `js/net/src/track.ts`. Delivery is recorded, not inferred from the cursor + position: an arrival-order or descending cursor delivers out of sequence + order, and #5929 re-offers below the position. The record only answers + "re-offer or not"; the serving and re-offer path stays #5929's. +- **The record is bounded by the pool, never by guessing.** Each run is + billed to the cache's `Pool` through a `Charge` at a fixed per-run cost + and is settled a run at a time, so a cursor's + state and CPU track its runs, never their span. The floor only rises over + a final prefix: the runs from the floor up to the lowest run that is not + final. Final means delivered, reported, or skipped as stale at the + track's max age bound: only a widening reverses a verdict, and no + widening reaches past the bound since every budget is clamped to it. A skipped stale run within reach is not final, and neither + is an unknown hole: a descending or arrival-order cursor can deliver H + while a lower G is still unknown, and folding H would silently treat G as + handled. When the pool evicts one of a cursor's runs: + - A run inside the final prefix folds into the floor, as `forget_below` + does today, and nothing is lost. + - Any other run cannot go without guessing, so the cursor fails: the + subscription ends with `TooFarBehind` (stream code 0x5, already meaning + a reader that fell too far behind and lost content), the subscriber + re-subscribes, and the cursor's whole record is released with it. + + `js/net` has no byte pool and gets no new cap: a cursor's runs expire + with the same idle window as the groups they sit between, under the same + fold-or-`TooFarBehind` rule, so there are at most about one per cached + sequence and the record grows no faster than the JS cache. The JS byte + bound is [JS unbounded groups](/quest/m2/unbounded-groups/js.md). +- **Drops are final for now**, decided by the maintainer on 2026-10-19. A + widening re-offers only groups this cursor never reported in + SUBSCRIBE_DROP; a reported group stays dropped, as today. That leaves a + known warm/cold gap: on lite-05 and lite-06 the publisher reports a stale + skip as soon as it gives it up, so a group already reported is not + re-offered, while a cold relay would serve it. Closing that gap needs + per-cursor judgement of missing sequences, planned in + [Drop set](/quest/m1/drop-set.md). +- **Cost, accepted.** Groups below a newcomer's floor that the wider budget + covers also come upstream, and the relay drops them locally. Bounded by the + max age and the cache. +- Rejected: sending min(S, floors) as the wire floor, which filters late + groups for the no-floor reader; a second upstream SUBSCRIBE bounded to the + gap, which costs a slot, puts two pulls into one copy, and misses the + no-floor case; and a new wire hint. + +Watch for: + +- **Every relay changes.** A widening now costs the publisher a scan of its + cached groups below the cursor. Keep it proportional to the groups + re-offered, not the cache, and make sure a reader joining with the same max + age causes no scan and no SUBSCRIBE_UPDATE. Extend `track_subscriber_churn` + in `rs/moq-net/benches/track.rs` with widening readers, swept over + subscriber count, cached groups, and runs per cursor record. +- **`js/net` has the same bug and is in scope.** `combineSubscriptions` in + `js/net/src/track.ts` already sends no floor when any subscriber has none, + and its publisher needs the same re-offer on a widened max age. Land a JS + regression test beside the Rust one. + +Test on mock time through a relay: a newcomer with a wider max age, with a +floor below the copy's start and with none, gets the groups upstream still +caches (fails on `main`); the same through a cold relay gives the same +answer; a late group below the copy's start still reaches the no-floor +reader; a reader joining with the same max age sends no SUBSCRIBE_UPDATE; and +a re-offered group is never delivered twice to one subscriber; and a widen, +shrink, evict, widen sequence at the relay gets the evicted groups back +through the range miss fill, not a re-offer; and a stale group already reported in +SUBSCRIBE_DROP stays dropped after a widening, never served after its drop; +and a cursor that delivered F, skipped its neighbour G as stale while G +stayed cached, and delivered H, gets only G after a widening that covers +all three, through a bigger budget or a wider range, never F or H again; +and a run skipped at the max age bound folds into the floor; and a +cursor whose final prefix is evicted under pool pressure folds it and +resends nothing; and a descending cursor that delivered H while a lower G +is still unknown or skipped within reach, under pool pressure that evicts +H's run, ends with `TooFarBehind` instead of skipping G. Cover the +warm-relay case in `js/net` too. + +Public API: none. Wire: no encoding change. A publisher now re-offers +cached groups it never reported dropped when a SUBSCRIBE_UPDATE widens the +max age or the range. + +## Required + +- [Dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) - #5929 lands the re-offer and serving path and the range miss fill this reuses + +## Related + +- [Archive producer](/quest/m1/archive/producer.md) - its resume floor crosses relays that viewers may have warmed +- [Drop set](/quest/m1/drop-set.md) - closes the gap this leaves for groups already reported dropped +- [JS unbounded groups](/quest/m2/unbounded-groups/js.md) - the JS byte bound for groups and the cursor runs between them