Skip to content

fix(net): request compressed prefixes for mid-group application ranges - #6242

Merged
kixelated merged 9 commits into
mainfrom
quest/m1/compress/app-range-widen
Oct 23, 2026
Merged

kixelated merged 9 commits into
mainfrom
quest/m1/compress/app-range-widen

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A Rust application joining a compressed lite-07 track at frame N asks for N even when its local copy lacks frames 0..N-1. Its plain reader cannot inflate that tail.

Approach

Normalize outgoing subscription bounds in the requesting model before lite-07 sends them. Keep N only when a cached copy holds the complete committed prefix, and record that as a read of the copy so normal eviction takes it last; otherwise request frame 0. Original application preferences still filter reads, and publishers still serve exactly the requested bounds.

The check uses one cache lookup and at most two copy checks, with no payload copying, pin, or added retained state. Regression coverage exercises absent, partial, complete, tail-only and aborted prefixes, a held prefix outliving pressure eviction, plus a two-hop application range and live update.

Scope

The quest also asked for moq-transport draft-17+ and js/net. Neither lands here:

  • js/net: nothing to mirror. Its subscriptions carry group ranges only (Subscription.groups), so a JS SUBSCRIBE always starts at frame 0, and a compressed group that does not is already refused with TooFarBehind.

  • moq-transport: kept open as quest/m1/compress/ietf-range-widen.md. I ran the two-hop app-range test across lite-07 and moq-transport-17/18/22 on either hop. The failures come from three gaps in the IETF requester, and plain tracks hit them too:

    • A cold SUBSCRIBE only learns that the track is compressed from SUBSCRIBE_OK. If it widened every cold mid-group start, plain tracks would pay for the prefix too.
    • IETF-received groups bypass the reader's frame window (group.relayed()). An application on an IETF hop already reads frames below its start on plain tracks.
    • REQUEST_UPDATE carries no range. moq-transport ranges already owns that.

    Widening the IETF start before those are fixed would add code that no test can check end to end. Our own IETF publisher already served whole groups in the matrix.

The epic now lists the moq-transport quest in place of the deleted one.

Impact

  • Consumer: Rust applications using mid-group track::Subscription ranges on compressed lite-07 tracks.
  • Public API: unchanged. widen_compressed is pub(crate); its benchmark bridge moq_net::fuzz::widen_compressed is under the existing hidden fuzz surface.
  • Wire format: unchanged. Existing lite-07 start-frame bounds widen only when the requester lacks the prefix. moq-transport requests and JS are unchanged.
  • Server-side range semantics and recovery FETCH behavior are unchanged.

Alternatives

Always requesting frame 0 wastes a complete cached prefix. Widening at the server would change the content it was asked to serve. Pinning the held prefix against cache policy would close the eviction window entirely, but it exempts groups from eviction and its accounting. Declined below.

Decisions

Codex's first P1: if a held prefix is evicted before its tail lands, the tail starts at N and a plain reader cannot inflate it.

Decided by the maintainer: keep asking from N when the prefix is cached and do not pin it. Refresh its access time first, as a FETCH hit does. If it still disappears before the tail lands, that group fails with Lagged. Only a consumer slower than the cache's turnover can trigger that. widen_compressed documents it.

Codex's second P1: a warm tail is rejected as Duplicate. This is a receive-path gap that predates this PR and also affects plain tracks: a stream that stopped at the request's end bound is finish()ed. It is recorded as quest/m1/subscribe-ranges/bounded-tail.md.

Made without the maintainer (sub-agent, recommended option):

  1. Keep the moq-transport half as its own quest instead of a partial IETF widening (above). ✅
  2. Put the bounded-tail quest under the subscribe-ranges epic, since frame-precise ranges belong to that epic. ✅

Validation

  • The two original regressions failed before the source fix. The eviction test fails without the refresh.
  • After merging current main, all passed: just check for the Rust and quest stages, quest check --base origin/main, and the full cross-language just test interop --all. just check's flake stage stops at the Nix runner-image build (unshare: write failed /proc/self/uid_map), which is a host restriction. CI runs that stage.
  • Earlier on this branch: criterion over 1/64/1,024 readers × 1/64/1,024 cached groups, with held and missing prefixes, measured about 37–51 ns per request with no linear scan over the audience or the cache.

Follow-ups

(Written by GPT-6, updated by Claude Opus 5.5)

kixelated and others added 3 commits October 21, 2026 20:38
Co-Authored-By: GPT-6 <noreply@openai.com>
Keep application frame filters intact while lite-07 requests widen to frame zero only when the requesting copy lacks the complete committed prefix.

Co-Authored-By: GPT-6 <noreply@openai.com>
Observe the propagated end while a retained route still holds the earlier floor, then verify only the application-requested frames are read.

Co-Authored-By: GPT-6 <noreply@openai.com>

Copy link
Copy Markdown
Collaborator Author

Implemented the settled current-main Rust lite-07 scope. Missing/partial/released prefixes widen outgoing requests to frame 0; a complete retained prefix keeps N. Local application filtering and server/recovery FETCH semantics stay intact. No new public API, wire fields, dependencies or retained history.

All 2,870 net tests, the full cross-language interop matrix and the 18-cell request benchmark passed. The final just ci attempt passed Rust/quest/drill checks and stopped at the existing Nix runner-image unshare uid_map permission restriction. Remaining CI stages have not run, so this stays draft. There are no open implementation choices; JS and IETF integration conditions are recorded in the PR body. No new quests were started or created.

(Written by GPT-6)

@kixelated
kixelated marked this pull request as ready for review October 22, 2026 06:30
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 13 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 25fd5806-816e-4817-b6c6-d10ade0048a0
📥 Commits

Reviewing files that changed from the base of the PR and between 1c72130 and 9f9daa4.

📒 Files selected for processing: 11
  • quest/m1/compress/README.md
  • quest/m1/compress/app-range-widen.md
  • quest/m1/compress/ietf-range-widen.md
  • quest/m1/subscribe-ranges/README.md
  • quest/m1/subscribe-ranges/bounded-tail.md
  • rs/moq-net/benches/codec.rs
  • rs/moq-net/src/fuzz.rs
  • rs/moq-net/src/lite/subscriber.rs
  • rs/moq-net/src/model/group.rs
  • rs/moq-net/src/model/track.rs
  • rs/moq-net/tests/integration/compress.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-22T22:12:03.437109Z 9f9daa4 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7b124da5d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-net/src/model/track.rs Outdated
@kixelated
kixelated marked this pull request as draft October 22, 2026 06:38
kixelated and others added 3 commits October 21, 2026 23:51
A request kept at frame N relies on the cached prefix, so record it as a read and normal eviction takes it last. A prefix still lost before the tail lands fails that group with Lagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 22, 2026 06:53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a2a750476

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-net/src/model/track.rs
kixelated and others added 2 commits October 22, 2026 14:45
…-range-widen

# Conflicts:
#	quest/m1/compress/README.md
#	quest/m1/compress/app-range-widen.md
#	rs/moq-net/src/fuzz.rs
#	rs/moq-net/tests/integration/compress.rs
Lite-07 widening lands in this PR. The IETF requester needs a cold
SUBSCRIBE to learn compression first and its readers to apply the frame
window, so that half stays as its own quest; js/net subscriptions carry
group ranges only and need no mirror.

Record the Duplicate rejection of a tail after a copy stopped at a
request's end bound, found while answering the review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit c443b5b Oct 23, 2026
12 checks passed
@kixelated
kixelated deleted the quest/m1/compress/app-range-widen branch October 23, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant