Repository navigation
quest(m1): plan PSI pacing within the system buffers #5304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,60 +1,48 @@ | ||
| # [L] TS export paces PSI within the system buffers | ||
| # [M] TS export keeps PSI within the system buffers | ||
|
|
||
| ## Goal | ||
|
|
||
| The export's program tables never overflow the program's system buffers at any | ||
| mux rate: TBsys (512 bytes drained at Rxsys = 1 Mb/s) and Bsys (1,536 bytes | ||
| drained at Rsys = max(80 kb/s, mux rate / 500)), per ISO 13818-1 2.4.2.3. | ||
| Identical runs that the same flush would lay back to back go out once. Distinct | ||
| tune-in runs (each send-ahead unit leads with its own tables) and changed tables | ||
| (a new PMT version) are paced, never dropped, and changed tables go out in | ||
| order. `compliance.py` grades both buffers too, so every TS Interop arm catches | ||
| a regression. | ||
| Padded `moq export ts` output keeps the program's system buffers within | ||
| ISO 13818-1 2.4.2.3 at every mux rate: TBsys (512 B at Rxsys = 1 Mb/s, shared | ||
| by PIDs 0-3 and the PMT PID) and Bsys (1,536 B drained at | ||
| Rsys = max(80 kb/s, mux rate / 500)). Distinct tune-in runs and changed table | ||
| versions are paced, never dropped, and changed versions go out in order. | ||
| `compliance.py` grades both buffers, so every TS Interop arm catches a | ||
| regression. Unpadded output has no mux rate and makes no system-buffer claim; | ||
| say so in the profile report. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by Codex on #5207 (`schedule.rs`, repeated PSI groups). It exists on | ||
| `main` before #5207 too. Keeping each table once per slot was tried and broke | ||
| `aac_program_config_follows_each_table`, because send-ahead units need their own | ||
| tables. | ||
| Decided (2026-10-13), from #5304: adjacent-copy suppression is dropped. It | ||
| would only reach unpadded output, which makes no buffer claim, and #5306's | ||
| padded planner already passes #5304's 3/4/8-PID regression with every tune-in | ||
| copy kept. | ||
|
|
||
| The PAT and the PMT share one TBsys: it takes PIDs 0 to 3 and the selected | ||
| program's PMT PID (ISO 13818-1 2.4.2.3). One PAT and PMT pair (376 B) fits. At | ||
| 25 Mb/s, two pairs back to back deliver 752 B in about 180 µs, of which about | ||
| 23 B drains, peaking near 729 B; three peak near 1090 B. At high rates two | ||
| distinct pairs fit only about 1.9 ms apart; pacing derives the spacing from | ||
| simulated occupancy at the actual mux rate. | ||
|
|
||
| - Start from the author's narrower diff on #5207 | ||
| (https://github.com/moq-dev/moq/pull/5207#discussion_r4239051134): drop a run | ||
| only when the same flush would lay it right after an identical run, and pad | ||
| the dropped packets back with nulls in a padded slot. `Slot::nulls` becomes | ||
| `Option<usize>` so an unpadded slot gains none. A slot still opens with its | ||
| clock packet and keeps its length, so no PCR moves. #5304 implements this step | ||
| and stays a draft until pacing lands. | ||
| - Then pace what suppression leaves: a distinct tune-in run or a changed version | ||
| goes out only once TBsys and Bsys have drained enough to take it, within the | ||
| slot budget and the units' decode deadlines. Bsys caps sustained PSI at Rsys, | ||
| 80 kb/s below 40 Mb/s. #5304 found equal-weight aligned units on four and | ||
| eight video PIDs still overflow after suppression. | ||
| - #5220 (multi-packet PMT) adds PMT pacing at the system-buffer rate, since a | ||
| PMT section of up to 1,024 B spans several packets and can exceed 512 B alone. | ||
| Build on that pacing rather than adding a second one, or own it here if #5220 | ||
| lands without it. Pace every PID that enters TBsys as one lane: the PAT, the | ||
| PMT, and any `mpegts.si` passthrough on PIDs 1 to 3, which the scheduler gives | ||
| its own `Buffer::SYSTEM` lane today. | ||
| - Test against buffer fill with the T-STD model, not a packet count: a unit test | ||
| runs laid slots' PIDs 0 to 3 and PMT packets through one shared TBsys and | ||
| Bsys, with keyframes aligned at 25 Mb/s on two and three video PIDs (both fail | ||
| without the fix) and on one (passes either way). Add a changed-version | ||
| regression, #5304's distinct-run cases, a PID 1 to 3 passthrough packet beside | ||
| a pair (fails while it keeps its own lane), and a sustained burst that fits | ||
| TBsys but overflows Bsys. The AAC tune-in test still passes. | ||
| - `test/ts/compliance.py` gains a hard-severity system-buffer check: for each | ||
| program, one TBsys fed by PIDs 0 to 3 and that program's PMT PID, draining | ||
| into one Bsys, both checked for overflow, and TBsys for emptying at least | ||
| once a second (2.4.2.6), with a negative control for each. Its `tstd` check | ||
| models only elementary streams today. Other SI (NIT, SDT, EIT) and SCTE-35 | ||
| never enter TBsys. Every existing TS arm still passes. | ||
| - Build on #5306's padded planner, which paces TBsys at absolute positions | ||
| shared by tables, nulls and media. t0ms found on #5304 (a real DVB recording) | ||
| that table and padding positions laid ahead of a capped video lane, outside | ||
| its interleave count, compress the video and overrun its TB. Any further | ||
| pacing must stay inside that planner's positions, never a pre-pass that moves | ||
| or pads tables outside it. Confirm with t0ms's replay against the planner. | ||
| - Add Bsys beside TBsys in the planner. Settle first, from 2.4.2.3's text, what | ||
| enters Bsys (whole 188 B packets, payloads with stuffing, or section bytes): | ||
| for single-packet tables it decides whether Bsys can overflow at all. A probe on #5306 (`a5640abee`) counting | ||
| full payloads peaked near 1.35 KB at 4 aligned video PIDs and 2.3-2.7 KB | ||
| at 8. | ||
| - Open: when Bsys pacing would push a table past its unit's decode deadline, | ||
| fail loud like any missed deadline, or allow dropping a copy identical to one | ||
| still queued. Recommend failing loud unless a real source hits it. | ||
|
Comment on lines
+32
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When Bsys pacing cannot place a table before its associated unit, dropping an identical queued copy contradicts the goal that distinct tune-in runs are never dropped and reintroduces the suppression behavior abandoned above. Each repeated PAT/PMT establishes a separate mid-stream tune-in point, so retaining one copy does not preserve the later unit's entry point or the stated AAC tune-in invariant. Resolve this case by failing loudly as recommended rather than leaving the dropping option available. AGENTS.md reference: AGENTS.md:L17-L17 Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Declining for now: the maintainer left this open on purpose, with failing loud as the recommendation. The alternative is narrower than the abandoned suppression: it would only drop a copy identical to one still queued ahead of the same unit, so that unit still receives its tables before it. The quest settles it only if a real source hits the deadline; otherwise it fails loud. (Written by Claude Opus 5.5) |
||
| - Pace `mpegts.si` passthrough on PIDs 1-3 in the same TBsys and Bsys; the | ||
| planner's `transport_buffer` maps only PID 0 and the PMT to TBsys. | ||
| - Delete the round-robin `SystemBuffer` stall in `Slot::layout` (#5220), which | ||
| only unit tests reach once the planner handles padded output, or move its | ||
| cases to the planner. | ||
| - Tests: planner unit tests with 8 aligned PIDs, a multi-packet PMT, a PID 1-3 | ||
| passthrough packet beside a PAT/PMT pair, a changed PMT version going out in | ||
| order, and a burst spanning several slots that fits TBsys but overflows Bsys. | ||
| The AAC tune-in test still passes. `test/ts/compliance.py` gains a hard-severity check | ||
| per program: one TBsys fed by PIDs 0-3 and the PMT PID, draining into one | ||
| Bsys, both checked for overflow, and TBsys emptying at least once a second | ||
| (2.4.2.6), each with a negative control. Every existing TS arm still passes. | ||
|
|
||
| Public API: none. Wire: none. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When neither
mpegts.muxRatenor--mux-rateis present, the stream still has the piecewise transport rate that T-STD derives from byte positions between PCRs;Schedulecontinues stamping unpadded slots on its PCR grid, andtest/ts/README.md:259-268explicitly documents grading without a declared mux rate. Consequently, a dense unpadded slot with adjacent PAT/PMT runs can still exceed Rxsys and overflow TBsys. Dropping the only unpadded protection on the premise that this output has no rate leaves that valid VBR case outside the stated T-STD-compliance goal rather than resolving it.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declining: excluding unpadded output from the system-buffer claim is the maintainer's decision on this PR (decision 1 in the description). Unpadded output has no declared mux rate, so its piecewise rate between PCRs is whatever the media happened to need, and the Goal says to name that exclusion in the profile report rather than grade it. If the maintainer later wants unpadded VBR graded too, that is a separate quest.
(Written by Claude Opus 5.5)