Skip to content

test(ietf): verify pinned Moxygen conformance under a relay profile - #6497

Merged
kixelated merged 5 commits into
mainfrom
quest/m1/ietf-extension/README
Oct 23, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m1/ietf-extension/README

Conversation

@kixelated

@kixelated kixelated commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The passthrough quest required pinned Moxygen sections 1-7 to pass through one relay and a two-relay cluster. The literal suite conflicts with two settled relay behaviors (track-level egress priority, and END_OF_TRACK), and its publisher cannot answer the cold metadata query the cluster sends. This PR lands the pinned harness, verifies under a relay conformance profile instead of the literal suite, gates the nightly on the cells that pass, and rewrites the quest to the work that remains.

Advances quest/m1/ietf-extension/README.md; the quest stays open with its remaining work.

Approach and consumer

  • Pin Moxygen c27ae6358df62a1195da479b2e07c50dd8174279 and its six matching dependency sources with fixed Nix hashes. Reuse upstream's standalone build, add its missing Folly ManualExecutor link dependency, and fix dependency script interpreters for Nix.
  • Conformance profile (nix/moxygen-profile.patch, client verifier only). It also accepts the track's advertised publisher priority (200) on any group, besides upstream's per-group value. It accepts one END_OF_TRACK on the subscription directly after the last object, either inline or opening the next group, and skips the END_OF_GROUP header expectation for that terminal stream, which ends no payload group. Any other priority, a second END_OF_TRACK, or one anywhere else still fails. Every other upstream check is unchanged: object IDs, properties, payloads, END_OF_GROUP status objects and header bits, FIRST_OBJECT.
  • Add just test ietf-conformance and a nightly workflow. Each draft runs a direct-origin control and one relay. The client uses the actual 16-field namespaces under the published moq-test-00 prefix. Two-relay cells are not run until a pinned publisher answers TRACK_STATUS (below); the quest records adding them back.
  • Select 29 upstream SUBSCRIBE cases per cell using sections 1-7 and upstream SKIP_FETCH. Explicitly exclude only modes 1/2, whose cases use nonzero subgroups. Keep all selected arguments, deadlines, and assertions. Refuse changed selection anchors, a crash after SUCCESS, and publisher reconnects.
  • Mock-clock integration tests prove prefix routing through both topologies on drafts 16 and 18.
  • Consumer: the nightly ietf-conformance job, running the pinned moqtest_client/moqtest_server. Ordinary checks and shells do not build the opt-in C++ package.

Measured conformance

Six consecutive runs, all passing:

Draft Direct origin One relay
16 29 pass 29 pass
18 29 pass 29 pass

Before the profile, both single-relay cells failed 29/29 on priority and terminal status. just test ietf-conformance now exits 0.

Remaining in the quest:

  • Cluster metadata: with two relays, all 29 cases failed before media on both drafts. The edge's TRACK_INFO query becomes TRACK_STATUS upstream, and the pinned MoQTestPublisher inherits NOT_SUPPORTED. The relay preserves that refusal; a_refused_track_status_never_subscribes forbids a SUBSCRIBE fallback, and moq-dev/moq#6465 owns the broader cold-metadata boundary. The mock-clock routing test still covers the lite-07 cluster path.
  • Direct draft-18 control: the checkpoint run failed "Large scale: Many groups and objects" with Group Mismatch: Actual=1 Expected=0 without a relay. It has not recurred in six runs since. The verifier expects groups in order although each arrives on its own stream, which its own source comment calls a spurious failure.

Decisions

  • ✅ Keep the approved relay behavior and adopt a conformance profile (maintainer, 2026-10-23). The profile understands track-level priority and legitimate END_OF_TRACK while still checking forwarded IDs, properties, END_OF_GROUP and FIRST_OBJECT. The quest Goal now names that profile instead of "the current selected suite passes".
  • Revisit per-group priority transport and terminal representation (not chosen).
  • ✅ Profile lives as a Nix patch on the pinned client (picked unattended, recommended). It needs no upstream post and keeps the server and protocol sources unchanged. Alternatives were classifying failures from client logs (fragile, hides failures) or carrying a fork of Moxygen.
  • ✅ Remaining work stays in the quest (maintainer): a publisher with metadata-query support, then the direct draft-18 control.
  • ✅ Nightly gates only on cells that can pass (picked unattended after the Codex finding, recommended). Two-relay cells fail every case until the publisher prerequisite lands, which would keep the whole nightly red and mask unrelated regressions. Alternatives were a permanently red job, or not wiring the nightly at all, which would leave the passing single-relay cells unrun.

Validation

  • Pinned Moxygen with the profile: Nix build passed. just test ietf-conformance: table above, exit 0; selection tests passed (29 cases selected). shellcheck on run.sh passed.
  • Routing integration tests: 2 passed.
  • just check: JS, Rust, quest, drill, OBS compile and OBS checks passed. The unchanged runner-image flake check fails on this host because systemd-repart cannot unshare its UID map (Operation not permitted); CI runs it.
  • just test interop --all: all checks passed (172 PASS lines, 0 FAIL).

Public API and wire impact

None. Production code, catalog formats, wire encoding, priorities and terminal behavior are unchanged. No migration entry is needed.

Follow-ups

Kept in the quest's Plan rather than new quests, per the maintainer: a Moxygen revision whose test publisher answers TRACK_STATUS (then bump pins and rebase the profile), and resolving the order-sensitive direct draft-18 control at the verifier.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits October 22, 2026 21:50
@kixelated kixelated changed the title moq-transport passthrough test(ietf): checkpoint pinned Moxygen conformance Oct 23, 2026
kixelated and others added 2 commits October 23, 2026 09:18
The pinned client now accepts the track's advertised priority on every
group and one END_OF_TRACK directly after the last object, and keeps every
other upstream check. Both single-relay cells pass on drafts 16 and 18.
The quest's Goal moves to that profile; the cluster's metadata query and
the direct draft-18 control remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title test(ietf): checkpoint pinned Moxygen conformance test(ietf): verify pinned Moxygen conformance under a relay profile Oct 23, 2026
@kixelated
kixelated marked this pull request as ready for review October 23, 2026 17:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T17:48:11.510108Z 2808a9a New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9cfeb323b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ietf-conformance.yml
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 910daba1-5ffa-4a0b-8b65-5efb5db5c7a8
📥 Commits

Reviewing files that changed from the base of the PR and between 397e251 and 9cfeb32.

📒 Files selected for processing: 16
  • .github/workflows/ietf-conformance.yml
  • .github/workflows/nightly.yml
  • flake.nix
  • nix/moxygen-profile.patch
  • nix/moxygen-sources.nix
  • nix/moxygen.nix
  • quest/m1/README.md
  • quest/m1/ietf-extension/README.md
  • rs/moq-net/tests/integration/ietf_conformance.rs
  • rs/moq-net/tests/integration/main.rs
  • test/README.md
  • test/ietf-conformance/README.md
  • test/ietf-conformance/prepare.py
  • test/ietf-conformance/run.sh
  • test/ietf-conformance/selection.test.py
  • test/justfile

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

Walkthrough

The change adds a pinned Moxygen package with a relay conformance profile, a harness for running selected conformance cases, and tests for relay routing. The harness covers drafts 16 and 18 with direct-origin controls, single-relay runs, and two-relay cluster runs. New integration tests exercise routing through one relay and a lite cluster. A reusable workflow runs the harness nightly and retains available reports and endpoint logs.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 9cfeb

The harness records the known two-relay failures without blocking other nightly jobs. No additional merge-blocking defect was established.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 5 files. (11 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title concisely identifies the main change: testing pinned Moxygen conformance with a relay profile. It matches the changeset.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The pull request description directly explains the Moxygen pin, conformance profile, nightly workflow, integration tests, measured results, validation, and remaining work described in the changeset.
Full details: Docstring Coverage

Explanation

Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 5 files. (11 skipped: 11 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Two-relay cells fail every case until a pinned test publisher answers
TRACK_STATUS, which would keep the whole nightly red and mask unrelated
regressions. Run the direct-origin controls and the single-relay cells,
which pass under the profile, and track the cluster cell in the quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. Adds the Moxygen conformance profile; the nightly gates on the direct and single-relay cells until upstream Moxygen answers TRACK_STATUS, as the maintainer chose. CI is green, the current head has a non-Grok review, and quest check passes against main.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 5e0af0b Oct 23, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/ietf-extension/README branch October 23, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant