Skip to content

examples/camera_streamer: carry a qualified sensor capture time to LiveKit - #1479

Closed
gareth-morgan-nv wants to merge 2 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingTiming
Closed

gareth-morgan-nv wants to merge 2 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingTiming

Conversation

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor

Summary

Receiver, mock and bridge side of capture-time alignment. Stacked on the bridge PR.

  • Wire: AVS1 bundle flag 0x0008: one 24-byte record per sample with the sensor capture instant on the sender's monotonic clock, its stated uncertainty, anchor age and a validity bit. The receiver parses and validates it and publishes the fields in the IPC metadata.
  • Mock: emits it from a synthetic sender clock (offset, drift, steps), writes the true capture instant to a ground-truth file, and serves the sender's stateless UDP clock responder (13-byte request, 29-byte reply) with mute, loss and reorder faults.
  • Bridge: measures the sender clock over that responder and publishes capture_monotonic_us only for frames that qualify: valid record, current mapping with warm-up, mapping plus sensor uncertainty within the budget, possible age. Otherwise no capture time is published and frame-ready time is never substituted. A reply counts only from the sender's address, of the exact size and tag, matching the outstanding request and the current session generation.

Limitations

A step of the sender clock shows at the next exchange, so frames in between (at most one exchange interval) carry the old offset; a learned age level cannot tell a step from a stall, so none is used. The sender side (real Orin) is not part of this PR and has not been qualified against it.

Testing

ctest 7/7 (clock-estimator golden vectors from egocap's estimator, qualifier, UDP client against a fake sender, plus the existing targets). A scenario test through the real receiver, bridge and a local LiveKit server (accuracy, step, restart, stale mapping, loss and reorder, invalid records, refusal) passed twice; with a 1.2 s offset and 50 ppm drift the published capture time is within about 100 us of ground truth. A take saved by the unchanged egocap timing recorder matched ground truth within 0.84 ms. The scenario test was reformatted after those runs and not rerun.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

gareth-morgan-nv and others added 2 commits November 11, 2026 00:37
…erve a clock responder

The AVS1 bundle gains an optional timing extension (flag 0x0008): one 24-byte
record per sample with the sensor capture instant on the sender's monotonic
clock, the sender's stated uncertainty, the anchor age and a validity bit. The
receiver parses and validates it and publishes the fields in the IPC metadata.

The mock sender emits it from a synthetic sender monotonic clock with its own
offset, drift and steps, writes the true capture instant on this host's clock to
a ground-truth file, and serves the sender's stateless UDP clock responder
(13-byte Q, 29-byte R) with mute, loss and reorder faults for tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ground truth

Scenarios through the real receiver, bridge and a local LiveKit server with a
sender clock offset and drift: accuracy, a clock step, a session restart, a
stale clock mapping, lost and reordered replies, invalid timing records, and
refusal without a sender clock. The mock's lint fix rides along.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@jsepulveda-nvidia

Copy link
Copy Markdown
Contributor

Review of eecc167: I compiled and ran clock_estimator, capture_qualifier, timestamp_basis, clock_sync, and sender_clock tests successfully (the UDP test requires loopback networking). Rejecting unqualified capture times without substituting frame-ready time is a good recording contract.

Two integration caveats: qualified sensor timing currently requires the SIPL sender extension in #1474; Argus does not yet populate it. Also, timestamp qualification depends on the pixels belonging to that metadata: the IPC slot-ownership concern posted on #1478 remains relevant here, but should be fixed once in the shared handoff, not duplicated in the timing code.

No new hardware validation was performed in this review. The mock timing accuracy is useful, but should not be presented as physical exposure alignment or photon-to-photon latency.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Thanks for the review and for running the bridge's timing tests. This PR is going to be closed, so no further changes will be made here. Your caveats are noted: qualified sensor timing currently needs the SIPL extension in #1474 (Argus does not populate it), the slot-ownership concern belongs in the shared handoff discussed on #1478 and not in the timing code, and the mock timing accuracy should not be presented as physical exposure alignment or photon-to-photon latency.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

Superseded by #1474 (sender capture timestamp and receiver parsing), #1518 (timesync module) and #1519 (bridge qualification against chrony); the launcher side is #1520. Closing without merging.

This branch had an error being deployed

1 failed deployment
dev — eecc167f Deployed Nov 11, 2026 by gareth-morgan-nv via publish-wheel #6782
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants