Skip to content

camera_livekit_bridge: qualify capture times against chrony - #1519

Open
gareth-morgan-nv wants to merge 10 commits into
gmorgan/TimeSyncfrom
gmorgan/TimeSyncIntegration
Open

gareth-morgan-nv wants to merge 10 commits into
gmorgan/TimeSyncfrom
gmorgan/TimeSyncIntegration

Conversation

@gareth-morgan-nv

@gareth-morgan-nv gareth-morgan-nv commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Makes the bridge's capture_monotonic_us timestamp basis use the timesync module: it polls this PC's chrony and the camera host's, qualifies each frame's capture time against a budget, and publishes a frame without a timestamp only when the conversion cannot be vouched for. A frame over the budget keeps its timestamp. Replaces the earlier clock-exchange approach for that basis.

Depends on #1518, which is this PR's base and carries src/core/timesync (and is itself stacked on the LiveKit bridge branch, #1478). This branch builds from a clean checkout of its own tree. It should be rebased onto main once #1478 and #1518 have landed.

What changed

  • New chrony_clock adapter (ChronyCaptureClock): background polling of the PC's and the camera host's chrony, the per-frame qualification, and the clock generation.
  • Bridge: --chrony-sender HOST[:PORT], --chrony-sender-refid HEX, --chrony-pc-port PORT, --chronyc PATH. capture_monotonic_us requires the first two. A frame whose conversion cannot be vouched for is published without a timestamp, so a recording take fails instead of recording a guessed time. A conversion that passes every check but is over the precision target (over_budget) is published with its timestamp and counted separately. Recording eligibility is permissive here: such a frame is recorded like a qualified one, because the verdict and bound are runtime diagnostics and are not carried with the video. That is this bridge's current policy, not the shared camera interface's precision admission; a consumer that must reject frames above the precision target needs the bound or a stricter publication policy. A detected wall-clock step on this host is a timing fault: for 8 s after it no frame is timestamped (longer than the recorder's 5 s silence limit, so a take cannot continue across it even if the refused frames are dropped), and chrony readings taken before it are never reused.
  • [stats] reports the qualified count, a count per refusal reason, the last combined uncertainty and the clock generation.
  • The mock sender can emit the realtime timing record (--timing-ext, --sensor-uncertainty-us) so the path runs without a camera.

Validation

  • timestamp_basis and chrony_clock tests pass; CLI refusal tests pass; the bridge builds. Built from a clean checkout of this branch, which carries the module: configure, build, all six bridge tests and the five timesync tests pass.
  • Real camera end to end with the launcher PR: about 30 fps, 6167 frames qualified and 110 refused as over budget in one run, combined uncertainty about 2.2 ms.
  • Over a Wi-Fi link every frame was over_budget (about 6 to 11 ms against the 5 ms target) and every one was published with its capture time; a 20 s take through the unchanged recorder finished with no timing fault.
  • The bridge's GPU timing tests were not run for this change.
  • CI on the restacked base has not been confirmed.

Integration boundary

The chrony_clock/ adapter and the capture-time publication in bridge/stream_worker.cpp are transitional: they connect the reusable timing library to the current video publication path and are intended to converge on the receiver backend's clock-mapping contract. With capture_monotonic_us, a mapping that cannot be vouched for publishes no timestamp and the frame-ready time is never offered in its place; a capture over the precision target keeps its timestamp. monotonic_us is a separate, declared basis.

Current path Intended owner or replacement Reuse, adapt or retire Follow-up and landing condition
chrony_clock/ and the capture-time publication in bridge/stream_worker.cpp The receiver backend's published clock mapping (offset, uncertainty, discontinuity epoch) Transitional: adapt, then retire the bridge-specific adapter Lands when consumers obtain equivalent timing without the bridge-only adapter.
src/core/timesync/ The estimator behind the receiver's clock mapping Reuse Keeps the chrony evidence handling and qualification behind the destination interface.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@jsepulveda-nvidia

Copy link
Copy Markdown
Contributor

Review of 7c46eea — keeping chrony polling off the frame path and preserving preview when timing is unqualified are good fits for this bridge. One correctness issue remains in ChronyCaptureClock::qualify():

A detected PC wall-clock step invalidates more than the single frame that detects it. The next call sees stepped == false and reuses chrony snapshots collected before the step.

I compiled the adapter and timesync sources using the PR's fake clock/chrony fixtures. Starting healthy, I kept the sender timestamp fixed (80 ms old), stepped only PC realtime backward 20 ms, and called qualify twice without polling again. Result: qualified → no_local_clock → qualified; the final monotonic capture value was wrong by 20 ms, with reported uncertainty 200060 ns. The existing test_realtime_step moves the simulated sender timestamp with the PC clock, masking this case.

Please invalidate pre-step evidence and keep timestamps unqualified until post-step synchronization evidence supports the budget. Also verify the generation change reaches the take-validity logic: recording it in worker stats alone cannot ensure an active take notices a discontinuity. Add a regression where the sender clock does not step with the PC and recovery requires new evidence. Video can continue without timestamps throughout.

Validation: synthetic C++ reproduction; no live clock or camera settings were changed.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Thanks, reproduced by reading the code: qualify() refused only the frame that revealed the step, and the next call reused chrony snapshots read before it. Fixed in 887a810.

ChronyCaptureClock now records the monotonic time of a detected PC realtime step and treats chrony evidence read at or before it as absent, for both the PC's and the camera host's. Frames then get NoPcEvidence/NoSenderEvidence (no timestamp) until both monitors have polled again after the step, so the stale-snapshot path (qualified, then no_local_clock, then qualified with the capture value off by the step) no longer exists. The header comment states this.

On whether a take notices: I did not add the generation to the published frame, because the recorded data and its per-frame metadata must not change. The mechanism is the existing one. The frame that reveals the step and every frame until fresh evidence arrives carry no timestamp, and a take in progress ends on the first of them with the missing-timestamp fault. After that a new take pins the new generation. If a step lands between two polls, the step is still detected by the next frame's local-clock sample, which is what sets the floor.

A regression with the sender timestamp not moving with the PC clock (healthy, step PC back 20 ms, qualify twice with no poll between, then poll and recover) is being added to test_chrony_capture_clock.cpp; I'll push it as a follow-up commit. Not run with a live clock step or on hardware.

@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/CameraStreamingLiveKit branch from 5e4cfbb to 17ff72a Compare November 13, 2026 16:44
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/TimeSyncIntegration branch from e6a719b to f305152 Compare November 13, 2026 16:44
@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Follow-up on the clock-step finding (pushed 621a31a, on top of the earlier 887a810 rebased as part of the stack rebase). My earlier reply said a take would notice a step through the one unstamped frame. A review of that pointed out it depends on that frame surviving the media path, and the recorder does not treat a lost frame as a fault, so I changed it.

After a detected PC realtime step the adapter now gives no frame a timestamp for step_hold_ns (default 8 s), whatever the chrony evidence says, and still ignores any chrony reading taken at or before the step. A take can only notice a step through an unstamped frame or through silence, and the recorder fails a take after 5 s without a frame, so the hold (longer than that) means a take cannot continue across the step even if every refused frame is dropped on the way. Frames and the preview keep flowing without timestamps meanwhile. No recorded field or per-frame metadata was added. The README states it.

Tests (all with the sender timestamp not moving with the PC clock): a frame 1 s after the step with fresh post-step evidence is still refused; just before the hold ends it is refused and just after it qualifies with the post-step generation and a capture time consistent with the new clock; with a 1 ms hold the old behaviour is reachable, so the hold is configurable. The chrony_clock test passes against the #1518 module sources with my pending review fixes. Not run on hardware or with a live clock step.

@farbod-nv

Copy link
Copy Markdown
Contributor

Rechecked 9795e94b6e25.

Integration gate: this branch's bridge CMake adds src/core/timesync, but that directory is not present in this head's base/tree. Please restack onto a base containing #1518 and validate a clean checkout/configure/build before merge. The portable adapter test passes when linked against an explicit #1518 overlay; that does not validate a self-contained checkout.

Please label chrony_clock and the current bridge-specific timestamp wiring as transitional adapters, with links and a migration follow-up to Juan's media bridge / the Camera Capture Interface. Preserve source timestamps and session identity; the target receiver contract publishes the mapping, uncertainty and discontinuity epoch separately.

Also reconcile the timing acceptance policy with that target contract: this PR deliberately emits timestamps for OverBudget frames, while their bound/verdict reaches bridge statistics rather than recording metadata. The camera-interface proposal fills a local sample time only while mapping uncertainty meets the configured budget and requires timing-sensitive consumers to reject unsuitable inputs. Please make the temporary behavior and its recording eligibility explicit so an over-budget time does not silently acquire the same meaning as an accepted common-interface sample.

Review double-checked with Codex on behalf of Farbod.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Re: @farbod-nv's review — #1519 (comment)

Addressed (head dcc0067d5).

  • Base. This PR is now stacked on timesync: qualify camera capture times against chrony #1518, which carries src/core/timesync (and is itself stacked on examples/camera_livekit_bridge: publish receiver frames to LiveKit #1478). A clean checkout of the stacked tree configured, built, and passed the bridge and timesync tests with no overlay (on the previous revision of this stack; the current head differs only by a documentation line in timesync: qualify camera capture times against chrony #1518's README).
  • Recording policy. The README and description now state it: an over_budget frame is converted validly, keeps its capture time and is published like a qualified one, so a recording that accepts every timestamped frame records it. The verdict and bound are runtime diagnostics, not carried with the video, so this is the bridge's permissive temporary policy and not the shared interface's precision admission. A frame without a timestamp (invalid mapping, stale or unsynchronised evidence, or a wall-clock step) is the timing fault. A consumer that must reject frames above the precision target needs the bound or a stricter publication policy.
  • Migration. The README marks chrony_clock and the bridge's timestamp publication as transitional, to be retired when the receiver-backend mapping supplies equivalent capture-time, validity and discontinuity behaviour; the reusable timing library stays separate. The migration notes are self-contained for public readers.

This does not close the GPU slot-ownership item or claim physical timing accuracy.

gareth-morgan-nv and others added 10 commits November 18, 2026 11:36
The bridge's capture_monotonic_us basis now converts the sender's wall-clock
capture time with the timesync library: a background poll of this PC's and
the sender's chrony, a per-frame qualification against a time budget, and a
refusal that publishes the frame without a timestamp. Verdict counts are
reported in the stream statistics. Frames carry no capture time unless
qualified, so a recording take fails rather than recording a guessed one.

The mock sender can emit the realtime timing record so the path runs without
a camera.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…s the precision target

A frame over the precision budget kept no capture time, so a consumer could
not use it. Publish the converted instant for every usable verdict; only a
conversion that cannot be vouched for publishes none.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…s published

A conversion that cannot be vouched for publishes no timestamp; one that
passes every check but is over the precision target keeps its timestamp and
is only counted differently in the statistics.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ck step

A detected PC wall-clock step refused only the frame that revealed it; the next
frame reused chrony snapshots collected before the step and qualified with a
capture time off by the step. Record the step time and ignore evidence read at
or before it, so frames stay without a timestamp until both hosts have been
read again. A take in progress then ends with the missing-timestamp fault.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
… not reused

With the sender capture timestamp not moving with the PC clock: a step is
refused, the next frames without a new poll are not usable, and qualification
recovers only after both hosts are read again, with the capture time correct
on the new clock.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…lock step

A take can only notice a step on this host through a frame without a
timestamp or through silence, and the recorder fails a take after 5 s of
silence. Refuse every frame for step_hold_ns (8 s) after a step, whatever the
chrony evidence says, so a take cannot continue across the step even if the
media path drops the refused frames. Evidence read before the step is still
never reused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
The bridge publishes either the frame-ready time or the chrony-qualified capture
time, both on the publisher's monotonic clock.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…tion boundary

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…tops qualifying

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>

This branch was successfully deployed

1 active deployment
dev — e9a6282d Deployed Nov 18, 2026 by gareth-morgan-nv via publish-wheel #7071
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants