Repository navigation
examples/camera_livekit_bridge: publish receiver frames to LiveKit - #1478
gareth-morgan-nv wants to merge 14 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
83b23e2 to
d950e1f
Compare
|
Review of d950e1f, focused on picture correctness and bounded latency. 1. Protect IPC slots during GPU reads. The post-copy notification check in stream_worker.cpp:189–215 cannot guarantee an intact picture: the receiver can begin overwriting a slot before its asynchronous notification arrives. A copy can therefore include overwritten pixels yet retain the earlier frame's metadata. This is already acknowledged in the PR description; I would treat it as a recording-correctness issue to address before depending on the bridge for data collection. Suggested direction: a bounded slot lease/read guard, released only after the consumer's GPU read completes. Drop new frames or skip unavailable slots under pressure rather than overwrite leased memory or create an unbounded queue. Test deliberate consumer stalls with a small ring and assert picture identity matches metadata; increasing ring size alone does not guarantee correctness. 2. Preserve color range through H.264 publication. The worker leaves EncoderConfig.full_range at false; NVENC then explicitly signals limited range. The new SIPL source produces full-range BT.709 NV12, and the copy/downscale does not perform range conversion. For full-range IPC input, this can give incorrect black/white levels in recordings. Please propagate range/colorimetry or expose a source setting until the IPC metadata carries it, then validate with grayscale ramps/color bars. Architecturally, the host decode → GPU processing → re-encode is justified when processing is needed. Encoded passthrough for unchanged pictures could be a later optimization, not a prerequisite for this PR. Keep performance comparisons explicit about that extra stage and their measurement endpoints. |
jsepulveda-nvidia
left a comment
There was a problem hiding this comment.
I left two important topics in a comment:
- Protect IPC slots during GPU reads.
- Preserve color range through H.264 publication.
These may be the most important issues I found across the stack, so they probably should be addressed.
|
🤖 Agent-generated (Claude Code · claude-sonnet-5-5) Re: 1. Protect IPC slots during GPU reads This is the same hazard raised on the receiver's PR (#1410) in this thread: a slot can be rewritten while a consumer is still reading it, so a delayed consumer can see newer pixels under older metadata, and a deeper ring only makes that less likely. The receiver's author answered it there: the ring is lossy by design, a slow consumer should take the newest frame and skip, and the default ring of 6 slots is meant to absorb about 100 ms of lag. The lease the review suggests is a receiver-side change, so it is not something this PR can add: a consumer cannot stop the receiver writing a slot, and nothing in the receiver's current code ( Why that is a reasonable design for this pipeline: the receiver never waits on a consumer, so one stalled consumer cannot back up decoding or the sender, and any number of consumers can read the same ring without coordinating. The cost is the contract the receiver documents: a consumer finishes with a slot within the ring's window or skips ahead. What the bridge does under that contract: it uses only the newest notice, starts the GPU copy at once on its own stream, waits for the copy to complete, and then drops the picture, never encodes it, when later notices show the ring may have wrapped. What this does not guarantee: the notice arrives asynchronously, so this check cannot prove a copy was clean. A bridge stall of most of the ring's window (about The colour-range point (2) is separate and is being handled on its own. |
|
🤖 Agent-generated (Claude Code · claude-sonnet-5-5) Re: 2. Preserve color range through H.264 publication — addressed in 5e4cfbb.
The slot-protection point (1) is answered in the comment above. |
1421797 to
6e422cb
Compare
5e4cfbb to
17ff72a
Compare
6e422cb to
2e07880
Compare
b1d98d9 to
a5006a9
Compare
a5006a9 to
91bb927
Compare
|
Rechecked [P1] Preserve IPC slot ownership until the GPU read completes. I agree with Juan's recording-correctness concern. The post-copy check still cannot establish that the pixels belong to the notice being encoded. The receiver can start overwriting a ring slot while the bridge is copying it; its replacement notice is generated after producer GPU completion and delivered asynchronously. The bridge can finish its copy and see no newer notice yet, accepting mixed/new pixels with the older frame ID and timestamp. Synchronizing the consumer stream does not synchronize the producer. A latest-frame/lossy policy still needs intact pixels and matching metadata. Please add a coordinated read lease/acknowledgement or equivalent ownership mechanism on both sides, releasing only after the final GPU read; skip unavailable slots under pressure. Test a deliberately stalled consumer and forced ring wrap, checking pixel identity against frame metadata. This is a requirement for using this bridge for recording even if the integration is temporary; I have not run that GPU test here. Please mark the transitional modules and their migration owners. Add a concise section in the affected module READMEs/PR descriptions linking Juan's Unified Media Bridge and the Camera Capture Interface, with module path, future owner, reuse/adapt/retire decision, and the landing condition that removes the temporary wiring:
Both designs explicitly preserve useful capture/encoder/decoder implementations and the AVS1/TCP baseline/mock as reusable foundations. Please distinguish reusable implementation from temporary integration and track the retirement work; the shared APIs are still being stabilized. Review double-checked with Codex on behalf of Farbod. |
Start a downstream consumer of camera_receiver's CUDA IPC output: a client that connects to the publisher's socket, subscribes, opens each slot's IPC handle and reads frame notices with their metadata, and a probe that prints them with a checksum of the first pixels. Pixels stay on the GPU. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…m to LiveKit Add the rest of the bridge: each camera stream's frames are copied on the GPU from the CUDA IPC slot into an NVENC input surface, encoded to H.264 and submitted to a LiveKit track as pre-encoded access units, with the frame's sequence, its sensor start time as the application timestamp and the sensor counter as user data. Frames reach the encoder without leaving the GPU. The NVENC encoder and the LiveKit session are small modules; the session also answers the clock-sync requests that arrive as data messages. Each stream runs on its own thread and encodes the newest picture, skipping older ones. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…oding Add a per-stream integer divisor (--stream NAME=TRACK:DIV) that averages each block of the NV12 picture on the GPU between the IPC slot and the encoder input, so a large camera stream can be published at a size the consumers can handle without the pixels leaving the GPU. This is also the point where a GPU operation on the picture belongs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Describe how the bridge moves frames from the IPC slot to a LiveKit track, how to build and run it, the per-stream shrink option, and the capacity measured against egocap's receiver and recording checks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…the pipeline Add unit tests for the GPU downscale (exact against a CPU reference), the clock-sync wire format (golden bytes) and the NVENC encoder (Annex-B output, SPS/PPS and IDR, pool behaviour, decodable by ffmpeg), a test that reads known GPU patterns through the IPC client and probe from a separate publisher process, and an end-to-end test of mock sender, receiver, bridge, a local LiveKit server and a subscriber that checks track names, sizes, frame ids, capture-time age and user data. Each skips with a stated reason when a prerequisite is missing. The probe's checksum now reads at most the slot's size instead of always 4096 bytes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Add --timestamp-basis: capture_unix_us (default, the sensor start-of-frame as Unix time), capture_monotonic_us (the same instant on this host's monotonic clock, derived as now minus the frame's age) and monotonic_us (the time the frame is published). A frame with no usable sensor start-of-frame carries no timestamp on the capture bases. The conversion is a small header-only helper with its own test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…te unqualified capture time Take the frame-ready instant when the picture is on the encoder's input, before encoding, so the monotonic timestamp does not include encode delay. Use the sender's sensor start-of-frame only when the receiver reports a realtime (Unix) sender clock; any other domain publishes no capture time instead of labelling it Unix. Refuse --timestamp-basis capture_monotonic_us: it moved the sensor time onto the host's monotonic clock by assuming the sender's wall clock agrees with the host's, with no bound on that error. It needs a measured sender-to-host clock mapping, which does not exist yet. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Start each stream's encoder at 2 Mbps and apply the target bitrate that the LiveKit congestion controller reports for the track (10% headroom, capped at --bitrate-mbps, floor 0.5 Mbps) by reconfiguring NVENC without a restart or an IDR. A fixed high rate sent into a connection whose bandwidth is not yet known queued frames behind the sender's pacer, making the first frames 1-2 s old and growing. With the adaptive start they are about 60-150 ms old and settle near 80 ms. The stats line reports the current rate, the transport target and the number of changes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
… the rate honest, add timing tests Report the sensor start-of-frame to frame-ready time in the stats line, from frames whose sensor time is on a realtime sender clock, so the latency before a consumer's own frame-ready clock starts can be added to what it reports. If NVENC rejects a bitrate change the encoder keeps its old rate, so the reported rate and the settings for the next change are restored too. Add tests: the bitrate setter and getter, zero rejected, no forced IDR after a reconfigure, smaller packets at a lower rate, and a stream that stays decodable across changes; and a timing test that runs egocap's clock-sync client and recording feed against the bridge on monotonic_us, refuses capture_monotonic_us, and checks that egocap's loader rejects capture_unix_us under its timing branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…e been overwritten The receiver overwrites a slot without notice once its ring wraps. After the GPU copy completes, the bridge now checks for a newer notice for the same slot, or as many newer notices as the ring has other slots, and drops the picture instead of encoding a possibly torn one. Dropped pictures are counted as slot overruns in the stats line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ces; document the slot contract --source-full-range marks the encoded H.264 as full range. The receiver's IPC catalog does not carry the range, so the operator says it; without the flag the stream signals limited range, which gives full-range SIPL pictures wrong black and white levels in players and recordings. The encoder copies levels as they are and does no range conversion. A test encodes a luma ramp and checks the SPS video_full_range_flag for both settings. The README now states the receiver's slot contract (slots are rewritten about ring_depth frames after they are announced and a consumer must finish in time or skip), what the bridge does under it, and that the notice-based check cannot prove a copy was clean. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…viour Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
9d829b1 to
012423b
Compare
|
🤖 Agent-generated (Claude Code · claude-sonnet-5-5) Re: @farbod-nv's review — #1478 (comment) The documentation/migration request is addressed; the GPU slot-ownership fix remains open. The current notification check cannot guarantee intact pixels with matching metadata. For the bounded set of consumers on one PC, we are proposing atomic acquisition of a specific slot generation, release only after the final GPU read completes, and a per-consumer outstanding-frame limit. With buffer capacity covering those limits plus producer working buffers, the producer can select an unheld slot without waiting for a slow consumer; stale acquisitions are rejected. This needs agreement on pool sizing, failure/reclamation and the protocol, plus a stalled-consumer/ring-wrap test checking pixel identity and producer throughput. It is not yet implemented or validated. Per-consumer snapshots remain an alternative with additional copy cost. The READMEs describe the temporary transport/receiver wiring, the reusable implementation, and application-owned GPU processing, and are self-contained so public readers do not need access to restricted design documents. |
…e ring's wrap time Each frame notice carries the receiver's announce time. After the GPU copy the bridge compares the copy's age against the time the publisher needs to wrap its ring at the stream's rate (or --max-notice-age-ms) and drops the picture with a prominent error when it is older, in addition to the newer-notice check. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
|
🤖 Agent-generated (Claude Code · claude-sonnet-5-5) Re: @farbod-nv's review (and @jsepulveda-nvidia's earlier point on slot protection) — #1478 (comment) Partly addressed in bfccaea. The residual risk is a receiver-side contract gap, which this PR cannot close from the consumer side. The failure case. The receiver rewrites its ring slots in rotation and never waits for a reader. It announces a slot only after it has finished writing it, so the notice that a slot was replaced arrives after the write. A consumer whose GPU copy is still reading a slot when the receiver wraps back to it can therefore publish a picture that mixes two frames under the first frame's ID and timestamp, and the "newer notice" check cannot see it if that notice has not arrived yet. What the bridge does now. Each notice carries the receiver's announce time ( What it cannot catch.
What needs a producer-side change. Reliable detection needs a marker the consumer can read after its copy. A per-slot write counter in shared memory, which the receiver bumps on the host before it enqueues a decode into that slot, would do it: the consumer re-reads it after its copy completes and drops the frame if it is not the generation from the notice. The receiver never waits, so a slow consumer still only affects itself, and there is nothing to release if a consumer crashes. The marker changes before any GPU write starts, so a write during the read is never missed; the worst case is dropping a frame that was clean. The cost is one 8-byte read per frame. Until the receiver publishes such a counter, recordings that need intact frames cannot be guaranteed by the consumer alone, and this stays a known limitation of the receiver contract rather than something this PR resolves. |
Summary
Adds
examples/camera_livekit_bridge, a standalone C++ bridge from the receiver's CUDA IPC frames to LiveKit: GPU copy or box downscale, NVENC H.264 (low latency, bitrate following the transport's target), pre-encoded publish with frame metadata, and a clock-sync responder. Pixels never reach host memory.--timestamp-basischooses what the frame timestamp means (capture_unix_usor the frame-readymonotonic_us). Stacked on the mock and receiver-fix PR.Known limitation
The receiver overwrites an IPC slot without notice once its ring wraps. After the GPU copy the bridge drops the picture if a newer notice for the same slot arrived, or as many newer notices as the ring has other slots (counted as "slot overruns"). This is a best-effort check, not exclusive slot ownership: the receiver queues notices asynchronously and may drop them, so a wrap that overlaps the copy can be undetected. No test forces a wrap; the drop path is exercised only by reading the code. A real fix needs a protocol change (reservation or a read guard), which this PR does not attempt.
Testing
ctest (gpu_ops, nvenc, clock_sync, timestamp_basis) and the Python IPC-client and mock-to-LiveKit pipeline tests. With the mock on an RTX PRO 5000: head 2560x1984 at 8 Mbps median capture-to-receive 75-83 ms, no missing frames, 0 slot overruns at 30 fps. Not run against real cameras.
Integration boundary
This example publishes receiver pictures to a browser through LiveKit. The session, publication and feedback wiring (
livekit_publisher/and the session handling inbridge/) is temporary integration, intended to move behind transport adapters that carry camera video and application messages over a selectable network transport and own sessions, admission limits and feedback. Camera buffer ownership and encode and decode responsibilities belong to the camera interface. A replacement must preserve browser delivery, bounded latency and safe GPU access; until receiver slots have read leases, the residual risk described in the README section "Reading the receiver's slots" applies.livekit_publisher/and the session, publication and feedback wiring inbridge/ipc_client/and the slot read inbridge/stream_worker.cpp🤖 Generated with Claude Code