Skip to content

examples/camera_livekit_bridge: run a series of GPU steps on each picture, with a sine-warp example - #1481

Open
gareth-morgan-nv wants to merge 5 commits into
gmorgan/TimeSyncIntegrationfrom
gmorgan/CameraStreamingSineWarp
Open

gareth-morgan-nv wants to merge 5 commits into
gmorgan/TimeSyncIntegrationfrom
gmorgan/CameraStreamingSineWarp

Conversation

@gareth-morgan-nv

@gareth-morgan-nv gareth-morgan-nv commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The copy between the receiver's CUDA IPC buffer and the encoder input becomes a series of GPU steps (GpuStep) that run in order on one CUDA stream, with no host copy. None run by default: with no flag the picture is only copied or shrunk, as before.

  • --gpu-step NAME[=ARGS] adds a step, repeatable, run in the order given at the source size; a stream's :DIV downscale (now itself a step) runs after them.
  • sine-warp is an example step only. It is not calibrated rectification; it shows the shape of a step.
  • Adding a step is one class plus one line in make_gpu_step(). GPU_PROCESSING.md describes the system, the step contract and how lens rectification would be written as a step (a remap table computed once from the calibration and sampled per frame for luma and chroma).
    Stacked on the capture-time PR (it touches the same bridge files).

Testing

ctest 8/8, including a new gpu_pipeline test (no steps copies, identity warp equals a copy, a visible change otherwise, steps run in series, bad names, parameters and sizes are refused). The series test composes a non-identity sine warp with a downscale and compares the result with a host-side box downscale of the separately warped picture; it also asserts that skipping the warp, or applying the steps in the other order, gives a different result. It passed on an RTX PRO 5000 on this head's stack; the warp used for the expected picture is the implementation's own, so this tests the step handoff, not the warp's mathematics. A live run through the mock, receiver and bridge with --gpu-step sine-warp=14,90 ran at 30 fps with 0 slot overruns and the published picture shows the ripple. The default path (no flag) is covered by the unit test; I did not repeat the live run without the flag on this head. Nothing here has run on real cameras.

Integration boundary

GPU operations run on camera images as application or derived-stream processing; running them inside this bridge is temporary wiring. The processing path keeps input and output on the GPU. The bridge's own scratch and encoder buffers are tied to GPU completion; the receiver's input slot is read without a lease today (the slot-ownership limitation described in #1478), so reading that input safely depends on the ownership contract being added on the receiver side. The processing must keep GPU-resident buffers and completion-based lifetime when it moves behind the camera reader interface. The sine-warp step demonstrates kernel integration; it is not a calibrated rectification implementation.

Current path Intended owner or replacement Reuse, adapt or retire Follow-up and landing condition
gpu_ops/ and the GPU step pipeline in bridge/stream_worker.cpp Application or derived-stream processing over camera data Reuse the kernels; adapt the pipeline; retire the bridge wiring Lands when GPU processing runs behind the camera reader with correct geometry and calibration and safe buffer lifetime.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

// rectification runs: see GpuStep in gpu_ops/gpu_step.hpp. The encode below waits for this stream.
const Nv12Planes in{ base + layout.y_offset, base + layout.uv_offset, layout.y_pitch, layout.uv_pitch };
const Nv12PlanesOut out{ surface.view.y, surface.view.uv, surface.view.y_pitch, surface.view.uv_pitch };
const cudaError_t copy_status = pipeline.enqueue(in, out, copy_stream);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NVshawnhwang this is where rectification goes. Between the receiver's CUDA IPC buffer and the encoder input, each picture now passes through a series of GPU steps (gpu_ops/inc/gpu_ops/gpu_step.hpp), none enabled by default. sine-warp is an example step only, not rectification. A rectification step is the same shape: compute the remap table from the camera calibration once in prepare(), upload it, and sample through it per frame for luma and chroma. SineWarpStep in gpu_ops/inc/gpu_ops/sine_warp.hpp is the template and its comment walks through it; GPU_PROCESSING.md covers how to add a step.

🤖 Agent-generated comment (Claude Code).

@NVdejung NVdejung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated first-pass review (Claude, for Dejun) — I'll follow up personally where needed.

Summary — This PR turns the copy from the receiver's IPC slot to the encoder input into an ordered GpuPipeline of GpuSteps on the worker's CUDA stream. It makes the :DIV box downscale the last step and adds an opt-in --gpu-step flag with a sine-warp example, a gpu_pipeline test and a design doc. I read every hunk plus the worker, encoder and CUDA-context code around them. With no flag the bridge issues the same copy and shrink calls as before. prepare() allocates and the worker thread launches in the same primary context, and the warp kernels stay in bounds. No blockers. The main gaps are a series test that cannot see a broken step-to-step hand-off, and a start-time size check that was lost.

Must fix
None found. Internal names: the scan of the added lines, title, description and commits found nothing, and the new docs name no people, stations or internal resources.

Suggestions

  • tests/gpu_pipeline/test_gpu_pipeline.cpp:120 — The series check chains sine-warp=0 with the downscale. The first step is an identity, so the test still passes if the downscale reads the source instead of m_scratch[0], or if the first step never runs. The hand-off that --gpu-step plus :DIV relies on is therefore untested.
    • The "two warps differ from one" check named in the comment does not exist.
    • in_series (L125) is computed and discarded, and shrunk holds 128x96.
    • Fix: compare sine-warp=8,32 followed by the downscale against the downscale of a separate sine-warp=8,32 output.
    • Give source and destination padded pitches as test_box_downscale.cpp does: every buffer here has pitch == width, so a width-for-pitch slip in a step would still pass.
    • The default path was rewritten and the description says the no-flag live run was not repeated on this head. One short run with and without :DIV would close that.
  • gpu_ops/gpu_pipeline.cpp:97 — Silent behaviour change. The old constructor ran box_downscale_supported(w, h, 1) on every stream, which refused an odd or zero source at start. Now, with no steps and no :DIV, nothing checks the source: the track is published and the worker then dies in NvencEncoder ("NV12 needs non-zero even dimensions"). Check for a non-zero even source at the top of GpuPipeline::prepare().
  • gpu_ops/gpu_pipeline.cpp:100 — prepare() clears m_inputs but not m_scratch. The contract ("once per picture size") invites a second call, which would leak the first buffers while enqueue() keeps using them, even if they are now too small. Free and clear m_scratch first, or document prepare() as one-shot.
  • bridge/main.cpp:125 — A bad --gpu-step is only parsed in the StreamWorker constructor, after the CUDA context and the LiveKit connection exist, and it exits 1 without usage. Call make_gpu_step() once while parsing and return 2 on std::invalid_argument, like --timestamp-basis.
  • GPU_PROCESSING.md:165 — The doc says to measure step cost with encode in the stats line, but encode times only encoder.encode() (bridge/stream_worker.cpp:250), after the stream sync. Step time never shows there. Only capture-to-ready includes it, and that needs a realtime sender clock. Fix the sentence, or add an enqueue-to-sync stat.
  • README.md:39 — The rectification walk-through is now written three times: in the README, in GPU_PROCESSING.md:133 and in the SineWarpStep comment (sine_warp.hpp:42). The step table appears twice. Keep one copy in GPU_PROCESSING.md and link to it. The README overview (L15, L18) and stream_worker.hpp:75 still say the picture is only copied or box-averaged.
  • bridge/stream_worker.cpp:177 — A failing step still reports "GPU copy from the IPC slot failed" without saying which step. Have GpuPipeline::enqueue() name the failing step, which will help whoever debugs a first rectification step.

Conflicts

  • With gmorgan/CameraStreamingTiming (the head of #1479): 0 commits behind and the local 3-way merge is clean, so no rebase is needed.
  • With open PRs: this sits on top of a stack (#1410 → #1476 → … → #1478 → #1479 → this). It shares 5 files with #1478 and 4 with #1479, all by the same author, with no textual conflicts. Semantic risk: this PR deletes the copy/shrink block and the :DIV start check in stream_worker.cpp that those PRs add, and moves them into copy_nv12 and BoxDownscaleStep. Any review change to that block in #1478 or #1479 has to be ported to gpu_pipeline.cpp, and this PR then rebased. Merge it after #1479. #1410 and #1476 share no files.
Change-by-change (12 files, paths under examples/camera_livekit_bridge/)
File / area What changed Makes sense?
GPU_PROCESSING.md (new) Where steps sit, CLI use, step contract, adding a step, rectification walk-through, what the bridge does around steps ⚠️ encode stat claim is wrong; overlaps README and header
README.md New "GPU steps" section: step table and a summary of how to add a step ⚠️ duplicates GPU_PROCESSING.md; overview lines not updated
bridge/main.cpp Repeatable --gpu-step NAME[=ARGS] into WorkerConfig::gpu_steps; usage text ✅ / ⚠️ validated only after connecting
bridge/stream_worker.hpp WorkerConfig::gpu_steps with doc ✅
bridge/stream_worker.cpp Per-stream GpuPipeline (steps, then box downscale when DIV > 1); size check via prepare(); encoder size from output_size(); per-frame copy lambda replaced by pipeline.enqueue() ✅ same calls on the default path / ⚠️ odd-source start check lost
gpu_ops/CMakeLists.txt Adds sine_warp.cu, gpu_pipeline.cpp ✅
gpu_ops/inc/gpu_ops/gpu_step.hpp (new) ImageSize, GpuStep, GpuPipeline, the two factories ✅ / ⚠️ includes box_downscale.hpp only for the plane types; a small nv12_planes.hpp would be cleaner
gpu_ops/gpu_pipeline.cpp (new) Plain copy, BoxDownscaleStep, pipeline prepare (sizes, scratch) and enqueue, make_gpu_step with the sine-warp parser ✅ / ⚠️ no source check with 0 steps; prepare() not re-entrant
gpu_ops/inc/gpu_ops/sine_warp.hpp (new) SineWarpParams, kernel API, SineWarpStep, rectification how-to comment ✅ clearly labelled as an example
gpu_ops/sine_warp.cu (new) Bilinear luma/chroma ripple, NaN-safe inside test, video-range fill, parameter checks ✅ bounds, rounding and chroma siting check out; zero amplitude is an exact identity
tests/gpu_pipeline/CMakeLists.txt (new) gpu_pipeline ctest, skip code 77 ✅ picked up by the tests/ glob (8 tests, matching the description)
tests/gpu_pipeline/test_gpu_pipeline.cpp (new) No-step copy, identity, ripple, chain, refusals ⚠️ chain check blind to the hand-off; dead in_series; unpadded pitches

Reviewed at e58a463 · 12 files · 2026-11-10 21:50 PST

@jsepulveda-nvidia

Copy link
Copy Markdown
Contributor

Review of e58a463: keeping GPU steps optional, on one CUDA stream, with no host pixel copy is a useful extension point. The explicit distinction between the sine-warp demonstration and calibrated rectification should remain.

The IPC slot-ownership concern on #1478 is inherited here: GPU work reading an imported slot must finish before the producer can reuse it. Please include a deliberately slow step/consumer in that shared ownership stress test; this is one underlying issue, not a request for a second fix in this PR.

For subsequent calibrated rectification, agreeing the frame/lease and calibration contracts with the camera-interface work should allow reuse of these operations without copying an entire pipeline. That coordination is a follow-up, not a request to delay this PR for a broad architecture rewrite. I have not run new GPU hardware tests for this review.

@NVshawnhwang

Copy link
Copy Markdown
Collaborator

Automated review

The PR turns the IPC-to-encoder copy into a configurable series of GPU steps (GpuPipeline) and adds an example sine-warp step. I read the pipeline, the step contract, the sine-warp kernels and the bridge wiring. I found no correctness bug on the default path or the steps path. The one remaining point is a small robustness gap in prepare(). I did not build or run anything.

  1. 🟡 prepare() is not idempotent: scratch buffers accumulate if it is called twice — examples/camera_livekit_bridge/gpu_ops/gpu_pipeline.cpp:107

    prepare() clears m_inputs but only appends to m_scratch, so a second call allocates a new set of buffers and enqueue still indexes the old ones at m_scratch[i]. The current caller invokes it once, so nothing breaks today. Free and clear m_scratch at the top of prepare(), or document that it is single-shot. Allocation also needs a current CUDA context on the calling thread. That holds today because CudaContext's constructor calls make_current() on the main thread, but nothing in the pipeline code states this requirement.

✅ Nothing blocking left from my side — auto-review of this one is now finished.


🤖 Sent by Shawn's Claude — automatic reply. Shawn hasn't reviewed this personally yet; treat it as a first pass.

@farbod-nv

Copy link
Copy Markdown
Contributor

Rechecked e58a46396d93.

Scope / code quality: the GPU processing hook is useful groundwork, but the sine-warp example is not calibrated rectification and should be optional to landing the minimal camera path. Please keep the synthetic warp as an example/test fixture and label the current bridge integration as transitional, with the follow-up to the Camera Capture Interface / Unified Media Bridge boundaries. Rectification remains an application/derived-stream operation consuming safely leased frames, rather than transport-specific camera capture code.

Please restack this onto the selected Chrony bridge; its current base is the closed/superseded UDP timing branch #1479.

For the pipeline test, use a non-identity first step in a composition with a known expected result. The current identity-warp-then-downscale check still passes if the first step is bypassed, so it does not establish the claimed step-to-step handoff. Keep the detailed extension tutorial in GPU_PROCESSING.md rather than duplicating it in the public header.

Review double-checked with Codex on behalf of Farbod; GPU execution not rerun locally.

@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/CameraStreamingSineWarp branch from e58a463 to 73a9a10 Compare November 13, 2026 23:31
@gareth-morgan-nv
gareth-morgan-nv changed the base branch from gmorgan/CameraStreamingTiming to gmorgan/TimeSyncIntegration November 13, 2026 23:31
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/TimeSyncIntegration branch from 45e906c to be6e29f Compare November 13, 2026 23:53
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/CameraStreamingSineWarp branch from 73a9a10 to 470c84e Compare November 13, 2026 23:53
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/TimeSyncIntegration branch from be6e29f to fd9c28a Compare November 16, 2026 17:07
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/CameraStreamingSineWarp branch 2 times, most recently from 63006de to 14674b8 Compare November 16, 2026 17:12
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/TimeSyncIntegration branch from 3b33025 to dcc0067 Compare November 16, 2026 19:30
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/CameraStreamingSineWarp branch from 74fbe2d to 245dec9 Compare November 16, 2026 19:30
@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Re: @farbod-nv's review — #1481 (comment)

Addressed (head 245dec9f1, stacked on #1519, the selected Chrony bridge).

  • Pipeline test. It now composes a non-identity sine warp with a downscale and compares the result with a host-side box downscale of the separately warped picture, and asserts that skipping the warp, or applying the steps in the other order, gives a different result. It passed on an RTX PRO 5000. The warp used for the expected picture is the implementation's own, so this checks the step handoff, not the warp's mathematics.
  • Scope. The sine warp stays an example and test fixture; it is not calibrated rectification.
  • Boundary. The README identifies the bridge wiring as transitional and GPU processing as an application/derived-stream responsibility. It distinguishes completion-protected bridge buffers from the receiver input, whose ownership issue remains open (examples/camera_livekit_bridge: publish receiver frames to LiveKit #1478).

gareth-morgan-nv and others added 5 commits November 18, 2026 11:36
…cture, with a sine-warp example

The copy between the receiver's IPC buffer and the encoder input becomes a
pipeline of GPU steps (GpuStep) that run in order, with no host copy. None run by
default; --gpu-step NAME[=ARGS] adds one, repeatable, before the stream's own
downscale. The existing downscale is now a step. sine-warp is an example step, not
calibrated rectification; its comments describe how a rectification step would be
written the same way (a remap table computed once from the camera calibration,
sampled per frame for luma and chroma). Adding a step is one class plus one line
in make_gpu_step().

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
How a picture flows from the receiver's IPC buffer through the GPU steps to the
encoder, the step contract, how to add a step, and how lens rectification would be
written as one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ation boundary

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…-identity first step

The pipeline test composed an identity warp with a downscale, which still passes when the first step is
bypassed. It now composes the sine warp with a downscale and compares against a host-side box downscale of the
separately warped picture, and asserts that skipping the warp or reversing the order gives a different result.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…eased receiver input

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>

This branch was successfully deployed

1 active deployment
dev — 09588a83 Deployed Nov 18, 2026 by gareth-morgan-nv via publish-wheel #7074
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants