Repository navigation
examples/camera_livekit_bridge: run a series of GPU steps on each picture, with a sine-warp example - #1481
examples/camera_livekit_bridge: run a series of GPU steps on each picture, with a sine-warp example#1481gareth-morgan-nv wants to merge 5 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
cef0850 to
e58a463
Compare
57c7ab4 to
eecc167
Compare
| // rectification runs: see GpuStep in gpu_ops/gpu_step.hpp. The encode below waits for this stream. | ||
| const Nv12Planes in{ base + layout.y_offset, base + layout.uv_offset, layout.y_pitch, layout.uv_pitch }; | ||
| const Nv12PlanesOut out{ surface.view.y, surface.view.uv, surface.view.y_pitch, surface.view.uv_pitch }; | ||
| const cudaError_t copy_status = pipeline.enqueue(in, out, copy_stream); |
There was a problem hiding this comment.
@NVshawnhwang this is where rectification goes. Between the receiver's CUDA IPC buffer and the encoder input, each picture now passes through a series of GPU steps (gpu_ops/inc/gpu_ops/gpu_step.hpp), none enabled by default. sine-warp is an example step only, not rectification. A rectification step is the same shape: compute the remap table from the camera calibration once in prepare(), upload it, and sample through it per frame for luma and chroma. SineWarpStep in gpu_ops/inc/gpu_ops/sine_warp.hpp is the template and its comment walks through it; GPU_PROCESSING.md covers how to add a step.
🤖 Agent-generated comment (Claude Code).
NVdejung
left a comment
There was a problem hiding this comment.
🤖 Automated first-pass review (Claude, for Dejun) — I'll follow up personally where needed.
Summary — This PR turns the copy from the receiver's IPC slot to the encoder input into an ordered GpuPipeline of GpuSteps on the worker's CUDA stream. It makes the :DIV box downscale the last step and adds an opt-in --gpu-step flag with a sine-warp example, a gpu_pipeline test and a design doc. I read every hunk plus the worker, encoder and CUDA-context code around them. With no flag the bridge issues the same copy and shrink calls as before. prepare() allocates and the worker thread launches in the same primary context, and the warp kernels stay in bounds. No blockers. The main gaps are a series test that cannot see a broken step-to-step hand-off, and a start-time size check that was lost.
Must fix
None found. Internal names: the scan of the added lines, title, description and commits found nothing, and the new docs name no people, stations or internal resources.
Suggestions
tests/gpu_pipeline/test_gpu_pipeline.cpp:120— The series check chainssine-warp=0with the downscale. The first step is an identity, so the test still passes if the downscale reads the source instead ofm_scratch[0], or if the first step never runs. The hand-off that--gpu-stepplus:DIVrelies on is therefore untested.- The "two warps differ from one" check named in the comment does not exist.
in_series(L125) is computed and discarded, andshrunkholds 128x96.- Fix: compare
sine-warp=8,32followed by the downscale against the downscale of a separatesine-warp=8,32output. - Give source and destination padded pitches as
test_box_downscale.cppdoes: every buffer here has pitch == width, so a width-for-pitch slip in a step would still pass. - The default path was rewritten and the description says the no-flag live run was not repeated on this head. One short run with and without
:DIVwould close that.
gpu_ops/gpu_pipeline.cpp:97— Silent behaviour change. The old constructor ranbox_downscale_supported(w, h, 1)on every stream, which refused an odd or zero source at start. Now, with no steps and no:DIV, nothing checks the source: the track is published and the worker then dies inNvencEncoder("NV12 needs non-zero even dimensions"). Check for a non-zero even source at the top ofGpuPipeline::prepare().gpu_ops/gpu_pipeline.cpp:100—prepare()clearsm_inputsbut notm_scratch. The contract ("once per picture size") invites a second call, which would leak the first buffers whileenqueue()keeps using them, even if they are now too small. Free and clearm_scratchfirst, or documentprepare()as one-shot.bridge/main.cpp:125— A bad--gpu-stepis only parsed in theStreamWorkerconstructor, after the CUDA context and the LiveKit connection exist, and it exits 1 without usage. Callmake_gpu_step()once while parsing and return 2 onstd::invalid_argument, like--timestamp-basis.GPU_PROCESSING.md:165— The doc says to measure step cost withencodein the stats line, butencodetimes onlyencoder.encode()(bridge/stream_worker.cpp:250), after the stream sync. Step time never shows there. Only capture-to-ready includes it, and that needs a realtime sender clock. Fix the sentence, or add an enqueue-to-sync stat.README.md:39— The rectification walk-through is now written three times: in the README, inGPU_PROCESSING.md:133and in theSineWarpStepcomment (sine_warp.hpp:42). The step table appears twice. Keep one copy inGPU_PROCESSING.mdand link to it. The README overview (L15, L18) andstream_worker.hpp:75still say the picture is only copied or box-averaged.bridge/stream_worker.cpp:177— A failing step still reports "GPU copy from the IPC slot failed" without saying which step. HaveGpuPipeline::enqueue()name the failing step, which will help whoever debugs a first rectification step.
Conflicts
- With
gmorgan/CameraStreamingTiming(the head of#1479): 0 commits behind and the local 3-way merge is clean, so no rebase is needed. - With open PRs: this sits on top of a stack (
#1410→#1476→ … →#1478→#1479→ this). It shares 5 files with#1478and 4 with#1479, all by the same author, with no textual conflicts. Semantic risk: this PR deletes the copy/shrink block and the:DIVstart check instream_worker.cppthat those PRs add, and moves them intocopy_nv12andBoxDownscaleStep. Any review change to that block in#1478or#1479has to be ported togpu_pipeline.cpp, and this PR then rebased. Merge it after#1479.#1410and#1476share no files.
Change-by-change (12 files, paths under examples/camera_livekit_bridge/)
| File / area | What changed | Makes sense? |
|---|---|---|
GPU_PROCESSING.md (new) |
Where steps sit, CLI use, step contract, adding a step, rectification walk-through, what the bridge does around steps | encode stat claim is wrong; overlaps README and header |
README.md |
New "GPU steps" section: step table and a summary of how to add a step | GPU_PROCESSING.md; overview lines not updated |
bridge/main.cpp |
Repeatable --gpu-step NAME[=ARGS] into WorkerConfig::gpu_steps; usage text |
✅ / |
bridge/stream_worker.hpp |
WorkerConfig::gpu_steps with doc |
✅ |
bridge/stream_worker.cpp |
Per-stream GpuPipeline (steps, then box downscale when DIV > 1); size check via prepare(); encoder size from output_size(); per-frame copy lambda replaced by pipeline.enqueue() |
✅ same calls on the default path / |
gpu_ops/CMakeLists.txt |
Adds sine_warp.cu, gpu_pipeline.cpp |
✅ |
gpu_ops/inc/gpu_ops/gpu_step.hpp (new) |
ImageSize, GpuStep, GpuPipeline, the two factories |
✅ / box_downscale.hpp only for the plane types; a small nv12_planes.hpp would be cleaner |
gpu_ops/gpu_pipeline.cpp (new) |
Plain copy, BoxDownscaleStep, pipeline prepare (sizes, scratch) and enqueue, make_gpu_step with the sine-warp parser |
✅ / prepare() not re-entrant |
gpu_ops/inc/gpu_ops/sine_warp.hpp (new) |
SineWarpParams, kernel API, SineWarpStep, rectification how-to comment |
✅ clearly labelled as an example |
gpu_ops/sine_warp.cu (new) |
Bilinear luma/chroma ripple, NaN-safe inside test, video-range fill, parameter checks | ✅ bounds, rounding and chroma siting check out; zero amplitude is an exact identity |
tests/gpu_pipeline/CMakeLists.txt (new) |
gpu_pipeline ctest, skip code 77 |
✅ picked up by the tests/ glob (8 tests, matching the description) |
tests/gpu_pipeline/test_gpu_pipeline.cpp (new) |
No-step copy, identity, ripple, chain, refusals | in_series; unpadded pitches |
Reviewed at e58a463 · 12 files · 2026-11-10 21:50 PST
|
Review of e58a463: keeping GPU steps optional, on one CUDA stream, with no host pixel copy is a useful extension point. The explicit distinction between the sine-warp demonstration and calibrated rectification should remain. The IPC slot-ownership concern on #1478 is inherited here: GPU work reading an imported slot must finish before the producer can reuse it. Please include a deliberately slow step/consumer in that shared ownership stress test; this is one underlying issue, not a request for a second fix in this PR. For subsequent calibrated rectification, agreeing the frame/lease and calibration contracts with the camera-interface work should allow reuse of these operations without copying an entire pipeline. That coordination is a follow-up, not a request to delay this PR for a broad architecture rewrite. I have not run new GPU hardware tests for this review. |
Automated reviewThe PR turns the IPC-to-encoder copy into a configurable series of GPU steps (
✅ Nothing blocking left from my side — auto-review of this one is now finished. 🤖 Sent by Shawn's Claude — automatic reply. Shawn hasn't reviewed this personally yet; treat it as a first pass. |
|
Rechecked Scope / code quality: the GPU processing hook is useful groundwork, but the sine-warp example is not calibrated rectification and should be optional to landing the minimal camera path. Please keep the synthetic warp as an example/test fixture and label the current bridge integration as transitional, with the follow-up to the Camera Capture Interface / Unified Media Bridge boundaries. Rectification remains an application/derived-stream operation consuming safely leased frames, rather than transport-specific camera capture code. Please restack this onto the selected Chrony bridge; its current base is the closed/superseded UDP timing branch #1479. For the pipeline test, use a non-identity first step in a composition with a known expected result. The current identity-warp-then-downscale check still passes if the first step is bypassed, so it does not establish the claimed step-to-step handoff. Keep the detailed extension tutorial in Review double-checked with Codex on behalf of Farbod; GPU execution not rerun locally. |
e58a463 to
73a9a10
Compare
45e906c to
be6e29f
Compare
73a9a10 to
470c84e
Compare
be6e29f to
fd9c28a
Compare
63006de to
14674b8
Compare
3b33025 to
dcc0067
Compare
74fbe2d to
245dec9
Compare
|
🤖 Agent-generated (Claude Code · claude-sonnet-5-5) Re: @farbod-nv's review — #1481 (comment) Addressed (head
|
…cture, with a sine-warp example The copy between the receiver's IPC buffer and the encoder input becomes a pipeline of GPU steps (GpuStep) that run in order, with no host copy. None run by default; --gpu-step NAME[=ARGS] adds one, repeatable, before the stream's own downscale. The existing downscale is now a step. sine-warp is an example step, not calibrated rectification; its comments describe how a rectification step would be written the same way (a remap table computed once from the camera calibration, sampled per frame for luma and chroma). Adding a step is one class plus one line in make_gpu_step(). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
How a picture flows from the receiver's IPC buffer through the GPU steps to the encoder, the step contract, how to add a step, and how lens rectification would be written as one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ation boundary Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…-identity first step The pipeline test composed an identity warp with a downscale, which still passes when the first step is bypassed. It now composes the sine warp with a downscale and compares against a host-side box downscale of the separately warped picture, and asserts that skipping the warp or reversing the order gives a different result. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…eased receiver input Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
dcc0067 to
e9a6282
Compare
245dec9 to
09588a8
Compare
Summary
The copy between the receiver's CUDA IPC buffer and the encoder input becomes a series of GPU steps (
GpuStep) that run in order on one CUDA stream, with no host copy. None run by default: with no flag the picture is only copied or shrunk, as before.--gpu-step NAME[=ARGS]adds a step, repeatable, run in the order given at the source size; a stream's:DIVdownscale (now itself a step) runs after them.sine-warpis an example step only. It is not calibrated rectification; it shows the shape of a step.make_gpu_step().GPU_PROCESSING.mddescribes the system, the step contract and how lens rectification would be written as a step (a remap table computed once from the calibration and sampled per frame for luma and chroma).Stacked on the capture-time PR (it touches the same bridge files).
Testing
ctest 8/8, including a new
gpu_pipelinetest (no steps copies, identity warp equals a copy, a visible change otherwise, steps run in series, bad names, parameters and sizes are refused). The series test composes a non-identity sine warp with a downscale and compares the result with a host-side box downscale of the separately warped picture; it also asserts that skipping the warp, or applying the steps in the other order, gives a different result. It passed on an RTX PRO 5000 on this head's stack; the warp used for the expected picture is the implementation's own, so this tests the step handoff, not the warp's mathematics. A live run through the mock, receiver and bridge with--gpu-step sine-warp=14,90ran at 30 fps with 0 slot overruns and the published picture shows the ripple. The default path (no flag) is covered by the unit test; I did not repeat the live run without the flag on this head. Nothing here has run on real cameras.Integration boundary
GPU operations run on camera images as application or derived-stream processing; running them inside this bridge is temporary wiring. The processing path keeps input and output on the GPU. The bridge's own scratch and encoder buffers are tied to GPU completion; the receiver's input slot is read without a lease today (the slot-ownership limitation described in #1478), so reading that input safely depends on the ownership contract being added on the receiver side. The processing must keep GPU-resident buffers and completion-based lifetime when it moves behind the camera reader interface. The sine-warp step demonstrates kernel integration; it is not a calibrated rectification implementation.
gpu_ops/and the GPU step pipeline inbridge/stream_worker.cpp🤖 Generated with Claude Code