Skip to content

[CameraInterface] Add camera_bench and log what each camera does - #1531

Closed
farbod-nv wants to merge 16 commits into
fm/camera_capture_m0_pluginfrom
fm/camera_capture_m0_bench
Closed

farbod-nv wants to merge 16 commits into
fm/camera_capture_m0_pluginfrom
fm/camera_capture_m0_bench

Conversation

@farbod-nv

Copy link
Copy Markdown
Contributor

Description

Ninth pull request of the camera interface, stacked on #1530 (the camera_plugin executable), whose branch is this one's base.

It adds the tool that measures the camera interface, and the plugin's own account of what its cameras do.

  • camera_bench reads one group from a running camera plugin, like any other reader, and prints where each frame set spent its time. Every frame set carries those times, so the bench only has to read them:

    $ camera_bench --endpoint=cameras.sock --group=head --frames=600 --check-synthetic
    head: 2 stream(s)
      head/left                1280x720 bgra8, 60.00 fps, 8 slots of 3686400 bytes
      head/right               1280x720 bgra8, 60.00 fps, 8 slots of 3686400 bytes
    read 600 frame sets in 10.00 s (60.0 per second), skipped 0
                                    p50        p90        p99        max   (microseconds)
      capture -> sdk ready        158.8      434.1     2496.3     3245.7
      sdk ready -> published        0.5        0.6        0.8        1.4
      published -> sent             1.2        1.6        2.3        3.7
      sent -> reader               46.0       63.4      121.8      184.7
      capture -> reader           210.0      494.2     2525.5     3273.3
    synthetic pattern: 0 image(s) did not match
    

    That run is the synthetic 720p pair at 60 fps read from a second process on an idle 24-core workstation with default scheduling. published -> sent and sent -> reader are what the plugin and the process boundary add; capture -> sdk ready is the synthetic camera filling its two images. With a second reader running at the same time the figures stayed in the same range, and the plugin used about 2% of one core.

  • Without --group the bench lists what the plugin publishes. --window and --order read as a recorder would. --check-synthetic checks every byte of every image against the synthetic backend's pattern, which makes the bench a correctness check as well.

  • The plugin logs what each camera did every ten seconds: frame sets published and readers connected. It warns when a camera had to drop frame sets because its pool was full, or when a reader did not take its events.

Type of change

  • New feature (non-breaking change which adds functionality)

Testing

  • tests/cpp/plugins/camera/test_camera_bench.cpp (Catch2, 1 case, tagged unit): with a synthetic camera served from the test process, the bench is run as a process of its own. It lists the group; reads 50 frame sets with every byte checked and reports each step; reads in order with a window of three; fails with the plugin's reason for an unknown group, with exit code 2 for arguments it cannot use, and with "no camera plugin answers" once the host has stopped. Run with ctest -L camera_plugin.
  • All 70 camera tests pass through CTest, and under AddressSanitizer with UBSan; the endpoint, reader and plugin suites ran three times each under ThreadSanitizer with no reports. 15 repeated runs of the plugin suite without a failure.
  • Built through the repository's own build (BUILD_PLUGINS=ON) with the project warning set promoted to errors.
  • Ubuntu 22.04 x86_64, GCC 11.4, Linux 6.8.
  • clang-format-14 reports no changes; SKIP=check-copyright-year pre-commit run --all-files passes.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation (src/plugins/camera/README.md)
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

🤖 Generated with Claude Code

farbod-nv and others added 16 commits November 13, 2026 16:28
… capture plugin

First step of the unified camera capture plugin (milestone M0). camera_core
tracks which pool slots hold which frame sets and who may still read them.
It uses the standard library only and never touches slot memory, so it
builds and is tested on every platform.

- A group is one or more hardware-synchronised streams published as one
  frame set.
- A reader subscribes with a window (frame sets kept for it at a time) and
  a delivery order, newest or in order.
- A subscription is accepted only when the pool covers it:
  slots >= ring history + sum of reader windows + backend reserve.
- A backend either takes a free slot and writes it, or hands over a slot
  the camera SDK filled and is told the moment nobody holds it any more.
- Every frame event carries its timeline, the group's generation and the
  number of frame sets the reader skipped.

Tests: tests/cpp/core/camera_core (Catch2), run with `ctest -L camera_core`.

Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…format

H.264 and H.265 are codecs, so they do not belong in PixelFormat. A stream
description now has a Codec (NONE for pixels) next to its PixelFormat, and
is_encoded() reads the codec.

Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Two more places where the stream description and the publisher were too
coarse for real cameras:

- A stream had a single stride. NV12 has two planes, and on Jetson each
  plane has its own offset and stride, so a stream now lists its planes.
  add_group rejects a pixel stream without a size or without one stride
  per plane of its format, and an encoded stream that lists planes.
- Counters were totals over every group. They are now kept per group, so
  a tool can tell which camera drops frames.

Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ory kind

The README explains the terms (stream, group, frame set, window, order,
lease, generation), the two ways a backend gets a slot, and the pool
budget, so a reviewer does not need anything outside the repository.

MemoryKind listed kinds that nothing implements yet. It now has HOST
only; each further kind is added together with its pool.

Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Core tests carry the unit tag, which becomes the CTest label that
`ctest -L unit` selects.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
The tests start threads. Without the link, std::thread fails at run time
on C libraries that keep threads in a separate library.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
add_group now checks that every plane of a pixel stream lies inside the
slot (row length against stride, last row against the slot size) and that
a pool can hold one frame set on top of the backend's reserve. publish
refuses an image larger than its slot.

EventSink::send_event is noexcept: it runs on the capture path with the
publisher's lock held, and a throwing sink would have left a slot held.

A group with a slot-free callback hands out its slots with claim_slot
only, so a slot cannot be taken again before its callback has run.

Adds agent notes for the module.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
HostPool is one stream's slots in shared host memory: a single anonymous
memory file with the slots a page-aligned stride apart. The owner maps it
writable and then seals it, so a process that is sent the descriptor can
map it for reading only and cannot resize it. HostPoolView is that
reader's mapping; both sides derive the layout from the slot count and
slot size. UniqueFd owns a file descriptor.

An anonymous sealed file is used rather than named shared memory: there
is no name to collide or to clean up after a crash, and only a process
that was handed the descriptor can map it.

Linux only, so the module and its tests are added under a platform check.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
MessageSocket and MessageListener are a connection between two processes
on one machine: a Unix domain socket of the sequenced-packet kind that
carries whole messages in order, each with optional file descriptors.
This is how a pool's descriptor, and later the per-frame events, reach a
reader.

- Sending never blocks. A full queue and a closed peer are statuses, so
  capture cannot stall on a reader that stopped reading.
- Receiving waits with a timeout and returns the descriptors as owned
  UniqueFds. A message that does not fit is an error, not a silent cut.
- A lock file next to the socket file decides which process owns the
  path. The kernel drops the lock when its holder dies, so a socket file
  found while holding it is stale and is replaced; anything that is not
  a socket is left alone.
- The socket file is created with mode 0600.

The README explains why a Unix socket and shared memory are used rather
than a messaging library.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
…reader

The plugin and a reader exchange six messages over their socket: Hello and
Welcome (the catalog of what is published), Subscribe and Subscribed (with
one pool descriptor per stream), then Frame per frame set and Release when
the reader is done with it. A connection carries one subscription.

The messages are FlatBuffers tables in a schema kept inside the module, so
a reader and a plugin from different builds keep understanding each other:
the schema only grows, and a message type a build does not know is ignored.
It is not under src/core/schema because these messages are never recorded.

WireEncoder and decode_wire_message convert to and from camera_core types,
so nothing outside the module sees FlatBuffers. Encoding a frame event
allocates nothing once the encoder's buffer has grown. Decoding treats its
input as coming from another process and checks it in full.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
Endpoint puts a Publisher on a socket and speaks the camera_wire messages:
it tells each reader what is published, and on a subscription sends the
group's pool descriptors and from then on its frame events.

One thread accepts readers and handles the handshake, subscriptions and
releases. Frame events are not sent from it: they are encoded and sent on
the publishing thread, inside Publisher::publish, with a send that does not
block, so nothing sits between a frame set being published and its event
being on the socket.

A closed connection ends the reader's subscription and returns every frame
set it held, whether the reader exited or was killed. A reader that breaks
the protocol is dropped without affecting the others.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
stop() ended the endpoint's thread but kept the listening socket, so a
reader that connected afterwards was accepted by the kernel and then waited
for an answer nobody would send. stop() now closes the listener, which
removes the socket file and releases the path: a reader that tries to
connect is refused at once. start() takes the path again.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
…meras

CameraReader::open subscribes to one group of the camera plugin and next()
hands out frame sets. A FrameSet is a hold on the frames: its images are
the plugin's own buffers, mapped read-only, and stay as they are until the
object is destroyed or released. Nothing is copied.

With newest order and a window larger than one, next() takes the newest
frame set that is waiting, gives the older ones back and counts them as
skipped. Frame sets that arrive before the plugin's answer to the
subscription are kept and handed out first.

A frame set can be given back from another thread and after its reader is
gone. A plugin that goes away is reported as CameraDisconnected; opening
fails with the plugin's own reason when a subscription is refused.

camera_core now exposes the two rules the reader shares with the publisher,
resolved_order and plane_size_of, instead of keeping them private.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
A backend opens cameras through one SDK. It implements CameraSource for a
configured camera: describe() says what the camera publishes, start() gets
a FrameOutput to acquire slots from and publish frame sets to.

CameraHost is the plugin process without its main(): it allocates each
camera's pools in shared host memory, registers the camera with one
publisher, and serves all cameras to readers on one endpoint.

A camera is configured as ID:BACKEND[:KEY=VALUE,...]. OptionReader makes a
backend refuse options it does not read, so a misspelt option is an error
instead of a setting that is silently ignored. A camera that names a
backend the build does not have is refused with the list of those it has.

The executable, the list of built backends and the synthetic backend follow
in the next pull request.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
camera_plugin owns the cameras named on its command line and serves their
frames to readers on one endpoint:

  camera_plugin --endpoint=PATH --camera=ID:BACKEND[:KEY=VALUE,...]

SIGINT and SIGTERM stop it cleanly. A misspelt argument is an error, and a
second plugin cannot take the endpoint of a running one.

Which backends a plugin has is decided when it is built: backends/ lists
them, each behind its own CMake option, and a backend that is off is not
configured at all, so its SDK is neither looked for nor built. The synthetic
backend needs no SDK and is always built. It publishes frames whose every
byte is known from the frame number, so a reader can check that it received
the whole of the frame it was told about.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
camera_bench reads one group from a running camera plugin, like any other
reader, and prints where each frame set spent its time: capture to SDK
ready, to published, to sent, to the reader, as percentiles. Without a
group it lists what the plugin publishes. It can read with a larger window
or in order, and check every byte against the synthetic backend's pattern.

The plugin now logs every ten seconds how many frame sets each camera
published, and warns when a camera dropped frame sets for lack of a free
slot or a reader did not take its events.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Farbod Motlagh <fmotlagh@nvidia.com>
@coderabbitai

coderabbitai Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@farbod-nv

Copy link
Copy Markdown
Contributor Author

Folded into #1530. The first milestone of the camera interface is now four pull requests instead of twelve: #1521, #1528, #1530 and #1534.

This branch was successfully deployed

1 active deployment
dev — 8e29524c Deployed Nov 14, 2026 by farbod-nv via publish-wheel #6956
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants