Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/build-ubuntu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,7 @@ jobs:
"deviceio_session",
"schema",
"plugin_manager",
"camera",
):
importlib.import_module(f"isaaccapture.{mod}")

Expand Down
4 changes: 3 additions & 1 deletion src/core/camera/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Read [`README.md`](README.md) first: it has the terms (stream, group, window, or
- **Releasing must never throw or block.** It runs in destructors, possibly on another thread; a plugin that is gone takes its frames back by itself.
- **The plugin is trusted for content, not for bounds.** Check a frame event against the group before handing out views, and check the handshake answer in `open` before mapping anything: the protocol version, that it is the group that was asked for, `problem_with` for the description, one pool per stream.
- **A newest-order reader asks; it does not drain.** `next` sends the request and takes the answer. Do not go back to reading whatever is queued and keeping the last: what is queued was held when it was published.
- **In a binding, whatever exposes image memory holds the frame set.** An array's base is the lease, not just the mapping, so an array never shows another frame; a scope that must not keep anything checks that count when it ends. Do not hand out a view that outlives its lease, and do not build a scope on something that can be released under a view (a `memoryview` released while `numpy.frombuffer` still points at it dangles).
- **A full window is an error, not a wait.** `next` by a reader that holds its window's worth throws `CameraWindowExhausted`; it does not time out as if no frame had come.
- **An empty location looks; a named one does not.** With no endpoint and no instance the reader asks every endpoint `default_endpoint_paths()` lists. With either, it talks to that one plugin alone: the full handshake timeout, and an error if the plugin is not there. Do not fall back from one to the other.
- **Looking stays plain.** The endpoints are asked one after another, each with a short fixed timeout. A refused connection is passed over silently. An endpoint that fails to say what it publishes (no answer in time, a refusal, a closed connection, any other failure) is passed over too, named with its reason in `open`'s error and left out by `list`; do not narrow that to the timeout. A group on two endpoints is an error, never a choice. No cache, no index file, no retry, no thread, and no option for any of it.
Expand Down Expand Up @@ -95,5 +96,6 @@ Read "What runs where" in the [`README.md`](README.md) before changing it.
- **A test does not assume which thread runs first.** A reader's request and the camera's publish come from two threads and are served in either order: have the reader ask for what comes after a frame set it names, not for "anything", when the test expects a particular one. Run the suite on one CPU (`taskset -c 0`) before pushing; CI's slower machines find these, a workstation does not.
- **A test waits for what the plugin's thread does; it does not assume it.** `next` returning says nothing about whether the plugin has taken the reader's request, so wait for the effect (`Served::wait_until_sent`) before counting what was sent.
- **A test's helper thread is stopped and joined however the test ends.** A failed `REQUIRE` throws, and a `std::thread` that is still joinable then aborts the process instead of reporting the failure. Use `std::jthread`, or a guard whose destructor joins.
- **Tests that look through the user's plugins, or let a plugin choose its endpoint or its configuration file, get a user's directories of their own.** Point `XDG_RUNTIME_DIR`, and for a plugin's configuration file `XDG_CONFIG_HOME`, at a short temporary directory and put them back, as `EndpointDirectory` (`tests/cpp/core/camera/endpoint_directory.hpp`) and `UserDirectories` (`tests/cpp/core/camera/test_camera_plugin.cpp`) do; a test that uses the real ones finds whatever camera plugin and `cameras.yaml` the developer has.
- **Tests that look through the user's plugins, or let a plugin choose its endpoint or its configuration file, get a user's directories of their own.** Point `XDG_RUNTIME_DIR`, and for a plugin's configuration file `XDG_CONFIG_HOME`, at a short temporary directory and put them back, as `EndpointDirectory` (`tests/cpp/core/camera/endpoint_directory.hpp`) and `UserDirectories` (`tests/cpp/core/camera/test_camera_plugin.cpp`) do, and the `endpoint_directory` fixture of `tests/python/core/camera` for a Python test and the plugins it starts; a test that uses the real ones finds whatever camera plugin and `cameras.yaml` the developer has.
- **A test that runs `run_camera_plugin` in the test process aims the stop signal at the plugin's thread**, and starts that thread with the signal blocked (`PluginRun` in `tests/cpp/core/camera/test_camera_plugin.cpp`). A signal sent to the process would end the test instead.
- **A Python test starts plugins through `start_plugin`.** The fixture stops every process it started and waits for it, also when the test fails, before the directory with the socket files is removed. A plugin started through the plugin manager is stopped in a `finally`.
4 changes: 4 additions & 0 deletions src/core/camera/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,7 @@
cmake_minimum_required(VERSION 3.20)

add_subdirectory(cpp)

if(BUILD_PYTHON_BINDINGS)
add_subdirectory(python)
endif()
53 changes: 51 additions & 2 deletions src/core/camera/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,45 @@ Not covered: a process that is stopped while it waits inside `next` is handed, w
continues, the frame set that arrived meanwhile. A reader that needs to know how old its
frame is has the times of every step in `timeline()`.

### Python

The same reader as `isaaccapture.camera`:

```python
from isaaccapture.camera import CameraReader

with CameraReader.open("head") as reader:
for frames in reader:
left = frames.image(0).array() # (height, width, 4) uint8, read-only
```

- An array is a view of the plugin's buffer, not a copy, and cannot be written to. **It
keeps its frame set held**: it shows that frame for as long as the array exists, after
`release()`, after the loop has moved on and after the reader is closed. The price is the
window: an array that is kept counts as a frame set the reader holds.
- `open` takes the window and the order (`"newest"`, `"in_order"`) as keyword arguments. The
window is two by default, not one as in C++: in the loop above `left` still names the array
of the frame before while the next frame set is asked for. The order is `"newest"` unless
`"in_order"` is given; a larger window alone does not change it, as it does in C++.
- **Which plugin** is as in C++ ("How a reader finds its camera"). `open` and `list` look
through this user's camera plugins; `instance=` or `endpoint=` names one plugin instead.
Each `GroupListing` of `list` has the `endpoint` of the plugin that publishes the group.
- A reader that holds its window's worth raises `CameraWindowExhausted` from `next()`, with
what to do about it: drop an array, or copy what is needed from it.
- `for frames in reader` lets go of the frame set before at each step. With `reader.next()`
the caller does: `frames.release()`, or a `with` block.
- `with frames.borrow() as frames:` is for code that must not keep anything: the frame set
goes back to the plugin when the block ends, and an array of it that still exists then
raises `BufferError` instead of holding the frame set without a word.
- Arrays are NumPy arrays over host memory. A DLPack export is added with the first pool
kind that is not host memory, where it is the way to hand a frame to a tensor library.
- `bgra8` and `rgba8` images are `(height, width, 4)`, `gray8` is `(height, width)`; `nv12`
has two planes, `plane(0)` and `plane(1)`. `data` is the flat bytes, also for encoded video.
- The synthetic camera plugin ships in the package under `plugins/camera_synthetic`. The
plugin manager, given `BUNDLED_PLUGINS_DIR` (the package's `plugins` directory, from
`isaaccapture.teleop_session_manager`) as its search path, starts it by its name,
`camera_synthetic`.

## Publishing cameras

A camera plugin is one vendor's or one SDK's capture code and this library. The capture code
Expand Down Expand Up @@ -297,7 +336,9 @@ its camera restarted.
vendor or SDK called `<name>`:

1. **Directory.** Copy `src/plugins/camera_synthetic` to `src/plugins/camera_<name>`, without
`tools/`: `camera_bench` reads any camera plugin and stays where it is.
`tools/`: `camera_bench` reads any camera plugin and stays where it is. Leave out the block
in `app/CMakeLists.txt` that bundles the plugin into the Python package as well: the wheel
does not ship a vendor's SDK.
2. **`core/`: the backend.** A static library `camera_<name>_core` that links
`camera::camera` and the SDK. It implements `CameraSource` for one camera and has one
function that returns the `Backend`, named `<name>`. No type of the SDK leaves this
Expand Down Expand Up @@ -697,7 +738,8 @@ its camera asks that of every plugin of its user. Nothing is logged per frame.

`tests/cpp/core/camera`, built as one executable, `camera_tests`. `ctest -L camera` runs them. A
label is a regular expression, so that also runs the tests of the camera plugins, whose labels
start with `camera_`; `ctest -L "^camera$"` is the form that selects this library's tests alone.
start with `camera_`, and in a build with the Python bindings the Python tests below
(`camera_python`); `ctest -L "^camera$"` is the form that selects this library's tests alone.

- **Unit** (`ctest -L unit`): everything that runs in one process. The endpoint's cases play the
reader's part with the socket and the codec directly. The reader's cases run it against a
Expand Down Expand Up @@ -725,3 +767,10 @@ start with `camera_`; `ctest -L "^camera$"` is the form that selects this librar
while it holds frame sets gives them back; a reader is told when the plugin's process is
killed and can still read what it holds; a reader finds two groups on two plugin processes by
name, and still finds the one after the other was killed.

`tests/python/core/camera` (`ctest -L camera_python`) reads from the synthetic camera plugin
that is bundled into the package, started as a process and, in one case, through the plugin
manager. Besides the reading, the cases cover the finding, each in a user's directories of
its own: a reader that is told nothing finds the plugin; a listing names the endpoint of its
plugin; two plugins side by side are both found by group; a group on two plugins, and a
group nobody publishes, are errors that name the endpoints and the directory.
18 changes: 18 additions & 0 deletions src/core/camera/python/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# The camera reader for Python: isaaccapture.camera._camera.
pybind11_add_module(camera_py
camera_bindings.cpp
)

target_link_libraries(camera_py
PRIVATE
camera::camera
)

set_target_properties(camera_py PROPERTIES
OUTPUT_NAME "_camera"
# Use genex in output directory - CMake won't add another config subdirectory
LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/isaaccapture/camera"
)
Loading
Loading