Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion src/core/camera/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,21 @@ Read [`README.md`](README.md) first: it has the terms (stream, group, window, or
- **An empty location looks; a named one does not.** With no endpoint and no instance the reader asks every endpoint `default_endpoint_paths()` lists. With either, it talks to that one plugin alone: the full handshake timeout, and an error if the plugin is not there. Do not fall back from one to the other.
- **Looking stays plain.** The endpoints are asked one after another, each with a short fixed timeout. A refused connection is passed over silently. An endpoint that fails to say what it publishes (no answer in time, a refusal, a closed connection, any other failure) is passed over too, named with its reason in `open`'s error and left out by `list`; do not narrow that to the timeout. A group on two endpoints is an error, never a choice. No cache, no index file, no retry, no thread, and no option for any of it.

## Publishing cameras

These rules hold for the publishing side of this library and for every camera plugin built on it (`src/plugins/camera_*`).

- **One camera plugin per vendor or SDK.** A plugin links exactly one camera SDK and hands exactly one `Backend` to `run_camera_plugin`. Do not add a second SDK to an existing plugin, a list of backends, or run-time loading of backends. A plugin that needs an SDK gets its own `BUILD_PLUGIN_*` option, off by default, like the repository's other vendor plugins. The steps are in [`README.md`](README.md), "Writing a camera plugin".
- **No SDK types outside a plugin's `core/` directory.** This library, a plugin's `app/` and its `plugin.yaml` see `CameraSource` and this library's own types only, and readers never link a camera SDK.
- **A plugin's `main` calls `run_camera_plugin` and does nothing else.** The arguments, the configuration file, the endpoint and the exit codes are written once, here, so they are the same for every camera plugin.
- **A plugin serves its own cameras.** The configuration file is shared by the camera plugins of a user, and its form is checked whole: an entry that is not an id with a `backend:` and `name: value` settings stops every plugin that reads the file, whichever backend it names. Of a well-formed file a plugin takes the cameras whose `backend:` is its backend's name and leaves the others alone: it never opens or lists them, except to say what the file names when none is its own.
- **A backend refuses what it does not understand.** Read options through `OptionReader` and call `finish()`, so a misspelt option is an error.
- **A camera drops a frame rather than wait.** When `acquire` has no free slot, abandon what was taken for that frame set and carry on.
- **A backend is opened once, with all its cameras.** What an SDK has open belongs to the backend and is shared by the sources it returns; a source is one camera and publishes one group. Do not open an SDK per camera when the SDK serves several from one object.
- **The backend interface shows what the `Publisher` can do, and no more.** `FrameOutput` passes through the publisher's two ways to get a slot, the slot-free callback and the restart, and `CameraSource::backend_reserve()` passes through the reserve. A further need (memory an SDK allocates, a readiness signal for GPU writes) is added with the pool and the backend that have it, not ahead of them.
- **`publish()` means complete.** A source publishes a slot only when everything that writes it has finished; readers look at it from that moment.
- **The usage is terminal output; everything else is logged.** `run_camera_plugin` prints the usage, and the refusal of an argument together with it, as one message on one channel. Every other line goes through `isaaccapture::Logger` under `isaaccapture.camera.<Name>`.

## Slots, windows and leases

- **The slot and lease code uses the standard library only.** `types.hpp`, `publisher.hpp` and their sources have no slot memory, sockets, logging or platform headers in them. Memory, connections and platform calls stay in the files of the next section.
Expand Down Expand Up @@ -80,4 +95,5 @@ Read "What runs where" in the [`README.md`](README.md) before changing it.
- **A test does not assume which thread runs first.** A reader's request and the camera's publish come from two threads and are served in either order: have the reader ask for what comes after a frame set it names, not for "anything", when the test expects a particular one. Run the suite on one CPU (`taskset -c 0`) before pushing; CI's slower machines find these, a workstation does not.
- **A test waits for what the plugin's thread does; it does not assume it.** `next` returning says nothing about whether the plugin has taken the reader's request, so wait for the effect (`Served::wait_until_sent`) before counting what was sent.
- **A test's helper thread is stopped and joined however the test ends.** A failed `REQUIRE` throws, and a `std::thread` that is still joinable then aborts the process instead of reporting the failure. Use `std::jthread`, or a guard whose destructor joins.
- **Tests that look through the user's plugins get a directory of their own.** `EndpointDirectory` (`tests/cpp/core/camera/endpoint_directory.hpp`) points `XDG_RUNTIME_DIR` at a temporary directory and puts it back; a test that uses the real one finds whatever camera plugin the developer has running.
- **Tests that look through the user's plugins, or let a plugin choose its endpoint or its configuration file, get a user's directories of their own.** Point `XDG_RUNTIME_DIR`, and for a plugin's configuration file `XDG_CONFIG_HOME`, at a short temporary directory and put them back, as `EndpointDirectory` (`tests/cpp/core/camera/endpoint_directory.hpp`) and `UserDirectories` (`tests/cpp/core/camera/test_camera_plugin.cpp`) do; a test that uses the real ones finds whatever camera plugin and `cameras.yaml` the developer has.
- **A test that runs `run_camera_plugin` in the test process aims the stop signal at the plugin's thread**, and starts that thread with the signal blocked (`PluginRun` in `tests/cpp/core/camera/test_camera_plugin.cpp`). A signal sent to the process would end the test instead.
190 changes: 187 additions & 3 deletions src/core/camera/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ What is in it, included as `<camera/NAME.hpp>` from the target `camera::camera`:
- `endpoint.hpp`: `Endpoint`, a camera plugin's end of the connection, which serves a
`Publisher`'s groups to readers.
- `camera_reader.hpp`: `CameraReader`, the reader's end and what an application uses.
- `camera_source.hpp`: `Backend`, `CameraSource` and `FrameOutput`, what a vendor's capture code
implements and what it publishes through.
- `camera_host.hpp`: `CameraHost`, which gives each camera its pools and puts all of them behind
one endpoint.
- `options.hpp`: `OptionReader`, how a backend reads a camera's options, so that one it does
not know is an error.
- `config.hpp`: `load_config_file`, the cameras a configuration file names for one backend.
- `camera_plugin.hpp`: `run_camera_plugin`, everything a camera plugin's `main` does.

## Terms

Expand Down Expand Up @@ -150,6 +158,172 @@ Not covered: a process that is stopped while it waits inside `next` is handed, w
continues, the frame set that arrived meanwhile. A reader that needs to know how old its
frame is has the times of every step in `timeline()`.

## Publishing cameras

A camera plugin is one vendor's or one SDK's capture code and this library. The capture code
opens the cameras and writes the images. The library gives each camera its pools, publishes the
frame sets, serves them on the plugin's [endpoint](#the-endpoint), and is the plugin's command
line and configuration.

```cpp
#include <camera/camera_plugin.hpp>

int main(int argc, char** argv)
{
return core::camera::run_camera_plugin(argc, argv, my_backend());
}
```

There is one camera plugin per vendor or SDK. Each is a small executable with its own
`plugin.yaml` and install folder, and its own build option when it needs an SDK. Readers find
the plugin that publishes a camera
([How a reader finds its camera](#how-a-reader-finds-its-camera)); none of them links a camera
SDK. [`camera_synthetic`](../../plugins/camera_synthetic) is the plugin that needs no hardware
and the one to copy; see [Writing a camera plugin](#writing-a-camera-plugin).

`run_camera_plugin` is everything a plugin's `main` does. It reads the arguments, chooses the
cameras, takes the endpoint, opens the cameras through the plugin's backend, serves until
`SIGINT` or `SIGTERM`, and returns the exit code: 0 after a clean stop or `--help`, 2 for a
command line it cannot parse, 1 when the plugin cannot run. An argument that parses and cannot
be honoured, such as an option its backend refuses, returns 1.

`CameraHost` gives each camera a pool of shared memory per stream ([Host pool](#host-pool)) and
puts all of them behind one endpoint. `run_camera_plugin` uses it; a test can use it without a
process.

### Running a plugin

What follows is true of every camera plugin, because all of it is this library.

**Which cameras.** A camera has an id that readers ask for and the options of the plugin's
backend. The plugin takes its cameras from, in this order:

1. `--camera=ID[:KEY=VALUE,...]` on the command line, once per camera;
2. `--config=FILE`, or without it `$XDG_CONFIG_HOME/isaaccapture/cameras.yaml`
(`~/.config/isaaccapture/cameras.yaml`) when that file exists;
3. otherwise, or with `--auto`, every camera its backend finds plugged in.

`--camera`, `--config` and `--auto` are alternatives: a command line that mixes them is
refused.

**The configuration file** is one for all the camera plugins of a user. A camera takes two
lines, its id and `backend:`, and one more for each option of that backend as `name: value`.
`slots:` at the top level sets the pool size.

```yaml
# ~/.config/isaaccapture/cameras.yaml
cameras:
head:
backend: synthetic
width: 1920
slots: 8
```

A plugin serves the cameras whose `backend:` is its own and leaves the others to their
plugins. A file that names cameras, but none for this plugin's backend, stops the plugin with
an error that names the file and lists its cameras with their backends. That is how a
misspelt backend shows.

**Where it listens.** On `camera-<instance>.sock` in the user's endpoint directory
([Where a plugin listens](#where-a-plugin-listens)). A plugin's instance name is the name of
its backend, so the synthetic plugin listens on `camera-synthetic.sock`. A second plugin of
the same backend takes another name with `--instance=NAME`. Give it its own cameras with
`--camera` or `--config` as well: the instance name does not change which cameras a plugin
takes, so without them it serves the same cameras as the first, and a reader that looks for
one of them finds it on two plugins and is refused. `--endpoint=PATH` sets the socket file
outright; a reader is then told the path. Only one plugin serves an endpoint at a time: a
plugin takes its endpoint before it opens its cameras, and accepts readers once they are
open ([When readers can reach it](#when-readers-can-reach-it)).

**Pool size.** `--slots=N`, 8 by default, is the number of frames kept per stream. A pool
holds the newest frame set, one slot for the camera to write ahead, and the windows of all
readers; a reader that does not fit is refused with the numbers. When readers want every
frame in order, the frame sets waiting for them are kept in place of the newest one: as many
as the largest of their windows ([Pool budget](#pool-budget)).

**Who starts it.** The application, through the plugin manager, like every other Isaac
Capture plugin: `plugin.yaml` sits next to the executable, and the handle the plugin manager
returns stops the plugin when it goes away. The `--plugin-root-id` the plugin manager adds is
accepted and not used. A plugin can also be run by hand or from a service. A reader never
starts it ([Reading cameras](#reading-cameras)).

A misspelt argument, option or setting is an error, not something ignored. `SIGINT` and
`SIGTERM` stop the plugin cleanly; readers see their connection close. `--help` prints the
arguments under the name of the executable.

Every ten seconds the plugin logs, per camera, how many frame sets it published. It warns
when a camera had to drop frame sets because its pool was full, or when a reader did not take
its events; both mean the pool or a reader's window needs another look. The loggers are
`isaaccapture.camera.run_camera_plugin` and `isaaccapture.camera.CameraHost`. The usage, and
the refusal of an argument that comes with it, are for the terminal and are not logged.

### What a backend implements

A backend is one camera SDK as its plugin sees it, and a plugin has exactly one. It is one
function that returns a `Backend` (`camera_source.hpp`):

- a `name`: what a camera's configuration says in `backend:`, and the plugin's instance name;
- a `description`: a short phrase without a full stop, which the plugin's `--help` shows
after the backend's name;
- `open`, which is called once with every camera the plugin is to serve and returns one
`CameraSource` per camera. What the SDK has open is the backend's, not a camera's: an SDK
that serves a whole rig from one object, or keeps state for the process, opens it once in
that call and lets the sources share it;
- optionally `discover`, which returns the cameras that are plugged in, for a plugin that is
not told which to serve.

Each source publishes one group. It reads its options through `OptionReader` and calls
`finish()`, so a misspelt option is refused.

A source gets the slot for an image through its `FrameOutput` in one of two ways:

- **The library picks the slot.** `acquire()` hands out a free slot, the source writes the
image into it and calls `publish()`. When nothing is free the frame is dropped. This is the
way to start with.
- **The SDK picks the buffer.** For an SDK that is given its buffers up front and says which
one it filled: the source hands the SDK the memory of every slot (`slot_count()`,
`slot_memory()`), asks to be told when a slot is free again (`on_slot_free()`), calls
`claim()` for the buffer the SDK filled, then `publish()`. `backend_reserve()` says how
many buffers the SDK needs in hand to keep capturing; readers are never given so many that
it is left with fewer.

In both, `publish()` says the images are complete. A slot is not handed out or given back to
the SDK while a reader holds it, also across `restart()`, which is how a source says that
its camera restarted.

### Writing a camera plugin

[`src/plugins/camera_synthetic`](../../plugins/camera_synthetic) is the template. For a
vendor or SDK called `<name>`:

1. **Directory.** Copy `src/plugins/camera_synthetic` to `src/plugins/camera_<name>`, without
`tools/`: `camera_bench` reads any camera plugin and stays where it is.
2. **`core/`: the backend.** A static library `camera_<name>_core` that links
`camera::camera` and the SDK. It implements `CameraSource` for one camera and has one
function that returns the `Backend`, named `<name>`. No type of the SDK leaves this
directory.
3. **`app/main.cpp`.** A few lines: `return run_camera_plugin(argc, argv, <name>_backend());`.
The executable is `camera_<name>_plugin`. There is nothing else to write for the command line,
the configuration file or the endpoint.
4. **`plugin.yaml`.** `name: camera_<name>`, `command: "./camera_<name>_plugin"`, a version, a
description and the `devices:` entry of type `camera`. `app/CMakeLists.txt` copies it next
to the built executable, so the plugin manager can start the plugin from the build tree
when it is given the directory above the executable's (`build/src/plugins/camera_<name>`),
as it is given `<install>/plugins` for an install. The plugin is installed to
`plugins/camera_<name>`.
5. **Build option.** A plugin that needs an SDK gets its own `BUILD_PLUGIN_<NAME>` option,
off by default, next to the other plugin options in the top-level `CMakeLists.txt`, and
`src/plugins/CMakeLists.txt` adds its directory only when the option is on. So nobody
builds or installs an SDK they do not use. `camera_synthetic` needs none and is built
whenever plugins are.
6. **Tests.** In `tests/cpp/plugins/camera_<name>`, wired under the same option, with a
CTest label that starts with `camera_`. Tag what runs without the camera `unit` (the
backend's options and what it describes) and what starts the plugin as a process
`integration`; the cases of `tests/cpp/plugins/camera_synthetic` show both.
7. **Documents.** A `README.md` with the backend's options and what the SDK needs, and an
`AGENTS.md` that points to "Publishing cameras" in [`AGENTS.md`](AGENTS.md) here for the
rules every camera plugin follows.

## How it works

### Slots, windows and leases
Expand Down Expand Up @@ -521,9 +695,9 @@ its camera asks that of every plugin of its user. Nothing is logged per frame.

## Tests

`tests/cpp/core/camera`, built as one executable, `camera_tests`. `ctest -L camera` runs them; a
label is a regular expression, so `ctest -L "^camera$"` is the form that selects this library's
tests alone.
`tests/cpp/core/camera`, built as one executable, `camera_tests`. `ctest -L camera` runs them. A
label is a regular expression, so that also runs the tests of the camera plugins, whose labels
start with `camera_`; `ctest -L "^camera$"` is the form that selects this library's tests alone.

- **Unit** (`ctest -L unit`): everything that runs in one process. The endpoint's cases play the
reader's part with the socket and the codec directly. The reader's cases run it against a
Expand All @@ -532,6 +706,16 @@ tests alone.
directory and put there what the case is about: plugins with different groups or the same
one, a plugin that has not started, a socket file nobody is behind, a plugin that does not
answer, a plugin that refuses the reader or goes away while it is asked.

The publishing side's cases use stand-in backends, and what they serve is read with a real
reader in the same process. Camera arguments, options and configuration files: a camera of
the command line is the plugin's backend's; a plugin takes its own cameras from a file it
shares with other plugins; a file with cameras and none for the plugin's backend is refused
with what it names. The host with a camera that publishes on demand, with one whose SDK owns
the buffers, with one that cannot start, and across a camera restart. `run_camera_plugin` on
a thread, stopped with the signal a plugin process is stopped with: the instance name is the
backend's unless it is given; the backend is opened once with all its cameras; what is
served without arguments; what ends a plugin with an error; what is answered with the usage.
- **Integration** (`ctest -L integration`): the cases that start a second process. One sends
it a pool over the socket: it maps the pool, sees the owner's writes and cannot write itself.
The others have the reader in the test process and `camera_test_peer`
Expand Down
10 changes: 10 additions & 0 deletions src/core/camera/cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,29 @@ find_package(Threads REQUIRED)

# The camera library; this module's README.md says what is in it. Linux only.
add_library(camera STATIC
camera_host.cpp
camera_plugin.cpp
camera_reader.cpp
codec.cpp
config.cpp
description.cpp
endpoint.cpp
endpoint_path.cpp
host_pool.cpp
message_socket.cpp
options.cpp
publisher.cpp
inc/camera/camera_host.hpp
inc/camera/camera_plugin.hpp
inc/camera/camera_reader.hpp
inc/camera/camera_source.hpp
inc/camera/config.hpp
inc/camera/endpoint.hpp
inc/camera/endpoint_path.hpp
inc/camera/host_pool.hpp
inc/camera/message_socket.hpp
inc/camera/messages.hpp
inc/camera/options.hpp
inc/camera/publisher.hpp
inc/camera/types.hpp
inc/camera/unique_fd.hpp
Expand All @@ -34,6 +43,7 @@ target_link_libraries(camera
PRIVATE
isaacteleop_schema
log_bridge::log_bridge_core
yaml-cpp::yaml-cpp
Threads::Threads
)

Expand Down
Loading
Loading