Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions src/core/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,17 @@ add_subdirectory(oxr_utils)
# Build Plugin Manager
add_subdirectory(plugin_manager)

# Camera capture core (standard library only)
add_subdirectory(camera_core)

# Camera frames and messages between processes (shared host memory, descriptor passing)
if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
add_subdirectory(camera_ipc)
add_subdirectory(camera_wire)
add_subdirectory(camera_endpoint)
add_subdirectory(camera_reader)
endif()

# Build OXR library (depends on oxr_utils for types)
add_subdirectory(oxr)

Expand Down
16 changes: 16 additions & 0 deletions src/core/camera_core/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# Agent notes — `camera_core`

**CRITICAL (non-optional):** Before editing this package, complete the mandatory **`AGENTS.md` preflight** in [`../../../AGENTS.md`](../../../AGENTS.md) (read every applicable `AGENTS.md` on your paths, not just this file).

Read [`README.md`](README.md) for the terms (stream, group, window, order, lease) before changing behaviour.

- **Standard library only.** No slot memory, sockets, logging or platform headers here; that keeps the module building and tested on every platform. Memory and connections live in `camera_ipc`.
- **One type or field per concept.** A codec is not a pixel format, each plane has its own layout, counters are per group. Do not give a field a second meaning, and do not carry a type over from a prototype without modelling it for this interface.
- **Nothing ahead of its user.** A memory kind, pixel format or flag is added in the change that brings the pool or backend that needs it.
- **`EventSink::send_event` runs on the capture path with the publisher's lock held.** It is `noexcept` and must not block or call back into the publisher. Callbacks to a backend (`SlotFreeCallback`) run outside the lock.
- **Reject bad input where it enters.** `add_group` and `publish` refuse a description or an image that a reader could run off the end of a slot with; readers do not re-check.
6 changes: 6 additions & 0 deletions src/core/camera_core/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

cmake_minimum_required(VERSION 3.20)

add_subdirectory(cpp)
71 changes: 71 additions & 0 deletions src/core/camera_core/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# Camera interface: core

The camera interface is the one place Isaac Capture applications get camera frames from. One
process owns the cameras and keeps their frames in fixed pools of shared buffers; viewers,
encoders and recorders in other processes read those buffers in place.

This module is the part with no memory and no sockets in it: it decides which pool slot holds
which frame set and who may still read it. It uses the standard library only. The shared memory
behind the slots and the connection to other processes live in [`camera_ipc`](../camera_ipc).

## Terms

- **Stream**: one sensor's images, or one encoded video stream. It has a pool of **slots**; a
slot holds one image, or one access unit when the stream is encoded.
- **Group**: one or more streams captured together, such as a stereo pair or a triggered rig.
A group publishes one **frame set** at a time: one image per stream, under one sequence number.
- **Window**: how many frame sets are kept for a reader at a time, unread and held together.
- **Order**: `NEWEST` hands the reader the latest frame set each time it has room and skips the
rest (a viewer, a live encoder). `IN_ORDER` hands over every frame set in sequence (a recorder).
The default is `NEWEST` for a window of one and `IN_ORDER` otherwise.
- **Lease**: a reader's hold on a frame set. Its slots are not reused until the reader releases
it or goes away.
- **Generation**: changes when a camera restarts, so a reader can tell a restart from a skip.

## How frames move

A backend gets a slot in one of two ways, then publishes:

```cpp
// The backend writes the image itself.
auto slot = publisher.acquire_slot(group, stream); // nothing: every slot is in use, drop the frame
// ... write the pixels into that slot ...
publisher.publish(group, images, timeline);

// The camera SDK owns the buffers and picked one.
publisher.claim_slot(group, stream, slot_the_sdk_filled);
publisher.publish(group, images, timeline);
```

For the second way, `set_slot_free_callback` tells the backend the moment a slot is held by
nobody, so the buffer can go back to the SDK. A group uses one way or the other: once it has
that callback, its slots are handed out with `claim_slot` only.

A reader subscribes with a window and an order, receives `FrameEvent`s through an `EventSink`,
and returns each one with `release`. Events carry slot indices, the frame set's timeline and the
number of frame sets the reader skipped. A slow reader skips; it never slows capture or another
reader, and no call blocks on a reader.

## Pool budget

A slot is in use while the ring of recent frame sets or any lease refers to it, so a subscription
is accepted only when the pool can cover it:

```text
slots >= ring history + sum of reader windows + backend reserve
```

Ring history is one frame set, or the largest window among in-order readers. The backend reserve
is the number of slots the backend needs for itself: one to write ahead, more for an SDK that
keeps several captures in flight. A subscription that does not fit is refused with a message that
gives these numbers.

## Tests

`tests/cpp/core/camera_core`, run with `ctest -L camera_core`. They are tagged `unit`, so
`ctest -L unit` runs them too.
16 changes: 16 additions & 0 deletions src/core/camera_core/cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

cmake_minimum_required(VERSION 3.20)

# Camera capture core: which pool slots hold which frame sets and who may still read them.
# Standard library only, so it builds and is tested on every platform.
add_library(camera_core STATIC
publisher.cpp
)

target_include_directories(camera_core INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/inc>
)

add_library(camera::camera_core ALIAS camera_core)
226 changes: 226 additions & 0 deletions src/core/camera_core/cpp/inc/camera_core/publisher.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

#pragma once

#include "types.hpp"

#include <array>
#include <cstddef>
#include <cstdint>
#include <deque>
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <optional>
#include <span>
#include <string>
#include <string_view>
#include <vector>

namespace core::camera
{

using GroupId = std::uint32_t;
using SubscriptionId = std::uint32_t; // 0 is never a valid subscription

/**
* @brief Where a reader's frame events go: a socket connection or an in-process queue.
*/
class EventSink
{
public:
virtual ~EventSink() = default;

/**
* @brief Hand one event to the reader. Called on the capture path with the publisher's
* lock held, so it must not block, throw or call back into the publisher.
* @return false when the event could not be handed over; the reader then gets a later one.
*/
virtual bool send_event(SubscriptionId subscription, const FrameEvent& event) noexcept = 0;
};

// Counters of one group since it was added.
struct PublisherStats
{
std::uint64_t published = 0; // frame sets published
std::uint64_t events_sent = 0;
std::uint64_t send_failures = 0;
std::uint64_t no_free_slot = 0; // acquire_slot calls that found every slot in use
};

struct SubscribeResult
{
SubscriptionId id = 0;
std::string error; // why the subscription was refused, when id is 0

explicit operator bool() const
{
return id != 0;
}
};

/**
* @brief Tracks which pool slots hold which frame sets and who may still read them.
*
* A group keeps a short ring of recent frame sets; a reader's window is how many are kept for
* it at a time, unread and held together. A subscription is accepted only when
* slots >= ring history + sum of reader windows + backend reserve. Events carry slot indices;
* slot memory is never touched here. Thread-safe; no call blocks on a reader.
*/
class Publisher
{
public:
using Clock = std::function<std::int64_t()>;
using SlotFreeCallback = std::function<void(std::size_t stream, std::uint32_t slot)>;

/**
* @param clock Host monotonic time in nanoseconds; defaults to std::chrono::steady_clock.
*/
explicit Publisher(Clock clock = {});

// ---- backend side ----------------------------------------------------------------------

/**
* @brief Declare a group and the size of its pools.
* @param backend_reserve Slots the backend needs beyond the ring and the readers: one to
* write ahead, more for an SDK that keeps several captures in flight in its own queue.
* @throws std::invalid_argument if the description is not usable.
*/
GroupId add_group(const GroupDescription& description, std::uint32_t backend_reserve = 1);

/**
* @brief Take a free slot to write the next image into.
* @return The slot, or nothing when every slot is in use (the frame should be dropped).
* @throws std::logic_error for a group that has a slot-free callback: its backend owns the
* pool and names slots with claim_slot.
*/
std::optional<std::uint32_t> acquire_slot(GroupId group, std::size_t stream);

/**
* @brief Mark a slot the SDK chose and filled, for a pool the SDK owns.
* @throws std::invalid_argument if the slot is still held by the ring or a reader.
*/
void claim_slot(GroupId group, std::size_t stream, std::uint32_t slot);

/**
* @brief Give back a slot taken with acquire_slot or claim_slot without publishing it.
*/
void abandon_slot(GroupId group, std::size_t stream, std::uint32_t slot);

/**
* @brief Be told when a published slot is no longer held by the ring or any reader, so a
* buffer the SDK owns can go back to it at once. Called outside the publisher's lock, on
* the thread whose call released the last hold. A group with this callback hands out its
* slots with claim_slot only, so a slot cannot be taken again before its callback has run.
*/
void set_slot_free_callback(GroupId group, SlotFreeCallback callback);

/**
* @brief Publish one frame set. images[i] belongs to stream i and names a slot taken with
* acquire_slot or claim_slot.
* @param timeline capture_ns and sdk_ready_ns from the backend; the rest is filled in here.
* @return The frame set's sequence number.
* @throws std::invalid_argument if the images do not match the group or do not fit its slots.
*/
std::uint64_t publish(GroupId group, std::span<const ImageInfo> images, const FrameTimeline& timeline);

/**
* @brief Tell readers the camera restarted: the generation changes and the ring is emptied.
* Frame sets readers already hold stay valid until released.
*/
void restart_group(GroupId group);

// ---- reader side -----------------------------------------------------------------------

/**
* @brief Start sending a group's frame events to a sink, which must outlive the subscription.
* @return The subscription, or the reason it does not fit the pool.
*/
SubscribeResult subscribe(EventSink& sink, GroupId group, const ReaderOptions& options);

/**
* @brief Return a frame set. The reader's next one, if any is waiting, is sent at once.
*/
void release(SubscriptionId subscription, std::uint64_t lease_id);

/**
* @brief End a subscription and take back every frame set it still holds.
*/
void unsubscribe(SubscriptionId subscription);

// ---- description -----------------------------------------------------------------------

std::size_t group_count() const;
GroupDescription group_description(GroupId group) const;
std::optional<GroupId> find_group(std::string_view id) const;
std::uint32_t generation(GroupId group) const;
PublisherStats stats(GroupId group) const;

private:
struct FrameSet
{
std::uint64_t sequence = 0;
std::uint32_t generation = 0;
FrameTimeline timeline;
std::array<ImageInfo, MAX_STREAMS_PER_GROUP> images;
};

struct Subscription
{
GroupId group = 0;
std::uint32_t window = 1;
DeliveryOrder order = DeliveryOrder::NEWEST;
EventSink* sink = nullptr;
std::uint64_t cursor = 0; // sequence of the last frame set sent to this reader
bool started = false;
std::map<std::uint64_t, FrameSet> leases; // sent and not yet released
};

struct Slot
{
std::uint32_t holds = 0; // the ring and every reader lease that names this slot
bool writing = false; // taken by the backend and not yet published
};

struct Group
{
GroupDescription description;
std::uint32_t reserve = 1;
std::uint32_t slot_count = 0; // the smallest pool of the group
std::uint32_t generation = 0;
std::uint64_t sequence = 0;
std::vector<std::vector<Slot>> slots; // per stream
std::deque<FrameSet> ring;
std::vector<SubscriptionId> subscriptions;
std::shared_ptr<const SlotFreeCallback> on_free;
PublisherStats stats;
};

struct FreedSlot
{
std::shared_ptr<const SlotFreeCallback> callback;
std::size_t stream = 0;
std::uint32_t slot = 0;
};

Group& group_at(GroupId group);
const Group& group_at(GroupId group) const;
static Slot& slot_at(Group& group, std::size_t stream, std::uint32_t slot);
std::size_t history_of(const Group& group) const;
void hold(Group& group, const FrameSet& set);
void unhold(Group& group, const FrameSet& set, std::vector<FreedSlot>& freed);
void trim_ring(Group& group, std::vector<FreedSlot>& freed);
void deliver(Group& group, SubscriptionId id, Subscription& subscription, std::vector<FreedSlot>& freed);
static void notify(const std::vector<FreedSlot>& freed);

Clock m_clock;
mutable std::mutex m_mutex;
std::vector<std::unique_ptr<Group>> m_groups;
std::map<SubscriptionId, Subscription> m_subscriptions;
SubscriptionId m_next_subscription = 1;
std::uint64_t m_next_lease = 1;
};

} // namespace core::camera
Loading
Loading