Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions examples/camera_livekit_bridge/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,18 @@ endif()
find_package(LiveKit CONFIG REQUIRED PATHS "${LIVEKIT_SDK_DIR}" NO_DEFAULT_PATH)
message(STATUS "LiveKit SDK=${LIVEKIT_SDK_DIR}")

# The time-sync library comes from the product tree; a build inside it already has the target.
if(NOT TARGET timesync::timesync)
cmake_path(GET CMAKE_CURRENT_SOURCE_DIR PARENT_PATH _examples_dir)
cmake_path(GET _examples_dir PARENT_PATH _repo_dir)
add_subdirectory(${_repo_dir}/src/core/timesync ${CMAKE_BINARY_DIR}/timesync)
endif()

add_subdirectory(video_types)
add_subdirectory(ipc_client)
add_subdirectory(gpu_ops)
add_subdirectory(nvenc)
add_subdirectory(chrony_clock)
add_subdirectory(livekit_publisher)
add_subdirectory(probe)
add_subdirectory(bridge)
Expand Down
50 changes: 42 additions & 8 deletions examples/camera_livekit_bridge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Each stream runs on its own thread. A frame is copied (or box-averaged down) on
the GPU from the IPC slot into an NVENC input surface, encoded, and submitted as
a pre-encoded access unit. Pixels never reach host memory. The frame's source
sequence is the track's `frame_id`, the sensor start time (when the receiver
derived one) is the `user_timestamp` in Unix microseconds, and the sensor
derived one) is the `user_timestamp` (what it means is set by `--timestamp-basis`, below), and the sensor
counter is the 8-byte `user_data`. The newest picture is encoded and older
queued ones are skipped. The receiver overwrites a slot without notice once its ring wraps, so after the GPU copy the
bridge checks for a newer notice for the same slot, or as many newer notices as the ring has other slots, and drops the
Expand All @@ -40,6 +40,13 @@ slots" applies.
| --- | --- | --- | --- |
| `livekit_publisher/` and the session, publication and feedback wiring in `bridge/` | Transport adapters | Adapt the useful implementation; retire the hard-coded transport glue | Map sessions, publications, encoder feedback and reconnect behaviour to the adapter boundary. Lands when real web clients receive streams, and feedback and recovery work, through an adapter. |
| `ipc_client/` and the slot read in `bridge/stream_worker.cpp` | Camera reader views and leases | Retire | Read frames through a lease held until the final GPU read. Lands with the receiver's lease support. |
| `chrony_clock/` and the capture-time publication in `bridge/stream_worker.cpp` | The receiver backend's published clock mapping (offset, uncertainty, discontinuity epoch) | Transitional: adapt, then retire the bridge-specific adapter | Carry mapped capture time, validity and discontinuity behaviour through the receiver contract. Lands when consumers obtain equivalent timing without the bridge-only adapter. |
| `src/core/timesync/` (used by `chrony_clock/`) | The estimator behind the receiver's clock mapping | Reuse | Keep the chrony evidence handling and qualification behind the destination interface. |

The `chrony_clock/` adapter connects the reusable timing library to the current video publication path and is intended
to converge on the receiver backend's clock-mapping contract. With `capture_monotonic_us`, a mapping that cannot be
vouched for publishes no timestamp, and the frame-ready time is never offered in its place; a capture over the
precision target keeps its timestamp. `monotonic_us` is a separate, declared basis.

## Build

Expand Down Expand Up @@ -78,13 +85,40 @@ the same basis:

| Basis | Value |
| --- | --- |
| `capture_unix_us` (default) | The sensor start-of-frame as Unix time on the sender's clock. |
| `monotonic_us` | This host's monotonic clock at the instant the picture is ready to encode (before encoding): frame-ready time, later than the sensor capture. |

`capture_unix_us` is used only when the receiver reports a realtime sender clock; a frame with no usable sensor
start-of-frame (missing, in the future, older than 5 s, or on another clock domain) carries no `user_timestamp`, so
a consumer that needs a capture time says so instead of using a guess. `capture_monotonic_us` is refused: it would
need a measured sender-to-host clock mapping with a stated uncertainty, which does not exist yet.
| `monotonic_us` (default) | This host's monotonic clock at the instant the picture is ready to encode (before encoding): frame-ready time, later than the sensor capture. |
| `capture_monotonic_us` | The sensor start-of-frame on this host's monotonic clock, converted from the sender's `CLOCK_REALTIME` with chrony evidence for both hosts (below). |

`capture_monotonic_us` uses the sensor start-of-frame that the receiver reads from the sender's AVS1 timing record
(`sensor_capture_realtime_ns`, `sensor_capture_uncertainty_ns`, `sensor_capture_valid`) and the time-sync library in
`src/core/timesync`. Each frame is converted from the sender's `CLOCK_REALTIME` to this host's monotonic clock, or
refused with a typed reason, using chronyd's own report for both hosts, a bracketed read of this host's clocks, and the
sender's stated uncertainty. Two outcomes are different in kind. A conversion that cannot be vouched for (a stale or
unsynchronised chrony report, another reference, an implausible age, an invalid sender record) is published without a
timestamp, and the frame-ready time is never offered in its place. A conversion that passes every one of those checks
but whose combined uncertainty is over the precision target (`over_budget`) still has a capture instant with a known
bound, so it is published with that timestamp; the target classifies the frame in the statistics and does not remove its
time. It needs:

| Option | Meaning |
| --- | --- |
| `--chrony-sender HOST[:PORT]` | The camera host's chronyd, read over chrony's monitoring interface (it must `cmdallow` this host). |
| `--chrony-sender-refid HEX` | The reference id this host's chronyd must have selected (an NTP source's reference id is its IPv4 address in hex). |
| `--chrony-pc-port PORT` | Read this host's chronyd on that command port over loopback, for a dedicated instance next to the system one. |
| `--chronyc PATH` | The `chronyc` binary (default `chronyc` on `PATH`). |

`src/core/timesync/README.md` describes the arrangement (this host measures the camera host, which serves its own clock) and how to set it up with `timesync_tool.py`. The
A realtime step of this host's wall clock is a timing fault: the frame that reveals it and every frame for the next 8 s are published without a timestamp (a longer silence than a recording take tolerates, so a take cannot continue across the step even if some of those frames are lost on the way), and chrony readings taken before the step are never reused. The
`[stats]` line gains the number of frames that met the precision target, a count per other verdict (`over_budget` frames
carry a timestamp; the rest do not), the last combined uncertainty and the clock generation the frame was converted
under. These are runtime diagnostics only and are not recorded with the video. The caller of a recording should treat a
frame without a timestamp as a timing fault.

Recording eligibility is permissive in this bridge: an `over_budget` frame is converted validly, keeps its capture
time and is published like a `qualified` one, so a recording that accepts every timestamped frame records it. The
timestamp alone does not say the precision target was met, and the verdict and bound are not carried with the video.
A frame without a timestamp (an invalid mapping, stale or unsynchronised evidence, or a wall-clock step) is the timing
fault. A consumer that must reject frames above the precision target needs the bound from the bridge's diagnostics
or a stricter publication policy; this bridge does not apply one.

`--source-full-range` says the receiver's pictures use the full 0-255 range, as SIPL sources do; the encoded stream
then signals full range, so players and recordings keep the black and white levels. Without it the stream signals
Expand Down
1 change: 1 addition & 0 deletions examples/camera_livekit_bridge/bridge/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ target_link_libraries(camera_livekit_bridge
camera_livekit_bridge::gpu_ops
camera_livekit_bridge::nvenc
camera_livekit_bridge::livekit_publisher
camera_livekit_bridge::chrony_clock
camera_livekit_bridge::video_types
CUDA::cudart
Threads::Threads
Expand Down
77 changes: 70 additions & 7 deletions examples/camera_livekit_bridge/bridge/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
#include <nvenc/cuda_context.hpp>

#include <atomic>
#include <cctype>
#include <chrono>
#include <csignal>
#include <cstdlib>
Expand All @@ -33,7 +34,9 @@ void usage()
{
std::cerr
<< "usage: camera_livekit_bridge --livekit-url URL --livekit-token-file FILE --stream NAME=TRACK[:DIV] [--stream ...]\n"
" [--socket PATH] [--fps N] [--bitrate-mbps N] [--idr-period N] [--max-notice-age-ms N] [--gpu N] [--source-full-range]\n [--timestamp-basis monotonic_us|capture_monotonic_us|capture_unix_us]\n"
" [--socket PATH] [--fps N] [--bitrate-mbps N] [--idr-period N] [--max-notice-age-ms N] [--gpu N] [--source-full-range]\n [--timestamp-basis monotonic_us|capture_monotonic_us]\n"
" capture_monotonic_us: --chrony-sender HOST[:PORT] --chrony-sender-refid HEX [--chrony-pc-port PORT]\n"
" [--chronyc PATH]\n"
" NAME is a stream in camera_receiver's IPC catalog; TRACK is the LiveKit track name egocap expects;\n DIV shrinks the picture on the GPU by that integer factor before encoding.\n";
}

Expand Down Expand Up @@ -67,6 +70,8 @@ int main(int argc, char** argv)
std::vector<StreamSpec> streams;
WorkerConfig defaults;
int gpu = 0;
ChronyCaptureClockConfig chrony_config;
bool use_chrony = false;
for (int i = 1; i < argc; ++i)
{
const std::string arg = argv[i];
Expand Down Expand Up @@ -100,18 +105,35 @@ int main(int argc, char** argv)
const auto basis = parse_timestamp_basis(next());
if (!basis)
{
std::cerr << "--timestamp-basis must be monotonic_us, capture_monotonic_us or capture_unix_us\n";
std::cerr << "--timestamp-basis must be monotonic_us or capture_monotonic_us\n";
return 2;
}
if (*basis == TimestampBasis::CaptureMonotonicUs)
defaults.timestamp_basis = *basis;
}
else if (arg == "--chrony-sender")
{
const std::string endpoint = next();
const auto colon = endpoint.rfind(':');
chrony_config.sender_host = colon == std::string::npos ? endpoint : endpoint.substr(0, colon);
if (colon != std::string::npos)
chrony_config.sender_port = std::atoi(endpoint.substr(colon + 1).c_str());
if (chrony_config.sender_host.empty())
{
std::cerr
<< "--timestamp-basis capture_monotonic_us is not available: it needs a measured sender-to-host clock "
"mapping, which is not implemented. Use monotonic_us (frame-ready time) or capture_unix_us.\n";
std::cerr << "--chrony-sender needs HOST or HOST:PORT, got '" << endpoint << "'\n";
return 2;
}
defaults.timestamp_basis = *basis;
use_chrony = true;
}
else if (arg == "--chrony-sender-refid")
{
chrony_config.expected_sender_reference_id = next();
for (auto& c : chrony_config.expected_sender_reference_id)
c = static_cast<char>(std::toupper(static_cast<unsigned char>(c)));
}
else if (arg == "--chrony-pc-port")
chrony_config.pc_port = std::atoi(next().c_str());
else if (arg == "--chronyc")
chrony_config.chronyc_path = next();
else if (arg == "--gpu")
gpu = std::atoi(next().c_str());
else if (arg == "--stream")
Expand Down Expand Up @@ -146,13 +168,35 @@ int main(int argc, char** argv)
return 2;
}

if (defaults.timestamp_basis == TimestampBasis::CaptureMonotonicUs && !use_chrony)
{
std::cerr
<< "--timestamp-basis capture_monotonic_us needs --chrony-sender HOST: the sender's capture time is only "
"used with chrony evidence for both hosts. Use monotonic_us for the frame-ready time.\n";
return 2;
}
if (use_chrony && chrony_config.expected_sender_reference_id.empty())
{
std::cerr << "--chrony-sender needs --chrony-sender-refid HEX: the reference id chronyd shows for its source "
"(chronyc tracking), so a host following another source is refused.\n";
return 2;
}

std::signal(SIGINT, on_signal);
std::signal(SIGTERM, on_signal);

try
{
CudaContext context(gpu);
auto session = LiveKitSession::connect(url, read_token(token_file));
std::unique_ptr<ChronyCaptureClock> chrony_clock;
if (use_chrony)
{
chrony_clock = std::make_unique<ChronyCaptureClock>(chrony_config);
chrony_clock->poll_once();
chrony_clock->start();
defaults.chrony_clock = chrony_clock.get();
}
std::vector<std::unique_ptr<StreamWorker>> workers;
for (const auto& spec : streams)
{
Expand Down Expand Up @@ -213,6 +257,25 @@ int main(int argc, char** argv)
<< static_cast<double>(stats.capture_to_ready_us_total.load()) / ready_samples / 1000.0
<< " ms mean " << stats.capture_to_ready_max_us.load() / 1000.0 << " ms max";
}
if (defaults.timestamp_basis == TimestampBasis::CaptureMonotonicUs)
{
std::cerr
<< ", capture claims: "
<< stats
.chrony_verdicts[static_cast<std::size_t>(isaaccapture::timesync::Verdict::Qualified)]
.load()
<< " qualified";
for (std::size_t v = 1; v < 10; ++v)
{
if (stats.chrony_verdicts[v].load() > 0)
std::cerr
<< ", "
<< isaaccapture::timesync::to_string(static_cast<isaaccapture::timesync::Verdict>(v))
<< " " << stats.chrony_verdicts[v].load();
}
std::cerr << ", combined uncertainty " << stats.last_uncertainty_ns.load() / 1000.0
<< " us, clock generation " << stats.last_generation.load();
}
std::cerr << "\n";
last_frames[i] = frames;
}
Expand Down
23 changes: 21 additions & 2 deletions examples/camera_livekit_bridge/bridge/stream_worker.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -326,8 +326,27 @@ struct StreamWorker::Impl
const bool wall_is_realtime = wall_domain != notice.metadata.end() && wall_domain->is_string() &&
wall_domain->get<std::string>() == "realtime";
const uint64_t sensor_sof_ns = wall_is_realtime ? unsigned_field(notice.metadata, "sensor_sof_ns") : 0;
frame.user_timestamp_us =
user_timestamp_us(config.timestamp_basis, sensor_sof_ns, ready_realtime_ns, ready_monotonic_ns);
frame.user_timestamp_us = config.timestamp_basis == TimestampBasis::MonotonicUs ?
static_cast<uint64_t>(ready_monotonic_ns / 1000) :
0;
if (config.timestamp_basis == TimestampBasis::CaptureMonotonicUs && config.chrony_clock != nullptr)
{
// The receiver's timing record: the sensor start of frame on the sender's CLOCK_REALTIME with the
// sender's stated uncertainty. A conversion that cannot be vouched for publishes no capture time; the
// frame-ready time is never offered in its place.
const auto valid = notice.metadata.find("sensor_capture_valid");
const bool sender_valid = valid != notice.metadata.end() && valid->is_boolean() && valid->get<bool>();
const auto result = config.chrony_clock->qualify(
sender_valid, static_cast<int64_t>(unsigned_field(notice.metadata, "sensor_capture_realtime_ns")),
static_cast<int64_t>(unsigned_field(notice.metadata, "sensor_capture_uncertainty_ns")));
++stats.chrony_verdicts[static_cast<std::size_t>(result.verdict)];
stats.last_uncertainty_ns = result.uncertainty_ns;
stats.last_generation = result.generation;
// A precision miss still has a converted instant with a stated bound, so the frame keeps its capture
// time; only a conversion that cannot be vouched for publishes none.
if (result.usable())
frame.user_timestamp_us = static_cast<uint64_t>(result.capture_monotonic_ns) / 1000;
}
if (sensor_sof_ns != 0)
{
const int64_t age_ns = ready_realtime_ns - static_cast<int64_t>(sensor_sof_ns);
Expand Down
10 changes: 9 additions & 1 deletion examples/camera_livekit_bridge/bridge/stream_worker.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

#pragma once

#include <chrony_clock/chrony_capture_clock.hpp>
#include <livekit_publisher/livekit_session.hpp>
#include <nvenc/cuda_context.hpp>
#include <video_types/timestamp_basis.hpp>
Expand Down Expand Up @@ -35,7 +36,9 @@ struct WorkerConfig
/// receiver's IPC catalog does not carry the range, so the operator says it here.
bool source_full_range = false;
/// What each frame's user_timestamp means; the consumer's streams file must name the same basis.
TimestampBasis timestamp_basis = TimestampBasis::CaptureUnixUs;
TimestampBasis timestamp_basis = TimestampBasis::MonotonicUs;
/// Chrony evidence for capture_monotonic_us; shared by every stream and owned by the caller.
ChronyCaptureClock* chrony_clock = nullptr;
};

struct WorkerStats
Expand All @@ -59,6 +62,11 @@ struct WorkerStats
std::atomic<uint64_t> capture_to_ready_us_total{ 0 };
std::atomic<uint64_t> capture_to_ready_samples{ 0 };
std::atomic<uint32_t> capture_to_ready_max_us{ 0 };
/// capture_monotonic_us: how many frames got each timesync::Verdict, and the last frame's combined uncertainty and
/// the PC clock generation it was converted under.
std::atomic<uint64_t> chrony_verdicts[16] = {};
std::atomic<int64_t> last_uncertainty_ns{ 0 };
std::atomic<uint64_t> last_generation{ 0 };
/// The latest target the transport reported (0 until it reports one).
std::atomic<uint32_t> target_bps{ 0 };
};
Expand Down
19 changes: 19 additions & 0 deletions examples/camera_livekit_bridge/chrony_clock/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Qualifies a sender CLOCK_REALTIME capture time against chrony's evidence for both hosts.
add_library(chrony_clock STATIC chrony_capture_clock.cpp)

target_include_directories(chrony_clock
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/inc>
)

target_link_libraries(chrony_clock
PUBLIC
timesync::timesync
)

target_compile_options(chrony_clock PRIVATE -Wall -Wextra)

add_library(camera_livekit_bridge::chrony_clock ALIAS chrony_clock)
Loading
Loading