Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 12 additions & 14 deletions examples/egocap/CAMERA_SETTINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ pass it with `--sender-config`.
| Component | Owns | Where it is set |
|---|---|---|
| SIPL platform file | Sensor geometry, sensor rate, trigger mode, link wiring | `sipl.platform_config_json` on the Orin (installed by the BSP or the camera deploy) |
| Sender (Orin) | Output frame rate and bitrate, codec, timing records, clock responder, TCP port | The sender profile, `configs/default-camera-settings.yaml` |
| Sender (Orin) | Output frame rate and bitrate, codec, timing records, TCP port | The sender profile, `configs/default-camera-settings.yaml` |
| Receiver (PC) | Decoding, the CUDA IPC output and its stream names | Generated by the launcher from the sender profile |
| Bridge (PC) | Publishing into LiveKit; timestamp basis; the sender-clock address; output rate | Command-line flags, built by the launcher |
| Bridge (PC) | Publishing into LiveKit; timestamp basis; the chrony addresses; output rate | Command-line flags, built by the launcher |
| CaptureServer | The stream descriptors the page reads (room, track, size, nominal fps, timestamp basis) | Generated by the launcher |

The sender never reads exposure, gain, white balance, trigger frequency or clock-pin settings from its YAML, and there
Expand All @@ -26,17 +26,16 @@ is no runtime settings API: add such keys to the file and nothing happens.
## The default sender profile

`configs/default-camera-settings.yaml` is the profile the launcher uses unless told otherwise. It describes one SHW5G head
camera on the SIPL backend: HEVC at 30 fps and 4 Mbit/s,
the AVS1 timing extension on, and the UDP clock responder on port 5611.
camera on the SIPL backend: HEVC at 30 fps and 4 Mbit/s.

The file is specific to that rig. A different SIPL platform or camera changes the `sipl` block (platform file, config
name, link masks, tuning directory) and possibly the stream id and size. The Argus backend has no `sipl` block: it
selects a sensor mode with `cameras[].mode`, can set `capture.mode` and `alignment`, enforces `output.fps` with a rate
limiter and can set output dimensions. Keep one file per rig and pass the right one with `--sender-config`. The launcher itself supports one stream per
file and requires `network.timing_extension: true`, `network.clock_port` and an explicit `output.fps`.
file and requires an explicit `output.fps`.

`tools/start_system.py` loads this file by default. Command-line overrides (`--fps`, `--bitrate-mbps`,
`--clock-port`, `--sender-port`) are applied to the loaded mapping, and the result is what is uploaded. The same
`--sender-port`) are applied to the loaded mapping, and the result is what is uploaded. The same
values feed the bridge and the CaptureServer descriptor, so they cannot disagree.

### How the file reaches the sender
Expand All @@ -46,8 +45,8 @@ The sender does not search for a camera file. The deploy tool uploads the chosen
it that are relative to the file (the remaps) resolve inside the release, and starts `./build/argus_sender --config
src/configs/run-config.yaml`. The file is read once at start. Editing it while the sender runs changes nothing: stop the
sender and start it again. A reconnecting receiver reuses the same in-memory configuration. Uploading a new file does
not rebuild the binary, so the release must already support the timing extension and the clock responder; an older
parser rejects keys it does not know instead of ignoring them. The platform JSON and tuning files are not uploaded
not rebuild the binary, so the release must already send the timing record (it is sent with every frame and has no
setting); an older parser rejects keys it does not know instead of ignoring them. The platform JSON and tuning files are not uploaded
and must already exist at the configured paths on the Orin.

### Frame rate
Expand All @@ -68,10 +67,11 @@ effective `output.fps`; the descriptor value is nominal and cannot make the came

The page shows latency from the sensor's capture time only when all of these hold:

1. The sender runs with `network.timing_extension: true` and `network.clock_port` set (the default profile does both),
from a release that includes the timing extension and the clock responder.
1. The sender is a release that includes the AVS1 timing extension; it sends the sensor capture time on its
`CLOCK_REALTIME` with every frame. A release whose parser still reads `network.timing_extension` is older than this profile and sends
no capture time unless that key is set to `true`, which this profile no longer carries; every take with it is refused.
2. The receiver passes the timing records through to the CUDA IPC output.
3. The bridge runs with `--timestamp-basis capture_monotonic_us --sender-clock ORIN_HOST:CLOCK_PORT`. For a SIPL profile
3. The bridge runs with `--timestamp-basis capture_monotonic_us --chrony-sender ORIN_HOST --chrony-sender-refid ID --chrony-pc-port PORT`. For a SIPL profile
the launcher also passes `--source-full-range`, because SIPL pictures use the full 0-255 range; without it the
encoded stream signals limited range and the page's blacks and whites are wrong. The bridge binary must support
that flag.
Expand Down Expand Up @@ -99,13 +99,11 @@ shipped profile.
| `sipl.link_masks` | Empty; required for SIPL | Sequence of unsigned masks, one per deserializer in transport order; passed to `ApplyMask`. Not an FPS knob. |
| `sipl.nito_dir` | `/var/nvidia/nvcam/settings/sipl` | ISP tuning directory; loads `<module-name>.nito` on the Orin. Not a YAML sensor-mode selector. |
| `sipl.isp0_buffers` | `6`; at least 2 | Number of ISP0 surfaces allocated/registered per sensor. Surface ownership and return timing affect pressure; increasing the count does not directly select FPS. |
| `network.bind` | `0.0.0.0`; nonempty | Sender TCP listening address; also used by the timing UDP responder, whose inspected implementation requires IPv4. |
| `network.bind` | `0.0.0.0`; nonempty | Sender TCP listening address. |
| `network.port` | `5000`; 1–65535 | AVS1 TCP listening port. Receiver must connect here. |
| `network.reconnect_ms` | `1000` | Retry delay for failed listen setup. Not a per-frame sleep or FPS limiter. |
| `network.dump_wire` | Empty / disabled | Writes the transmitted framed stream to a diagnostic file on the sender. Avoid enabling by default; storage can add work. |
| `network.send_buffer_kb` | `0` | Kernel TCP send-buffer request in KiB. Zero leaves OS behavior; positive values set `SO_SNDBUF`. Bounds one source of hidden buffering. |
| `network.timing_extension` | `false` | Emits AVS1 sensor timing qualification records; use `true` for the capture-time chain. Requires a compatible sender and receiver. |
| `network.clock_port` | `0` / disabled | UDP clock-exchange responder; 1–65535 enables it. Use 5611 in this profile and the same port in the bridge's sender-clock address. |
| `runtime.frames` | `0` / unlimited | Finite run limit. Argus counts bundles; SIPL counts successful frame submissions across streams. Not a time duration or FPS setting. |
| `runtime.kpi` | `true` | Enables periodic KPI collection/reporting; detailed output also depends on `--verbose`. |
| `runtime.log_interval_s` | `1` | Periodic logging interval in seconds; zero disables interval-based summaries at sites guarded by a positive interval. |
Expand Down
22 changes: 15 additions & 7 deletions examples/egocap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,16 +105,24 @@ timestamp because the mock sender has no clock to measure. A recording made in t
Without `--orin` or `--mock` only the PC side starts. `--skip-deploy` reuses the previous release. The sender runs
for `--seconds` (default one hour) and the launcher stops the whole stack when it ends, so a take in progress is
cut off; the launcher prints the stop time at startup. The sender release the profile describes, its deploy tool, the
LiveKit server and the bridge are not in this tree, so pass their paths (or set `EGOCAP_DEPLOY_SCRIPT`,
`EGOCAP_LIVEKIT_SERVER`, `EGOCAP_BRIDGE`). `--orin` needs an IPv4 address, because the bridge measures the sender's
clock at that address.
timesync tool, the LiveKit server and the bridge are not in this tree's base, so pass their paths (or set
`EGOCAP_DEPLOY_SCRIPT`, `EGOCAP_TIMESYNC_TOOL`, `EGOCAP_LIVEKIT_SERVER`, `EGOCAP_BRIDGE`); `--orin` needs the timesync
tool and refuses to start without it. `--orin` also needs an IPv4 address, because the bridge finds the sender's chrony
by it. After the bridge starts, the launcher runs time sync through `src/core/timesync/tools/timesync_tool.py` (see that
module's README): `ensure-local` (which may install chrony on this PC), an unprivileged chronyd on this PC that measures
the Orin's clock, and `configure-remote` on the Orin; each step has a timeout. These steps run before the page address
is printed, and a failure leaves the preview up with capture times unavailable (`--require-chrony` makes it fatal). For
the SIPL backend the launcher also stops `nvargus-daemon` on the Orin before starting the sender, since it holds the
cameras; it does not restart it.

The page needs the frontend built once (`cd frontend && npm install && npm run build`); the launcher refuses to start
without `frontend/dist`.
without `frontend/dist`. For the default receiver, the launcher runs the receiver's `setup.sh` (venv, requirements,
native decoder) when the venv or decoder is missing and the camera chain starts (`--orin`, or `--mock` with the
bridge); `CAMERA_RECEIVER_VENV` and `CAMERA_RECEIVER_PYTHON` select the environment as they do for the receiver's own
scripts.

The sender settings come from the checked-in profile `configs/default-camera-settings.yaml` (30 fps, 4 Mbit/s HEVC,
timing extension and clock responder on). Pass another file with `--sender-config`, or override single values with
`--fps`, `--bitrate-mbps`, `--clock-port` and `--sender-port`; the receiver, bridge and CaptureServer descriptor are
The sender settings come from the checked-in profile `configs/default-camera-settings.yaml` (30 fps, 4 Mbit/s HEVC). Pass another file with `--sender-config`, or override single values with
`--fps`, `--bitrate-mbps` and `--sender-port`; the receiver, bridge and CaptureServer descriptor are
built from the same effective values. [`CAMERA_SETTINGS.md`](CAMERA_SETTINGS.md) says which component owns which
setting and lists every key.

Expand Down
6 changes: 6 additions & 0 deletions examples/egocap/backend/egocap_web/livekit_recording.py
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,12 @@ async def prepare(self) -> None:
)
if frame is None:
raise RuntimeError(f"{stream.name}: the LiveKit track was lost before recording started")
# A take records the publisher's capture time, so a source that sends none cannot start one.
if frame.user_timestamp_us is None:
raise RuntimeError(
f"{stream.name}: the first frame carries no capture time, so it cannot be recorded "
"(the bridge publishes none while time sync is not up)"
)
if frame.image.shape[:2] != (stream.height, stream.width):
raise RuntimeError(
f"{stream.name}: frames are {frame.image.shape[1]}x{frame.image.shape[0]}, "
Expand Down
5 changes: 0 additions & 5 deletions examples/egocap/configs/default-camera-settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,6 @@ sipl:
network:
bind: 0.0.0.0
port: 5000 # TCP port the PC receiver connects to
# The two settings below are what let the page show latency measured from the sensor's capture time: the timing
# records carry the capture instant, and the clock responder lets the PC map the sender's clock onto its own.
# The bridge connects to clock_port, so the launcher takes it from here.
timing_extension: true
clock_port: 5611 # UDP

runtime:
frames: 0 # 0 = run until stopped
Expand Down
2 changes: 1 addition & 1 deletion examples/egocap/frontend/src/LiveKitCameraTile.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ export function LiveKitCameraTile({
title={
latencyLabel === NO_LATENCY
? "No latency for the displayed frame: it needs the publisher's per-frame stamp (matched by RTP timestamp), a current clock mapping to the publisher, and a stream whose timestamp is on the publisher's monotonic clock."
: `The publisher's frame stamp (frame-ready or sensor capture, per the stream) to the frame's expected display on this page, including receive, decode and render (${latencyDetail}); ± is half the clock-exchange round trip, the drift the exchange's age allows, and this browser's timer resolution`
: `The publisher's frame stamp (frame-ready or sensor capture, per the stream) to the frame's expected display on this page, including receive, decode and render (${latencyDetail}); ± is only this browser's clock mapping to the publisher (half the clock-exchange round trip, the drift the exchange's age allows, and this browser's timer resolution). It does not include the sensor-to-publisher clock conversion or how closely the sensor's start-of-frame marks the exposure`
}
>
{latencyLabel}
Expand Down
6 changes: 6 additions & 0 deletions examples/egocap/frontend/src/displayLatency.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ describe("latencySample", () => {
expect(latencySample(Number.NaN, { userTimestamp: 1_000_000_000n }, mapping())).toBeNull();
});

it("is null for a stamp of zero, which the browser SDK reports for a frame that carries no capture time", () => {
// Without this a frame with only a frame id gives the publisher's whole monotonic uptime as its latency.
expect(latencySample(997_072, { userTimestamp: 0n }, mapping())).toBeNull();
expect(latencySample(997_072, { userTimestamp: -1n }, mapping())).toBeNull();
});

it("is null when display precedes capture by more than the uncertainty, but keeps a small negative", () => {
expect(latencySample(996_990, { userTimestamp: 1_000_000_000n }, mapping())).toBeNull(); // -10 ms
expect(latencySample(996_999, { userTimestamp: 1_000_000_000n }, mapping())?.ms).toBeCloseTo(-1, 6);
Expand Down
2 changes: 2 additions & 0 deletions examples/egocap/frontend/src/displayLatency.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ export function latencySample(
mapping: ClockMapping | null,
): LatencySample | null {
if (stamp === undefined || mapping === null || !Number.isFinite(expectedDisplayTime)) return null;
// The browser SDK reports a frame that carries no capture time (only a frame id) as a stamp of 0, not as absent.
if (stamp.userTimestamp <= 0n) return null;
const displayRemote = toRemoteNs(mapping, BigInt(Math.round(expectedDisplayTime * 1e6)));
const ms = Number(displayRemote - stamp.userTimestamp * 1000n) / 1e6;
const uncertaintyMs = Number(mapping.uncertaintyNs) / 1e6;
Expand Down
20 changes: 18 additions & 2 deletions examples/egocap/tests/backend/test_livekit_recording_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,14 @@ def jpeg_camera(stream_id: int = 1) -> p.CameraStream:
)


def frame(stream_id: int, *, width=6, height=4) -> ReceivedFrame:
def frame(stream_id: int, *, width=6, height=4, stamped=True) -> ReceivedFrame:
return ReceivedFrame(
stream_id=stream_id,
image=np.zeros((height, width, 3), dtype=np.uint8),
# Captured just now on this host's clock (the fake mapping is the identity): passes the frame-age check.
receive_time_ns=time.monotonic_ns(),
transport_timestamp_us=1,
user_timestamp_us=time.monotonic_ns() // 1000,
user_timestamp_us=time.monotonic_ns() // 1000 if stamped else None,
frame_id=None,
user_data=None,
)
Expand Down Expand Up @@ -225,6 +225,22 @@ async def test_a_wrong_sized_first_frame_fails_naming_the_stream_and_leaves_the_
assert FakeReceiver.instances[0].stops == 1


async def test_a_first_frame_without_a_capture_time_fails_prepare_and_leaves_the_room():
streams = [livekit_stream(16)]
source = make_source(streams)
with pytest.raises(RuntimeError, match=r"cam16: the first frame carries no capture time"):
await prepare_with_frames(source, streams, stamped=False)
assert FakeReceiver.instances[0].stops == 1


async def test_a_first_frame_with_a_capture_time_prepares():
streams = [livekit_stream(16)]
source = make_source(streams)
await prepare_with_frames(source, streams, stamped=True)
assert FakeReceiver.instances[0].stops == 0
await source.stop()


async def test_a_lost_track_fails_naming_the_stream_and_leaves_the_room():
streams = [livekit_stream(16)]
source = make_source(streams)
Expand Down
75 changes: 75 additions & 0 deletions examples/egocap/tests/backend/test_recording_session_livekit.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,12 @@
import asyncio
from pathlib import Path

import numpy as np
import pytest

from capture_server import protocol as p
from egocap_web.livekit_receiver import ReceivedFrame, StreamCounters
from egocap_web.livekit_recording import LiveKitRecordingFeed
from egocap_web.recording_session import CoordinatorBusyError, RecordingSessionCoordinator
from recording_service import (
Disposition,
Expand Down Expand Up @@ -254,3 +258,74 @@ async def test_an_interrupted_takes_evidence_does_not_describe_the_next_take(tmp

assert calls == [(two.episode_id, {"event_time": tracking})]
await coordinator.close()


class _UnstampedReceiver:
"""Delivers one frame with no publisher timestamp, then stays quiet."""

class _Clock:
discontinuity = False

def mapping(self):
return None

def __init__(self) -> None:
self.clock_sync = self._Clock()
self.counters = {16: StreamCounters()}
self._sent = False

async def next_frame_or_lost(self, stream_id, timeout=None):
if self._sent:
await asyncio.Event().wait()
self._sent = True
return ReceivedFrame(
stream_id=stream_id,
image=np.zeros((4, 6, 3), dtype=np.uint8),
receive_time_ns=0,
transport_timestamp_us=1,
user_timestamp_us=None,
frame_id=None,
user_data=None,
)


class _RealFeedSource(FakeSource):
"""A source whose ``start`` runs the real feed, so its own failure reason reaches the coordinator."""

def start(self, ingress, on_failure) -> None:
super().start(ingress, on_failure)
stream = p.LiveKitCameraStream(
stream_id=16,
name="head",
url="ws://lk",
room="r",
publisher="o",
track="t",
timestamp_basis=p.LiveKitTimestampBasis.SENSOR_CAPTURE_MONOTONIC_US,
)
self.feed = LiveKitRecordingFeed(
_UnstampedReceiver(), [stream], ingress, {16: "head"}, on_failure=on_failure
)
self.feed.start()

async def stop(self) -> None:
await super().stop()
await self.feed.stop()


async def test_a_frame_without_a_publisher_timestamp_ends_the_take_and_leaves_it_recoverable(tmp_path):
source = _RealFeedSource([])
coordinator = make_coordinator(tmp_path, source)
await coordinator.open()
coordinator.set_task("t")
await coordinator.start_episode()
for _ in range(300):
snap = coordinator.snapshot()
if snap.service.phase is RecordingPhase.RECOVERY_REQUIRED and snap.active_operation is None:
break
await asyncio.sleep(0.01)
snap = coordinator.snapshot()
assert snap.service.phase is RecordingPhase.RECOVERY_REQUIRED
assert "a frame arrived without the publisher's frame timestamp" in (snap.issue or "")
assert snap.service.last_result is None or snap.service.last_result.disposition is not Disposition.KEPT
await coordinator.close()
Loading
Loading