Skip to content

feat(observability): supervisor OTLP telemetry relay - #3196

Open
rhuss wants to merge 8 commits into
NVIDIA:mainfrom
rhuss:feat/supervisor-otlp-relay
Open

feat(observability): supervisor OTLP telemetry relay#3196
rhuss wants to merge 8 commits into
NVIDIA:mainfrom
rhuss:feat/supervisor-otlp-relay

Conversation

@rhuss

@rhuss rhuss commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Add an OTLP relay to the sandbox supervisor that accepts agent-emitted traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocol
  • Enable zero-config observability for any OTel-instrumented agent inside network-isolated sandboxes without requiring egress policy exceptions
  • Validate via spike (feat(observability): agent-level trace collection via supervisor OTLP relay #2641) with zero measurable latency impact up to 10,000 spans/s on the shared session channel

Related Issue

Closes #2641

Changes

Proto (proto/openshell.proto)

  • TelemetryData message with sandbox_id, trace_data (serialized OTLP), and ocsf_events
  • capabilities field on SupervisorHello and SessionAccepted for feature negotiation
  • Telemetry variant on SupervisorMessage

Supervisor OTLP Module (crates/openshell-supervisor-network/src/otlp/)

  • receiver.rs: hyper-based HTTP server accepting POST /v1/traces (protobuf + JSON), TCP_NODELAY, dual spawn variants for netns vs direct bind
  • enrichment.rs: Injects sandbox resource attributes (openshell.sandbox.id, .workspace_id, .policy, .user, .image, .driver) and openshell.telemetry.source: "agent" routing marker
  • buffer.rs: Bounded channel (4096 slots) with drop-newest semantics, atomic drop counter and queue depth gauge
  • mod.rs: TelemetryRelay orchestrator, RelayHandle for lifecycle, RateLimitedOcsfSink for OCSF events, forwarder with non-blocking try_send

Supervisor Process Integration

  • child_env.rs: OTEL env var injection helper
  • process.rs / ssh.rs: Set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL for agent processes
  • supervisor_session.rs: Advertise telemetry_relay capability, gate forwarding on gateway confirmation, log capability negotiation outcome

Sandbox Lifecycle (crates/openshell-sandbox/src/lib.rs)

  • Relay startup with driver-appropriate binding (netns for Docker/Podman, direct for K8s/VM)
  • Graceful shutdown: drain relay before network teardown so short-lived agents don't lose final spans

Gateway (crates/openshell-server/)

  • telemetry_relay.rs: Dedicated TelemetryRelayExporter with separate gRPC client to OTLP collector (preserves supervisor-enriched resource attributes), info-level logging on connect
  • supervisor_session.rs: Capability negotiation with logging (confirm telemetry_relay only when OTLP is configured, log why when not confirmed)

Configuration

  • deploy/docker/gateway.toml: Commented-out [openshell.gateway.otlp] section as setup reference

Supporting

  • openshell-otel/propagation.rs: HeaderMapInjector and inject_traceparent_if_missing() for W3C context propagation
  • openshell-ocsf: OcsfRelaySink trait and OcsfRelayLayer (implemented but not wired into subscriber, follow-up)
  • architecture/sandbox.md: Telemetry relay section with data flow diagram

Testing

  • cargo check passes for all relay crates
  • 17 unit tests covering buffer, enrichment, rate limiter, and forwarder
  • Local Docker smoke test: traces flow from sandbox agent curl through supervisor relay to Jaeger via gateway
  • End-to-end validated on SAW cluster (demo.redhat.com OCP 4.22.8) with traces flowing to MLflow + Jaeger
  • Performance: zero latency impact at sustained 10k spans/s (spike report)

Checklist

  • Tests pass
  • Documentation updated (architecture/sandbox.md)
  • Changes are scoped to the issue
  • No secrets or credentials committed

@copy-pr-bot

copy-pr-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rhuss
rhuss marked this pull request as draft September 5, 2026 05:44
rhuss added 4 commits September 5, 2026 08:50
@rhuss
feat(observability): supervisor OTLP telemetry relay
f5784ab 
Add an OTLP relay to the sandbox supervisor that accepts agent-emitted
traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes,
and forwards them to the gateway over the existing session protocol.

The relay enables zero-config observability for OTel-instrumented agents
inside network-isolated sandboxes. Agents export to the standard OTLP
endpoint; the supervisor handles enrichment, buffering, and forwarding
without requiring egress policy exceptions.

Key components:
- OTLP HTTP receiver (protobuf + JSON) with per-driver binding (netns
  for Docker/Podman, direct for K8s/VM)
- Span enrichment with sandbox identity (sandbox_id, workspace_id,
  policy, user, image, driver) and telemetry source marker
- Bounded buffer (4096 slots) with non-blocking try_send forwarding
- Gateway-side TelemetryRelayExporter with dedicated gRPC client that
  preserves supervisor-enriched resource attributes
- Capability negotiation: supervisor advertises telemetry_relay, gateway
  confirms only when OTLP export is configured
- Graceful shutdown ordering: relay drains before network teardown

Proto: TelemetryData message, capabilities on SupervisorHello and
SessionAccepted.

Closes: NVIDIA#2641

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
fix(observability): address review findings in OTLP relay
144f674 
Fix Critical and Important findings from multi-agent code review:

- Fix shutdown deadlock: drop telemetry sender before awaiting forwarder
  to allow the mpsc channel to close
- Limit OTLP request body to 4 MiB (http_body_util::Limited), return
  413 Payload Too Large when exceeded
- Replace expect() with ConnectError on malformed OTLP endpoint URI to
  prevent gateway panic on startup
- Add 10s timeout on telemetry export gRPC calls to prevent unbounded
  task accumulation during collector brownout
- Validate OCSF events as JSON before logging and emit as structured
  field to prevent log injection via embedded newlines
- Cap concurrent OTLP receiver connections at 64 via semaphore
- Strip existing trusted keys before enrichment to prevent agent-supplied
  attribute spoofing
- Fix rate limiter TOCTOU race with fetch_update CAS loop
- Log session drop counter during shutdown for observability
- Log on supervisor session try_send failure instead of silent discard

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
docs(observability): clarify relay port binding and topology awareness
b7881f9 
Clarify in architecture/sandbox.md that the OTLP receiver only binds
port 4318 when the relay is active (gateway has OTLP configured and
confirms the telemetry_relay capability). Document topology constraints:
all current topologies keep the process supervisor co-located with the
agent, so 127.0.0.1 is correct. Note that future topologies moving the
process supervisor out of the workload pod would need a different address.

Remove the netns_fd gate on OTEL env var injection in process.rs since
all topologies keep the supervisor co-located and the env vars are
harmless when the relay is not running (OTel SDKs handle unreachable
endpoints gracefully).

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
fix: revert accidental podman driver and mise.lock changes
e6cada9 
The cherry-pick conflict resolution for podman container.rs used
--theirs which in a cherry-pick context takes the spike's version,
not main's. This accidentally included unrelated spike changes
(TLS secret prefix removals, SPIFFE mount changes). Revert to
upstream/main's version since our PR has no podman changes.

Also revert mise.lock changes that were auto-generated when the
pre-commit hook installed tools.

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
rhuss force-pushed the feat/supervisor-otlp-relay branch from 2038d0d to e6cada9 Compare September 5, 2026 06:51
rhuss added 3 commits September 5, 2026 08:55
@rhuss
refactor(observability): extract OTLP receiver address constants
cb274db 
Replace hardcoded "127.0.0.1:4318" with constants in sandbox_env.rs
(OTLP_RECEIVER_ADDR and OTLP_RECEIVER_ENDPOINT) so the bind address
and agent env var are defined in one place. Future topologies that move
the process supervisor out of the agent's network namespace can update
these constants or derive the address from the topology.

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
test(observability): add unit tests for OTLP relay buffer, enrichment…
db992a8 
…, and forwarder

Add 13 new tests bringing the relay test suite from 4 to 17 tests:

Buffer (4 new):
- depth_tracks_send_and_recv: queue depth accuracy across operations
- drop_count_increments_on_each_overflow: counter accuracy under sustained overflow
- metrics_shared_across_clones: shared metrics between sender clones
- recv_returns_none_when_all_senders_dropped: channel close behavior

Enrichment (4 new):
- enrichment_strips_agent_supplied_trusted_keys: dedup prevents spoofing
- enrichment_preserves_non_trusted_agent_attributes: custom attrs kept
- enrichment_handles_json_content_type: JSON input, protobuf output
- enrichment_rejects_invalid_protobuf: error handling for bad input

Rate limiter (3 new):
- rate_limiter_acquires_initial_tokens: exact token count
- rate_limiter_drops_when_exhausted: OcsfRelaySink drop counting
- rate_limiter_refills_after_time: time-based token refill

Forwarder (2 new):
- forwarder_constructs_telemetry_data_messages: correct message construction
- forwarder_increments_session_drop_counter: session channel backpressure

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
feat(observability): add OTLP relay capability negotiation logging
48e9ece 
The OTLP relay was silently disabled when the gateway config lacked an
[openshell.gateway.otlp] section, with no log output at any decision
point. This made it impossible to diagnose why telemetry data was not
flowing from supervisor to gateway.

Add info-level logging when the relay exporter connects and when
capabilities are confirmed, and debug-level logging for the normal
inactive paths. Add a commented-out OTLP section to the Docker gateway
config as a setup reference. Fix pre-existing rustfmt and clippy issues
in OTLP relay code.

Signed-off-by: Roland Huß <rhuss@redhat.com>
@rhuss
rhuss marked this pull request as ready for review September 5, 2026 13:38
@rhuss

rhuss commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@krishicks @drew this is a PR for introducing an OTEL relay to the supervisor, forwarding OTLP traces to a collector via the gateway. This is the hardened OTEL-relay spike that I did for creating this report in #2641 (comment)

It would be great if you could put your review agent army on this PR (I did some self-review, but we all know the more reviews the merrier :)

See also https://github.com/rhuss/OpenShell/blob/48e9ece979889aa49b1df146c3d0d08e7118496a/architecture/sandbox.md#telemetry-relay for a recap of the overall architecture.

@rhuss rhuss mentioned this pull request Sep 5, 2026
Open
2 tasks
@rhuss
refactor(observability): rename TelemetryData to OtelExportData
250fee2 
Rename the proto message and all related types to clarify that this
carries OpenTelemetry signal data destined for an OTLP collector, not
product analytics telemetry. Add a `oneof signal` wrapper around
`trace_data` so future metrics and logs signals can be added without
breaking the wire format.

Renames:
- Proto: TelemetryData -> OtelExportData, oneof payload field
  telemetry -> otel_export
- Capability: "telemetry_relay" -> "otel_export"
- Gateway: TelemetryRelayExporter -> OtelRelayExporter,
  telemetry_relay.rs -> otel_relay.rs
- Supervisor: TelemetryRelay -> OtelRelay
- All related variables, functions, and log messages

Signed-off-by: Roland Huß <rhuss@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(observability): agent-level trace collection via supervisor OTLP relay

1 participant

@rhuss