Configurable Latency Budget Policies for Real-Time Agent Turns

[bhargavikalicheti]

Feature Type

Nice to have

Feature Description

Feature: Configurable Latency Budget Policies for Real-Time Agent Turns

Is your feature request related to a problem?

LiveKit Agents provides useful latency and performance measurements across the real-time voice pipeline, including end-of-utterance timing, LLM time-to-first-token, TTS time-to-first-byte, and turn-level metrics.

These metrics provide good observability into agent performance, but applications currently have limited framework-level support for reacting to latency degradation while a session is running.

For production real-time voice applications, latency is often an operational constraint rather than only an observability metric. An application may have a conversational latency target and may want to change its behavior when that target is approached or exceeded.

For example, an application might want to:

• emit telemetry when conversational latency approaches an SLO;
• select a lower-latency execution path;
• reduce optional processing;
• use a fallback model/provider;
• adjust application-specific response behavior; or
• record a latency-budget violation for subsequent analysis.

Today, applications can consume metrics and implement this logic independently, but doing so requires users to build their own correlation, threshold evaluation, state management, and callback mechanisms.

Describe the solution you'd like

I would like to propose an optional latency-budget mechanism for real-time agent turns.

The intent would not be for LiveKit to prescribe what an application should do when latency is high. Instead, LiveKit could provide a lightweight mechanism for:

1. defining latency targets or thresholds;
2. evaluating supported turn-level latency measurements against those thresholds;
3. exposing structured information when a threshold is approached or exceeded; and
4. allowing application code to determine the appropriate response.

Conceptually:

Agent turn
    │
    ├── latency measurement
    │
    ▼
Latency budget
    │
    ├── within budget ────────> normal execution
    │
    ├── warning threshold ────> callback/event
    │
    └── budget exceeded ──────> callback/event
                                      │
                                      ▼
                              application-defined action

For example, an application could configure a target and react to a violation:

# Conceptual API only — not proposing a specific interface.

async def on_latency_budget_exceeded(event):
    # Application-defined behavior:
    # - emit telemetry
    # - choose a faster execution path
    # - modify subsequent agent behavior
    # - trigger application-specific fallback logic
    pass

I intentionally haven't proposed a concrete API because this may fit better as an AgentSession option, event, callback, metrics extension, or another abstraction consistent with the current architecture.

A minimal first version could potentially support:

• a configurable turn-latency threshold;
• an optional warning threshold;
• structured threshold/breach information;
• an application-defined callback or event;
• integration with existing turn-level latency measurements rather than introducing another independent metrics system.

More advanced adaptive behavior could remain entirely application-defined.

Describe alternatives you've considered

1. Consuming metrics directly

Applications can subscribe to existing metrics and implement threshold evaluation themselves.

This provides maximum flexibility, but every latency-sensitive application needs to independently implement correlation, threshold handling, state management, and policy execution.

2. External observability/SLO systems

Latency metrics can be exported to external monitoring platforms and alerts can be created there.

This is useful for operational monitoring but generally happens outside the conversational execution path and therefore is less suitable when the application needs to adapt behavior during an active session.

3. Hard-coded automatic fallback behavior

LiveKit could automatically switch LLMs, TTS providers, or other components when latency becomes excessive.

I don't think this should be the initial approach because appropriate fallback behavior is highly application-specific. A generic threshold/event mechanism would keep the framework flexible while allowing developers to implement their own policies.

Additional context

The motivation is production real-time voice applications where conversational responsiveness is part of the application's reliability requirements.

I think this could complement LiveKit's existing latency observability by introducing a distinction between:

observability

"How long did this turn take?"

and

runtime policy

"The latency budget was exceeded; should the application react?"

The feature should preferably reuse LiveKit's existing canonical turn-level latency measurements rather than defining a new latency formula.

I'd be happy to work on an initial implementation, tests, documentation, and an example if this direction fits the Agents architecture.

Before implementing anything, I'd appreciate maintainer feedback on:

1. Does a latency-budget abstraction belong in LiveKit Agents, or is consuming the existing metrics API considered the preferred approach?
2. If it belongs in Agents, would an event/callback mechanism be preferable to a dedicated policy abstraction?
3. Which existing turn-level latency measurement should be considered canonical for threshold evaluation?
4. Would you prefer the initial implementation to expose only threshold events and leave all adaptive behavior to application code?

Workarounds / Alternatives

No response

Additional Context

No response