[wip] Add session observability with tags and LLM-based evals

[bcherry]

Summary

Adds comprehensive session observability to the agent by implementing session end callbacks that tag conversations and evaluate quality using LLM-as-judge verdicts.

Key Changes

• New on_session_end callback: Implements session lifecycle observability with:

  • Automatic success/failure tagging based on conversation engagement (3+ messages threshold)
  • LLM-based evaluation using built-in judges (safety, relevancy, conciseness)
  • Graceful handling of sessions that fail during setup

• Judge integration: Evaluates every session using JudgeGroup with three production-ready judges:

  • safety_judge() - Ensures responses are safe and appropriate
  • relevancy_judge() - Validates response relevance to user queries
  • conciseness_judge() - Checks response conciseness

• Session registration: Wires the callback into the agent via on_session_end parameter in @server.rtc_session decorator

• Documentation update: Enhanced README to explicitly mention session tags and production evals as key features

Implementation Details

• Session reports are safely generated with error handling for sessions that never started
• Chat history is filtered to exclude function calls and system instructions for meaningful engagement detection
• Each judge verdict is automatically tagged as lk.judge.<name> for filtering and analysis in LiveKit Cloud
• The implementation follows the production evals pattern documented in LiveKit's observability guide

https://claude.ai/code/session_01Mkt7G86CNXSTmhRrwHcL2A