feat(narratives): add HMAC-chained transcripts, state slots, and multi-turn context

[nick-nlb]

Overview

This PR is the fourth stage of the Narratives model loop migration. It replaces the unsigned, client-assembled history array on /agent/chat/stream with an HMAC-SHA256-chained Transcript window, extracts grounded ConversationStateSlots (QueryScope entries) from each turn's MCP tool results and validated chart configurations, compacts turns beyond the six-turn sliding window into a bounded summary, and passes verified conversation context into both the MCP tool loop and the synthesis phase.

Isolating conversation history behind Transcript (load_transcript, transcript_contents, and finalize_turn) keeps the model loop independent of how history is transported, so moving conversation history to server-side session storage later only requires updating load_transcript and finalize_turn. On the client, useSseChat persists the signed fields returned on each complete terminal frame (turn_index, hmac, state_slots, compacted_summary, and the re-signed window array) in localStorage and sends only verified turns on subsequent requests.

Related Issues

This PR follows #494, #495, and #496 in the Narratives model loop migration.

Changes Made

• Settings: Added TRANSCRIPT_HMAC_SECRET (Settings.transcript_hmac_secret as a SecretStr, validated to at least 32 UTF-8 bytes when set) in settings.py, with a per-process random fallback key for local development and a startup warning on Cloud Run (K_SERVICE) when unset.
• State-slot extraction (workflows/state_slots.py): Added extract_state_slots to build ConversationStateSlots (QueryScope records mapping place DCIDs, parent-place cohorts, and statistical variable DCIDs to single-line display names, plus min-to-max observation date ranges) strictly from MCP observation calls that returned rows and rendered chart configurations whose DCIDs were grounded in tool payloads. Added has_observation_rows in mcp/data_utils.py.
• Transcript verification, formatting, and compaction (workflows/transcript.py): Added Turn, Transcript, SignedWindow, TranscriptError, load_transcript, transcript_contents, and finalize_turn. Each turn's hmac chains over the domain tag (narratives.transcript.turn.v1), the predecessor's hmac, the turn's fields, and the window's compacted_summary. When a completed turn pushes the window past MAX_VERBATIM_TURNS (6), finalize_turn summarizes evicted turns via Gemini (with a 5s timeout and a deterministic fallback summary) and re-signs the retained window.
• Chat route and workflows (server/routes/chat.py, workflows/chat_pipeline.py, workflows/mcp_loop.py): Bounded request bodies to MAX_REQUEST_BYTES (4 MiB, returning HTTP 413 request_too_large), replaced ChatRequest.history with turns and compacted_summary validated by load_transcript before opening the SSE stream (returning HTTP 400 transcript_invalid on failure), passed Transcript into execute_mcp_tool_loop and run_synthesis_phase, and populated turn_index, hmac, state_slots, compacted_summary, and window on the complete terminal frame.
• UI (ui/src/hooks/use_sse_chat.ts): Added transcriptRequestFields and applySignedWindow to store signed turns in ChatSessionProvider (localStorage), send turns and compacted_summary instead of history, update stored signatures when compaction re-signs the window, and clear stored signatures when the agent returns HTTP 400 transcript_invalid.
• Tests and docs: Added unit tests in transcript_test.py, state_slots_test.py, settings_test.py, chat_test.py, chat_pipeline_test.py, mcp_loop_test.py, use_sse_chat.test.ts, and chat_session_context.test.ts, and updated README.md and docs/smoke.sh.

Behavior changes

• Verified multi-turn context in the MCP tool loop and synthesis: Follow-up queries (such as "compare them to Germany" or "what about 2020?") now receive up to 6 verbatim prior turns (MCP_HISTORY_RESPONSE_CHARS = 2_000 per prior answer in the MCP loop; full answers in synthesis), along with the compacted summary of older turns and the structured QueryScope descriptions wrapped in <conversation_context> tags.
• Pre-stream request validation and rejection recovery: Requests over 4 MiB fail before streaming with HTTP 413 (request_too_large). Requests carrying an altered turn, a mismatched signature chain, non-contiguous turn indexes, duplicate idempotency keys, or fields outside schema caps fail before streaming with HTTP 400 (transcript_invalid). When the UI receives HTTP 400 transcript_invalid (for example, after a key rotation or an unconfigured server restart), it clears stored signatures so the next message starts a fresh context window rather than failing every subsequent turn.
• Sliding-window compaction: After 6 signed turns, the oldest turn is compacted into compacted_summary (capped at 8,000 characters) and the retained 6 turns are re-signed from the front of the window. If the compaction model call times out (5s) or errors, a deterministic fallback summary is built from the evicted turns so compaction never fails a turn whose answer has already streamed.
• Oversized answers complete unsigned: If a synthesized answer exceeds MAX_RESPONSE_CHARS (32,000 characters) or signing fails unexpectedly, the turn still completes normally without signing fields on the terminal frame, and the browser omits that turn from future context requests while retaining earlier signed turns.

Testing Done

Verified the Python agent locally from narratives/agent/:

uv sync --frozen
uv lock --check
uv run ruff format --check .
uv run ruff check .
uv run mypy
uv run pytest -v

Ran the UI type-check, the full workspace test suite, and the production build from narratives/ with the Node version from .nvmrc:

nvm use
pnpm -C ui run lint
pnpm test
pnpm build

All 146 UI tests and 611 agent tests pass.

Manual verification against a live MCP server:

1. In narratives/agent/, export MCP_SERVER_URL=https://api.datacommons.org/mcp, DATA_PLANE_URL=https://api.datacommons.org, DATA_PLANE_WEB_URL=https://datacommons.org, DC_API_KEY, and TRANSCRIPT_HMAC_SECRET=$(openssl rand -hex 32), then run uv run narratives-agent-dev.
2. In narratives/, run pnpm -C ui run dev and open http://localhost:3000.
3. Ask "What is the population of France?" and confirm that the terminal SSE frame includes turn_index: 0, hmac, state_slots, and window.
4. Ask a follow-up question that relies on prior context, such as "How does that compare to Germany?", and confirm that the request body carries turns (with turn 0's signature and state slots) and that the agent resolves the comparison variable and entities from the first turn.

• Unit tests passed
• Integration tests passed [NOT APPLICABLE]
• Manual verification
• Any updated goldens or fixtures were reviewed and are intentional [NOT APPLICABLE]

Risk & Rollback

Low risk. Narratives has no active deployments. For multi-instance Cloud Run deployments, TRANSCRIPT_HMAC_SECRET should be configured so all instances share the same signing key; if unset, each instance generates an ephemeral key and logs a warning at startup. To roll back, revert the merge commit on narratives-dev.

Follow-ups

• Update the default Gemini model configuration in the next stage of the migration.

Checklist

• I have read AGENTS.md and followed CODING_GUIDELINES.md, plus FRONTEND.md for UI changes.
• I have run the app's lint, test, and build commands, as documented in that application's guide.
• I have commented my code, particularly in hard-to-understand areas.
• My changes generate no new warnings.

---

Note: Only Maintainers can approve and merge PRs. Expected initial review time: 3 business days.

[gemini-code-assist]

n/a

• This review was triggered when the PR was incorrectly based off of main. None of the comments are relevant. I will request another.

[nick-nlb]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements secure transcript signing, verification, and compaction to support multi-turn context in the Narratives agent. It introduces a sliding window of signed turns (up to 6) using HMAC-SHA256 to prevent client-side tampering, and folds older turns into a compacted summary when the window overflows. The backend endpoint /chat/stream is updated to enforce request size limits and verify transcripts before starting a turn, while the frontend is updated to store and round-trip these signed transcripts. Feedback on the newly added state_slots.py suggests simplifying the _clean_name whitespace collapsing, making the _strings helper more robust for dictionary values, and avoiding redundant _display_name evaluations by checking dictionary membership first.

[juliawu]

This looks great! Left some comments found with the Gemini skill.