diff --git a/.config.yaml.template b/.config.yaml.template index ded4c2d5..46b645f8 100644 --- a/.config.yaml.template +++ b/.config.yaml.template @@ -82,6 +82,10 @@ ASR: # Optional decoder bias — short list of canonical phrases improves # recall on rare names. Keep under ~200 chars to avoid prompt bleed. initial_prompt: null + # Bias the decoder toward the wake-name so it stops coming back as + # Jody / Donny / Duddy (#164). null disables. Needs faster-whisper >= 1.0 + # (ignored with a warning on older builds). + hotwords: SenseVoiceOnnx: # int8 sherpa-onnx SenseVoiceSmall — no PyTorch, lighter for Pi-class hosts # (#135). Opt-in: set selected_module.ASR: SenseVoiceOnnx to use it. diff --git a/.env.example b/.env.example index 4a929828..7386f0fa 100644 --- a/.env.example +++ b/.env.example @@ -91,6 +91,16 @@ PORT=8081 # Same key as above works if your provider supports vision models. # VISION_API_KEY=sk-or-v1-... +# Keep camera frames and room audio on the LAN (dotty-behaviour). When set, +# any VLM / audio-caption / narrative URL that is not loopback or a private +# address is disabled at startup instead of falling back to the cloud +# default, and weather + calendar fetches are switched off. Point the VLM at +# a local OpenAI-compatible vision model to keep the camera working: +# DOTTY_LOCAL_ONLY=1 +# VLM_API_URL=http://127.0.0.1:8080/v1/chat/completions +# VLM_MODEL= +# VLM_API_KEY=local # any non-empty value; local servers ignore it + # --- Google Calendar integration (optional) --- # Comma-separated Google Calendar IDs for context injection. diff --git a/.github/workflows/firmware-release.yml b/.github/workflows/firmware-release.yml index 409d0987..43178079 100644 --- a/.github/workflows/firmware-release.yml +++ b/.github/workflows/firmware-release.yml @@ -1,6 +1,12 @@ # Build StackChan firmware and publish a GitHub Release. # -# Triggered by tags matching fw-v* (e.g. fw-v1.0.0). +# Manual only (2026-10-21): the firmware's server address is compiled in +# (CONFIG_OTA_URL), so these binaries only work on the network they were +# built for. Nothing here stops that: a manual run publishes whatever +# CONFIG_OTA_URL the pinned source has, so don't run it until #190 makes a +# prebuilt that can be pointed at any server, then restore the fw-v* tag +# trigger. A manual run names the release after the ref it runs on, so run +# it from a tag. # Uses the Espressif IDF Docker container to build the ESP32-S3 firmware, # then attaches the binaries + SHA256 checksums to a GitHub Release. # @@ -11,9 +17,7 @@ name: Firmware Release on: - push: - tags: - - 'fw-v*' + workflow_dispatch: permissions: contents: write diff --git a/.gitignore b/.gitignore index bafd54a5..4a7562d7 100644 --- a/.gitignore +++ b/.gitignore @@ -38,6 +38,23 @@ sbom-server.json htmlcov/ .pytest_cache/ +# Local UAT evidence. Keep compact manifests/results/draft issue summaries +# visible for selective review, but never offer raw household media, event +# streams, health snapshots, or container logs for an accidental commit. +/uat-sessions/**/video/ +/uat-sessions/**/clips/ +/uat-sessions/**/logs/ +/uat-sessions/**/ndjson/ +/uat-sessions/**/snapshots/ + +# Generated audit/session outputs. These can contain machine-local paths, +# prompts, or deployment observations; regenerate or curate before publishing. +/AUDIT-REPORT.md +/audit-draft-issues.md +/pi-session-*.html +# Session-id file a test writes when a DB path is ":memory:". +*.ses + # Frontend build deps for the dashboard's vendored Tailwind bundle. # package.json + tailwind.config.js are committed; build artifacts and # the npm install state are not. Re-run `npm install && npm run build:css` diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f22feb5..3ae2b171 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +## [server-v0.2.0-rc.1] - 2026-10-24 + ### Added +- **Release test tooling** (#182): `scripts/dotty_integration_check.py` checks the live stack over SSH without opening the microphone: xiaozhi admin routes, dotty-behaviour and the dashboard, with a small head nudge, toggle flips and set-state idle that are all restored afterwards, and `scripts/dashboard_smoke.py` is a read-only checker for every `/ui` fragment, stream and asset that also flags an old bridge build. `scripts/dotty_render_prompts.py` renders the A/V prompts with Piper. The A/V catalogue now covers conversation, all seven voice tools, the states with a safe way back to idle, the cold wake word and "sleep ignores voice". See `docs/dotty-overnight.md`. +- **Dashboard replays recent turns** (#183): `/ui/events` replays the last turns when the Turns tab opens and resumes from the last one seen after a reconnect. +- **`POST /xiaozhi/admin/end-session`**: closes the device's listening session. Setting the state to idle or sleep from the dashboard or admin API now does the same, so the microphone closes instead of waiting for the 120 s silence timeout. +- **Overnight A/V runner can run unattended** (#182) — opt-in `--mic-opener admin_say` opens the microphone through `/xiaozhi/admin/say` before a warm case; playback is also proven by the robot's own recognition; reference-microphone slips and extra speech in the room are `INCONCLUSIVE` rather than failures; a case the evidence cannot settle is set aside after three attempts. `dotty_av_clips.py export --crop` fills the 9:16 frame from a portrait source region. See `docs/dotty-av-tests.md`. - **Kid Mode: blocked-words content filter on the live voice path** (#157, closes the gap tracked in #138) — the pure matcher (three severity tiers + the kid-safe replacement) moved from `bridge/text.py` into the shared `custom-providers/textUtils.py`, the single source of truth both containers import; the bridge keeps its metrics/safety-ring/logging wrapper on top with unchanged behaviour. Both live LLM providers (`PiVoiceLLM`, `OpenAICompat`) now wrap their TTS-bound streams in `filter_tts_stream()`. In Kid Mode it drains and checks the complete response before TTS, then emits the original clean chunks or atomically replaces a blocked turn; outside Kid Mode it remains a transparent streaming passthrough. Full-turn consumption also prevents PiVoiceLLM from abandoning the RPC iterator before `agent_end` and leaking stale frames into the next turn. Honest caveat (see `docs/faq.md`): a word-level blocklist is a weak, bypassable backstop — prompt steering remains the primary defence. **Bench: needs on-device red-team verification before release sign-off.** - **Admin-API auth: `X-Admin-Token` across the whole stack** (#149, #150, #151, #152) — xiaozhi-server's `/xiaozhi/admin/*` routes now accept an `X-Admin-Token` shared secret (`DOTTY_ADMIN_TOKEN`, timing-safe compare, permissive when unset), and all three callers — the bridge dashboard, dotty-behaviour's `XiaozhiAdminClient`, and the dotty-pi voice tools' `adminFetch` — send it. The secret is now **plumbed end-to-end**: `make setup` generates one into `.env`, the compose template / all-in-one pass it to xiaozhi-server, the three service compose files document where their copy lives, and `.env.example` + `SETUP.md` §10 describe the enable-everywhere-or-nowhere semantics. Previously the code shipped with no config path, so every deploy silently stayed permissive. - **PersonResolver — one answer to "who is this?"** (`dotty-behaviour/household/resolver.py`) — identity resolution was smeared across consumers, and the 2026-06-06 audit found four separate identity bugs because of it. All resolution now funnels through one module with `Person.id` as the canonical key space. Fixed by the consolidation: **room_view roster recognition silently failing whenever `id != display_name`** (the VLM echoes display names; validation compared ids — confirmed 3/3, both the core and greeter paths), **multi-word display names never matching** (the NAME parser was single-token, so "Mary Anne" was a 100% silent miss — confirmed 3/3), **the greeter's calendar lookup dropping a person's own events on a case mismatch** (`[Hudson]` ≠ `hudson` — confirmed 2/3), and **bracketless `calendar_prefix:` YAML never matching**. `summarize_for_prompt` now matches person tags case-insensitively and accepts the resolver's tag set; the room_view test fakes were also fixed to carry real ids (the old fakes re-derived ids from display names, which is exactly what masked the bug). @@ -15,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Bridge systemd unit loads API keys from `${BRIDGE_DIR}/.env`** (#15) — `zeroclaw-bridge.service.template` and `scripts/install-bridge.sh` now emit `EnvironmentFile=-${BRIDGE_DIR}/.env`. `install-bridge.sh` creates a mode-0600 stub `.env` containing `OPENROUTER_API_KEY=` (and commented `VISION_API_KEY` / `VLM_API_KEY` placeholders) when one isn't already present, so the missing-vision-key failure surfaces as the bridge's existing ERROR ("camera offline") instead of a silent confabulation. Existing `.env` files are preserved. ### Changed +- **README status note** cut to one short paragraph that matches the project as it is now; the stale rough-edges list now points at the issue tracker. `SETUP.md` names the SoftAP portal's **Advanced** tab correctly (#172). - **Whole-repo vision-alignment pass (2026-05-29)** — a multi-agent audit found ~50 verified code/doc-drift findings from the post-#36 and #115 cutovers; this pass resolves them against six locked maintainer decisions (see `docs/vision-alignment-review-2026-05-29.md`). Highlights: dashboard port references corrected to **:8081** repo-wide (`:8080` is the llama-swap endpoint); perception re-attributed from the retired `bridge.py` `_perception_*` methods to the **11 dotty-behaviour consumer classes**; the brain's voice-tool count corrected to **7** (a nonexistent `set_led` tool was removed from the docs, the real `remember` restored); kid-safety wording fixed to state the live PiVoiceLLM sandwich ships while the blocked-words content filter remains the open gap; `make doctor`/`status`/`audit`/`setup` and `dotty_doctor.py` repointed off the retired ZeroClaw RPi at `:8080` to the dashboard (`:8081`) and dotty-behaviour (`:8090`). - **kid/smart-mode state shared across containers** — `receiveAudioHandle.py` (xiaozhi container) and the bridge dashboard now resolve the toggle state files to the same `/var/lib/dotty-bridge/state` mount, fixing a desync where the firmware LED pips drifted from the dashboard on every reconnect. - **Documentation reconciled to the post-#36 architecture** — `README.md`, `CLAUDE.md`, and the `docs/` tree previously described the retired ZeroClaw bridge and its Raspberry-Pi brain host. They now describe the live stack: the `dotty-pi` pi-agent container (the voice brain, reached via the `PiVoiceLLM` provider), `dotty-behaviour` (perception bus + ambient consumers + greeter, port 8090), and `bridge.py` as the admin dashboard service (port 8081). `.config.yaml.template` and `docker-compose.yml.template` updated to match — `vision_explain` / `VISION_BRIDGE_URL` now point at dotty-behaviour, and the `zeroclaw` provider mount + `ZeroClawLLM` config block are gone. The #36 cutover was executed 2026-05-19; this is the follow-up doc sweep its runbook deferred. @@ -26,6 +33,53 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`docs/multi-daemon-split.md`, `docs/advanced/multi-host.md`** — both documented ZeroClaw-host topologies that no longer exist. ### Fixed +- **Saved robot state survives a bad entry**: dotty-behaviour threw away every device's saved mutex state (the robot's sleep or security included) if any one device had an unexpected state, and a corrupt state file stopped the service from starting. Invalid entries are now dropped one by one and an unreadable file starts empty with a warning. +- **Kid Mode reaches vision descriptions and the greeter**: dotty-behaviour read Kid Mode from a flag nothing ever set (always off), so every room and photo description and every proactive greeting was written with the adult prompt. Both now read the shared guardian file on each use and treat a missing or unreadable file as Kid Mode on. +- **Injected turns no longer count as barge-ins** (#105): admin inject-text (the face greeter, the dashboard) and the end-of-session prompt went through the same path as speech, so injecting while Dotty spoke counted toward the four-interruptions-in-60-s close and could end a child's conversation. Only the user's own speech counts now. +- **An aborted tool loop can't spoil the next turn**: when the tool-call cap aborted a turn and pi's run did not end within 5 s (think_hard ignores the abort and can wait 90 s on a cold model), its late frames reached the next voice turn, which then said "(no response)" or "brain offline". PiClient now restarts pi in that case, and each reader thread delivers only to the queue of the process it reads. +- **Kid Mode retells think_hard answers under the rules**: the direct "think hard" route spoke the 27B reasoner's answer as-is, and since the voice rules moved into the 4B's system prompt the reasoner never saw them; only the word-level filter stood between its answer and a child. In Kid Mode the answer now goes to the 4B as context and is retold under the rules (the filter still checks it). The new context block is kept out of memory like the camera and search blocks. +- **Kid Mode danger filter blocks invitations, not facts** (#178): the imitable-danger tier blocked any listed verb near a hazard word, so "Helicopters fly using spinning blades", "Geckos can climb … ceilings", "The Little Match Girl" (`lit` matched "little") and "take your medicine" were all replaced, while "Climb up high on the shelf!" and "Grown-ups are boring, let's play with matches!" got through. A phrase now counts only when the sentence invites copying it (I/we/let's/you before the action, or a bare command); a safety cue exempts only the action it governs (a negation just before the verb, a warning after it, or asking a grown-up); swallowing hazards must be the verb's object; verb forms are exact; shelves, counters and "up high" are hazards. +- **No LAN-locked firmware releases**: pushing an `fw-v*` tag published binaries with the maintainer's server address compiled in. The firmware release workflow is now manual-only until a prebuilt can be pointed at any server (#190), and the `fw-v1.3.1`, `fw-v1.3.3-rc.1` and `fw-v1.3.3` releases are now source-only. +- **Quickstart builds the firmware from source**: step 1 told new users to download the `fw-v1.3.3` release binaries, which are compiled for the maintainer's own server and can't reach anyone else's. It now points to the build-from-source steps in SETUP.md. +- **No runaway tool loops**: with the rules in its system prompt the 4B answered "Repeat these five words" by calling `think_hard` 19 times in 30 s until the context overflowed and Dotty said nothing. `PiClient` now steers pi to answer after 3 tool calls in one reply and aborts the turn after 6, and the routing rules say `think_hard` is for hard multi-step questions only, each tool at most once per reply. +- **Kid Mode repeats ordinary words**: the jailbreak rule made the 4B refuse "Repeat these five words: purple, robot, seven, window, banana" as "part of a command to bypass my rules". Rule 7 now says repeating ordinary words is a game: say them back (mentioning spelling made it spell them out letter by letter); dangerous phrases are still stopped by rule 12 and the output filter. +- **First word no longer clipped**: xiaozhi kept only ~0.6 s of audio from before the VAD fired, so a soft first word was lost ("Do you remember…" reached ASR as "remember…"). A build-time patch (`scripts/patch-asr-preroll.py`) keeps ~1.2 s. +- **Dotty calls its tools again** (#165, #187): the ~2.8K-character tool-routing and HARD CONSTRAINTS block appended after every user turn stopped the 4B model calling tools at all; it answered "Please put on the song called tetris" without `play_song` 0 times in 6, and twice claimed actions it never took. The same rules now go in pi's system prompt (after `personas/pi_voice.md`), the user's words reach pi bare, and a Kid Mode change respawns pi with the matching rules before the next turn. The kid-mode output filter (#157) is unchanged. +- **Reproducible xiaozhi image**: the Dockerfile's pip packages were unpinned, so a rebuild could change faster-whisper or CTranslate2 underneath WhisperLocal. They are now pinned to the versions in the image that passed the 2026-10-20 A/V run. +- **No repeated line after a tool call**: after `play_song` the 4B model often said its pre-tool line again, so Dotty said "Let me play that song for you" twice. `PiClient` now holds a later agent turn's text while it only repeats the previous turn's words and drops the repeat; anything new after it is still spoken. +- **"Use your camera" takes a photo instead of inventing a room**: "Please use your camera and tell me about the room" did not match the direct camera route, and the 4B model replied "I took a snapshot and see a cozy room…" without calling `take_photo`. "Use your camera", "describe the room" and "tell me about the room" now route to `take_photo`. PiVoiceLLM's own log lines are f-strings, since loguru ignores `%s` arguments, and successful direct tools are logged. +- **PiClient log lines reach the xiaozhi log**: `PiClient` logs through stdlib `logging`, which xiaozhi-server leaves at WARNING, so `PiClient: tool call name=…` and the other INFO lines were dropped and the A/V cases could not see which tool the agent chose. They now go to the provider's logger. +- **Memory lookup survives one misheard word**: `memory_lookup` is an exact phrase search, so "the test work for tonight" found nothing for "the test word for tonight". When the phrase misses, it now retries on the query's content words and keeps only memories containing at least two of them and 60%; pending person facts stay excluded (#53). +- **No reply to a phantom "Dotty"** (#105, #164): the Whisper hotword bias turned idle servo noise into a lone "Dotty" (no-speech 0.43) and Dotty answered it. A transcript that is only the hotword now has to score no-speech 0.3 or below. +- **WhisperLocal works on the GPU after an image rebuild**: CTranslate2 loads `libcublas.so.12` by name and never found the copy in the pip CUDA wheels, so a freshly built xiaozhi image failed every utterance ("libcublas.so.12 is not found"). The Dockerfile now puts those library directories on `LD_LIBRARY_PATH`. +- **Explicit tool requests route reliably** (#165): xiaozhi's `[ROOM_VIEW]` and speaker prefixes stopped the direct tool routes matching and were written into memory. pi now runs with only the seven voice tools (no file or shell tools) and without auto-compaction, which had fired every turn on the 4B model. Camera and room views are kept out of `brain.db`. +- **`play_song` works again**: `dotty-pi` had no `XIAOZHI_HOST`, so the tool always answered "can't reach xiaozhi-server". The songs catalogue now lists and plays the MIDI songs (rendered with fluidsynth), not only `purr.opus`. A song requested mid-reply now starts after the reply finishes (`play-asset` `after_turn`); before, the reply's own speech state dropped it. **Bench: T07.** +- **Server-pushed speech is no longer dropped** (#104): `admin/say` and greeter audio were discarded by `sendAudioMessage`'s stale-turn check. Needs the xiaozhi image rebuilt (the patch is applied at build time). +- **Sticky state survives a reconnect** (#170): security, sleep and story time are restored to the device when its WebSocket reconnects. +- **Whisper hears "Dotty"** (#164): WhisperLocal takes a `hotwords` bias (`hotwords: ` in the template). Name corrections now apply only when the robot is being addressed, so "Goodnight Dolly" no longer puts it to sleep. +- **A storm of interrupted replies ends the session** (#105): four barge-ins within 60 s close the session instead of looping on room noise or the robot's own echo. +- **OpenAICompat follows the live kid-mode toggle**: it read `DOTTY_KID_MODE` once at startup, so a dashboard flip never reached the fallback provider. It now reads the shared toggle file every turn, like `PiVoiceLLM`. +- **Wake-word head turns stay out of dance and security**: `WakeWordTurner` had no state gate. (`SoundTurner` already ran only in idle.) +- **Face greetings only happen in idle and talk**: `FaceGreeter` only checked for a dance, so it could greet someone in security, story time or sleep. Both the bare and the named greeting now check the state first. +- **Head pets purr only in idle and talk**: `PurrPlayer` had no state gate, so it could purr over a dance or story, in security, or when a touch woke Dotty from sleep. +- **A lost face no longer cuts off a story or a dance**: `FaceLostAborter` aborted speech in any state shortly after a greeting. It now acts only in idle and talk, and checks the state again before the abort fires. +- **Dashboard State, Kid Mode and toggles stay in sync** (#169, #21): the State card and sensor staleness only judge the live robot, the Kid Mode card shows the persisted guardian policy, and the bridge pushes the toggle pips to the robot when it starts. +- **Test probe devices expire** from dotty-behaviour's per-device state, so a leftover probe no longer turns the robot's status dot amber. +- **Reading the latest photo no longer deletes it**: `GET /api/vision/latest/{device}` used to pop the cached photo. +- **Closing a dashboard tab no longer freezes the dashboard**: the perception feed relay could block the bridge for up to 15 s when a client disconnected. +- **Surprise and thinking faces render again** (#168) — the leading-face guard turned every emoji outside the nine supported ones into neutral, so a model reaching for `😲` or `🧐` showed no face change. `enforce_face_stream()` (shared by `PiVoiceLLM` and `OpenAICompat`) now maps near-miss emoji onto the supported set, keeps exactly one face, and puts it on the first streamed chunk as soon as it arrives. **Bench: confirm the surprise and thinking faces on the device.** +- **Kid Mode output filter covers every PiVoiceLLM path and fails closed** (#157, #138) — `think_hard` results reached TTS without the filter; all PiVoiceLLM output now runs through `guard_tts_stream()`. A matcher error replaces the turn instead of speaking unchecked text. +- **Kid Mode: on-topic replies and no imitable danger** (#178) — three new kid-mode prompt rules (answer what was asked, ask again when the words make no sense, never describe copyable physical danger) and a `danger` filter tier for phrases like "dancing around the fan blades". **Bench: re-run the #178 prompts.** +- **Voice: one abort no longer silences every later reply** (#176) — a device or admin abort set `conn.client_abort` and nothing on the `nointent` chat path cleared it, so ASR and the LLM kept running but no TTS was sent until the robot reconnected. `startToChat` now clears the flag when a new user turn is accepted. Reproduced and verified on the physical robot. +- **Asking about a friend always checks memory**: the 4B answered "What do you know about my friend Sam Baker?" with "I don't have any information" without calling `recall_person`. "What do you know about my friend / neighbour / teacher … " now runs `recall_person` directly. Nothing known gets "I don't know anything about yet", which is also what's said about a fact still waiting for review. Found by the release A/V run. +- **Named songs play** (#192): "Please put on the song called tetris" got "Tetris is a puzzle game, not a song" with no `play_song` call. "Play / put on the song (or tune) called " now runs `play_song` directly, and the catalogue decides: "Here's tetris!", or "I don't have a song called .". +- **The dashboard can wake a sleeping Dotty again**: an admin `sleep` closed the WebSocket, and a sleeping robot never reconnects, so `set-state idle` from the dashboard had no device to reach. Even without that close, xiaozhi's inactivity check dropped a sleeping robot after about 3 minutes. Now `sleep` keeps the connection, and the server keeps a sleeping device's connection alive. The firmware privacy gate still keeps the mic and wake word off. Found by the release A/V run (S05). +- **Server speech after an abort is no longer lost**: the same `client_abort` flag also silenced `/xiaozhi/admin/say`, so a greeting after `FaceLostAborter` or a dashboard stop never played until the next chat turn. The say route now clears the flag before queueing, as play-asset already did. Found by the release A/V run, where the mic opener's "Ready." after a song was dropped. +- **Live voice path now has a Dotty system prompt** (#177) — `PiClient` spawned pi with no `--system-prompt`, so pi's built-in coding-assistant prompt was the only identity the model saw ("I'm just a tiny coding assistant…"). `personas/pi_voice.md` is now passed via `--system-prompt` (`DOTTY_PI_SYSTEM_PROMPT_FILE` overrides; a missing file keeps the old behaviour). The persona also tells the 4B model to answer briefly and stop. +- **WhisperLocal drops near-silence hallucinations** (#105) — transcripts whose mean `no_speech_prob` exceeds `no_speech_threshold` (default 0.6, `1.0` disables) are discarded instead of being answered ("Thank you." after the robot stops speaking). +- **Memory: prompt scaffolding is no longer stored or spoken** (#186) — every voice turn was logged to `brain.db` with the per-turn tool-routing and HARD CONSTRAINTS text attached, and an explicit recall ("do you remember…") spoke `memory_lookup`'s raw rows. The turn logger now stores only what the person said, `memory_lookup` cleans older rows at read time, and recall passes the search results to the model as context so the answer is phrased naturally. +- **think_hard no longer fails on a cold start** (#187) — its request timeout defaulted to 30 s, shorter than the reasoner's 30–50 s cold load, so the first think-hard after an idle period was cancelled mid-load. Default is now 90 s (as the pre-cutover bridge unit had it), the direct tool runner allows 105 s, and Dotty says "Let me think hard about that." before the wait. +- **PiVoice RPC turns are serialized** (#175, thanks @0Downtime) — one connection can no longer interleave two pi transactions and consume each other's frames. - **No-GPU ASR path no longer crash-loops on first run** (#124, #136) — `make fetch-models` was requesting two SenseVoiceSmall filenames that don't exist on Hugging Face (`tokens.json` and `chn_jpn_yue_eng_ko_spectral.fbank.conf.yaml`); the real SentencePiece tokenizer is `chn_jpn_yue_eng_ko_spectok.bpe.model`. Both 404s were silently saved as 15-byte "Entry not found" stubs (curl had no `--fail`), so funasr loaded with `bpemodel=None` and `xiaozhi-esp32-server` crash-looped on every GPU-less host. The file list is corrected; **all `fetch-models` downloads now fail loudly** (`curl --fail --retry` + a size floor + delete-on-failure) instead of saving junk; and **`make doctor` now size-checks the required SenseVoice assets** so a corrupt download FAILs instead of passing. Huge thanks to **[@miltieIV2](https://github.com/miltieIV2)** — a meticulous bug report *and* a self-driven root-cause that pinned it on the `HAS_CUDA=0` FunASR switch. A lighter int8 sherpa-onnx SenseVoice runtime (no PyTorch) for Pi-class hosts has landed as the opt-in `SenseVoiceOnnx` provider (#135; see the Added section above). ## [server-v0.1.0] - 2026-05-17 diff --git a/CLAUDE.md b/CLAUDE.md index ee7f4c57..8bfc22d8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ The voice path runs through a single LLM provider — `PiVoiceLLM`, selected via ``` StackChan hardware → configured persona - │ ESP32-S3, xiaozhi firmware (built from m5stack/StackChan source) + │ ESP32-S3, xiaozhi firmware (pinned BrettKinny/StackChan@dotty fork) │ WiFi / WebSocket (Xiaozhi protocol) ▼ xiaozhi-esp32-server (Docker) @@ -197,7 +197,8 @@ For hardware specs, protocol details, model internals, latent capabilities, and - xiaozhi-esp32-server: https://github.com/xinnan-tech/xiaozhi-esp32-server - xiaozhi-esp32 firmware (upstream): https://github.com/78/xiaozhi-esp32 -- StackChan (hardware + firmware patches): https://github.com/m5stack/StackChan +- StackChan upstream (hardware + base firmware): https://github.com/m5stack/StackChan +- Dotty firmware fork (the pinned build source): https://github.com/BrettKinny/StackChan/tree/dotty - Emotion protocol: https://xiaozhi.dev/en/docs/development/emotion/ ## Agent skills diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 84d56c50..07a621e3 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -13,11 +13,16 @@ For protocol wire formats see `docs/protocols.md`. | Component | Current Version | Protocol / Interface | Breaking Change Policy | |---|---|---|---| -| StackChan firmware (m5stack/StackChan v1.2.4) | v1.2.4 | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0) | Pin firmware to a known-good build; do not OTA-update without verifying server compatibility first | +| StackChan firmware (`BrettKinny/StackChan@dotty`) | `fw-v1.3.3` release; submodule `969c2b2` | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0), StateManager event contract | Build the pinned submodule; do not substitute the official upstream tree or OTA-update without coordinated verification | | xiaozhi-esp32-server (local build) | `xiaozhi-esp32-server-piper:local` | Custom LLM provider API, `.config.yaml` schema, Xiaozhi WS server | Rebuild image only after checking upstream changelog for provider API or config schema changes | -| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the five `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing | +| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the seven `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing | | dotty-behaviour | `dotty-behaviour:0.1.0` | HTTP API (`/api/perception/*`, `/api/vision/*`, `/api/audio/*`, `/health`) | Endpoint signatures stable; perception event-schema changes require firmware + xiaozhi review | -| bridge.py (dashboard) | unversioned (HEAD) | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers | +| bridge.py (dashboard) | `dotty-bridge:0.1.0` image from repo HEAD | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers | + +The public `fw-v1.3.3` superproject tag (commit `24a009c`, 2026-07-12) is the +current coordinated release pointer. It pins firmware submodule `969c2b2` and +the matching server-side patches. Draft PRs and a dirty submodule are not a +released compatibility set. ## What counts as a breaking change @@ -42,27 +47,28 @@ Any of the following require coordinated updates across components: ## Versioning strategy -No formal versioning is adopted yet (tracked in -[ROADMAP.md](ROADMAP.md#community-wishlist) under "Firmware/server -compatibility matrix"). When adopted, the plan is: +The repo uses separate tag namespaces: + +- `server-vX.Y.Z` for server-only release milestones. +- `fw-vX.Y.Z` for coordinated firmware release pointers in this superproject. -- Separate tag namespaces: `server-vX.Y.Z` and `fw-vX.Y.Z`. -- This matrix will document which server versions are compatible with which - firmware versions. -- The bridge will carry its own version once it moves to a tagged release - cadence. +Container image tags remain `0.1.0` today and are not sufficient by themselves +to identify the exact source revision; retain the Git commit/tag alongside a +deployment record. ## Upgrade guidance 1. **Check this matrix first.** Confirm the component you are upgrading is compatible with the versions of the other components you are running. -2. **Back up before upgrading.** Run `scripts/backup.sh` (or the equivalent - manual steps) to snapshot config, persona files, and bridge state. +2. **Back up before upgrading.** Manually snapshot `.env`, rendered + `data/.config.yaml`, persona/household files, `brain.db` (including WAL/SHM + companions), and the bridge state directory. This repo does not currently + ship an automated backup script. 3. **Upgrade one component at a time.** Validate with a health check (`curl http://:8090/health` and `:8081/health`) plus a live voice turn before moving to the next component. -4. **Tail logs during validation.** Watch both the xiaozhi-server container - logs and the bridge journal simultaneously to catch mismatches early. +4. **Tail logs during validation.** Watch the xiaozhi-server, dotty-pi, + dotty-behaviour, and bridge container logs together to catch mismatches. 5. **Roll back if broken.** Restore from the backup taken in step 2 and revert to the previous image or binary. @@ -96,7 +102,11 @@ Server and firmware are versioned independently: The docs live as plain markdown under `docs/` and render on GitHub directly — there is no separate docs-site build to deploy. -GitHub Actions handles image builds and artifact publishing from the tag. +Pushing a tag does not trigger any build or publish. The firmware release +workflow (`firmware-release.yml`) is manual-only, and no firmware binaries are +published at the moment, because the server address is compiled in; a +portable prebuilt is tracked in +[#190](https://github.com/BrettKinny/dotty-stackchan/issues/190). ### Compatibility matrix updates @@ -106,4 +116,4 @@ versions work with which firmware versions. --- -Last verified: 2026-05-22. +Last verified against the repository and `fw-v1.3.3` pin: 2026-07-16. diff --git a/Dockerfile b/Dockerfile index 08b9caa9..a3f5de61 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,28 @@ # Pinned 2026-05-17 from :server_0.9.3 FROM ghcr.io/xinnan-tech/xiaozhi-esp32-server@sha256:3accd82a7d1a6c01c58f32f6199400a11655d607780b80219493123dedbb347e -RUN pip install --no-cache-dir piper-tts scipy numpy mido faster-whisper sherpa-onnx==1.13.2 +# Versions pinned to the image that passed the 2026-10-20 A/V run; an unpinned +# rebuild could otherwise change the Whisper stack underneath WhisperLocal. +RUN pip install --no-cache-dir piper-tts==1.8.0 scipy==1.15.3 numpy==1.26.4 mido==1.3.3 \ + faster-whisper==1.2.1 ctranslate2==4.8.3 sherpa-onnx==1.13.2 + +# CTranslate2 (faster-whisper) dlopens libcublas.so.12 and cuDNN by name and +# doesn't look inside the pip CUDA wheels, so WhisperLocal on device=cuda fails +# every utterance with "libcublas.so.12 is not found" without this. +ENV LD_LIBRARY_PATH=/usr/local/lib/python3.10/site-packages/nvidia/cublas/lib:/usr/local/lib/python3.10/site-packages/nvidia/cudnn/lib + +# Patch upstream TTS consumers and the audio sender to treat explicitly +# registered server-push sentence IDs as independent from chat-turn +# stale-message arbitration (#104). +COPY scripts/patch-tts-server-push.py /tmp/patch-tts-server-push.py +RUN python /tmp/patch-tts-server-push.py \ + /opt/xiaozhi-esp32-server/core/providers/tts \ + /opt/xiaozhi-esp32-server/core/handle/sendAudioHandle.py + +# Keep ~1.2 s of audio from before the VAD fires (upstream ~0.6 s), so a soft +# first word still reaches ASR. +COPY scripts/patch-asr-preroll.py /tmp/patch-asr-preroll.py +RUN python /tmp/patch-asr-preroll.py /opt/xiaozhi-esp32-server/core/providers/asr/base.py # fluidsynth + General MIDI soundfont for runtime rendering of dance/song MIDI # files to Opus. Installed as the LAST layer so iteration on Python deps above diff --git a/Makefile b/Makefile index 4f798bc0..aa182ebc 100644 --- a/Makefile +++ b/Makefile @@ -221,9 +221,10 @@ setup: _preflight-compose ## Interactive first-run wizard (re-runnable; remember echo -e "$(GREEN)$(BOLD)Setup complete.$(RESET)"; \ echo ""; \ echo "Next steps:"; \ - echo " 1. Flash the StackChan firmware (see SETUP.md or m5stack/StackChan repo)."; \ - echo " 2. In the device's Advanced Options, set the OTA URL to:"; \ - echo " http://$$XIAOZHI_HOST:8003/xiaozhi/ota/"; \ + echo " 1. Build and flash StackChan firmware with this compiled setting:"; \ + echo " CONFIG_OTA_URL=\"http://$$XIAOZHI_HOST:8003/xiaozhi/ota/\""; \ + echo " See SETUP.md. The on-device Settings app has no OTA URL editor."; \ + echo " 2. Provision the robot's 2.4 GHz Wi-Fi using its displayed setup flow."; \ echo " 3. Run 'make doctor' to verify everything is healthy."; \ echo "" diff --git a/README.md b/README.md index df050558..9641bbf0 100644 --- a/README.md +++ b/README.md @@ -4,119 +4,115 @@ # Dotty -**Your self-hosted [StackChan](https://github.com/m5stack/StackChan) robot assistant — kid-minded by default, hackable by design, private by architecture.** +**Your self-hosted [StackChan](https://github.com/m5stack/StackChan) robot assistant: kid-minded by default, hackable by design, private by architecture.** -> ⚠️ **Heads up: this is not a stable project yet.** Dotty is buggy, frequently broken, and actively changing day-to-day. End-to-end behaviour works on the maintainer's hardware but regressions land all the time, the API and config surface shifts without notice, and a fresh deploy on someone else's gear has not been verified. Treat this as a hobby-grade work-in-progress, not a polished product. Bugs, PRs, and "this didn't work for me" issues all very welcome. -> -> **Known rough edges:** face emoji rendering is missing visual differentiation for 4 of 9 emotions (sad / surprise / love / laughing); sound-direction localizer has a hardware-AEC-related left-bias on M5Stack CoreS3 (energy detection works, direction is unreliable); kid-voice ASR on the SenseVoice CPU default still garbles some short utterances (WhisperLocal, auto-selected on GPU hosts, handles high-pitched kid speech better). -> -> **Where it's at:** the first stretch of this was vibe-coded pretty hard — I moved fast, chased ideas, and let the scope sprawl to prove out the whole end-to-end loop. That phase is over. I'm now deliberately pulling it back: deleting half-built features, trimming the docs, and hardening what's left so the core is solid rather than sprawling. Fewer things, done properly. +> **Status:** a hobby project with one maintainer, built with heavy AI assistance ([AI_TRANSPARENCY.md](./AI_TRANSPARENCY.md)). It runs end to end on the maintainer's own robot and server, but nobody has verified it on anyone else's hardware yet, and config can still change between releases. Known problems are in the [issue tracker](https://github.com/BrettKinny/dotty-stackchan/issues); bug reports and "this didn't work for me" issues are welcome. -Dotty is a fully self-hosted voice stack for the M5Stack StackChan desktop robot. Open-source firmware on the robot, [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) for voice I/O, and a local **pi** coding agent as the brain. ASR, TTS, and session state all run on your own hardware. The LLM is pluggable — the shipped default runs a small fast model for plain conversation and escalates hard questions to a more capable model, with [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) as the recommended local backend. Swap in [Ollama](./docs/cookbook/run-fully-local.md) for the simpler single-binary option, or point at OpenRouter / any OpenAI-compatible API if you'd rather use the cloud. +Dotty is a fully self-hosted voice stack for the M5Stack StackChan desktop robot. Open-source firmware runs on the robot, [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) handles voice I/O, and a local **pi** coding agent is the brain. ASR, TTS and session state all run on your own hardware. The LLM is pluggable. The shipped default runs a small, fast model for plain conversation and hands hard questions to a more capable one, with [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) as the recommended local backend. [Ollama](./docs/cookbook/run-fully-local.md) is the simpler single-binary option, or you can point it at OpenRouter or any OpenAI-compatible API if you'd rather use the cloud. -Out of the box, Dotty ships in **Kid Mode** — the persona and per-turn prompt sandwich steer responses toward age-appropriate topics, while a thin blocked-words filter replaces matching spoken turns before TTS. The filter is a bypassable backstop, not a safety guarantee or substitute for supervision. Disable Kid Mode for a general-purpose assistant. +Out of the box, Dotty runs in **Kid Mode**. The persona and the Kid Mode rules in pi's system prompt steer replies toward age-appropriate topics, and a blocked-words filter replaces a matching LLM reply before it is spoken. The filter is a backstop that can be bypassed, not a safety guarantee or a substitute for supervision. Turn Kid Mode off for a general-purpose assistant. ## Why I built this -I didn't like the idea of a camera and microphone running in my house unless I could (1) self-host the whole stack end-to-end and (2) understand the whole stack end-to-end. Off-the-shelf voice assistants fail both tests — audio leaves the house, the model is opaque, and you're trusting a vendor's privacy posture forever. +I didn't want a camera and microphone running in my house unless I could host the whole stack myself and understand how every part of it works. Off-the-shelf voice assistants fail on both counts. Your audio leaves the house, the model is a black box, and you're stuck trusting a vendor's privacy policy for as long as you use it. -So Dotty is the version that passes: every component runs on hardware I own, every seam is documented and swappable, and the only thing that can leave the LAN is whatever LLM call I explicitly route out (and even that swaps to a local model with a config change). It's also meant to be fun — a friendly desktop robot for the kids, and an interesting hobby project to keep building on. +Dotty is my attempt at one that passes. Every component runs on hardware I own, and every piece is documented and can be swapped out. The only thing that can leave my network is an LLM call I've chosen to send out, and a config change swaps even that for a local model. It's also meant to be fun: a friendly desktop robot for the kids, and a hobby project I can keep building on. ## AI transparency -**Yes — Dotty is built with AI assistance, and it says so out loud.** Coding agents help write the code, draft the docs, and triage issues — a deliberate choice, not a secret. The rule: **anything an AI agent authors is acknowledged as such** (model-named `Co-Authored-By:` trailers, AI-assisted PR notes, marked AI-drafted docs), with a human reviewing and accountable for everything that lands. +**Yes, Dotty is built with AI assistance, and I'm open about it.** Coding agents help me write the code, draft the docs and triage issues. That's a deliberate choice, not something I'm hiding. The rule: **anything an AI agent authors is acknowledged as such** (model-named `Co-Authored-By:` trailers, AI-assisted PR notes, marked AI-drafted docs), with a human reviewing and accountable for everything that lands. Full policy: [`AI_TRANSPARENCY.md`](./AI_TRANSPARENCY.md). ## Features -- **Kid Mode (on by default)** — age-appropriate responses via persona + per-turn prompt steering, backed by a thin blocked-words filter on TTS-bound output ([#157](https://github.com/BrettKinny/dotty-stackchan/issues/157)). It remains bypassable and is not a substitute for supervision. Toggle off for general-purpose use. See [`docs/kid-mode.md`](./docs/kid-mode.md). -- **Local ASR** — FunASR SenseVoiceSmall by default, no cloud transcription. WhisperLocal auto-selects on GPU hosts (better kid-speech accuracy); SenseVoiceOnnx is a lighter low-RAM option. -- **Local or cloud TTS** — Piper (offline) or EdgeTTS (cloud). Swap with a config change. -- **Streaming responses** — the bridge streams LLM output to the voice pipeline for lower perceived latency. -- **Emoji expressions** — every response starts with an emoji that the firmware maps to a face animation (smile, laugh, sad, surprise, thinking, angry, love, sleepy, neutral). -- **Voice tools** — the pi agent can search its memory, escalate hard questions to a bigger model, take a photo, and play songs, all mid-conversation. -- **States, toggles & LEDs** — a six-state mutex (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. Shipped on the active firmware fork (commit `d78118b`, 2026-04-27); the `firmware/firmware/` submodule pin in this repo lags, so flash from the active fork to get it. See "States, Toggles & LEDs" below and [`docs/modes.md`](./docs/modes.md). -- **Vision (camera)** — the robot's built-in camera can capture images for multimodal LLM queries. -- **Privacy LEDs** — hardware-bound mic (green) and camera (red) indicators on the LED ring. They light from the codec/camera enable signals via RAII guards, so a misbehaving server or model can't capture with the lights off. -- **Calendar context** — optional calendar integration feeds upcoming events into the conversation context. -- **Hackable** — every seam is swappable: LLM, TTS, ASR, agent framework. Fork it, rip out what you don't want, wire in your own. +- **Kid Mode (on by default):** age-appropriate replies through the persona and the Kid Mode rules in pi's system prompt, backed by a blocked-words filter on every LLM voice reply ([#157](https://github.com/BrettKinny/dotty-stackchan/issues/157)). If the filter itself fails, the reply is replaced rather than spoken. Speech the server pushes directly (the proactive greeter, admin `say`) does not pass through the filter. It can still be bypassed and is not a substitute for supervision. See [`docs/kid-mode.md`](./docs/kid-mode.md). +- **Local ASR:** FunASR SenseVoiceSmall by default, with no cloud transcription. WhisperLocal is picked automatically on GPU hosts and handles kids' voices better; SenseVoiceOnnx is a lighter option for low-RAM hosts. +- **Local or cloud TTS:** Piper (offline) or EdgeTTS (cloud), switched with a config change. +- **Streaming replies:** the voice provider streams the model's reply into TTS sentence by sentence, so Dotty starts talking before the whole answer is written. +- **Face expressions:** every reply starts with an emoji that the firmware turns into a face (smile, laugh, sad, surprise, thinking, angry, love, sleepy, neutral). The server makes sure each reply leads with exactly one face the firmware supports. +- **Voice tools:** mid-conversation, the pi agent can remember things and people and look them up later, pass a hard question to a bigger model, take a photo and describe it, and play a song. +- **States, toggles and LEDs:** six states (`idle / talk / story_time / security / sleep / dance`) plus two independent toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and shown on the 12-pixel LED ring. See [`docs/modes.md`](./docs/modes.md). +- **Privacy sleep:** in `sleep`, the firmware turns off face detection, voice processing and wake-word detection. Only a head touch or an explicit dashboard state change wakes it, not a face or a voice. This stops the normal camera and microphone paths; it is not a hardware power cut. +- **Vision:** the robot's camera can take a photo for a vision model to describe. The default is a cloud API (OpenRouter); to keep frames on your LAN, see [Keeping vision and audio on the LAN](./docs/brain.md#keeping-vision-and-audio-on-the-lan). +- **Calendar context:** optional calendar integration adds upcoming events to the conversation. +- **Hackable:** every seam can be swapped: LLM, TTS, ASR, agent framework. Fork it, rip out what you don't want, wire in your own. ## States, Toggles & LEDs -Behaviour is a **six-state mutex** (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager (shipped on the active fork in commit `d78118b`, 2026-04-27; bench checks tracked in [#38](https://github.com/BrettKinny/dotty-stackchan/issues/38)). Voice phrases, camera edges, and dashboard controls all flow through it. +Behaviour is **one of six states** (`idle / talk / story_time / security / sleep / dance`) plus two independent toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager. Voice phrases, camera events and dashboard controls all go through it. Bench checks are tracked in [#38](https://github.com/BrettKinny/dotty-stackchan/issues/38). -> Note: the `firmware/firmware/` submodule pin in this repo deliberately lags the active fork — flashing from the submodule won't give you Phase 4 yet. See the "Firmware iteration" section in [`CLAUDE.md`](./CLAUDE.md) and the submodule-pin caveat in [`docs/modes.md`](./docs/modes.md). +> Note: `firmware/` is a release pointer, not the active development checkout. The pinned build (`969c2b2`) includes the StateManager, but later fixes may only be on the active fork, so check your actual revision. See "Firmware iteration" in [`CLAUDE.md`](./CLAUDE.md) and the submodule note in [`docs/modes.md`](./docs/modes.md). -The 12-pixel LED ring shows the current state at a glance. **Left ring 0-5 is the state arc** — all six pixels paint the state colour, matching the dashboard's state buttons: +The 12-pixel LED ring shows the current state at a glance. **Left ring pixels 0-5 are the state arc.** All six show the state colour, matching the dashboard's state buttons: | | State | |---|---| -| ⚫ | `idle` — ambient awareness; ring off. Default. | -| 🟢 | `talk` — conversation engaged. | -| 🟠 | `story_time` — long-running interactive story. | -| ⚪ | `security` — watching the room (1 Hz white flash). | -| 🔵 | `sleep` — quiescent, mic open for "wake up". | -| 🟣 | `dance` — rainbow sweep + choreography. | +| ⚫ | `idle`: ambient awareness; ring off. Default. | +| 🟢 | `talk`: in a conversation. | +| 🟠 | `story_time`: a long-running interactive story. | +| ⚪ | `security`: watching the room (1 Hz white flash). | +| 🔵 | `sleep`: privacy sleep (see Features). Wake by head touch or a dashboard state change. | +| 🟣 | `dance`: rainbow sweep and choreography. | -On the right ring, **indices 8-9 are toggle pips** for kid_mode (salmon pink) and smart_mode (orange), and **index 11 (bottom) lights red while you have the turn** (LISTENING). The `idle → talk` transition fires on `face_detected` from the firmware; VLM identity recognition runs in parallel and feeds the LLM context. +On the right ring, **pixel 6 (top) is the face light**: yellow when a face is detected, green once it has been recognised. **Pixels 8 and 9 are the toggle lights** for kid_mode (salmon pink) and smart_mode (orange), and **pixel 11 (bottom) lights red while it's your turn to speak**. Pixels 7 and 10 are reserved and stay off. Seeing a face moves Dotty from `idle` to `talk`; recognising who it is runs alongside and feeds the conversation. -> Heads up: that right-ring layout is the active-fork StateManager. On the firmware **submodule pinned in this repo**, pixels 6 and 11 instead drive the **privacy LEDs** — 6 = mic (green), 11 = camera (red) — and the StateManager pips arrive once the submodule catches up to the active fork. +> Older firmware from before the StateManager (the historical `35f701a` pin) uses pixels 6 and 11 differently: 6 is a green microphone light and 11 a red camera light. The pinned and current firmware use the layout above. -Full state taxonomy, colour palette, transition diagram, and per-state backing architecture: [`docs/modes.md`](./docs/modes.md). +Full state list, colour palette, transition diagram and how each state is implemented: [`docs/modes.md`](./docs/modes.md). ## Web dashboard (locally hosted) -The dashboard service serves a web dashboard at `http://:8081/ui` — host status, mode toggles (Kid Mode / Smart Mode), state switcher, perception card (face / identity), emoji presets, and a live event log (turns, perception events, errors). Light and dark themes follow the system preference. It runs as a small FastAPI service (`bridge.py`) on your own hardware — no external service ever sees your data. +The dashboard is served at `http://:8081/ui`. It shows host status, the Kid Mode and Smart Mode toggles, a state switcher, a perception card (face and identity), emoji presets, and a live log of turns, perception events and errors. Light and dark themes follow the system setting. It runs as a small FastAPI service (`bridge.py`) on your own hardware, and no external service sees your data.

- Dotty dashboard — light theme + Dotty dashboard, light theme   - Dotty dashboard — dark theme + Dotty dashboard, dark theme

## Reference deployment -- **Hardware**: M5Stack StackChan (CoreS3 + servo kit), firmware built from `m5stack/StackChan`. -- **Brain**: a **pi** coding agent running in the `dotty-pi` container. It runs `qwen3.5:4b` on local [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) for the conversation loop and escalates hard questions to `qwen3.6:27b-think` (also on llama-swap) via its `think_hard` tool. xiaozhi-server's `PiVoiceLLM` provider hands each voice turn to the agent. See [`docs/brain.md`](./docs/brain.md). -- **Voice I/O**: xiaozhi-esp32-server on Docker (any Linux Docker host). +- **Hardware:** M5Stack StackChan (CoreS3 + servo kit), running firmware built from the [`BrettKinny/StackChan`](https://github.com/BrettKinny/StackChan) fork of `m5stack/StackChan`. +- **Brain:** a **pi** coding agent in the `dotty-pi` container. It runs `qwen3.5:4b` on local [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) for the conversation and passes hard questions to `qwen3.6:27b-think` (also on llama-swap) through its `think_hard` tool. xiaozhi-server's `PiVoiceLLM` provider hands each voice turn to the agent. See [`docs/brain.md`](./docs/brain.md). +- **Voice I/O:** xiaozhi-esp32-server in Docker (any Linux Docker host). ## What runs where | Component | Host | Notes | |---|---|---| -| StackChan (device) | ESP32-S3 on the desk | Firmware built from `m5stack/StackChan` (see `SETUP.md`) | -| xiaozhi-esp32-server | server (``) | Docker — voice I/O, ports 8000 + 8003 | -| dotty-pi | server (``) | Docker — the pi agent, Dotty's voice brain | -| dotty-behaviour | server (``) | Docker — FastAPI: perception bus, ambient consumers, vision, greeter; port 8090 | -| dashboard service | server (``) | Docker — FastAPI admin dashboard (`bridge.py`); port 8081 | -| Admin workstation | any LAN box | Development / `ssh` only | +| StackChan (device) | ESP32-S3 on the desk | Firmware from the `BrettKinny/StackChan` fork (see `SETUP.md`) | +| xiaozhi-esp32-server | server (``) | Docker: voice I/O, ports 8000 and 8003 | +| dotty-pi | server (``) | Docker: the pi agent, Dotty's brain | +| dotty-behaviour | server (``) | Docker, FastAPI: perception, ambient behaviour, vision, greeter; port 8090 | +| dashboard service | server (``) | Docker, FastAPI: the admin dashboard (`bridge.py`); port 8081 | +| Admin workstation | any LAN box | Development and `ssh` only | ## Get it running -The stack is the device plus four server-side pieces — xiaozhi-server (voice I/O), `dotty-pi` (the pi agent brain), `dotty-behaviour` (perception, ambient behaviour, and the proactive greeter), and the admin dashboard service. The four server pieces run as Docker containers on a single Docker host, alongside a local model backend ([llama-swap](./docs/cookbook/llama-swap-concurrent-models.md), or [Ollama](./docs/cookbook/run-fully-local.md) for the simpler single-binary option). +The stack is the robot plus four server-side pieces: xiaozhi-server (voice I/O), `dotty-pi` (the pi agent brain), `dotty-behaviour` (perception, ambient behaviour and the greeter) and the admin dashboard. All four run as Docker containers on one host, next to a local model backend ([llama-swap](./docs/cookbook/llama-swap-concurrent-models.md), or [Ollama](./docs/cookbook/run-fully-local.md) for the simpler single-binary option). Then: -- [`docs/quickstart.md`](./docs/quickstart.md) — 15-minute happy path: flash, configure, first turn. Includes placeholder substitution table, deployment layout, endpoints, reboot survival, and common ops snippets. -- [`docs/troubleshooting.md`](./docs/troubleshooting.md) — symptom-first lookup for common (and obscure) failure modes. +- [`docs/quickstart.md`](./docs/quickstart.md): the 15-minute happy path from flashing to a first conversation, with the placeholder table, deployment layout, endpoints, reboot survival and common ops snippets. +- [`docs/troubleshooting.md`](./docs/troubleshooting.md): look up a symptom to find common (and obscure) failures. ## Support -Questions, bugs, or "this didn't work for me" → [open a GitHub issue](https://github.com/BrettKinny/dotty-stackchan/issues). That's the best way to reach me. Occasional updates also go up on [r/stackchan](https://www.reddit.com/r/stackchan/). +Questions, bugs, or "this didn't work for me": [open a GitHub issue](https://github.com/BrettKinny/dotty-stackchan/issues). That's the best way to reach me. Occasional updates also go up on [r/stackchan](https://www.reddit.com/r/stackchan/). ## Deeper reference -For what the stack *is* underneath — hardware specs, protocol docs, model facts, and features we aren't using — see [`docs/`](./docs/README.md): +For what the stack *is* underneath (hardware specs, protocol docs, model facts, and features we aren't using), see [`docs/`](./docs/README.md): -- [docs/architecture.md](./docs/architecture.md) — end-to-end data flow, topology, deployment files, admin surface, perception bus, threat model. -- [docs/hardware.md](./docs/hardware.md) — M5Stack StackChan body + firmware lineage + on-device MCP tool catalog. -- [docs/voice-pipeline.md](./docs/voice-pipeline.md) — xiaozhi-esp32-server internals, FunASR/SenseVoice, VAD, TTS. -- [docs/brain.md](./docs/brain.md) — model matrix, the pi agent runtime, and how voice turns reach it. -- [docs/protocols.md](./docs/protocols.md) — Xiaozhi WS framing, MCP-over-WS, pi RPC, the dashboard HTTP API, emotion channel. -- [docs/modes.md](./docs/modes.md) — behavioural mode taxonomy + LED contract + transition diagram (with shipped-vs-planned breakdown). -- [docs/latent-capabilities.md](./docs/latent-capabilities.md) — features upstream supports that we aren't using yet. -- [docs/references.md](./docs/references.md) — canonical upstream URLs, model cards, licenses. +- [docs/architecture.md](./docs/architecture.md): end-to-end data flow, topology, deployment files, admin surface, perception bus, threat model. +- [docs/hardware.md](./docs/hardware.md): the M5Stack StackChan body, firmware lineage and on-device MCP tool catalogue. +- [docs/voice-pipeline.md](./docs/voice-pipeline.md): xiaozhi-esp32-server internals, FunASR/SenseVoice, VAD, TTS. +- [docs/brain.md](./docs/brain.md): the model matrix, the pi agent runtime, and how voice turns reach it. +- [docs/protocols.md](./docs/protocols.md): Xiaozhi WS framing, MCP over WS, pi RPC, the dashboard HTTP API, the emotion channel. +- [docs/modes.md](./docs/modes.md): states, the LED contract and the transition diagram, with what has shipped and what is planned. +- [docs/latent-capabilities.md](./docs/latent-capabilities.md): features upstream supports that we aren't using yet. +- [docs/references.md](./docs/references.md): upstream URLs, model cards, licences. ## References diff --git a/SETUP.md b/SETUP.md index fee1f2e6..14b445bd 100644 --- a/SETUP.md +++ b/SETUP.md @@ -48,34 +48,41 @@ There is **no SoftAP captive portal** in stock firmware (some older third-party xiaozhi builds had one; the shipped firmware does not). To run the StackChan **fully self-hosted** (no phone-app account, no vendor cloud, -your own xiaozhi-server as the endpoint), you need to **reflash the device -with firmware built from the open source tree**. - -The upstream firmware lives at **https://github.com/m5stack/StackChan**: -- `firmware/` — M5Stack's patches + ESP-IDF project wrapper -- `firmware/fetch_repos.py` — pulls `78/xiaozhi-esp32` as a dependency and - applies StackChan-specific patches, adding the board target - `CONFIG_BOARD_TYPE_M5STACK_STACK_CHAN` +your own xiaozhi-server as the endpoint), reflash it with Dotty's pinned +`BrettKinny/StackChan@dotty` firmware fork. The official `m5stack/StackChan` +tree is its upstream, but it does not contain Dotty's state, motion, LED, and +perception changes. + +The StackChan's on-device Settings app has no **Advanced Options** or OTA URL +editor. The similarly named **Advanced** tab documented by Xiaozhi belongs to +its browser-based SoftAP captive portal and appears only when that provisioning +mode is active. The server address is compiled into the firmware. Earlier +`fw-v*` GitHub releases carried binaries locked to the maintainer's LAN +server; those binaries have been removed and the releases are now +source-only. A portable prebuilt build is tracked in +[#190](https://github.com/BrettKinny/dotty-stackchan/issues/190). Build from +source with your own `CONFIG_OTA_URL` as shown below. + +The exact firmware source is the `firmware/` submodule of this repo. It pins +`BrettKinny/StackChan@dotty`, whose `firmware/fetch_repos.py` pulls +`78/xiaozhi-esp32` v2.2.4 and applies the StackChan integration patch. --- ## 2. Build and flash open firmware -> **Note — build flow documented from first-pass session findings; not yet -> end-to-end verified. Will be updated after a successful first flash.** - Requires **ESP-IDF v5.5.4**. Easiest path: the official `espressif/idf:v5.5.4` Docker image. ### 2a. Clone and configure ```bash -git clone https://github.com/m5stack/StackChan.git -cd StackChan/firmware +git clone --recursive https://github.com/BrettKinny/dotty-stackchan.git +cd dotty-stackchan/firmware/firmware ``` Point the firmware at your xiaozhi-server for OTA. Edit -`firmware/sdkconfig.defaults` and add (or modify) the line: +`sdkconfig.defaults` and add (or modify) the line: ``` CONFIG_OTA_URL="http://:8003/xiaozhi/ota/" @@ -83,10 +90,20 @@ CONFIG_OTA_URL="http://:8003/xiaozhi/ota/" Trailing slash matters — that's the path the server exposes. +`sdkconfig.defaults` seeds a newly generated `sdkconfig`; it does not overwrite +an existing one. On a repeat build, either make the same change through +`idf.py menuconfig` (**Xiaozhi Assistant → Default OTA URL**) or remove the +generated `sdkconfig` before rebuilding. Verify the effective value after +configuration with: + +```bash +grep '^CONFIG_OTA_URL=' sdkconfig +``` + ### 2b. Build inside the IDF container ```bash -docker run --rm -it -v "$PWD/..":/project -w /project/firmware \ +docker run --rm -it -v "$PWD":/project -w /project \ espressif/idf:v5.5.4 \ bash -c "python3 fetch_repos.py && idf.py build" ``` @@ -101,7 +118,7 @@ Linux (`/dev/cu.usbmodem*` on macOS — adapt the `--device` flag). ```bash docker run --rm -it --device=/dev/ttyACM0 \ - -v "$PWD/..":/project -w /project/firmware \ + -v "$PWD":/project -w /project \ espressif/idf:v5.5.4 \ idf.py -p /dev/ttyACM0 flash ``` @@ -111,14 +128,23 @@ flow), use that instead. ### 2d. First boot after flash -- No pairing-code screen -- No BLE provisioning step -- The device boots, loads WiFi credentials compiled into the firmware - (or, if you left WiFi unconfigured, whatever fallback the upstream - build offers — consult the xiaozhi-esp32 README for the current default - behaviour) -- It POSTs to `http://:8003/xiaozhi/ota/`, gets back the - WebSocket endpoint, and connects +Seeing **“Welcome! Let's get started.” is expected on a fresh flash**. It is +the StackChan app-configuration wizard, controlled by an NVS flag; it does not +mean the compiled `CONFIG_OTA_URL` was ignored. + +For a self-hosted setup: + +1. Tap **Skip** on the welcome screen. This bypasses the M5Stack account/BLE + wizard for the current boot. +2. If the launcher remains visible, open **SETUP**, then use its home control + to exit. Closing SETUP starts the Xiaozhi voice application. +3. With no saved SSID, the pinned build enters its default Xiaozhi hotspot + provisioner. Join the temporary `Xiaozhi-*` Wi-Fi network and open the URL + shown on the robot (normally `http://192.168.4.1`). +4. Select the robot's 2.4 GHz Wi-Fi network. The browser portal also has an + **Advanced** tab where you can confirm or override `ota_url`. +5. After Wi-Fi connects, the device POSTs to the compiled/persisted OTA URL, + receives the WebSocket endpoint, and connects. Tail the server logs while the device boots so you can watch the handshake happen (see step 4 below). @@ -127,18 +153,13 @@ handshake happen (see step 4 below). ## 3. WiFi credentials -Two options, depending on what the upstream xiaozhi-esp32 build exposes at -the version you pulled: +The pinned firmware does **not** define `CONFIG_WIFI_SSID` or +`CONFIG_WIFI_PASSWORD`; adding those names to `sdkconfig.defaults` has no +effect. Wi-Fi credentials live in NVS and are supplied through the Xiaozhi +hotspot portal described above (or retained from an earlier compatible +provisioning). -- **Compile-time WiFi credentials** — set `CONFIG_WIFI_SSID` and - `CONFIG_WIFI_PASSWORD` in `sdkconfig.defaults`. Simplest for a static - home setup; easy to forget they're in the binary. -- **Fallback SoftAP or BLE provisioning** — some upstream builds include a - fallback provisioning flow if no credentials are saved. Check the - upstream README for what your commit supports. - -Either way, the device must land on a **2.4 GHz** network. ESP32-S3 does -not do 5 GHz. +The device must land on a **2.4 GHz** network. ESP32-S3 does not do 5 GHz. --- @@ -227,13 +248,16 @@ Some older xiaozhi builds expose a SoftAP captive portal on first boot: 2. Join it from your phone; a captive portal typically auto-pops. If not, browse to `http://192.168.4.1`. 3. Pick your 2.4 GHz LAN SSID, enter the password. -4. Click "Advanced Options" and paste the OTA URL: +4. Open the portal's **Advanced** tab and paste the OTA URL: `http://:8003/xiaozhi/ota/` (trailing slash required). 5. Save. Device reboots and connects. If you have a build where this works, it's the fastest provisioning flow. It just isn't what M5Stack ships today. +This **Advanced** control is a tab in the browser portal at `192.168.4.1`, not +an item in the StackChan's on-device Settings app. + --- ## 9. When it's working: bookmark these diff --git a/bridge.py b/bridge.py index b4783d23..0d36c4af 100644 --- a/bridge.py +++ b/bridge.py @@ -15,12 +15,14 @@ """ import asyncio +import hmac import json import logging import os import re import sys import time +from collections import deque from contextlib import asynccontextmanager from pathlib import Path from typing import Any @@ -42,6 +44,7 @@ ) from bridge.csrf import CSRFMiddleware +from bridge.devices import candidate_devices, pick_live_device from bridge.text import ( # noqa: F401 (used by bridge.dashboard via re-imports) CONTENT_FILTER_REPLACEMENT, MAX_SENTENCES, @@ -89,24 +92,48 @@ def _safe_metric(fn, *args, **kwargs) -> None: ) +def _read_toggle_file(path: Path) -> bool | None: + """Parse a "true"/"false" toggle state file; None when it is missing, + unreadable or holds anything else.""" + try: + v = path.read_text().strip().lower() + except (OSError, UnicodeError): + return None + if v in ("true", "1", "yes"): + return True + if v in ("false", "0", "no"): + return False + return None + + +def _write_toggle_file(path: Path, enabled: bool) -> None: + """Atomically replace a toggle state file. xiaozhi-server, PiVoiceLLM + and dotty-behaviour re-read it on every turn from their own mounts, so + a reader must never catch it empty or half-written.""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(f".{path.name}.tmp") + tmp.write_text("true" if enabled else "false") + os.replace(tmp, path) + + +def _kid_mode_saved() -> bool | None: + """The persisted guardian policy, or None when nothing is saved yet.""" + return _read_toggle_file(_KID_STATE_FILE) + + def _read_kid_mode() -> bool: """State file overrides env var so the dashboard can flip kid-mode and - survive a restart without editing the unit. Format: "true" or "false".""" - if _KID_STATE_FILE.exists(): - try: - v = _KID_STATE_FILE.read_text().strip().lower() - if v in ("true", "1", "yes"): - return True - if v in ("false", "0", "no"): - return False - except OSError: - pass + survive a restart without editing the unit. Format: "true" or "false". + Same precedence as PiVoiceLLM's reader, so the dashboard shows what the + voice path enforces.""" + saved = _kid_mode_saved() + if saved is not None: + return saved return os.environ.get("DOTTY_KID_MODE", "true").lower() in ("1", "true", "yes") def _write_kid_mode(enabled: bool) -> None: - _KID_STATE_FILE.parent.mkdir(parents=True, exist_ok=True) - _KID_STATE_FILE.write_text("true" if enabled else "false") + _write_toggle_file(_KID_STATE_FILE, enabled) if _METRICS_AVAILABLE: _safe_metric(dotty_kid_mode_active.set, 1 if enabled else 0) @@ -131,21 +158,25 @@ def _apply_kid_mode(enabled: bool) -> None: def _read_smart_mode() -> bool: - if _SMART_STATE_FILE.exists(): - try: - v = _SMART_STATE_FILE.read_text().strip().lower() - if v in ("true", "1", "yes"): - return True - if v in ("false", "0", "no"): - return False - except OSError: - pass - return False + saved = _read_toggle_file(_SMART_STATE_FILE) + return bool(saved) def _write_smart_mode(enabled: bool) -> None: - _SMART_STATE_FILE.parent.mkdir(parents=True, exist_ok=True) - _SMART_STATE_FILE.write_text("true" if enabled else "false") + _write_toggle_file(_SMART_STATE_FILE, enabled) + + +def _dashboard_kid_mode_getter() -> bool: + """Kid Mode as the dashboard shows it: re-read from the persisted policy + on every poll, so a flip made elsewhere (POST /admin/kid-mode, a hand + edit, another bridge) shows within one refresh. Re-applies the + kid-derived globals and the Prometheus gauge when the file moved.""" + enabled = _read_kid_mode() + if enabled != KID_MODE: + _apply_kid_mode(enabled) + if _METRICS_AVAILABLE: + _safe_metric(dotty_kid_mode_active.set, 1 if enabled else 0) + return enabled # --------------------------------------------------------------------------- @@ -382,6 +413,44 @@ def _voice_memory_delete_blocking(mem_id: str) -> bool: log = logging.getLogger("dotty-bridge") +_STARTUP_RESYNC = os.environ.get("DOTTY_BRIDGE_STARTUP_RESYNC", "1") != "0" +_STARTUP_RESYNC_DELAY_SEC = float( + os.environ.get("DOTTY_BRIDGE_STARTUP_RESYNC_DELAY_SEC", "3") +) + + +async def _startup_resync() -> dict: + """Bring the robot back in line with persisted state after a restart (#21). + + The bridge keeps no state of its own any more: the State card reads + dotty-behaviour's per-device ``current_state`` live (persisted across + behaviour restarts by #173) and the toggles live in the shared state + files. What a restart can leave diverged is the firmware's kid / smart + LED pips, e.g. after a flip that landed while the robot held no WS. So + push both persisted toggles once. Best-effort: with no robot connected + xiaozhi answers 503 and receiveAudioHandle repaints the pips on the + next connection anyway. Returns what was pushed, for tests and logs.""" + kid_on = _read_kid_mode() + smart_on = _read_smart_mode() + result = {"kid_mode": kid_on, "smart_mode": smart_on, "pushed": False} + if not _XIAOZHI_HOST: + return result + kid_ok = await _dispatch_set_toggle("", "kid_mode", kid_on) + smart_ok = await _dispatch_set_toggle("", "smart_mode", smart_on) + result["pushed"] = kid_ok and smart_ok + if result["pushed"]: + log.info( + "startup resync: pushed kid_mode=%s smart_mode=%s to the robot", + kid_on, smart_on, + ) + else: + log.info( + "startup resync: robot not connected; pips sync on its next " + "connection (kid_mode=%s smart_mode=%s)", kid_on, smart_on, + ) + return result + + @asynccontextmanager async def lifespan(app: FastAPI): """Lifespan slimmed to dashboard-only. @@ -389,12 +458,25 @@ async def lifespan(app: FastAPI): The voice / perception / VLM machinery that used to spin up here moved to dotty-pi and dotty-behaviour in the #36 cutover; PR #109 then containerized this file, and #111 ripped the dormant code paths so the - container doesn't double-fire consumers. The startup voice-model - reconcile was dropped in the 2026-05-29 alignment pass along with - Tier1Slim — the live PiVoiceLLM path has no runtime model swap (v2 - scope), so there is no startup work left to do. + container doesn't double-fire consumers. The only startup work left is + the one-shot toggle resync (#21), run in the background so a slow or + absent xiaozhi never delays the dashboard coming up. """ - yield + task: asyncio.Task | None = None + if _STARTUP_RESYNC: + async def _delayed() -> None: + await asyncio.sleep(_STARTUP_RESYNC_DELAY_SEC) + try: + await _startup_resync() + except Exception: + log.exception("startup resync failed") + + task = asyncio.create_task(_delayed(), name="bridge-startup-resync") + try: + yield + finally: + if task is not None and not task.done(): + task.cancel() app = FastAPI(title="Dotty Bridge", lifespan=lifespan) @@ -575,20 +657,22 @@ def _dashboard_perception_recent_getter( def _dashboard_state_getter() -> str: - """Return the current State mirrored by dotty-behaviour. - - The deployment has one robot, so use the first valid per-device state. - If behaviour is unavailable or has not observed a state transition yet, - retain the dashboard's historical safe fallback of ``idle``. + """Return the robot's current State as mirrored by dotty-behaviour. + + dotty-behaviour tracks ``current_state`` per device from the firmware's + ``state_changed`` events. Its state also holds ids no robot owns (test + probes, the ``unknown`` fallback), so pick the most recently active + real device rather than whichever key comes first. If behaviour is + unavailable or has not observed a state transition yet, keep the + dashboard's historical safe fallback of ``idle``. """ states = _dashboard_perception_state_getter() - for device_state in states.values(): - if not isinstance(device_state, dict): - continue - current = device_state.get("current_state") - if isinstance(current, str) and current: - return current - return "idle" + with_state = { + did: s for did, s in candidate_devices(states).items() + if isinstance(s.get("current_state"), str) and s["current_state"] + } + did = pick_live_device(with_state) + return with_state[did]["current_state"] if did else "idle" def _dashboard_last_user_line_getter(device_id: str) -> dict | None: @@ -596,25 +680,19 @@ def _dashboard_last_user_line_getter(device_id: str) -> dict | None: return None -def _dashboard_sound_balance_series() -> list[float]: - """Live sound_event balance series from dotty-behaviour. +def _dashboard_open_perception_feed() -> Any: + """Open dotty-behaviour's perception SSE stream for the dashboard proxy. - Picks the first device with perception state (single-robot - deployment heuristic — matches the device-picking strategy used - for the perception card) and reads its sound-balance ring via - /api/perception/sound-balance/{device_id}. Same cache + timeout + - circuit-breaker contract as the other dotty-behaviour-backed - getters — returns ``[]`` on any failure.""" - state = _dashboard_perception_state_getter() - device_id = next(iter(state), None) if isinstance(state, dict) else None - if not device_id: - return [] - result = _dotty_behaviour_get( - f"/api/perception/sound-balance/{device_id}", - {"limit": 30}, - [], + Returns a streaming ``requests.Response`` (caller closes it). The read + timeout sits above the upstream's 15 s keepalive so an idle-but-healthy + stream never trips it. Raises on connect failure / non-2xx.""" + r = requests.get( + f"{DOTTY_BEHAVIOUR_URL}/api/perception/feed", + stream=True, + timeout=(_DOTTY_BEHAVIOUR_TIMEOUT_SEC, 30.0), ) - return result if isinstance(result, list) else [] + r.raise_for_status() + return r def _dashboard_vision_failures_last_hour() -> dict[str, int]: @@ -631,10 +709,43 @@ def _identity_display_name(identity: str) -> str | None: return None -# Stub SSE plumbing — kept so /ui/events still wakes the queue handler -# (sends heartbeats) when the browser subscribes. No producer is wired in -# bridge.py post-#111; convo turns are owned by dotty-pi now. +# Turn stream — /ui/events subscribers plus the recent-turns ring behind +# the Turns tab, Errors count and error report. Voice turns are owned by +# dotty-pi since #111, so the PiVoiceLLM provider reports each completed +# turn to /api/voice/turn (below). In-memory ONLY: the ring is lost on +# restart and transcripts are never written to disk. _dashboard_event_listeners: list[asyncio.Queue] = [] +_DASHBOARD_TURNS_MAX = 200 +_dashboard_recent_turns: "deque[dict]" = deque(maxlen=_DASHBOARD_TURNS_MAX) + + +_dashboard_last_turn_id = 0 + + +def _dashboard_publish_turn(record: dict) -> None: + """Record a completed turn and fan it out to /ui/events subscribers. + A slow subscriber's full queue drops the event rather than blocking. + + Each turn gets an ``id`` (milliseconds since the epoch, bumped to stay + strictly increasing) that /ui/events sends as the SSE event id, so a + reconnecting browser's Last-Event-ID replays only what it missed, even + across a bridge restart.""" + global _dashboard_last_turn_id + _dashboard_last_turn_id = max( + _dashboard_last_turn_id + 1, int(time.time() * 1000), + ) + record = {**record, "id": _dashboard_last_turn_id} + _dashboard_recent_turns.append(record) + for q in list(_dashboard_event_listeners): + try: + q.put_nowait(record) + except asyncio.QueueFull: + pass + + +def _dashboard_recent_turns_getter() -> list[dict]: + """Recent turns, oldest first.""" + return list(_dashboard_recent_turns) def _dashboard_subscribe_events() -> asyncio.Queue: @@ -658,13 +769,18 @@ async def _dashboard_set_kid_mode(enabled: bool) -> dict: custom-providers/textUtils.py and the persona prompts.""" _write_kid_mode(enabled) _apply_kid_mode(enabled) - ok = await _dispatch_set_toggle("", "kid_mode", enabled) - if not ok: - return { - "ok": False, - "error": "firmware did not acknowledge — LED + on-device toggle stale; bridge state is flipped", - } - return {"ok": True} + # The saved policy is what every voice turn and camera request + # reads, so the flip is in force once the file is written. The pip + # push only fails when the robot holds no WS (xiaozhi's WS is + # session-scoped); receiveAudioHandle repaints it on reconnect. + pushed = await _dispatch_set_toggle("", "kid_mode", enabled) + result: dict = {"ok": True, "device_pushed": pushed} + if not pushed: + result["note"] = ( + "saved and in force; the robot's LED pip updates when it " + "next connects" + ) + return result async def _dashboard_abort_device(*, device_id: str = "") -> dict: """Fire-and-forget POST to xiaozhi-server's admin abort route.""" @@ -726,13 +842,13 @@ async def _dashboard_set_smart_mode(enabled: bool) -> dict: Tier1Slim hot-swap path was removed in the 2026-05-29 alignment pass.""" _write_smart_mode(enabled) - dispatch_ok = await _dispatch_set_toggle("", "smart_mode", enabled) - if not dispatch_ok: - return { - "ok": False, - "error": "firmware did not acknowledge set_toggle (LED pip stale)", - } - return {"ok": True} + pushed = await _dispatch_set_toggle("", "smart_mode", enabled) + result: dict = {"ok": True, "device_pushed": pushed} + if not pushed: + result["note"] = ( + "saved; the robot's LED pip updates when it next connects" + ) + return result async def _dashboard_memory_records() -> list[dict]: """All per-person memory rows (approved + pending) for /ui/memory.""" @@ -754,7 +870,8 @@ async def _dashboard_memory_redact(mem_id: str) -> dict: vision_cache_getter=_dashboard_vision_cache_getter, audio_cache_getter=_dashboard_audio_cache_getter, scene_synthesis_cache_getter=_dashboard_scene_synthesis_cache_getter, - kid_mode_getter=_read_kid_mode, + kid_mode_getter=_dashboard_kid_mode_getter, + kid_mode_saved_getter=_kid_mode_saved, kid_mode_setter=_dashboard_set_kid_mode, smart_mode_getter=_read_smart_mode, smart_mode_setter=_dashboard_set_smart_mode, @@ -764,6 +881,7 @@ async def _dashboard_memory_redact(mem_id: str) -> dict: abort_device=_dashboard_abort_device, subscribe_events=_dashboard_subscribe_events, unsubscribe_events=_dashboard_unsubscribe_events, + recent_turns_getter=_dashboard_recent_turns_getter, perception_state_getter=_dashboard_perception_state_getter, perception_recent_getter=_dashboard_perception_recent_getter, memory_records_getter=_dashboard_memory_records, @@ -771,11 +889,74 @@ async def _dashboard_memory_redact(mem_id: str) -> dict: memory_redact=_dashboard_memory_redact, identity_display_name=_identity_display_name, last_user_line_getter=_dashboard_last_user_line_getter, - sound_balance_getter=_dashboard_sound_balance_series, + perception_feed_opener=_dashboard_open_perception_feed, vision_failures_getter=_dashboard_vision_failures_last_hour, ) +# --------------------------------------------------------------------------- +# /api/voice/* — turn + filter-hit reports from the voice provider +# --------------------------------------------------------------------------- +# PiVoiceLLM runs inside the xiaozhi container, so the bridge only learns +# about a voice turn if the provider tells it. Guarded by the same shared +# X-Admin-Token as /xiaozhi/admin/* when DOTTY_ADMIN_TOKEN is set; open on +# the LAN otherwise, matching the rest of the stack's unset-token posture. + +_VOICE_TEXT_MAX = 2000 + + +def _voice_require_admin_token(request: Request) -> None: + if not _ADMIN_TOKEN: + return + supplied = request.headers.get("X-Admin-Token", "") + if not hmac.compare_digest(supplied.encode(), _ADMIN_TOKEN.encode()): + raise HTTPException(status_code=401, detail="bad admin token") + + +class _VoiceTurnIn(BaseModel): + request_text: str = "" + response_text: str = "" + latency_ms: int | None = None + error: str | None = None + channel: str = "dotty" + + +class _VoiceFilterHitIn(BaseModel): + tier: str + rule: str = "" + prefix: str = "" + + +_voice_router = APIRouter( + prefix="/api/voice", dependencies=[Depends(_voice_require_admin_token)], +) + + +@_voice_router.post("/turn") +async def _voice_turn(payload: _VoiceTurnIn) -> dict: + _dashboard_publish_turn({ + "ts": time.time(), + "channel": payload.channel[:32] or "dotty", + "request_text": payload.request_text[:_VOICE_TEXT_MAX], + "response_text": payload.response_text[:_VOICE_TEXT_MAX], + "latency_ms": payload.latency_ms, + "error": (payload.error or "")[:500] or None, + }) + return {"ok": True} + + +@_voice_router.post("/filter-hit") +async def _voice_filter_hit(payload: _VoiceFilterHitIn) -> dict: + from bridge.text import record_content_filter_hit + record_content_filter_hit( + payload.tier[:16], payload.rule[:64], payload.prefix, + ) + return {"ok": True} + + +app.include_router(_voice_router) + + # --------------------------------------------------------------------------- # /admin/* — localhost-only runtime configuration mutations # --------------------------------------------------------------------------- diff --git a/bridge/dashboard.py b/bridge/dashboard.py index 6ff97548..07b95f67 100644 --- a/bridge/dashboard.py +++ b/bridge/dashboard.py @@ -16,6 +16,8 @@ import os import re import secrets +import socket +import threading import time import requests from datetime import datetime @@ -27,6 +29,13 @@ from fastapi.security import HTTPBasic, HTTPBasicCredentials from fastapi.templating import Jinja2Templates +from bridge.devices import ( + candidate_devices, + is_probe_device, + pick_live_device, + sensor_health, +) + log = logging.getLogger("dashboard") # Bridge wires its in-process message handler in via configure(). Lets the @@ -37,6 +46,7 @@ "audio_cache_getter": None, "scene_synthesis_cache_getter": None, "kid_mode_getter": None, + "kid_mode_saved_getter": None, "kid_mode_setter": None, "smart_mode_getter": None, "smart_mode_setter": None, @@ -46,6 +56,7 @@ "abort_device": None, "subscribe_events": None, "unsubscribe_events": None, + "recent_turns_getter": None, "perception_state_getter": None, "perception_recent_getter": None, "identity_display_name": None, @@ -53,7 +64,7 @@ "memory_records_getter": None, "memory_approve": None, "memory_redact": None, - "sound_balance_getter": None, + "perception_feed_opener": None, "vision_failures_getter": None, } @@ -62,11 +73,13 @@ def configure(*, send_message: Any = None, vision_cache_getter: Any = None, audio_cache_getter: Any = None, scene_synthesis_cache_getter: Any = None, kid_mode_getter: Any = None, kid_mode_setter: Any = None, + kid_mode_saved_getter: Any = None, smart_mode_getter: Any = None, smart_mode_setter: Any = None, state_getter: Any = None, state_setter: Any = None, inject_to_device: Any = None, abort_device: Any = None, subscribe_events: Any = None, unsubscribe_events: Any = None, + recent_turns_getter: Any = None, perception_state_getter: Any = None, perception_recent_getter: Any = None, identity_display_name: Any = None, @@ -74,7 +87,7 @@ def configure(*, send_message: Any = None, vision_cache_getter: Any = None, memory_records_getter: Any = None, memory_approve: Any = None, memory_redact: Any = None, - sound_balance_getter: Any = None, + perception_feed_opener: Any = None, vision_failures_getter: Any = None) -> None: """Register bridge state with the dashboard. Idempotent.""" if send_message is not None: @@ -89,6 +102,8 @@ def configure(*, send_message: Any = None, vision_cache_getter: Any = None, _state["kid_mode_getter"] = kid_mode_getter if kid_mode_setter is not None: _state["kid_mode_setter"] = kid_mode_setter + if kid_mode_saved_getter is not None: + _state["kid_mode_saved_getter"] = kid_mode_saved_getter if smart_mode_getter is not None: _state["smart_mode_getter"] = smart_mode_getter if smart_mode_setter is not None: @@ -105,6 +120,8 @@ def configure(*, send_message: Any = None, vision_cache_getter: Any = None, _state["subscribe_events"] = subscribe_events if unsubscribe_events is not None: _state["unsubscribe_events"] = unsubscribe_events + if recent_turns_getter is not None: + _state["recent_turns_getter"] = recent_turns_getter if perception_state_getter is not None: _state["perception_state_getter"] = perception_state_getter if perception_recent_getter is not None: @@ -119,8 +136,8 @@ def configure(*, send_message: Any = None, vision_cache_getter: Any = None, _state["memory_approve"] = memory_approve if memory_redact is not None: _state["memory_redact"] = memory_redact - if sound_balance_getter is not None: - _state["sound_balance_getter"] = sound_balance_getter + if perception_feed_opener is not None: + _state["perception_feed_opener"] = perception_feed_opener if vision_failures_getter is not None: _state["vision_failures_getter"] = vision_failures_getter @@ -298,7 +315,6 @@ def _xiaozhi_admin_headers() -> dict[str, str]: is set (matches the xiaozhi-server middleware); empty otherwise.""" return {"X-Admin-Token": _ADMIN_TOKEN} if _ADMIN_TOKEN else {} LOG_DIR = Path(os.environ.get("CONVO_LOG_DIR", "/var/lib/dotty-bridge/logs")) -VOICE_CHANNELS = ("dotty", "stackchan") _START_TIME = time.time() @@ -340,12 +356,27 @@ def _humanize_age(seconds: float) -> str: return f"{s // 86400}d" -def _today_log_path() -> Path: - return _log_path_for(datetime.now().strftime("%Y-%m-%d")) +def _recent_turns() -> list[dict]: + """Recent voice turns from the bridge's in-memory ring, oldest first. + Empty after a bridge restart — nothing is persisted.""" + getter = _state.get("recent_turns_getter") + try: + return list(getter()) if getter else [] + except Exception: + return [] -def _log_path_for(date_str: str) -> Path: - return LOG_DIR / f"convo-{date_str}.ndjson" +def _errored_turns_today() -> list[dict]: + """Today's (local date) errored turns, newest first.""" + today = datetime.now().date() + out: list[dict] = [] + for rec in reversed(_recent_turns()): + ts = rec.get("ts") + if not rec.get("error") or not isinstance(ts, (int, float)): + continue + if datetime.fromtimestamp(ts).date() == today: + out.append(rec) + return out def _clean_request_text(s: str) -> str: @@ -372,37 +403,36 @@ def _clean_request_text(s: str) -> str: return after -def _parse_ts(ts: str) -> float | None: - if not ts: - return None +def _stackchan_last_seen() -> float | None: + """Timestamp of the most recent voice activity. + + Prefers the newest reported turn; falls back to dotty-behaviour's + per-device `last_chat_t`, which also covers the window after a bridge + restart when the turn ring is empty.""" + return _last_turn_ts() or _perception_last_chat_ts() + + +def _perception_last_chat_ts() -> float | None: + getter = _state.get("perception_state_getter") try: - return datetime.fromisoformat(ts.replace("Z", "+00:00")).timestamp() + pstate = (getter() if getter else None) or {} except Exception: return None + # Real devices only, like every other device read: a probe or the + # "unknown" id must not mark the robot as recently active. + stamps = [ + d["last_chat_t"] for d in candidate_devices(pstate).values() + if isinstance(d, dict) and isinstance(d.get("last_chat_t"), (int, float)) + ] + return max(stamps) if stamps else None -def _stackchan_last_seen() -> float | None: - """Timestamp of the most recent voice-channel turn in today's log.""" - path = _today_log_path() - if not path.exists(): - return None - try: - data = path.read_bytes() - except OSError: - return None - last_voice_ts: float | None = None - for line in data.splitlines(): - if not line.strip(): - continue - try: - rec = json.loads(line) - except Exception: - continue - if rec.get("channel") in VOICE_CHANNELS: - ts = _parse_ts(rec.get("ts", "")) - if ts is not None: - last_voice_ts = ts - return last_voice_ts +def _last_turn_ts() -> float | None: + for rec in reversed(_recent_turns()): + ts = rec.get("ts") + if isinstance(ts, (int, float)): + return float(ts) + return None @router.get("", response_class=HTMLResponse, include_in_schema=False) @@ -435,27 +465,48 @@ async def dashboard(request: Request) -> Any: # never touches the files itself — it asks xiaozhi to list them via the admin # endpoint, and asks xiaozhi to play one via /xiaozhi/admin/play-asset. _SONGS_BASE_PATH = "/opt/xiaozhi-esp32-server/config/assets/songs" -_SONG_OK_EXT = {".opus", ".ogg", ".wav", ".mp3"} +_SONG_OK_EXT = {".opus", ".ogg", ".wav", ".mp3", ".mid", ".midi"} # = xiaozhi play-asset -async def _xiaozhi_device_count() -> int | None: - """Count active StackChan WS connections via the admin endpoint. - Returns None if xiaozhi is unreachable.""" +_connected_cache: tuple[float, list[str] | None] | None = None + + +async def _xiaozhi_connected_devices() -> list[str] | None: + """Device ids holding a live StackChan WS, via the admin endpoint. + Returns None if xiaozhi is unreachable (unknown, not "none"). Cached for + _PROBE_TTL: the status strip re-renders on every perception event in + every open tab, and each render used to make this request.""" + global _connected_cache if not XIAOZHI_HOST: return None + now = time.monotonic() + if _connected_cache and now - _connected_cache[0] < _PROBE_TTL: + return _connected_cache[1] url = f"http://{XIAOZHI_HOST}:{XIAOZHI_OTA_PORT}/xiaozhi/admin/devices" import urllib.request - def _fetch() -> int | None: + def _fetch() -> list[str] | None: try: req = urllib.request.Request(url, headers=_xiaozhi_admin_headers()) with urllib.request.urlopen(req, timeout=2) as r: if r.status != 200: return None data = json.loads(r.read()) - return len(data.get("devices", [])) + devices = data.get("devices", []) + if not isinstance(devices, list): + return None + return [str(d) for d in devices] except Exception: return None - return await asyncio.to_thread(_fetch) + result = await asyncio.to_thread(_fetch) + _connected_cache = (time.monotonic(), result) + return result + + +async def _xiaozhi_device_count() -> int | None: + """Count active StackChan WS connections via the admin endpoint. + Returns None if xiaozhi is unreachable.""" + devices = await _xiaozhi_connected_devices() + return None if devices is None else len(devices) @router.get("/device-status", response_class=HTMLResponse, include_in_schema=False) @@ -479,7 +530,7 @@ async def device_status(request: Request) -> Any: @router.get("/alerts/count", response_class=HTMLResponse, include_in_schema=False) async def alerts_count(request: Request, chip: int = 0) -> Any: - """Q6: count today's errored turns from the convo log. + """Q6: count today's errored turns from the recent-turns ring. Two render modes share the same count so the dashboard polls one URL: - default: legacy floating ``alerts_badge.html`` (kept for @@ -490,22 +541,7 @@ async def alerts_count(request: Request, chip: int = 0) -> Any: the chip. innerHTML swap target is the chip itself, so the script runs against its own parent element. """ - today = datetime.now().strftime("%Y-%m-%d") - path = _log_path_for(today) - n = 0 - if path.exists(): - try: - for line in path.read_bytes().splitlines(): - if not line.strip(): - continue - try: - rec = json.loads(line) - except Exception: - continue - if rec.get("error"): - n += 1 - except OSError: - pass + n = len(_errored_turns_today()) if chip: # Don't tint the chip red — selection (`btn-primary` via setFeedFilter) # is the only "selected" cue; the chip just shows a warning glyph + @@ -526,37 +562,15 @@ async def alerts_count(request: Request, chip: int = 0) -> Any: async def alerts_detail(request: Request) -> Any: """F13: render today's errored turns. Opened via the alerts-badge modal in dashboard.html.""" - today = datetime.now().strftime("%Y-%m-%d") - path = _log_path_for(today) entries: list[dict[str, Any]] = [] - if path.exists(): - try: - lines = path.read_bytes().splitlines() - except OSError: - lines = [] - for line in reversed(lines): - if not line.strip(): - continue - try: - rec = json.loads(line) - except Exception: - continue - if not rec.get("error"): - continue - ts = rec.get("ts", "") - try: - time_str = datetime.fromisoformat( - ts.replace("Z", "+00:00") - ).astimezone().strftime("%H:%M:%S") - except Exception: - time_str = ts[-8:] if ts else "?" - entries.append({ - "time": time_str, - "channel": rec.get("channel") or "?", - "request": _clean_request_text(rec.get("request_text") or "")[:400], - "response": (rec.get("response_text") or "")[:300], - "error": str(rec.get("error"))[:500], - }) + for rec in _errored_turns_today(): + entries.append({ + "time": datetime.fromtimestamp(rec["ts"]).astimezone().strftime("%H:%M:%S"), + "channel": rec.get("channel") or "?", + "request": _clean_request_text(rec.get("request_text") or "")[:400], + "response": (rec.get("response_text") or "")[:300], + "error": str(rec.get("error"))[:500], + }) return templates.TemplateResponse( request, "alerts_detail.html", {"entries": entries}, @@ -663,17 +677,13 @@ def _post() -> dict: return templates.TemplateResponse(request, "say_result.html", result) -_INJECT_WAIT_SEC = 8.0 # Q4: how long to wait for Dotty's reply before - # showing "no response in time" fallback. - - async def _inject_or_error(request: Request, text: str, label: str) -> Any: """Helper for action endpoints that fire text into xiaozhi-server's pipeline so the device actually speaks/emotes/runs MCP tools. - Q4: subscribes to the bridge's event stream BEFORE injecting, then - waits up to ~8s for the next turn so the dashboard can show what Dotty - actually said (not just "Sent…").""" + Returns as soon as the inject is accepted. It used to wait ~8 s for the + reply on the bridge's turn stream, but turns are owned by dotty-pi since + #111 and nothing publishes them here — the wait only ever timed out.""" inject = _state.get("inject_to_device") if inject is None: return templates.TemplateResponse( @@ -681,38 +691,22 @@ async def _inject_or_error(request: Request, text: str, label: str) -> Any: {"ok": False, "error": "Inject path not configured (xiaozhi admin patch missing)."}, ) - subscribe = _state.get("subscribe_events") - unsubscribe = _state.get("unsubscribe_events") - queue = subscribe() if subscribe else None try: - try: - result = await inject(text=text) - except Exception as exc: - log.exception("dashboard inject failed") - return templates.TemplateResponse( - request, "say_result.html", - {"ok": False, "error": f"Bridge error: {exc.__class__.__name__}"}, - ) - if not result.get("ok"): - return templates.TemplateResponse( - request, "say_result.html", - {"ok": False, "error": result.get("error", "unknown injection failure")}, - ) - # Wait for the next completed turn (likely ours — single device). - response_text = "Sent — no reply in 8s." - if queue is not None: - try: - event = await asyncio.wait_for(queue.get(), timeout=_INJECT_WAIT_SEC) - response_text = event.get("response_text") or "(no text)" - except asyncio.TimeoutError: - pass + result = await inject(text=text) + except Exception as exc: + log.exception("dashboard inject failed") return templates.TemplateResponse( request, "say_result.html", - {"ok": True, "sent": label, "response": response_text}, + {"ok": False, "error": f"Bridge error: {exc.__class__.__name__}"}, ) - finally: - if queue is not None and unsubscribe is not None: - unsubscribe(queue) + if not result.get("ok"): + return templates.TemplateResponse( + request, "say_result.html", + {"ok": False, "error": result.get("error", "unknown injection failure")}, + ) + return templates.TemplateResponse( + request, "say_result.html", {"ok": True, "sent": label}, + ) def _kid_blocked(text: str) -> bool: @@ -919,11 +913,25 @@ def _host_detail_llm_label(smart_on: bool | None = None) -> str: @router.get("/kid-mode", response_class=HTMLResponse, include_in_schema=False) async def kid_mode_partial(request: Request) -> Any: + """Kid Mode card. Re-read from the persisted guardian policy on every + poll (the same file PiVoiceLLM, the camera gate and dotty-behaviour + read each turn), so it always shows what is being enforced. ``saved`` + is False when no policy file exists yet: the voice path then falls back + to its startup default while the camera gate stays denied.""" getter = _state.get("kid_mode_getter") - enabled = bool(getter()) if getter else True + try: + enabled = bool(getter()) if getter else True + except Exception: + log.exception("kid_mode getter failed") + enabled = True + saved_getter = _state.get("kid_mode_saved_getter") + try: + saved = (saved_getter() is not None) if saved_getter else True + except Exception: + saved = False return templates.TemplateResponse( request, "kid_mode.html", - {"enabled": enabled, "available": getter is not None}, + {"enabled": enabled, "available": getter is not None, "saved": saved}, ) @@ -954,29 +962,6 @@ async def kid_mode_partial(request: Request) -> Any: } -def _sound_balance_sparkline() -> dict | None: - """Build the State-tile sound-balance sparkline (#69) — an SVG - polyline over the last 60 s of localizer `balance` samples. Returns - None when there's no recent sound data (fewer than 2 points).""" - getter = _state.get("sound_balance_getter") - series = getter() if getter else [] - if not series or len(series) < 2: - return None - width, height = 100.0, 24.0 - n = len(series) - pts: list[str] = [] - for i, bal in enumerate(series): - x = (i / (n - 1)) * width - y = (1.0 - max(0.0, min(1.0, float(bal)))) * height - pts.append(f"{x:.1f},{y:.1f}") - return { - "points": " ".join(pts), - "last": series[-1], - "width": width, - "height": height, - } - - def _vision_failures_count() -> int: """Total vision-capture failures in the last hour (#74) — summed across error kinds. 0 when the getter is unwired or the window is @@ -1002,21 +987,25 @@ async def state_partial(request: Request) -> Any: "states": _STATES, "labels": _STATE_LABELS, "descriptions": _STATE_DESCRIPTIONS, - "sound_spark": _sound_balance_sparkline(), }, ) # #65 tier 1 — voice tool inventory. Hardcoded list of the tools shipped # by dotty-pi-ext (see dotty-pi-ext/package.json). Per the issue body the -# static list is intentional: "hardcoded; same five values until a sixth -# tool ships". Tier 2 (call counts via PiClient stdout parsing) and Tier 3 -# (safety-denial counts) are deferred follow-ups. +# static list is intentional — keep it in step with dotty-pi-ext/src/tools/ +# (seven tools since #53 added recall_person / remember_person). Tier 2 +# (call counts via PiClient stdout parsing) and Tier 3 (safety-denial +# counts) are deferred follow-ups. _VOICE_TOOLS: list[dict[str, str]] = [ {"name": "memory_lookup", "description": "Search Dotty's long-term memory by keyword"}, {"name": "remember", "description": "Save a new memory (\"Brett's birthday is …\")"}, + {"name": "recall_person", + "description": "Look up what Dotty knows about a named person"}, + {"name": "remember_person", + "description": "Save a fact about a named person (kid-mode: pending review)"}, {"name": "think_hard", "description": "Hand a hard reasoning task to the bigger 27B model"}, {"name": "take_photo", @@ -1176,11 +1165,13 @@ def _pick_perception_device_id() -> str | None: to feed into the Perception card builder. Priority: 1. The device with a fresh _vision_cache entry (most likely the one actively perceiving). - 2. Any device in _perception_state (single robot ➜ single key). + 2. The most recently active real device in dotty-behaviour's + per-device state (test probes / the "unknown" id are skipped). Returns None when the bridge has not yet seen any device — the card then renders empty states across the board. """ - vc = _vision_cache_snapshot() + vc = {did: e for did, e in _vision_cache_snapshot().items() + if not is_probe_device(did)} if vc: try: return max(vc.items(), key=lambda kv: kv[1].get("wall_ts", 0.0))[0] @@ -1192,8 +1183,7 @@ def _pick_perception_device_id() -> str | None: states = psg() or {} except Exception: states = {} - for did in states.keys(): - return did + return pick_live_device(states) return None @@ -1437,9 +1427,10 @@ async def smart_mode_set(request: Request, enabled: str = Form("")) -> Any: request, "smart_mode_result.html", {"ok": False, "error": err or "Smart mode toggle failed."}, ) + note = result.get("note") if isinstance(result, dict) else None return templates.TemplateResponse( request, "smart_mode_result.html", - {"ok": True, "new_state": new_state}, + {"ok": True, "new_state": new_state, "note": note}, ) @@ -1472,9 +1463,10 @@ async def kid_mode_set(request: Request, enabled: str = Form("")) -> Any: request, "kid_mode_result.html", {"ok": False, "error": err or "Kid mode toggle failed."}, ) + note = result.get("note") if isinstance(result, dict) else None return templates.TemplateResponse( request, "kid_mode_result.html", - {"ok": True, "new_state": new_state}, + {"ok": True, "new_state": new_state, "note": note}, ) @@ -1697,25 +1689,24 @@ async def status_strip(request: Request, placement: str = "header") -> Any: # (face/sound events stopped flowing past PERCEPTION_STALE_THRESHOLD_S # in bridge.py). This is the only path that can degrade the robot # dot below "ok" in the redesigned binary model. + # + # Only real devices that have streamed since dotty-behaviour started + # and (when xiaozhi answers) hold a live WS are judged — test + # probes, the "unknown" fallback id, devices restored from disk and + # a robot that is simply offline/asleep must not raise the warning. psg = _state.get("perception_state_getter") if psg is not None: try: pstate = psg() or {} except Exception: pstate = {} - stale_devs = [ - did for did, s in pstate.items() - if s and s.get("sensor_stale") - ] - fresh_devs = [ - did for did, s in pstate.items() - if s and not s.get("sensor_stale") - ] + connected = await _xiaozhi_connected_devices() + judged, stale_devs, fresh_devs = sensor_health(pstate, connected) if stale_devs and sc_status in ("ok", "unknown"): sc_status = "warn" sc_tip = ( f"Dotty: perception sensors stale " - f"({len(stale_devs)}/{len(pstate)} dev) — " + f"({len(stale_devs)}/{len(judged)} dev) — " f"firmware bus may be hung" ) elif fresh_devs and sc_status == "unknown": @@ -1815,7 +1806,7 @@ async def host_detail(request: Request, slug: str) -> Any: smart_on = bool(smart_getter()) if smart_getter else None active_llm = _host_detail_llm_label(smart_on) facts = [ - ("Device", "Raspberry Pi"), + ("Device", "Docker container (dotty-bridge)"), ("Status", "online"), ("Version", BRIDGE_VERSION), ("Uptime", _humanize_age(time.time() - _START_TIME)), @@ -1885,7 +1876,8 @@ async def host_detail(request: Request, slug: str) -> Any: seen = f"{_humanize_age(time.time() - last_seen_ts)} ago" getter = _state.get("state_getter") current = (getter() if getter else None) or "idle" - n = await _xiaozhi_device_count() + connected = await _xiaozhi_connected_devices() + n = None if connected is None else len(connected) # Perception bus liveness — separate from voice activity. Stale # means firmware-side face/sound events have stopped flowing # (face_detected, sound_event, state_changed) past the bridge's @@ -1899,31 +1891,33 @@ async def host_detail(request: Request, slug: str) -> Any: pstate = psg() or {} except Exception: pstate = {} - if pstate: - stale = [ - did for did, s in pstate.items() - if s and s.get("sensor_stale") - ] + judged, stale, fresh = sensor_health(pstate, connected) + if not judged and candidate_devices(pstate): + perception_label = ( + "not streaming (robot offline or idle)" + if connected is not None else "no events since restart" + ) + elif judged: if stale: - ages = [ - s.get("sensor_age_s") for _, s in pstate.items() - if s and s.get("sensor_stale") + ages = [pstate[did].get("sensor_age_s") for did in stale] + finite_ages = [ + a for a in ages + if isinstance(a, (int, float)) and a != float("inf") ] - finite_ages = [a for a in ages if a not in (None, float("inf"))] if finite_ages: oldest = max(finite_ages) perception_label = ( - f"stale ({len(stale)}/{len(pstate)} dev, " + f"stale ({len(stale)}/{len(judged)} dev, " f"oldest {_humanize_age(oldest)})" ) else: perception_label = ( - f"stale ({len(stale)}/{len(pstate)} dev, never)" + f"stale ({len(stale)}/{len(judged)} dev, never)" ) else: youngest = min( - (s.get("sensor_age_s", float("inf")) - for s in pstate.values() if s), + (pstate[did].get("sensor_age_s", float("inf")) + for did in fresh), default=float("inf"), ) if youngest != float("inf"): @@ -2099,40 +2093,162 @@ async def security_recent(request: Request, device_id: str) -> Any: return templates.TemplateResponse(request, "security_panel.html", ctx) +# --- Perception feed proxy ------------------------------------------------ + +def _abort_upstream(upstream: Any) -> None: + """Shut down a streaming ``requests`` response's socket (waking any + thread blocked reading it), then close the response. Best-effort.""" + for path in (("raw", "_connection", "sock"), + ("raw", "_fp", "fp", "raw", "_sock")): + obj = upstream + for attr in path: + obj = getattr(obj, attr, None) + if obj is None: + break + if isinstance(obj, socket.socket): + try: + obj.shutdown(socket.SHUT_RDWR) + except OSError: + pass + break + try: + upstream.close() + except Exception: + pass + + +@router.get("/perception/feed", include_in_schema=False) +async def perception_feed_proxy(request: Request) -> StreamingResponse: + """Relay dotty-behaviour's perception SSE stream to the browser. + + The perception bus moved to dotty-behaviour in #36, but the page can + only open same-origin EventSources — so the bridge relays the stream + line-for-line. If the upstream is down or drops, the stream just ends + and EventSource reconnects after the `retry` interval (a non-200 would + stop it retrying for good).""" + opener = _state.get("perception_feed_opener") + + async def gen(): + yield b"retry: 5000\n\n" + if opener is None: + return + try: + upstream = await asyncio.to_thread(opener) + except Exception as exc: + log.warning("perception feed upstream unavailable: %s", exc) + return + lines = upstream.iter_lines(chunk_size=1) + reader_idle = True # False while a worker thread is inside next() + try: + while True: + if await request.is_disconnected(): + break + reader_idle = False + try: + line = await asyncio.to_thread(next, lines, None) + except Exception as exc: + log.warning("perception feed upstream dropped: %s", exc) + break + reader_idle = True + if line is None: + break + yield line + b"\n" + finally: + if reader_idle: + upstream.close() + else: + # The browser left while a worker thread is blocked reading + # the upstream (dotty-behaviour only sends a keepalive every + # 15 s). Closing here would wait on that read's buffer lock + # and freeze the event loop — every other dashboard request + # with it — until the next upstream line. Unblock the read + # with a socket shutdown and close off the loop instead. + threading.Thread( + target=_abort_upstream, args=(upstream,), + name="perception-feed-close", daemon=True, + ).start() + + return StreamingResponse( + gen(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) + + # --- P13 + P12: SSE event stream for live log + error toasts ------------- +# Turns replayed to a freshly opened page so the Turns tab is not blank +# until the next voice turn. +_EVENTS_REPLAY_MAX = 20 + @router.get("/events", include_in_schema=False) async def events_stream(request: Request) -> StreamingResponse: """Server-Sent Events stream of completed conversation turns. - Each event is one JSON object: {ts, channel, request_text, response_text, - latency_ms, error, emoji_used}. The bridge's ConvoLogger broadcasts on - every turn. Heartbeats every 15s keep proxies / browsers awake. + Each event is one JSON object: {id, ts, channel, request_text, + response_text, latency_ms, error}. Published when the voice provider + reports a turn to /api/voice/turn. Heartbeats every 15s keep proxies / + browsers awake. + + On connect the stream first replays recent turns from the in-memory + ring, flagged ``"replay": true`` so the page shows them without raising + the error toast: the last _EVENTS_REPLAY_MAX turns for a fresh page, or + only the turns newer than the browser's Last-Event-ID on a reconnect. """ subscribe = _state.get("subscribe_events") unsubscribe = _state.get("unsubscribe_events") if subscribe is None or unsubscribe is None: raise HTTPException(503, "event broadcast not configured") + try: + last_seen_id: int | None = int(request.headers.get("last-event-id", "")) + except ValueError: + last_seen_id = None + # Subscribe before reading the ring so a turn landing in between is + # caught by the queue; the id check below drops the overlap. queue = subscribe() + backlog = [t for t in _recent_turns() if isinstance(t.get("id"), int)] + if last_seen_id is None: + backlog = backlog[-_EVENTS_REPLAY_MAX:] + else: + backlog = [t for t in backlog if t["id"] > last_seen_id] + + def _frame(event: dict, *, replay: bool) -> bytes: + # Strip the heavy [Context] block before pushing — clients only + # want the cleaned user payload. + event = {**event, + "request_text": _clean_request_text( + event.get("request_text") or "")} + if replay: + event["replay"] = True + payload = json.dumps(event, ensure_ascii=False) + event_id = event.get("id") + prefix = f"id: {event_id}\n" if isinstance(event_id, int) else "" + return f"{prefix}data: {payload}\n\n".encode() async def gen(): + sent_id = max((t["id"] for t in backlog), default=0) try: # Tell EventSource how long to wait before reconnecting on drop. yield "retry: 5000\n\n".encode() + for turn in backlog: + yield _frame(turn, replay=True) while True: if await request.is_disconnected(): break try: event = await asyncio.wait_for(queue.get(), timeout=15.0) - # Strip the heavy [Context] block before pushing — clients - # only want the cleaned user payload. - event = {**event, - "request_text": _clean_request_text( - event.get("request_text") or "")} - payload = json.dumps(event, ensure_ascii=False) - yield f"data: {payload}\n\n".encode() except asyncio.TimeoutError: yield b": heartbeat\n\n" + continue + event_id = event.get("id") + if isinstance(event_id, int): + if event_id <= sent_id: + continue + sent_id = event_id + yield _frame(event, replay=False) finally: unsubscribe(queue) diff --git a/bridge/devices.py b/bridge/devices.py new file mode 100644 index 00000000..807434b4 --- /dev/null +++ b/bridge/devices.py @@ -0,0 +1,118 @@ +"""Pick the real robot out of dotty-behaviour's per-device perception state. + +dotty-behaviour keys its state by device id and keeps every id it has ever +seen (and persists their mutex state across restarts, #173). That includes +ids no robot owns: acceptance probes such as ``automation-persistence-probe``, +the ``unknown`` fallback id, and devices restored from disk that have sent +nothing since. The dashboard must not let any of those decide the State +card, the robot status dot, or the "sensors stale" warning. + +Pure functions over the ``/api/perception/state`` payload so they are easy +to test; no I/O here. +""" + +from __future__ import annotations + +import math +import os +import re +from typing import Any, Iterable + +# Same default as dotty-behaviour's PERCEPTION_EPHEMERAL_DEVICE_RE. Mirrored +# here so the dashboard ignores probes even against an older dotty-behaviour +# that does not expire them yet. +_PROBE_RE_SRC = os.environ.get( + "PERCEPTION_EPHEMERAL_DEVICE_RE", + r"(automation|test|probe|smoke|ci|uat)[-_].*|unknown", +) +try: + _PROBE_RE: re.Pattern[str] | None = re.compile(_PROBE_RE_SRC) +except re.error: + _PROBE_RE = None + + +def is_probe_device(device_id: str) -> bool: + """True for test / automation / fallback ids that are not a robot.""" + if not device_id: + return True + return bool(_PROBE_RE and _PROBE_RE.fullmatch(device_id)) + + +def _num(value: Any) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + value = float(value) + return value if math.isfinite(value) else None + + +def activity_t(device_state: dict) -> float: + """Newest wall-clock timestamp the device has produced (0 when none).""" + stamps = ( + _num(device_state.get(key)) + for key in ("last_event_t", "last_state_change_t", "last_chat_t") + ) + return max((t for t in stamps if t is not None), default=0.0) + + +def sensor_seen(device_state: dict) -> bool: + """Has this device sent a sensor event since dotty-behaviour started? + + Uses dotty-behaviour's ``sensor_seen`` when present, else infers it from + a finite ``sensor_age_s`` / a ``last_event_t``.""" + seen = device_state.get("sensor_seen") + if isinstance(seen, bool): + return seen + if _num(device_state.get("sensor_age_s")) is not None: + return True + return _num(device_state.get("last_event_t")) is not None + + +def candidate_devices(pstate: dict) -> dict[str, dict]: + """Per-device entries that could be the real robot.""" + if not isinstance(pstate, dict): + return {} + return { + did: s for did, s in pstate.items() + if isinstance(did, str) and isinstance(s, dict) and not is_probe_device(did) + } + + +def pick_live_device( + pstate: dict, connected: Iterable[str] | None = None, +) -> str | None: + """The device the dashboard should describe. + + A device with a live xiaozhi WebSocket wins; otherwise the one with the + newest activity. Probes and fallback ids are never picked.""" + devices = candidate_devices(pstate) + if not devices: + return None + connected_set = set(connected or ()) + return max( + devices, + key=lambda did: (did in connected_set, activity_t(devices[did])), + ) + + +def sensor_health( + pstate: dict, connected: Iterable[str] | None = None, +) -> tuple[list[str], list[str], list[str]]: + """Split devices into (judged, stale, fresh) for the stale-sensor check. + + A device is judged only when it is a real device, has sent a sensor + event since dotty-behaviour started and, when the connected list is + known (``connected`` is not None), currently holds a xiaozhi WebSocket. + A robot that is not connected is offline or asleep, not "hung"; a + device that never streamed cannot have a stuck stream.""" + connected_set = None if connected is None else set(connected) + judged: list[str] = [] + stale: list[str] = [] + fresh: list[str] = [] + for did, s in candidate_devices(pstate).items(): + if not sensor_seen(s): + continue + if connected_set is not None and did not in connected_set: + continue + judged.append(did) + (stale if s.get("sensor_stale") else fresh).append(did) + return judged, stale, fresh diff --git a/bridge/templates/dashboard.html b/bridge/templates/dashboard.html index 075eceeb..4c705b48 100644 --- a/bridge/templates/dashboard.html +++ b/bridge/templates/dashboard.html @@ -561,7 +561,7 @@

Latest vision capture

} (function () { if (typeof EventSource === 'undefined') return; - var psrc = new EventSource('/api/perception/feed'); + var psrc = new EventSource('/ui/perception/feed'); psrc.onmessage = function (ev) { var data; try { data = JSON.parse(ev.data); } catch (e) { return; } @@ -655,6 +655,9 @@

Latest vision capture

var data; try { data = JSON.parse(ev.data); } catch (e) { return; } prependFeedTurn(data); + // Replayed turns (history sent on connect / reconnect) fill the feed + // only — no card refresh storm and no toast for an old error. + if (data && data.replay) return; // Trigger refresh on polling tiles (status-strip, etc.) so they // pick up new state immediately on every turn. window.dispatchEvent(new CustomEvent('dotty-refresh')); diff --git a/bridge/templates/kid_mode.html b/bridge/templates/kid_mode.html index ea0470b4..6401d8b1 100644 --- a/bridge/templates/kid_mode.html +++ b/bridge/templates/kid_mode.html @@ -1,4 +1,4 @@ -
+
@@ -6,6 +6,7 @@
Kid Mode
{% if enabled %}content sandwich, camera denied, kid persona{% else %}guardrails off{% endif %} + {% if not saved %}· default, not saved{% endif %}
diff --git a/bridge/templates/kid_mode_result.html b/bridge/templates/kid_mode_result.html index 8a3079bd..589fa7f6 100644 --- a/bridge/templates/kid_mode_result.html +++ b/bridge/templates/kid_mode_result.html @@ -1,9 +1,11 @@ {% if ok %}
+ hx-get="/ui/kid-mode" hx-trigger="load delay:{{ '5s' if note else '300ms' }}" hx-swap="innerHTML" hx-target="#kid-card-body"> Kid Mode → {{ "on" if new_state else "off" }} + {% if note %}({{ note }}){% endif %}
- +{# A dotty-refresh would reload this card at once and hide the note. #} +{% if not note %}{% endif %} {% else %}
{{ error or "Kid mode toggle failed." }}
{% endif %} diff --git a/bridge/templates/say_result.html b/bridge/templates/say_result.html index a5e61f35..d99f54db 100644 --- a/bridge/templates/say_result.html +++ b/bridge/templates/say_result.html @@ -1,10 +1,12 @@ {% if ok %}
-
you said
+
{% if response %}you said{% else %}sent to Dotty{% endif %}
{{ sent }}
+ {% if response %}
Dotty replied
{{ response }}
+ {% endif %}
{% else %} diff --git a/bridge/templates/smart_mode.html b/bridge/templates/smart_mode.html index f5110c0a..4c4cdc7c 100644 --- a/bridge/templates/smart_mode.html +++ b/bridge/templates/smart_mode.html @@ -1,4 +1,4 @@ -
+
diff --git a/bridge/templates/smart_mode_result.html b/bridge/templates/smart_mode_result.html index 74c0a1a6..738811e6 100644 --- a/bridge/templates/smart_mode_result.html +++ b/bridge/templates/smart_mode_result.html @@ -1,9 +1,11 @@ {% if ok %}
+ hx-get="/ui/smart-mode" hx-trigger="load delay:{{ '5s' if note else '300ms' }}" hx-swap="innerHTML" hx-target="#smart-card-body"> Smart Mode → {{ "on" if new_state else "off" }} + {% if note %}({{ note }}){% endif %}
- +{# A dotty-refresh would reload this card at once and hide the note. #} +{% if not note %}{% endif %} {% else %}
{{ error or "Smart mode toggle failed." }}
{% endif %} diff --git a/bridge/templates/songs.html b/bridge/templates/songs.html index a8c4ee76..614b8f31 100644 --- a/bridge/templates/songs.html +++ b/bridge/templates/songs.html @@ -14,7 +14,7 @@ {% else %}
- No songs yet. Drop .opus / .mp3 / .wav files into + No songs yet. Drop .mid / .opus / .mp3 / .wav files into <XIAOZHI_PATH>/songs/ on the Docker host.
{% endif %} diff --git a/bridge/templates/state.html b/bridge/templates/state.html index be57f940..0724261f 100644 --- a/bridge/templates/state.html +++ b/bridge/templates/state.html @@ -44,21 +44,6 @@
{% endif %} - {% if sound_spark %} - {# #69 — sound-localizer balance over the last 60 s. 0.5 is centred; - a flat line near 1.0 is the stuck-left bias tracked in #27. #} -
- Sound balance - - {{ "%.2f"|format(sound_spark.last) }} -
- {% endif %} -