feat(studio): controller-fault detection, post-mortem probes, soak driver, GET_HEALTH decode — Studio v0.76 (fw #50) - #198
Open
mbreiser wants to merge 14 commits into
Open
feat(studio): controller-fault detection, post-mortem probes, soak driver, GET_HEALTH decode — Studio v0.76 (fw #50)#198mbreiser wants to merge 14 commits into
mbreiser wants to merge 14 commits into
Conversation
…iver, GET_HEALTH decode — Studio v0.76 (fw #50) A Mode-3 controller wedge (firmware #50) used to be invisible: the closed-loop drain loop swallowed every 0x70 timeout and a dead-arena run "completed" and auto-committed. Now: - fictrac-bridge-client: rolling fault detector (>=3 failed applies in the last 10, not "3 consecutive" — a late same-opcode reply can satisfy the next request) → latched `fault` event, apply forced OFF. - arena-runner: `fault(reason, detail)` wakes the trial wait and unwinds; `summary.fault` / `summary.faultDetail` / `summary.stopAcked`; a timed-out protocol command (the 0x08 after the 0x70s) is labelled the same. - arena-session: bridge fault → runner.fault + `fault` event; runner sends go through session.send (a link facade) so trialParams/STOP land in the run log; `reconnect()` / `flushRx()`; `dt` on the monotonic clock; terminal summary + fault detail in the sanitized runner rows. - arena-link: `reconnect()` from granted ports (navigator.serial.getPorts, VID/PID match, ambiguity refused, re-enumeration poll); the response deadline now covers a stalled write; `flushRx()`. - run-log / adapter: outcome CONTROLLER_FAULT (auto-commits; ABORTED_BY_USER still doesn't); `fault` phase in the transcript. - studio-postmortem (new, classic dual-export): quiet period + rx flush → confirmation 0xC2 → probe window (5 s timeouts, typed `probe` rows with decoded + raw hex) → halt | SYSTEM_RESET → reconnect → MAC verify → post-reset probe. - wire: GET_HEALTH 0xCA encode/decode (55 B + optional 11 B slowest-op tail), GET_FRAME_POSITION 0x72, capability bit 7 `health`. - Studio: fault banner; `runOnce()` (the no-dialog half of beginRun); soak driver (File ▾ → Soak…, `?soak=1`, advanced-only; refuses without a behavior_v2 ack or bridge frames; halt-first; never auto-commits); Debug ▾ → Controller health; URL-state `soak` flag; v0.76 footer. - scripts/wedge-scan.py + tests/test-wedge-scan.py: per-run wedge/onset table over v1/v2/gz/.runlog.json (found a sixth, earlier occurrence: rig2 2026-07-11). - fictrac_sim.py: --turn-sigma / --jump-every / --jump-deg. - protocols/soak_mode3_closed_loop.yaml; docs/development/mode3-wedge-soak-plan.md (findings, hypothesis ladder, campaign spec, bench recipe); release notes; CLAUDE.md rule. Tests: full `pixi run test` green (new: fault latch, runner fault, session facade + fault routing, link reconnect/write-deadline, wire GET_HEALTH vectors, adapter/run-log outcome, postmortem lifecycle, URL soak flag, wedge-scan). Not bench-tested. Codex (gpt-6-astra) plan review reconciled in .codex-review/report-20260911-wedge-soak.md (gitignored). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…havior_v2; honest stop reason Bench 2026-09-11 (10-10 controller, sim): the first smoke iteration ran clean (2,001 × 0x70, RTT 2/6 ms) but (1) the protocol's last step disconnected the fictrac plugin — the bridge IS the run's logger, so the run's terminal event never reached the log and the soak loop then waited on frames from a disconnected bridge; (2) a remembered behavior_v1 session setting overrode the soak preflight's v2 negotiation at run start. Drop fictrac.disconnect from the soak protocol, reconnect the bridge before the freshness check, force behavior_v2 while a soak runs, and report the operator's stop reason instead of "no-frames". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…roller (silent, 0x01 no ack); flash mishap note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… an apply failure
Bench 2026-09-12 07:00 ET: a firmware built for a 4x10 arena rejected every
pattern on the 2x10 card, and the runner streamed 21k rejected 0x70s
("no pattern selected", status 1) for minutes without tripping the fault
detector — it only counted throws (timeouts/transport). The session's bridge
applyFrame now decodes the reply and throws on a non-ok status, so rejects feed
the ≥3-of-10 window like timeouts do (one #199 reject per trial start stays
well below the threshold).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…RMWARE_VERSION 0xCB) in every run log
No G6 controller build was identifiable until now: 0xC2 carries a format byte
that is always 1, 0xE3 is the panel image footer, and run_metadata.firmware
held "v1" (or null after a gesture-free reconnect). Firmware
feat/controller-health adds GET_FIRMWARE_VERSION 0xCB — git SHA, dirty flag,
branch, build date, arena rows×cols, compiled in by a PlatformIO pre-script.
- wire: encodeGetFirmwareVersion / decodeFirmwareVersion (46 B payload →
{sha, dirty, debug, date, branch, arena, label}), golden tests.
- Studio: Studio.refreshFirmwareVersion() on the Connect button, the Console
info command, and a state hook that covers every session.reconnect() (the
post-mortem/soak paths, which used to leave firmware null); gated on the
`health` capability bit because older firmware flashes CE 01 on the arena
for an unknown opcode. Studio.firmware → run_metadata.firmware → run log.
- post-mortem probe set includes firmware_version (post-reset probe records
which build wedged); wedge-scan.py gains an `fw` column.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ost side (fw feat/telemetry-ring)
The controller keeps CMD/FRAME/STATE records in reboot-surviving OCRAM
(proposal § 3.1); the host drains them with GET_TELEMETRY_BLOCK 0xA9 using an
ack cursor (lossless, framing A) and writes compact rows into the run log.
- js/arena-telemetry.js (new, classic dual-export): parseBlock (18 B header +
records: CMD/FRAME/STATE/PAD), toRows → ["cc"|"cf"|"cs", rx, t_us, seq, …],
createDrainer (ack cursor, seq gaps, controller drop counter, drainAll crash
dump), createPoller (single-flight, gated). tests/test-arena-telemetry.js.
- wire: SET_TELEMETRY 0xA8 / GET_TELEMETRY_BLOCK 0xA9 encoders + tests.
- bridge 3.1: {type:'rows'} → each shape-checked array row written verbatim
(write_rows); fictrac-bridge-client.logRows(); Python test.
- Studio: Studio.initTelemetry() on every link-up (health cap + 0xA8 ack),
10 Hz drain while connected (pauses under the post-mortem), stream_schema
line per log, minute stats, Studio.drainTelemetry() wired as the
post-mortem's afterReconnect → the ring's last records after a reset.
- wedge-scan.py: ctl recs / sd max us / ctl states / ctl rejects columns and,
with --verbose, the last 40 controller records before a wedge.
- docs: plan § 11 (T4 as built), release notes, CLAUDE.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…before apply (#199) The bridge applies a new `frames` modulus one frame late, so the first index after a trial whose pattern has a different frame count carried the previous modulus → "index out of range" + an error glyph once per trial (bench 2026-09-11: 2 of 110,643). Now that controller rejects feed the fault window, a protocol with short alternating trials could even trip a false CONTROLLER_FAULT. Wrap locally (idx mod frames) before the consumer clamp; no configured count → untouched. run_metadata gains `telemetry: ring-10hz|off` so the instrumentation factor is visible in analysis. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red ring drain, deferred fault commit, reader safety Reconciles the gpt-6-astra status + diff reviews (.codex-review/report-20260912-status.md): - runner: terminal event emitted after the best-effort STOP so the serialized summary carries stopAcked; Studio defers a CONTROLLER_FAULT run's export/commit until the post-mortem (probes + ring dump) has finished - post-mortem: identity requires a decoded 0xC2 reply and the same MAC; "recovered" requires zero timeouts/errors and every required probe ok (SD-image probes are optional); capability-gated probes are skipped + recorded when caps are unknown; evidence first after reset (health → ring drain → probes), duplicate drain removed - telemetry: FRAME sd_load_us u32, header flag bits 2/3, maxChunks 40 / maxBytes 178, ack only when the sink stored the rows (notStored), drainer.idle(); Studio gates SET_TELEMETRY on GET_FIRMWARE_VERSION flags bit 2 (never on `health`), on/off switch (Debug ▾), stop path, poller yields to the post-mortem; analog-in too - readers: runlog-replay + dashboard analysis-core skip string-tagged rows - bridge 3.1: per-tag minimum row lengths; logRows returns acceptance - wedge-scan: fault-declared outcome, pre-boot ring tails, malformed rows counted - soak: per-iteration applied_0x70 / duration / telemetry stats, `unexercised` - docs: status doc §4b/§4c/§6, plan bench log, release notes, CLAUDE.md rules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… capability Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…(re)connect and post-reboot, duplicates dropped, fresh ring resets the cursor Codex firmware review: an old ack sent to a power-cycled controller would free unseen records. The Studio now rearms the drainer on every link-up and before the post-reboot crash-dump drain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 12, 2026
…refused-row spin) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines the Mode-3 controller-wedge bug (reiserlab/LED-Display_G6_Firmware_Arena#50) with the telemetry plan from #192. Not bench-tested — the whole chain needs a night on the CSHL 10-10 controller with the simulator. Design, findings, hypothesis ladder and the campaign spec:
docs/development/mode3-wedge-soak-plan.md. Codex (gpt-6-astra) plan review reconciled in.codex-review/report-20260911-wedge-soak.md(local, gitignored) — its corrections are folded in (0x82 is a RAM lookup, so the issue's "SD tier" is a queueing artifact; late same-opcode replies defeat a consecutive counter; faulted runs must produce a durable artifact; hold the firmware frame-cache candidate until a baseline wedge reproduces).What changes for every run (fail closed)
CONTROLLER_FAULT(never "aborted by user") and the log auto-commits. Before this a wedged 40-minute run completed and committed looking normal.summary.stopAckedrecords whether the best-effort STOP got through ("fail closed" = the host stopped; quiescence only if acked).session.send, so they appear asarena_commandrows.dtis measured on the monotonic clock.js/studio-postmortem.jsruns: quiet ≥1 s + rx flush → confirmation 0xC2 → ~60 s probe window (0xC2/0x33/0x72/0xCA/0x88/0xE3, 5 s timeouts, typedproberows with decoded + raw hex) → halt (outside a soak).Soak driver (advanced-only)
File ▾ → Soak (repeat protocol)… or
?soak=1: repeats the open protocol via the newStudio.runOnce()(N times / H hours / gap), refuses without abehavior_v2ack or without bridge frames, pauses (not faults) when the simulator dies, halts on the first fault by default (evidence first), then SYSTEM_RESET →ArenaLink.reconnect()from granted ports (VID/PID, ambiguity refused) → MAC verified → post-reset probe → new iteration (≤3 resets). Soak runs never auto-commit.protocols/soak_mode3_closed_loop.yaml= 20 × 60 s Mode-3 trials alternating the 200-frame bar and the 20-frame grating.fictrac_sim.pygains--turn-sigma/--jump-every/--jump-deg.Analyzer
scripts/wedge-scan.py(v1/v2/gz/.runlog.json, streaming) — over the course repo it found a sixth, earlier occurrence (rig2, 2026-07-11, third controller, identical signature) and reproduced the issue's clean-control numbers exactly. The five reported wire logs are not in the repo (aborted runs never auto-committed).Firmware side
reiserlab/LED-Display_G6_Firmware_Arena#53: GET_HEALTH 0xCA (loop timing, SD read stats + error code, counters, reset cause, reset-surviving breadcrumb + slowest op) and
scripts/soak_mode3.py(browser-free driver, same lifecycle/log format). Decoded here byencodeGetHealth/decodeHealth(55 B + 11 B tail) and Console → Debug ▾ → Controller health. Follow-ups filed: fw #51 (flushResponses short write), fw #52 (SD path), #197 here (self-healing runner), Linear LAB-212.Tests
Full
pixi run testgreen. New: fault latch (incl. interleaved + window aging), runnerfault()mid-wait + timeout labelling, session facade + fault routing + sanitizer, linkreconnect()(same port / re-enumerated / ambiguous / timeout) + write-stall deadline, wire GET_HEALTH + 0x72 vectors, adapter/run-logCONTROLLER_FAULT, postmortem lifecycle (transient / halt / dead / reset-continue / reset-failed / wrong MAC / operator abort / capability gate), URLsoakflag,tests/test-wedge-scan.py(97 checks).Bench gate (before merge)
git pull, restartpixi run bridge, hard-refresh; log-level mirror readsbehavior_v2.DISCONNECTED(unchanged path); Node tests cover the timeout path.iterations=1 hours=0 gap=5 first=halt) → one clean iteration,soakrows in the bridge log, no auto-commit.breadcrumb_valid+prev_slow_op.docs/development/mode3-wedge-soak-plan.md§ 6.🤖 Generated with Claude Code