feat(telemetry): binary telemetry service + WebUSB transport (Phase 2); rename serial_plotter→telemetry

[finger563]

Phase 2 of the Serial Plotter (#781): a firmware-side binary telemetry transport the web app plots over WebUSB, alongside the existing text/CSV Web Serial path.

Component rename (breaking): now that this PR adds the firmware library, the component is renamed serial_plotter → telemetry (espp names components after their C++ library, espp::Telemetry). include/telemetry_service.hpp → include/telemetry.hpp, the web app web/serial_plotter.html → web/telemetry.html (hosted at apps/telemetry.html), and doc/en/serial_plotter/ → doc/en/telemetry/. All git-tracked as renames. The web app keeps its on-screen name "espp Serial Plotter".

Firmware: espp::Telemetry (include/telemetry_service.hpp)

A small device→host protocol on the stream_frame framing (dispatcher module 3). It is transport-agnostic — it can share a Dispatcher with other modules over any byte stream (USB vendor/WebUSB, CDC, UART, a socket) — though the web app's binary path reads it over WebUSB. Firmware declares named float channels (SCHEMA) and pushes SAMPLE frames — a device timestamp (u32 µs) + one float per channel, batchable — with emit(...). Host requests: GET_SCHEMA, SET_STREAM (enable/disable + rate).

Follows the CoreDumpService pattern: frames built under a mutex, the send callback invoked with the lock released; usable with a Dispatcher (handle()) or standalone (feed()). emit() and request handling are concurrency-safe.

The component now ships firmware, so it registers include/ and depends on base_component + stream_frame again (the deps removed when it was webapp-only).

Example (example/)

An esp32s3 app streams four synthetic channels (sine, cosine, noise, ramp) over the USB vendor (WebUSB) interface (the console/logs stay on the separate USB-Serial-JTAG). It wires one Dispatcher on that vendor stream — module 3 + discovery advertising app="serial_plotter.html" — and honors GET_SCHEMA / SET_STREAM. (CDC is enabled in sdkconfig.defaults only so the usb_device component compiles; the example creates no CDC interface.)

Web app (web/serial_plotter.html)

A new USB button connects over WebUSB (claims the vendor 0xFF interface's bulk IN/OUT pair), requests the schema, starts the stream, decodes stream_frame frames (vendored codec matching components/stream_frame), and maps SCHEMA → series and SAMPLE → the same ring buffers / uPlot plot, using the device timestamp as X. Serial and USB are mutually exclusive; pause / clear / filter / save / 2D / 3D modes all work over USB too.

Docs

New doc/en/serial_plotter/ component page (Telemetry API via include-build-file) + example include, registered in the main toctree and the Doxygen INPUT list; web_apps.rst, the README, and the app <meta> note the WebUSB path.

Testing

• Firmware: the example/ builds clean against ESP-IDF v6.1 (esp32s3) — image generated.
• Protocol match: the web app's crc32 matches the C++ golden (0xCBF43926); SCHEMA/SAMPLE frames build + parse + decode correctly (headless).
• Web app: loads clean with the USB button, no console errors; the shared plotting path is confirmed via the CSV/serial path (regression). Live WebUSB enumeration/transfer uses the same UsbTransport pattern as the other espp consoles (dispatcher_hub.html); an on-device WebUSB smoke test is the one thing not exercisable in CI.

🤖 Generated with Claude Code

[github-actions]

✅Static analysis result - no issues found! ✅

[Copilot]

🟡 Changes recommended

telemetry_service.hpp has likely C++20 build breaks (missing direct includes and std::vector passed to a std::span parameter) that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Phase 2 of the serial_plotter component by introducing a firmware-side binary telemetry service (espp::Telemetry) and a matching WebUSB transport in the existing browser plotter, plus docs and an ESP32-S3 USB streaming example.

Changes:

• Add espp::Telemetry header-only service using stream_frame (module 3) with SCHEMA/SAMPLE and GET_SCHEMA/SET_STREAM handling.
• Extend serial_plotter.html with a WebUSB transport that decodes stream_frame telemetry and feeds the existing plotting pipeline.
• Add example + documentation wiring (Sphinx + Doxygen) and update component metadata to include firmware dependencies.

File summaries

File	Description
doc/en/web_apps.rst	Mentions Serial Plotter’s new WebUSB binary telemetry path in the web apps overview.
doc/en/serial_plotter/serial_plotter.rst	New component docs page describing Web Serial vs WebUSB and the Telemetry protocol.
doc/en/serial_plotter/serial_plotter_example.md	Adds docs include of the example README.
doc/en/serial_plotter/index.rst	Adds Serial Plotter section entry to the docs tree.
doc/en/index.rst	Registers Serial Plotter docs section in the main docs toctree.
doc/Doxyfile	Adds Telemetry header + example source to Doxygen inputs/example paths.
components/serial_plotter/web/serial_plotter.html	Implements WebUSB connect + stream_frame parsing and telemetry schema/sample decoding.
components/serial_plotter/README.md	Updates README to describe both transports and the Telemetry firmware path.
components/serial_plotter/include/telemetry_service.hpp	Introduces the espp::Telemetry firmware protocol/service implementation.
components/serial_plotter/idf_component.yml	Adds firmware deps and example metadata now that the component ships code.
components/serial_plotter/example/sdkconfig.defaults	Configures ESP32-S3 USB + TinyUSB vendor/CDC settings for telemetry streaming.
components/serial_plotter/example/README.md	Documents how to build/use the USB telemetry example with the web app.
components/serial_plotter/example/main/telemetry_example.cpp	Implements ESP32-S3 demo producing synthetic channels over vendor + CDC with Dispatcher routing.
components/serial_plotter/example/main/CMakeLists.txt	Registers the example’s main component requirements.
components/serial_plotter/example/CMakeLists.txt	Adds a managed-deps-friendly example project configuration and component search path narrowing.
components/serial_plotter/CMakeLists.txt	Updates component registration to export include/ and depend on base_component + stream_frame.

Review details

Suppressed comments (2)

components/serial_plotter/include/telemetry_service.hpp:180

• send_fn expects a std::span, but this call passes a std::vector<uint8_t>, which can fail to compile under C++20. Pass an explicit span over the vector storage.

    s(frame);

components/serial_plotter/include/telemetry_service.hpp:283

• send_fn expects a std::span, but send_frame passes a std::vector<uint8_t>. This can fail to compile under C++20; pass an explicit span over the vector bytes.

    if (s)
      s(frame);

• Files reviewed: 16/16 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

WebUSB lifecycle and pause behavior, CDC compatibility, schema bounds, concurrency guarantees, and CI coverage need correction.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

components/serial_plotter/example/main/telemetry_example.cpp:52

• This binary CDC path is not consumable by the Serial Plotter web app: its Web Serial loop decodes every chunk as UTF-8 and calls feedText() (serial_plotter.html:764-799), while only UsbTransport feeds SfParser. Consequently selecting the example's CDC port cannot request a schema or plot telemetry. Either add a binary Web Serial mode or remove the CDC telemetry path and its claims.
  components/serial_plotter/include/telemetry_service.hpp:91
• Concurrent emit() and request handling can invoke this callback simultaneously because each path releases mutex_ before sending. The type does not document that send must be thread-safe, despite promising concurrent-safe use; a transport without the example's separate TX mutex can race or interleave frames. Serialize callback invocations internally, or explicitly require callers to provide a serialized/thread-safe callback.
  components/serial_plotter/web/serial_plotter.html:946
• open() can fail after the device has been opened or an interface claimed, but this catch drops t without closing it. That leaves the WebUSB device/interface open and can make the next connection attempt fail; close t before returning, as the other web transports do.
  components/serial_plotter/web/serial_plotter.html:958
• SET_STREAM is sent only during connect/disconnect. The shared Pause handler merely toggles paused and drops incoming samples, so a WebUSB device continues transmitting while paused and is never re-enabled through this request on resume. Send SET_STREAM with enabled=0/1 when Pause changes in USB mode (while retaining local-only pause for text serial).
  components/serial_plotter/example/CMakeLists.txt:7
• This raises the example to C++23 even though the repository standard and the directly related examples use C++20 (components/dispatcher/example/CMakeLists.txt:22, components/stream_frame/example/CMakeLists.txt:21, and components/usb_device/example/CMakeLists.txt:35). This code needs no C++23 feature, so the higher setting unnecessarily narrows compiler/ESP-IDF compatibility.
  components/serial_plotter/example/CMakeLists.txt:49
• The new firmware example is absent from .github/workflows/build.yml; the matrix jumps from seeed-studio-round-display to serialization. Add an alphabetically placed components/serial_plotter/example entry targeting esp32s3 (using the manager-disabled command if intended) so this firmware path is continuously built.

• Files reviewed: 16/16 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Concurrency hazards, schema boundary failures, and several WebUSB plotting state bugs can corrupt or stall telemetry sessions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (7)

components/serial_plotter/include/telemetry_service.hpp:113

• Config documents at least one channel, but construction accepts an empty vector. That emits a zero-channel schema; the new browser rejects it (labels.length is false) and retries until “no schema,” so this valid-looking configuration can never connect. Reject empty schemas or define and support zero-channel behavior explicitly.

  explicit Telemetry(const Config &config)
      : BaseComponent("Telemetry", config.log_level)
      , channels_(config.channels)
      , send_(config.send)
      , streaming_(config.stream_on_start)
      , period_ms_(config.period_ms) {
    clamp_channels();

components/serial_plotter/include/telemetry_service.hpp:272

• The per-name 255-byte cap does not keep the whole SCHEMA within stream_frame::kMaxPayloadSize (4096). For example, sixteen 255-byte names produce 4,115 bytes, so build_frame() returns an empty vector and send_schema() silently delivers no frame. Enforce the aggregate payload limit (and report truncation/rejection) while building the schema.

      const uint8_t len = static_cast<uint8_t>(std::min<size_t>(name.size(), 255));
      p.push_back(len);
      p.insert(p.end(), name.begin(), name.begin() + len);
    }

components/serial_plotter/include/telemetry_service.hpp:317

• deliver() holds a non-recursive mutex while invoking user code. If the send callback synchronously re-enters Telemetry and reaches emit(), send_schema(), or a request reply, the nested deliver() blocks forever, contradicting the class-level guarantee that a re-entrant transport cannot deadlock. Drain a serialized outbound queue without holding its guard during the callback, or explicitly disallow re-entry and update the contract.

  void deliver(const send_fn &s, const std::vector<uint8_t> &frame) const {
    std::lock_guard<std::mutex> lock(send_mutex_);
    s(std::span<const uint8_t>(frame));

components/serial_plotter/example/main/telemetry_example.cpp:170

• The timer snapshots period_ms() only during construction. A later nonzero SET_STREAM updates Telemetry::period_ms_, but the producer remains at the original 10 ms cadence, so the example does not honor the requested rate as described. Detect period changes and call gen_timer.set_period(...), or remove the rate-honoring claim.

  espp::Timer gen_timer(
      {.period = std::chrono::milliseconds(telemetry.period_ms()),
       .callback = [&]() -> bool {

components/serial_plotter/web/serial_plotter.html:772

• The rolling window assumes the first and last X values bound the selected samples. In 2D mode or with “First column” selected, X can be non-monotonic (the example’s sine channel is one case), yielding reversed or clipped scales. Compute the minimum and maximum X across [lo, n) instead.

            plot.setScale("x", { min: xs[Math.min(lo, n - 1)], max: xs[n - 1] });

components/serial_plotter/web/serial_plotter.html:786

• This live-edge shift also runs in 2D X–Y mode whenever the hidden xMode is not firstcol. There, xs is an arbitrary selected signal and is not chronological, so changes in its last value spuriously pan the user's view. Restrict this scrolling behavior to time-series mode.

          if (n && lastLiveX != null && curMin != null && curMax != null &&
              $("xMode").value !== "firstcol") {

components/serial_plotter/web/serial_plotter.html:1204

• This unconditionally re-enables the Serial controls after a USB disconnect, even when navigator.serial is unavailable. That leaves a nonfunctional Connect Serial button in WebUSB-only browsers. Preserve the feature-detection result when resetting the UI.

      $("connect").disabled = false; $("baud").disabled = false;

• Files reviewed: 17/17 changed files
• Comments generated: 7
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The 2D draw path currently throws, and telemetry schema validation and callback locking have correctness risks.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

components/serial_plotter/web/serial_plotter.html:777

• Using only the first and last X values does not define the range for XY mode, whose selected X column is explicitly allowed to be non-monotonic. A sine-like X can yield reversed or tiny bounds and hide most samples in the requested time window. Compute the extrema over the time-selected slice instead.
  components/serial_plotter/example/README.md:21
• The app never requests a rate—it always sends a zero period—and the example reads period_ms() only once when constructing its timer, so this rate claim is not implemented. Remove it unless a rate control and dynamic producer-period update are added.

components/serial_plotter/include/telemetry_service.hpp:305

• The channel validation does not enforce the wire constraints. An empty channel list is accepted even though the browser rejects zero-channel schemas, and a legal count with long names can exceed stream_frame::kMaxPayloadSize (for example, 16 names of 255 bytes produce 4,115 payload bytes). build_frame() then returns an empty vector and the host retries forever. Validate the minimum count and cumulative encoded schema size, then reject or explicitly truncate invalid configurations before sending.

  void clamp_channels() {
    if (channels_.size() > kMaxChannels) {
      logger_.warn("{} channels exceeds the {}-channel SCHEMA limit; truncating", channels_.size(),
                   kMaxChannels);
      channels_.resize(kMaxChannels);

• Files reviewed: 17/17 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The X–Y redraw path crashes, schema sizing can suppress replies, and telemetry locking can deadlock re-entrant callbacks.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

components/telemetry/web/telemetry.html:767

• In X–Y mode buildData() returns [null, [x, y], ...], so data[0] is null and this xs.length access throws on the first redraw. The same mode-1 assumption is repeated in the non-following branch and final lastLiveX update. Derive the shared X array from data[1][0] for mode 2 (and decide whether the time-window logic should be disabled or slice mode-2 data) before using it in all three places.
  components/telemetry/include/telemetry.hpp:295
• The schema can exceed stream_frame::kMaxPayloadSize long before the documented 255-channel limit (for example, 17 channels with 255-byte names). build_frame() then returns an empty vector, and send_schema() invokes the transport with an empty frame, so the host can never complete its handshake. Budget channel-name bytes against the framing payload cap while reserving the type/length bytes for every remaining channel.

    for (size_t i = 0; i < n; i++) {
      const auto &name = channels_[i];
      p.push_back(static_cast<uint8_t>(ChannelType::F32));
      const uint8_t len = static_cast<uint8_t>(std::min<size_t>(name.size(), 255));
      p.push_back(len);
      p.insert(p.end(), name.begin(), name.begin() + len);
    }

components/telemetry/include/telemetry.hpp:63

• The callback is still invoked while send_mutex_ is held (for example at line 158), so a callback that synchronously re-enters emit(), send_schema(), or request handling deadlocks on that mutex. This contradicts the stated re-entrancy guarantee and the CoreDumpService pattern. Serialize frames with an outbound queue/state flag, but release every internal lock before invoking user code.

/// on a transport RX task; both are safe to call concurrently. Frames are built
/// under one mutex and transmitted under a separate send mutex, so the user
/// `send` callback is (a) never invoked while the build mutex is held — a
/// re-entrant transport cannot deadlock — and (b) never invoked concurrently, so
/// a `send` that is not itself thread-safe still cannot interleave the bytes of
/// two frames.

doc/Doxyfile:434

• The new INPUT entry is not alphabetical because it precedes serialization and the seeed-* entries. Per the repository Doxyfile convention, move telemetry.hpp to after the task headers and before thermistor.

  $(PROJECT_PATH)/components/telemetry/include/telemetry.hpp \

components/telemetry/include/telemetry.hpp:306

• Config::channels promises at least one channel, but this validation only enforces the upper bound, so both construction and set_channels({}) accept an empty schema. The WebUSB client rejects that schema (labels.length is false) and retries until “no schema.” Add an explicit non-empty validation/error path and apply it consistently to construction and runtime replacement.

  /// Truncate channels_ to the u8 SCHEMA limit (caller holds mutex_, or is the
  /// constructor). Warns when channels are dropped.
  void clamp_channels() {
    if (channels_.size() > kMaxChannels) {
      logger_.warn("{} channels exceeds the {}-channel SCHEMA limit; truncating", channels_.size(),
                   kMaxChannels);
      channels_.resize(kMaxChannels);
    }

• Files reviewed: 20/21 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Concurrency and transport-state races can emit after stopping or corrupt/reset active plotting sessions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

components/telemetry/web/telemetry.html:958

• Switching from WebUSB to Serial leaves the old binary schema and device-time samples intact: usbResetUI() does not clear them, and connect() only changes the controls. If the Serial source has the same header (or starts with numeric rows), ingestLine() never calls setSchema(), so browser-relative timestamps are appended to a ring still identified as webusb, producing a non-monotonic/mislabeled X axis. Reset the binary dataset/source when starting Serial.
  components/telemetry/web/telemetry.html:1214
• Clearing usb before this asynchronous teardown finishes allows another click to start a new connection immediately. The old disconnect then unconditionally runs usbResetUI(), clearing usbHaveSchema and resetting controls/status for the new transport; similarly, the old read-loop catch currently checks only whether any usb exists and can tear down the replacement. Serialize connect/disconnect with a busy or generation token, and gate callbacks/UI reset on usb === t.

• Files reviewed: 20/21 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Several edge cases can corrupt labels, freeze plots, hide cleared captures, or produce non-atomic USB frames.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

components/telemetry/web/telemetry.html:1289

• Once the ring reaches cap, tRing.count stops changing, so the 3D cache key also stops changing and drawScene() reuses stale points forever. WebUSB streaming in 3D therefore freezes at the retention limit. Invalidate sceneCache in the shared sample-ingestion path or include a ring generation/write index in its key.
  components/telemetry/web/telemetry.html:1429
• Clearing while Follow is disabled leaves lastLiveX and the non-following state from the previous capture. For WebUSB's absolute device time, the emptied plot resets to an unrelated scale; the first new sample updates lastLiveX, but later samples never satisfy the live-edge test, so the new capture remains off-screen. Treat Clear as a fresh followed dataset and reset the stale live-edge marker.

components/telemetry/web/telemetry.html:381

• Removing every % corrupts legitimate unit labels. Existing producers such as bq27220/example/main/bq27220_example.cpp:77 emit %time(s), ..., SoC (%), so this turns SoC (%) into SoC () (and %/hr into /hr). Strip only the leading header marker.
  components/telemetry/include/telemetry.hpp:318
• An empty channel set breaks the firmware/browser contract rather than merely producing a useless schema. set_channels({}) sends a zero-column schema, but the web client rejects it via labels.length and retains the previous width, so subsequent zero-channel samples are silently dropped against that stale schema. Reject empty construction/updates, or explicitly clear the browser schema when zero channels are supported.

    if (channels_.empty()) {
      // The config contract asks for >= 1 channel; a zero-channel SCHEMA is
      // ignored by the web client. Warn rather than emit a silently-useless one.
      logger_.warn("Telemetry configured with no channels; SCHEMA will be empty");
      return;

• Files reviewed: 20/21 changed files
• Comments generated: 1
• Review effort level: Balanced