From 39131d738b8671f0e1ac6a83fdb192fbe774123b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 14:39:31 +0000 Subject: [PATCH 1/3] docs(design): 060 tracing off by default, wall-clock epoch in trace IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - trace-context is off by default; every crate that depends on core forwards the feature, and libraries never enable it (§2.6) - the trace ID's reserved bytes hold an epoch from unix_time(), so trace IDs do not repeat across a restart on instances with a wall clock (§4.2) - span IDs may still repeat across a restart; consumers bound parent walks (§6) Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WGkDJ8UNN7mWpJEEj5zxzX --- docs/design/060-trace-context.md | 206 +++++++++++++++++++++---------- 1 file changed, 144 insertions(+), 62 deletions(-) diff --git a/docs/design/060-trace-context.md b/docs/design/060-trace-context.md index 787592da..e5fe231d 100644 --- a/docs/design/060-trace-context.md +++ b/docs/design/060-trace-context.md @@ -11,7 +11,8 @@ single-input async derivations. Rerun taps get exact causal chains, and agents can follow them through the writes they captured or can still read (§6). Breaking for Rust APIs (join handlers, `WriteMetadata`, session engine, `AimDbBuilder`); additive on the wire (the graph shapes do not -change). Ships as `aimdb-core` 4.0 (§8.1). +change). Ships as `aimdb-core` 4.0 (§8.1). Tracing is off by default and +enabled with the `trace-context` feature (§2.6). Two throwaway prototypes tested this design. The second implements all of it except Zenoh, the CLI/MCP display and Rerun; §13 lists what it showed, and @@ -138,7 +139,8 @@ tc.parent(); // Option: None for an origin tc.depth(); // u8 ``` -- **In process:** every value read from a local record has `Some`. +- **In process:** with `trace-context` enabled (§2.6), every value read from + a local record has `Some`. - **`aimdb-client`:** `Some` when the peer sent a `trace`, `None` when it did not (an older peer, or a link with tracing off). The client never invents an ID, so two reads of the same stored value report the same thing. @@ -146,30 +148,42 @@ tc.depth(); // u8 `produce_from` takes `&WriteMetadata`. If its trace is `None` (metadata decoded from a peer and passed back in), the write starts a trace. -### 2.6 Opting out - -Tracing is on by default, behind the `trace-context` feature. Turning it -off removes the trace from `WriteMetadata` (back to 8 bytes, §4.3), -`trace()` returns `None`, AimX omits the field and inbound traces are -ignored. No instance tag is required: `instance_tag(…)` is still accepted -and ignored, so the same application code builds with tracing on and off. -This is the opt-out for MCUs that cannot spare the RAM. - -**What "on by default" reaches.** Cargo features unify across the -application's dependency graph, so tracing is on when any crate in it enables -`aimdb-core/trace-context`. The feature is in the default features of -`aimdb-core`, `aimdb-embassy-adapter` and `aimdb-wasm-adapter`. But 25 of the -44 crates in this workspace that depend on core do so with -`default-features = false`, and every Embassy example and connector disables -the Embassy adapter's defaults. In the prototype that left tracing off in -all four Embassy examples and in `aimdb-bench`, silently. So: - -- **Applications that disable default features list `trace-context`.** The - Embassy examples do (phase 1), and the Embassy getting-started docs show - it with `.instance_tag(…)`. -- **Library crates (connectors, `aimdb-data-contracts`) never enable or - disable it.** They follow the application. -- **Benchmarks and tests that measure tracing enable it explicitly.** +### 2.6 Opting in + +Tracing is off by default. It sits behind the `trace-context` feature of +`aimdb-core`, which is in no crate's default features. Without it, +`WriteMetadata` stays at 8 bytes (§4.3), `trace()` returns `None`, AimX +omits the field and inbound traces are ignored. The API is the same either +way: `DerivedProducer`, `produce_from` and `instance_tag(…)` exist with the +feature off and do nothing extra, so the same application code builds with +tracing on and off, and an MCU that cannot spare the RAM pays nothing. + +**Turning it on.** Every crate that depends on core forwards the feature: + +```toml +# in each adapter, connector, aimdb-client, aimdb-sync, aimdb-data-contracts +[features] +trace-context = ["aimdb-core/trace-context"] +``` + +An application enables `trace-context` on any AimDB crate it already lists, +usually its runtime adapter. Cargo unifies features across the dependency +graph, so that one line turns tracing on in core and in every connector the +application uses. (The prototype had it on by default instead; because most +crates in this workspace take core with `default-features = false`, that +left tracing off in all four Embassy examples and in `aimdb-bench`, +silently.) + +- **Library crates forward the feature and never enable it.** No crate lists + `trace-context` in its defaults or turns it on in a dependency. Enabling it + is the application's decision, because it also makes the instance tag + mandatory (§4.2): a library that enabled it would make a working + application fail at `build()`. +- **Applications that want tracing enable it and set a tag.** The examples + that show tracing do both; the getting-started docs show the feature line + next to `.instance_tag(…)`. +- **Benchmarks and tests that measure tracing enable it explicitly**, and + fail if they are built without it. ## 3. Dependency graph @@ -233,17 +247,31 @@ full span ID, including the instance it was written on: ```text counter = instance write counter, starting at 1 at build(), + 1 per write +epoch = Unix time in seconds (u32) when the instance first had a wall clock; 0 before span = instance_tag (32 bits) ‖ counter (low 32 bits) -trace_id = instance_tag (4 bytes) ‖ 0 (4 bytes) ‖ counter (8 bytes) (origin writes only) +trace_id = instance_tag (4 bytes) ‖ epoch (4 bytes) ‖ counter (8 bytes) (origin writes only) ``` There is no mixing step and no randomness: two instances with different tags can never produce the same span or trace ID, whatever their counters. (A mixed 32-bit span cannot give that guarantee: `0x10 ⊕ 1` and `0x13 ⊕ 2` are the same input to the mix, so two instances can emit the same span, and -a mirrored write can end up with `span == parent`.) The four zero bytes of -the trace ID are reserved; a later exporter may fill them, for example with -a boot value (below). +a mirrored write can end up with `span == parent`.) + +**Epoch.** The epoch makes trace IDs differ from one run of an instance to +the next. It comes from the instance clock's `unix_time()` (058): + +- The first origin write that finds `unix_time()` available stores its + seconds as the epoch; from then on the epoch does not change in that run. + On Tokio and WASM that is the first write. On Embassy it is the first + write after `set_unix_time` (an RTC read at start, or SNTP). +- Until then the epoch is 0. Changing from 0 to the stored value during a + run is safe: the counter never repeats within a run, so `(epoch, counter)` + pairs stay distinct. +- The clock is read once per run, not per write. An origin write costs one + more atomic load. +- The epoch is in the trace ID only. A span has no room for it (§4.3), so + span IDs still repeat across restarts (below). **Instance tags are the application's choice.** The application assigns each instance its tag with `AimDbBuilder::instance_tag(NonZeroU32)`, for @@ -255,29 +283,47 @@ read in a trace, and stays the same across restarts. Tags are `NonZero`, so a span is never all zero, including at the counter wrap. The `test_support` builders set a fixed tag, so tests that do not care about it need no change. -**Guarantee.** Within one trace, two writes have different span IDs if +**Guarantee: spans.** Within one trace, two writes have different span IDs +if 1. the instances that wrote them have different instance tags, or 2. they were written by the same instance in the same run, fewer than 2³² writes apart. +**Guarantee: trace IDs.** Two origin writes have different trace IDs if + +1. the instances that wrote them have different instance tags, or +2. they were written by the same instance in the same run, or +3. they were written by the same instance in different runs, both runs had + a wall clock when they wrote, and the two epochs differ. + Keeping tags distinct across instances is the application's responsibility; §4.2.1 lists what the framework detects when it is not. -**Restarts are not protected.** The counter starts at 1 on every `build()`, -so a restarted instance repeats its span and trace IDs from before the -restart. A trace from before a restart can still reach the instance (a -retained MQTT message, an AimX late-join snapshot, a peer's buffered mirror -write) and mix with new writes; such a trace may then contain two writes -with the same span. AimDB instances are long-running and restarts are rare, -so this is accepted. One case is still prevented, because it would make a -write its own parent: +**Restarts.** The counter starts at 1 on every `build()`. + +- **Trace IDs do not repeat** across a restart when the instance has a wall + clock: the new run has a later epoch, so its traces never merge with + traces from before the restart. Three cases are not covered and are + accepted: an instance that never gets a wall clock (epoch 0 in every + run), two runs whose first clock readings fall in the same second, and a + clock set back to an earlier run's epoch. In those cases a new trace can + reuse the ID of an old one, as it would without the epoch. +- **Span IDs repeat.** A restarted instance reuses its spans from before the + restart. A trace from before a restart can still reach the instance (a + retained MQTT message, an AimX late-join snapshot, a peer's buffered + mirror write) and be continued by new writes; such a trace may then + contain two writes with the same span, and in rare cases a parent chain + that returns to a span it already passed. Consumers that walk parents + must therefore bound the walk (§6). + +One case is prevented in the writer, because it would make a write its own +parent: - **A new span would equal its cause's span.** The writer takes the next counter value instead. A write never stores `span == parent`. Without this, a restarted instance that ingests its own retained value `tag‖1` as - its first write would store `tag‖1` with parent `tag‖1`, a cycle that - breaks every consumer walking parents. + its first write would store `tag‖1` with parent `tag‖1`. ### 4.2.1 Detecting duplicate tags @@ -312,7 +358,8 @@ per write or at `build()`, so no runtime adapter needs an entropy source. ### 4.3 Memory and time `WriteMetadata` grows from 8 to 48 bytes (8 stamp + 40 trace: 16 trace ID, -8 span, 8 parent, 1 depth, padded to 8). Every buffer slot pays it: +8 span, 8 parent, 1 depth, padded to 8) with `trace-context` enabled. Every +buffer slot then pays it: | `T` | `Stamped` (058) | `Stamped` (060) | Added per slot | |---|---|---|---| @@ -323,7 +370,7 @@ per write or at `build()`, so no runtime adapter needs an entropy source. A 16-slot `f32` ring on Embassy goes from 256 to 896 bytes. 32-bit write identities would not save space: with the 8-byte stamp, the struct pads to 48 bytes either way once a parent carries its instance. With -`trace-context` off (§2.6), nothing changes. The 058 `size_of` assertions +`trace-context` off, the default (§2.6), nothing changes. The 058 `size_of` assertions are updated; the allocation gates must show no added allocation. The sizes are compile-time assertions in `aimdb-core` (48 bytes with @@ -343,8 +390,8 @@ origins and continued writes alike, since every write now draws a span. No allocation is added on the produce, consume and connector paths (`aimdb-bench` B0 suites), nor on AimX subscription events. `record.get` gains one allocation per call (§5). The Embassy benchmark -(`embassy-bench-stm32h5`) runs in phase 1 to document the MCU cost; the 48 -bytes are accepted, so it does not reopen the decision. +(`embassy-bench-stm32h5`) runs in phase 1, with the feature enabled, to +document the MCU cost for applications that opt in. ## 5. Wire @@ -529,6 +576,12 @@ span equals its `parent`. That holds in the diamond case (`Raw → A`, `Raw → B`, `A + B → C`): each write of `C` names the write of `A` or `B` that triggered it. +**Walking parents.** A consumer that follows parent links stops when the +next write's depth is not lower than the current one's (below 255), or when +it meets a span it has already visited. Span IDs can repeat across a restart +(§4.2), so a chain is not guaranteed to be free of cycles; with this rule a +walk always ends. + When the parent was not captured (a record without a Rerun tap, a tap that skipped values, or an instance without one), Rerun still knows that the parent exists and its span, and the graph names the records it can be on. @@ -565,6 +618,7 @@ layout) is a separate design. | WebSocket connector | its public `Snapshot` type gains the trace; its fan-out carries it (§5.2) | | Serial connector | tracing off by default; `.trace_default(bool)` on the server (§5.0) | | `AimDbBuilder` | `instance_tag(NonZeroU32)`; with `trace-context` enabled, `build()` fails without it (§4.2) | +| Cargo features | `trace-context` on `aimdb-core`, off by default; every crate that depends on core forwards it (§2.6) | | `InboundDispatch` | `dispatch` takes `Option`; no untraced variant (§5.2) | | AimX handshake | `hello` and `welcome` gain an optional `instance_tag` (§4.2.1) and `trace` (§5.0) | | MQTT, Tokio backend | MQTT 5 instead of 3.1.1; 3.1.1-only brokers are no longer supported (§5.3) | @@ -575,9 +629,9 @@ layout) is a separate design. In the prototype, the user code that had to change was the join handlers (three weather-mesh stations, the Tokio and WASM join tests: `produce` → `produce_from(…, trigger.metadata())`) and the codegen join stub -(`Producer` → `DerivedProducer`). The required instance tag adds one -line to every application builder; the prototype derived tags from a seed -and did not measure that. +(`Producer` → `DerivedProducer`). An application that enables +`trace-context` also adds one `instance_tag(…)` line to its builder; the +prototype derived tags from a seed and did not measure that. ### 8.1 Release @@ -595,8 +649,9 @@ The CHANGELOG lists every row of the table above under "Breaking changes". 1. **Core:** envelope with span and parent, IDs and the required instance tag, automatic propagation in `map` / `with_state`, the single-input `on_values` body, - `DerivedProducer` in joins, `Producer::produce_from`, the `trace-context` - feature and its listing in the Embassy examples (§2.6), tests on Tokio, + `DerivedProducer` in joins, `Producer::produce_from`, the epoch from + `unix_time()`, the `trace-context` feature, off by default, and its + forwarding feature in every dependent crate (§2.6), tests on Tokio, Embassy and WASM. Enough for Rerun taps on one instance. 2. **AimX:** `trace` field on values and on mirror `write` frames, session engine (WebSocket fan-out included), client, CLI, MCP; Serial off by @@ -618,8 +673,18 @@ The CHANGELOG lists every row of the table above under "Breaking changes". - **Tag required:** with `trace-context` enabled, `build()` without `instance_tag` fails; with it off, it succeeds, and `instance_tag(…)` is accepted and ignored. The `test_support` builders build without one. -- **Origin trace ID:** an origin write with tag 7 and counter 5 has trace ID - `00000007 00000000 0000000000000005`. +- **Origin trace ID:** without a wall clock, an origin write with tag 7 and + counter 5 has trace ID `00000007 00000000 0000000000000005`; with + `unix_time()` at 1 700 000 000 s it has `00000007 6553f100 0000000000000005`. +- **Epoch:** it is read once: a later clock change does not change the trace + IDs of the run. On a clock that gains wall time after three origin writes, + those three have epoch 0 and the following ones the stored epoch; all + trace IDs of the run are distinct. +- **Restart with a wall clock:** an instance with tag 7 writes, is rebuilt + with a clock one second later and writes again; no trace ID of the second + run equals one of the first. Without a wall clock the IDs repeat. +- **Parent walk ends:** on a recording with a repeated span that closes a + parent cycle, a walk stops at the repeated span (§6). - **Retained message after reboot:** a device with tag 7 publishes a retained MQTT value with span `7‖1`, restarts, and ingests the retained value as its first write. It reports `OwnTagInbound` and stores span `7‖2` @@ -639,9 +704,11 @@ The CHANGELOG lists every row of the table above under "Breaking changes". - **`on_values` after lag:** a burst larger than the input ring makes the receiver lag; it reports `Lagged`, continues, and a later overheat still emits an alarm. `recv()` returns `None` only when the input closes. -- Every locally written value has `trace()` = `Some`. -- The same tag and the same writes give identical IDs across runs. -- `trace-context` off: `size_of::() == 8`, `trace()` is +- With `trace-context` enabled, every locally written value has `trace()` = + `Some`. +- The same tag, the same writes and a clock without wall time (or a fixed + one) give identical IDs across runs. +- `trace-context` off (the default): `size_of::() == 8`, `trace()` is `None`, AimX frames carry no `trace`, and an Embassy build without `instance_tag` succeeds on `thumbv7em-none-eabihf`. - A malformed `trace` field on a value or `write` frame (including a tag-zero @@ -681,8 +748,11 @@ The CHANGELOG lists every row of the table above under "Breaking changes". - `size_of::() == 48` (8 with `trace-context` off), asserted at compile time on every target; allocation gates unchanged, including `record.get`. -- **Feature reach:** the Embassy examples build with `trace-context` on; a - benchmark that measures tracing fails if it builds without it. +- **Feature reach:** a default build of every crate has tracing off + (`size_of::() == 8`). Enabling `trace-context` on any one + adapter, connector or `aimdb-client` turns it on in core. An Embassy + example builds with it on, and a benchmark that measures tracing fails if + it builds without it. - **Counter on Cortex-M:** writing a record from an interrupt handler while the main context writes does not deadlock. - **MQTT oversize (Tokio):** a publish over the broker's maximum packet size @@ -736,9 +806,20 @@ The CHANGELOG lists every row of the table above under "Breaking changes". 14. **Uniqueness across restarts** with a persisted counter high-water mark (`counter_start`) or a distinct tag per boot. Needs persistent storage or provisioning on every device, and only covers the restarts the - application handles correctly. Rejected: AimDB instances are - long-running and restarts are rare; repeats across a restart are - accepted, and only `span == parent` is prevented (§4.2). + application handles correctly. Rejected: the trace ID takes an epoch + from the wall clock the instance already has (§4.2), which needs neither. + Spans have no room for it, so span repeats across a restart are + accepted; `span == parent` is prevented and parent walks are bounded + (§6). +15. **Tracing on by default.** Every application would pay 40 bytes per + buffer slot and need an instance tag without asking for either, and a + feature that makes `build()` fail without new configuration must not be + reachable through a dependency's defaults. Rejected: off by default, + forwarded by every dependent crate (§2.6). +16. **A timestamp in every span or on every write.** A span would need more + than 64 bits to hold tag, time and counter, which grows the envelope and + leaves the W3C span size; reading the wall clock per write costs time on + the hot path. Rejected: one epoch per run, in the trace ID only. ## 12. Decisions @@ -752,9 +833,10 @@ All earlier open questions are settled: | Single-input async transform | ships now, as `on_values` (§2.2) | | Parent identity | exact, across instances: 64-bit span ID = instance tag ‖ counter (§4.2) | | Instance tag | assigned by the application, required with `trace-context`; no seed, no entropy source | -| Restarts | IDs may repeat across a restart; only `span == parent` is prevented | +| Restarts | trace IDs carry a wall-clock epoch and do not repeat across a restart when the instance has a wall clock; span IDs may repeat; `span == parent` is prevented and parent walks are bounded | +| Default | `trace-context` is off by default; every crate that depends on core forwards the feature (§2.6) | | Release | `aimdb-core` 4.0.0; AimX 4.2 | -| Envelope size | 48 bytes accepted everywhere; `trace-context` off is the opt-out for small MCUs | +| Envelope size | 48 bytes for applications that enable `trace-context`; 8 bytes otherwise | No open questions remain. From e07866d0068ee7977248315a9e6cbc6ea3f71e78 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 14:42:55 +0000 Subject: [PATCH 2/3] docs(design): 060 resolve review points - joins: one parent per write, stated as a limit; handlers take ctx - single-input async body renamed on_stream, same shape as on_triggers - WebSocket duplicate-tag check through the upgrade URL, server side - Zenoh: zenoh:// carries the trace as the attachment, ros2:// does not - record.set accepts an optional trace, so a client can name a cause - carrier ignores trailing fields; AimX minor depends on 059's order - all three phases ship in one 4.0.0 release - tags for unprovisioned instances, OwnTagInbound logging, sync API, portable-atomic reuse, record.get wording Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WGkDJ8UNN7mWpJEEj5zxzX --- docs/design/060-trace-context.md | 221 +++++++++++++++++++++++-------- 1 file changed, 166 insertions(+), 55 deletions(-) diff --git a/docs/design/060-trace-context.md b/docs/design/060-trace-context.md index e5fe231d..a187a7bf 100644 --- a/docs/design/060-trace-context.md +++ b/docs/design/060-trace-context.md @@ -7,7 +7,7 @@ trace ID, the write's own span ID, its parent's span ID, and a depth. It is carried in `WriteMetadata` (058) and propagated along derivations and across connectors that can carry it. Propagation goes through the existing transform, join and producer APIs, plus one new transform body for -single-input async derivations. Rerun taps get exact causal chains, and +single-input async derivations. Rerun taps get exact parent chains, and agents can follow them through the writes they captured or can still read (§6). Breaking for Rust APIs (join handlers, `WriteMetadata`, session engine, `AimDbBuilder`); additive on the wire (the graph shapes do not @@ -67,8 +67,8 @@ through a `DerivedProducer`: builder.configure::(AlarmKey::Overheat, |reg| { reg.buffer(BufferCfg::SpmcRing { capacity: 16 }) .transform::(TempKey::Calibrated, |t| { - t.on_values(|ctx, mut rx, producer| async move { - while let Some(temp) = rx.recv().await { // Stamped + t.on_stream(|ctx, mut rx, producer| async move { + while let Ok(temp) = rx.recv().await { // Stamped if temp.value.celsius > 80.0 { producer.produce_from(Alarm::Overheat, temp.metadata()); } @@ -84,21 +84,28 @@ builder.configure::(AlarmKey::Overheat, |reg| { - In the graph it is the existing `RecordOrigin::Transform { input }`, so the graph shapes on the wire do not change. - The receiver is typed: no boxing, no downcast. -- `rx.recv()` returns `Some(Stamped)` for each value and `None` only when - the input closes. A lagging receiver skips the values it missed, reports +- The name is `on_stream`, not `on_values`: `with_state(…).on_value(…)` + already exists as the synchronous per-value closure, and one letter is too + small a difference for a body that runs its own loop. +- The handler has the same shape as a join's (§2.3): `|ctx, rx, producer|`, + and `rx.recv()` returns `Ok(Stamped)` for each value and `Err` only when + the input closes, as `JoinEventRx::recv` does. A lagging receiver skips the values it missed, reports them (`TransformEvent::Lagged`, as `map` and `with_state` do) and continues, so a burst cannot stop the transform. ### 2.3 Joins `on_triggers` handlers receive a `DerivedProducer` instead of a -`Producer`. `JoinTrigger` already carries the input's `WriteMetadata`: +`Producer`, and the runtime context as a new first argument, so that a +join can use timers (the case `produce_origin` exists for) and has the same +shape as `on_stream`. `JoinTrigger` already carries the input's +`WriteMetadata`: ```rust reg.transform_join(|b| b .input::(TempKey::Gamma) .input::(HumidityKey::Gamma) - .on_triggers(|mut rx, producer| async move { + .on_triggers(|ctx, mut rx, producer| async move { while let Ok(trigger) = rx.recv().await { // … update state … producer.produce_from(dew_point(&t, &h), trigger.metadata()); @@ -116,6 +123,14 @@ impl DerivedProducer { `DerivedProducer` has no plain `produce`, so a join or an async transform cannot forget its cause. +**A write has one parent.** A join output names the write that triggered +it. The other inputs' values, which the handler kept as state, are not +recorded: in the example above, a dew point triggered by a temperature +continues the temperature's trace and does not mention the humidity write +it was computed with. When the inputs come from different origins, the +other input's trace ends at the join. Recording several causes per write +(OTEL "links") is not part of this design (§6). + ### 2.4 Producers `Producer` keeps `produce`, which starts a trace, and gains @@ -124,6 +139,10 @@ example a task that bridges an external system. Such writes have an exact parent, but the graph has no edge for them (the record stays `Passive`). Derivations from another record should use a transform (§2.2). +`aimdb-sync` is unchanged: `SyncProducer::set` and `set_value` start a +trace, like `produce`. Synchronous code sits outside the graph and has no +cause to pass; a sync `produce_from` can follow if a user needs it. + ### 2.5 Reading ```rust @@ -231,12 +250,13 @@ continues `c` with `parent = c.span`. Only | `map`, `with_state` | the input | | `Producer::produce_from`, `DerivedProducer::produce_from` | the given metadata | | inbound connector with a trace on the wire | the wire trace | -| AimX `write` from a peer's outbound mirror, with a trace | the wire trace (§5.1) | -| inbound connector or AimX `write` without one, AimX `record.set`, persistence restore | `None` | +| AimX `write` or `record.set` with a trace | the wire trace (§5.1) | +| inbound connector, AimX `write` or `record.set` without one, persistence restore | `None` | An inbound value without a trace therefore gets a new ID when, and only -when, it is inserted into a local record. Every AimX `record.set` starts a -trace: a remote client cannot continue one. +when, it is inserted into a local record. A peer that writes can name a +cause on either path, a mirror `write` or a `record.set`; the trace is +advisory and is not authenticated on either. ### 4.2 IDs @@ -283,6 +303,16 @@ read in a trace, and stays the same across restarts. Tags are `NonZero`, so a span is never all zero, including at the counter wrap. The `test_support` builders set a fixed tag, so tests that do not care about it need no change. +**Instances without provisioning.** Browser tabs running the WASM adapter, +scaled replicas and short-lived tools have no table to take a tag from. The +application still supplies the tag, and may draw it at random when the +instance starts (`crypto.getRandomValues` in a browser, the OS RNG on a +host). That is the application's choice and the application's risk: random +tags are distinct only with high probability (100 instances collide with +probability about 1 in 860,000), and §4.2.1 reports a collision when it can +see one. The framework never draws a tag itself, so core and the adapters +need no entropy source. + **Guarantee: spans.** Within one trace, two writes have different span IDs if @@ -331,24 +361,38 @@ Two instances with the same tag break condition 1 silently in general. The framework reports what it can see, without changing any write: - **An inbound span carries the instance's own tag** - (`TraceEvent::OwnTagInbound`, counted with `observability`). It means - either another instance uses the same tag, or the instance's own write - from before a restart has come back (a retained message after a reboot). - It is a diagnostic, not an error: the second cause is normal operation. + (`TraceEvent::OwnTagInbound`, counted with `observability`). It means one + of three things: another instance uses the same tag; the instance's own + write from before a restart has come back (a retained message after a + reboot); or the instance receives its own publishes (it subscribes to a + topic it also publishes on). It is a diagnostic, not an error: the last + two are normal operation. Every occurrence is counted; it is logged once + per inbound route per connection, so an echoing topic does not fill the + log. - **An AimX peer announces the same tag** (`TraceEvent::DuplicateTag`). `hello` and `welcome` gain an optional `instance_tag` field, sent by instances with `trace-context` enabled (a CLI or MCP client sends none). Whichever side sees its own tag reports it and keeps the connection: the - trace is advisory and never costs a link. The WebSocket server does not - read `hello`, so over WebSocket only the client side checks. + trace is advisory and never costs a link. +- **Over WebSocket there is no `hello` and no `welcome`** (the client runs + `sends_hello: false`, the server `reads_hello: false`), so the tag travels + in the upgrade URL, next to the version: `…/ws?v=4.2&tag=0000a1f3`. + `ws_url_with_version`, which every first-party dialer already uses, adds + it when `trace-context` is enabled. Only the server checks, at the + upgrade; it reports `DuplicateTag` and accepts the connection. A browser + cannot read the upgrade response, so the client side learns of a + duplicate only through `OwnTagInbound`. MQTT and Zenoh peers announce nothing, so a duplicate tag behind a broker shows up only as `OwnTagInbound`. ### 4.2.2 Counter -The counter is a 64-bit atomic where the target has one, and a -`critical-section` lock elsewhere (Cortex-M has no native 64-bit atomics). +The counter is a `portable_atomic::AtomicU64`: a native 64-bit atomic where +the target has one, and a `critical-section` lock elsewhere (Cortex-M has +no native 64-bit atomics). `trace-context` enables `portable-atomic`'s +`fallback` and `critical-section` features, exactly as the `observability` +feature already does, so no new dependency or lock type is needed. It must be a critical section, not a spin lock: a record written from an interrupt handler while the main context holds a spin lock would deadlock a single-core MCU. (The prototype used a spin lock; it is not acceptable for @@ -388,8 +432,9 @@ origins and continued writes alike, since every write now draws a span. | `produce` + `try_recv` | 112.8 ns | 125.2 ns | No allocation is added on the produce, consume and connector paths -(`aimdb-bench` B0 suites), nor on AimX subscription events. `record.get` -gains one allocation per call (§5). The Embassy benchmark +(`aimdb-bench` B0 suites), nor on AimX subscription events. In the +prototype `record.get` gained one allocation per call; the implementation +must not (§5). The Embassy benchmark (`embassy-bench-stm32h5`) runs in phase 1, with the feature enabled, to document the MCU cost for applications that opt in. @@ -405,7 +450,8 @@ One carrier value everywhere: | AimX over TCP, UDS, WebSocket | optional `trace` field next to `ts` on every value | | AimX over Serial | off by default; a client turns it on for its session in `hello` (§5.0), the server's configuration can change the default | | MQTT v5 (both backends, §5.3) | user property `aimdb-trace` | -| Zenoh | attachment key `aimdb-trace` | +| Zenoh, `zenoh://` | the publication's attachment (§5.4) | +| Zenoh, `ros2://` | not carried (§5.4) | | KNX | not carried | ```json @@ -413,9 +459,15 @@ One carrier value everywhere: "trace":"5a07b3f1000000000000000000000002/3/9c41e2d000000a1f/5a07b3f100000004","data":{"celsius":21.5}} ``` +- **The carrier can grow.** A decoder reads the first four `/`-separated + fields and ignores any that follow, so a later version can append fields + (flags, for example) without older peers treating the value as malformed. - The field is optional: an older peer or a link with tracing off sends - none. AimX decoders ignore unknown fields, so this is the next minor - version after 059's 4.1. Nothing else on the wire changes. + none. AimX decoders ignore unknown fields, so this is a minor version. + `PROTOCOL_VERSION` is 4.0 today and 059, which claims 4.1, is not merged: + 060 takes the next free minor when it lands, 4.2 after 059 or 4.1 before + it (059 then takes 4.2). This document writes 4.2. Nothing else on the + wire changes. - Absence means different things on the two paths, with the same frame: - **observing** (client `get`, `watch`, `drain`): the value has no trace, `trace()` is `None`; @@ -493,8 +545,12 @@ receiving dispatch passes only the value to `set_record_from_json`. through to the record write, which continues it or, without one, starts a trace. -A user-issued `record.set` is not a mirror write and always starts a trace -(§4.1). +`record.set` takes the same optional field in its parameters: +`{"name":…,"value":…,"trace":"…"}`. A client that sets a record because of +a value it read passes that value's trace and the write continues it; +without the field the write starts a trace. This is what lets an agent's +action stay in the chain that prompted it (§7). A session with tracing off +(§5.0) ignores the field on both frames. ### 5.2 Session engine @@ -563,6 +619,24 @@ Both MQTT backends speak MQTT 5 and carry the trace as one user property. connector reports it as an MQTT version error naming the broker. Mosquitto 1.6+, EMQX, HiveMQ and AWS IoT Core speak MQTT 5. +### 5.4 Zenoh + +Not prototyped; phase 3 starts with a prototype of this section. + +- **`zenoh://`:** a Zenoh 1.x publication has one attachment, an opaque + byte string with no keys. The connector sets it to the carrier value as + ASCII bytes (68 to 70 bytes). Inbound, an attachment that does not parse + as a carrier is treated as absent, like any malformed trace (§5): the + value is kept. Publications from non-AimDB peers that use the attachment + for something else are therefore ingested untraced. +- **`ros2://`:** not carried. The attachment belongs to rmw_zenoh, which + requires its fixed 33-byte layout (sequence number, source timestamp, + GID) on every publication, and ROS 2 peers reject any other. A value + ingested from ROS 2 starts a trace; a value published to ROS 2 ends its + trace there. +- **Embedded backend:** the `zenoh-nostd` backend does not exist yet; it + carries the trace the same way once it does. + ## 6. Rerun taps and agents A Rerun tap is a read-only `tap` on a Tokio instance. For each value it logs @@ -571,10 +645,11 @@ that instance's own timeline. Each instance also logs its dependency graph once. MCUs run no Rerun taps; their values show up at the hub with their trace intact. -**Parents are exact.** A write's parent is the write in the same trace whose -span equals its `parent`. That holds in the diamond case (`Raw → A`, +**Parents are exact, and there is one per write.** A write's parent is the +write in the same trace whose span equals its `parent`. That holds in the diamond case (`Raw → A`, `Raw → B`, `A + B → C`): each write of `C` names the write of `A` or `B` -that triggered it. +that triggered it. It does not name the other one: a chain shows what +triggered each write, not every value that went into it (§2.3). **Walking parents.** A consumer that follows parent links stops when the next write's depth is not lower than the current one's (below 255), or when @@ -607,12 +682,15 @@ layout) is a separate design. trace ID, span, parent and depth, or `no trace` for a value the peer sent without one. - MCP `get_record` includes the trace, or `null`. +- `aimdb set … --cause ` and MCP `set_record` with an + optional `cause` pass a trace to `record.set` (§5.1), so a write made in + response to a value continues that value's trace. ## 8. What breaks | Area | Change | |---|---| -| Join handlers | `on_triggers` receives `DerivedProducer` instead of `Producer` | +| Join handlers | `on_triggers` receives `(ctx, rx, DerivedProducer)` instead of `(rx, Producer)` | | `WriteMetadata` | 48 bytes; `trace() -> Option`; `from_wire_stamp` becomes `from_wire(stamp, Option)` | | Session engine | new fields and parameters (§5.2); every `Session` and `EnvelopeCodec` implementation updates | | WebSocket connector | its public `Snapshot` type gains the trace; its fan-out carries it (§5.2) | @@ -620,16 +698,19 @@ layout) is a separate design. | `AimDbBuilder` | `instance_tag(NonZeroU32)`; with `trace-context` enabled, `build()` fails without it (§4.2) | | Cargo features | `trace-context` on `aimdb-core`, off by default; every crate that depends on core forwards it (§2.6) | | `InboundDispatch` | `dispatch` takes `Option`; no untraced variant (§5.2) | -| AimX handshake | `hello` and `welcome` gain an optional `instance_tag` (§4.2.1) and `trace` (§5.0) | +| AimX handshake | `hello` and `welcome` gain an optional `instance_tag` (§4.2.1) and `trace` (§5.0); the WebSocket upgrade URL gains an optional `tag` parameter | +| AimX `record.set` | optional `trace` parameter (§5.1) | +| Zenoh connector | `zenoh://` publications carry the trace as their attachment (§5.4) | | MQTT, Tokio backend | MQTT 5 instead of 3.1.1; 3.1.1-only brokers are no longer supported (§5.3) | -| `TransformBuilder`, `Producer`, `TopicMatch` | additive: `on_values`, `produce_from`, `trace()` | +| `TransformBuilder`, `Producer`, `TopicMatch` | additive: `on_stream`, `produce_from`, `trace()` | | `aimdb-client` | `RecordUpdate` gains `trace` | | AimX | optional `trace` field, minor version; graph shapes unchanged | In the prototype, the user code that had to change was the join handlers (three weather-mesh stations, the Tokio and WASM join tests: `produce` → `produce_from(…, trigger.metadata())`) and the codegen join stub -(`Producer` → `DerivedProducer`). An application that enables +(`Producer` → `DerivedProducer`). The `ctx` argument adds one name to +each of those closures. An application that enables `trace-context` also adds one `instance_tag(…)` line to its builder; the prototype derived tags from a seed and did not measure that. @@ -640,30 +721,39 @@ prototype derived tags from a seed and did not measure that. - `aimdb-core`: 4.0.0. The adapter and connector crates take their next major version with it. - `aimdb-client`: 0.9.0 (a breaking bump for a 0.x crate). -- AimX protocol: 4.2, the next minor after 059's 4.1. The wire change is - additive; the protocol version is independent of the crate version. +- AimX protocol: the next free minor, 4.2 if 059 has landed (§5). The wire + change is additive; the protocol version is independent of the crate + version. +- **One release for all three phases.** The phases of §9 are a merge order, + not releases: the session engine and `InboundDispatch` changes land in + phase 2 and the MQTT change in phase 3, and all of them are breaking. 4.0.0 + is published only after phase 3 is merged, so no second major release is + needed. The CHANGELOG lists every row of the table above under "Breaking changes". ## 9. Phases +The phases are merged in this order and released together (§8.1). + 1. **Core:** envelope with span and parent, IDs and the required instance tag, automatic - propagation in `map` / `with_state`, the single-input `on_values` body, + propagation in `map` / `with_state`, the single-input `on_stream` body, `DerivedProducer` in joins, `Producer::produce_from`, the epoch from `unix_time()`, the `trace-context` feature, off by default, and its forwarding feature in every dependent crate (§2.6), tests on Tokio, Embassy and WASM. Enough for Rerun taps on one instance. 2. **AimX:** `trace` field on values and on mirror `write` frames, session engine (WebSocket fan-out included), client, CLI, MCP; Serial off by - default with the per-session request in `hello` (§5.0); the `record.get` reply without an added allocation; tags in the - handshake. Enough for + default with the per-session request in `hello` (§5.0); the `record.get` + reply without an added allocation; `record.set` with a cause; tags in the + handshake and in the WebSocket upgrade URL. Enough for chains across AimX-linked instances in both directions. 3. **MQTT v5** on both backends, with the Tokio oversize check, and - **Zenoh**. + **Zenoh**, starting with a prototype of §5.4. ## 10. Tests -- Source → `map` → `with_state` → `on_values` → join keeps one trace ID; depth +- Source → `map` → `with_state` → `on_stream` → join keeps one trace ID; depth grows by one per write; each write's `parent` is its cause's `span`. - Every write gets a new span; within one run and the §4.2 bound (2³² writes) spans in one instance do not repeat, and they are never zero. @@ -695,15 +785,18 @@ The CHANGELOG lists every row of the table above under "Breaking changes". tag and aligned counters store no `span == parent` and report `OwnTagInbound`. - **Duplicate tag over AimX:** two instances with the same tag connect; both - sides (client side only over WebSocket) report `DuplicateTag`, and values - still flow. A CLI client without a tag triggers nothing. + sides report `DuplicateTag`, and values still flow. Over WebSocket the + client's upgrade URL carries `tag`, and only the server reports it. A CLI + client without a tag triggers nothing. +- **Own publishes:** an instance subscribed to a topic it publishes on + counts `OwnTagInbound` for every message and logs it once. - `produce_origin` and `Producer::produce` start new traces with no parent; `Producer::produce_from` continues one. -- `on_values`: typed receiver; zero, one and several writes per input; the graph +- `on_stream`: typed receiver; zero, one and several writes per input; the graph shows `Transform { input }`. -- **`on_values` after lag:** a burst larger than the input ring makes the +- **`on_stream` after lag:** a burst larger than the input ring makes the receiver lag; it reports `Lagged`, continues, and a later overheat still - emits an alarm. `recv()` returns `None` only when the input closes. + emits an alarm. `recv()` returns `Err` only when the input closes. - With `trace-context` enabled, every locally written value has `trace()` = `Some`. - The same tag, the same writes and a clock without wall time (or a fixed @@ -728,14 +821,24 @@ The CHANGELOG lists every row of the table above under "Breaking changes". `"trace":true`; the next client's `hello` without the field is untraced again. Over UDS, `"trace":false` turns tracing off for that session only. A server built with `trace-context` off answers `"trace":false`. -- `record.set` always starts a trace. +- `record.set` without `trace` starts a trace; with one, the write continues + it with the given span as parent; a malformed one is treated as absent. +- **Carrier growth:** a carrier with a fifth field decodes to the same trace + as the first four alone. +- **Join, one parent:** a dew point triggered by a temperature write has the + temperature's trace ID and span as parent, and nothing of the humidity + write. +- **Zenoh:** a `zenoh://` publication carries the carrier as its attachment + and the receiver continues the trace; a foreign attachment is ingested + untraced; a `ros2://` publication keeps rmw_zenoh's attachment and the + receiver starts a trace. - **MQTT v5:** a publish carries `aimdb-trace`; a receiving instance continues the trace; a publish without the property starts one. A publish that fits the broker's maximum packet size without the property but not with it is delivered untraced and counted as `trace_omitted`; one that does not fit without the property is rejected as oversize. - `graph.nodes` and `graph.edges` replies are byte-identical to AimX 4.1 for a - fixture with a transform, an `on_values` transform and a join. + fixture with a transform, an `on_stream` transform and a join. - A value that is migrated on the inbound path keeps its trace. - **Exact parents:** in the diamond `Raw → A, Raw → B, A + B → C`, each write of `C` names the span of the write of `A` or `B` that triggered it; with @@ -763,7 +866,7 @@ The CHANGELOG lists every row of the table above under "Breaking changes". 1. **Merge `transform` and `transform_join` into one builder.** Changes the graph shapes on the wire and every transform call, and makes a single-input caller downcast its own declared type through the boxed join - path. Rejected: a typed `on_values` body on the existing single-input builder + path. Rejected: a typed `on_stream` body on the existing single-input builder (§2.2). 2. **Parent candidates from the graph instead of stored parents.** Costs nothing per slot, but is ambiguous in diamonds, which some pipelines @@ -829,16 +932,21 @@ All earlier open questions are settled: |---|---| | Tokio MQTT backend | moves to MQTT 5 and requires it; no 3.1.1 mode | | Serial links | trace off by default; a client turns it on per session in `hello`, the server's configuration changes the default (§5.0) | -| `record.set` | every remote set starts a trace | -| Single-input async transform | ships now, as `on_values` (§2.2) | +| `record.set` | continues a trace when the caller passes one, otherwise starts one (§5.1) | +| Single-input async transform | ships now, as `on_stream`, with the join handler's shape (§2.2) | +| Causes per write | one: the triggering write; a join's other inputs are not recorded (§2.3) | +| Zenoh | `zenoh://` carries the trace as the attachment; `ros2://` does not (§5.4) | +| WebSocket tag check | tag in the upgrade URL, checked by the server only (§4.2.1) | | Parent identity | exact, across instances: 64-bit span ID = instance tag ‖ counter (§4.2) | | Instance tag | assigned by the application, required with `trace-context`; no seed, no entropy source | | Restarts | trace IDs carry a wall-clock epoch and do not repeat across a restart when the instance has a wall clock; span IDs may repeat; `span == parent` is prevented and parent walks are bounded | | Default | `trace-context` is off by default; every crate that depends on core forwards the feature (§2.6) | -| Release | `aimdb-core` 4.0.0; AimX 4.2 | +| Release | `aimdb-core` 4.0.0, published once after all three phases; AimX takes the next free minor (4.2 after 059) | | Envelope size | 48 bytes for applications that enable `trace-context`; 8 bytes otherwise | -No open questions remain. +No open questions remain. Two parts are decided but not prototyped, and +their phases start by checking them: the tag exchange in the handshake and +the upgrade URL (§4.2.1), and the Zenoh attachment (§5.4). ## 13. Prototype results @@ -846,11 +954,14 @@ The second throwaway prototype implemented §2 to §5 except Zenoh and the CLI/MCP display. It was not merged. It derived tags from a per-boot seed and had `counter_start` for restarts; both have since been replaced (§4.2, alternatives 12 and 14), and the rows below for them are kept as history. -The handshake tag exchange (§4.2.1) was not prototyped. +The handshake tag exchange (§4.2.1) was not prototyped, nor were the +wall-clock epoch (§4.2), `record.set` with a cause (§5.1) and the `ctx` +argument of join handlers (§2.3); the prototype's single-input body was +called `on_values` and returned `Option` from `recv`. | Claim | Result | |---|---| -| Propagation through `map`, `with_state`, `on_values` and joins; parent = cause's span | holds | +| Propagation through `map`, `with_state`, `on_stream` and joins; parent = cause's span | holds | | Exact parents in the diamond | holds, over 20 diamonds | | Cross-instance parents over AimX (UDS), both directions | holds | | Cross-instance parents over MQTT 5, both backends, real broker | holds | @@ -858,7 +969,7 @@ The handshake tag exchange (§4.2.1) was not prototyped. | Same tag, aligned counters | collision reported, no self-parent | | Retained value after reboot | collision reported, no self-parent | | Counter wrap, tag zero | no zero span | -| `on_values` after lag | skips, reports, a later value still produces | +| `on_stream` after lag | skips, reports, a later value still produces | | Tracing off without an ID source | builds; no_std suites pass with tracing on and off | | 48 / 8 bytes | asserted on host, Cortex-M, wasm32 | | Wire cost | AimX +79 bytes, MQTT +84 bytes | From 3ef53bf3ab99635e44f91804aca80f3873753ba1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 15:03:29 +0000 Subject: [PATCH 3/3] docs(design): 060 spike results A throwaway spike tested the opt-in feature, the epoch, on_stream, ctx in join handlers, record.set with a cause, handshake tags and the Zenoh attachment. Findings written into the design: - a fallback tag in test builds: the mandatory tag failed 41 core, 52 Tokio adapter and 11 Zenoh tests with tracing on - the ID source is compiled only with the feature (Cortex-M build) - parent walk checks trace ID and exact depth; a cycle forms after three derived writes - record.set: cause through JsonRecordAccess, trace in the reply - tag text form; ws_url_with_version takes the tag - second cost measurement, Zenoh attachment result Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WGkDJ8UNN7mWpJEEj5zxzX --- docs/design/060-trace-context.md | 157 ++++++++++++++++++++++++++----- 1 file changed, 132 insertions(+), 25 deletions(-) diff --git a/docs/design/060-trace-context.md b/docs/design/060-trace-context.md index a187a7bf..67d3ee23 100644 --- a/docs/design/060-trace-context.md +++ b/docs/design/060-trace-context.md @@ -14,9 +14,12 @@ engine, `AimDbBuilder`); additive on the wire (the graph shapes do not change). Ships as `aimdb-core` 4.0 (§8.1). Tracing is off by default and enabled with the `trace-context` feature (§2.6). -Two throwaway prototypes tested this design. The second implements all of -it except Zenoh, the CLI/MCP display and Rerun; §13 lists what it showed, and -its measurements are quoted in §4.3, §5 and §5.3. +Two throwaway prototypes and one spike tested this design. The second +prototype implements all of it except Zenoh, the CLI/MCP display and Rerun; +its measurements are quoted in §4.3, §5 and §5.3. The spike tested what +changed after it: the opt-in feature, the epoch, `on_stream`, `ctx` in join +handlers, `record.set` with a cause, the handshake tags and the Zenoh +attachment. §13 lists what both showed. --- @@ -84,6 +87,9 @@ builder.configure::(AlarmKey::Overheat, |reg| { - In the graph it is the existing `RecordOrigin::Transform { input }`, so the graph shapes on the wire do not change. - The receiver is typed: no boxing, no downcast. +- The handler must be `Send + Sync`, like `map` and `with_state` closures: + the single-input pipeline stores it in a `Send + Sync` factory. A join + handler only needs `Send`. - The name is `on_stream`, not `on_values`: `with_state(…).on_value(…)` already exists as the synchronous per-value closure, and one letter is too small a difference for a body that runs its own loop. @@ -300,8 +306,18 @@ With `trace-context` enabled, `build()` fails with a configuration error if no tag is set. The framework does not derive tags from randomness, the chip ID or the runtime: a tag the user chose is distinct by construction, can be read in a trace, and stays the same across restarts. Tags are `NonZero`, so -a span is never all zero, including at the counter wrap. The `test_support` -builders set a fixed tag, so tests that do not care about it need no change. +a span is never all zero, including at the counter wrap. + +**Tags in tests.** Most tests build through `AimDbBuilder::new()` directly, +not through a `test_support` builder. In the spike, with `trace-context` +enabled and the tag mandatory, 41 of 299 `aimdb-core` unit tests, 52 of 102 +Tokio adapter tests and all 11 Zenoh `native_session` tests failed at +`build()`, every one on the missing tag. So in test builds a missing tag +falls back to a fixed one: in core's own tests, and in any crate that +enables core's `test-utils` feature. With that fallback the same core (299) +and Tokio adapter (110) suites pass with tracing on. A crate that tests +with tracing on enables `test-utils` or sets a tag; the Zenoh suite, which +does neither, still failed. Outside test builds there is no fallback. **Instances without provisioning.** Browser tabs running the WASM adapter, scaled replicas and short-lived tools have no table to take a tag from. The @@ -343,9 +359,11 @@ responsibility; §4.2.1 lists what the framework detects when it is not. restart. A trace from before a restart can still reach the instance (a retained MQTT message, an AimX late-join snapshot, a peer's buffered mirror write) and be continued by new writes; such a trace may then - contain two writes with the same span, and in rare cases a parent chain - that returns to a span it already passed. Consumers that walk parents - must therefore bound the walk (§6). + contain two writes with the same span, and a parent chain that returns + to a span it already passed. This is not exotic: in the spike a retained + value with span `7‖3`, continued by three derived writes after the + restart, was enough, because the third of them is `7‖3` again. Consumers + that walk parents must therefore bound the walk (§6). One case is prevented in the writer, because it would make a write its own parent: @@ -372,14 +390,21 @@ framework reports what it can see, without changing any write: - **An AimX peer announces the same tag** (`TraceEvent::DuplicateTag`). `hello` and `welcome` gain an optional `instance_tag` field, sent by instances with `trace-context` enabled (a CLI or MCP client sends none). + The tag is written as 8 hex digits everywhere it appears in text + (`"instance_tag":"0000a1f3"`, `tag=0000a1f3`), the form it has inside a + span. Whichever side sees its own tag reports it and keeps the connection: the trace is advisory and never costs a link. - **Over WebSocket there is no `hello` and no `welcome`** (the client runs `sends_hello: false`, the server `reads_hello: false`), so the tag travels in the upgrade URL, next to the version: `…/ws?v=4.2&tag=0000a1f3`. - `ws_url_with_version`, which every first-party dialer already uses, adds - it when `trace-context` is enabled. Only the server checks, at the - upgrade; it reports `DuplicateTag` and accepts the connection. A browser + `ws_url_with_version` is a free function that knows no instance, so it + takes the tag as a new argument. The connector's `WsDialer` is built + where the database is at hand and passes its tag; the WASM bridge does + the same. The server's upgrade handler holds only a type-erased + `Dispatch`, so its `ServerState` gains the instance's tag. Only the server + checks, at the upgrade; it reports `DuplicateTag` and accepts the + connection. A browser cannot read the upgrade response, so the client side learns of a duplicate only through `OwnTagInbound`. @@ -392,7 +417,11 @@ The counter is a `portable_atomic::AtomicU64`: a native 64-bit atomic where the target has one, and a `critical-section` lock elsewhere (Cortex-M has no native 64-bit atomics). `trace-context` enables `portable-atomic`'s `fallback` and `critical-section` features, exactly as the `observability` -feature already does, so no new dependency or lock type is needed. +feature already does, so no new dependency or lock type is needed. The +counter and the rest of the ID source are compiled only with +`trace-context`: without the fallback, `portable_atomic::AtomicU64` does not +exist on `thumbv7em-none-eabihf`, and the spike's first feature-off build +for that target failed on it. It must be a critical section, not a spin lock: a record written from an interrupt handler while the main context holds a spin lock would deadlock a single-core MCU. (The prototype used a spin lock; it is not acceptable for @@ -431,6 +460,19 @@ origins and continued writes alike, since every write now draws a span. | `produce_from` | n/a | 115.0 ns | | `produce` + `try_recv` | 112.8 ns | 125.2 ns | +The spike measured more, with the epoch and a different harness (Tokio +adapter, 16-slot `f32` ring with one subscriber, 5 million writes, best of +7, one shared machine): 16 to 20 ns per write. + +| Operation | feature off | feature on | +|---|---|---| +| `produce` (origin) | 91.9 ns | 111.7 ns | +| `produce_from` | 90.4 ns | 107.7 ns | + +The two harnesses are not comparable to each other; `aimdb-bench` decides +the number in phase 1. The spike did not compare its feature-off build +with `main`. + No allocation is added on the produce, consume and connector paths (`aimdb-bench` B0 suites), nor on AimX subscription events. In the prototype `record.get` gained one allocation per call; the implementation @@ -549,7 +591,14 @@ receiving dispatch passes only the value to `set_record_from_json`. `{"name":…,"value":…,"trace":"…"}`. A client that sets a record because of a value it read passes that value's trace and the write continues it; without the field the write starts a trace. This is what lets an agent's -action stay in the chain that prompted it (§7). A session with tracing off +action stay in the chain that prompted it (§7). The reply carries the new +write's `trace` next to `ts`, so the caller can pass it on. + +A `record.set` reaches the buffer through `JsonRecordAccess::set_from_json`, +a public trait. The cause is a new parameter of that method, not a second, +defaulted method: the spike used a default that falls back to the untraced +write, and an implementation that does not override it loses the cause +without notice (the same reason as §5.2). A session with tracing off (§5.0) ignores the field on both frames. ### 5.2 Session engine @@ -621,7 +670,14 @@ Both MQTT backends speak MQTT 5 and carry the trace as one user property. ### 5.4 Zenoh -Not prototyped; phase 3 starts with a prototype of this section. +The outbound half was tested in the spike against an in-process Zenoh +router: the connector's publication arrived with a 68-byte attachment that +parsed to the write's trace. Of three publications from another session, the +one with a carrier parsed, and the one with no attachment and the one with +rmw_zenoh's 33-byte layout both read as untraced. Passing an inbound trace +on to the record write needs `dispatch` with a trace (§5.2) and was not +part of the spike. The spike allocates the attachment per publish; the +implementation has to meet the connector allocation gate. - **`zenoh://`:** a Zenoh 1.x publication has one attachment, an opaque byte string with no keys. The connector sets it to the carrier value as @@ -651,11 +707,14 @@ write in the same trace whose span equals its `parent`. That holds in the diamon that triggered it. It does not name the other one: a chain shows what triggered each write, not every value that went into it (§2.3). -**Walking parents.** A consumer that follows parent links stops when the -next write's depth is not lower than the current one's (below 255), or when -it meets a span it has already visited. Span IDs can repeat across a restart -(§4.2), so a chain is not guaranteed to be free of cycles; with this rule a -walk always ends. +**Walking parents.** A write found by its span is accepted as the parent +only if it has the same trace ID and its depth is exactly one lower (at +depth 255: 254 or 255), and only if it was not visited before. Otherwise +the walk stops and the hop is reported as not available. Span IDs can +repeat across a restart (§4.2), so a span lookup can return a different +write than the one meant, and a chain can close on itself; the trace ID and +depth check rejects the wrong write, and the visited check ends a cycle. In +the spike's three-write cycle the walk stopped after the three writes. When the parent was not captured (a record without a Rerun tap, a tap that skipped values, or an instance without one), Rerun still knows that the @@ -699,7 +758,9 @@ layout) is a separate design. | Cargo features | `trace-context` on `aimdb-core`, off by default; every crate that depends on core forwards it (§2.6) | | `InboundDispatch` | `dispatch` takes `Option`; no untraced variant (§5.2) | | AimX handshake | `hello` and `welcome` gain an optional `instance_tag` (§4.2.1) and `trace` (§5.0); the WebSocket upgrade URL gains an optional `tag` parameter | -| AimX `record.set` | optional `trace` parameter (§5.1) | +| AimX `record.set` | optional `trace` parameter; the reply carries the new write's `trace` (§5.1) | +| `JsonRecordAccess` | `set_from_json` takes the cause (§5.1) | +| `ws_url_with_version` | takes the instance tag (§4.2.1) | | Zenoh connector | `zenoh://` publications carry the trace as their attachment (§5.4) | | MQTT, Tokio backend | MQTT 5 instead of 3.1.1; 3.1.1-only brokers are no longer supported (§5.3) | | `TransformBuilder`, `Producer`, `TopicMatch` | additive: `on_stream`, `produce_from`, `trace()` | @@ -710,7 +771,9 @@ In the prototype, the user code that had to change was the join handlers (three weather-mesh stations, the Tokio and WASM join tests: `produce` → `produce_from(…, trigger.metadata())`) and the codegen join stub (`Producer` → `DerivedProducer`). The `ctx` argument adds one name to -each of those closures. An application that enables +each of those closures. In the spike the five join handlers of the Tokio +adapter's tests took 15 changed lines in three files, and the compiler +named every `produce` that had to become `produce_from`. An application that enables `trace-context` also adds one `instance_tag(…)` line to its builder; the prototype derived tags from a seed and did not measure that. @@ -762,7 +825,8 @@ The phases are merged in this order and released together (§8.1). is the zero span. - **Tag required:** with `trace-context` enabled, `build()` without `instance_tag` fails; with it off, it succeeds, and `instance_tag(…)` is - accepted and ignored. The `test_support` builders build without one. + accepted and ignored. In a test build (core's tests, or core's + `test-utils` feature) a builder without a tag gets the fixed fallback tag. - **Origin trace ID:** without a wall clock, an origin write with tag 7 and counter 5 has trace ID `00000007 00000000 0000000000000005`; with `unix_time()` at 1 700 000 000 s it has `00000007 6553f100 0000000000000005`. @@ -944,9 +1008,10 @@ All earlier open questions are settled: | Release | `aimdb-core` 4.0.0, published once after all three phases; AimX takes the next free minor (4.2 after 059) | | Envelope size | 48 bytes for applications that enable `trace-context`; 8 bytes otherwise | -No open questions remain. Two parts are decided but not prototyped, and -their phases start by checking them: the tag exchange in the handshake and -the upgrade URL (§4.2.1), and the Zenoh attachment (§5.4). +No open questions remain. Three parts are decided but not run, and their +phases start by checking them: the tag in the WebSocket upgrade URL +(§4.2.1), the per-session trace request (§5.0), and ingesting a trace from a +Zenoh attachment (§5.4). ## 13. Prototype results @@ -982,3 +1047,45 @@ Findings that changed this document: tracing's reach through the counter (§4.2.2); the `record.get` allocation (§5); the WebSocket fan-out and snapshot cache (§5.2); the Tokio MQTT oversize check (§5.3); the adapter and connector API changes (§8). + +### 13.1 Spike + +A third throwaway branch tested what changed after the second prototype. +It was written from scratch on `main` (the prototypes were not merged) and +covers core, the Tokio adapter and the Zenoh connector's outbound path. It +does not cover the AimX value field, mirror `write` frames, the session +engine or MQTT, which the second prototype covered, nor the per-session +trace request, the WebSocket upgrade URL, the CLI/MCP display, Rerun or an +on-target Embassy run. + +| Claim | Result | +|---|---| +| Off by default: same graph code builds and runs untraced, with or without a tag; `trace()` is `None` | holds | +| Forwarding: `trace-context` on the Tokio adapter alone, or on the Zenoh connector alone, turns it on in core | holds | +| 48 / 8 bytes, as compile-time assertions | hold on the host, `thumbv7em-none-eabihf` and `wasm32-unknown-unknown`; `Stamped` 56 / 16 | +| Feature off on Cortex-M | failed first: the ID source needs a 64-bit atomic; fixed by compiling it only with the feature (§4.2.2) | +| Origin IDs `tag ‖ epoch ‖ counter`; epoch stored once; a run that gains a clock keeps distinct IDs | holds | +| Restart with a later wall clock: no trace ID of the first run repeats; without a clock, or in the same second, they repeat | holds | +| Tokio clock: the epoch in a real trace ID is the current Unix time | holds | +| No wall clock: the same tag and writes give identical IDs in two runs | holds | +| Counter wrap; retained value after reboot; tags `0x10` and `0x13` | no zero span, no self-parent, distinct spans | +| Source → `map` / `with_state` → join and `on_stream`: one trace ID, depth + 1, parent = cause's span | holds | +| Exact parents in the diamond, 20 origins and 40 join outputs; every span distinct | holds | +| `on_stream`: 0, 1 and 3 writes per input; a burst of 300 into a 64-slot ring, then a later value still produces | holds | +| Join handler with `ctx`: sleeps on the runtime, then `produce_origin` starts a trace | holds | +| `Producer::produce_from` continues; with untraced metadata it starts a trace | holds | +| Carrier: 68 / 69 / 70 bytes; round trip; trailing fields ignored; ten malformed forms rejected | holds | +| Parent walk on a cycle after a restart | ends; rule tightened to trace ID and exact depth (§6) | +| `record.set` with a cause continues it; without or with a malformed one starts a trace; a cause with the instance's tag is counted | holds | +| `hello` with a tag: `welcome` returns the server's; an equal tag is counted, a different or absent one is not | holds | +| Zenoh attachment, outbound, real router | holds; inbound parsing of carrier, absent and rmw_zenoh attachments holds (§5.4) | +| Mandatory tag against the existing suites | 41 of 299 core, 52 of 102 Tokio adapter and 11 of 11 Zenoh `native_session` tests fail on the missing tag; with the test fallback, core and the adapter pass with tracing on (§4.2) | +| Write time | +16 to 20 ns per write in the spike's harness (§4.3) | + +Findings that changed this document: the fallback tag in test builds +(§4.2); the ID source compiled only with the feature (§4.2.2); the second +cost measurement (§4.3); the stricter parent walk and how easily a cycle +forms (§4.2, §6); the cause as a parameter of `JsonRecordAccess` and the +`trace` in the `record.set` reply (§5.1); the text form of the tag and the +tag as an argument of `ws_url_with_version` (§4.2.1); the `Sync` bound on +`on_stream` handlers (§2.2); the Zenoh result (§5.4).