Repository navigation
feat(observed): add Windows performance counter backend - #817
Ralf Biedert (ralfbiedert) wants to merge 40 commits into
Conversation
Add whole-event numeric metric bindings and native Perflib V2 publishing for integer counters, up/down deltas, and gauges. Generate matching deployment manifests and document Windows counter registration. Use performables for private shared state while preserving existing public Arc interfaces. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
The substantial unsafe Windows FFI and callback synchronization require human validation, particularly with the installed-schema consumer test remaining opt-in.
0 open findings
What changed in this PR
Adds typed metric bindings and a native 64-bit Windows Perflib V2 backend for observed.
Changes:
- Generates typed metric metadata and allocation-free numeric slot access.
- Adds Windows counter publishing, manifests, validation, tests, and examples.
- Migrates private sink/enrichment ownership to
performables::Arc.
| File | Description |
|---|---|
.spelling |
Adds Windows counter terminology. |
Cargo.toml |
Registers the backend and dependencies. |
Cargo.lock |
Locks new backend dependencies. |
crates/performables/src/arc/mod.rs |
Supports unsized standard Arc wrapping. |
crates/observed/Cargo.toml |
Registers the counter benchmark. |
crates/observed/README.md |
Documents whole-event bindings. |
crates/observed/benches/observed_counters.rs |
Benchmarks bound counter emission. |
crates/observed/src/enrichment/enrich_ext.rs |
Migrates enrichment ownership. |
crates/observed/src/enrichment/entry.rs |
Migrates stored values to performables. |
crates/observed/src/enrichment/slot.rs |
Migrates enrichment slots. |
crates/observed/src/event.rs |
Adds metric metadata and slot access. |
crates/observed/src/interop.rs |
Adds dynamic numeric slot forwarding. |
crates/observed/src/lib.rs |
Documents typed metric bindings. |
crates/observed/src/metadata/metric.rs |
Defines numeric metric representations. |
crates/observed/src/metadata/mod.rs |
Exports metric value types. |
crates/observed/src/processing/event_state.rs |
Forwards metric values during dispatch. |
crates/observed/src/processing/event_view.rs |
Exposes numeric measurements. |
crates/observed/src/processing/metric_bindings.rs |
Implements typed event bindings. |
crates/observed/src/processing/mod.rs |
Exports binding APIs. |
crates/observed/src/sink/core.rs |
Migrates private sink ownership. |
crates/observed/tests/counter_bindings.rs |
Tests binding and delivery behavior. |
crates/observed_perf_counters/Cargo.toml |
Defines the new backend crate. |
crates/observed_perf_counters/README.md |
Documents backend deployment and use. |
crates/observed_perf_counters/examples/consumer_manifest.rs |
Generates consumer-test manifests. |
crates/observed_perf_counters/examples/publish.rs |
Demonstrates counter publishing. |
crates/observed_perf_counters/src/error.rs |
Defines backend errors. |
crates/observed_perf_counters/src/lib.rs |
Exposes and documents the backend. |
crates/observed_perf_counters/src/native/consumer_tests.rs |
Tests native consumer integration. |
crates/observed_perf_counters/src/native.rs |
Implements Perflib publishing. |
crates/observed_perf_counters/src/processor.rs |
Implements processor configuration and updates. |
crates/observed_perf_counters/src/schema.rs |
Builds stable schemas and manifests. |
crates/observed_perf_counters/tests/allocations.rs |
Verifies allocation-free emission. |
crates/observed_perf_counters/tests/configuration.rs |
Tests configuration validation. |
crates/observed_macros_impl/src/event.rs |
Generates metric metadata and accessors. |
crates/observed_macros_impl/tests/expansion.rs |
Adds macro expansion coverage. |
crates/observed_macros_impl/tests/snapshots/expansion__a_fieldless_counter_becomes_the_event_level_metric.snap |
Captures fieldless-counter generation. |
crates/observed_macros_impl/tests/snapshots/expansion__a_macro_substituted_type_is_seen_through_its_invisible_group.snap |
Captures macro-substituted numeric access. |
crates/observed_macros_impl/tests/snapshots/expansion__an_instrument_looks_through_parentheses_around_its_value_type.snap |
Captures parenthesized numeric types. |
crates/observed_macros_impl/tests/snapshots/expansion__an_instrument_without_a_name_is_named_after_the_event.snap |
Captures default instrument naming. |
crates/observed_macros_impl/tests/snapshots/expansion__every_instrument_kind_records_the_field_it_names.snap |
Captures all metric representations. |
crates/observed_macros_impl/tests/snapshots/expansion__whole_event_metadata_does_not_generate_conflicting_associated_keys.snap |
Covers conflicting metric names. |
crates/observed_macros_impl/tests/snapshots/expansion__whole_event_metadata_preserves_non_identifier_counter_names.snap |
Covers non-identifier names. |
crates/observed_macros_impl/tests/snapshots/expansion__whole_event_metadata_preserves_raw_fields_and_instrument_names.snap |
Covers raw fields and names. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #817 +/- ##
=========================================
Coverage 99.9% 100.0%
=========================================
Files 808 241 -567
Lines 68565 15971 -52594
=========================================
- Hits 68556 15971 -52585
+ Misses 9 0 -9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Exercise numeric defaults, pointer-width code generation, counter identities, sticky errors, callback faults, native startup failures and teardown diagnostics. Add a private native API boundary so lifecycle and reference-cell tests run under Miri without Perflib syscalls. Keep native allocation and OS integration checks intact, run portable arithmetic tests on Linux, and limit Linux mutations to code actually compiled there. Preserve the 100 percent coverage gate and exclude only the equivalent default-constructor mutation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
Preserve the counter backend's equivalent-mutation and Linux platform rules alongside the updated metabench rules from main. Retain the base branch's dependency and benchmark changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
Declare alloc_tracker only for 64-bit Windows, matching the native allocation test. Linux no longer receives an unused dev-dependency; Windows retains the allocation-free emission contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
… tracing Separate atomic collection admission from the provider registry so registry contention can reuse a frozen snapshot and matching collection-end notifications never need that lock. Return retryable busy only while a snapshot writer is active. Store sticky processor errors before tracing and release both ordinary and poisoned guards before invoking subscribers. Add contention recovery, concurrent reference-read, reentrant subscriber, and panic-retention regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
Validate the final description after appending unit metadata, preserving the existing component checks and UTF-16 limit. Cover the reported overflow, exact-limit manifest output, and one-code-unit overflow for ASCII and supplementary Unicode units. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
Extract the zero-copy standard Arc conversion for unsized slices and trait objects, with regression tests, as a prerequisite for #817. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c45a6d66-dac9-4b55-beda-d5786efb3c2d
Distinguish missing manifest, category, and instance diagnostics. Require the Bytes Sent duplicate-destination error rather than accepting any invalid configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
Exposing Rust AtomicU64 storage directly to Perflib still requires native-safety review of the Windows-specific read/shutdown contract and an installed-schema consumer run, neither established by mocks or Miri.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
Update async-executor to the compatible non-yanked 1.14.1 release. Exercise the allocator head-lease handoff and Tokio startup abort paths without depending on thread scheduling. Keep the existing coverage and dependency gates unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
Direct AtomicU64 publication to Perflib requires Windows maintainer review of native-read and shutdown guarantees, while SDK manifest compilation and installed-schema consumer validation remain unrun.
No open issues 🎉
Review effort: Balanced
Resolve the lockfile conflict by keeping main's removal of obsolete executor dependencies while preserving counter-backend dependencies. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
Assert complete primary errors for unknown counter sets and unmapped instruments so another configuration failure cannot satisfy these tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
AtomicU64 publication and shutdown safety depend on a Windows-specific FFI contract requiring maintainer review, while SDK manifest compilation and installed-consumer validation remain unverified.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
Keep the new workspace release versions and exact observed macro pins, preserve the counter backend entry, and regenerate the observed README from its documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
The unsafe Perflib boundary relies on Windows atomic-read and shutdown guarantees that mocks cannot establish, requiring platform-owner review and the still-unrun installed-schema consumer validation.
No open issues 🎉
Review effort: Balanced
Keep the Gamma migration and allocator handoff regression from main. Adapt the incoming dynamic-event test to the private metadata constructors and disambiguate the native backend dependency in the merged lockfile. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
Direct Perflib access to Rust AtomicU64 storage requires human review of the native FFI safety contract and installed-manifest consumer validation on Windows, neither of which mocks or Miri establish.
No open issues 🎉
Review effort: Balanced
There was a problem hiding this comment.
cargo-gamma found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Adopt the exact upstream executor test fix from c27b005 as approved by the operator. Tighten disabled-event, nonzero-slot, native lifecycle, startup, and XML diagnostic assertions. Document only observed equivalent mutations, invariant-only error paths, and native-boundary or timeout limitations beside their exact source sites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
The unsafe Perflib backend requires a Windows maintainer to assess native atomic-access and shutdown-safety guarantees and validate installed-manifest consumer readings on a provisioned Windows host.
No open issues 🎉
Review effort: Balanced
|
Keep the upstream testable Tokio startup coordinator and preserve this branch's existing startup regression checks. Adapt the incoming observed oracle fixtures to the hidden metadata constructors without changing their assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 740c130d-6b52-40ad-92aa-da53562338f7
There was a problem hiding this comment.
🔵 Needs a closer look
The unsafe Perflib backend needs Windows/Perflib maintainer review because its native atomic-read contract is platform-specific and installed-manifest consumer validation remains unrun.
No open issues 🎉
Review effort: Balanced



Publish typed
observedinteger metrics to Windows performance-counter consumers, including PerfSpy deployments, using startup-prepared numeric targets and the existing lazy field visitor.Event metadata and binding
MetricDescriptionBuilderis deleted. No replacement construction API is exposed under interop. This intentionally changes the previous metadata construction API.EventDescription::fields()supplies the static routed-field inventory. SeparateEvent::METRICSandmetric_valueaccessors are removed. Fieldless counters remain inEventDescription::metric()and add one per emission.MetricDescription::value_kind()returns requiredMetricValueKind, not an option. The separate publicMetricValueenum is removed: bindings deliver the existingValue, normalizing unsigned measurements without losing integer precision. Invalid values fail explicitly.MetricBindings::bind_event::<E>()prepares targets at startup. Recording selects type identities and field indices, enumerating routed fields but evaluating only bound numeric getters. There is no string matching, hashing, allocation, redaction, or enrichment extraction during successful numeric recording.Windows definitions and behavior
observed_perf_countersimports an externally authored V2.manthroughmanifest_file(path)?.category(name)selects a declared set;build()validates exact instrument-name mappings and starts the provider on 64-bit Windows. Supplied provider/set GUIDs, counter IDs and definitions remain authoritative. Unbound declared counters remain zero. There is no XML export, identity generation, or catalog installation at startup.The supported profile remains a user-mode provider with a custom callback, multiple-instance sets, and inline-named, standard-detail, zero-scale
perf_counter_large_rawcountreference counters. The public backend surface remainsError,PerfCounterProcessorandPerfCounterProcessorBuilder.Unsigned counters add, signed up/down counters adjust a non-negative total, and unsigned gauges replace readings. Overflow/underflow preserve the prior value and surface through tracing and
Sink::flush(). Histograms, floats and signed gauges are rejected. Normal sink timestamp/context/sampling behavior remains; use an unsampled sink for deltas. Enrichment opt-out is deferred.Deployment compiles and installs the same supplied manifest and resource DLL separately; PerfSpy also needs its collection service configured. This imports V2 definitions, not classic .NET custom counter categories.
Direct atomic publication
Commit
15aedd8areplaces the callback snapshot/admission scheme. Perflib references now point directly at the existing, alignedAtomicU64totals. The ordinaryUnsafeCell<u64>publication buffer, snapshot copying, reader gate and sticky callback errors are removed. The custom callback does no work: delayed or overlapping START/END notifications cannot resume a publication copy. Emission retains its existing checked atomic updates, with no additional native calls or operations. Manifest formats are unchanged; no native setter loop is introduced.The documented FFI argument combines:
PerfSetCounterRefValue's dereference of a four- or eight-byte unsigned integer; this backend accepts only eight-byte counters.AtomicU64size/bit-validity/alignment guarantees, exclusively full-width Rust atomic writes, and retained storage through successful native shutdown. Failed shutdown retains storage instead of freeing potentially registered addresses.This is a Windows-specific scalar-access contract, not a claim that ordinary Rust raw-pointer reads are atomic or that arbitrary mixed-size/native writes are supported. No side payload relies on counter memory ordering. Individual readings are atomic; multiple counters are not a transactional snapshot. .NET's V2 CounterData implementation supplies supporting reference-publication precedent, not a replacement for that contract.
The design guide documents the argument and validation boundaries.
Integration and validation
Prerequisites #821 and #822 are merged. Commit
6de372c2preserves upstream sampling/sink-builder changes and migrates counter tests, examples and benchmark setup toSink::builder.Commit
7ffeb04emerges current main and repairs CI integration with the newly upstreamed sink-interest API. GitHub tested a merge ref containingsink_interest.rs, which was absent from the earlier local head; its tests and doctest still used the removed metadata constructor. They now use macro-generated event descriptions. The same compilation error was verified in Anvil and all four external Linux/Windows dev/release builds; no pipeline checks, assertions or exclusions were weakened.ddd0d444adds a private injectable manifest opener/reader. Load-once, file-error, malformed-input and byte-budget tests use controlled in-memory inputs without fixture writes or native startup. Public file-loading smoke coverage reads the checked-in manifest unchanged. Unsupported numeric tests assert exact diagnostics so missing manifest names cannot mask a regression. The unused backend tempfile dev-dependency is removed; no public construction API is added..manwith Windows SDK tools and running the installed-schema consumer scenario remain deployment validation tasks on a suitably provisioned host. The opt-in consumer test is not claimed to have run here. An earlier optional Ubuntu rerun was blocked before compilation by an unavailable kache wrapper; no cache bypass was applied.