Repository navigation
Observe executor resources and current counter attachment - #363
Merged
Merged
Conversation
added 2 commits
October 5, 2026 09:46
Operators could see how long an executor reported holding a disclosure back, but not whether the keys its schedule makes due actually reached the dispatcher. The metrics endpoint now exports executor_disclosure_lag_seconds: the longest time, across registered executors, since the oldest due key became disclosable on the dispatcher's clock without this dispatcher having verified and recorded it, together with executors_disclosure_lag_unknown. The lag is judged by the key store's in-memory record of received keys only; collection reads no database and contacts no executor. Nothing is due before the first disclosable epoch, so a new chain is a known zero; once a key is due, an executor is unknown until its chain's first disclosure reaches this process, which an honest executor delivers within one heartbeat. DebugletDisclosureUnhealthy additionally fires when the lag exceeds 90 seconds, three maximal heartbeat intervals, or is unknown, and keeps firing while the sample is missing. The rule fixtures cover pending, firing, unknown, missing and recovery; the documentation states what the lag does and does not measure. GitHub issues #116 and #119.
ctfbruce
marked this pull request as ready for review
October 5, 2026 07:58
The metrics document described a zero lag as every due key being stored. The observation covers only the chain an executor announced for its current registered session: a restart announces a new chain, and the lag neither follows retained earlier chains nor certifies that their undisclosed final keys were delivered. Say so where the gauge is defined. GitHub issue #116.
2 tasks
1 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executor operators could see a selected eBPF mode but not whether its owned TCX links remained attached; dispatcher metrics lacked executor host-resource observations and any measure of whether disclosed keys were actually reaching the dispatcher. The existing bounded control report now includes daemon RSS/FDs, state-filesystem capacity/available space, and the current interface/hook/program match for both owned counter links. The authenticated metrics endpoint publishes fixed-cardinality aggregates with explicit unknown counts; missing, malformed, stale and disconnected reports cannot appear healthy.
Disclosure delivery lag. The exporter adds
executor_disclosure_lag_secondsandexecutors_disclosure_lag_unknown: for each registered executor's current announced chain, the time since the oldest key its schedule makes due became disclosable (dispatcher clock, no acceptance skew) without this dispatcher having verified and recorded it, read from the key store's in-memory record only. Before any key is due the lag is a known zero; once a key is due, an executor is unknown until the first disclosure of that chain reaches this dispatcher process (at most one heartbeat for an honest executor, one heartbeat after a dispatcher restart). The observation covers the current registered session's chain only: it does not iterate retained previous chains, certify delivery of their undisclosed tail keys, verify tags or captured traffic.Alerts. The opt-in required-counter alert now requires fresh attachment presence from every registered executor.
DebugletDisclosureUnhealthyadditionally fires when the lag exceeds 90 seconds (three maximal heartbeat intervals, equal to the report lifetime) or is unknown, and keeps firing while the sample is missing. A new executor-storage alert retains failures through missing reports and failed scrapes until fresh recovery. Attachment presence does not certify packet coverage or effective policing; daemon RSS/FDs exclude separate guest workers. No packet policy, enforcement algorithm, admission, schema or public discovery API changed.Launcher preflight. The foreground launcher rejected recoverable interrupted SQLite writes before the daemon could apply its existing recovery. Its schema preflight now verifies a private recovery copy and leaves the original database/journal unchanged; the daemon retains ownership of actual recovery. Managed reinstall and strict read-only checks keep their boundaries.
Validation. Health slice: 49 focused race-test events with zero skips; three race-instrumented real TCX detach/recovery repetitions releasing every process/BPF handle; a production-node Hello check observing real resources and attached versus fallback counters; Prometheus rule tests with nine failure/recovery fixtures; 33 race-test events verifying recoverable preflight, unsupported-schema refusal and unchanged original/journal bytes. Lag slice: gofmt/vet/build and race tests over internal/dispatcher, internal/dispatcher/tag and transport/api;
promtool check rulesandpromtool test ruleswith the pinned Prometheus 3.15.0, including before-change runs that fail as expected. Candidate CI on the reviewed health head f7f147c passed all 16 jobs (run 37014456246); the composed head with the lag slice passed all 16 jobs (run 37279734390).Refs #116, #119. Completes the executor resource, attachment-presence and current-chain disclosure-lag observations. Denied-traffic measurements, independently measured clock uncertainty and settlement backlog remain outside the supported observation contract.