Skip to content

Observe executor resources and current counter attachment - #363

Merged
ctfbruce merged 5 commits into
mainfrom
feat/executor-health-observations
Oct 5, 2026
Merged

ctfbruce merged 5 commits into
mainfrom
feat/executor-health-observations

Conversation

@ctfbruce

@ctfbruce ctfbruce commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Executor operators could see a selected eBPF mode but not whether its owned TCX links remained attached; dispatcher metrics lacked executor host-resource observations and any measure of whether disclosed keys were actually reaching the dispatcher. The existing bounded control report now includes daemon RSS/FDs, state-filesystem capacity/available space, and the current interface/hook/program match for both owned counter links. The authenticated metrics endpoint publishes fixed-cardinality aggregates with explicit unknown counts; missing, malformed, stale and disconnected reports cannot appear healthy.

Disclosure delivery lag. The exporter adds executor_disclosure_lag_seconds and executors_disclosure_lag_unknown: for each registered executor's current announced chain, the time since the oldest key its schedule makes due became disclosable (dispatcher clock, no acceptance skew) without this dispatcher having verified and recorded it, read from the key store's in-memory record only. Before any key is due the lag is a known zero; once a key is due, an executor is unknown until the first disclosure of that chain reaches this dispatcher process (at most one heartbeat for an honest executor, one heartbeat after a dispatcher restart). The observation covers the current registered session's chain only: it does not iterate retained previous chains, certify delivery of their undisclosed tail keys, verify tags or captured traffic.

Alerts. The opt-in required-counter alert now requires fresh attachment presence from every registered executor. DebugletDisclosureUnhealthy additionally fires when the lag exceeds 90 seconds (three maximal heartbeat intervals, equal to the report lifetime) or is unknown, and keeps firing while the sample is missing. A new executor-storage alert retains failures through missing reports and failed scrapes until fresh recovery. Attachment presence does not certify packet coverage or effective policing; daemon RSS/FDs exclude separate guest workers. No packet policy, enforcement algorithm, admission, schema or public discovery API changed.

Launcher preflight. The foreground launcher rejected recoverable interrupted SQLite writes before the daemon could apply its existing recovery. Its schema preflight now verifies a private recovery copy and leaves the original database/journal unchanged; the daemon retains ownership of actual recovery. Managed reinstall and strict read-only checks keep their boundaries.

Validation. Health slice: 49 focused race-test events with zero skips; three race-instrumented real TCX detach/recovery repetitions releasing every process/BPF handle; a production-node Hello check observing real resources and attached versus fallback counters; Prometheus rule tests with nine failure/recovery fixtures; 33 race-test events verifying recoverable preflight, unsupported-schema refusal and unchanged original/journal bytes. Lag slice: gofmt/vet/build and race tests over internal/dispatcher, internal/dispatcher/tag and transport/api; promtool check rules and promtool test rules with the pinned Prometheus 3.15.0, including before-change runs that fail as expected. Candidate CI on the reviewed health head f7f147c passed all 16 jobs (run 37014456246); the composed head with the lag slice passed all 16 jobs (run 37279734390).

Refs #116, #119. Completes the executor resource, attachment-presence and current-chain disclosure-lag observations. Denied-traffic measurements, independently measured clock uncertainty and settlement backlog remain outside the supported observation contract.

@ctfbruce ctfbruce added the Ready for review Reviewed, green candidate; maintainer review and dependency checks still required. label Oct 2, 2026
TheodorAdrienIsaak Mattli added 2 commits October 5, 2026 09:46
Compose the health observations and the launcher recovery correction with the
CILogon deployment changes that reached main after this branch was opened; no
file is changed by both sides.

GitHub issues #116 and #119.
Operators could see how long an executor reported holding a disclosure
back, but not whether the keys its schedule makes due actually reached
the dispatcher. The metrics endpoint now exports
executor_disclosure_lag_seconds: the longest time, across registered
executors, since the oldest due key became disclosable on the
dispatcher's clock without this dispatcher having verified and recorded
it, together with executors_disclosure_lag_unknown. The lag is judged by
the key store's in-memory record of received keys only; collection
reads no database and contacts no executor. Nothing is due before the
first disclosable epoch, so a new chain is a known zero; once a key is
due, an executor is unknown until its chain's first disclosure reaches
this process, which an honest executor delivers within one heartbeat.

DebugletDisclosureUnhealthy additionally fires when the lag exceeds 90
seconds, three maximal heartbeat intervals, or is unknown, and keeps
firing while the sample is missing. The rule fixtures cover pending,
firing, unknown, missing and recovery; the documentation states what the
lag does and does not measure.

GitHub issues #116 and #119.
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 07:58
The metrics document described a zero lag as every due key being stored.
The observation covers only the chain an executor announced for its
current registered session: a restart announces a new chain, and the lag
neither follows retained earlier chains nor certifies that their
undisclosed final keys were delivered. Say so where the gauge is defined.

GitHub issue #116.
@ctfbruce
ctfbruce merged commit c5b4e1d into main Oct 5, 2026
16 checks passed
@vincent10400094
vincent10400094 deleted the feat/executor-health-observations branch October 8, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Reviewed, green candidate; maintainer review and dependency checks still required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant