feat(observe): export drain-marked members independently of held-out - #403
Merged
Conversation
A full online drain keeps platform health up but left that exception invisible: HeldOutMembers stays empty because the listing is complete. Snapshot schema 16 and Prometheus now name every drain-marked member, online or off. Queue and assigned pages still fire.
Runbooks now name drain-marked members as the authorized-hold signal and keep held-out as the offline listing-incomplete subset.
The provider already filters user.gha_pressure.reason through that prefix. The observer now does the same, so a pressure-closed or operator-typed string cannot look like an authorized hold. A blank member listing is fail-closed: it is not evidence every member was marked.
count_over_time of (healthy < 1) is empty on a quiet fleet, and hold greater than eval then wraps that empty vector into error 20008. sum_over_time of the bool keeps a matrix. Host-signal tickets and dashboards use last-min of the cumulative counter so a collector restart or series reset is not an unsigned burst. Thresholds are unchanged.
Queued-delivery is thirty minutes, slow-burn is the five-to-thirty band, created-unbound is fifteen minutes. Inventory-gap pages on any of the three mins, not each of them. Alert rule count is thirty-one.
host_utilization and job_lifecycle were already in the contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Public #402 keeps platform health up during a full online drain, but that exception was invisible:
HeldOutMembersstays empty because the listing is complete. Operators could not see the authorized hold except by inferring it from health staying one.Snapshot schema 16 and Prometheus now name every drain-marked member, online or offline. Only a reason with the
drained:prefix counts; an empty member listing is not a full drain.gha_fleet_visibility_held_out_membersis still the offline listing-incomplete subset. Queue and assigned pages still fire. No alert threshold was retuned.Also in this PR, from the same bake-window audit:
fleet_health_flappingusessum_over_timeofhealthy < bool 1so a quiet fleet is not an empty vector (OpenObserve 20008 →alert_evaluation_failed).Verify
gofmt -l internal/fleetobservego vet ./internal/fleetobserve/go test ./internal/fleetobserve/ ./internal/observabilitydashboards/ ./internal/observabilityrules/ ./cmd/gha-fleet-observer/