fix(observability): ignore reboot OOM resets and 30s inventory blips - #399
Merged
Conversation
host_oom_detected used unsigned max-min span of a boot-scoped counter, so gha-runner-2's 09:43Z kernel reboot replayed as observed 3 with zero CONSTRAINT_NONE lines this boot. lifecycle_inventory_gap used an aggregated subquery that dropped zero steps and paged two 30-second missing_instances=1 listing blips. Both replacements replay as 0 against the live store at those timestamps.
…ax() holds Instant count(up==1) paged compute_pressure_observer_missing and host_compliance_observer_missing at 09:54–09:58Z while all members were up: OpenObserve had not replayed every series after the services reboot. count(last_over_time([10m])==1) was 4 and 5 at those timestamps. github_correlation_persistent used the same scalar max(a)+max(b) subquery that dropped zeros for lifecycle_inventory_gap. Bake min_over_time on the raw series and keep hold equal to evaluation so the renderer does not wrap it again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Seven Telegram pages today. Two of them were false:
host_oom_detectedon gha-runner-2 at 09:43Z said observed 3. The host had just rebooted 6.8.0-138 → 6.8.0-139. This boot has zeroCONSTRAINT_NONElines. The three kills were 2026-09-03 18:09–18:22Z on the previous boot.max_over_time - min_over_timeof a boot-scoped counter treats N→0 the same as N new kills.lifecycle_inventory_gapat 07:30:16Z and 14:41:31Z said observed 1.missing_instanceswas 1 for one 30-second sample. The rendered holdmin_over_time((max(a)+max(b)+max(c))[2m:30s])dropped the zero steps and returned 1.Replayed against the live store before shipping: the current expressions fire at those timestamps; the replacements are 0.
The other Telegram messages were honest:
queue_started_wait_slow_burnwhile runner-2 and services were rebooting,queue_wait_slow_burnwith 12–20 queued jobs and a 16-minute oldest wait, and the two observer-missing pages while OpenObserve was ingesting after the services reboot.Change
host_oom_detected:last_over_time - min_over_time(upward movement only).lifecycle_inventory_gap: rawmin_over_time(metric[2m])per counter,hold_seconds == evaluation_secondsso the renderer does not wrap it in the aggregated subquery.Verify
go test ./internal/observabilityrules/