Repository navigation
Log audit: catch OOM kills and retry give-ups logged with no severity - #732
Merged
Merged
Conversation
#730) gunicorn's 'Worker ... was sent SIGKILL! Perhaps out of memory?' and a worker's 'Failed on previous attempt' reach stderr with no severity, so the severity>=WARNING filter dropped them and the 2026-10-15 treevox OOM kill showed up only as an unexplained 503. Match both at any severity and count them as ERROR. Also record a missing severity as DEFAULT, not 'None', in the client auth path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #730.
gunicorn's
Worker (pid:N) was sent SIGKILL! Perhaps out of memory?and a worker'sFailed on previous attempt, marking as failedare written to stderr with no severity. Theseverity>=WARNINGfilter dropped both, so the 2026-10-16 audit reported the 2026-10-15 treevox OOM kill (#729) as an HTTP 503 with "cause unknown".Changes
build_filteralso matchesLOST_JOB_MARKERS(was sent SIGKILL,Failed on previous attempt) intextPayloadorjsonPayload.message, at any severity.to_entrycounts a matching line with no severity asERROR, so it lands in the ERROR+ column and its own cluster.iter_clientrecords a missing severity asDEFAULTinstead of the string"None".Verified
Ran against the incident window (
--auth rest --services treevox-v2-prod griddle-v2-prod --end 2026-10-15T21:00:00Z --hours 9). The digest now lists[ERROR] x1: [{ts} +{n}] [{n}] [ERROR] Worker (pid:{n}) was sent SIGKILL! Perhaps out of memory?and treevox'sFailed on previous attempt×2, neither of which appeared before. The--auth clientpath was not run (no ADC on this machine).The nightly routine clones
main, so this takes effect on the first run after merge.