fix: capacity-burst timeout pages and queue slow-burn repeats - #404
Merged
Conversation
A create that reclaimed a warm slot and hit context deadline exceeded was classed as timeout, which pages and can open a 24-hour circuit. Name the refusal insufficient-memory, delete with an independent stop-plus-delete budget, and classify the same phrases in the GARM overlay so a rebuild agrees.
Fifteen-minute silence re-announced one memory-bound afternoon six times while assigned age stayed under the page and queued wait stayed under thirty minutes. Stall pages still evaluate on their own cadence.
Record that warm-preemption deadlines are capacity, and that queue slow-burn tickets repeat once per burst.
Stamp the derivative from the warm-preemption capacity source commit and advance the example platform pins with it.
Stamp overlay digests and the rebuilt GARM binary so warm-preemption deadlines class as capacity. Classify "timed out" with timeout, not the default provider class.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Live Drakkars tickets from 2026-09-05/06 were a real memory ceiling on the four members (queued waits 9–18 minutes, assigned age under the stall page). Two defects made that afternoon worse than the contract:
context deadline exceededwas classed astimeout, soprovider_retry_error_persistentpaged and three attempts can open a 24-hour circuit. The same create refused forinsufficient-memoryis a ticket.Stall pages (
lifecycle_queued_delivery_stall,lifecycle_assigned_stall, real identity/provider/timeout retries) are unchanged.What
context.WithoutCancel).insufficient-memoryso the live GARM classifier records capacity without a GARM rebuild.nddevProviderErrorClassclassifies preemption/delete-wait/busy-start as capacity; bare deadlines still page.queue_wait_slow_burnandqueue_started_wait_slow_burnrepeat every four hours.v0.1.5-nddev.124stamped from28bea87a(two matching rebuilds).Verify
Targeted:
go test ./internal/garmproviderincus/provider/ ./internal/observabilityrules/ ./internal/providerrelease/ ./internal/fleetcontract/Deploy (estate, after merge)
provider_versionandprovider-incus.tomltov0.1.5-nddev.124.deploy/github-actions/scripts/rollout-provider.sh <worktree> v0.1.5-nddev.124config/observability-rules.yaml.