Repository navigation
fix(sentry): keep the throttle's count across a container restart - #3435
Open
vpetersson-bot wants to merge 1 commit into
Open
vpetersson-bot wants to merge 1 commit into
vpetersson-bot wants to merge 1 commit into
Conversation
The repetition throttle added in #3400 counts occurrences in a dict at module scope, so its memory ends when the process does -- and the issues that spend the most quota are precisely the ones that end with the process going away. A board whose webview will not start raises WebviewLaunchError out of the viewer's spawn retry loop, the viewer exits, the container restarts, and the new process starts from an empty dict. Occurrence 1 is paid for on every restart and the powers-of-two thinning never engages at all, which is the one shape of loop the throttle was built for. In the last window the project recorded in full, the launch-failure issues (ANTHIAS-D / F / 1W / 1C / 4V) were an eighth of everything accepted, from a handful of boards doing exactly this. Keep the count in redis instead, which outlives the container -- it is already the broker, the channel layer and the viewer's signalling bus, and the telemetry cooldown is kept there for this same reason. The window becomes the key's TTL, which also sidesteps the wall-clock problem the monotonic timer was there for. The unit of accounting becomes the device rather than the process, which is the unit the quota is actually spent in: server, celery and viewer looping on one issue now share a budget instead of each being allowed their own. SET NX EX runs before the INCR, pipelined into a single round trip, so the counter can never exist without a TTL -- an INCR/EXPIRE pair that lost its process in between would silence that issue on that device permanently, whereas losing an increment costs one extra reported event. before_send runs inline on the reporting thread, so redis gets a half-second leash and no connection retries (redis-py's default three-with-backoff turns that leash into a ~4s stall, which is long enough to matter inside a celery task's soft limit), and a failure parks redis for a minute rather than being retried per event. When redis cannot answer, the existing in-process counter takes over unchanged: a throttle must never be the reason an event is lost. Verified against a real redis: the key always carries its 3600s TTL, the count survives the in-memory state being cleared, and an event stream spanning 32 simulated restarts is thinned to 1, 2, 4, 8, 16, 32 with each survivor tagged with its true running count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What is wrong today
The per-issue repetition throttle added in #3400 counts occurrences in
a module-scope dict, so its memory ends when the process does. The
issues that spend the most of the project's error allowance are
precisely the ones that end with the process going away: a board whose
webview will not start raises
WebviewLaunchErrorout of_spawn_webview_with_retry, that propagates out of the viewer, thecontainer restarts, and the new process comes back to an empty dict.
Occurrence 1 is paid for on every restart, so the powers-of-two
thinning never engages on the one shape of loop it was built for. In
the most recent window the project recorded in full, the launch-failure
issues (Sentry
ANTHIAS-D/F/1W/1C/4V) were an eighth ofeverything accepted, almost all of it from a handful of boards in that
restart loop.
What this changes
The count moves to redis, which outlives the container — it is already
the broker, the channel layer and the viewer's signalling bus, and the
telemetry cooldown lives there for this same reason. The hour-long
window becomes the key's TTL, which also sidesteps the wall-clock step
the monotonic timer was guarding against.
That makes the unit of accounting the device rather than the process,
which is the unit the allowance is really spent in: server, celery and
viewer looping on the same issue now share one budget instead of each
being granted its own. Nothing else about the thinning changes —
occurrences 1, 2, 4, 8, … still survive, each one past the first still
carries its true running count in the
throttle.occurrencetag, andevery device still reports every distinct issue it hits.
Details that matter:
SET NX EXbefore theINCR, pipelined into one round trip, sothe counter can never exist without a TTL. An
INCRfollowed by aseparate
EXPIREwould leave a key that never expires if the processdied between the two, silencing that issue on that device for good.
Losing an increment instead costs one extra reported event.
before_sendruns inline on the reporting thread, so redis gets ahalf-second socket leash and no connection retries. redis-py's default
(three attempts with backoff) turns that leash into a ~4s stall,
which is long enough to matter inside a celery task's soft limit —
measured at 3.18s before the change and 0.001s after.
event, and is deliberately not logged above
debug: anERRORherewould become another event, decided by this same function.
unchanged. A throttle must never be the reason an event is lost, and
a device whose redis is down is already having a worse day than its
quota.
IS_TEST):conftestmocks theshared
connect_to_redishelper, which this path deliberately doesnot go through.
The handle is built straight from
redis.Redisrather than throughanthias_common.utils.connect_to_redis, because this module has to stayimport-light (importing
anthias_common.utilshere crashes django-stubs'mypy plugin in CI's slim environment) and the shared helper sets no
timeouts.
stubs/redis-stubs/growspipeline/incrand the tworetry classes to match.
How I know it is right
Six new tests in
tests/test_sentry.py, three of which fail without thechange: a restart loop thinned across processes, the TTL invariant, the
fallback to the local counter when redis refuses, and that a refusal is
asked once rather than per event.
Also driven against a real redis rather than only the fake: the key
always carries its 3600s TTL, the count survives the in-memory state
being cleared, and an event stream spanning 32 simulated restarts is
thinned to 1, 2, 4, 8, 16, 32 with each survivor correctly tagged.
Full non-integration suite (2595 passed, 3 skipped),
ruff check, andmypy .across 219 source files are all clean locally.🤖 Generated with Claude Code