Skip to content

fix(sentry): keep the throttle's count across a container restart - #3435

Open
vpetersson-bot wants to merge 1 commit into
masterfrom
agent/anthias-claude/bd6edf627474
Open

vpetersson-bot wants to merge 1 commit into
masterfrom
agent/anthias-claude/bd6edf627474

Conversation

@vpetersson-bot

Copy link
Copy Markdown
Contributor

What is wrong today

The per-issue repetition throttle added in #3400 counts occurrences in
a module-scope dict, so its memory ends when the process does. The
issues that spend the most of the project's error allowance are
precisely the ones that end with the process going away: a board whose
webview will not start raises WebviewLaunchError out of
_spawn_webview_with_retry, that propagates out of the viewer, the
container restarts, and the new process comes back to an empty dict.

Occurrence 1 is paid for on every restart, so the powers-of-two
thinning never engages on the one shape of loop it was built for. In
the most recent window the project recorded in full, the launch-failure
issues (Sentry ANTHIAS-D / F / 1W / 1C / 4V) were an eighth of
everything accepted, almost all of it from a handful of boards in that
restart loop.

What this changes

The count moves to redis, which outlives the container — it is already
the broker, the channel layer and the viewer's signalling bus, and the
telemetry cooldown lives there for this same reason. The hour-long
window becomes the key's TTL, which also sidesteps the wall-clock step
the monotonic timer was guarding against.

That makes the unit of accounting the device rather than the process,
which is the unit the allowance is really spent in: server, celery and
viewer looping on the same issue now share one budget instead of each
being granted its own. Nothing else about the thinning changes —
occurrences 1, 2, 4, 8, … still survive, each one past the first still
carries its true running count in the throttle.occurrence tag, and
every device still reports every distinct issue it hits.

Details that matter:

  • SET NX EX before the INCR, pipelined into one round trip, so
    the counter can never exist without a TTL. An INCR followed by a
    separate EXPIRE would leave a key that never expires if the process
    died between the two, silencing that issue on that device for good.
    Losing an increment instead costs one extra reported event.
  • before_send runs inline on the reporting thread, so redis gets a
    half-second socket leash and no connection retries. redis-py's default
    (three attempts with backoff) turns that leash into a ~4s stall,
    which is long enough to matter inside a celery task's soft limit —
    measured at 3.18s before the change and 0.001s after.
  • A failure parks redis for a minute rather than being retried per
    event, and is deliberately not logged above debug: an ERROR here
    would become another event, decided by this same function.
  • When redis cannot answer, the existing in-process counter takes over
    unchanged.
    A throttle must never be the reason an event is lost, and
    a device whose redis is down is already having a worse day than its
    quota.
  • The unit suite opts out explicitly (IS_TEST): conftest mocks the
    shared connect_to_redis helper, which this path deliberately does
    not go through.

The handle is built straight from redis.Redis rather than through
anthias_common.utils.connect_to_redis, because this module has to stay
import-light (importing anthias_common.utils here crashes django-stubs'
mypy plugin in CI's slim environment) and the shared helper sets no
timeouts. stubs/redis-stubs/ grows pipeline/incr and the two
retry classes to match.

How I know it is right

Six new tests in tests/test_sentry.py, three of which fail without the
change: a restart loop thinned across processes, the TTL invariant, the
fallback to the local counter when redis refuses, and that a refusal is
asked once rather than per event.

Also driven against a real redis rather than only the fake: the key
always carries its 3600s TTL, the count survives the in-memory state
being cleared, and an event stream spanning 32 simulated restarts is
thinned to 1, 2, 4, 8, 16, 32 with each survivor correctly tagged.

Full non-integration suite (2595 passed, 3 skipped), ruff check, and
mypy . across 219 source files are all clean locally.

🤖 Generated with Claude Code

The repetition throttle added in #3400 counts occurrences in a dict
at module scope, so its memory ends when the process does -- and the
issues that spend the most quota are precisely the ones that end with
the process going away. A board whose webview will not start raises
WebviewLaunchError out of the viewer's spawn retry loop, the viewer
exits, the container restarts, and the new process starts from an
empty dict. Occurrence 1 is paid for on every restart and the
powers-of-two thinning never engages at all, which is the one shape
of loop the throttle was built for. In the last window the project
recorded in full, the launch-failure issues (ANTHIAS-D / F / 1W / 1C
/ 4V) were an eighth of everything accepted, from a handful of boards
doing exactly this.

Keep the count in redis instead, which outlives the container -- it
is already the broker, the channel layer and the viewer's signalling
bus, and the telemetry cooldown is kept there for this same reason.
The window becomes the key's TTL, which also sidesteps the wall-clock
problem the monotonic timer was there for. The unit of accounting
becomes the device rather than the process, which is the unit the
quota is actually spent in: server, celery and viewer looping on one
issue now share a budget instead of each being allowed their own.

SET NX EX runs before the INCR, pipelined into a single round trip,
so the counter can never exist without a TTL -- an INCR/EXPIRE pair
that lost its process in between would silence that issue on that
device permanently, whereas losing an increment costs one extra
reported event. before_send runs inline on the reporting thread, so
redis gets a half-second leash and no connection retries (redis-py's
default three-with-backoff turns that leash into a ~4s stall, which
is long enough to matter inside a celery task's soft limit), and a
failure parks redis for a minute rather than being retried per event.
When redis cannot answer, the existing in-process counter takes over
unchanged: a throttle must never be the reason an event is lost.

Verified against a real redis: the key always carries its 3600s TTL,
the count survives the in-memory state being cleared, and an event
stream spanning 32 simulated restarts is thinned to 1, 2, 4, 8, 16,
32 with each survivor tagged with its true running count.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
Copilot AI lite review requested due to automatic review settings October 29, 2026 09:16
@vpetersson-bot
vpetersson-bot requested a review from a team as a code owner October 29, 2026 09:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants