Skip to content

Claude/focused pasteur hwbd8l - #191

Merged
maximusunc merged 3 commits into
develfrom
claude/focused-pasteur-hwbd8l
Oct 28, 2026
Merged

maximusunc merged 3 commits into
develfrom
claude/focused-pasteur-hwbd8l

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

No description provided.

Persistence: turn on the append-only file (fsync every second, RDB
preamble, rewrite only past 1gb) so a broker crash loses ~1s of writes
instead of everything since the last snapshot, and drop snapshots to
hourly -- every snapshot forks, and fork copy-on-write under write load
is the likeliest way the container gets OOM-killed. Cap client buffer
memory with maxmemory-clients so big concurrent transfers drop one client
instead of evicting data.

Monitoring: the memory alert only compared used_memory to maxmemory,
which can't see RSS, fragmentation or fork copy-on-write -- what the
kernel actually OOM-kills on. With REDIS_MEMORY_LIMIT set to the pod's
memory limit, it now also tracks that footprint and alerts on whichever
is higher, and at 98% it fires on the first tick instead of after the
120s grace. New alert when a snapshot or AOF write fails. The restart
alert says how much was lost (AOF vs snapshot) and what the broker looked
like on the last tick before it went down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRBBfCFE4VQ3gQA6hWm
pg_database_size() timing out on a stalled volume read as a 0-byte
database, which was cached and hid the database from the db_capacity
alert exactly while the disk was struggling. Keep the last good reading
and log the failure instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRBBfCFE4VQ3gQA6hWm
Conflicts were both additions at the same spot: devel's ARS payload-volume
collector and alert tests next to this branch's Postgres-size fix and
Redis memory tests. Kept both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRBBfCFE4VQ3gQA6hWm
@maximusunc
maximusunc merged commit 010f7ef into devel Oct 28, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/focused-pasteur-hwbd8l branch October 28, 2026 20:59
@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.80952% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 53.94%. Comparing base (f25b8e8) to head (1d5ed6c).
⚠️ Report is 7 commits behind head on devel.

Files with missing lines Patch % Lines
workers/monitor/worker.py 0.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
shepherd_utils/config.py 98.80% <100.00%> (+0.06%) ⬆️
workers/monitor/alerts.py 68.61% <100.00%> (+2.68%) ⬆️
workers/monitor/poller.py 50.68% <100.00%> (+6.17%) ⬆️
workers/monitor/recovery.py 90.47% <100.00%> (+2.01%) ⬆️
workers/monitor/worker.py 0.00% <0.00%> (ø)

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a72b360...1d5ed6c. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants