Repository navigation
fix: keep the bundled Redis accepting writes when a snapshot fails [ENG-1707] - #56
Merged
Merged
Conversation
…NG-1707] The chart's redis.commonConfiguration replaces Bitnami's default, so the bundled Redis runs its built-in RDB save schedule with stop-writes-on-bgsave-error yes. One failed background save (a full disk, or the fork killed for memory) then rejects every write with MISCONF. A self-hosted customer hit this at 24 GB: the director's artifact transaction failed with EXECABORT and the director pods crashlooped. Snapshots stay on, because Redis holds in-progress runs and delayed queue jobs that nothing recreates after a restart. Refusing writes loses every incoming result for certain; continuing only loses data if Redis restarts before a snapshot succeeds. Document the memory and disk headroom snapshots need, and that operators should alert on rdb_last_bgsave_status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The patch release carries the director and pm2 log file fixes for the same outage, the Sentry removal for on-prem dashboard builds, and two ENG-1535 TTL changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keeps the bundled Redis accepting writes when an RDB snapshot fails, and moves the app to the
2026-07-26-006patch release. Today one failed snapshot makes it reject every write, which stops ingestion and crashloops the director.Found on a self-hosted install (chart 0.8.0, app
2026-07-26-004, ENG-1707). Redis reached 24 GB with a full 50 GB volume, and every write returned:Why it happens
redis.commonConfigurationreplaces Bitnami's default (appendonly yes,save "") with just theloadmoduleline. So the bundled Redis runs its built-insaveschedule (3600 1 300 100 60 10000) withstop-writes-on-bgsave-error yes. A snapshot that fails because the disk is full, or because its fork is killed for memory, turns into a full write outage until a later snapshot succeeds.The change
stop-writes-on-bgsave-error no. Snapshots stay on: Redis holdsrun:*:cachefor in-progress runs and delayed BullMQ jobs (run timeouts,syncRunDB/syncInstance), and nothing recreates either after a restart. Refusing writes loses every incoming result for certain; continuing only loses data if Redis restarts before a snapshot succeeds.used_memory, volume at least 2× the snapshot file) and therdb_last_bgsave_statuscheck, invalues.yamlanddocs/eks/dependencies.md.An external Redis is unaffected.
App version 2026-07-26-006
appVersionandcurrents.imageTaggo from2026-07-26-004to2026-07-26-006(patches/release-2026-07-26-006in currents). On top of 005's on-prem seat check fix, it adds:nextinstead of exiting. This is the crashloop from the same outage.PM2_HOMEemptyDir filling up.tr-steps:*:upload-stateTTL 24h → 2h.term_markerkeys (TTL stays 24h).Testing
helm lintpasses, andhelm template --set redis.enabled=truerenders the line intoredis.conf.redis/redis-stack-server:7.2.0-v15with/dataon a 1 MB tmpfs, 30k keys, thenBGSAVE("No space left on device"):SET,JSON.SETand the director'sMULTI/HSETNX/EXPIRE/EXECall fail with MISCONF, thenEXECABORT, matching the customer's log.rdb_last_bgsave_status:err, and all three writes succeed.redis-cli CONFIG SET stop-writes-on-bgsave-error nounblocks writes without a restart (checked on the current config).Release Plan
Merge after the
2026-07-26-006images are published by theon-premise-releasepipeline. The chart version stays 0.8.0 for the release PR to bump.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.