Skip to content

fix: keep the bundled Redis accepting writes when a snapshot fails [ENG-1707] - #56

Merged
twk3 merged 2 commits into
mainfrom
fix/eng-1707-redis-keep-writes-on-snapshot-failure
Oct 7, 2026
Merged

twk3 merged 2 commits into
mainfrom
fix/eng-1707-redis-keep-writes-on-snapshot-failure

Conversation

@twk3

@twk3 twk3 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Keeps the bundled Redis accepting writes when an RDB snapshot fails, and moves the app to the 2026-07-26-006 patch release. Today one failed snapshot makes it reject every write, which stops ingestion and crashloops the director.

Found on a self-hosted install (chart 0.8.0, app 2026-07-26-004, ENG-1707). Redis reached 24 GB with a full 50 GB volume, and every write returned:

MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk. ... (stop-writes-on-bgsave-error option)

Why it happens

redis.commonConfiguration replaces Bitnami's default (appendonly yes, save "") with just the loadmodule line. So the bundled Redis runs its built-in save schedule (3600 1 300 100 60 10000) with stop-writes-on-bgsave-error yes. A snapshot that fails because the disk is full, or because its fork is killed for memory, turns into a full write outage until a later snapshot succeeds.

The change

  • Add stop-writes-on-bgsave-error no. Snapshots stay on: Redis holds run:*:cache for in-progress runs and delayed BullMQ jobs (run timeouts, syncRunDB/syncInstance), and nothing recreates either after a restart. Refusing writes loses every incoming result for certain; continuing only loses data if Redis restarts before a snapshot succeeds.
  • Document the headroom a snapshot needs (memory limit about 2× used_memory, volume at least 2× the snapshot file) and the rdb_last_bgsave_status check, in values.yaml and docs/eks/dependencies.md.

An external Redis is unaffected.

App version 2026-07-26-006

appVersion and currents.imageTag go from 2026-07-26-004 to 2026-07-26-006 (patches/release-2026-07-26-006 in currents). On top of 005's on-prem seat check fix, it adds:

  • currents#4301: director passes async handler errors to next instead of exiting. This is the crashloop from the same outage.
  • currents#4302: pm2 stops writing log files for the writer and director ECS. This is the writer's PM2_HOME emptyDir filling up.
  • currents#4304: the dashboard no longer initializes Sentry in on-prem builds.
  • ENG-1535 #3905: tr-steps:*:upload-state TTL 24h → 2h.
  • ENG-1535 #3907: the term value is hashed in term_marker keys (TTL stays 24h).

Testing

  • helm lint passes, and helm template --set redis.enabled=true renders the line into redis.conf.
  • redis/redis-stack-server:7.2.0-v15 with /data on a 1 MB tmpfs, 30k keys, then BGSAVE ("No space left on device"):
    • Current config: SET, JSON.SET and the director's MULTI / HSETNX / EXPIRE / EXEC all fail with MISCONF, then EXECABORT, matching the customer's log.
    • This config: rdb_last_bgsave_status:err, and all three writes succeed.
  • On a running install, redis-cli CONFIG SET stop-writes-on-bgsave-error no unblocks writes without a restart (checked on the current config).

Release Plan

Merge after the 2026-07-26-006 images are published by the on-premise-release pipeline. The chart version stays 0.8.0 for the release PR to bump.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

twk3 and others added 2 commits October 7, 2026 08:33
…NG-1707]

The chart's redis.commonConfiguration replaces Bitnami's default, so the
bundled Redis runs its built-in RDB save schedule with
stop-writes-on-bgsave-error yes. One failed background save (a full disk, or
the fork killed for memory) then rejects every write with MISCONF. A
self-hosted customer hit this at 24 GB: the director's artifact transaction
failed with EXECABORT and the director pods crashlooped.

Snapshots stay on, because Redis holds in-progress runs and delayed queue
jobs that nothing recreates after a restart. Refusing writes loses every
incoming result for certain; continuing only loses data if Redis restarts
before a snapshot succeeds. Document the memory and disk headroom snapshots
need, and that operators should alert on rdb_last_bgsave_status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The patch release carries the director and pm2 log file fixes for the same
outage, the Sentry removal for on-prem dashboard builds, and two ENG-1535
TTL changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@twk3
twk3 marked this pull request as ready for review October 7, 2026 20:25
@twk3
twk3 merged commit a2ee577 into main Oct 7, 2026
2 checks passed
@twk3
twk3 deleted the fix/eng-1707-redis-keep-writes-on-snapshot-failure branch October 7, 2026 20:27
@twk3 twk3 mentioned this pull request Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant