fix(automation): keep syncing after profile renames and reload the crontab without a restart - #386
Merged
Merged
Conversation
…ontab without a restart On 2026-09-24 the sync-anchor profile `multisig` was renamed to `ten_minute`. supercronic still ran the crontab from its last start, `run multisig` failed with "unknown profile" before its git sync, and the box stopped pulling code for about 45 hours. Stables and Safe monitoring were down that whole time. - After each successful sync, re-render the crontab from the pulled jobs.yaml and rewrite $CRONTAB_PATH when it differs; supercronic now runs with -inotify and reloads it. - `run <unknown profile>` still syncs before exiting 2, so a stale crontab cannot stop the box from pulling code. - Move crontab rendering into automation/crontab.py; update runbook and jobs.yaml notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unknown-profile recovery hard-resets the checkout to origin/main, so a typo in a local `python -m automation run` could discard tracked edits. Gate it on CRONTAB_PATH, which only the monitoring systemd unit sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
On 2026-09-24 at 11:20 UTC the box pulled #6fe7b6c, which renamed the sync-anchor profile
multisigtoten_minute. The crontab is only rebuilt when the service restarts, so supercronic kept callingrun multisig. That failed withunknown profile 'multisig'before its git sync ran, and the box stopped pulling code until someone pulled and restarted by hand on 2026-09-26 08:09 UTC (about 45 h). Stables and Safe monitoring did not run at all during that window.Fix
automation/crontab.pyre-renders the crontab from the pulledjobs.yamland rewrites$CRONTAB_PATHwhen it differs. supercronic now runs with-inotify, so profile adds, removes, renames and cadence changes apply without a restart. Ifjobs.yamlis invalid, the current schedule stays.run <unknown profile>still syncs before exiting 2, so a stale crontab can't stop code sync.jobs.yamlnotes. Dependency changes still needuv syncplus a restart.Testing
reloading crontab, and the new job runs.--dry-rundoesn't).pytest: 1397 passed. ruff is clean.Deploy note (one time)
The unit file changed (
CRONTAB_PATHenv and-inotify). After merging, re-render/etc/systemd/system/monitoring.servicewithdeploy/install.sh, or add the two lines by hand. Then runsudo systemctl daemon-reload && sudo systemctl restart monitoring. Without that step, the sync-on-unknown-profile part still protects the box.🤖 Generated with Claude Code