Skip to content

Keep submitting last-good payouts during reward database outages - #466

Merged
dylanuys merged 2 commits into
testnetfrom
fix/skip-weights-on-reward-read-failure
Sep 28, 2026
Merged

dylanuys merged 2 commits into
testnetfrom
fix/skip-weights-on-reward-read-failure

Conversation

@dylanuys

@dylanuys dylanuys commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Propagate reward database read failures instead of treating them as an empty reward window.
  • Persist the last-good payable scores by hotkey. On reward-data failure, leave scores unchanged and continue submitting weights from that snapshot, while refreshing discriminator kings normally.
  • Resolve cached hotkeys to current UIDs so replacements cannot inherit rewards; exclude positive EMA history that was not eligible for payout.
  • Keep genuine empty-window and intentional full-burn behavior unchanged. Skip only when no safe payout snapshot exists.

Fixes the September 27 incident where a SQLite read error caused zero generator weights to be submitted, without stopping validator weight updates during subsequent database outages. This does not repair underlying storage faults.

Existing snapshots remain compatible and keep their EMA history. The first successful scoring run populates the new fallback; no database migration is required.

Validation

  • 242 tests passed; CI lint and diff checks passed.
  • Covers database errors/partial reads, repeated fallback, unchanged scores, restart persistence, UID reassignment, excluded EMA history, fresh kings, legacy/invalid snapshots, legitimate empty payouts, and recovery.

@dylanuys dylanuys changed the title Preserve generator weights when reward database reads fail Keep submitting last-good payouts during reward database outages Sep 28, 2026
@dylanuys
dylanuys merged commit eb3288f into testnet Sep 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant