feat: add leader election and fencing for singleton background jobs

[Olabsy19]

Overview

This PR adds DB/Redis lease-based leader election with fencing tokens for singleton background jobs (migrations, rebuilds, retention, schedulers, and reconciliation). At most one current leader mutates a singleton job; stale leaders are fenced, lease loss halts work safely, and takeover is bounded and audited.

Related Issue

Changes

🔒 Leader Election & Fencing

• [ADD] src/blockchain/leader-election.service.ts

  • Acquires and renews a lease (DB/Redis) for singleton jobs and issues monotonically increasing fencing tokens.
  • Rejects stale-token writes so a paused/partitioned former leader cannot mutate state.
  • Stops work safely on lease loss and bounds/audits takeover.

• [MODIFY] src/blockchain/reconciliation.service.ts

  • Runs reconciliation only while holding a current, valid lease; fences writes with the active token.

• [MODIFY] src/blockchain/blockchain-indexer.service.ts

  • Gates indexer/rebuild work behind the leader lease and fails closed on stale or ambiguous leadership.

• [MODIFY] src/audit/services/audit-retention.service.ts

  • Retention runs as a singleton job under the lease, with fencing on mutations and audited takeover.

• [MODIFY] src/claims/claim-projector.service.ts

  • Projector mutations require a current leader token; stale leaders stop work safely.

• [MODIFY] src/blockchain/blockchain.module.ts

  • Registers LeaderElectionService and wires it into the singleton job providers.

Verification Results

npm run build
✅ compiles

npm test
✅ leader election / fencing suites pass (two workers, lease expiry, clock skew, partition, slow holder, restart, stale token)

Acceptance Criteria	Status
At most one current leader mutates a singleton job	✅ Lease-guarded jobs; concurrent workers cannot both hold the lease
Stale leaders are fenced	✅ Monotonic fencing tokens reject stale-token writes
Lease loss stops work safely	✅ Services halt on lease loss before further mutation
Takeover is bounded/audited	✅ Bounded takeover with audit events

Closes #666

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Repository: DigiNodes/truthbounty-api/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: d4c3994b-10f0-412b-8f0b-4e57b5fc6451

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[dDevAhmed]

resolve conflicts @Olabsy19