Skip to content

Restart safety, storage cleanup and worker fixes - #70

Merged
Zacgoose merged 14 commits into
mainfrom
dev
Nov 11, 2026
Merged

Zacgoose merged 14 commits into
mainfrom
dev

Conversation

@Zacgoose

@Zacgoose Zacgoose commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Makes container restarts and overlapping deployments safe for background work, stops orchestrator storage from growing without limit, and fixes several worker problems.

Shutdown and restarts

  • The HTTP API and frontend keep serving while background work drains on SIGTERM. Kestrel stops only after the drain ends. The pump stops claiming new work, hands back claims it hasn't started, and gives running tasks most of the grace period to finish.
  • The grace period comes from ShutdownGraceSeconds, then WEBSITES_CONTAINER_STOP_TIME_LIMIT, then 30 s. The host shutdown timeout uses the same value. The drain (ShutdownDrainSeconds) defaults to the grace minus a margin for Kestrel, so 120 → 110 and 30 → 25. It also waits for scheduled starters.
  • The instance lock is handed over as soon as the drain finishes, not after Kestrel stops. A waiting successor checks for it every 2 s.
  • Each timer slot is claimed with an ETag-guarded row (StateTable, TimerSlot partition). When the old and new containers overlap during a restart, only one of them fires a slot. The scheduler keeps firing during shutdown, and anything it starts is saved to storage for the successor.

Orchestration

  • A sequential step that is still running is never reclaimed. Before, a step that ran longer than the driver lease was taken back and run a second time.
  • Runs still unfinished RetentionHours after they started are abandoned. They are deleted with no post-execution, at startup before the pump claims anything and again at every sweep. The periodic sweep skips a run while any of its tasks holds a live lease. The startup pass abandons it regardless.
  • The startup retention sweep no longer delays claiming or timers.
  • A daily sweep deletes rows that no index can reach once they are more than an hour old: Work partitions with no run header, and Results partitions and name rows whose run is gone. Runs still on the active list are left to index repair.

Storage

  • When a bulk delete hits a row that's already gone, the delete no longer leaves rows behind. The failed transaction's missing row is dropped and the rest are sent again. DeletePartitionAsync now uses the same path instead of skipping the whole batch.

Workers

  • HTTP burst workers are added only after every pool is built.
  • Global JSON preloads now survive the per-invocation global sweep.
  • The periodic memory trim no longer runs in the request's reclaim path. Its cooldown starts at process start.

Behaviour changes

  • RetentionHours default goes from 48 → 24. It now also sets how long a run may stay unfinished before it is abandoned. Instances that set it explicitly keep their value.
  • New optional settings: ShutdownGraceSeconds and ShutdownDrainSeconds. Both are documented in docs/configuration.md and appsettings.example.jsonc.

Tests

New or extended: AzureTableStoreDeleteTests, OrchestrationLifecycleTests, WorkStoreTests, PowerShellWorkerRunspaceHygieneTests and SchedulerTimezoneJitterE2ETests.

…unning

- a step outliving the driver lease was reclaimed and run twice, because only its own lease is renewed
…hutdown

- the pump drains in StoppingAsync, before Kestrel stops: no new claims, unstarted claims handed back, running tasks get up to ShutdownDrainSeconds (25)
- the scheduler fires nothing once shutdown starts
- the startup retention sweep no longer holds up claiming and timers
…s before claiming

- stale runs are deleted at startup before the pump claims and at every sweep, without running post-execution
- RetentionHours defaults to 24
- each due slot is claimed with an ETag-guarded row, so the old and new container of a restart never both fire it
- the scheduler keeps firing during shutdown; what it starts lands in storage for the successor
…grace

- grace from ShutdownGraceSeconds, else WEBSITES_CONTAINER_STOP_TIME_LIMIT, else 30; the host shutdown timeout follows it
- the drain defaults to the grace less a margin for Kestrel and also waits for scheduled starters
- the periodic sweep skips a run while any task holds a live lease; the startup pass still abandons regardless
…already gone

- a delete transaction fails whole on one missing row; drop the row the service names and resend the rest
- DeletePartitionAsync shares that path instead of skipping the whole batch
…ends

- released once no claimed task is in flight, not after Kestrel stops; a waiting successor looks every 2s
…ounded

- Work partitions without a run header, and Results partitions and name rows of runs that have none, are deleted once older than an hour
- runs on the active list are left to index repair
@Zacgoose
Zacgoose requested a review from JohnDuprey November 9, 2026 15:17
@Zacgoose
Zacgoose merged commit 15f0ae8 into main Nov 11, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants