Skip to content

fix: leave the date intact when a rerun cannot start - #172

Merged
dwsmith1983 merged 1 commit into
mainfrom
fix/rerun-failed-start
Nov 22, 2026
Merged

dwsmith1983 merged 1 commit into
mainfrom
fix/rerun-failed-start

Conversation

@dwsmith1983

Copy link
Copy Markdown
Owner

When a rerun's Step Functions execution could not start, the stream-router deleted the date's trigger row and kept the rerun record it had already written, along with the rerun-accepted row and RERUN_ACCEPTED. The budget was spent, the date lost its record, and with the default budget of 1 the retried request was rejected.

  • Order. The rerun record is written before the start. The rerun-accepted row, RERUN_ACCEPTED and the post-run baseline delete come after a successful start.
  • Undo. On a failed start the record is deleted (store.DeleteRerun), and on a failed start or rerun write the date's previous status, COMPLETED or FAILED_FINAL, is put back. The undo runs on its own 5-second context detached from the request's cancellation, and the error is returned so the stream retries the request, up to its 3 retries. A record that cannot be deleted is logged and still counts against the budget; the status is restored either way.
  • Execution name. <id>-<schedule>-<date>-rerun<attempt>-<unix> now takes <unix> from the stream record's creation time. A retry after a start that reached Step Functions but reported an error reuses the name, so Step Functions returns the running execution instead of starting a second one, and ExecutionAlreadyExists counts as started.
  • Consistent reads. GetTrigger and the rerun budget count read consistently, so a retry arriving right after the undo sees the restored row and the deleted record.
  • Docs. The pipelines page and the stream-router table say what a failed start leaves behind, and which cases still leave the date RUNNING until the watchdog's stale-trigger check closes it with SFN_TIMEOUT: a failed status restore, a crash between the lock reset and the start, or an earlier start that finished before the retry arrived.

A rerun whose Step Functions execution could not start deleted the
date's trigger row and kept the rerun record it had already written, so
the budget was spent, the date lost its record and the retried request
was rejected.

The rerun record is now written before the start and the accepted job
row, the RERUN_ACCEPTED event and the post-run baseline delete after it.
When the start fails, the record is deleted and the date's previous
status (COMPLETED or FAILED_FINAL) is restored on a context that
survives the request's cancellation, and the error is returned so the
stream retries the request. The execution name comes from the stream
record's creation time, so a retry after a start that did reach Step
Functions reuses the same execution, and ExecutionAlreadyExists counts
as started.

The trigger read and the rerun budget count use consistent reads, so a
retry arriving straight after the undo sees the restored row and the
deleted record.
@github-actions github-actions Bot added tests Test changes lambda Lambda handlers docs Documentation labels Nov 22, 2026
@dwsmith1983 dwsmith1983 self-assigned this Nov 22, 2026
@dwsmith1983
dwsmith1983 merged commit feecdf2 into main Nov 22, 2026
12 checks passed
@dwsmith1983
dwsmith1983 deleted the fix/rerun-failed-start branch November 22, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation lambda Lambda handlers tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant