Skip to content

fix: keep a running trigger row until the watchdog closes it - #173

Merged
dwsmith1983 merged 1 commit into
mainfrom
fix/trigger-stale-deadline
Nov 22, 2026
Merged

dwsmith1983 merged 1 commit into
mainfrom
fix/trigger-stale-deadline

Conversation

@dwsmith1983

Copy link
Copy Markdown
Owner

A running trigger row's stale deadline was its ttl, the attribute DynamoDB TTL deletes on. DynamoDB could delete a dead run's row before the watchdog's next pass reported it: no SFN_TIMEOUT was sent, and reconciliation, finding a ready sensor and no trigger row, started the date a second time.

  • staleAt. A running row carries its deadline as staleAt (epoch seconds, sfn_timeout_seconds plus 30 minutes from the latest attempt, as before) and no trigger row carries ttl. Taking, resetting or refreshing the lock writes staleAt and removes any ttl; COMPLETED and FAILED_FINAL remove both. DynamoDB never deletes a trigger row: a live run's row is closed by its own execution, and a dead run's only by the watchdog.
  • Stale check. The watchdog treats a running row as dead once staleAt has passed. The SFN_TIMEOUT detail key ttlExpired is now staleAt, and the actionHint reads "step function passed its stale deadline: check SFN execution history" (listed under Changed (BREAKING)).
  • Dry run. The stale check no longer skips dry-run pipelines. A dry-run pipeline never takes a trigger lock, so a running row for one is left over from before it was switched to dry-run, and without ttl it would otherwise stay running for good.
  • Docs. Every page that called the deadline the trigger lock's TTL now names staleAt, and the watchdog page says a dead run stays RUNNING while the watchdog is stopped.

Deploy with no run in flight: a running row written by the previous build keeps its ttl and has no staleAt until a writer touches it, so the watchdog does not report it and DynamoDB can still delete it.

A running trigger row's stale deadline was its ttl, the attribute
DynamoDB TTL deletes on, so DynamoDB could delete a dead run's row
before the watchdog reported it. No SFN_TIMEOUT was sent, and the
watchdog's reconciliation then started the date a second time.

A running row now carries the deadline as staleAt and no trigger row
carries ttl: taking, resetting or refreshing the lock writes staleAt and
removes any ttl, and a terminal status removes both. The stale check
reads staleAt, and its dry-run skip is gone so a pipeline switched to
dry-run has its leftover running row closed too. The SFN_TIMEOUT detail
key ttlExpired is now staleAt.
@github-actions github-actions Bot added tests Test changes lambda Lambda handlers docs Documentation e2e End-to-end tests types Public types (pkg/types) labels Nov 22, 2026
@dwsmith1983 dwsmith1983 self-assigned this Nov 22, 2026
@dwsmith1983
dwsmith1983 merged commit f8d36d4 into main Nov 22, 2026
12 checks passed
@dwsmith1983
dwsmith1983 deleted the fix/trigger-stale-deadline branch November 22, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation e2e End-to-end tests lambda Lambda handlers tests Test changes types Public types (pkg/types)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant