Repository navigation
fix: keep a running trigger row until the watchdog closes it - #173
Merged
Merged
Conversation
A running trigger row's stale deadline was its ttl, the attribute DynamoDB TTL deletes on, so DynamoDB could delete a dead run's row before the watchdog reported it. No SFN_TIMEOUT was sent, and the watchdog's reconciliation then started the date a second time. A running row now carries the deadline as staleAt and no trigger row carries ttl: taking, resetting or refreshing the lock writes staleAt and removes any ttl, and a terminal status removes both. The stale check reads staleAt, and its dry-run skip is gone so a pipeline switched to dry-run has its leftover running row closed too. The SFN_TIMEOUT detail key ttlExpired is now staleAt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A running trigger row's stale deadline was its
ttl, the attribute DynamoDB TTL deletes on. DynamoDB could delete a dead run's row before the watchdog's next pass reported it: noSFN_TIMEOUTwas sent, and reconciliation, finding a ready sensor and no trigger row, started the date a second time.staleAt. A running row carries its deadline asstaleAt(epoch seconds,sfn_timeout_secondsplus 30 minutes from the latest attempt, as before) and no trigger row carriesttl. Taking, resetting or refreshing the lock writesstaleAtand removes anyttl;COMPLETEDandFAILED_FINALremove both. DynamoDB never deletes a trigger row: a live run's row is closed by its own execution, and a dead run's only by the watchdog.staleAthas passed. TheSFN_TIMEOUTdetail keyttlExpiredis nowstaleAt, and theactionHintreads "step function passed its stale deadline: check SFN execution history" (listed under Changed (BREAKING)).ttlit would otherwise stay running for good.staleAt, and the watchdog page says a dead run staysRUNNINGwhile the watchdog is stopped.Deploy with no run in flight: a running row written by the previous build keeps its
ttland has nostaleAtuntil a writer touches it, so the watchdog does not report it and DynamoDB can still delete it.