Repository navigation
fix: leave the date intact when a rerun cannot start - #172
Merged
Merged
Conversation
A rerun whose Step Functions execution could not start deleted the date's trigger row and kept the rerun record it had already written, so the budget was spent, the date lost its record and the retried request was rejected. The rerun record is now written before the start and the accepted job row, the RERUN_ACCEPTED event and the post-run baseline delete after it. When the start fails, the record is deleted and the date's previous status (COMPLETED or FAILED_FINAL) is restored on a context that survives the request's cancellation, and the error is returned so the stream retries the request. The execution name comes from the stream record's creation time, so a retry after a start that did reach Step Functions reuses the same execution, and ExecutionAlreadyExists counts as started. The trigger read and the rerun budget count use consistent reads, so a retry arriving straight after the undo sees the restored row and the deleted record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a rerun's Step Functions execution could not start, the stream-router deleted the date's trigger row and kept the rerun record it had already written, along with the
rerun-acceptedrow andRERUN_ACCEPTED. The budget was spent, the date lost its record, and with the default budget of 1 the retried request was rejected.rerun-acceptedrow,RERUN_ACCEPTEDand the post-run baseline delete come after a successful start.store.DeleteRerun), and on a failed start or rerun write the date's previous status, COMPLETED or FAILED_FINAL, is put back. The undo runs on its own 5-second context detached from the request's cancellation, and the error is returned so the stream retries the request, up to its 3 retries. A record that cannot be deleted is logged and still counts against the budget; the status is restored either way.<id>-<schedule>-<date>-rerun<attempt>-<unix>now takes<unix>from the stream record's creation time. A retry after a start that reached Step Functions but reported an error reuses the name, so Step Functions returns the running execution instead of starting a second one, andExecutionAlreadyExistscounts as started.GetTriggerand the rerun budget count read consistently, so a retry arriving right after the undo sees the restored row and the deleted record.SFN_TIMEOUT: a failed status restore, a crash between the lock reset and the start, or an earlier start that finished before the retry arrived.