Skip to content

AWS Batch: Add timeout before canceling stuck jobs #551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
victorlin wants to merge 1 commit into
base: master
Choose a base branch
from
+20 −5
Open

AWS Batch: Add timeout before canceling stuck jobs #551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ development source code and as such may not be routinely kept up to date.

# __NEXT__

## Bug fixes

* `nextstrain build --aws-batch` now waits 10 minutes before automatically
canceling any jobs that are stuck as RUNNABLE due to misconfiguration.
([#551](https://github.com/nextstrain/cli/pull/551))

# 11.1.1 (22 October 2026)

Expand Down
6 changes: 6 additions & 0 deletions doc/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ development source code and as such may not be routinely kept up to date.
(v-next)=
## __NEXT__

(v-next-bug-fixes)=
### Bug fixes

* `nextstrain build --aws-batch` now waits 10 minutes before automatically
canceling any jobs that are stuck as RUNNABLE due to misconfiguration.
([#551](https://github.com/nextstrain/cli/pull/551))

(v11-1-1)=
## 11.1.1 (22 October 2026)
Expand Down
8 changes: 6 additions & 2 deletions nextstrain/cli/runner/aws_batch/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,11 +420,15 @@ def interrupt_signaled(sig, frame):
if job.status_changed and not job.is_complete:
print_stage("Job now %s" % job.status)

# Cancel jobs that are stuck due to misconfiguration.
# <https://docs.aws.amazon.com/batch/latest/userguide/job_stuck_in_runnable_misconfiguration.html>
# Cancel jobs that are stuck due to misconfiguration.¹ The 10 minute
# timeout is buffer time for EC2 instance scale-up, etc. and mirrors the
# minimum duration of the AWS-side equivalent.²
# ¹ <https://docs.aws.amazon.com/batch/latest/userguide/job_stuck_in_runnable_misconfiguration.html>
# ² <https://docs.aws.amazon.com/batch/latest/userguide/job_stuck_in_runnable_time_limit_actions.html>
if (job.status == "RUNNABLE"
and job.status_reason is not None
and job.status_reason.startswith("MISCONFIGURATION")
and job.elapsed_time >= 600
and not job.stop_sent):
warn(f"Job is stuck in RUNNABLE. Reason: {job.status_reason}")
print_stage("Automatically canceling stuck job…")
Expand Down
6 changes: 3 additions & 3 deletions nextstrain/cli/runner/aws_batch/jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,9 @@ def is_failed(self) -> bool:
@property
def elapsed_time(self) -> float:
# Timestamps from the job state are in milliseconds
created = self.state.get("createdAt", float("NaN")) / 1000
stopped = self.state.get("stoppedAt", time() * 1000) / 1000
return stopped - created
start = self.state.get("createdAt", float("NaN")) / 1000
end = self.state.get("stoppedAt", time() * 1000) / 1000
return end - start

@property
def log_stream(self) -> Optional[str]:
Expand Down
Loading