Skip to content

Wait for other jobs only before submitting to the runner - #4390

Merged
un-def merged 1 commit into
masterfrom
issue_4331_runner_submission_context
Oct 20, 2026
Merged

un-def merged 1 commit into
masterfrom
issue_4331_runner_submission_context

Conversation

@un-def

@un-def un-def commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

_prepare_startup_context ran on every PROVISIONING and PULLING tick and returned early whenever the runner submission had to wait for other jobs: replica provisioning, the router IP, or referenced group IPs. The early return also skipped the status handlers' own work: copying the instance provisioning data into the job, failing the job when its instance is terminated, the provisioning timeout, and shim polling. With replica IP refs on newly provisioned instances, jobs waited for IPs that only the skipped copy would set, so the run stayed in provisioning forever.

  • Replace _prepare_startup_context with _prepare_runner_submission, called right before the runner submission once the runner is up.
  • Load volumes and interpolate secrets in the shim branch, which needs the interpolated registry_auth. The shim no longer waits for other jobs, so the image pull starts earlier.
  • Terminate workers whose router failed or did not get an IP in time in _terminate_if_router_unavailable, before both statuses as before.
  • Wait for every job in the replica to have a hostname, not only provisioning data, since shim submission no longer ensures it.
  • Drop the unused secrets parameter of _submit_job_to_runner.

Fixes: #4331

`_prepare_startup_context` ran on every PROVISIONING and PULLING tick
and returned early whenever the runner submission had to wait for other
jobs: replica provisioning, the router IP, or referenced group IPs. The
early return also skipped the status handlers' own work: copying the
instance provisioning data into the job, failing the job when its
instance is terminated, the provisioning timeout, and shim polling.
With replica IP refs on newly provisioned instances, jobs waited for
IPs that only the skipped copy would set, so the run stayed in
provisioning forever.

* Replace `_prepare_startup_context` with `_prepare_runner_submission`,
  called right before the runner submission once the runner is up.
* Load volumes and interpolate secrets in the shim branch, which needs
  the interpolated `registry_auth`. The shim no longer waits for other
  jobs, so the image pull starts earlier.
* Terminate workers whose router failed or did not get an IP in time
  in `_terminate_if_router_unavailable`, before both statuses as
  before.
* Wait for every job in the replica to have a hostname, not only
  provisioning data, since shim submission no longer ensures it.
* Drop the unused `secrets` parameter of `_submit_job_to_runner`.

Fixes: #4331
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@un-def
un-def merged commit 7077f6e into master Oct 20, 2026
26 checks passed
@un-def
un-def deleted the issue_4331_runner_submission_context branch October 20, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Service with replica IP refs stuck in provisioning when instances are newly provisioned

1 participant