Skip to content

Schedule parallel catchup infra pods ahead of the workers - #448

Merged
dmkozh merged 3 commits into
stellar:mainfrom
Jonathan-Eid:jonathan/catchup-infra-priority
Oct 26, 2026
Merged

dmkozh merged 3 commits into
stellar:mainfrom
Jonathan-Eid:jonathan/catchup-infra-priority

Conversation

@Jonathan-Eid

Copy link
Copy Markdown
Contributor

Sets priorityClassName: catchup-infra on the redis, preload and job monitor pods so the scheduler places them before the 1024 workers.

  • build 1968 failed "job monitor not reachable": per the API audit log the scheduler didn't attempt those 3 pods until 15:36, 9 min after creation, while it cycled through workers waiting on Karpenter nodes. The untainted nodes had room. Redis bound at 15:36, the driver's 10-min /status timeout fired at 15:38:07
  • the class (value 1000, preemptionPolicy: Never) comes from stellar/kube#5671. Merge that first and confirm it exists on ssc-eks and ssc-test, or every pod using it is rejected
  • workers keep default priority

Testing:

  • helm template: the 3 infra pods get catchup-infra, workers none
  • ssc-test run (class applied there): redis/preload/monitor at priority 1000 Never, workers 0, all scheduled

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jonathan-Eid and others added 2 commits October 26, 2026 12:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dmkozh
dmkozh merged commit f5985e2 into stellar:main Oct 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants