feat(kubernetes): retry Job Pods lost to infrastructure disruptions

[yuechao-qin]

One Ignore rule on the DisruptionTarget Pod condition. Kubernetes creates a replacement Pod
when the cluster takes a Pod away. Nothing else changes.

This is the upstream documentation's own worked example, verbatim
(Handling retriable and non-retriable Pod failures), on top of the backoff_limit_per_index=0
the launcher already sets. 3 functional lines in
cloud_pipelines_backend/launchers/kubernetes_launchers.py.

What changes

Pod stops because	Before	After
Scheduler preemption	index fails → task FAILED	♻️ replacement Pod
Eviction API / autoscaler	task fails	♻️ replacement Pod
NoExecute taint, node NotReady	task fails	♻️ replacement Pod
Node deleted (Pod GC)	task fails	♻️ replacement Pod
Graceful node shutdown	task fails	♻️ replacement Pod
Node out of memory, kubelet evicts the Pod	task fails	♻️ replacement Pod

What does not change

Pod stops because	Result
main exits non-zero	index fails on the first attempt — identical
Container exceeds its own memory limit (OOMKilled)	index fails on the first attempt — identical
Image cannot be pulled, Pod never scheduled	stays Pending — no Pod failure, no policy involved
Orchestrator or launcher raised (SYSTEM_ERROR)	out of scope

No exit code is ever read, so a failing task is never rerun.

Why this works

Claim	Reference
"Ignore: use to indicate that the counter towards the .spec.backoffLimit should not be incremented and a replacement Pod should be created."	Job → Pod failure policy
"specifying the Ignore action for failed Pods with condition DisruptionTarget excludes Pod disruptions from being counted towards the .spec.backoffLimit limit of retries"	Job → Pod failure policy
DisruptionTarget is set only for disruptions initiated by the cluster — preemption, eviction, taint-based deletion, node loss, graceful node shutdown	Pods → Pod disruption conditions
Rules match on condition type and status only, never on a reason string	Job → Pod failure policy
podReplacementPolicy is not set: with a Pod failure policy present the API server defaults it to Failed and permits no other value	Job → Delayed creation of replacement pods
Requires the Job's restartPolicy: Never — already the case	Job → Pod failure policy

Stable since Kubernetes 1.31; the feature gate has been removed.

Verification

Upstream's own procedure, against a canary Job:

kubectl drain nodes/$nodeName --ignore-daemonsets --grace-period=0
kubectl get jobs -l job-name=<job> -o yaml   # .status.failed is NOT incremented

The replacement Pod carries batch.kubernetes.io/job-index-ignored-failure-count, which already
appears in GET /api/executions/{id}/container_state — no new instrumentation needed.