[Kevin Cho (chokevin)]

Project Signal had no upstream end-to-end path from a certification request to a bounded group of node workloads and durable, typed results. This adds the controller, grouped admission contract, and optional Pod/MPIJob adapter while keeping diagnostic scripts, approval decisions, and environment-specific deployment policy outside the repository.

Behavior

• A CertificationProfile can select 1-64 participants. The requesting Node is index 0; every participant records its immutable Node UID plus capacity and allocatable snapshots.
• Admission reserves the complete group atomically, rolls back partial reservations, and validates runtime stage and deadline contracts before reserving any Node.
• Script and evidence digests use one strict algorithm-qualified representation: sha256:<64 lowercase hex>. Bare, uppercase, wrong-algorithm, truncated, extended, and whitespace-padded values are rejected.
• The optional adapter runs one independently Node-pinned Pod or kubeflow.org/v2beta1 MPIJob per participant. MPI execution uses one launcher and one GPU worker per tested Node.
• An immutable workloadPolicy carries bounded resources, security context, working directory, results storage, volumes, host networking, and typed Kueue topology intent.
• workloadPolicy.kueueTopology.requiredLevel requires an explicit queue and renders kueue.x-k8s.io/podset-required-topology only on ordinary participant Pods and MPIJob worker templates, never the unpinned launcher.
• Stage-aware scripts declare script.requiresRuntimeStage: true and a bounded lowercase DNS-label runtimeStage.name. The stage is snapshotted Profile-to-Run-to-evidence and exported as PROJECT_SIGNAL_STAGE_NAME only to ordinary participant Pods and MPI workers, never the launcher. Non-stage scripts omit it.
• backendActiveDeadline is distinct from outer deadline. It must be positive whole seconds and leave at least 60 seconds for adapter result projection, evidence, and cleanup. The outer value becomes Workflow activeDeadlineSeconds; the backend value becomes Pod/MPIJob activeDeadlineSeconds.
• Diagnostic results use an exact verdict/exit matrix: Pass=0, Fail=1, Inconclusive=2. A syntactically and semantically valid bounded result remains authoritative when exit 1 or 2 makes the backend terminally failed. Missing, malformed, stale, contradictory, or unexpected-code results remain fail-closed as backend execution failures.
• Completion records API-assigned workload UIDs and exact participant coverage in bounded backend-resources. A grouped Pass requires complete backend and result coverage.
• Recovery releases every reservation in an abandoned group, and foreground deletion plus Workflow ownership provides cleanup across normal and interrupted execution.
• Each profile/run selects one immutable script key and backend. FullCertification and stagePlan are evidence metadata, distinct from the optional single-script runtime stage.

Scope

This PR does not publish certification scripts, runner packaging, hardware thresholds, deployment overlays, queue, flavor, topology, or stage values, resource quantities, mount paths, capabilities, tolerations, cluster or node identifiers, credentials, or endpoints. It does not add TrainJob support, approve privileged or host access, synthesize a cross-run suite verdict, or claim live-cluster validation.

Evidence

Grouped admission, rollback, backend identity, and failure-domain behavior (11 seconds):

signal-grouped-evidence.mp4

Pod/MPIJob rendering, fail-closed aggregation, and cleanup behavior (15 seconds):

project-signal-adapter.mp4

Typed workload policy, Pod queue labeling, MPI worker isolation, and focused contract tests (15 seconds):

signal-workload-policy-evidence.mp4

Typed host-network field, participant-only Pod/MPI rendering, launcher isolation, and focused tests (15 seconds):

project-signal-host-network.mp4

The first downstream CertificationRun exposed the digest mismatch before backend creation. Commit 3406e82258971c3b4049ab70a606b50a1772049a aligns the canonical digest contract.

Representative Kueue admission then exposed missing explicit TAS intent. Commit 5b117beffd65e6d45c15892f163bb64096b194e9 adds the typed topology policy and participant-only rendering.

A4 execution then exposed that valid Inconclusive result files followed by exit 2 were discarded as backend failures. Exact head f6c11877808a10ae59760149da4676e15e00dd22 makes validated verdict-associated nonzero exits authoritative, rejects contradictory or missing results, and adds the typed runtime-stage and split-deadline contracts required by the downstream scripts.

Exact-head pre-merge images:

• Controller index: ghcr.io/chokevin/project-signal-controller@sha256:60ef8509c3aa12761c5dc2db4e7679d69106a61d7e29606148876ea2d8792839
• Controller linux/amd64 manifest: ghcr.io/chokevin/project-signal-controller@sha256:e42aa8887f6b2fee1aef99c117f672a63027f7bdd398bab5f4ee35f4f49cc785
• Adapter index: ghcr.io/chokevin/project-signal-adapter@sha256:f3b93286383416cd1b7fb4ba9e633e9cc71837cebba2b883fc6a15812a692ba5
• Adapter linux/amd64 manifest: ghcr.io/chokevin/project-signal-adapter@sha256:1268386ce2148b598cf6beafa7684d5981e9327705bdf1de59bc583e2d0ef752

Verification

• Passed: go test -race ./api/signal/v1alpha1 ./internal/signalworkload ./internal/signal ./internal/signaladapter ./cmd/project-signal-controller ./cmd/project-signal-adapter ./deploy/signal
• Passed: GOOS=linux GOARCH=amd64 golangci-lint run -c .golangci.yaml ./api/signal/v1alpha1 ./internal/signalworkload ./internal/signal ./internal/signaladapter ./cmd/project-signal-controller ./cmd/project-signal-adapter ./deploy/signal
• Passed: GOOS=linux GOARCH=amd64 go build ./cmd/project-signal-controller ./cmd/project-signal-adapter
• Passed: go generate ./api/signal/v1alpha1; generated CRDs contain canonical digest, bounded topology, runtime-stage, deadline-ordering, and immutability constraints.
• Passed: make signal-manifests with no rendered manifest drift.
• Passed: git diff --check.
• Passed: exact-head fork image workflows for controller and adapter; OCI indexes and linux/amd64 child manifests resolved independently.
• Passed: all current PR checks. The unrelated Racer default-feature test descriptor_half_close_matches_simulated_and_host_sockets failed once and passed on the failed-job rerun.
• Not rerun: live downstream execution; this fix did not deploy to or mutate downstream environments.

Risk

The highest-risk boundaries are translating asynchronous Kubernetes and MPI Operator state into a terminal diagnostic result, preserving operator-supplied participant policy without leaking it onto the launcher, and separating backend execution time from the outer evidence/cleanup window. The adapter requires exact backend identity and participant coverage, clears reserved result files before invocation, validates bounded result fields plus the exact verdict/exit matrix, checks result exit status against Kubernetes terminal state, revalidates runtime-stage and deadline ordering, and fails closed when identity, status, outputs, cleanup, or contracts cannot be proven.

Review guide

1. api/signal/v1alpha1/ and internal/signal/controller.go: immutable API snapshots, runtime stage/deadline validation, canonical digests, grouped admission, Workflow arguments, and evidence.
2. internal/signalworkload/policy.go: bounded resources, security context, paths, volume unions, and Kueue topology validation.
3. internal/signal/policy.go: whole-group recurring and failure-domain accounting.
4. internal/signaladapter/adapter.go: result-record contract, verdict/exit matrix, Pod/MPI rendering, participant-only stage/topology/policy application, and bounds.
5. internal/signaladapter/runner.go: backend terminal observation, result authority, deadline separation, UID collection, and foreground cleanup.
6. deploy/signal/: WorkflowTemplate wiring, generated CRDs, operator contract, and single-run semantics.

AI assistance: GitHub Copilot assisted with implementation, tests, investigation, documentation, evidence preparation, and PR drafting. The author verified the claims and final text.

[Kevin Cho (chokevin)]

Typed workload policy, Pod queue labeling, MPI worker isolation, and focused contract tests (15 seconds):

signal-workload-policy-evidence.mp4