Add the SWE-bench dataset builder, image mirror script, exclusion list, loader Job and README

[Vicente Ferrara (vicentefb)]

Third of three. Everything needed to prepare and run the benchmark:

• make-dataset.py builds the manifest from the SWE-bench/SWE-bench_Verified dataset (the SWE-bench org copy carries the harness eval scripts and log parser names; the older princeton-nlp copy does not) and resolves each task image to a digest with crane. --image-ref maps an instance to an image reference; the default matches a mirror with one tag per instance.
• mirror-images.sh copies images by digest into a registry the nodes can read, for projects where the node identity cannot pull from the shared mirror.
• offline-exclusions.txt: the 55 Verified instances whose evaluation needs network egress (all 34 matplotlib instances pip-install in their eval script; requests, django and sphinx have socket tests). Actors have no egress until an egress policy exists for them, which the env SDK cannot create yet. The list is generated by swebench-loader report -unresolved from a full pass and reviewed by diff.
• manifests/loader-job.yaml: runs the loader next to ate-env-api for the 500-wide shapes.
• tests/test_grade.py: the grader's parser and resolution rules.
• README.md: the workloads, the events and their meaning, prepare and run steps, and what the first runs showed.

[Vicente Ferrara (vicentefb)]

Updated after the first full-shape runs on cluster (445 images x 6 at width 500: 2,670/2,670; 445 x 2 x 8 steps with suspend parking: 7,120 rollouts, 0 failed, barriers flat). Changes on b157230: the loader runs commands as start/poll/read and retries a task's first command within a wake budget; grading matches the harness; trajectories gained a once-per-environment setup section that baselines the image's working tree, after sphinx showed that resetting to git HEAD loses edits SWE-bench bakes into the image; an in-cluster run script; a connection pool across API replicas.

[Tomer Glottmann (tomergee)]

Review at b157230, for this PR's own commits: 669 lines in 7 files under benchmarking/swebench/ and one line in .gitignore. The GitHub diff also shows #94 and #95. Merge it after #95.

Blocking

B1. .gitignore:5. The file conflicts with main, which added uv.lock in #82. Because of the conflict, the ci workflow (Go and Python jobs) did not run on this PR. Rebase, keep both lines, and let CI run.

B2. run-in-cluster.sh:47 and :50. The script runs cd "${ROOT}" before cp "${DATASET}", thus a relative dataset path resolves against the repository root. From /tmp/x, run-in-cluster.sh r1 ds.jsonl fails with cp: ds.jsonl: No such file. Make DATASET absolute before the cd.

Before merge

M1. run-in-cluster.sh:109-111. The until loop has no deadline. On ImagePullBackOff or a deleted pod, exitCode stays empty and the script waits with no limit. Add a timeout and print the pod status.

M2. run-in-cluster.sh:112-113. If the loader exits before it writes the events file, kubectl cp fails and errexit stops the script, but the Job stays. The keep sidecar then holds the pod and its 2 CPU / 2Gi request for 48 hours. Add a trap that deletes the Job.

M3. run-in-cluster.sh:60. Arguments go into the YAML inside double quotes with no escaping. -instances=a"b gives YAML that does not parse. Encode each argument as a JSON string, or apply the manifest as JSON.

M4. run-in-cluster.sh:76-96. Pin busybox by digest; Docker Hub also limits unauthenticated pulls. Add a securityContext (runAsNonRoot, allowPrivilegeEscalation: false, drop all capabilities), set automountServiceAccountToken: false, and set memory limits.

M5. make-dataset.py:91 and :109. The dataset load is not pinned, and the Hugging Face repository changed on 2026-08-16. eval_script comes from make_test_spec with whatever swebench is installed, but the dataset rows already carry eval_script. Two operators can thus build different manifests. Add a --revision flag, record the swebench version, and use row["eval_script"] or check that the two are the same.

M6. README.md:67-79. Step 1 copies to us-central1-docker.pkg.dev/PROJECT/swebench, but step 2 resolves us-docker.pkg.dev/PROJECT/swebench-mirror/swebench-verified. As written, each image is reported missing. Use one path and document how to make tags.txt.

M7. mirror-images.sh:17 and :50. The header and the description say "by digest", but the script copies SRC:tag. Copy by digest, or correct the text.

M8. benchmarking/swebench/tests. CI and the Makefile do not run these tests. Add them to ci.yaml next to clients/python (or move them to #94, which has the grader).

Nits

• make-dataset.py:85. The message says swebench>=3; the docstring says >=5.
• README.md:23 and :109. With the exclusion list, -images 500 gives 445 images and Select cuts the list with no message. The real count is 2,670 tasks, not 3,000.
• README.md:68. The source project gke-ai-eco-dev is internal; external users cannot read it.
• run-in-cluster.sh:107. grep -v '"step failed"' hides failures from the operator.
• The description lists manifests/loader-job.yaml, which commit 0a2baa4 deleted, and its list of network causes omits astropy and pylint.

Verified

• go build, go vet, go test ./internal/benchmarking/... pass on the head. pytest benchmarking/swebench/tests: 10 passed. shellcheck is clean.
• offline-exclusions.txt: 55 ids in 57 lines; the format matches ReadIDList.
• The fields log_parser, eval_type and eval_script exist in SWE-bench/SWE-bench_Verified. No credentials appear in the scripts or the Job spec.

[Vicente Ferrara (vicentefb)]

Addressed at d00f349. B1: rebased on main with both .gitignore lines. B2: the dataset path is made absolute before the cd. M1: the start wait has a deadline and prints the pod on timeout; the exit wait fails if the pod is gone or Failed. M2: a trap deletes the Job on any exit. M3: arguments are JSON-encoded. M4: busybox pinned by digest, non-root, no privilege escalation, capabilities dropped, no service account token, memory limits. M5: --revision, with the dataset revision and swebench version written to OUT.meta.json; the row's eval_script is checked to be contained in make_test_spec's, which adds the header and the exit-code marker the grader reads. M6: one registry path and tags.txt documented. M7: copies by digest and skips on equal digest. M8: the tests moved to #94. Nits: version message, counts, placeholders instead of the internal project, the log is no longer filtered. The inline comment on loader-job.yaml refers to a file 0a2baa4 deleted; the sidecar is now in the script with the deadline above.

[Tomer Glottmann (tomergee)]

Approved at d00f349. Both blocking items are gone: the branch is rebased with both .gitignore lines and CI runs green, and the dataset path is absolute before the cd. All eight before-merge items are in: bounded waits that print the pod on timeout, an exit trap that deletes the Job, JSON-encoded arguments, a digest-pinned non-root sidecar with dropped capabilities and no token, --revision with a provenance file and an eval-script check, one registry path with tags.txt documented, mirroring by digest with a skip on equal digest, and the tests moved to #94. Verified on the head: shellcheck, bash -n, py_compile, go test -race, gofmt and license headers; 13 checks green. One leftover for the description only: it still lists manifests/loader-job.yaml, which the branch deleted.