Add the swebench-loader runner and binary that drive SWE-bench images through the ate-env API

[Vicente Ferrara (vicentefb)]

Second of three. The runner executes the two Fast-benchmark shapes through the Go SDK (clients/go), so it measures what a trainer sees:

• eval: images x tasks-per-image at a given width; each task creates an environment from the digest-pinned task image (the API derives and bakes a template per image), replays the trajectory, records the reward and deletes.
• steps: images x rollouts created once and driven through N training steps; each step replays the trajectory on the whole fleet behind a barrier, then parks the fleet (suspend, delete or none).

Two things the first runs forced:

• Commands run as start, poll, read rather than through the SDK's streaming Shell: the router's five-minute route timeout cuts a single streamed request, and SWE-bench test runs take up to nine minutes.
• A task's first command retries transient gRPC errors within a wake budget (default 3 min): a cold first command outlasts the router's parking budget (Unavailable), and a first create races the image's golden bake (Unknown). The attempt count is recorded on the ttfi / wake event.

The backend is an interface; tests run the eval and steps shapes against an in-memory fake (width is enforced, environments are created once in steps mode, suspend/delete counts per park mode, reward and expectation handling, retry budget, validation).

Measured on a GKE cluster (139 n2-standard-8 workers, substrate ac5c5ee0): 445 Verified instances at width 100, 445 resolved, 0 failed, 11.4 min; steps 50 images x 2 rollouts x 3 steps with suspend parking, 300 of 300 resolved, park 2-4 s per fleet, wake p50 9.5 s.

[Vicente Ferrara (vicentefb)]

Updated after the first full-shape runs on cluster (445 images x 6 at width 500: 2,670/2,670; 445 x 2 x 8 steps with suspend parking: 7,120 rollouts, 0 failed, barriers flat). Changes on fbf8c09: the loader runs commands as start/poll/read and retries a task's first command within a wake budget; grading matches the harness; trajectories gained a once-per-environment setup section that baselines the image's working tree, after sphinx showed that resetting to git HEAD loses edits SWE-bench bakes into the image; an in-cluster run script; a connection pool across API replicas.

[Tomer Glottmann (tomergee)]

Review at fbf8c09, for this PR's own commits: cmd/benchmarking/swebench-loader/main.go and internal/benchmarking/swebench/{backend,run}.go with their tests, about 1550 lines. The GitHub diff also shows #94, because both PRs target main. Merge #94 first.

Blocking: none.

Before merge

M1. run.go:282. Eval mode leaks environments on cancel. runEval calls r.remove(ctx, ...) with the run context, thus after SIGINT, SIGTERM or a deadline each Delete fails. At width 500, a Ctrl-C leaves up to 500 environments. Steps mode (run.go:347-355) already uses context.WithoutCancel. Use the same cleanup context in eval mode. A test with a context-aware fake shows two leaked environments in eval mode and none in steps mode.

M2. run.go:374-386. A failed or timed-out create is never deleted. A create error leaves e.sb nil, thus remove skips the environment, but the server can still finish the create. After the 5-minute CreateTimeout, the environment appears later and leaks, and with -park delete a later create with the same id can get AlreadyExists. After a create error, delete the environment by id and ignore NotFound.

M3. backend.go:140-177. Run does not stop the guest process when it returns early. After a cancel, or after the poll gives up, the process continues. doStep (run.go:504) can then start the same command again while the first copy runs, for example two git processes on one index.lock. Kill the process with a short context.WithoutCancel context before Run returns an error.

M4. backend.go:140-205. The start, poll and read path is the main change of this PR, but the tests cover only the in-memory fake. Add tests against a fake gRPC server through ClientOptions.GRPCConn: exit, transient poll errors, the retry limit and output collection.

M5. Design. Each SDK user meets the router's route timeout, the nemo-gym integration for one. Move the start, poll and read logic into clients/go as an option of Env.Run, or record why it stays here. The retry on Unavailable for the first command overlaps the wake-wait in #86; say in the doc which one is the long-term fix.

Nits

• run.go:106 and :115. staticcheck ST1005: start each error string with a lowercase letter.
• main.go:84. The -wake-backoff default is 2s, but Config.defaults (run.go:145) uses 1s. Use one value.
• main.go:101. -run-id has no validation, and EnvID (run.go:247) adds it to each id. An uppercase or long run id makes each create fail. Check it against the Substrate name rule.
• run.go:476. Steps mode stops counting an environment after its first failure, thus later steps have fewer tasks and their resolved rates are not comparable. Report the dropped environments for each step.
• events.go:89 (from #94) and main.go:139. Record discards encode errors and the f.Close error is not checked. A full disk loses events with no message.
• main.go:72 and :75. The flag help names make-dataset.py and offline-exclusions.txt, which only #96 adds.

Verified

• go build, go vet pass on the head and on the merge with main at 1c08a7a. go test -race -count=3 passes; coverage 83.7%. gofmt and make verify-gofmt verify-boilerplate are clean. CI passes.
• status.FromError unwraps the SDK's wrapped errors, thus isTransient sees the correct codes.
• Not verified: the cluster results in the description (445/445 and 300/300 resolved).

[Vicente Ferrara (vicentefb)]

Addressed at 375e9e2. M1: eval mode deletes with a context that survives cancellation, tested with a context-aware fake that cancels from inside the first create. M2: a failed create is deleted by name, NotFound ignored. M3: Run kills the process before any early return. M4: tests against an in-process gRPC API over bufconn through ClientOptions.GRPCConn: exit and output, transient polls, retry limit with kill, cancel with kill, delete by name. M5: recorded in Run's doc comment: start/poll/read belongs in clients/go as an option of Env.Run and lives here until that lands; with #86 merged the server waits for the wake, and the loader's first-command retry remains the client-side safety, counted per event. Nits: lowercase error strings, one wake-backoff default, run id validated against the name rule and capped at 20 characters, live fleet size per step in the report, recorder and output-file errors fail the run, flag help no longer names #96 files. Inline: reset after a delete park was already gated at 88b539a and the setup/reset split makes it explicit; think time is skipped before the first command; f.Close is checked.

[Tomer Glottmann (tomergee)]

Approved at 375e9e2. All five before-merge items and the nits are in the code: eval mode cleans up with a cancellation-proof context, a failed create is deleted by name, Run kills the process before an early return, the backend has tests against an in-process gRPC API, and the start/poll/read design is recorded in the Run doc comment. Verified on the head: build, vet, go test -race, gofmt, license headers, and the 11 grader tests pass; CI is green.