serve: lame-duck on failed initial fetch, log stale-lock heals; retry Alpine bazel probe

[tinder-maxwellelliott]

Why

Two scheduled workflows are failing:

• Serve Stress has been red every day since 2026-09-10. ci: retry flaky bazelisk/coverage steps; add Rust vs Kotlin serve stress #490 pointed the stress harness at the Rust //src:bazel-diff serve (and Remove the Kotlin implementation; the Rust CLI is now the only one #498 then removed the Kotlin leg), but the Rust port was missing two behaviours the harness asserts and the README documents.
• Serve Stress (Alpine 8 GiB, real repos) failed on 2026-09-14 on all three legs. The small leg died in the Alpine bazel probe after GitHub Releases returned a 504 for rules_java; medium and bazel got past the probe and hit the same missing heal log as above.

What changed

src/server.rs

• A failed initial git fetch no longer exits serve() before the HTTP server binds. New perform_initial_fetch logs the error and leaves the instance lame-ducked: /health and query endpoints return 503, /metrics is still served with ready: false. This restores the RFC RFC: Bazel Query Service #29 behaviour the Kotlin serve had and the README promises.
• Clearing an orphaned .git/index.lock and retrying the checkout now logs cleared stale git index.lock in <workspace>; retrying checkout of <sha> so the harness (and operators) can see the self-heal path fire.
• New unit test initial_fetch_failure_lame_ducks_instead_of_exiting covering the un-ready responses and the --noInitialFetch path.

tools/serve_stress_alpine.sh

• probe_bazel retries up to 4 times with backoff when the probe output matches a download / 5xx / timeout error, and still fails fast on any other error so a genuinely non-working binary falls through to the next flavor quickly.

Verification

• cargo fmt --check, cargo clippy --all-targets, and the unit tests pass (107 in the lib).
• Ran the stress harness locally against the patched server: tools/serve_stress.py --quick --only lock and --only lameduck pass every check, including the four that failed in CI (self-heal path logged, startup with remote down -> lame-duck, query against lame-duck instance -> 503, /metrics still served while lame-ducked).
• The Alpine probe change is shell-only and exercised by the next serve-stress-alpine* cron runs; I could not reproduce the 504 locally.

Reviewer notes

• The Serve Stress workflow still sets up a JDK it no longer needs after Remove the Kotlin implementation; the Rust CLI is now the only one #498; left alone here as cosmetic.

🤖 Generated with Claude Code