Repository navigation
ci(coverage): allow checkout retries before the step deadline - #464
Conversation
XuPeng-SH
left a comment
There was a problem hiding this comment.
APPROVE — no blocking findings at 0928c128ba036858d03aff276d3a71b24b9c67b4.
The design keeps retry ownership in the pinned checkout action rather than adding another retry loop. Both network checkouts receive step-scoped Git low-speed limits and a 20-minute outer deadline. This leaves room for the action's existing three attempts while bounding connection stalls and local Git work. Healthy checkout does not acquire a fixed delay. The 60-second low-speed setting is not a universal connection timeout; CONNECT/TLS failures are still bounded by the transfer stack and outer step limit.
The reporting change is a useful simplification: add the existing summary script to the already-pinned sparse helper checkout, then remove the redundant failure-only checkout. Failure reporting consumes the same trusted workflow revision and no longer needs another network fetch after tests fail.
Reviewed the complete YAML delta and its callers/failure gates. No test scope, coverage threshold, test deadline, credentials or runner selection is weakened. The 60-minute suite deadline / 70-minute fallback remain unchanged; 330 minutes is the job ceiling, not permission for tests to run that long. The operational tradeoff is that a broken network can now occupy up to 20 minutes per checkout instead of five.
Validation:
- Reused the successful hosted workflow validation for this PR and inspected the exact pinned checkout retry/child-environment implementation.
- Recalculated all configured budgets: cache off success/failure 280/305 minutes; cache on success/failure 300/325, below the 330-minute job ceiling.
- Real sparse checkout of this exact commit yielded precisely the five configured helper files. Executed the actual changed summary shell with both a missing report and a failed/panicking controlled-timeout report; expected summaries were published successfully.
- Latest main
d4972582ea25495d9a037f9d272188225b8a065bchanges this workflow only in an independent CGo rpath line. A merge-tree check succeeds without conflict and retains that main-side fix. - Existing Python helpers are unchanged; no unrelated Go/race run is warranted for this YAML-only change. Worktree is clean and diff whitespace checks pass. No outstanding review threads or comments were present.
Limit: Guangzhou end-to-end recovery remains unverified until a fresh caller run uses the merged workflow. The old incident's raw logs have expired (HTTP 410), so its detailed timing was not independently recertified. This is a bounded checkout recovery improvement, not proof that the underlying Guangzhou proxy/network fault is repaired.
What type of PR is this?
Which issue(s) this PR fixes:
Coverage checkout failures affecting MatrixOne PR #30021.
What this PR does / why we need it:
The Guangzhou coverage runner repeatedly exhausts its five-minute source-checkout budget inside the first Git fetch. Checkout is cancelled before its existing retry helper can recover, so Go setup, tests and coverage-profile publication never start. The later CI-helper checkout has the same short budget on the same network path.
Give both source and CI-helper checkouts a bounded 10-minute step budget and step-scoped Git HTTP low-speed limits. A connected transfer below one byte per second for 60 seconds aborts and returns control to the pinned checkout action's existing retry helper. Proxy CONNECT and TLS handshakes remain subject to Git/libcurl's connection timeout; the step deadline bounds the complete checkout, including multiple connections and local Git work.
Recent checkout logs show successful recovery after 5m31s and 7m04s: job 121078420439 and job 121085665239. The 10-minute cap accommodates these observed recoveries while bounding runner occupancy; it can still interrupt three persistently slow attempts.
Include the UT failure-summary script in the existing CI-helper sparse checkout at the reusable workflow's exact commit. Run the summary from that checkout and remove the redundant failure-only checkout, so reporting needs no additional Git fetch and uses the same trusted helper revision.
Account for every configured step limit on the cache-enabled failure path: 20 minutes for the two checkouts, 20 for cache seeding, 190 for coverage, 15 for Go/Python setup, 5 for cache status, and 55 for failure reports, artifacts, final status and failure propagation. Set the job limit to 310 minutes, leaving five minutes above the 305-minute step total for overhead and post-action cleanup. Preserve the source head, credentials, runner selection, test deadlines, coverage thresholds and failure gates.
Validation
Failure evidence: MatrixOne head
59741ad63a, run38442514176, coverage job115452608775. Source checkout timed out exactly at five minutes and test steps were skipped; the hosted shared-build job checked out the same source head in nine seconds.Validation after the 10-minute budget adjustment: YAML semantic comparison confirms only the two checkout deadlines and the derived job deadline changed; cache on/off and coverage success/failure totals were recalculated. Test scope, controlled test deadlines and low-speed settings are unchanged. Hosted workflow validation passed on commit
42345f2: run 40258389589.