Skip to content

ci(coverage): allow checkout retries before the step deadline - #464

Merged
XuPeng-SH merged 3 commits into
matrixorigin:mainfrom
aunjgr:fix/coverage-checkout-retry-budget
Oct 26, 2026
Merged

XuPeng-SH merged 3 commits into
matrixorigin:mainfrom
aunjgr:fix/coverage-checkout-retry-budget

Conversation

@aunjgr

@aunjgr aunjgr commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • BUG
  • Test and CI

Which issue(s) this PR fixes:

Coverage checkout failures affecting MatrixOne PR #30021.

What this PR does / why we need it:

The Guangzhou coverage runner repeatedly exhausts its five-minute source-checkout budget inside the first Git fetch. Checkout is cancelled before its existing retry helper can recover, so Go setup, tests and coverage-profile publication never start. The later CI-helper checkout has the same short budget on the same network path.

Give both source and CI-helper checkouts a bounded 10-minute step budget and step-scoped Git HTTP low-speed limits. A connected transfer below one byte per second for 60 seconds aborts and returns control to the pinned checkout action's existing retry helper. Proxy CONNECT and TLS handshakes remain subject to Git/libcurl's connection timeout; the step deadline bounds the complete checkout, including multiple connections and local Git work.

Recent checkout logs show successful recovery after 5m31s and 7m04s: job 121078420439 and job 121085665239. The 10-minute cap accommodates these observed recoveries while bounding runner occupancy; it can still interrupt three persistently slow attempts.

Include the UT failure-summary script in the existing CI-helper sparse checkout at the reusable workflow's exact commit. Run the summary from that checkout and remove the redundant failure-only checkout, so reporting needs no additional Git fetch and uses the same trusted helper revision.

Account for every configured step limit on the cache-enabled failure path: 20 minutes for the two checkouts, 20 for cache seeding, 190 for coverage, 15 for Go/Python setup, 5 for cache status, and 55 for failure reports, artifacts, final status and failure propagation. Set the job limit to 310 minutes, leaving five minutes above the 305-minute step total for overhead and post-action cleanup. Preserve the source head, credentials, runner selection, test deadlines, coverage thresholds and failure gates.

Validation

  • All workflows pass repository-pinned actionlint 1.7.12; its release checksum was verified.
  • YAML semantic comparison isolates the changes to checkout configuration, job budget and failure-summary script delivery. Diff whitespace checks pass.
  • Budget validation covers cache seeding enabled/disabled and coverage success/failure: step totals are 280/305 minutes with cache seeding and 260/285 without, all below the 310-minute job limit.
  • A real local Git sparse checkout of the candidate commit delivers exactly the five configured helper files. Executing the workflow's failure-summary shell publishes the expected failed-test/panic summary and the missing-report summary without a separate checkout.
  • Reused passing coverage-helper evidence: six execution-helper tests and five coverage-summary tests. Their source and test inputs are unchanged.
  • The pinned checkout retry helper with real local Git fetches succeeds immediately on a healthy request, recovers a transient transfer stall on attempt two, and propagates persistent failure after attempt three. This probe uses a two-second low-speed interval and the stock 10-20-second retry delays.
  • Separate localhost proxy probes confirm that the low-speed interval does not bound stalled CONNECT or TLS handshakes; external deadlines terminate those probes. No TLS verification settings are weakened.
  • Guangzhou end-to-end verification remains pending upstream deployment and a fresh MatrixOne caller run. Rerunning only failed jobs retains the reusable workflow revision from the original attempt.

Failure evidence: MatrixOne head 59741ad63a, run 38442514176, coverage job 115452608775. Source checkout timed out exactly at five minutes and test steps were skipped; the hosted shared-build job checked out the same source head in nine seconds.

Validation after the 10-minute budget adjustment: YAML semantic comparison confirms only the two checkout deadlines and the derived job deadline changed; cache on/off and coverage success/failure totals were recalculated. Test scope, controlled test deadlines and low-speed settings are unchanged. Hosted workflow validation passed on commit 42345f2: run 40258389589.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — no blocking findings at 0928c128ba036858d03aff276d3a71b24b9c67b4.

The design keeps retry ownership in the pinned checkout action rather than adding another retry loop. Both network checkouts receive step-scoped Git low-speed limits and a 20-minute outer deadline. This leaves room for the action's existing three attempts while bounding connection stalls and local Git work. Healthy checkout does not acquire a fixed delay. The 60-second low-speed setting is not a universal connection timeout; CONNECT/TLS failures are still bounded by the transfer stack and outer step limit.

The reporting change is a useful simplification: add the existing summary script to the already-pinned sparse helper checkout, then remove the redundant failure-only checkout. Failure reporting consumes the same trusted workflow revision and no longer needs another network fetch after tests fail.

Reviewed the complete YAML delta and its callers/failure gates. No test scope, coverage threshold, test deadline, credentials or runner selection is weakened. The 60-minute suite deadline / 70-minute fallback remain unchanged; 330 minutes is the job ceiling, not permission for tests to run that long. The operational tradeoff is that a broken network can now occupy up to 20 minutes per checkout instead of five.

Validation:

  • Reused the successful hosted workflow validation for this PR and inspected the exact pinned checkout retry/child-environment implementation.
  • Recalculated all configured budgets: cache off success/failure 280/305 minutes; cache on success/failure 300/325, below the 330-minute job ceiling.
  • Real sparse checkout of this exact commit yielded precisely the five configured helper files. Executed the actual changed summary shell with both a missing report and a failed/panicking controlled-timeout report; expected summaries were published successfully.
  • Latest main d4972582ea25495d9a037f9d272188225b8a065b changes this workflow only in an independent CGo rpath line. A merge-tree check succeeds without conflict and retains that main-side fix.
  • Existing Python helpers are unchanged; no unrelated Go/race run is warranted for this YAML-only change. Worktree is clean and diff whitespace checks pass. No outstanding review threads or comments were present.

Limit: Guangzhou end-to-end recovery remains unverified until a fresh caller run uses the merged workflow. The old incident's raw logs have expired (HTTP 410), so its detailed timing was not independently recertified. This is a bounded checkout recovery improvement, not proof that the underlying Guangzhou proxy/network fault is repaired.

@XuPeng-SH
XuPeng-SH merged commit 49251b9 into matrixorigin:main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants