Skip to content

ci: docs-only PRs run the full test matrix unless the title has [doc-only] or main has a green baseline #3480

Open

Description

@Andy-Jost

What happens

Docs-only PRs often run the full build and test matrix. Two mechanisms are supposed to prevent that, and both miss in common cases:

  1. The title tag. should-skip in .github/workflows/ci.yml skips the non-docs builds and all test jobs only when the PR title contains the literal, case-sensitive [doc-only]. Most docs PRs use a conventional prefix such as docs(cuda.core): ... instead, which does not match. The tag is not documented anywhere in the repo.
  2. The path planner. ci/tools/compute_ci_plan.py already treats <package>/docs/**, *.md and a few other paths as needing no tests. But it uses the changed paths only when the merge base has a successful main CI run with its artifacts. Without one, main() passes an empty path list and force_all is set ("this run will build and test everything").

Lately main rarely has that baseline: of the main push runs since 2026-10-17, 35 succeeded, 44 failed and 75 were cancelled. So an untagged docs PR now gets the full matrix.

The rule is the same for cuda.core, cuda.bindings and cuda.pathfinder (one ci.yml, one planner). The habit differs: cuda.bindings and cuda.pathfinder docs PRs mostly carry [doc-only], while cuda.core docs PRs mostly use docs(cuda.core): ....

Evidence (PRs opened since 2026-10-17)

Cost

A docs run that skips tests costs about 12-50 job-minutes (build-linux-64 for the docs, the docs build, the checks). A full run costs about 1,050 job-minutes, more than 20 times as much. It also occupies the shared GPU runners, so other PRs wait longer in the queue.

Options

  1. Detect docs-only changes from the paths, with or without a baseline. Always compute the changed paths against the merge base. If every path is one the planner already ignores, skip the tests even when there is no baseline. The [doc-only] tag becomes unnecessary. Drawback: the ignore list becomes the only safeguard, so a wrong entry would skip tests on real code. The same list already makes this decision whenever a baseline exists.
  2. Also accept the conventional docs(...): / doc(...): prefix, when the paths agree. Drawback: it keeps CI behavior tied to title conventions. It still misses untagged docs PRs with other titles (Bring over 13.4.4 release notes to main #3264, Omit autosummary pages in PR previews #3244, [doc-only] docs(release): tag cuda-core releases on a release branch #3431), and it needs the path check of option 1 anyway to be safe.
  3. A check that flags a mismatch between the title and the paths. For example, [doc-only] with code changes, or a docs-only change without the tag. Drawback: it adds one more check that authors must react to, and it does not save any runner time by itself.

I would do option 1, and honor [doc-only] only when the paths agree, so the tag can no longer skip tests on code changes. Option 1 fixes the problem at the planner, where the ignore list already lives, for all three packages and any title style. The [no-ci] escape hatch can stay as it is.

Activity

  1. changed the title [-]ci: docs-only PRs run the full test matrix unless the title has [doc-only] and main has a green baseline[/-] [+]ci: docs-only PRs run the full test matrix unless the title has [doc-only] or main has a green baseline[/+] on Nov 17, 2026
  2. added
    P2Low priority - Nice to have
    CI/CDCI/CD infrastructure
    and removed
    triageNeeds the team's attention
    on Nov 17, 2026
  3. self-assigned this
    on Nov 17, 2026
  4. added this to the CI milestone on Nov 17, 2026
  5. added theissue type on Nov 17, 2026
  6. lxcxjxhx commented on Nov 18, 2026

    @lxcxjxhx

    I read option 1 as the right one and implemented it on a local branch, not pushed — per CONTRIBUTING I'd like the assignment (or a different direction) before opening a PR.

    Mechanism confirmed in the code. main() computed the diff only when a reusable baseline existed:

    reusable_baseline = bool(merge_base and baseline_run_id and not release_tag)
    paths, linked_paths = _changed_paths(merge_base) if reusable_baseline else ([], set())

    so without a baseline the planner ran with paths == [] and compute_workplan took its force_all branch. The diff was never consulted, which is why an untagged docs PR gets the matrix even though select_source_work would have selected nothing.

    Change (3 files, +60/−16):

    • ci/tools/compute_ci_plan.py: compute the diff whenever merge_base and no release tag, and export doc-only from documentation_only(...), which is select_source_work asked directly (not full_rebuild and not builds and not tests and not platforms). No second copy of the ignore list.
    • .github/workflows/ci.yml: detect-changes exports doc-only; the 9 consumers read needs.detect-changes.outputs.doc-only. The [doc-only] title check is removed from should-skip, so a title can no longer skip tests on .pyx/.pyi changes.
    • ci/tools/tests/test_compute_ci_plan.py: six cases pinning docs paths → True, _device.py/_memory.pxi/ci/tools/ci_env.py/empty → False.

    force_all itself is unchanged: without artifacts every package is still rebuilt (the docs build needs build-linux-64). The paths only decide whether a test job could observe the change.

    Verification. pytest ci/tools/tests/test_compute_ci_plan.py → 38 passed, 1 failed. The failure is test_symlinked_package_inputs_are_not_mistaken_for_documentation_only on Windows (WinError 1314, symlink privilege), it fails the same way on unmodified main here, and CI runs Linux.

    End-to-end, on this clone, with BASELINE_RUN_ID="":

    docs-only, no baseline   doc-only=true    (README.md changed)
    code path, no baseline   doc-only=false   (cuda_core/cuda/core/system/_memory.pxi changed)
    

    That is the two halves of your example: #3479/#3431 would now skip, #3304/#3302 would no longer skip. A workflow-lint pass also checks that every job reading doc-only has detect-changes in needs, and that nothing still reads it from should-skip.

    Two things I could not check: whether the fork's workflow runs need approval (that decides whether a PR can show its own CI here at all), and the real job-minute saving — the 12-50 vs 1050 figures are yours from the issue.

    Should I open the PR, or would you rather see the branch first?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP2Low priority - Nice to have

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions