Skip to content

feat!: remove PR auto-merge; flag green task PRs ready for a human merge (0.10.0) - #89

Merged
stackbilt-admin merged 8 commits into
mainfrom
fix/automerge-check-runs
Oct 17, 2026
Merged

stackbilt-admin merged 8 commits into
mainfrom
fix/automerge-check-runs

Conversation

@stackbilt-admin

@stackbilt-admin stackbilt-admin commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Why

pr-automerge merged completed docs/tests/research task PRs. None of the Stackbilt repos have branch protection (GitHub's free plan for private repos), so it was the only control between a task PR and main, and it didn't work:

  • CI wasn't really checked. It read only the legacy combined-status API, which omits GitHub Actions check runs and returns {state: 'pending', total_count: 0} on an Actions-only repo. That was treated as "no CI configured" and passed, so a PR whose checks all failed could be merged. (aegis PR #819's head shows exactly that shape.)
  • Hardening didn't converge. Four adversarial review rounds on a stricter gate showed that no file category is safe to merge by rule here:
    • docs can rewrite CLAUDE.md, AGENTS.md, reviewer prompts and skills that steer every later agent;
    • Markdown under src/pages is a live page on the Astro sites, and .mdx runs at build time;
    • a new test file is code that runs on main in CI, in aegis on the same runner just before the deploy step receives the Cloudflare token;
    • research PRs could add config;
    • and a test with a write-capable token could forge an Actions check.

It has never merged anything: 740 heartbeat runs in the Stackbilt deployment, no "Auto-merged by AEGIS" commit in the org. It was stalled re-reading old closed PRs.

Change

  • Removed pr-automerge (and AUTOMERGE_SAFE_CATEGORIES).

  • Added pr-ready (every 2 h). It never merges. It labels a completed task's PR aegis:ready-to-merge when all of these hold:

    • Provenance: the head is an auto/ branch in the same repo (fork PRs, including deleted forks, are refused), and the base is the repo's default branch.
    • Mergeable: GitHub reports mergeable === true; unknown isn't enough.
    • Doesn't edit its own workflows: no changed file under .github/ (case-insensitive, including a rename's old path). A PR that edits .github/workflows/ decides which checks run on it. Refused when not every changed file could be read.
    • CI on the head SHA: every commit status is success, every check run completed with success/neutral/skipped, and at least one GitHub Actions check run succeeded (app.slug === 'github-actions'). Optional per-repo required checks (integrations.github.prReady.requiredChecks) must each be a successful Actions run. Statuses and check runs are read up to 100 each, and more is refused. Check runs use GitHub's default filter=latest, so a passing re-run replaces a failed attempt.
    • Labels follow the PR: a labelled PR that stops qualifying (e.g. a new commit fails CI), or can't be re-checked (any GitHub error), loses the label; a label applied while the head moved is taken back off; an already-labelled PR isn't re-labelled.
    • What the label means: "CI reported green on the head commit at the last check." It can lag a push by up to one run (2 h), covers the newest 25 tracked PRs (a reopened PR or one outside the window isn't re-checked), and CI is reported by code the PR can change (package scripts, test config). Re-check CI when merging. It applies to any completed task's PR, not only docs/tests/research.
    • Starvation fixed: up to 25 tracked PRs are re-evaluated per run, newest first. Closed or merged PRs, and rows whose pr_url isn't a GitHub PR, get /closed appended to the task's pr_url and stop being read. (GitHub redirects /pull/N/closed to the PR, so the link still works.)
  • github.ts, all additive:

    • getCheckRuns() (with app slug);
    • PullRequestStats.headSha, headRepo and baseDefaultBranch;
    • getPullRequestFileDetails() (status and previous name), removeLabelFromIssue() (a missing label isn't an error), PullRequestStats.labels;
    • per_page=100 on getCombinedStatus().

    The merge_pull_request and get_ci_status tools are unchanged. Known gap, filed as Scheduled LLM runs can merge PRs: merge_pull_request has no approval gate #90: merge_pull_request is available to scheduled LLM runs with no approval gate, so "nothing merges without a human" holds for this heartbeat, not yet for core as a whole.

  • README scheduled-task row updated; 0.10.0 with a Breaking note.

Tests

web/tests/pr-ready.test.ts (26):

  • ciVerdict:
    • no CI;
    • only a status, only another app's run, or only skipped runs;
    • a failed, in-progress or non-success check;
    • a missing, skipped or status-only required check;
    • pagination;
    • all green.
  • End to end with mocked GitHub:
    • a green PR is labelled, and no merge endpoint is ever called;
    • the task query reads newest first and skips finished PRs;
    • a closed PR is marked /closed;
    • no label for: a fork, a deleted fork, a non-auto/ branch, a non-default base, unknown mergeability, conflicts, a failed check next to a green one, no CI, a workflow edit (including a rename away from .github/ and a different case), more than 100 files, a missing required check. Each case asserts the logged reason;
    • a labelled PR whose new commit fails loses the label; an already-labelled PR isn't re-labelled; the label comes back off when the head moved mid-check or the PR can't be re-checked (403 on check runs); a non-PR pr_url is marked done.

19 mutations (each rule removed in turn) all fail the suite. tsc --noEmit is clean. Full suite: 1894 pass, 14 fail. The same 14 tests (recall-baseline, memory/graph, recall-mindspring, recall-pipeline) fail identically in a clean clone of main on this Windows machine, and main's GitHub CI passes them.

Release

After merge: tag v0.10.0 to publish. For the AEGIS daemon (on ^0.9.2): bump to ^0.10.0, update docs that describe auto-merge (aegis#819 does), deploy. prReady.requiredChecks is optional. If it's used, the daemon's local web/src/operator/types.ts needs the field, since the daemon types its config against a local copy. The daemon's scripts/sync-oss.sh would copy its stale files over this repo if run with --sync; it should be retired.

🤖 Generated with Claude Code

kovermier and others added 5 commits October 16, 2026 16:29
… mergeable state, pin the head SHA (0.9.5)

The gate read only the legacy combined-status API, which omits GitHub
Actions check runs and reports pending/0 on Actions-only repos; that was
treated as "no CI" and passed, so a docs/tests/research PR whose checks
all failed could be auto-merged. Now every status and check run on the
head commit must pass and at least one must exist; mergeable null waits;
the merge is pinned to the checked SHA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; skipped isn't passed

Review of #89: one green signal isn't CI passing (a job waiting on another,
a path-filtered workflow, a fast external status), and these repos have no
branch protection to name the checks that matter. The operator config now
lists required checks per repo (integrations.github.automerge.requiredChecks);
each must succeed on the head commit and a repo with no entry never
auto-merges. Commit statuses are read 100 per page and the gate refuses
when there are more; a skipped or neutral required check doesn't count.
Tests assert the logged skip reason, not just the absence of a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st be Actions runs (0.10.0)

Second review of #89: the gate matched required checks by name, and the
PR controls what runs under that name.
- Per-category file allowlist replaces the sensitive-file list: docs may
  touch Markdown and docs/ text/images, research research/ and Markdown,
  tests may only add new test files. .github/, dotfiles, existing tests and
  config mean no auto-merge. Renames are judged by both paths; refuse when
  not every changed file could be read.
- A required check counts only as a completed github-actions check run
  concluding success (not a status or another app's run).
- 0.10.0 with a Breaking note: auto-merge is off until configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stackbilt-admin stackbilt-admin changed the title fix(automerge): fail closed on CI — read check runs, require known mergeable state, pin head SHA (0.9.5) fix(automerge): fail closed — Actions checks per repo, per-category file allowlist, pinned head SHA (0.10.0) Oct 16, 2026
…rge (0.10.0)

Four review rounds on a hardened auto-merge gate showed no file category
is safe to merge by rule without branch protection: docs steer later
agents, Markdown is served on some sites, and new tests are code that runs
in CI beside deploy credentials. The pr-automerge heartbeat is replaced by
pr-ready, which labels a PR aegis:ready-to-merge when it comes from an
auto/ branch in the same repo, targets the default branch, is mergeable,
and its CI on the head commit passed (at least one successful GitHub
Actions run; optional per-repo required checks). It never merges. Tasks
are read newest first and closed PRs are marked done, ending starvation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stackbilt-admin stackbilt-admin changed the title fix(automerge): fail closed — Actions checks per repo, per-category file allowlist, pinned head SHA (0.10.0) feat!: remove PR auto-merge; flag green task PRs ready for a human merge (0.10.0) Oct 16, 2026
…re-evaluate 25 newest

Review of #89:
- A flagged PR that stops qualifying (a new commit fails CI) loses the
  label; an already-labelled PR isn't re-labelled.
- A PR touching .github/ (case-insensitive, incl. a rename's old path) is
  never flagged: editing its own workflows decides which checks run.
  Refuse when not every changed file could be read.
- Re-evaluate up to 25 tracked PRs per run; rows whose pr_url isn't a PR
  are marked done.
- Drop the unused sha parameter on mergePullRequest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad moved mid-check

Round-six review of #89: withdrawing a label failed open (an error while
re-checking left it on), and a push between the CI check and labelling
left the label on an unchecked head. Remove the label on any evaluation
error (best effort), and re-read the head after labelling. Docs now say
what the label means: CI reported green at the last check, can lag a push
by one run, and is reported by code the PR can change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stackbilt-admin
stackbilt-admin merged commit 5bf356a into main Oct 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants