Repository navigation
fork-ci pr-queue 2/4: the queue's merge path (pr_queue.py) - #19
Conversation
|
lens: security SEC-2 · reviewer: opus high (t3lens-stack-sec2) · read at: 2d1c5b4 |
|
lens: security SEC-1 · reviewer: opus high (t3lens-stack-sec1) · read at: 2d1c5b4 |
2d1c5b4 to
2e661e1
Compare
|
lens: security SEC-1 · reviewer: opus high (t3lens-stack-sec1) · read at: 2e661e1 |
|
lens: security SEC-2 · reviewer: opus high (t3lens-stack-sec2) · read at: 2e661e1 |
2e661e1 to
5e94aa7
Compare
da09388 to
a7c8b7e
Compare
|
lens: security SEC-1 · reviewer: opus high (t3lens-stack-sec1-r3) · read at: 5e94aa7 |
|
lens: design · reviewer: opus high (t3lens-stack-design) · read at: 5e94aa7 |
|
lens: correctness (#19) · reviewer: opus high (t3lens-18-19-corr) · read at: 5e94aa7 |
|
lens: security SEC-2 · reviewer: opus high (t3lens-stack-sec2-r3) · read at: 5e94aa7 |
5e94aa7 to
efae48e
Compare
|
lens: tests · reviewer: opus high (t3lens-stack-tests) · read at: 5e94aa7 |
|
lens: correctness (#19) · reviewer: opus high (t3lens-stack-r4-corr) · read at: efae48e |
|
lens: security · reviewer: opus high (t3lens-stack-r4-sec) · read at: efae48e |
1f3382b to
d104868
Compare
efae48e to
1b5250b
Compare
One run brings the oldest armed pull request that is BEHIND its base up to date: auto-merge off, gh pr update-branch, and auto-merge back on with --match-head-commit only for a new head that holds exactly the change the review read merged with the base (git merge-tree --write-tree, tree for tree). Anything else leaves it off with a comment. The plan is on disk (--state-file) before the disarm; a run stopped half-way is finished by the next run first, against the review as it is then, and no other pull request is touched until it has been re-armed or told why not. Ported from the source project's queue with its tests. Project facts come from the config; STATE names its repository; exit 3 (was 2) is "may be off and silent" and 2 is "refused to start"; the adoption of an older queue's state file is gone. The queue's own advice keeps its backticks and <sha> placeholder (lens.no_lens_text keep_markdown), which the source stripped from comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security review fixes for the queue's merge path. - The review counts only comments of lens_author_associations, shown and unedited (the lens.Review rule from 1/4 is wired in). A stranger's lens comment neither vouches for a head nor disarms one. - git runs with no hooks, no fsmonitor and no replace refs on every call. The cache is used only as the queue made it: a directory of this user with mode 0700, a bare repository, and the queue's own config. Anything else is set aside and rebuilt; a new cache gets no template hooks. - STATE is written 0600 (O_NOFOLLOW). recover reads it only if this user owns it and no group or other may write it; anything else is kept as STATE.bad, loudly, like an unreadable plan. - recover checks the change against the base GitHub shows now, not the base stored in the plan (the pull request may have been retargeted). - A short read gets the way out in the comment: re-post it with the full sha. - README: the queue's runner runs no pull_request job and no other repository's job, and its state and cache folders are its user's alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a cache with grafts or alternates The queue's comment for a PR left off by a hidden lens comment names the way out (that lens posts again, or the override, also for a lens no longer used). The cache is rebuilt when it holds info/grafts, shallow or objects/info alternates, which change what its objects say and the queue never writes. Tests for the round-2 security findings: Sec2 and Sec1R2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ache (SEC-2 r3 minor) A planted objects/info/commit-graph(s) could change the parents git reports; core.commitGraph=false joins the options every git call gets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1b5250b to
bb10beb
Compare
Plan row 14, part 2 of 4: the queue's merge path,
fork-ci/actions/pr-queue/pr_queue.py. One run brings the oldest armed pull request that isBEHINDits base up to date. It turns auto-merge off and runsgh pr update-branch. Then it turns auto-merge back on (with--match-head-commit) only for a new head that holds exactly the change the review read, merged with the base. Anything else leaves auto-merge off with a comment that names the way out. Usage, exit codes andSTATE: README, "The queue". This is the tier-3 part: it merges and comments.Stack (each PR's base branch is the one before it; review and merge them in order):
fork-ci/pr-queue-lens→main: the lens review rules and a strict configfork-ci/pr-queue-core→ 1/4: the queue's merge path (pr_queue.py)fork-ci/pr-queue-sweep→ 2/4: armed pull requests that cannot merge are told and passed over (SkyRPG [Feature]: escape button to go back to previous thread from settings page pingdotgg/t3code#1350)fork-ci/pr-queue-workflow→ 3/4: "red only because the base was red" (SkyRPG Fix plan composer layout by preventing flex overflow in chat/sidebar pingdotgg/t3code#706 / feat: Codex OpenAI API key and base URL overrides from Settings pingdotgg/t3code#1174),action.yml,tested-base/and the reusable workflow.github/workflows/fork-pr-queue.ymlOnce all four are open, a comment links them.
Why this approach
git merge-tree --write-tree) and compares the result, tree for tree, with the new head. It then re-arms pinned to that head (--match-head-commit). A force-push, a web conflict fix or an applied suggestion is a different tree, so it stays off with a comment.lens_author_associations(1/4) count, and only when shown and never edited. Anyone else's comment is ignored. It neither vouches for a head nor disarms a pull request.--state-fileholds{repo, pr, old, base, read}, written and synced before the first GitHub write. A run that stops half-way is finished by the next run first: against the review as it is then, and before any other pull request is touched. A pull request is never left off and silent.--cache), with credentials fromgh auth git-credentialonly, never a helper the machine configured. Every git call runs with no hooks, no fsmonitor and no replace refs, because the token is in git's environment. The cache is used only as the queue made it: mode0700, this user's, its own config.STATEis written0600and read only if no one else could have written it. The runner must run nopull_requestjob and no other repository's job (README, "The queue"). It refuses to start (exit 2, touching nothing) when the runner is GitHub-hosted, a path is relative or inside the job's workspace,--repois notowner/name, the config is bad, or git is older than 2.38.Rejected alternatives
merge_group): it needs an organization-owned repository, and both the fork and SkyRPG are user-owned.pr_queue.py: see Size.What it replaces in SkyRPG (the switch-over is row 15, not this PR)
At
dd7244bd7e,scripts/skyrpg-dev/pr_queue.py::64-224(state paths,gh/git, the own clone,fetch,same_change,holds),:317-543(viewthroughrecover),:565-594(main,dry_run) andrun()'s update path (:595-627, without the red and stuck branches, which are 3/4) becomepr_queue.py.LOCK:48,STATE:51andCACHE:54(paths fixed for one machine) become--lock-file,--state-fileand--cache.adopt_old_state(:456-471), which adopted an older queue's state file. That migration is SkyRPG's.start()andcancel_superseded(:544-563). They cancel SkyRPG's own queuedci.ymlruns to free its build runners, so they belong to that project's runners, not the queue.scripts/skyrpg-dev/tests/test_pr_queue.pyUpdate:54,Main:203,ReviewedHead:267,QueueKeepsTheReadChange:304,Recover:333, plus the queue cases oftest_lens542_r2_di1.pyandtest_lens542_r3_di1.py(AStopHalfWayLeavesATrace,WhitespaceIsPartOfTheChange,AMovedHunkIsAnotherChange,PrefixesOfOneShaAreOneHead,AFailedRecoverIsNotDropped). All are ported under the same names inpr_queue.test.py, exceptPrefixesOfOneShaAreOneHead. It becameShortReadsNameNoHead, because a read must now be the full sha (1/4).What the port changed: everything project-specific comes from the config;
STATEnames its repository; exit 3 (was 2) means "may be off and silent", so that 2 can mean "refused"; and the queue's own advice keeps its backticks and<sha>placeholder (lens.no_lens_text(keep_markdown=True)), which the source stripped from comments. After the security review, the port also differs from the source in these ways: only write-access lens comments count, git runs with no hooks, the cache andSTATEmust be this user's alone, andrecoveruses the base GitHub shows rather than the stored one.State: what happens when this runs again, stops half-way, or meets existing/old/empty state?
The writes are: the lock,
STATE, the cache, auto-merge off,update-branch, auto-merge on, and a comment.Main.test_a_held_lock_runs_nothingSTATEwas written before the disarm. The next run re-arms at a checked head, or leaves it off with a comment, and touches no other PR until thenAStopHalfWayLeavesATrace.test_killed_after_the_disarm_then_the_next_run_re_arms_or_says_it_is_off,Recover.test_an_unfinished_recover_ends_the_run,Recover.test_the_plan_survives_a_new_process_and_an_emptied_lock_folderUpdate.test_a_failed_disarm_means_no_update,Update.test_rerun_after_any_stop_touches_nothingUpdate.test_a_new_head_with_other_content_stays_offqueue empty, exit 0, nothing writtenMain.test_an_empty_queue_does_nothinggh pr update-branchfails, or the re-arm (gh pr merge --auto --match-head-commit) failsSTATEis removed once the comment is postedUpdate.test_failed_update_or_arm_stays_offwait_secondsSTATEstays, and the next run checks the head GitHub shows thenUpdate.test_no_new_head_in_time_keeps_the_plan_for_the_next_runrecoverreads the review as it is now, not the stored readRecover.test_a_lens_posted_after_the_stop_countsRecover.test_a_lens_comment_edited_after_the_stop_reads_nothing_until_that_lens_posts_again,WhoseCommentsCount.test_an_edited_lens_comment_does_not_countlens_author_associations)WhoseCommentsCount.test_a_strangers_lens_comment_never_moves_the_read_head,WhoseCommentsCount.test_a_strangers_junk_comment_cannot_disarm_a_reviewed_pr,WhoseCommentsCount.test_a_lens_comment_whose_author_does_not_count_is_named_in_the_run_logWhoseCommentsCount.test_a_hidden_lens_comment_reads_nothing,Sec2.test_all_lens_comments_hidden_as_outdated_does_not_vouch_for_an_unread_push,Sec2.test_a_hidden_override_does_not_count,Sec1R2.test_a_hidden_override_vouches_for_nothingShortReadsNameNoHead.test_a_short_read_of_the_head_is_left_off_with_the_way_outrecoverchecks against the base GitHub shows now, not the stored oneRecover.test_the_base_comes_from_github_not_from_the_planSTATEstays and the next run says it, onceUpdate.test_a_failed_comment_keeps_the_state,Recover.test_a_gh_error_keeps_the_state_and_says_so_once,Recover.test_a_fetch_error_that_does_not_heal_says_so_onceSTATE(the source's format, norepo)Recover.test_a_plan_without_repo_from_the_source_queue_is_finishedSTATE(two queues on one file)Recover.test_a_plan_of_another_repository_is_refused_and_touches_nothingSTATESTATE.bad. The run fails, and every later run warnsRecover.test_an_unreadable_state_is_kept_aside_and_never_silentSTATEanother user could have written (another owner, or writable by group or others; also an older queue's file with a looser mode)STATE.bad, the run fails, nothing is re-armed. A newSTATEis0600Recover.test_a_plan_another_user_could_write_is_kept_aside_and_not_followed,Recover.test_the_plan_is_this_users_alonegit initor fetch)Recover.test_a_stale_ref_lock_in_the_cache_is_removed,Recover.test_a_half_made_cache_with_an_undeletable_file_is_set_aside,Recover.test_leftovers_are_removed_by_the_next_rebuildinfo/grafts,shallowor alternates)0700with the queue's config and no template hooksOwnCache.test_a_planted_hook_never_runs,OwnCache.test_a_cache_with_another_config_or_open_to_others_is_rebuilt,OwnCache.test_a_cache_with_grafts_shallow_or_alternates_is_rebuilt,OwnCache.test_a_new_cache_is_this_users_alone_with_the_queues_configSTATE, no cache, no commentUpdate.test_dry_run_changes_nothing,Update.test_dry_run_leaves_an_existing_cache_as_it_was,Main.test_a_dry_run_takes_no_lock_and_shows_a_stopped_planRound 2 (security reviews SEC-1 and SEC-2, round 2)
Rebased onto
mainafter #15 and #16 merged (they changed the same test runner): the runner ismain's (presentTests),.github/fork-ci-tests.txtlists the tests of both, and this stack's end-to-end runner test stays besidemain's. The lens fixes are in 1/4. Here:Sec2.test_all_lens_comments_hidden_as_outdated_does_not_vouch_for_an_unread_pushSec2.test_a_hidden_override_does_not_count,Sec1R2.test_a_hidden_override_vouches_for_nothingWhoseCommentsCount.test_a_lens_comment_whose_author_does_not_count_is_named_in_the_run_logcache_problemdid not checkinfo/grafts,shalloworobjects/info/alternatesobjects/info/http-alternates) is set aside and rebuilt. The README still says a same-user job can change the cache after the check: only the runner rule stops thatOwnCache.test_a_cache_with_grafts_shallow_or_alternates_is_rebuiltWhat happens when this runs again, stops half-way, or meets existing/old/empty state? No new write step. The cache rebuild is the existing set-aside-and-rebuild path with three more reasons: a rerun finds the new cache clean and uses it; a rebuild stopped half-way leaves
CACHE.new, which the next run sets aside like before; an old cache from an earlier queue version holds none of these files and is kept. A PR left off because of a hidden lens comment gets one comment, and the next run reads the review as it is then (a new comment of that lens, or an override, re-arms it through the same-change check).Round 3 (SEC-2 round 3, #19 (comment))
lens.py): a stranger's huge comment stalled every runobjects/info/commit-graph(s)could change the parents git reportscore.commitGraph=falsejoinsGIT_SAFE(every git call)OwnCacheargv test pins itGeneral vs project
STATEplan and its recovery, the cache and its repair, the lock, the refusals and the exit codes.STATE, the cache and the lock live, and the runner.Tests
pr_queue.test.py: 98 tests on real commits in a temporary world. A local bare repository stands in for GitHub, andghis a fake. A guard fails the whole run if anything starts the realgh.lens.test.py: 33.node .github/scripts/fork-ci-tests.cjsexits 0 at this commit, run from a clean copy withGH_TOKEN/GITHUB_TOKENunset and an emptyGH_CONFIG_DIR.Size
pr_queue.pyis 555 lines;lens.pychanges by 18. That is above the ~400-line rule, and there is no split, for these reasons:pr_queue.py.There are also 1292 test lines.
Round 4 (last round) and merge
r4 lenses: correctness SAFE (#21 (comment)), security SAFE (#21 (comment)), at heads 1f3382b/efae48e8/5480bf0a/11e9ca28. Then rebased on main after #27 merged: the only conflict was the sorted test list
.github/fork-ci-tests.txt(both sides added lines; kept both). All suites pass at each new head.Minors left at merge (follow-up row with the other stack minors):
pr_queue.py:64: the cache-rebuild file list lacksinfo/attributes(a planted* merge=unionturns a conflict clean; only a same-user job could plant it, which the runner rule excludes).🤖 Generated with Claude Code