Repository navigation
shape-doctor.py: every git the doctor starts runs with GIT_NO_LAZY_FETCH=1, so a read of an object a partial clone lacks fails rather than fetches, and pin_restore names the fetch where a root's partial clone lacks its pin's history (#787) - #791
Conversation
…lacks objects at, by id, from the promisor remote whose branches hold it, so it no longer loops where the pinned commit sits only on another remote or another branch, and its `why` says where the clone does not hold the commit at all (#754) RV-f's review of #742, measured with git 2.43: - S2: the NEXT was `fetch --refetch --no-filter <the first promisor remote>`, the remote whole. With the pinned commit on a second promisor remote's branches only, it exited 0 and the row came back the same; in a `--single-branch` clone whose pin is on another branch, the clone still did not hold the commit afterwards. The NEXT now names each lacking commit by id, and `remote_holding` picks the promisor remote whose tracking refs hold them all (`for-each-ref --contains`, which reads commits alone and fetches nothing). Where none does -- always for a commit the clone does not hold -- it asks the first, and the `why` names every promisor remote. - S1: where `objects_lacking` is None (the clone does not hold the commit), the `why` says so. - N1: the reason says "at least", since a missing tree hides what is under it from `rev-list`. - N2: the `why` says `--refetch` needs git 2.36 or later. tests/test_doctor_partial_clone.py: a `--single-branch` partial clone of a standard whose `main` lacks C1 (exit 3, `lacking == {C1: None}`, C1 still not held), its NEXT run as printed, two promisor remotes with C1 on the second's branches only, and the NEXT and "at least" of the existing case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hold as absent on every git, so from git 2.45 on it is no longer counted as one object lacked and handed to `for-each-ref --contains`, which fetches it (#754, round 2) PR #773's CI, on git 2.55.0, failed three tests at setup with "the clone holds C1". The clone did not hold it. From git 2.45 on, `rev-list --missing=print` takes a missing tip as missing too: it prints `?<commit>` and exits 0 (git's rev-list documentation since v2.45.0, "If some tips passed to the traversal are missing"), where git 2.43, this machine's, refuses it, `bad object`, exit 128. The tests' `holds` read exit 0 as "held". The doctor read the same command. On git 2.45 and later `objects_lacking` counted the absent commit as one object lacked, not None, so the row said "lacks at least 1 object" where the clone does not hold the commit at all, and `remote_holding` asked `for-each-ref --contains` of a commit the clone lacks, a read that fetches it. A `?<commit>` line for the commit itself is now None, as the refusal is. tests/test_doctor_partial_clone.py: `holds` takes both answers as "not held"; `elsewhere` puts C1 on its `pinned` branch only once `single_branch`'s clone exists, so no clone flag of any git can bring it in, and the precondition stays; `objects_lacking` is held to all four answers it can get, the two for a missing commit included, through the module, since one machine has one git. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re on different remotes' branches, the NEXT fetches from the remote holding the pinned commit only the commits that remote holds, and the `why` says which remote holds which, so the fetch no longer fails `not our ref` and comes back unchanged (#754, round 3) RV-f's review of #773 rounds 1 and 2 (SHOULD S2): a partial clone whose HEAD lacked objects on `origin`'s branches alone, with the pinned commit on `second`'s alone, got `fetch --refetch --no-filter origin <pinned> <HEAD>`. git refuses a fetch by id whole where the remote lacks one id, so the row came back the same, its `why` saying no branch held them. - `partial_clone_fetch` picks the remote and the commits: the first remote whose branches hold every lacking commit some remote holds, else the first holding the pinned commit, and only the commits that remote holds. The doctor run again names the rest. Where no remote's branches hold any, it is the first promisor remote and every commit, as before. - The `why` says where each commit is (`where_lacked`). Where the remote is a guess, it spells out the same fetch from each other promisor remote as a line to paste (N1). - `remotes_holding` (was `remote_holding`) lists every remote whose branches hold one commit, and is still never asked of a commit the clone does not hold. - SonarCloud S3776 and S3358 (S1): the reason, the `why` and the fetch are built by helpers, with no nested conditional, and the nested `def which` is gone (N2). Tests: `test_commits_on_two_remotes_are_fetched_from_each_in_turn` runs RV-f's case's NEXT twice to an answer, and `test_the_why_spells_out_the_fetch_from_each_other_promisor_remote` runs the `why`'s command for a never-fetched `second`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ETCH=1`, so a read of an object a partial clone lacks fails rather than fetches, and `pin_restore` answers `partial` and names the fetch where a root's own partial clone lacks its pin's history (#787) "It never fetches" held read by read: #683 and #754 ask a partial clone of the standard with `rev-list --missing=print` before anything reads it. A read nothing asked about first still fetched. Measured with git 2.43: a partial clone of a ROOT whose HEAD commit broke its pin made one lazy fetch (`blob:none`) or two (`tree:0`), when `pin_restore` walked the pin's history. - `lazy_fetch_off` sets `GIT_NO_LAZY_FETCH=1` in this process's environment for the run of `main`, and puts it back after. Every git inherits it: the doctor's own reads, `update-shape.py`'s `Upstream` as the doctor drives it in-process, the validators and their gits. No printed line carries it, so a NEXT `fetch --refetch` still fetches. git 2.45.0 and later honour it, and the 2024-05-14 security releases back to 2.39.4 (CVE-2024-32465); an older git ignores it, harmlessly. - With it, `pin_restore`'s walk read a failed read as "no pin there" and said `none`: no commit holds a pin to put back, which is not so. `history_left_out` asks, without fetching, whether a partial clone lacks the pin's history. Where it does, `restore` is `partial`, and the NEXT is `git -C <root> fetch --refetch --no-filter <remote>`. Run as printed, the doctor run again names the commit (`committed`). Tests (`tests/test_doctor_no_lazy_fetch.py`): - no row starts a fetch from a partial clone of the standard (trace2); - the report is byte-identical with the variable ignored by every git, on a full clone and on two partial clones of the standard; - a partial root whose pin history is not here answers `partial`; - `lazy_fetch_off` sets the variable and puts it back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideThe PR makes the doctor’s no-fetch guarantee structural by disabling Git lazy fetching for every process it starts, adds a Sequence diagram for no-lazy-fetch doctor executionsequenceDiagram
participant User
participant Doctor as shape-doctor.py
participant Git as Git child process
participant Remote as Promisor remote
User->>Doctor: main(argv)
activate Doctor
Doctor->>Doctor: lazy_fetch_off()
Doctor->>Git: git read with GIT_NO_LAZY_FETCH=1
Git-->>Doctor: object data or read failure
Git--x Remote: no lazy fetch
Doctor->>Git: start validator with inherited environment
Git-->>Doctor: validator result
Doctor-->>User: report and optional NEXT command
Doctor->>Doctor: restore previous environment
deactivate Doctor
Sequence diagram for partial pin-history diagnosissequenceDiagram
participant Doctor as pin_restore
participant Git as Git rev-list
participant Root as Partial root
participant User
participant Remote as Promisor remote
Doctor->>Git: rev-list --objects --missing=print -n 51 HEAD -- pin
Git->>Root: inspect pin history with lazy fetching disabled
Root-->>Git: missing object marker or read failure
Git-->>Doctor: history is unavailable
Doctor->>Doctor: history_left_out(root, pin)
Doctor-->>User: restore=partial and NEXT fetch command
User->>Remote: git fetch --refetch --no-filter
Remote-->>Root: restore missing history
User->>Doctor: run doctor again
Doctor->>Git: walk restored pin history
Git-->>Doctor: identify pin commit to restore
Flow diagram for partial-clone finding and recoveryflowchart TD
A[Doctor checks pin restore] --> B[pin_restore walk finds no readable pin]
B --> C[history_left_out runs rev-list --missing=print]
C --> D{Missing history or failed read?}
D -- No --> E[Return restore none]
D -- Yes --> F[Return restore partial]
F --> G[Print git fetch --refetch --no-filter]
G --> H[User runs printed fetch]
H --> I[User runs doctor again]
I --> J[Doctor reads complete history and names pin commit]
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
(lane openRepoShape-3, 2026-10-17T10:26:57Z) Independent review of #791 by RV-f (opus), on Rulings on the risks:
NITs:
Runs:
CI on git 2.55 and SonarCloud are green. Lane: openreposhape-3 (openRepoShape-3) |
…ncluded, is not all in the clone, and spells out the fetch from each other promisor remote; a test records that every git a doctor run starts, its validators' and `Upstream`'s included, has `GIT_NO_LAZY_FETCH=1` (#787, RV-f's NITs on #791) - N1: "which commit broke the pin cannot be read here" was not so where HEAD was moved back without a checkout. There the doctor run after the fetch answers `uncommitted`. The `why` now says "the pin's history, HEAD's included, is not all in this clone" (`partial_history_why`), and a `reset --soft` case is tested. - N2: `test_every_git_the_run_starts_has_the_variable` puts a recording `git` first on `PATH`. Every git the run starts has the variable: `validate-pins.py`'s, started by `run_validator`, and `Upstream.blob`'s `cat-file blob` as the doctor drives it in-process. - N3: `history_left_out` returns every promisor remote. The `why` spells out the same fetch from each one after the first, where the history is on another remote's branches alone. Tested with `origin` naming another repository: the NEXT leaves the row the same, and the `why`'s `second` line ends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
(lane openRepoShape-3, 2026-10-17T12:00:24Z) Re-verification of #791 round 2 by RV-f (opus), on head 1. The shim test cannot pass vacuously (
2. Each printed fetch line pastes and works. RV-f built a
3. Nothing else changed since 4. Merges:
CI: run 39026676188's one error, NIT: git 2.43's Runs:
Lane: openreposhape-3 (openRepoShape-3) |
|
|
Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 2 days and 11 hours by commenting |
…and its round 2, and run to main 33c5128 (#883) Round 2b of #883. It answers RV883's full NIT list on ede678c (lane 3, 2026-10-18T09:29:28Z) and RV883's round 2 on 86baec5 (#883 issuecomment-6248362246). Each finding was checked against the code at main 33c5128 before the text changed: - The #843 limit is two: what `adopt-work-dir-shared` leaves open (group write, Windows, root as the other user, this user's own processes, the default work directory), and what it now refuses that v1.1.0 used (another user's directory, a run as root included, a 0777 directory without the sticky bit, a drive mounted without ownership), with the way out. "What a script sees" names `adopt-work-dir-shared` and `adopt-work-dir-occupied`. - The doctor's NEXT lines, not its `members` row, read no url from an unmerged `.gitmodules`; the `make siblings` limit names that row. `siblings.py` has no unmerged rule; the `validate.yml` templates do. - `validate-family.py`'s `family ok` is scoped to `--no-members` and to a holder whose members are checked out. - The wording points: `update-pr-no-repository` under `--pr`, family.py's `git rm -r --cached` line, `MSYSTEM` set and not empty, `..` and `naming-unclassified`, the keys git refuses with no value, the url the root's config keeps, `contracts: cannot be listed`, install.sh's newest tag and redirect location, the scan test's name, `update-rewrite-symlink`, `os.access` for root, `LC_ALL=C`, `update-validators-red`, `heads/main` in both keys, where `plan written to` came, junctions on Python 3.12, the approval click and the run's approvals, and the back-port route's one run. - #457's L1-17 is narrowed by #825; #853 and #854 join #834 as test limits. The refresh past c5359df adds #870 (#864), #876 (#839), #791 (#787), #859 (#838), #780 (#776), #873 (#857) and #875 (#874): 57 pull requests. It drops the limits for #776, #839 and #874's ref rule, rewrites the #838 clauses, and adds the limits those squashes name (#863, #879, #884, an older git without `GIT_NO_LAZY_FETCH`, family.py's link guard). The open list is the 14 that the 09:42Z milestone read leaves after #875: #675, #707, #739, #755, #764, #834, #840, #853, #854, #856, #863, #878, #879 and #884. The six-and-six shape-copy lists stand: from c5359df to 33c5128 the templates and repo_shape.py changed only in the family root's bootstrap.py and validate-family.py, both listed. No other file changes. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



Lane: openreposhape-3 (openRepoShape-3)
Claim: #787 (comment)
Closes #787.
Stacked on #773; lands after it. The base is
main, so until #773 lands this diff also shows #773's commits (l3-754-promisor-next, head296a6dd6). This PR's own changes are the three commits on top:152e9c39;19665420, which moves one function ("Merging with main" below); and1d774ab3, the NIT round ("Round 2 (NIT)" below).Lands after the v1.1.0 tag.
Round 2 (NIT) (head
1d774ab3, on19665420)RV-f's review (comment) was READY with three NITs. This round takes all three.
partialwhy. It said "which commit broke the pin cannot be read here", which is wrong where HEAD was moved back without a checkout: there the doctor run after the fetch answersuncommitted. It now says "the pin's history, HEAD's included, is not all in this clone" (partial_history_why,shape-doctor.py:4802, wording at:4820).pin_restore's docstring entry (:3850) says the same.reset --softcase, which answersuncommittedafter the fetch (PARTIAL_ROOTS,tests/test_doctor_no_lazy_fetch.py:164).test_every_git_the_run_starts_has_the_variable(:242) puts a recordinggitfirst onPATH(recording,:222). For each git the run starts, it logs the variable's value, the parent's command line and the arguments. Every one has1, including:validate-pins.pystarts, whichrun_validatorlaunched;Upstream.blob'scat-file blob <commit>:templates/…, which the doctor drives in its own process.history_left_out(:4834) returns every promisor remote, not the first. Thewhyadds "where<first>'s branches do not hold it, the same from<name>:git -C … fetch --refetch --no-filter <name>" for each other one (:4825).test_the_why_spells_out_the_fetch_from_each_other_promisor_remote(:200) pointsoriginat another repository and adds the history's remote as an unfetched promisorsecond.partial. Thewhy'ssecondline, run as printed, givescommitted.Each helper sits beside
history_left_out, afterremotes_holding, so the branch still merges clean with mainbf1f099b.Mutants (each run against
tests/test_doctor_no_lazy_fetch.py):mainnot underlazy_fetch_off0partialanswertree:0)run_validatordrops the variablewhywordswhyhistory_left_outreturns the first remote onlyResults on
1d774ab3, with the same stubghand command as below:tests/test_doctor_no_lazy_fetch.py,tests/test_doctor_pin_restore_merges.py,tests/test_doctor_partial_clone.py,tests/test_repo_hygiene.pyandtests/test_cli_reference.py, together:162 passed in 204.36s (0:03:24).tests/test_shape_doctor.py -k "pin or partial":88 passed, 1 skipped, 351 deselected in 217.43s (0:03:37).1d774ab3into mainbf1f099b,tests/test_doctor_no_lazy_fetch.py,tests/test_doctor_pin_restore_merges.pyandtests/test_doctor_partial_clone.py:69 passed in 202.20s (0:03:22).What was measured first (base
296a6dd6, git 2.43.0)A probe ran the doctor,
--jsonand the table, underGIT_TRACE2_EVENT, which logs every git process and everyfetchchild it starts. Each of 14 scenarios ran twice: as is, and with agitshim first onPATHthat unsetsGIT_NO_LAZY_FETCH, which is the doctor before this change.A partial clone of the standard: no row fetches today. That covers
blob:noneandtree:0, behind the pin or pinned at its own HEAD, with the remote in reach or not, plus two full clones. In all ten scenarios:--jsonand the table.The guards from #683 and #754 already cover every read there. For the standard, this change makes the promise hold by construction rather than read by read.
A partial clone of the ROOT whose HEAD commit broke its pin: the doctor fetched.
pin_restorewalks the pin's history, and today that made 1 lazy fetch (blob:none) or 2 (tree:0):git -c fetch.negotiationAlgorithm=noop fetch origin … --stdin.What changed (
shape-doctor.py)lazy_fetch_off(:10100, the variable's nameNO_LAZY_FETCHat:10096) is a context manager. It setsGIT_NO_LAZY_FETCH=1in this process's environment and puts back what was there on the way out.mainruns under it (@lazy_fetch_off(),:10146). Every git child inherits it:git_out(fromscripts/repo_shape.py, which is unchanged),git_answer, and its directsubprocess.runof git;update-shape.py'sUpstreamreads (cat-file,ls-tree,rev-parse,tree_digest), which the doctor loads and drives in this process;run_validatorstarts with a copy ofos.environ, and their gits.fetch --refetch --no-filterstill fetches when pasted.:10128):environment.c. v2.45.1 and later, up to v2.55.0 (CI's), read it inpromisor-remote.c'sfetch_objects, with the warning "lazy fetching disabled; some objects may not be available".1:2.43.0-1ubuntu7.1and later carry that patch (CVE-2024-32465.patch), which is why this machine's 2.43.0 honours it. Measured:cat-fileof a missing blob gives exit 128, the warning, "could not fetch … from promisor remote", and no fetch child.pin_restorehas a new answer,partial(:4062, documented at:3850). Where the walk finds no pin to put back,history_left_out(:4834) asks without fetching whether the root is a partial clone that lacks the pin's history:rev-list --objects --missing=print -n 51 HEAD -- <pin>;?<id>line means it lacks the history; so does a failure, which is what atree:0clone gives, since that walk cannot read the trees.restoreispartialand the NEXT isgit -C <root> fetch --refetch --no-filter <first promisor remote>. Thewhy(partial_history_why,:4802) says the pin's history, HEAD's included, is not all in this clone, and spells out the same fetch from each other promisor remote (round 2).manifest kindsrow's files too, since that row uses the samepin_restore.:31) says every git the doctor starts runs with the variable.Pinned copies: none.
shape-doctor.pyis not copied into projects, andscripts/repo_shape.pyandupdate-shape.pyare unchanged.Docs: no change. README, AGENTS.md and the handbook say the doctor fetches nothing, which now holds by construction. No doc lists
pin_restore'srestorevalues, anddocs/cli.mdtakes only the docstring's first line.Tested (
tests/test_doctor_no_lazy_fetch.py, new)test_every_row_from_a_partial_clone_of_the_standard_starts_no_fetch(:88;blob:noneandtree:0, remote out of reach and in reach):fetch(trace2child_start);rev-list --objects --all --missing=printis the same before and after.test_the_report_is_the_same_byte_for_byte_with_the_variable_ignored(:131): the full-clone control, plusblob:noneandtree:0partial clones of the standard.--jsonand the table are run as is and through the shim, and stdout, stderr and the exit are the same.test_a_partial_root_whose_pin_history_is_not_here_names_the_fetch(:172;blob:none,tree:0, and HEAD moved back byreset --soft):restore == "partial";whysays the pin's history, HEAD's included, is not all in this clone;git -C <root> fetch --refetch --no-filter origin;committed, oruncommittedwhere HEAD was moved back.test_the_why_spells_out_the_fetch_from_each_other_promisor_remote(:200, round 2).test_every_git_the_run_starts_has_the_variable(:242, round 2).test_lazy_fetch_off_sets_the_variable_and_puts_it_back(:266): with the variable unset, and with it set to0.Mutants (each run against
tests/test_doctor_no_lazy_fetch.py):mainnot underlazy_fetch_offcommitted)0partialanswernone)tree:0Rows on
tests/test_shape_doctor.py: that file makes no partial clone (--filteroccurs 0 times), so the variable cannot change a git answer there.296a6dd6:436 passed, 3 skipped, 1 xfailed in 327.40s (0:05:27).436 passed, 3 skipped, 1 xfailed in 331.04s (0:05:31).Run with a stub
ghfirst on PATH,GH_TOKEN=bogusand an emptyGH_CONFIG_DIR:Results:
152e9c39, the seven filestests/test_doctor_no_lazy_fetch.py,tests/test_doctor_partial_clone.py,tests/test_update_shape_merged_away.py,tests/test_repo_hygiene.py,tests/test_cli_reference.py,tests/test_doctor_pin_restore_merges.pyandtests/test_doctor_next_is_a_command.py, together:230 passed in 175.17s (0:02:55).152e9c39,tests/test_shape_doctor.py:436 passed, 3 skipped, 1 xfailed in 331.04s (0:05:31).19665420(the move), the same seven files:230 passed in 167.23s (0:02:47).19665420into mainbf1f099b, the same seven files:236 passed in 183.89s (0:03:03).tests/test_shape_doctor.py:436 passed, 3 skipped, 1 xfailed in 308.14s (0:05:08).19665420(run 39021769167, git 2.55.0, onpull/791/mergeb558d917):testsgives11556 passed, 144 skipped, 1 xfailed in 2295.67s (0:38:15).tests-git-bash,packageand SonarCloud pass. SonarCloud's analysis of19665420: 0 open issues, quality gate OK, 0.0% new duplication.Merging with main
The first push,
152e9c39, conflicted with mainbf1f099binshape-doctor.py, so GitHub ran no CI.other_sides_copies(shape-doctor.pypin_restore: a copy conflicted beside the shape pin that only the side NOT taken has afiles:row for is neither taken nor named, so the conflicted pin'swhyandleft_for_a_personsay nothing of a file the merge still holds unmerged #687) took the place right afterpin_restore, where this PR had puthistory_left_out.19665420moveshistory_left_outnext to the other partial-clone readers, afterremotes_holding. It is a pure move: every line removed is added back unchanged.bf1f099b, and so does shape-doctor.py: the partial-clone row's NEXT fetches each commit it lacks objects at, by id, from the promisor remote whose branches hold it, so it no longer loops where the pinned commit sits only on another remote or another branch, and itswhysays where the clone does not hold the commit at all (#754) #773's head.git merge-treewrites forbf1f099band19665420, run as a detached worktree of this branch's clone.🤖 Generated with Claude Code
Summary by Sourcery
Ensure shape-doctor never lazily fetches missing objects and identify partial clone pin-history gaps with actionable refetch commands.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests: