From c2f288483c5e6693fe20945842b32635fa9f4551 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Thu, 15 Oct 2026 22:59:25 +0000 Subject: [PATCH 01/15] validate-family.py and family.py: a row's gitlink that no section of HEAD's `.gitmodules` gives is `member-gitmodules-absent`, so a holder whose HEAD lost the file no longer says `family ok`, `remove` refuses it before `deinit` instead of leaving the mount empty, and the gitlink remedies name a case variant and the commit `add` needs (#611) 571(b). In a holder whose HEAD lost `.gitmodules` (the commit #407 describes), `family.py remove --dry-run` exited 0, and the real run did `git submodule deinit` and `git rm`, then died at its commit's `git --literal-pathspecs add -f -- .gitmodules family.yaml` (`pathspec '.gitmodules' did not match any files`); the rollback left `members/

` empty. `_checked_gitmodules_held` now refuses that holder, before any git command that writes and under `--dry-run` alike: no `.gitmodules` in the work tree, the index or HEAD while the index records the member's gitlink. Refused, not committed without the file, as #407 refuses a `.gitmodules` the work tree lacks. The remedy writes back, with `git config -f .gitmodules`, the section `family.py add` writes for each member whose gitlink the index records, then `git add -f .gitmodules`; `remove` then runs as it always has. 571(c), #407's decision 3. `validate-family.py` reads HEAD's `.gitmodules` (`_head_gitmodules`, HEAD's root tree listed with no pathspec) and makes `member-gitmodules-absent` for each row at `members/` whose gitlink no section gives, in every mode, `--no-members` included. Its two lines write the section back as `family.py add` writes one, keeping the url of a section of that name that has one. A member it names that is not checked out is not refused `member-uninitialized`: that refusal named a `git submodule update` that exits 0 there and dropped the finding, which is what a fresh clone and CI met. It reads HEAD, not the work tree, so a deletion nobody committed, an unstaged edit and a sparse checkout that leaves the file out are no finding, and a holder with no members is asked nothing. 574/2 (#363's item 3). A row whose gitlink HEAD records only at a case variant of its path (`Members/

`) is refused `member-gitlink-absent` naming each variant, and the move that puts it back in the index alone: `git rm -r --cached` of the variants and `git update-index --add --cacheinfo 160000,,members/

`, then a commit. `family.py add`, the plain refusal's remedy, cannot help there: the row exists. 574/3. A hand mount's `member-gitlink-unnamed` remedy says to commit the `git rm` before the `add`: since #574, `add` refuses a path under another spelling of `members/` that HEAD holds and the index no longer does. `templates/family-root/scripts/validate-family.py` is a pinned copy: each holder sees it as `upstream-changed` on its next `update-shape.py check`. Tests: tests/test_family_head_lost_gitmodules.py (new, 24); the pinned remedy text and the case-variant states in tests/test_validate_family_unnamed_mounts.py; a docstring in tests/test_family_remove_gitmodules.py. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- scripts/family.py | 73 ++- .../family-root/scripts/validate-family.py | 214 ++++++- tests/test_family_head_lost_gitmodules.py | 596 ++++++++++++++++++ tests/test_family_remove_gitmodules.py | 12 +- tests/test_validate_family_unnamed_mounts.py | 34 +- 5 files changed, 902 insertions(+), 27 deletions(-) create mode 100644 tests/test_family_head_lost_gitmodules.py diff --git a/scripts/family.py b/scripts/family.py index a3eb0e08..85b2f87a 100755 --- a/scripts/family.py +++ b/scripts/family.py @@ -5209,7 +5209,8 @@ def _checked_gitmodules_as_staged(root: Path, verb: str) -> None: its line endings is refused here as `git rm` refuses it, and the refusal says so where `git add` would stage the index's own blob from it. With neither the index, HEAD nor the work tree holding a `.gitmodules`, - nothing is asked. + nothing is asked here; `remove` then asks `_checked_gitmodules_held` + (#611). `add` LEAVES A MISSING FILE TO GIT. `git submodule add` refuses to write a `.gitmodules` the work tree lacks while the index or HEAD holds one, @@ -5403,6 +5404,73 @@ def _checked_one_section(root: Path, project: str, path: str) -> None: f"{GITMODULES}` — and run the same command again.") +def _checked_gitmodules_held(family: Family, path: str) -> None: + """`member-gitmodules-absent`: `remove` refuses, before any git command + that writes and under `--dry-run` alike, a holder that holds no + `.gitmodules` at all, in the work tree, the index or HEAD, while its + index records the member's gitlink (#611). + + A HOLDER WHOSE HEAD LOST `.gitmodules`, as #407's commit left one. `git + rm` edits only a `.gitmodules` the work tree has, and `remove` commits + the file as `git rm` leaves it, so with none anywhere the commit's `git + add` died (`pathspec '.gitmodules' did not match any files`) after + `deinit` and `git rm` had emptied the member, and the rollback left its + directory empty; the dry run exited 0. `_checked_gitmodules_as_staged` + asks nothing here, since no copy exists to compare. + + REFUSED, NOT COMMITTED WITHOUT IT, as #407 refuses a `.gitmodules` the + work tree lacks: the holder is left byte for byte as it was. A clone + checks out none of its members, and the holder's own + `validate-family.py` finds `member-gitmodules-absent` for each, so the + remedy writes back, as `family.py add` writes one, the section of every + member whose gitlink the index records, this one included, which + `remove` then takes out as it always has. Staged, the file goes into + `remove`'s commit, as #407's untracked remedy has it, unless the person + commits it on its own first. + """ + root = family.root + if os.path.lexists(root / GITMODULES): + return + records = _index_records(root) + if GITMODULES.encode() in records or _head_oid(root, GITMODULES): + return + gitlinks = {name.decode("utf-8", "surrogateescape") + for name, held in records.items() + if any(record.startswith(b"160000 ") + and record.partition(b"\t")[0].endswith(b" 0") + for record in held)} + if path not in gitlinks: + return + sections = [] + for row in family.members: + project = str(row.get("project") or "") + member = f"{MEMBERS_DIR}/{project}" + if not project or member not in gitlinks: + continue + url = _member_url(str(row.get("repository") or ""), None) + sections += [f"`git config -f {GITMODULES} " + f"{shlex.quote(f'submodule.{member}.{key}')} " + f"{shlex.quote(value)}`" + for key, value in (("path", member), ("url", url))] + raise Refusal( + "member-gitmodules-absent", + f"the holder has no {GITMODULES}: neither the work tree, the index " + f"nor HEAD holds one, though the index records a gitlink at {path}. " + f"`git rm` edits only a {GITMODULES} the work tree has, and `remove` " + f"commits {GITMODULES} as `git rm` leaves it, so with none to commit " + f"it would fail after `git submodule deinit` and `git rm` had emptied " + f"{path}. A recursive clone of the holder checks out none of its " + "members, and its validate-family.py finds member-gitmodules-absent " + "for each. No git command that writes was run and nothing was " + "written.", + "Remediation: write back the section `family.py add` writes for " + "each member the index records, this one included, in a POSIX shell " + f"— {' and '.join(sections)} — then stage it, `git add -f " + f"{GITMODULES}` (`-f`, since an operator's excludes may match it), " + "and commit it on its own first if the command's commit should " + "carry only its own change, and run the same command again.") + + def _checked_index_unlocked(root: Path, path: str) -> None: """`family-index-locked`: the real `remove` refuses a held `index.lock` BEFORE it unmounts anything (#365). @@ -5591,6 +5659,9 @@ def cmd_remove(args) -> int: # carry a change nobody staged. Where the index holds a copy, every # reading of the work tree's file after this one reads that copy. _checked_gitmodules_as_staged(family.root, "remove") + # A holder that holds no `.gitmodules` anywhere, as one whose HEAD lost + # it does, is refused next (#611): the commit could not stage one. + _checked_gitmodules_held(family, path) # A `.gitmodules` that gives the path to a second submodule is refused # here, before any git command that writes (#360), so the dry run makes # the same refusal, as it makes every one above. diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index 450bac3d..e4c2200c 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -40,7 +40,9 @@ project the row claims. A row whose `path:` is not `members/` is the `member-path` finding and none of the three is asked of it: nothing read through a - path the row may not have is a fact about the member (#367). + path the row may not have is a fact about the member (#367). A row + whose gitlink is recorded only at a case variant of its path is refused + naming that variant, and the index move that puts it back (#611). And, across the rows, that no two record ONE `id`: that is one project pinned twice, however each row spelled its repository (#206). This one reads `family.yaml` alone, so `--no-members` asks it too. @@ -57,6 +59,13 @@ asks them too. A path a row names in another case is still one of these, and what it says to do depends on which of the two is the member's own: it names no command that would un-register the member. + - that a section of HEAD's `.gitmodules` gives the path of every row's + gitlink (#611, #407's decision 3). A recursive clone reads HEAD's file, + so a gitlink no section of it gives is left empty there, by CI and by + `make bootstrap`, though the holder it was committed in keeps its + checkout. It needs no member checked out, so `--no-members` asks it + too, and a member it names that is not checked out is not then refused + as uninitialized. - the COPIED shape files, each against its `sha256` row in `contracts/shape-pin.yaml`. Editing a copy in place is drift and the exit is to carry it upstream, never to update the digest. @@ -381,7 +390,7 @@ def _pin_commit(pin: dict, project: str, report: Report) -> str: def _check_member_gitlink(root: Path, path: str, commit: str, project: str, - report: Report) -> None: + gitlinks: dict[str, str], report: Report) -> None: """The submodule gitlink this repository records at `path`, against the pin — THE LOCKSTEP RULE. Split from `_check_member` for #137. Raises `Refusal` when there is no gitlink at all: a row naming a member this @@ -389,9 +398,15 @@ def _check_member_gitlink(root: Path, path: str, commit: str, project: str, `git switch -c` that `add` needs first (#539) and the `add`, from a checkout of openRepoShape and naming this holder by its path, as the holder's other remedies do: `--family-root .` named the checkout the - line runs from.""" + line runs from. Where `gitlinks` (`_member_gitlinks`) holds one at a + case variant of `path`, the refusal names that instead (#611).""" gitlink = recorded_gitlink(root, path) if gitlink is None: + variants = sorted(other for other in gitlinks if other != path + and other.casefold() == path.casefold()) + if variants: + raise _gitlink_in_another_case(path, commit, project, variants, + gitlinks) raise Refusal( "member-gitlink-absent", f"no gitlink (mode 160000) recorded at {path!r}. {project} is " @@ -414,6 +429,45 @@ def _check_member_gitlink(root: Path, path: str, commit: str, project: str, report.note(f"{path}: gitlink == pin {commit[:12]}") +def _gitlink_in_another_case(path: str, commit: str, project: str, + variants: list[str], + gitlinks: dict[str, str]) -> Refusal: + """`member-gitlink-absent` for a row whose gitlink this repository + records only at a case variant of its path (#611, #363's item 3), as a + holder committed in #363's state does: `Members/

`, under a row and a + `.gitmodules` section that say `members/

`. A fresh clone checks out + nothing at the row's path, and the plain refusal's `family.py add` + cannot help: the row is there already. + + EACH VARIANT IS NAMED, AND THE GITLINK IS MOVED IN THE INDEX ALONE, as + #574's remedies act on an index entry under another spelling of + `members/` (Brett Heap, 2026-10-14, "Refuse on every disk"): `git rm -r + --cached` takes each variant out and touches no file, and `git + update-index --cacheinfo` records the gitlink at the row's path, at the + commit the row pins, reading no disk either. So the two lines work on + every disk, wherever the checkout stands, and the commit after them + records the member once, at its own path. `member-gitlink-unnamed` + prints no command for the same gitlink, since taking it out ALONE would + un-register the member; these take it out and record it together. + """ + found = ", ".join(f"{variant} (at {gitlinks[variant]})" + for variant in variants) + taken_out = " ".join(shlex.quote(variant) for variant in variants) + return Refusal( + "member-gitlink-absent", + f"no gitlink (mode 160000) recorded at {path!r}. {project} is named " + f"in {MANIFEST}, and this repository records its gitlink only at a " + "case variant of that path, which macOS and Windows hold as one " + f"directory with it: {found}. A recursive clone checks out nothing " + f"at {path}.", + "Remediation: the member is recorded in the wrong case. Move its " + "gitlink to the row's path, at the commit the row pins, in the index " + "alone, which touches no file, and commit that:\n" + f" git rm -r --cached -- {taken_out}\n" + " git update-index --add --cacheinfo " + f"{shlex.quote(f'160000,{commit},{path}')}") + + def _check_member_tree(member: Path, pin: dict, path: str, commit: str, project: str, report: Report) -> None: """Is `member`'s own object store readable at `commit`, and does its @@ -467,7 +521,8 @@ def _check_member_tree(member: Path, pin: dict, path: str, commit: str, report.note(f"{path}: tree digest recomputes ({recorded[:12]}…)") -def _check_member(root: Path, row: dict, seen: dict, report: Report) -> None: +def _check_member(root: Path, row: dict, seen: dict, report: Report, + gitlinks: dict[str, str], unregistered: set[str]) -> None: """One member: the row itself, the gitlink, the digest, the identity. NOTHING IS READ THROUGH A PATH THE ROW MAY NOT HAVE (#367). A hand-edited @@ -482,6 +537,14 @@ def _check_member(root: Path, row: dict, seen: dict, report: Report) -> None: The git each check runs is run inside `_git_runs`, which names the check when there is no git to run (#368); the identity check reads a file and runs none. + + A MEMBER NO SECTION OF HEAD'S `.gitmodules` GIVES (`unregistered`, the + `member-gitmodules-absent` finding, #611) that is not checked out stops + after its gitlink. Refused `member-uninitialized`, it named `git + submodule update --init`, which exits 0 there and checks nothing out, + and the refusal dropped the finding that says why: the state a fresh + clone of a holder whose HEAD lost `.gitmodules` is in, and CI's. The + finding is made all the same, so the run still fails. """ project = _member_project_name(row, seen, report) if project is None: @@ -500,10 +563,12 @@ def _check_member(root: Path, row: dict, seen: dict, report: Report) -> None: # ---- the gitlink ------------------------------------------------------- with _git_runs(f"the gitlink recorded at {path} could not be read"): - _check_member_gitlink(root, path, commit, project, report) + _check_member_gitlink(root, path, commit, project, gitlinks, report) # ---- the digest and the identity -------------------------------------- member = root / path + if path in unregistered and not (member / ".git").exists(): + return with _git_runs(f"{project}'s pinned commit and its tree digest could not " f"be read in {path}"): _check_member_tree(member, pin, path, commit, project, report) @@ -822,6 +887,13 @@ def _gitmodules_sections(root: Path) -> dict[str, dict[str, str]]: "holder's submodules can be cloned from it as it stands. Mend " f"the file by hand (`git config -f {GITMODULES} --list` says " "where it breaks), then run this again.") from exc + return _sections_listed(listed) + + +def _sections_listed(listed: bytes) -> dict[str, dict[str, str]]: + """`{name: {key: value}}` of every `submodule` section in what `git + config -z --list` printed, each key in the order git lists it: one + reading for the work tree's `.gitmodules` and HEAD's (#611).""" sections: dict[str, dict[str, str]] = {} for record in listed.split(b"\0"): key, _, value = record.decode("utf-8", @@ -833,6 +905,48 @@ def _gitmodules_sections(root: Path) -> dict[str, dict[str, str]]: return sections +def _head_gitmodules(root: Path + ) -> tuple[bool, dict[str, dict[str, str]]] | None: + """`(whether HEAD holds a .gitmodules, its sections)`, read as + `_gitmodules_sections` reads the work tree's, or None for a HEAD git + cannot list, a holder before its first commit, which a clone cannot + take either (#611). + + HEAD'S, NOT THE WORK TREE'S: a recursive clone of this holder, and CI's + checkout, read the file HEAD holds. A work tree's copy nobody has + committed is no fact about what a clone gets, and neither is one a + sparse checkout leaves out of the work tree or a deletion nobody has + committed. HEAD's ROOT TREE is listed and the entry picked by name, with + no pathspec, as `_member_gitlinks` lists HEAD. One git cannot read is + `gitmodules-unreadable`, as the work tree's is. + """ + with _git_runs(f"HEAD's {GITMODULES} could not be read as git reads it"): + try: + tree = git_out(["ls-tree", "-z", "HEAD"], cwd=root, binary=True) + except Refusal: + return None + blob = next((meta.split()[2].decode("ascii") + for meta, _, name in (record.partition(b"\t") + for record in tree.split(b"\0")) + if name == GITMODULES.encode() + and meta.split()[1:2] == [b"blob"]), None) + if blob is None: + return False, {} + try: + listed = git_out(["config", "-z", "--blob", blob, "--list"], + cwd=root, binary=True) + except Refusal as exc: + raise Refusal( + "gitmodules-unreadable", + f"HEAD's {GITMODULES} could not be read as git reads it: " + f"{exc.detail}", + "Remediation: git cannot read it either, so none of this " + "holder's submodules can be cloned from it. Mend the file by " + f"hand (`git config -f {GITMODULES} --list` says where it " + "breaks), commit it, then run this again.") from exc + return True, _sections_listed(listed) + + def _check_gitmodules_paths(sections: dict[str, dict[str, str]], report: Report) -> None: """No two sections of `.gitmodules` give ONE path (#300). @@ -1079,7 +1193,10 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], which refuses while its checkout holds changes, then mounted with `family.py add`: `add` cannot adopt an existing mount, and after `git rm --cached` alone it refuses the checkout left in the way - (`member-path-taken`). `add` then names the object store `git rm` leaves + (`member-path-taken`). The removal is committed between the two (#611): + `add` refuses a path under another spelling of `members/` that HEAD + holds and the index no longer does (#574), which an uncommitted `git rm + -- Members/

` leaves. `add` then names the object store `git rm` leaves in `.git/modules`, if it is still there. `--member` is quoted as the paths are, its project alone: `` is the operator's to fill in. A BRANCH FIRST (#539): `add` refuses the holder's tracking branch @@ -1119,10 +1236,13 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], "with the first command below, since family.py add refuses the " "holder's tracking branch, take it out with the second, which " f"refuses while its checkout holds changes and takes its " - f"{GITMODULES} section out too, then mount it with the third, " - "from a checkout of openRepoShape: family.py add writes its row " - "and its gitlink in one commit, and names the object store git " - "rm leaves under .git/modules if one is still there.\n" + f"{GITMODULES} section out too, and commit that removal, since " + "family.py add refuses a path under another spelling of members/ " + "that HEAD holds and the index no longer does, then mount it " + "with the third, from a checkout of openRepoShape: family.py add " + "writes its row and its gitlink in one commit, and names the " + "object store git rm leaves under .git/modules if one is still " + "there.\n" f" {_switch_line(root, 'add', project)}\n" f" git rm -- {shlex.quote(path)}\n" f" {PYTHON} scripts/family.py add --family-root " @@ -1201,6 +1321,73 @@ def _check_gitmodules_named(sections: dict[str, dict[str, str]], f"of them, then commit {GITMODULES}:{removal}") +def _check_gitlinks_registered(root: Path, members: list, + gitlinks: dict[str, str], + report: Report) -> set[str]: + """Every row's gitlink is at a path a section of HEAD's `.gitmodules` + gives — the `member-gitmodules-absent` finding (#611, #407's decision + 3). Returns the paths it was made for, of which `_check_member` asks no + checkout that is not there. + + A recursive clone checks a gitlink out only where a section gives its + path, so a gitlink with none is left empty in every clone, by CI and by + `make bootstrap` there, whose `git submodule update` exits 0 and checks + nothing out. The holder the commit was made in keeps its checkout, and + its git config the registration, so every other check here passed it: + `family.py remove` committed a `.gitmodules` the work tree had lost + (#407) and the gate said `family ok`. + + HEAD's file (`_head_gitmodules`), and the rows that `_check_member` + reads at their path: one whose `path:` is not `members/` is + `member-path`'s, and one with no gitlink there is + `member-gitlink-absent`'s (`gitlinks`, read as `recorded_gitlink` reads + one). A holder with no members holds no gitlink and is asked nothing. + + The remedy writes the section as `family.py add` writes it, named after + its path, with the url it gives a member, `https://github.com/` and the + row's `repository:`, or `` where that is not `/`. A + section of that name that has a url keeps it: only its path is written. + """ + head = _head_gitmodules(root) + if head is None: + return set() + held, sections = head + given = {keys["path"] for keys in sections.values() if "path" in keys} + unregistered: set[str] = set() + for row in members: + project = row.get("project") if isinstance(row, dict) else None + if not isinstance(project, str) or not project: + continue + path = f"{DEFAULT_MEMBERS_DIR}/{project}" + if (str(row.get("path") or path) != path or path in unregistered + or path not in gitlinks or path in given): + continue + unregistered.add(path) + repository = row.get("repository") + keys = {"path": shlex.quote(path)} + if "url" not in sections.get(path, {}): + keys["url"] = ( + shlex.quote(f"https://github.com/{repository}.git") + if isinstance(repository, str) + and QUALIFIED_RE.match(repository) else "") + lines = "".join( + f"\n git config -f {GITMODULES} " + f"{shlex.quote(f'submodule.{path}.{key}')} {value}" + for key, value in keys.items()) + report.finding( + "member-gitmodules-absent", + f"{path} is a gitlink at {gitlinks[path]} that a row of members: " + "names, and " + + (f"no section of HEAD's {GITMODULES} gives that path" if held + else f"HEAD holds no {GITMODULES}") + + ", so a recursive clone of this holder checks out nothing " + "there, and neither does make bootstrap in that clone, whatever " + "checkout this holder has of it. Write its section back by hand, " + "in a POSIX shell, as family.py add writes one, then stage and " + f"commit {GITMODULES}:{lines}") + return unregistered + + def _check_members(root: Path, members: list, no_members: bool, report: Report) -> None: """Every row in `members:`, or the `--no-members` skip line. @@ -1209,7 +1396,8 @@ def _check_members(root: Path, members: list, no_members: bool, out comes first, so `--no-members` asks them too: the ones that read a file alone, `family.yaml`'s names (#331) and ids (#206) and the paths `.gitmodules` gives twice (#300), then the gitlinks and sections no row - names, which read the index and HEAD as well (#361). + names, which read the index and HEAD as well (#361), and the gitlinks + no section of HEAD's `.gitmodules` gives (#611). """ _check_member_names_unique(members, report) _check_member_ids_unique(root, members, report) @@ -1219,6 +1407,8 @@ def _check_members(root: Path, members: list, no_members: bool, gitlinks = _member_gitlinks(root) _check_gitlinks_named(root, gitlinks, named, report) _check_gitmodules_named(sections, gitlinks, named, report) + unregistered = _check_gitlinks_registered(root, members, gitlinks, + report) if no_members: print(f" -- {len(members)} member(s) SKIPPED (--no-members): " "their gitlinks, digests and project ids are not checked " @@ -1230,7 +1420,7 @@ def _check_members(root: Path, members: list, no_members: bool, report.finding("member-row", f"a member is not a mapping: {row!r}") continue - _check_member(root, row, seen, report) + _check_member(root, row, seen, report, gitlinks, unregistered) if not members: report.note("no members yet: a family with none is empty, not " "wrong") diff --git a/tests/test_family_head_lost_gitmodules.py b/tests/test_family_head_lost_gitmodules.py new file mode 100644 index 00000000..159082b9 --- /dev/null +++ b/tests/test_family_head_lost_gitmodules.py @@ -0,0 +1,596 @@ +# SPDX-License-Identifier: Apache-2.0 +"""A holder whose HEAD lost `.gitmodules`, and the gate's gitlink remedies +(#611). + +THE ISSUE'S FOUR LINES, each reproduced on main before this change: + +- 571(b). In a holder whose HEAD already lost `.gitmodules` (the commit #407 + describes), `family.py remove --dry-run` exited 0, and the real run did + `git submodule deinit` and `git rm`, then died at its commit's `git + --literal-pathspecs add -f -- .gitmodules family.yaml` (`pathspec + '.gitmodules' did not match any files`). Its rollback put HEAD and the + index back and left the member's directory EMPTY. `remove` now refuses + that holder before any git command that writes, `member-gitmodules-absent`, + under `--dry-run` alike, and its remedy writes back each member's section. +- 571(c). `validate-family.py` passed a `members:` row whose gitlink no + section of HEAD's `.gitmodules` gives (#407's decision 3): `family ok` in + the holder, `--no-members` too, while a fresh recursive clone left every + such member empty and its gate refused `member-uninitialized`, naming a `git + submodule update` that exits 0 and checks nothing out. It is the finding + `member-gitmodules-absent` now, in every mode, read from HEAD, and a member + it names that is not checked out is not refused as uninitialized. +- 574/2. `member-gitlink-absent` said nothing of a gitlink HEAD holds at a + case variant of the row's path (#363's item 3), and named `family.py add`, + which cannot help: the row is there. It names the variant now, and the + index move that records the member at its own path. +- 574/3. A hand mount's `member-gitlink-unnamed` remedy said `git rm`, then + `family.py add`. `add` refuses a path under another spelling of `members/` + that HEAD holds and the index no longer does (#574), so it says to commit + the removal between them now. + +THE SAME FAMILY, OFFLINE RULES AND READINGS as +`test_family_remove_gitmodules.py` (#407), whose helpers are imported. Every +remedy is typed as printed: split as a POSIX shell splits it, run with no +shell. The url a written-back section gives is the one `family.py add` +writes, `https://github.com//.git`, and a clone reaches the +fixture's local remotes through it by a `url..insteadOf` for the one +command, as `clone_just_pushed` reaches the legs. The case-variant states are +index records and commits (`stage_gitlink`), never two directories whose +names differ only in case, and each is asked again with +`core.ignorecase=true`, how macOS and Windows clone, so every one runs on all +three systems. +""" + +from __future__ import annotations + +import re +import shlex +from pathlib import Path + +import pytest + +from conftest import clone_just_pushed, run_script +from repo_shape import PYTHON +#: The family fixture, its names, the offline environment and the readings +#: of a holder, imported from the modules that own them rather than rebuilt. +from test_family import ( # noqa: F401 + FAMILY, NAME, ORG, SPARE, family, validate, +) +from test_family_add_excludes import empty_holder # noqa: F401 +from test_family_add_refusals import holder, offline # noqa: F401 +from test_family_dry_run import isolated +from test_family_member_path import MEMBER, OTHER, raw +from test_family_remove_gitmodules import ( + deleted, edited, head_paths, left_out, ran, refused_alike, + typed_as_printed, verb_args, +) +from test_scaffold_ignored_shape_files import excluding +from test_validate_family_unnamed_mounts import ( + commands, findings, spare_commits, stage_gitlink, +) + +GITMODULES = ".gitmodules" +MODES = pytest.mark.parametrize( + "extra", [(), ("--pins",), ("--no-members",)], + ids=["members-checked", "pins", "no-members"]) + +#: The two disks: git's own reading of a case-sensitive one, and how macOS +#: and Windows clone (`core.ignorecase=true`), set for every git command the +#: test types. +DISKS = pytest.mark.parametrize("ignorecase", ["false", "true"], + ids=["linux", "folding"]) + + +def on_disk(env: dict, ignorecase: str) -> dict: + """`env` with `core.ignorecase` set to `ignorecase`, one more + `GIT_CONFIG_*` entry after the ones it already holds.""" + index = int(env.get("GIT_CONFIG_COUNT", "0")) + return {**env, "GIT_CONFIG_COUNT": str(index + 1), + f"GIT_CONFIG_KEY_{index}": "core.ignorecase", + f"GIT_CONFIG_VALUE_{index}": ignorecase} + + +def url(project: str) -> str: + """The url `family.py add` writes for a member of the fixture's org.""" + return f"https://github.com/{ORG}/{project}.git" + + +#: What `remove` says in a holder that holds no `.gitmodules` anywhere. +REMOVE_REFUSED = ( + "REFUSED member-gitmodules-absent: the holder has no .gitmodules: neither " + "the work tree, the index nor HEAD holds one, though the index records a " + "gitlink at members/{member}. `git rm` edits only a .gitmodules the work " + "tree has, and `remove` commits .gitmodules as `git rm` leaves it, so " + "with none to commit it would fail after `git submodule deinit` and `git " + "rm` had emptied members/{member}. A recursive clone of the holder checks " + "out none of its members, and its validate-family.py finds " + "member-gitmodules-absent for each. No git command that writes was run " + "and nothing was written.") + + +def remove_remedy(*projects: str) -> str: + """`remove`'s remediation, writing back the section of each of + `projects`, in that order.""" + written = " and ".join( + f"`git config -f .gitmodules submodule.members/{project}.{key} " + f"{value}`" for project in projects + for key, value in (("path", f"members/{project}"), + ("url", url(project)))) + return ("Remediation: write back the section `family.py add` writes for " + "each member the index records, this one included, in a POSIX " + f"shell — {written} — then stage it, `git add -f .gitmodules` " + "(`-f`, since an operator's excludes may match it), and commit it " + "on its own first if the command's commit should carry only its " + "own change, and run the same command again.") + + +def absent(project: str, commit: str, *, held: bool, + with_url: bool = True) -> str: + """The `member-gitmodules-absent` finding for `project`'s gitlink at + `commit`, WHOLE: its line and the commands under it. `held` is whether + HEAD holds a `.gitmodules` at all; `with_url` whether the section to + write has no url of its own.""" + path = f"members/{project}" + lines = f"\n git config -f .gitmodules submodule.{path}.path {path}" + if with_url: + lines += (f"\n git config -f .gitmodules submodule.{path}.url " + f"{url(project)}") + return (f"FINDING member-gitmodules-absent: {path} is a gitlink at " + f"{commit} that a row of members: names, and " + + ("no section of HEAD's .gitmodules gives that path" if held + else "HEAD holds no .gitmodules") + + ", so a recursive clone of this holder checks out nothing " + "there, and neither does make bootstrap in that clone, whatever " + "checkout this holder has of it. Write its section back by hand, " + "in a POSIX shell, as family.py add writes one, then stage and " + f"commit .gitmodules:{lines}") + + +# --- the states -------------------------------------------------------------- + +def commit_all(root: Path, env: dict, message: str) -> None: + raw(root, env, "commit", "-q", "-m", message) + + +def head_lost(root: Path, env: dict) -> None: + """THE ISSUE'S STATE: `.gitmodules` deleted and the deletion committed, + as the commit #407 describes left it. The registrations stay in the + holder's git config, and its checkouts on disk.""" + raw(root, env, "rm", "-q", "--", GITMODULES) + commit_all(root, env, "The commit #407 describes") + + +def committed_gitmodules(root: Path, env: dict, data: bytes) -> None: + """`data` committed as `.gitmodules`, hashed into git's objects as + BYTES, and the same bytes on disk, so no platform rewrites a line + ending.""" + blob = raw(root, env, "hash-object", "-w", "--stdin", + stdin=data).stdout.decode("ascii").strip() + raw(root, env, "update-index", "--cacheinfo", + f"100644,{blob},{GITMODULES}") + commit_all(root, env, "A .gitmodules edited by hand") + (root / GITMODULES).write_bytes(data) + + +def without_section(root: Path, env: dict, project: str) -> None: + """HEAD's `.gitmodules` with `project`'s section taken out by hand and + committed, the other member's left as `family.py add` wrote it.""" + data = raw(root, env, "show", f"HEAD:{GITMODULES}").stdout + section = re.compile(rb'\[submodule "members/' + project.encode() + + rb'"\]\n(?:\t[^\n]*\n)*') + assert len(section.findall(data)) == 1, data + committed_gitmodules(root, env, section.sub(b"", data)) + + +def pinned(root: Path, project: str) -> str: + """The commit `family.yaml` pins `project` at, as git records its + gitlink after `family.py add`.""" + return raw(root, {}, "rev-parse", f"HEAD:members/{project}").stdout.decode( + "ascii").strip() + + +def typed(root: Path, env: dict, finding: str) -> None: + """Every command `finding` prints, typed as printed.""" + for command in commands(finding): + argv = shlex.split(command) + assert argv[0] == "git", command + raw(root, env, *argv[1:]) + + +def staged_and_committed(root: Path, env: dict, message: str) -> None: + """What a finding's "then stage and commit .gitmodules" says.""" + raw(root, env, "add", "--", GITMODULES) + commit_all(root, env, message) + + +def assert_passes_in_every_mode(root: Path, members: int) -> None: + for extra in ((), ("--pins",), ("--no-members",)): + result = validate(root, *extra) + assert result.returncode == 0, (extra, result.stderr + result.stdout) + assert (f"family ok: {NAME} (inkrouter), {members} member(s)" + in result.stdout), extra + + +def github_to(remotes: Path) -> list[str]: + """The `-c` that makes a clone read `https://github.com//` from the + fixture's local remotes, for the one command.""" + uri = remotes.resolve().as_uri().rstrip("/") + "/" + return ["-c", f"url.{uri}.insteadOf=https://github.com/{ORG}/"] + + +def pushed_and_cloned(root: Path, env: dict, family, tmp_path: Path, + branch: str) -> Path: + """HEAD pushed to `branch` of the family's own remote, then a fresh + `clone --recurse-submodules` of it, which must exit 0.""" + remote = family["remotes"] / f"{NAME}.git" + raw(root, env, "push", "-q", str(remote), f"HEAD:refs/heads/{branch}") + clone = tmp_path / f"clone-{branch.replace('/', '-')}" + cloned = clone_just_pushed(remote, clone, branch, + config=github_to(family["remotes"])) + assert cloned.returncode == 0, cloned.stdout + cloned.stderr + return clone + + +# --- 571(b): remove in a holder whose HEAD lost .gitmodules ------------------ + +@pytest.mark.parametrize("rules", [b"", b"*\n"], ids=["no-excludes", "star"]) +def test_remove_refuses_a_holder_whose_head_lost_gitmodules( + family, holder, offline, tmp_path, rules): + """`member-gitmodules-absent`, by `remove` and its dry run alike, exit 2, + with the holder byte for byte as it was, MEMBER's checkout included: + the real run used to empty it and fail at its commit. Typed as printed, + under an operator whose excludes hold `*` too (hence the `-f`), the + remedy writes both members' sections back and stages them; `remove` + then commits MEMBER's removal with OTHER's section back in HEAD, and + the gate passes.""" + env = {**isolated(offline, tmp_path), **excluding(tmp_path / "op", rules)} + head_lost(holder, env) + args = verb_args("remove", holder, family) + + said, remediation = refused_alike(holder, env, args) + + assert said == REMOVE_REFUSED.format(member=MEMBER) + assert remediation == remove_remedy(MEMBER, OTHER) + assert (holder / "members" / MEMBER / "project.yaml").is_file() + assert typed_as_printed(holder, env, remediation)[-1] == ( + "git add -f .gitmodules") + ran(holder, env, args) + assert head_paths(holder, env) == [f"members/{OTHER}"] + assert_passes_in_every_mode(holder, 1) + + +def test_the_holder_remove_leaves_clones_with_the_other_member( + family, holder, offline, tmp_path): + """THE ISSUE'S CLONE. After the remedy and the `remove`, a push and a + fresh `clone --recurse-submodules` check OTHER out, from the url the + written-back section gives, and the clone's own gate passes, where + before the change a clone of this holder checked out no member at all.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + args = verb_args("remove", holder, family) + typed_as_printed(holder, env, refused_alike(holder, env, args)[1]) + ran(holder, env, args) + + clone = pushed_and_cloned(holder, env, family, tmp_path, "w611/removed") + + assert (clone / "members" / OTHER / "project.yaml").is_file() + assert not (clone / "members" / MEMBER).exists() + gate = validate(clone) + assert gate.returncode == 0, gate.stdout + gate.stderr + + +def test_remove_of_a_member_whose_section_alone_is_gone_runs_as_before( + family, holder, offline, tmp_path): + """The refusal is for a holder with NO `.gitmodules`, where the commit + cannot stage one. Where HEAD's file lacks only the member's own section, + `git rm` finds none to take out and `remove` commits the gitlink and the + row, as it did before: the holder it leaves passes the gate.""" + env = isolated(offline, tmp_path) + without_section(holder, env, MEMBER) + + removed = run_script(FAMILY, *verb_args("remove", holder, family), + env=env) + + assert removed.returncode == 0, removed.stdout + removed.stderr + assert head_paths(holder, env) == [f"members/{OTHER}"] + assert_passes_in_every_mode(holder, 1) + + +# --- 571(c): a gitlink no section of HEAD's .gitmodules gives --------------- + +@MODES +def test_a_head_that_lost_gitmodules_is_a_finding_per_member(holder, offline, + tmp_path, extra): + """THE ISSUE'S GATE: `family ok` before the change, in every mode. One + finding per member, each naming its gitlink and the two lines that write + its section back, in every mode, since none needs a member checked out. + Typed as printed, staged and committed, the holder passes again.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + + result = validate(holder, *extra) + + assert result.returncode == 1, result.stderr + result.stdout + assert "REFUSED" not in result.stderr, result.stderr + assert findings(result) == [ + absent(project, pinned(holder, project), held=False) + for project in (MEMBER, OTHER)] + for finding in findings(result): + typed(holder, env, finding) + staged_and_committed(holder, env, "Write the sections back as printed") + assert_passes_in_every_mode(holder, 2) + + +@MODES +def test_a_section_head_lost_alone_is_one_finding(holder, offline, tmp_path, + extra): + """One member's section taken out of HEAD's `.gitmodules` by hand: a + recursive clone checks that member out nowhere, though the file is + there. One finding, for it alone, and the other member is asked + nothing.""" + env = isolated(offline, tmp_path) + without_section(holder, env, OTHER) + + result = validate(holder, *extra) + + assert result.returncode == 1, result.stderr + result.stdout + (finding,) = findings(result) + assert finding == absent(OTHER, pinned(holder, OTHER), held=True) + typed(holder, env, finding) + staged_and_committed(holder, env, "Write the section back as printed") + assert_passes_in_every_mode(holder, 2) + + +def test_a_section_of_the_name_add_gives_keeps_its_url(holder, offline, + tmp_path): + """A section named `members/

`, as `family.py add` names one, whose + path is another case (`members/irss`): no section gives OTHER's path, + so a clone checks it out nowhere. The section's url is kept, and only + its path is written, which clears this finding and the section's own + `gitmodules-path-unnamed` beside it.""" + env = isolated(offline, tmp_path) + data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout + old = f"\tpath = members/{OTHER}\n".encode() + assert data.count(old) == 1, data + committed_gitmodules(holder, env, data.replace( + old, f"\tpath = members/{OTHER.lower()}\n".encode())) + + result = validate(holder) + + assert result.returncode == 1, result.stderr + result.stdout + section, finding = findings(result) + assert section.startswith("FINDING gitmodules-path-unnamed: "), section + assert finding == absent(OTHER, pinned(holder, OTHER), held=True, + with_url=False) + typed(holder, env, finding) + staged_and_committed(holder, env, "Write the path back as printed") + assert_passes_in_every_mode(holder, 2) + + +def test_a_fresh_clone_of_it_says_the_finding_and_not_uninitialized( + family, holder, offline, tmp_path): + """BOOTSTRAP, VALIDATE AND A FRESH RECURSIVE CLONE AGREE. Pushed, the + holder whose HEAD lost `.gitmodules` clones with exit 0 and both + members empty; `make bootstrap`'s script there checks neither out, and + says so for each; and the clone's gate names both, members checked or + not, where before the change it refused `member-uninitialized`, whose + `git submodule update` exits 0 there. With the sections written back + as printed and pushed, a fresh clone checks both out and its gate + passes.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + clone = pushed_and_cloned(holder, env, family, tmp_path, "w611/lost") + expected = [absent(project, pinned(holder, project), held=False) + for project in (MEMBER, OTHER)] + + assert [path.name for path in (clone / "members").iterdir() + if any(path.iterdir())] == [] + booted = run_script(clone / "scripts" / "bootstrap.py", cwd=clone) + for project in (MEMBER, OTHER): + assert f" [{project}] {project}: NOT CHECKED OUT; skipped." in ( + booted.stdout), booted.stdout + for extra in ((), ("--no-members",)): + gate = validate(clone, *extra) + assert gate.returncode == 1, (extra, gate.stderr + gate.stdout) + assert "member-uninitialized" not in gate.stderr, gate.stderr + assert findings(gate) == expected, extra + + for finding in expected: + typed(holder, env, finding) + staged_and_committed(holder, env, "Write the sections back as printed") + mended = pushed_and_cloned(holder, env, family, tmp_path, "w611/mended") + for project in (MEMBER, OTHER): + assert (mended / "members" / project / "project.yaml").is_file() + assert_passes_in_every_mode(mended, 2) + + +#: `(set-up, what it is)`: the work tree's `.gitmodules` is not HEAD's, and +#: HEAD's gives every member's path, so a clone checks each out. +NOT_HEADS = { + "deleted, not committed": deleted, + "edited, not staged": edited, + "left out by a sparse checkout": left_out, +} + + +@pytest.mark.parametrize("set_up", NOT_HEADS.values(), ids=NOT_HEADS) +def test_a_work_tree_copy_that_is_not_heads_is_no_finding(holder, offline, + tmp_path, set_up): + """No misfire: the finding reads HEAD's file, which a clone reads, and + never the work tree's. A deletion nobody committed, an edit nobody + staged and a sparse checkout that leaves the file out are each no fact + about what a clone gets.""" + env = isolated(offline, tmp_path) + set_up(holder, env) + + for extra in ((), ("--no-members",)): + result = validate(holder, *extra) + assert result.returncode == 0, (extra, result.stderr + result.stdout) + + +def test_a_holder_with_no_members_is_asked_nothing(empty_holder): + """No misfire: a holder `init` made holds no `.gitmodules` and no + gitlink, and passes in every mode.""" + for extra in ((), ("--pins",), ("--no-members",)): + result = validate(empty_holder, *extra) + assert result.returncode == 0, (extra, result.stderr + result.stdout) + + +def test_an_untracked_copy_put_back_is_still_a_finding_until_committed( + holder, offline, tmp_path): + """A `.gitmodules` put back by hand, untracked: the holder's own + checkouts still work, but HEAD, which a clone reads, holds none, so the + finding stands until the file is committed, and the holder then + passes.""" + env = isolated(offline, tmp_path) + kept = (holder / GITMODULES).read_bytes() + head_lost(holder, env) + (holder / GITMODULES).write_bytes(kept) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stderr + result.stdout + assert [finding.split(":", 1)[0] for finding in findings(result)] == [ + "FINDING member-gitmodules-absent"] * 2 + staged_and_committed(holder, env, "Put .gitmodules back") + assert_passes_in_every_mode(holder, 2) + + +# --- 574/2: a gitlink recorded only at a case variant of the row's path ----- + +def variant_refused(path: str, commit: str, variants: list[str], + gitlinks: dict[str, str]) -> list[str]: + """`member-gitlink-absent`'s lines for a row whose gitlink is recorded + only at `variants`.""" + project = path.split("/", 1)[1] + found = ", ".join(f"{variant} (at {gitlinks[variant]})" + for variant in variants) + return [ + f"REFUSED member-gitlink-absent: no gitlink (mode 160000) recorded at " + f"'{path}'. {project} is named in family.yaml, and this repository " + "records its gitlink only at a case variant of that path, which " + f"macOS and Windows hold as one directory with it: {found}. A " + f"recursive clone checks out nothing at {path}.", + "Remediation: the member is recorded in the wrong case. Move its " + "gitlink to the row's path, at the commit the row pins, in the index " + "alone, which touches no file, and commit that:", + " git rm -r --cached -- " + " ".join(variants), + f" git update-index --add --cacheinfo 160000,{commit},{path}"] + + +def recorded_at(root: Path, env: dict, project: str, + *spellings: str) -> str: + """`project`'s gitlink taken out of the index at its own path and + recorded at each of `spellings`, then committed: index records alone, + so no disk holds two names and the checkout does not move. Returns the + commit it names.""" + commit = pinned(root, project) + raw(root, env, "rm", "-q", "--cached", "--", f"members/{project}") + for spelling in spellings: + stage_gitlink(root, spelling, commit) + commit_all(root, env, f"{project} recorded at {', '.join(spellings)}") + return commit + + +@DISKS +@pytest.mark.parametrize("spellings", [ + ("Members/{p}",), ("MEMBERS/{p}", "Members/{p}")], ids=["one", "two"]) +def test_a_gitlink_only_at_a_case_variant_is_named_and_moved_as_printed( + holder, offline, tmp_path, ignorecase, spellings): + """#363's item 3: HEAD records MEMBER's gitlink at `Members/IRRS` (and, + in the second case, at `MEMBERS/IRRS` too) while its row and section say + `members/IRRS`. The refusal names each variant with its commit and, + since `family.py add` cannot help, prints the move: the variants out of + the index, the gitlink in at the row's path at the pinned commit. Typed + as printed and committed, on either disk, HEAD records the member once, + at its own path, and the holder passes in every mode.""" + env = on_disk(isolated(offline, tmp_path), ignorecase) + variants = sorted(spelling.format(p=MEMBER) for spelling in spellings) + commit = recorded_at(holder, env, MEMBER, *variants) + + result = validate(holder) + + assert result.returncode == 2, result.stderr + result.stdout + lines = result.stderr.splitlines() + path = f"members/{MEMBER}" + assert lines == variant_refused(path, commit, variants, + dict.fromkeys(variants, commit)) + for line in lines[2:]: + raw(holder, env, *shlex.split(line)[1:]) + commit_all(holder, env, "Move the gitlink as printed") + listed = raw(holder, env, "ls-tree", "-r", "--name-only", "HEAD").stdout + assert f"{path}\n".encode() in listed + for variant in variants: + assert f"{variant}\n".encode() not in listed + assert_passes_in_every_mode(holder, 2) + + +def test_a_fresh_clone_of_a_case_variant_says_it_too(family, holder, offline, + tmp_path): + """The same state pushed: a fresh clone names the variant, where the + refusal named `family.py add` and nothing of `Members/`. MEMBER is the + first row, so its gitlink is asked before any member's checkout.""" + env = isolated(offline, tmp_path) + commit = recorded_at(holder, env, MEMBER, f"Members/{MEMBER}") + remote = family["remotes"] / f"{NAME}.git" + raw(holder, env, "push", "-q", str(remote), "HEAD:refs/heads/w611/case") + clone = tmp_path / "clone" + cloned = clone_just_pushed(remote, clone, "w611/case", recurse=False) + assert cloned.returncode == 0, cloned.stdout + cloned.stderr + + result = validate(clone) + + assert result.returncode == 2, result.stderr + result.stdout + variant = f"Members/{MEMBER}" + assert result.stderr.splitlines() == variant_refused( + f"members/{MEMBER}", commit, [variant], {variant: commit}) + + +# --- 574/3: a hand mount's remedy commits the removal before the add --------- + +@DISKS +def test_a_hand_mount_under_members_spelt_another_way_is_added_as_printed( + family, holder, offline, tmp_path, ignorecase): + """A gitlink at `Members/IRQS` that no row names, with its section, as a + hand mount leaves one (index records, so no disk holds `Members/`). + Typed as printed, the branch, the `git rm`, the commit the text says + the removal needs, and the `add` with the org filled in mount IRQS at + `members/IRQS` with its row, and the holder passes in every mode. With + the removal left uncommitted, as the text said before, `add` refuses: + HEAD still holds `Members/IRQS`.""" + env = on_disk(isolated(offline, tmp_path), ignorecase) + raw(holder, env, "switch", "-q", "-C", "main") + _, tip = spare_commits(family) + hand = f"Members/{SPARE}" + stage_gitlink(holder, hand, tip) + data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout + ( + f'[submodule "{hand}"]\n\tpath = {hand}\n\turl = {url(SPARE)}\n' + .encode()) + committed_gitmodules(holder, env, data) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stderr + result.stdout + gitlink = findings(result)[0] + assert gitlink.startswith(f"FINDING member-gitlink-unnamed: {hand} ") + assert (", and commit that removal, since family.py add refuses a path " + "under another spelling of members/ that HEAD holds and the " + "index no longer does, then mount it with the third, ") in gitlink + switch, rm, add = commands(gitlink) + raw(holder.parent, env, *shlex.split(switch)[1:]) + raw(holder, env, *shlex.split(rm)[1:]) + argv = [arg.replace("", ORG) for arg in shlex.split(add)] + assert argv[:3] == [PYTHON, "scripts/family.py", "add"], add + + def add_as_printed(): + return run_script(FAMILY, *argv[2:], "--local-remote-dir", + str(family["remotes"]), env=env) + + uncommitted = add_as_printed() + assert uncommitted.returncode == 2, uncommitted.stdout + uncommitted.stderr + assert "in HEAD and no longer in the index" in uncommitted.stderr + commit_all(holder, env, "Take the hand mount out, as printed") + added = add_as_printed() + assert added.returncode == 0, added.stdout + added.stderr + assert (holder / "members" / SPARE / "project.yaml").is_file() + assert_passes_in_every_mode(holder, 3) diff --git a/tests/test_family_remove_gitmodules.py b/tests/test_family_remove_gitmodules.py index cc7594f8..282a6b82 100644 --- a/tests/test_family_remove_gitmodules.py +++ b/tests/test_family_remove_gitmodules.py @@ -651,12 +651,12 @@ def test_a_holder_whose_head_has_no_gitmodules_is_not_asked( """Where none of the index, HEAD and the work tree holds a `.gitmodules` there is no copy to compare, and no `gitmodules-` refusal is made. - THE REAL RUN STILL FAILS HERE, a known case from before #407 that goes - with the gate's to #459: in a holder whose HEAD already lost the file, - `remove` runs `deinit` and `git rm`, then dies at its commit's `git add - -f -- .gitmodules family.yaml` and puts back only part of the unmount, - while its dry run exits 0. So this pins only that #407's check asks - nothing, and not the dry run's answer (the review's N3).""" + A holder whose HEAD already lost the file is #611's: `remove` and its + dry run refuse it `member-gitmodules-absent`, before any git command + that writes (`tests/test_family_head_lost_gitmodules.py`), where the + real run used to die at its commit's `git add -f -- .gitmodules + family.yaml` with the member emptied. So this pins only that #407's + check asks nothing (the review's N3).""" env = isolated(offline, tmp_path) deletion_staged(holder, env) raw(holder, env, "commit", "-q", "-m", "The commit #407 describes") diff --git a/tests/test_validate_family_unnamed_mounts.py b/tests/test_validate_family_unnamed_mounts.py index bcce579c..7c4babf5 100644 --- a/tests/test_validate_family_unnamed_mounts.py +++ b/tests/test_validate_family_unnamed_mounts.py @@ -88,9 +88,11 @@ "was mounted by hand, make a branch with the first command below, since " "family.py add refuses the holder's tracking branch, take it out with " "the second, which refuses while its checkout holds changes and takes " - "its .gitmodules section out too, then mount it with the third, from a " - "checkout of openRepoShape: family.py add writes its row and its gitlink " - "in one commit, and names the object store git rm leaves under " + "its .gitmodules section out too, and commit that removal, since " + "family.py add refuses a path under another spelling of members/ that " + "HEAD holds and the index no longer does, then mount it with the third, " + "from a checkout of openRepoShape: family.py add writes its row and its " + "gitlink in one commit, and names the object store git rm leaves under " ".git/modules if one is still there.") @@ -523,17 +525,24 @@ def test_a_member_recorded_only_in_another_case_says_so_and_prints_no_command( none and is the refusal `member-gitlink-absent`, which prints no finding, so only `--no-members` shows the finding; a git that matched the pathspec in any case would find the variant and print it, so either is accepted - there, and what no run may do is print a command.""" + there. What no run may do is print a command that takes the gitlink out + ALONE: since #611 the refusal names the variant and prints the move that + takes it out of the index and records it at the row's path together + (`tests/test_family_head_lost_gitmodules.py` types it).""" named = record_in_another_case(holder, "IRSS", "members/irss") result = validate(holder, *extra) - assert "git rm" not in result.stderr + result.stdout assert "--unset-all" not in result.stderr + result.stdout if result.returncode == 2: assert extra != ("--no-members",), result.stderr assert result.stderr.startswith( "REFUSED member-gitlink-absent: no gitlink (mode 160000) " "recorded at 'members/IRSS'."), result.stderr + assert result.stderr.splitlines()[-2:] == [ + " git rm -r --cached -- members/irss", + " git update-index --add --cacheinfo " + f"160000,{named},members/IRSS"], result.stderr return + assert "git rm" not in result.stderr + result.stdout assert result.returncode == 1, result.stderr + result.stdout (gitlink,) = findings(result) assert gitlink == GITLINK_SAID.format( @@ -546,13 +555,20 @@ def test_a_sections_only_path_in_another_case_says_so_and_prints_no_command( holder, extra): """The section's own state (ii): IRSS's row is canonical, its gitlink is where it belongs, and the ONLY section that registers it gives - `members/irss`. `--unset-all` of that left the member with no section.""" + `members/irss`. `--unset-all` of that left the member with no section. + Since #611 no section of HEAD's file giving `members/IRSS` is + `member-gitmodules-absent` beside it, whose one line writes the path + back into that section and keeps its url.""" url = url_of(holder, "IRSS") with_path_replaced(holder, b"\tpath = members/IRSS\n", b"\tpath = members/irss\n") result = validate(holder, *extra) assert result.returncode == 1, result.stderr + result.stdout - (section,) = findings(result) + section, unregistered = findings(result) + assert unregistered.startswith( + "FINDING member-gitmodules-absent: members/IRSS is a gitlink at ") + assert commands(unregistered) == [ + "git config -f .gitmodules submodule.members/IRSS.path members/IRSS"] assert section == SECTION_SAID.format( path="members/irss", name="members/IRSS", url=f"url {url}" ) + WRONG_CASE.format(row_path="members/IRSS", project="IRSS", @@ -685,7 +701,8 @@ def test_a_hand_mount_is_taken_out_and_added_back_as_printed(family, holder): """ROW (a) FOR REAL, and the gitlink's remedy run as printed, from the holder's TRACKING BRANCH, where `init` leaves it and `add` refuses to commit (#487): the first line makes the branch `add` suggests (#539). - `git rm` takes the checkout and its section out; `family.py add`, run + `git rm` takes the checkout and its section out, and the removal is + committed, as the text says since #611; `family.py add`, run as printed with the org filled in, then refuses `member-git-dir-cached` for the object store `git rm` leaves, as the remedy says it will name; with that gone, the same `add` mounts and pins the member with its row @@ -707,6 +724,7 @@ def test_a_hand_mount_is_taken_out_and_added_back_as_printed(family, holder): assert commands(findings(result)[1]) == [] git(*shlex.split(switch)[1:], cwd=holder.parent) git(*shlex.split(rm)[1:], cwd=holder) + commit(holder, "Take the hand mount out, as printed") assert not (holder / "members" / SPARE).exists() assert f"members/{SPARE}" not in (holder / ".gitmodules").read_text( encoding="utf-8") From 6f938ed1a385890cdb4c001e85b304c2618f638d Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Thu, 15 Oct 2026 23:11:21 +0000 Subject: [PATCH 02/15] tests/test_family_head_lost_gitmodules.py: the rows `member-gitmodules-absent` reads and the members `remove`'s refusal is for are pinned, so a finding for a row with no gitlink or a hand-edited path, or the refusal over a member with no gitlink, no longer passes (#611) Two tests. In a holder whose HEAD lost `.gitmodules`: a row with no gitlink (`member-gitlink-absent`'s) and a row whose `path:` was edited by hand (`member-path`'s, its mount `member-gitlink-unnamed`'s) get no `member-gitmodules-absent`; and `remove` of a member with no gitlink is not that refusal, but the dry run's `did not match any file(s) known to git`, as before. Each holds a condition of the first commit that its other tests did not: both pass on BASE 396c0ff4 and on that commit. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- tests/test_family_head_lost_gitmodules.py | 43 +++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/tests/test_family_head_lost_gitmodules.py b/tests/test_family_head_lost_gitmodules.py index 159082b9..62337ef1 100644 --- a/tests/test_family_head_lost_gitmodules.py +++ b/tests/test_family_head_lost_gitmodules.py @@ -279,6 +279,25 @@ def test_the_holder_remove_leaves_clones_with_the_other_member( assert gate.returncode == 0, gate.stdout + gate.stderr +def test_remove_of_a_member_with_no_gitlink_is_not_this_refusal( + family, holder, offline, tmp_path): + """The refusal is for a member whose gitlink the index records: that is + the unmount the commit's failure emptied. With none, the run fails at + its first git call, which writes nothing, and the dry run says so by + git's own words, as before; the remedy of this refusal could not help.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + raw(holder, env, "rm", "-q", "--cached", "--", f"members/{MEMBER}") + commit_all(holder, env, "MEMBER's gitlink taken out") + + dry = run_script(FAMILY, *verb_args("remove", holder, family), + "--dry-run", env=env) + + assert dry.returncode == 2, dry.stdout + dry.stderr + assert "member-gitmodules-absent" not in dry.stderr, dry.stderr + assert "did not match any file(s) known to git" in dry.stderr + + def test_remove_of_a_member_whose_section_alone_is_gone_runs_as_before( family, holder, offline, tmp_path): """The refusal is for a holder with NO `.gitmodules`, where the commit @@ -436,6 +455,30 @@ def test_a_holder_with_no_members_is_asked_nothing(empty_holder): assert result.returncode == 0, (extra, result.stderr + result.stdout) +def test_a_row_the_gitlink_checks_do_not_read_is_asked_nothing( + holder, offline, tmp_path): + """The rows `_check_member` reads at their path, and no other, in a + holder whose HEAD lost `.gitmodules`. OTHER's gitlink taken out too: a + row with no gitlink is `member-gitlink-absent`'s, members checked. + MEMBER's row given another `path:` by hand: that row is `member-path`'s, + and its mount `member-gitlink-unnamed`'s.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + raw(holder, env, "rm", "-q", "--cached", "--", f"members/{OTHER}") + commit_all(holder, env, "OTHER's gitlink taken out") + rows = (holder / "family.yaml").read_bytes() + line = f" path: members/{MEMBER}\n".encode() + assert rows.count(line) == 1, "fixture drift: MEMBER's path line" + (holder / "family.yaml").write_bytes(rows.replace( + line, f" path: members/{MEMBER}-moved\n".encode())) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stderr + result.stdout + assert "member-gitlink-unnamed" in result.stderr, result.stderr + assert "member-gitmodules-absent" not in result.stderr, result.stderr + + def test_an_untracked_copy_put_back_is_still_a_finding_until_committed( holder, offline, tmp_path): """A `.gitmodules` put back by hand, untracked: the holder's own From 6aace429305eb4f0c9e1e3f7cfa05091b8fbb67f Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 01:35:46 +0000 Subject: [PATCH 03/15] family.py and validate-family.py: `remove` refuses a holder with no `.gitmodules` over an unmerged gitlink too, and the case-variant move prints `--ignore-unmatch` and the `mv` a case-sensitive disk needs, so the mount is no longer emptied there and the printed lines work half typed or with both directories on disk (#611) Fix round 1, from the review of 6f938ed1. `_checked_gitmodules_held` (scripts/family.py) took the member's gitlink only at stage 0. `git submodule deinit` and `git rm` run on an unmerged gitlink as on a merged one, so with `members/

` held only at stages 1, 2 and 3 in a holder whose HEAD lost `.gitmodules`, the dry run exited 0 and the real run emptied the mount and died at its commit. It now takes a gitlink at any stage, and both runs refuse `member-gitmodules-absent` before `deinit`, with the holder byte for byte as it was. The docstring says so. The case-variant refusal of `member-gitlink-absent` (templates/family-root/scripts/validate-family.py, pinned) now prints `git rm -r --cached --ignore-unmatch -- `: a variant whose removal is staged and not committed is still named from HEAD, and the plain line failed there (`pathspec ... did not match`). Its remediation also says what is left on disk: where nothing is checked out at the row's path, as in a fresh clone, `git submodule update --init --recursive` then checks the member out; on a case-sensitive disk a directory left at a variant stays, untracked, and `family.py add` refuses it as `family-members-dir-variant`, so it gives an `mv` of each out of the holder, to `../`, and says to leave it on macOS and Windows, where it is the row's own directory. The validator's module docstring names `member-gitmodules-absent`, says what a staged, uncommitted `.gitmodules` does (HEAD's file is what counts, as #407's decision 3 says: an uncommitted `add` is the finding for its member, a staged section removal is none, a staged restore is still the finding), and gives EXIT CODES' one exception, an unregistered member that is not checked out (exit 1, not 2). Tests (tests/test_family_head_lost_gitmodules.py, 12 more, 38 in all): the unmerged gitlink refused alike and repaired as printed; one test per condition no test held (a checked-out member still gets its tree check beside the finding; HEAD's unreadable file refused in every mode; the `` placeholder; `remove`'s remedy names only members with a gitlink in the index; one finding per path for a duplicate row); the variant move half typed (both disks); and, on a case-sensitive disk, every printed line typed with both directories present, after which the gate passes and another member is added. tests/test_validate_family_unnamed_mounts.py pins the `--ignore-unmatch` line. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- scripts/family.py | 12 +- .../family-root/scripts/validate-family.py | 60 +++- tests/test_family_head_lost_gitmodules.py | 292 ++++++++++++++++-- tests/test_validate_family_unnamed_mounts.py | 2 +- 4 files changed, 322 insertions(+), 44 deletions(-) diff --git a/scripts/family.py b/scripts/family.py index 85b2f87a..32cc1e34 100755 --- a/scripts/family.py +++ b/scripts/family.py @@ -5408,7 +5408,7 @@ def _checked_gitmodules_held(family: Family, path: str) -> None: """`member-gitmodules-absent`: `remove` refuses, before any git command that writes and under `--dry-run` alike, a holder that holds no `.gitmodules` at all, in the work tree, the index or HEAD, while its - index records the member's gitlink (#611). + index records the member's gitlink at any stage (#611). A HOLDER WHOSE HEAD LOST `.gitmodules`, as #407's commit left one. `git rm` edits only a `.gitmodules` the work tree has, and `remove` commits @@ -5416,7 +5416,11 @@ def _checked_gitmodules_held(family: Family, path: str) -> None: add` died (`pathspec '.gitmodules' did not match any files`) after `deinit` and `git rm` had emptied the member, and the rollback left its directory empty; the dry run exited 0. `_checked_gitmodules_as_staged` - asks nothing here, since no copy exists to compare. + asks nothing here, since no copy exists to compare. It asks only where + none of the work tree, the index and HEAD holds a `.gitmodules`, and the + index records the member's gitlink at any stage: `deinit` and `git rm` + run on a merged or an unmerged gitlink alike, and with no `.gitmodules` + anywhere the commit cannot stage one. REFUSED, NOT COMMITTED WITHOUT IT, as #407 refuses a `.gitmodules` the work tree lacks: the holder is left byte for byte as it was. A clone @@ -5436,9 +5440,7 @@ def _checked_gitmodules_held(family: Family, path: str) -> None: return gitlinks = {name.decode("utf-8", "surrogateescape") for name, held in records.items() - if any(record.startswith(b"160000 ") - and record.partition(b"\t")[0].endswith(b" 0") - for record in held)} + if any(record.startswith(b"160000 ") for record in held)} if path not in gitlinks: return sections = [] diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index e4c2200c..d4283032 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -42,7 +42,8 @@ finding and none of the three is asked of it: nothing read through a path the row may not have is a fact about the member (#367). A row whose gitlink is recorded only at a case variant of its path is refused - naming that variant, and the index move that puts it back (#611). + naming that variant, the index move that puts it back, and what to do + with a directory the variant leaves on disk (#611). And, across the rows, that no two record ONE `id`: that is one project pinned twice, however each row spelled its repository (#206). This one reads `family.yaml` alone, so `--no-members` asks it too. @@ -60,12 +61,18 @@ and what it says to do depends on which of the two is the member's own: it names no command that would un-register the member. - that a section of HEAD's `.gitmodules` gives the path of every row's - gitlink (#611, #407's decision 3). A recursive clone reads HEAD's file, - so a gitlink no section of it gives is left empty there, by CI and by - `make bootstrap`, though the holder it was committed in keeps its - checkout. It needs no member checked out, so `--no-members` asks it - too, and a member it names that is not checked out is not then refused - as uninitialized. + gitlink, the `member-gitmodules-absent` finding (#611, #407's decision + 3). A recursive clone reads HEAD's file, so a gitlink no section of it + gives is left empty there, by CI and by `make bootstrap`, though the + holder it was committed in keeps its checkout. It needs no member + checked out, so `--no-members` asks it too, and a member it names that + is not checked out is not then refused as uninitialized. A + `.gitmodules` staged and not committed counts for nothing until it is + committed, since HEAD's file is the one a clone reads: a `family.py + add` staged and not committed is this finding for its own member; a + section taken out and staged, not committed, is no finding; and a file + put back and staged, not committed, in a holder whose HEAD lost it is + still this finding for every member. - the COPIED shape files, each against its `sha256` row in `contracts/shape-pin.yaml`. Editing a copy in place is drift and the exit is to carry it upstream, never to update the digest. @@ -74,8 +81,10 @@ 0 the manifest is valid, every member is in lockstep, every copy matches 1 a FINDING: drift a reviewer can act on 2 a REFUSAL: the question could not be asked — no manifest, an - uninitialized member, an unreadable pin, no git to run. An - unanswerable question is never an implicit pass. + uninitialized member (but for one no section of HEAD's `.gitmodules` + gives: that is `member-gitmodules-absent`, exit 1), an unreadable + pin, no git to run. An unanswerable question is never an implicit + pass. """ from __future__ import annotations @@ -449,10 +458,26 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, records the member once, at its own path. `member-gitlink-unnamed` prints no command for the same gitlink, since taking it out ALONE would un-register the member; these take it out and record it together. + `--ignore-unmatch`, since a variant whose removal is staged and not yet + committed is still named here, from HEAD, and the plain `git rm` of it + failed (`pathspec ... did not match`). + + WHAT IS LEFT ON DISK. Where nothing is checked out at the row's path, + as in a fresh clone, `git submodule update --init --recursive` checks + the member out there once the move is committed. On a case-sensitive + disk a directory at a variant (the empty one a fresh clone leaves, or a + checkout) stays behind, untracked, and `family.py add` refuses it as + `family-members-dir-variant` (#574), so the text says to move it out of + the holder, by an `mv` with nothing to fill in. On macOS and Windows + that directory is the row's own, and the text says to leave it there. """ found = ", ".join(f"{variant} (at {gitlinks[variant]})" for variant in variants) taken_out = " ".join(shlex.quote(variant) for variant in variants) + moved = " and ".join( + f"`mv -- {shlex.quote(variant)} " + f"{shlex.quote('../' + variant.replace('/', '-'))}`" + for variant in variants) return Refusal( "member-gitlink-absent", f"no gitlink (mode 160000) recorded at {path!r}. {project} is named " @@ -462,8 +487,16 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, f"at {path}.", "Remediation: the member is recorded in the wrong case. Move its " "gitlink to the row's path, at the commit the row pins, in the index " - "alone, which touches no file, and commit that:\n" - f" git rm -r --cached -- {taken_out}\n" + "alone, which touches no file, with the two lines below, and commit " + f"that. Where nothing is checked out at {path}, as in a fresh clone, " + "`git submodule update --init --recursive` then checks the member " + "out there. On a case-sensitive disk, as Linux's usually is, a " + f"directory left at {' or '.join(variants)} stays there, untracked, " + "and family.py add refuses it as family-members-dir-variant: move " + f"each one that is there out of the holder, {moved}. On macOS and " + "Windows it is the row's own directory, so leave it. The two " + "lines:\n" + f" git rm -r --cached --ignore-unmatch -- {taken_out}\n" " git update-index --add --cacheinfo " f"{shlex.quote(f'160000,{commit},{path}')}") @@ -1342,6 +1375,11 @@ def _check_gitlinks_registered(root: Path, members: list, `member-path`'s, and one with no gitlink there is `member-gitlink-absent`'s (`gitlinks`, read as `recorded_gitlink` reads one). A holder with no members holds no gitlink and is asked nothing. + THREE SOURCES: the gitlinks from the index first, the rows from the + work tree's `family.yaml`, and the sections from HEAD alone, as #407's + decision 3 has it, so a `.gitmodules` staged and not committed counts + for nothing until it is committed (the module docstring gives the three + cases). The remedy writes the section as `family.py add` writes it, named after its path, with the url it gives a member, `https://github.com/` and the diff --git a/tests/test_family_head_lost_gitmodules.py b/tests/test_family_head_lost_gitmodules.py index 62337ef1..85f35308 100644 --- a/tests/test_family_head_lost_gitmodules.py +++ b/tests/test_family_head_lost_gitmodules.py @@ -28,6 +28,12 @@ that HEAD holds and the index no longer does (#574), so it says to commit the removal between them now. +AND THE STATES BESIDE THEM that the first round's review found: a member's +gitlink held only at the conflict stages, which `remove` emptied as it +emptied a merged one (its S1); a condition of each new check that no test +held (its S2 and N5); and the variant move typed half way, or leaving a +directory on a case-sensitive disk (its N1 and N2). + THE SAME FAMILY, OFFLINE RULES AND READINGS as `test_family_remove_gitmodules.py` (#407), whose helpers are imported. Every remedy is typed as printed: split as a POSIX shell splits it, run with no @@ -45,6 +51,8 @@ import re import shlex +import shutil +import subprocess from pathlib import Path import pytest @@ -54,19 +62,21 @@ #: The family fixture, its names, the offline environment and the readings #: of a holder, imported from the modules that own them rather than rebuilt. from test_family import ( # noqa: F401 - FAMILY, NAME, ORG, SPARE, family, validate, + FAMILY, NAME, ORG, SPARE, family, manifest, validate, ) from test_family_add_excludes import empty_holder # noqa: F401 from test_family_add_refusals import holder, offline # noqa: F401 -from test_family_dry_run import isolated +from test_family_dry_run import hand_edit_digest, isolated from test_family_member_path import MEMBER, OTHER, raw from test_family_remove_gitmodules import ( deleted, edited, head_paths, left_out, ran, refused_alike, typed_as_printed, verb_args, ) from test_scaffold_ignored_shape_files import excluding +from test_validate_family_duplicates import EXACT, irrs_row, rewrite from test_validate_family_unnamed_mounts import ( - commands, findings, spare_commits, stage_gitlink, + codes, commands, findings, spare_commits, stage_gitlink, + stage_unmerged_gitlink, ) GITMODULES = ".gitmodules" @@ -125,16 +135,17 @@ def remove_remedy(*projects: str) -> str: def absent(project: str, commit: str, *, held: bool, - with_url: bool = True) -> str: + with_url: bool = True, printed_url: str | None = None) -> str: """The `member-gitmodules-absent` finding for `project`'s gitlink at `commit`, WHOLE: its line and the commands under it. `held` is whether HEAD holds a `.gitmodules` at all; `with_url` whether the section to - write has no url of its own.""" + write has no url of its own; `printed_url` the url line's value where + it is not `url(project)`.""" path = f"members/{project}" lines = f"\n git config -f .gitmodules submodule.{path}.path {path}" if with_url: lines += (f"\n git config -f .gitmodules submodule.{path}.url " - f"{url(project)}") + f"{printed_url or url(project)}") return (f"FINDING member-gitmodules-absent: {path} is a gitlink at " f"{commit} that a row of members: names, and " + ("no section of HEAD's .gitmodules gives that path" if held @@ -315,6 +326,54 @@ def test_remove_of_a_member_whose_section_alone_is_gone_runs_as_before( assert_passes_in_every_mode(holder, 1) +def test_remove_refuses_it_where_the_members_gitlink_is_unmerged( + family, holder, offline, tmp_path): + """THE REVIEW'S S1: HEAD lost `.gitmodules`, and MEMBER's gitlink is held + only at the conflict stages 1, 2 and 3 (`stage_unmerged_gitlink`). + `deinit` and `git rm` run on an unmerged gitlink as on a merged one, so + while the refusal asked only for one at stage 0, the dry run exited 0 + and the real run emptied the mount and died at its commit. Both refuse + now, with the holder byte for byte as it was, the mount's files and the + three stages included. Typed as printed, the remedy writes both + sections back, and `remove` then commits.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + path = f"members/{MEMBER}" + stage_unmerged_gitlink(holder, path, pinned(holder, MEMBER)) + listed = raw(holder, env, "ls-files", "-s", "--", path).stdout + assert [line.split()[2] for line in listed.splitlines()] == [ + b"1", b"2", b"3"], "premise: unmerged, no stage 0" + args = verb_args("remove", holder, family) + + said, remediation = refused_alike(holder, env, args) + + assert said == REMOVE_REFUSED.format(member=MEMBER) + assert remediation == remove_remedy(MEMBER, OTHER) + assert (holder / path / "project.yaml").is_file() + typed_as_printed(holder, env, remediation) + ran(holder, env, args) + assert head_paths(holder, env) == [f"members/{OTHER}"] + assert_passes_in_every_mode(holder, 1) + + +def test_removes_remedy_names_only_members_whose_gitlink_is_indexed( + family, holder, offline, tmp_path): + """The remedy writes back the section of each member whose gitlink the + index records, and of no other. OTHER's gitlink taken out and its row + left: a section for it would register a path the holder records + nothing at, so the remedy writes MEMBER's section alone.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + raw(holder, env, "rm", "-q", "--cached", "--", f"members/{OTHER}") + commit_all(holder, env, "OTHER's gitlink taken out") + + said, remediation = refused_alike(holder, env, + verb_args("remove", holder, family)) + + assert said == REMOVE_REFUSED.format(member=MEMBER) + assert remediation == remove_remedy(MEMBER) + + # --- 571(c): a gitlink no section of HEAD's .gitmodules gives --------------- @MODES @@ -499,6 +558,104 @@ def test_an_untracked_copy_put_back_is_still_a_finding_until_committed( assert_passes_in_every_mode(holder, 2) +@pytest.mark.parametrize("extra", [(), ("--pins",)], + ids=["members-checked", "pins"]) +def test_a_checked_out_member_it_names_still_has_its_tree_checked( + holder, offline, tmp_path, extra): + """Only a member it names that is NOT checked out stops after its + gitlink. One that is checked out is asked its tree as before: MEMBER's + digest edited by hand, in a holder whose HEAD lost `.gitmodules`, is + `member-digest-mismatch` beside the two `member-gitmodules-absent`.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + hand_edit_digest(holder, env) + + result = validate(holder, *extra) + + assert result.returncode == 1, result.stderr + result.stdout + assert codes(result) == ["member-gitmodules-absent"] * 2 + [ + "member-digest-mismatch"] + assert findings(result)[2].startswith( + f"FINDING member-digest-mismatch: {MEMBER}: pin.tree_sha256 " + f"{'b' * 64}\n"), result.stderr + + +@MODES +def test_a_head_gitmodules_git_cannot_read_is_refused(holder, offline, + tmp_path, extra): + """HEAD's `.gitmodules` committed broken, the work tree's mended and not + committed. A clone reads HEAD's, which git cannot read either, so the + gate refuses `gitmodules-unreadable` in every mode, saying to commit the + mended file, and never reads the broken file as one that gives no + section. Committed, the holder passes.""" + env = isolated(offline, tmp_path) + mended = (holder / GITMODULES).read_bytes() + committed_gitmodules(holder, env, + f'[submodule "members/{MEMBER}"\n\tpath = x\n' + .encode("utf-8")) + (holder / GITMODULES).write_bytes(mended) + + result = validate(holder, *extra) + + assert result.returncode == 2, result.stderr + result.stdout + lines = result.stderr.splitlines() + assert lines[0].startswith( + "REFUSED gitmodules-unreadable: HEAD's .gitmodules could not be read " + "as git reads it: `git config -z --blob "), result.stderr + assert lines[-1] == ( + "Remediation: git cannot read it either, so none of this holder's " + "submodules can be cloned from it. Mend the file by hand (`git config " + "-f .gitmodules --list` says where it breaks), commit it, then run " + "this again.") + staged_and_committed(holder, env, "Commit the mended .gitmodules") + assert_passes_in_every_mode(holder, 2) + + +def test_a_repository_not_org_and_name_prints_the_url_placeholder( + holder, offline, tmp_path): + """The url line gives `https://github.com/.git`, the url + `family.py add` writes, only where the row's `repository:` is + `/`. From any other value that would be a url nobody chose, + so the line prints `` for the person to fill in.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + rows = (holder / "family.yaml").read_bytes() + line = f" repository: {ORG}/{MEMBER}\n".encode("utf-8") + assert rows.count(line) == 1, "fixture drift: MEMBER's repository line" + (holder / "family.yaml").write_bytes(rows.replace( + line, f" repository: {MEMBER}\n".encode("utf-8"))) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stderr + result.stdout + assert findings(result) == [ + absent(MEMBER, pinned(holder, MEMBER), held=False, + printed_url=""), + absent(OTHER, pinned(holder, OTHER), held=False)] + + +def test_a_row_written_twice_is_one_finding_per_path(holder, offline, + tmp_path): + """MEMBER's row written twice in a holder whose HEAD lost `.gitmodules`: + `member-duplicate` once, and `member-gitmodules-absent` once per path, + not once per row.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + row = irrs_row(holder) + assert row["project"] == MEMBER, "fixture drift: MEMBER is IRRS" + other = next(r for r in manifest(holder)["members"] + if r["project"] == OTHER) + rewrite(holder, [row, dict(row), other]) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stderr + result.stdout + assert findings(result) == [ + EXACT, + absent(MEMBER, pinned(holder, MEMBER), held=False), + absent(OTHER, pinned(holder, OTHER), held=False)] + + # --- 574/2: a gitlink recorded only at a case variant of the row's path ----- def variant_refused(path: str, commit: str, variants: list[str], @@ -508,6 +665,8 @@ def variant_refused(path: str, commit: str, variants: list[str], project = path.split("/", 1)[1] found = ", ".join(f"{variant} (at {gitlinks[variant]})" for variant in variants) + moved = " and ".join(f"`mv -- {variant} ../{variant.replace('/', '-')}`" + for variant in variants) return [ f"REFUSED member-gitlink-absent: no gitlink (mode 160000) recorded at " f"'{path}'. {project} is named in family.yaml, and this repository " @@ -516,11 +675,44 @@ def variant_refused(path: str, commit: str, variants: list[str], f"recursive clone checks out nothing at {path}.", "Remediation: the member is recorded in the wrong case. Move its " "gitlink to the row's path, at the commit the row pins, in the index " - "alone, which touches no file, and commit that:", - " git rm -r --cached -- " + " ".join(variants), + "alone, which touches no file, with the two lines below, and commit " + f"that. Where nothing is checked out at {path}, as in a fresh clone, " + "`git submodule update --init --recursive` then checks the member " + "out there. On a case-sensitive disk, as Linux's usually is, a " + f"directory left at {' or '.join(variants)} stays there, untracked, " + "and family.py add refuses it as family-members-dir-variant: move " + f"each one that is there out of the holder, {moved}. On macOS and " + "Windows it is the row's own directory, so leave it. The two lines:", + " git rm -r --cached --ignore-unmatch -- " + " ".join(variants), f" git update-index --add --cacheinfo 160000,{commit},{path}"] +def refused_at(root: Path, commit: str, variants: list[str]) -> list[str]: + """The members-checked gate in `root`, where MEMBER's gitlink is recorded + only at `variants`, at `commit`: exit 2, in `variant_refused`'s lines, + which it returns.""" + result = validate(root) + assert result.returncode == 2, result.stderr + result.stdout + lines = result.stderr.splitlines() + assert lines == variant_refused(f"members/{MEMBER}", commit, variants, + dict.fromkeys(variants, commit)) + return lines + + +def moved_as_printed(root: Path, env: dict, lines: list[str], + variants: list[str]) -> None: + """The refusal's two command lines typed as printed, and the commit it + says to make: HEAD then records MEMBER once, at its own path, and at + none of `variants`.""" + for line in lines[2:]: + raw(root, env, *shlex.split(line)[1:]) + commit_all(root, env, "Move the gitlink as printed") + listed = raw(root, env, "ls-tree", "-r", "--name-only", "HEAD").stdout + assert f"members/{MEMBER}\n".encode() in listed + for variant in variants: + assert f"{variant}\n".encode() not in listed + + def recorded_at(root: Path, env: dict, project: str, *spellings: str) -> str: """`project`'s gitlink taken out of the index at its own path and @@ -551,23 +743,74 @@ def test_a_gitlink_only_at_a_case_variant_is_named_and_moved_as_printed( variants = sorted(spelling.format(p=MEMBER) for spelling in spellings) commit = recorded_at(holder, env, MEMBER, *variants) - result = validate(holder) + lines = refused_at(holder, commit, variants) - assert result.returncode == 2, result.stderr + result.stdout - lines = result.stderr.splitlines() - path = f"members/{MEMBER}" - assert lines == variant_refused(path, commit, variants, - dict.fromkeys(variants, commit)) - for line in lines[2:]: - raw(holder, env, *shlex.split(line)[1:]) - commit_all(holder, env, "Move the gitlink as printed") - listed = raw(holder, env, "ls-tree", "-r", "--name-only", "HEAD").stdout - assert f"{path}\n".encode() in listed - for variant in variants: - assert f"{variant}\n".encode() not in listed + moved_as_printed(holder, env, lines, variants) assert_passes_in_every_mode(holder, 2) +@DISKS +def test_a_variant_whose_removal_is_staged_is_moved_as_printed( + holder, offline, tmp_path, ignorecase): + """HALF REPAIRED (the review's N1): the variant's removal staged and not + committed. HEAD still records the gitlink there, so the refusal still + names it, and the plain `git rm -r --cached` of it failed, `pathspec + ... did not match`. With `--ignore-unmatch` the lines typed as printed + and committed move the gitlink, and the holder passes.""" + env = on_disk(isolated(offline, tmp_path), ignorecase) + variant = f"Members/{MEMBER}" + commit = recorded_at(holder, env, MEMBER, variant) + raw(holder, env, "rm", "-q", "--cached", "--", variant) + + lines = refused_at(holder, commit, [variant]) + + moved_as_printed(holder, env, lines, [variant]) + assert_passes_in_every_mode(holder, 2) + + +@pytest.fixture +def case_sensitive_disk(tmp_path): + """Skips where `Members/` and `members/` would be one directory, as + macOS and Windows hold them by default (nothing is left behind there to + move), or where there is no `mv` to type.""" + (tmp_path / "CaseProbe").mkdir() + if (tmp_path / "caseprobe").exists(): + pytest.skip("this disk is case-insensitive: Members/IRRS is " + "members/IRRS here, so no second directory can exist") + if shutil.which("mv") is None: + pytest.skip("no mv on PATH to type the printed line with") + + +def test_a_variant_checkout_left_on_disk_is_moved_out_as_printed( + family, holder, offline, tmp_path, case_sensitive_disk): + """THE REVIEW'S N2, in its C3 layout: MEMBER's gitlink recorded only at + `Members/IRRS`, its checkout still at `members/IRRS`, and a second + checkout at `Members/IRRS`, as a case-sensitive disk can hold. The + index move leaves that directory on disk, untracked, where `family.py + add` refused any other member past it (`family-members-dir-variant`). + Every line the refusal prints, typed as printed: the two lines, their + commit, and the `mv` it gives for a case-sensitive disk. The gate then + passes in every mode, and SPARE is added.""" + env = isolated(offline, tmp_path) + variant = f"Members/{MEMBER}" + commit = recorded_at(holder, env, MEMBER, variant) + raw(holder, env, "clone", "-q", "--no-checkout", + str(family["remotes"] / f"{MEMBER}.git"), variant) + raw(holder / variant, env, "checkout", "-q", "--detach", commit) + + lines = refused_at(holder, commit, [variant]) + + moved_as_printed(holder, env, lines, [variant]) + (move,) = re.findall(r"`(mv [^`]*)`", lines[1]) + subprocess.run(shlex.split(move), cwd=str(holder), check=True) + assert not (holder / variant).exists() + assert (tmp_path / f"Members-{MEMBER}" / "project.yaml").is_file() + assert_passes_in_every_mode(holder, 2) + added = run_script(FAMILY, *verb_args("add", holder, family), env=env) + assert added.returncode == 0, added.stdout + added.stderr + assert_passes_in_every_mode(holder, 3) + + def test_a_fresh_clone_of_a_case_variant_says_it_too(family, holder, offline, tmp_path): """The same state pushed: a fresh clone names the variant, where the @@ -581,12 +824,7 @@ def test_a_fresh_clone_of_a_case_variant_says_it_too(family, holder, offline, cloned = clone_just_pushed(remote, clone, "w611/case", recurse=False) assert cloned.returncode == 0, cloned.stdout + cloned.stderr - result = validate(clone) - - assert result.returncode == 2, result.stderr + result.stdout - variant = f"Members/{MEMBER}" - assert result.stderr.splitlines() == variant_refused( - f"members/{MEMBER}", commit, [variant], {variant: commit}) + refused_at(clone, commit, [f"Members/{MEMBER}"]) # --- 574/3: a hand mount's remedy commits the removal before the add --------- diff --git a/tests/test_validate_family_unnamed_mounts.py b/tests/test_validate_family_unnamed_mounts.py index 7c4babf5..3144f710 100644 --- a/tests/test_validate_family_unnamed_mounts.py +++ b/tests/test_validate_family_unnamed_mounts.py @@ -538,7 +538,7 @@ def test_a_member_recorded_only_in_another_case_says_so_and_prints_no_command( "REFUSED member-gitlink-absent: no gitlink (mode 160000) " "recorded at 'members/IRSS'."), result.stderr assert result.stderr.splitlines()[-2:] == [ - " git rm -r --cached -- members/irss", + " git rm -r --cached --ignore-unmatch -- members/irss", " git update-index --add --cacheinfo " f"160000,{named},members/IRSS"], result.stderr return From d660ec564e652be9e2d44de93c53f22b962e757c Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 12:12:05 +0000 Subject: [PATCH 04/15] bootstrap.py and validate-family.py: a member no credential would check out fails the family bootstrap, every row's gitlink is asked before any row's checkout, and `--no-members` prints the case-variant move, so a clone of a holder whose HEAD lost `.gitmodules` or of #363's state no longer ends `family bootstrap ok` or hides the variant behind `member-uninitialized` (#648) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (a) `templates/family-root/scripts/bootstrap.py` said "every member and leg fetched" and ended `family bootstrap ok`, exit 0, over members it had not checked out: in a fresh recursive clone of a holder whose HEAD lost `.gitmodules` (git exits 0), and in a plain clone of that holder or of #363's state, where git dies `No url found for submodule path` (exit 128). After the fetch it now asks this holder's own records of each row whose member is not checked out: no gitlink at the row's path (`recorded_gitlink`, as `member-gitlink-absent` reads it), or no section of `.gitmodules` giving that path, or, where git exited 0, any such row. Each is named on one `NOT CHECKED OUT, and not for want of a credential` line with why, which names `validate-family.py`, and the run ends `family bootstrap FAILED:

(not checked out)`, exit 1. "every member and leg fetched" is said only where git exited 0 and every row is checked out. A row git could not fetch for want of a credential is still the skip it was (`test_bootstrap_degrades_when_a_member_cannot_be_fetched`), and a good holder's run prints what it printed. (b) `validate-family.py` asked each row whole, gitlink then checkout, before the next. In a plain clone of #363's state plus `git submodule update --init --recursive`, which dies at `Members/

` before it checks out any member, an earlier row was refused `member-uninitialized` (naming that same update) and the later row's variant refusal was never reached. `_check_member` now stops at the gitlink and returns the member (`Gitlinked`); `_check_member_checkout` asks the digest and identity of each once every row's gitlink has been asked. (c) Under `--no-members`, `member-gitlink-unnamed` for a member's only gitlink, in the wrong case, printed no command. Where the row's path is its canonical one and its pin a commit, it now prints the index move the row's members-mode refusal prints, word for word (`_wrong_case_move`, shared by both), to the pinned commit, every variant on one line. Docs: the validator's WHAT IS CHECKED docstring (the order; the move under --no-members) and bootstrap's (a) and (c). README.md's "A member it cannot fetch is reported and skipped, never a failure" is no longer true, so it says "One it lacks a credential for is reported and skipped, not a failure", in the same line count (README is at its 1449-line cap), and docs/handbook.html says the same, with its § Families marker set to the segment's new blob. Pinned copies changed: templates/family-root/scripts/bootstrap.py and templates/family-root/scripts/validate-family.py. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/handbook.html | 4 +- templates/family-root/scripts/bootstrap.py | 135 ++++- .../family-root/scripts/validate-family.py | 177 +++++-- tests/test_family_bootstrap_gate_order.py | 468 ++++++++++++++++++ tests/test_validate_family_unnamed_mounts.py | 32 +- 6 files changed, 728 insertions(+), 90 deletions(-) create mode 100644 tests/test_family_bootstrap_gate_order.py diff --git a/README.md b/README.md index 2664a842..4ff2989f 100644 --- a/README.md +++ b/README.md @@ -942,7 +942,7 @@ at the right commit. **A family pins assembly roots, never legs**: the GitHub App or `SHAPE_LEGS_TOKEN` credential the workflow does, as a member is a private assembly root mounting two private legs — then runs each member's own `make bootstrap`; `make validate` runs the family validator, then each -member's. A member it cannot fetch is reported and skipped, never a failure. +member's. One it lacks a credential for is reported and skipped, not a failure. The holder carries the same COPY pin an assembly root does, so `update-shape.py` re-syncs it the same way; it mirrors into `family.yaml` diff --git a/docs/handbook.html b/docs/handbook.html index 801cc4ac..82ec73d9 100644 --- a/docs/handbook.html +++ b/docs/handbook.html @@ -1268,7 +1268,7 @@

Adopting an existing repository

- +

Reference

Families: a holder for projects that ship separately

@@ -1361,7 +1361,7 @@

The InkRouter example

resolving the same GitHub App or SHAPE_LEGS_TOKEN credential the workflow does — and then runs each member’s own make bootstrap. make validate runs the family validator and then each member’s own - make validate. A member that could not be fetched is reported and skipped, never + make validate. A member it lacks a credential for is reported and skipped, not a failure. The holder carries the same COPY pin an assembly root does, so update-shape.py re-syncs it the same way; it mirrors into family.yaml instead of project.yaml and must leave diff --git a/templates/family-root/scripts/bootstrap.py b/templates/family-root/scripts/bootstrap.py index d9ddcc30..e3fb3b10 100755 --- a/templates/family-root/scripts/bootstrap.py +++ b/templates/family-root/scripts/bootstrap.py @@ -27,6 +27,18 @@ refused to do anything because one member is unreachable would be worse than one that says which. + "every member and leg fetched" is said only where git exited 0 AND + every row's member is checked out (#648). A member left NOT CHECKED + OUT where no credential would check it out is said here instead, with + why, and fails the run (c): this holder records no gitlink at the + row's path (a gitlink recorded only in another case, #363), no + section of `.gitmodules` gives that path (a holder whose HEAD lost the + file, #611), or git exited 0 and checked nothing out there. Git exits + 0 over the second where a recursive clone ran first, and 128 (`No url + found for submodule path`) in a plain clone, as it does for a missing + credential, so what the holder records is the test, never git's + words. The line names `validate-family.py`, which says what to commit. + (b) RUNS EACH MEMBER'S OWN `make bootstrap`. Each member is a whole project and knows how to bootstrap itself — its legs onto their tracking branches at their pinned commits, its own validators, its own @@ -35,9 +47,10 @@ `--members-make validate` runs `make validate` instead, which is what the family Makefile's `validate` target uses. - (c) EXITS 0 unless a member's own command failed. A member that is not - checked out is REPORTED and skipped, not a failure: it is the state a - missing credential leaves, and it is already reported in (a). + (c) EXITS 0 unless a member's own command failed, or (a) left a member not + checked out that no credential would check out. Any other member that + is not checked out is REPORTED and skipped, not a failure: it is the + state a missing credential leaves, and it is already reported in (a). THE FAMILY CONFERS NOTHING and neither does this command. It fetches, it delegates, and it says what it could not reach. @@ -54,10 +67,12 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from repo_shape import ( # noqa: E402 - Refusal, escape_unencodable_stdout, find_repo_root, load_yaml, spelled, + PYTHON, Refusal, escape_unencodable_stdout, find_repo_root, git_out, + load_yaml, recorded_gitlink, spelled, ) MANIFEST = "family.yaml" +GITMODULES = ".gitmodules" DEFAULT_MEMBERS_DIR = "members" #: The two environment variables a leg/member credential may arrive in, in the @@ -79,8 +94,13 @@ def credential() -> tuple[str, str]: return "none", "" -def fetch_members(root: Path) -> bool: - """(a) `git submodule update --init --recursive`, authenticated or not.""" +def fetch_members(root: Path, rows: list[dict]) -> list[str]: + """(a) `git submodule update --init --recursive`, authenticated or not. + + Returns a failure line for each of `rows` whose member it leaves not + checked out where no credential would check it out (`unfetchable`), for + the run to end FAILED on (#648); every member and leg is said to be + fetched only where there is none and git exited 0.""" source, token = credential() args = ["git"] if token: @@ -92,21 +112,83 @@ def fetch_members(root: Path) -> bool: sys.stdout.flush() proc = subprocess.run(args, cwd=str(root), capture_output=True, text=True, encoding="utf-8", errors="replace", check=False) - if proc.returncode == 0: + if proc.returncode != 0: + # The token can appear in git's own error text (it is in the URL it + # tried), so the output is scrubbed before it is printed. + message = (proc.stderr + proc.stdout).strip() + if token: + message = message.replace(token, "") + print(f" FETCH INCOMPLETE (git exit {proc.returncode}) using " + f"credential source '{source}':\n{message}", file=sys.stderr) + print(" If the members are PRIVATE, set SHAPE_LEGS_TOKEN (a " + "fine-grained PAT with contents:read on them) or export an " + "App-minted SHAPE_LEGS_APP_TOKEN, then re-run. What did arrive " + "is bootstrapped below.", file=sys.stderr) + left = unfetchable(root, rows, proc.returncode == 0) + if left: + print(" NOT CHECKED OUT, and not for want of a credential: " + + ", ".join(f"{spelled(path)} ({why})" for _, path, why in left) + + f". `{PYTHON} scripts/validate-family.py` names the cause and " + "the commit that cures it.", file=sys.stderr) + elif proc.returncode == 0: print(" every member and leg fetched") - return True - # The token can appear in git's own error text (it is in the URL it tried), - # so the output is scrubbed before it is printed. - message = (proc.stderr + proc.stdout).strip() - if token: - message = message.replace(token, "") - print(f" FETCH INCOMPLETE (git exit {proc.returncode}) using credential " - f"source '{source}':\n{message}", file=sys.stderr) - print(" If the members are PRIVATE, set SHAPE_LEGS_TOKEN (a fine-grained " - "PAT with contents:read on them) or export an App-minted " - "SHAPE_LEGS_APP_TOKEN, then re-run. What did arrive is bootstrapped " - "below.", file=sys.stderr) - return False + return [f"{spelled(row.get('project') or '?')} (not checked out)" + for row, _, _ in left] + + +def gitmodules_paths(root: Path) -> set[str]: + """Every path a section of `.gitmodules` gives, read as git reads it: + the work tree's file, the one `git submodule update` reads, which a + fresh clone takes from HEAD. None where there is no such file, or one + git cannot read: git has no url for any submodule then either.""" + if not os.path.lexists(root / GITMODULES): + return set() + try: + listed = git_out(["config", "-z", "--file", GITMODULES, "--list"], + cwd=root, binary=True) + except Refusal: + return set() + paths: set[str] = set() + for record in listed.split(b"\0"): + key, _, value = record.decode("utf-8", + "surrogateescape").partition("\n") + if key.startswith("submodule.") and key.endswith(".path"): + paths.add(value) + return paths + + +def unfetchable(root: Path, rows: list[dict], + exited_0: bool) -> list[tuple[dict, str, str]]: + """`(row, path, why)` for each row whose member (a) left NOT CHECKED OUT + where no credential would check it out (#648), in row order. + + Asked of this holder's own records, never of git's words: a row at + whose path the holder records no gitlink (`recorded_gitlink`, + as `validate-family.py`'s `member-gitlink-absent` reads one), or whose + gitlink no section of `.gitmodules` gives a path (as its + `member-gitmodules-absent` reads HEAD's file, which is the work tree's + in a clone); and, where git exited 0, any row at all, since git then + lacked nothing a credential gives. Any other row not checked out is the + missing credential's, and stays the skip it always was. A path two rows + give is said once.""" + left: dict[str, dict] = {} + for row in rows: + if not (member_path(root, row) / ".git").exists(): + left.setdefault(row_path(row), row) + if not left: + return [] + given = gitmodules_paths(root) + found: list[tuple[dict, str, str]] = [] + for path, row in left.items(): + if recorded_gitlink(root, path) is None: + found.append((row, path, "no gitlink is recorded at that path")) + elif path not in given: + found.append((row, path, f"no section of {GITMODULES} gives that " + "path")) + elif exited_0: + found.append((row, path, "git exited 0 and checked nothing out " + "there")) + return found def members_of(root: Path) -> tuple[str, list[dict]]: @@ -128,9 +210,14 @@ def members_of(root: Path) -> tuple[str, list[dict]]: return name, rows +def row_path(row: dict) -> str: + """The path a row mounts its member at, as the holder spells it.""" + return str(row.get("path") + or f"{DEFAULT_MEMBERS_DIR}/{row.get('project')}") + + def member_path(root: Path, row: dict) -> Path: - return root / str(row.get("path") - or f"{DEFAULT_MEMBERS_DIR}/{row.get('project')}") + return root / row_path(row) def run_member_target(root: Path, row: dict, target: str) -> str | None: @@ -187,13 +274,13 @@ def main(argv: list[str] | None = None) -> int: f"{len(rows)} member(s)") print("\n(a) members and their legs") + failed: list[str] = [] if args.no_fetch: print(" skipped (--no-fetch)") else: - fetch_members(root) + failed += fetch_members(root, rows) print(f"\n(b) each member's own `make {spelled(args.members_make)}`") - failed: list[str] = [] if args.skip_members: print(" skipped (--skip-members)") elif not rows: diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index d4283032..aca84f2b 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -43,7 +43,11 @@ path the row may not have is a fact about the member (#367). A row whose gitlink is recorded only at a case variant of its path is refused naming that variant, the index move that puts it back, and what to do - with a directory the variant leaves on disk (#611). + with a directory the variant leaves on disk (#611). The first is asked + of every row before the other two are asked of any (#648), so a later + row's gitlink refusal is never hidden behind an earlier row's + `member-uninitialized`, as a plain clone whose `git submodule update` + died at that gitlink left it. And, across the rows, that no two record ONE `id`: that is one project pinned twice, however each row spelled its repository (#206). This one reads `family.yaml` alone, so `--no-members` asks it too. @@ -59,7 +63,10 @@ them ever asked. Neither needs a member checked out, so `--no-members` asks them too. A path a row names in another case is still one of these, and what it says to do depends on which of the two is the member's own: - it names no command that would un-register the member. + it names no command that would un-register the member, and for a member + recorded only in the wrong case it names the index move that records it + at its row's path, as the row's own refusal does members checked + (#648). - that a section of HEAD's `.gitmodules` gives the path of every row's gitlink, the `member-gitmodules-absent` finding (#611, #407's decision 3). A recursive clone reads HEAD's file, so a gitlink no section of it @@ -456,11 +463,12 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, commit the row pins, reading no disk either. So the two lines work on every disk, wherever the checkout stands, and the commit after them records the member once, at its own path. `member-gitlink-unnamed` - prints no command for the same gitlink, since taking it out ALONE would - un-register the member; these take it out and record it together. - `--ignore-unmatch`, since a variant whose removal is staged and not yet - committed is still named here, from HEAD, and the plain `git rm` of it - failed (`pathspec ... did not match`). + prints the same move for the same gitlink under `--no-members` (#648), + and no command that takes it out ALONE, which would un-register the + member; these take it out and record it together. `--ignore-unmatch`, + since a variant whose removal is staged and not yet committed is still + named here, from HEAD, and the plain `git rm` of it failed (`pathspec + ... did not match`). WHAT IS LEFT ON DISK. Where nothing is checked out at the row's path, as in a fresh clone, `git submodule update --init --recursive` checks @@ -473,11 +481,6 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, """ found = ", ".join(f"{variant} (at {gitlinks[variant]})" for variant in variants) - taken_out = " ".join(shlex.quote(variant) for variant in variants) - moved = " and ".join( - f"`mv -- {shlex.quote(variant)} " - f"{shlex.quote('../' + variant.replace('/', '-'))}`" - for variant in variants) return Refusal( "member-gitlink-absent", f"no gitlink (mode 160000) recorded at {path!r}. {project} is named " @@ -485,17 +488,31 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, "case variant of that path, which macOS and Windows hold as one " f"directory with it: {found}. A recursive clone checks out nothing " f"at {path}.", - "Remediation: the member is recorded in the wrong case. Move its " - "gitlink to the row's path, at the commit the row pins, in the index " - "alone, which touches no file, with the two lines below, and commit " - f"that. Where nothing is checked out at {path}, as in a fresh clone, " - "`git submodule update --init --recursive` then checks the member " - "out there. On a case-sensitive disk, as Linux's usually is, a " - f"directory left at {' or '.join(variants)} stays there, untracked, " - "and family.py add refuses it as family-members-dir-variant: move " - f"each one that is there out of the holder, {moved}. On macOS and " - "Windows it is the row's own directory, so leave it. The two " - "lines:\n" + "Remediation: the member is recorded in the wrong case. " + + _wrong_case_move(path, commit, variants)) + + +def _wrong_case_move(path: str, commit: str, variants: list[str]) -> str: + """The index move for a member recorded only at `variants`, case + variants of its row's `path`, to `path` at the pinned `commit`, with + what it leaves on disk: what `_gitlink_in_another_case` says, and + `member-gitlink-unnamed` under `--no-members` (#648), in one text.""" + taken_out = " ".join(shlex.quote(variant) for variant in variants) + moved = " and ".join( + f"`mv -- {shlex.quote(variant)} " + f"{shlex.quote('../' + variant.replace('/', '-'))}`" + for variant in variants) + return ( + "Move its gitlink to the row's path, at the commit the row pins, in " + "the index alone, which touches no file, with the two lines below, " + f"and commit that. Where nothing is checked out at {path}, as in a " + "fresh clone, `git submodule update --init --recursive` then checks " + "the member out there. On a case-sensitive disk, as Linux's usually " + f"is, a directory left at {' or '.join(variants)} stays there, " + "untracked, and family.py add refuses it as " + "family-members-dir-variant: move each one that is there out of the " + f"holder, {moved}. On macOS and Windows it is the row's own " + "directory, so leave it. The two lines:\n" f" git rm -r --cached --ignore-unmatch -- {taken_out}\n" " git update-index --add --cacheinfo " f"{shlex.quote(f'160000,{commit},{path}')}") @@ -554,9 +571,22 @@ def _check_member_tree(member: Path, pin: dict, path: str, commit: str, report.note(f"{path}: tree digest recomputes ({recorded[:12]}…)") +class Gitlinked(NamedTuple): + """A member `_check_member` asked up to its gitlink, with what + `_check_member_checkout` reads its checkout by.""" + + row: dict + project: str + path: str + pin: dict + commit: str + + def _check_member(root: Path, row: dict, seen: dict, report: Report, - gitlinks: dict[str, str], unregistered: set[str]) -> None: - """One member: the row itself, the gitlink, the digest, the identity. + gitlinks: dict[str, str]) -> Gitlinked | None: + """One member up to its checkout: the row itself and the gitlink. The + member, for `_check_member_checkout` to ask its digest and identity of, + or None where nothing more is asked of it. NOTHING IS READ THROUGH A PATH THE ROW MAY NOT HAVE (#367). A hand-edited `path:` is `member-path`, and this member's checks stop once its pin is @@ -570,18 +600,10 @@ def _check_member(root: Path, row: dict, seen: dict, report: Report, The git each check runs is run inside `_git_runs`, which names the check when there is no git to run (#368); the identity check reads a file and runs none. - - A MEMBER NO SECTION OF HEAD'S `.gitmodules` GIVES (`unregistered`, the - `member-gitmodules-absent` finding, #611) that is not checked out stops - after its gitlink. Refused `member-uninitialized`, it named `git - submodule update --init`, which exits 0 there and checks nothing out, - and the refusal dropped the finding that says why: the state a fresh - clone of a holder whose HEAD lost `.gitmodules` is in, and CI's. The - finding is made all the same, so the run still fails. """ project = _member_project_name(row, seen, report) if project is None: - return + return None path = _check_member_repository_and_path(row, project, report) @@ -589,23 +611,38 @@ def _check_member(root: Path, row: dict, seen: dict, report: Report, if not isinstance(pin, dict): report.finding("member-pin", f"{project}: `pin:` is {pin!r}, expected " "a mapping") - return + return None commit = _pin_commit(pin, project, report) if path is None: - return + return None # ---- the gitlink ------------------------------------------------------- with _git_runs(f"the gitlink recorded at {path} could not be read"): _check_member_gitlink(root, path, commit, project, gitlinks, report) + return Gitlinked(row, project, path, pin, commit) + - # ---- the digest and the identity -------------------------------------- - member = root / path - if path in unregistered and not (member / ".git").exists(): +def _check_member_checkout(root: Path, asked: Gitlinked, report: Report, + unregistered: set[str]) -> None: + """One member's digest and identity, read in its checkout, asked once + every row's gitlink has been (`_check_members`, #648). + + A MEMBER NO SECTION OF HEAD'S `.gitmodules` GIVES (`unregistered`, the + `member-gitmodules-absent` finding, #611) that is not checked out is + asked nothing here. Refused `member-uninitialized`, it named `git + submodule update --init`, which exits 0 there and checks nothing out, + and the refusal dropped the finding that says why: the state a fresh + clone of a holder whose HEAD lost `.gitmodules` is in, and CI's. The + finding is made all the same, so the run still fails. + """ + member = root / asked.path + if asked.path in unregistered and not (member / ".git").exists(): return - with _git_runs(f"{project}'s pinned commit and its tree digest could not " - f"be read in {path}"): - _check_member_tree(member, pin, path, commit, project, report) - _check_member_identity(member, row, project, report) + with _git_runs(f"{asked.project}'s pinned commit and its tree digest " + f"could not be read in {asked.path}"): + _check_member_tree(member, asked.pin, asked.path, asked.commit, + asked.project, report) + _check_member_identity(member, asked.row, asked.project, report) def _check_member_identity(member: Path, row: dict, project: str, @@ -1074,11 +1111,14 @@ def _member_gitlinks(root: Path) -> dict[str, str]: class RowPath(NamedTuple): """What a row names by the path it gives: `who` names the row in a - finding, and `canonical` is the one path its project may have, - `members/`, or None for a row with no `project:`.""" + finding, `canonical` is the one path its project may have, + `members/`, or None for a row with no `project:`, and `pin` is + the commit its `pin:` names, lowered, or None where that is no 40-hex + commit (#648).""" who: str canonical: str | None + pin: str | None = None def _row_paths(members: list) -> dict[str, RowPath]: @@ -1109,9 +1149,11 @@ def _row_paths(members: list) -> dict[str, RowPath]: path = canonical else: continue + pin = row.get("pin") + commit = str(pin.get("commit") or "") if isinstance(pin, dict) else "" named.setdefault(path, RowPath( f"the row for {project}" if project else "a row with no project:", - canonical)) + canonical, commit.lower() if COMMIT_RE.match(commit) else None)) return named @@ -1190,6 +1232,10 @@ def _variant_advice(kind: str, path: str, row_path: str, row: RowPath, is wrong, and where to put its path back; a `WRONG_CASE` says the member is recorded in the wrong case, and sends the operator to `git log -p -- family.yaml`, as an unnamed path that no row folds to does (`RESTORE_ROW`). + A `WRONG_CASE` gitlink whose row's path is its canonical one and whose + pin is a commit never comes here: `_check_gitlinks_named` prints the + index move for it, which records the member as it takes the gitlink out + (#648). """ if kind == ROW_EDITED: return (f" It is {path}, the one path {row.who} may have, and the row " @@ -1220,9 +1266,15 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], still a finding, and says which of three it is (`_variant_case`). A second path to the member's own directory is taken out of the index alone with `git rm --cached`, which touches no file, and committed. The - member's own gitlink, which the ROW misspells, and the only record of a - member there is, in the wrong case, print no command: either could - un-register the member. Any other is taken out with a plain `git rm`, + member's own gitlink, which the ROW misspells, prints no command, since + taking it out could un-register the member. The only record of a member + there is, in the wrong case, prints the index move members mode's + `member-gitlink-absent` prints (`_wrong_case_move`, #648), which records + the member at its row's path as it takes the variants out, so + `--no-members`, the run CI falls back to when the members cannot be + checked out, names the cure too; where the row's path is not its + canonical one, or its pin no commit, it prints no command either. Any + other is taken out with a plain `git rm`, which refuses while its checkout holds changes, then mounted with `family.py add`: `add` cannot adopt an existing mount, and after `git rm --cached` alone it refuses the checkout left in the way @@ -1247,6 +1299,16 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], if variant is not None: row_path, row = variant kind = _variant_case(path, row_path, row, holds) + if kind == WRONG_CASE and row_path == row.canonical and row.pin: + variants = sorted(other for other in holds + if other != row_path + and other.casefold() == row_path.casefold()) + report.finding( + "member-gitlink-unnamed", + f"{said}{_variant_said(row_path, row)} The member is " + "recorded in the wrong case. " + + _wrong_case_move(row_path, row.pin, variants)) + continue if kind != STRAY: report.finding( "member-gitlink-unnamed", @@ -1436,6 +1498,16 @@ def _check_members(root: Path, members: list, no_members: bool, `.gitmodules` gives twice (#300), then the gitlinks and sections no row names, which read the index and HEAD as well (#361), and the gitlinks no section of HEAD's `.gitmodules` gives (#611). + + THEN EVERY ROW'S GITLINK, AND ONLY THEN ANY ROW'S CHECKOUT (#648). Each + row used to be asked whole before the next was asked at all, so in a + plain clone of a holder whose later row's gitlink is recorded only in + another case, where `git submodule update --init --recursive` dies at + that gitlink (`No url found for submodule path`, exit 128) before it + checks out any member, an earlier row was refused + `member-uninitialized`, naming that same update, and the later row's + `member-gitlink-absent`, which names the move that cures both, was + never reached. """ _check_member_names_unique(members, report) _check_member_ids_unique(root, members, report) @@ -1453,12 +1525,17 @@ def _check_members(root: Path, members: list, no_members: bool, "here") return seen: dict[str, bool] = {} + gitlinked: list[Gitlinked] = [] for row in members: if not isinstance(row, dict): report.finding("member-row", f"a member is not a mapping: {row!r}") continue - _check_member(root, row, seen, report, gitlinks, unregistered) + asked = _check_member(root, row, seen, report, gitlinks) + if asked is not None: + gitlinked.append(asked) + for asked in gitlinked: + _check_member_checkout(root, asked, report, unregistered) if not members: report.note("no members yet: a family with none is empty, not " "wrong") diff --git a/tests/test_family_bootstrap_gate_order.py b/tests/test_family_bootstrap_gate_order.py new file mode 100644 index 00000000..9962f731 --- /dev/null +++ b/tests/test_family_bootstrap_gate_order.py @@ -0,0 +1,468 @@ +# SPDX-License-Identifier: Apache-2.0 +"""A holder's own bootstrap and gate after a clone, in every order and mode +(#648). + +THE ISSUE'S THREE LINES, each reproduced on #639's head before this change: + +- (a) `scripts/bootstrap.py` ended `family bootstrap ok`, exit 0, having + checked no member out. In a fresh recursive clone of a holder whose HEAD + lost `.gitmodules`, `git submodule update` exits 0, and it said "every + member and leg fetched", `NOT CHECKED OUT; skipped` for each member, then + ok. In a plain clone of that holder, or of #363's state (a member's + gitlink recorded only at `Members/

`), git dies `No url found for + submodule path` (exit 128), and it still ended ok. It fails now, saying + which members no credential would check out and why, and names + `validate-family.py`. A missing credential is still the skip it was. +- (b) In a plain clone of #363's state followed by `git submodule update + --init --recursive`, as CI and `make bootstrap` run it, the update dies at + `Members/

` before it checks out any member, so where that row is not + the first, the gate refused `member-uninitialized` for an earlier row, + naming that same update, and never reached the variant's refusal. Every + row's gitlink is asked before any row's checkout now. +- (c) Under `--no-members`, the mode CI falls back to there, + `member-gitlink-unnamed` for that gitlink printed no command. It prints + the index move members mode prints now. + +THE SAME FAMILY, OFFLINE RULES AND STATES as +`test_family_head_lost_gitmodules.py` (#611), whose helpers are imported. +Every remedy is typed as printed. The case-variant states are index records +and commits, never two directories whose names differ only in case, and +the holder's own are asked again with `core.ignorecase=true`. A clone's +checkout is git's: on a disk that folds case, `Members/

` IS the row's +directory, so the `mv` the text gives for a case-sensitive disk is typed +only where the disk holds the two apart, as the text says. +""" + +from __future__ import annotations + +import itertools +import os +import re +import shlex +import subprocess +from pathlib import Path + +import pytest + +from conftest import clone_just_pushed, git, run_script +from repo_shape import PYTHON +from test_family import NAME, family, validate # noqa: F401 +from test_family_add_refusals import holder, offline # noqa: F401 +from test_family_dry_run import isolated +from test_family_head_lost_gitmodules import ( + DISKS, assert_passes_in_every_mode, commit_all, head_lost, on_disk, + pinned, recorded_at, variant_refused, +) +from test_family_member_path import MEMBER, OTHER, raw +from test_validate_family_unnamed_mounts import ( + GITLINK_SAID, VARIANT, WRONG_CASE, commands, findings, stage_gitlink, +) + +#: The line `(a)` prints for the members it left NOT CHECKED OUT where no +#: credential would check them out, and the one the run then ends on. +NOT_FETCHED = (" NOT CHECKED OUT, and not for want of a credential: {left}. " + f"`{PYTHON} scripts/validate-family.py` names the cause and " + "the commit that cures it.") +NO_SECTION = "no section of .gitmodules gives that path" +NO_GITLINK = "no gitlink is recorded at that path" +EXITED_0 = "git exited 0 and checked nothing out there" + +#: Each test pushes a branch of its own into the module's family remote. Not +#: `tmp_path.name`: a passed test's directory is removed under +#: `tmp_path_retention_policy=failed`, and the next test is given its number. +BRANCHES = itertools.count() + +#: What members mode's variant refusal says before the move it prints. +WRONG_CASE_LEAD = "Remediation: the member is recorded in the wrong case. " + +#: `bootstrap.py --skip-members`'s stdout after its first line, in a clone +#: where every member was checked out, and where not every one was: the +#: second says nothing of (a)'s fetch on stdout, and ends on stderr. +FETCHED = ["", "(a) members and their legs", " credential source: none", + " every member and leg fetched", "", + "(b) each member's own `make bootstrap`", + " skipped (--skip-members)", "", "family bootstrap ok", ""] +NOT_ALL_FETCHED = ["", "(a) members and their legs", + " credential source: none", "", + "(b) each member's own `make bootstrap`", + " skipped (--skip-members)", "", ""] + + +def with_config(env: dict, key: str, value: str) -> dict: + """`env` with `key` set to `value`, one more `GIT_CONFIG_*` entry after + the ones it already holds, for every git the run starts.""" + index = int(env.get("GIT_CONFIG_COUNT", "0")) + return {**env, "GIT_CONFIG_COUNT": str(index + 1), + f"GIT_CONFIG_KEY_{index}": key, f"GIT_CONFIG_VALUE_{index}": value} + + +def fetching(env: dict) -> dict: + """`env` under which `git submodule update` may clone the fixture's + local members: `protocol.file.allow`, as `test_family.py` gives the + holder's own bootstrap.""" + return with_config(env, "protocol.file.allow", "always") + + +def variant(holder: Path, env: dict, project: str = OTHER) -> str: + """#363's state: `project`'s gitlink recorded only at `Members/

`, + committed, its row and section still `members/

`. OTHER is the + second row, so a plain clone's update dies before the first row's + checkout. Returns the commit.""" + return recorded_at(holder, env, project, f"Members/{project}") + + +def cloned(holder: Path, env: dict, family, tmp_path: Path, + *, recurse: bool) -> Path: + """HEAD pushed to a branch of the family's own remote that no other + test pushes (`BRANCHES`), then a fresh clone of it, recursive or plain, + which must exit 0.""" + remote = family["remotes"] / f"{NAME}.git" + branch = f"w648/{next(BRANCHES)}" + raw(holder, env, "push", "-q", str(remote), f"HEAD:refs/heads/{branch}") + clone = tmp_path / "clone" + made = clone_just_pushed(remote, clone, branch, recurse=recurse) + assert made.returncode == 0, made.stdout + made.stderr + return clone + + +def bootstrap(root: Path, env: dict, *extra: str): + return run_script(root / "scripts" / "bootstrap.py", *extra, cwd=root, + env=env) + + +def not_fetched(*left: tuple[str, str]) -> str: + """`NOT_FETCHED` naming each `(path, why)`.""" + return NOT_FETCHED.format( + left=", ".join(f"{path} ({why})" for path, why in left)) + + +def failed_on(*projects: str) -> str: + return "family bootstrap FAILED: " + ", ".join( + f"{project} (not checked out)" for project in projects) + + +def after_first_line(result) -> list[str]: + """Bootstrap's stdout after its first line, which names the holder and + its two members.""" + lines = result.stdout.split("\n") + assert lines[0].startswith(f"bootstrap: {NAME} ("), result.stdout + assert lines[0].endswith(" 2 member(s)"), result.stdout + return lines[1:] + + +def other_older(family) -> str: + """A commit of OTHER that is not the one the holder pins.""" + return git("rev-parse", "main~1", + cwd=family["remotes"] / f"{OTHER}.git").stdout.strip() + + +def type_line(root: Path, env: dict, line: str) -> None: + """One printed `git` line, typed as printed: split as a POSIX shell + splits it, run with no shell.""" + argv = shlex.split(line) + assert argv[0] == "git", line + raw(root, env, *argv[1:]) + + +def moved_out_where_apart(root: Path, remediation: str) -> None: + """Each backticked `mv` the text gives for a case-sensitive disk, typed + as printed where the disk holds its source apart from the row's own + directory, as the text says; on a disk that folds case it is the row's + directory, and the text says to leave it.""" + for move in re.findall(r"`(mv -- [^`]*)`", remediation): + argv = shlex.split(move) + source = root / argv[2] + own = root / "members" / argv[2].split("/", 1)[1] + if not source.exists() or (own.exists() + and os.path.samefile(source, own)): + continue + subprocess.run(argv, cwd=str(root), check=True) + assert not source.exists() + + +# --- (a) bootstrap ----------------------------------------------------------- + +#: `(set-up, flow, the members (a) names with why, the rows it fails on)`. +UNFETCHED = { + "head-lost-recursive": ( + head_lost, True, + ((f"members/{MEMBER}", NO_SECTION), (f"members/{OTHER}", NO_SECTION)), + (MEMBER, OTHER)), + "head-lost-plain": ( + head_lost, False, + ((f"members/{MEMBER}", NO_SECTION), (f"members/{OTHER}", NO_SECTION)), + (MEMBER, OTHER)), + "variant-recursive": ( + variant, True, ((f"members/{OTHER}", NO_GITLINK),), (OTHER,)), + "variant-plain": ( + variant, False, ((f"members/{OTHER}", NO_GITLINK),), (OTHER,)), +} + + +@pytest.mark.parametrize("set_up, recurse, left, projects", + UNFETCHED.values(), ids=UNFETCHED) +def test_bootstrap_fails_over_members_no_credential_would_check_out( + family, holder, offline, tmp_path, set_up, recurse, left, projects): + """THE ISSUE'S (a), both of rv611b's states, after a recursive clone and + after a plain one. Git exits 0 over a holder whose HEAD lost + `.gitmodules` where the clone was recursive, and 128 (`No url found for + submodule path`) where it was plain, and 128 over #363's state in a + plain clone: the run said ok each time. It names the members no + credential would check out, says why, names `validate-family.py`, and + fails; it never says every member was fetched.""" + env = isolated(offline, tmp_path) + set_up(holder, env) + clone = cloned(holder, env, family, tmp_path, + recurse=recurse) + + result = bootstrap(clone, fetching(env), "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert after_first_line(result) == NOT_ALL_FETCHED + errors = result.stderr.splitlines() + assert errors[-2:] == [not_fetched(*left), failed_on(*projects)], ( + result.stderr) + if not recurse: + assert errors[0].startswith(" FETCH INCOMPLETE (git exit 128) "), ( + result.stderr) + assert "No url found for submodule path" in result.stderr + + +def test_a_good_holder_bootstraps_as_it_did(family, holder, offline, + tmp_path): + """A GOOD HOLDER'S RUN IS KEPT: a fresh recursive clone checks every + member out, (a) says every member and leg was fetched, and the run ends + ok with nothing on stderr.""" + env = isolated(offline, tmp_path) + clone = cloned(holder, env, family, tmp_path, recurse=True) + + result = bootstrap(clone, fetching(env), "--skip-members") + + assert result.returncode == 0, result.stdout + result.stderr + assert after_first_line(result) == FETCHED + assert result.stderr == "" + + +def test_a_missing_credential_is_still_the_skip_it_was(family, holder, + offline, tmp_path): + """No misfire: a plain clone whose update git refuses for want of a + credential (no `protocol.file.allow` for the local members, as a + private member with no token is refused) records every gitlink at its + row's path, and `.gitmodules` gives each. A credential is the cure, so + the members are skipped, the run ends ok, and no line says otherwise.""" + env = isolated(offline, tmp_path) + clone = cloned(holder, env, family, tmp_path, + recurse=False) + + result = bootstrap(clone, env) + + assert result.returncode == 0, result.stdout + result.stderr + assert "FETCH INCOMPLETE" in result.stderr + assert "not for want of a credential" not in result.stderr + for project in (MEMBER, OTHER): + assert (f" [{project}] {project}: NOT CHECKED OUT; skipped. See the " + "fetch above.") in result.stdout.splitlines() + assert result.stdout.endswith("\nfamily bootstrap ok\n"), result.stdout + + +def test_an_update_that_exits_0_and_skips_a_member_fails(family, holder, + offline, tmp_path): + """Git exited 0, so it lacked nothing a credential gives, though the + holder's records give OTHER: here `submodule..update=none` set + for the run skips it. (a) does not say every member was fetched, and + the run fails on OTHER alone.""" + env = isolated(offline, tmp_path) + clone = cloned(holder, env, family, tmp_path, + recurse=False) + skip = with_config(fetching(env), f"submodule.members/{OTHER}.update", + "none") + + result = bootstrap(clone, skip, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert (clone / "members" / MEMBER / "project.yaml").is_file() + assert "every member and leg fetched" not in result.stdout + assert result.stderr.splitlines() == [ + not_fetched((f"members/{OTHER}", EXITED_0)), failed_on(OTHER)] + + +def test_a_row_written_twice_is_named_once(family, holder, offline, + tmp_path): + """OTHER's row written twice in a holder whose HEAD lost `.gitmodules`: + one path, so one name in (a) and one failure.""" + env = isolated(offline, tmp_path) + head_lost(holder, env) + rows = (holder / "family.yaml").read_bytes() + start = rows.index(f" - project: {OTHER}\n".encode()) + (holder / "family.yaml").write_bytes(rows + rows[start:]) + raw(holder, env, "add", "--", "family.yaml") + commit_all(holder, env, "OTHER's row written twice") + clone = cloned(holder, env, family, tmp_path, recurse=True) + + result = bootstrap(clone, fetching(env), "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert result.stderr.splitlines() == [ + not_fetched((f"members/{MEMBER}", NO_SECTION), + (f"members/{OTHER}", NO_SECTION)), + failed_on(MEMBER, OTHER)] + + +# --- (a) and (b) together: the plain clone of #363's state, typed through --- + +def test_a_plain_clone_of_a_case_variant_is_named_and_mended_as_printed( + family, holder, offline, tmp_path): + """The round trip a person makes. In a plain clone of #363's state, with + the variant at the SECOND row, `make bootstrap`'s script fails naming + OTHER and the validator; the validator, members checked, names the + variant and the move, where it refused `member-uninitialized` for the + first row before. Typed as printed in the clone, with the commit and + the `mv` where the disk holds the two apart, the same script checks + both members out and runs each one's own bootstrap, and the gate passes + in every mode.""" + env = fetching(isolated(offline, tmp_path)) + commit = variant(holder, env) + clone = cloned(holder, env, family, tmp_path, recurse=False) + + booted = bootstrap(clone, env) + + assert booted.returncode == 1, booted.stdout + booted.stderr + assert booted.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{OTHER}", NO_GITLINK)), failed_on(OTHER)] + gate = validate(clone) + assert gate.returncode == 2, gate.stdout + gate.stderr + lines = gate.stderr.splitlines() + assert lines == variant_refused(f"members/{OTHER}", commit, + [f"Members/{OTHER}"], + {f"Members/{OTHER}": commit}) + for line in lines[2:]: + type_line(clone, env, line) + commit_all(clone, env, "Move the gitlink as printed") + moved_out_where_apart(clone, lines[1]) + + again = bootstrap(clone, env) + + assert again.returncode == 0, again.stdout + again.stderr + assert " every member and leg fetched" in again.stdout.splitlines() + for project in (MEMBER, OTHER): + assert f" --- {project}: make bootstrap ---" in again.stdout + assert again.stdout.endswith("\nfamily bootstrap ok\n"), again.stdout + assert_passes_in_every_mode(clone, 2) + + +# --- (b) the gate asks every row's gitlink before any row's checkout -------- + +@pytest.mark.parametrize("project", [MEMBER, OTHER], ids=["first", "second"]) +def test_a_plain_clone_after_the_update_names_the_variant_in_any_row( + family, holder, offline, tmp_path, project): + """THE ISSUE'S (b), as CI and `make bootstrap` run it: a plain clone, + then `git submodule update --init --recursive`, which dies at + `Members/

` (exit 128) before it checks out either member. Members + checked and `--pins` alike, the gate refuses the variant with the move, + whichever row it is; for the second it refused `member-uninitialized` + for the first row, naming that same update. Under `--no-members`, CI's + fallback, the finding prints the same two lines (c); typed as printed in + the clone and committed, the update checks both out and the gate + passes.""" + env = fetching(isolated(offline, tmp_path)) + commit = variant(holder, env, project) + clone = cloned(holder, env, family, tmp_path, recurse=False) + update = raw(clone, env, "submodule", "update", "--init", "--recursive", + check=False) + assert update.returncode == 128, update.stderr + assert (f"No url found for submodule path 'Members/{project}'".encode() + in update.stderr) + expected = variant_refused(f"members/{project}", commit, + [f"Members/{project}"], + {f"Members/{project}": commit}) + + for extra in ((), ("--pins",)): + gate = validate(clone, *extra) + assert gate.returncode == 2, (extra, gate.stdout + gate.stderr) + assert gate.stderr.splitlines() == expected, extra + fallback = validate(clone, "--no-members") + assert fallback.returncode == 1, fallback.stdout + fallback.stderr + (finding,) = findings(fallback) + assert commands(finding) == [line.strip() for line in expected[2:]] + + for line in commands(finding): + type_line(clone, env, line) + commit_all(clone, env, "Move the gitlink as CI's gate printed") + moved_out_where_apart(clone, finding) + raw(clone, env, "submodule", "update", "--init", "--recursive") + assert_passes_in_every_mode(clone, 2) + + +# --- (c) `--no-members` prints the move members mode prints ----------------- + +def unnamed_moved(variant_path: str, at: str, pin: str, + variants: list[str]) -> str: + """`member-gitlink-unnamed` WHOLE for the gitlink at `variant_path` (at + `at`), OTHER recorded only at `variants`: what every unnamed gitlink + says, the row it is a variant of, and members mode's move to the pin, + word for word.""" + path = f"members/{OTHER}" + refused = variant_refused(path, pin, variants, + dict.fromkeys(variants, at)) + assert refused[1].startswith(WRONG_CASE_LEAD + "Move its gitlink "), ( + "premise: members mode's remedy text") + return "\n".join([ + GITLINK_SAID.format(path=variant_path, commit=at) + + VARIANT.format(row_path=path, project=OTHER) + + " The member is recorded in the wrong case. " + + refused[1][len(WRONG_CASE_LEAD):], *refused[2:]]) + + +@DISKS +@pytest.mark.parametrize("spellings", [ + ("Members/{p}",), ("MEMBERS/{p}", "Members/{p}")], ids=["one", "two"]) +def test_no_members_prints_the_move_members_mode_prints( + family, holder, offline, tmp_path, ignorecase, spellings): + """THE ISSUE'S (c). OTHER's only gitlink at a case variant (two, in the + second case), at a commit that is not its pin. Under `--no-members` each + variant's `member-gitlink-unnamed` printed no command; it prints the + move members mode prints now, to the commit the row PINS, every variant + taken out on one line. Typed as printed and committed, on either disk, + the holder passes in every mode.""" + env = on_disk(isolated(offline, tmp_path), ignorecase) + variants = sorted(spelling.format(p=OTHER) for spelling in spellings) + at = other_older(family) + pin = pinned(holder, OTHER) + assert pin != at, "premise: the variant is not at the pin" + raw(holder, env, "rm", "-q", "--cached", "--", f"members/{OTHER}") + for spelling in variants: + stage_gitlink(holder, spelling, at) + commit_all(holder, env, f"{OTHER} recorded in another case") + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert findings(result) == [unnamed_moved(path, at, pin, variants) + for path in variants] + for line in commands(findings(result)[0]): + type_line(holder, env, line) + commit_all(holder, env, "Move the gitlink as printed") + assert_passes_in_every_mode(holder, 2) + + +def test_a_row_whose_pin_is_no_commit_still_prints_no_command( + holder, offline, tmp_path): + """The move records the gitlink at the commit the row pins, so a row + whose `pin.commit` is no 40-hex commit (members checked, the refusal + `member-pin-commit`) gets the text it had, and no command.""" + env = isolated(offline, tmp_path) + commit = variant(holder, env) + rows = (holder / "family.yaml").read_bytes() + line = f' commit: "{commit}"\n'.encode() + assert rows.count(line) == 1, "fixture drift: OTHER's pin line" + (holder / "family.yaml").write_bytes( + rows.replace(line, f' commit: "{commit[:12]}"\n'.encode())) + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stdout + result.stderr + (finding,) = findings(result) + assert finding == GITLINK_SAID.format( + path=f"Members/{OTHER}", commit=commit) + WRONG_CASE.format( + row_path=f"members/{OTHER}", project=OTHER, what="gitlink") + assert "git rm" not in result.stderr, result.stderr diff --git a/tests/test_validate_family_unnamed_mounts.py b/tests/test_validate_family_unnamed_mounts.py index 3144f710..a9ab815c 100644 --- a/tests/test_validate_family_unnamed_mounts.py +++ b/tests/test_validate_family_unnamed_mounts.py @@ -28,7 +28,9 @@ the wrong case: the same command, run in either, takes the member's own record out of the index, after which `--no-members`, the run a private family's CI falls back to, says `family ok` over a member that is no longer -there. Those two print no command. +there. Those two print no such command: the first none, and the second, +since #648, the index move members mode prints, which records the member +at its row's path as it takes the variant out. When a hand mount's gitlink says to `git rm` it, its section finding says that the `git rm` takes the section out too and prints no `--unset-all` line: run @@ -512,14 +514,13 @@ def test_a_row_hand_edited_into_a_case_variant_of_its_mount_is_the_edit( @MODES -def test_a_member_recorded_only_in_another_case_says_so_and_prints_no_command( +def test_a_member_recorded_only_in_another_case_says_so_and_prints_the_move( holder, extra): """STATE (ii), REVIEW SHOULD-1: IRSS's row is canonical, and the only gitlink the holder records for it is at `members/irss`. `git rm --cached` of that took the member's only record out, and `--no-members` then said `family ok` over a holder that records one gitlink fewer than it has rows. - It says the member is recorded in the wrong case instead, and sends the - operator to git's history, as every other unnamed gitlink does. + It says the member is recorded in the wrong case instead. Members checked, the row's own gitlink is read at its exact path, finds none and is the refusal `member-gitlink-absent`, which prints no finding, @@ -528,26 +529,31 @@ def test_a_member_recorded_only_in_another_case_says_so_and_prints_no_command( there. What no run may do is print a command that takes the gitlink out ALONE: since #611 the refusal names the variant and prints the move that takes it out of the index and records it at the row's path together - (`tests/test_family_head_lost_gitmodules.py` types it).""" + (`tests/test_family_head_lost_gitmodules.py` types it), and since #648 + the finding prints the same two lines + (`tests/test_family_bootstrap_gate_order.py` types them).""" named = record_in_another_case(holder, "IRSS", "members/irss") result = validate(holder, *extra) assert "--unset-all" not in result.stderr + result.stdout + move = [ + " git rm -r --cached --ignore-unmatch -- members/irss", + f" git update-index --add --cacheinfo 160000,{named},members/IRSS"] if result.returncode == 2: assert extra != ("--no-members",), result.stderr assert result.stderr.startswith( "REFUSED member-gitlink-absent: no gitlink (mode 160000) " "recorded at 'members/IRSS'."), result.stderr - assert result.stderr.splitlines()[-2:] == [ - " git rm -r --cached --ignore-unmatch -- members/irss", - " git update-index --add --cacheinfo " - f"160000,{named},members/IRSS"], result.stderr + assert result.stderr.splitlines()[-2:] == move, result.stderr return - assert "git rm" not in result.stderr + result.stdout assert result.returncode == 1, result.stderr + result.stdout (gitlink,) = findings(result) - assert gitlink == GITLINK_SAID.format( - path="members/irss", commit=named) + WRONG_CASE.format( - row_path="members/IRSS", project="IRSS", what="gitlink") + said = GITLINK_SAID.format(path="members/irss", commit=named) + ( + VARIANT.format(row_path="members/IRSS", project="IRSS")) + assert gitlink.startswith( + said + " The member is recorded in the wrong case. Move its gitlink " + "to the row's path, at the commit the row pins, in the index " + "alone, "), gitlink + assert gitlink.splitlines()[1:] == move, gitlink @MODES From 73fa3ff4ee605a6634d17efb3342a5fa547d3416 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 17:33:37 +0000 Subject: [PATCH 05/15] README.md, the handbook, AGENTS.md and the family-root docs: `init`'s `--into` sentence names the `family-landing-in-standard` refusal, `make siblings` is described as it stops before cloning, and `members/` has its one-spelling rule, so three sentences that fell behind #488, #578 and #574 are true of main (#695) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md and docs/handbook.html § Families: - The `--into` sentence says the default (the directory you are standing in) is refused inside a checkout of this standard, and names the code. - The `make siblings` paragraph says what it does since #578: the holder's parent folder must be named EXACTLY after its repository, case included, on every disk; where it is not, and no link of that name beside it leads to it, it WARNS, prints the exact `mv` and clones nothing until it is run, so the next run clones each member once. Lane 3's two #460 sentences (L20, L21) ride in it: a holder at `~/projects/InkRouter` is moved to `~/InkRouter/InkRouter`, beside that folder, not inside it. The one-spelling rule, one sentence in the same words in AGENTS.md (family item 2), README.md § Families, the handbook, the holder's README and the holder's pinned AGENTS-shape.md: a holder keeps `members/` in ONE spelling on every disk, and `add` refuses `family-members-dir-variant` (`member-path-taken` for a variant of the member's own path) until a human renames what sits under another spelling. Neither cap is raised. README.md stays at its cap of 1459 lines and AGENTS.md at 492, by cutting wording that is stated elsewhere: README § Families lost the "same rule projects have" aside, the tool's rationale after "Nothing ever moves a checkout" (the handbook, the holder's README and the tool's own output keep it), "not merely a repository at the right commit", the private-legs reason in the `make bootstrap` paragraph and the policy history in the naming paragraph; AGENTS.md lost the converse in the family intro, the `bump --trailer` sentence (stated in "Advancing a leg" and in AGENTS-shape.md) and "on your own initiative" in "Updating" step 2. The handbook marker for README § Families is recomputed. templates/family-root/AGENTS-shape.md is pinned: every holder re-pins it (`upstream-changed`). tests/test_family_docs_one_spelling.py holds the sentences to the five places and the codes to family.py. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 12 +-- README.md | 54 +++++------ docs/handbook.html | 23 +++-- templates/family-root/AGENTS-shape.md | 5 +- templates/family-root/README.md | 5 +- tests/test_family_docs_one_spelling.py | 126 +++++++++++++++++++++++++ 6 files changed, 183 insertions(+), 42 deletions(-) create mode 100644 tests/test_family_docs_one_spelling.py diff --git a/AGENTS.md b/AGENTS.md index 07f60556..48bb220d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -290,8 +290,7 @@ git -C push -u origin shape/update- # then open a pull request `upstream-removed`, `unmapped` or `copy-missing`; `apply` refuses the last three. A file with no row is not a shape copy and is none of your business. 2. **Show the human the verdicts and let them say yes.** Re-pinning records - which openRepoShape this project is a copy of. Do not pass `--yes` on your - own initiative. + which openRepoShape this project is a copy of. Never pass `--yes` yourself. 3. **Never pass `--accept-local` to make a refusal go away.** It re-pins a locally edited file FROM THE PROJECT'S OWN BYTES, which makes the drift `validate-pins.py` reports today invisible. Whether the project keeps its @@ -366,8 +365,7 @@ git -C push -u origin # then open a pull request A FAMILY is a holder that pins other projects' ASSEMBLY ROOTS as submodules under `members/`. It is not a project — no legs — and membership confers -nothing. Use one when the parts DEPLOY SEPARATELY; one project is what parts -that ship together already are. +nothing. Use one when the parts DEPLOY SEPARATELY. ```sh python3 scripts/family.py init --org --family --into .. [--reuse-empty-repo] @@ -391,13 +389,15 @@ python3 scripts/family.py siblings --family-root deliberate and reads like a mistake until somebody explains it. 2. **A family pins ASSEMBLY ROOTS, never legs.** `--member /`, never `-spec`: a leg has no `project.yaml`, and is rightly refused. + A holder keeps `members/` in ONE spelling on every disk: `add` refuses + `family-members-dir-variant` (`member-path-taken` for a variant of the + member's own path) until a human renames what sits under another spelling. 3. **Each of `add`, `bump` and `remove` writes ONE commit** moving the gitlink and the pin together, on a branch you make first (see above): the tracking branch is refused (`family-on-tracking-branch`). Push the branch as its `NEXT` line says and land it as a pull request; never suggest a push to the default branch. Never hand-edit `family.yaml`'s `members:` - block — the tool rewrites it wholesale. `bump` takes the same repeatable - `--trailer ": "` as `update-shape.py apply`. + block — the tool rewrites it wholesale. 4. **The member must be scaffolded or adopted FIRST.** A family cannot make a project out of a repository; it can only pin one that already is. 5. **`make siblings` in the holder — or `family.py siblings --family-root diff --git a/README.md b/README.md index 2dc61e44..80c14dde 100644 --- a/README.md +++ b/README.md @@ -870,10 +870,9 @@ is reviewed identically to one in this family. | what the holder owns | the legs | nothing but pins and utilities | **Use a family when the parts deploy separately.** Eight services that are -eight APIs are eight projects: folding them into one assembly root would give -them one gate, one release and one pin, which is the opposite of what "deploy -separately" means. **Use one project when the parts ship together** — that is -what the spec and code legs already are. +eight APIs are eight projects: one assembly root would give them one gate, one +release and one pin, the opposite of "deploy separately". **Use one project +when the parts ship together**: that is what the spec and code legs are. ### The workstation layout: a plain folder, and two copies of every member @@ -891,13 +890,13 @@ InkRouter/ a PLAIN FOLDER, not a git repository ``` `scripts/family.py init` lands the holder there: `--into

` is the PARENT -directory (the directory you are standing in, by default), the folder +directory (the directory you are standing in, by default; refused inside a +checkout of this standard, `family-landing-in-standard`), the folder `/` is created in it, and the holder is cloned INSIDE that at -`//` — the same "lands where you were standing" rule -projects have. Standing in the family folder already, it does not nest a -second one. `--work-dir` remains the override that lands `/` -and makes no folder. The doubled name is kept on purpose: the holder keeps its -own repository name and `family.yaml` is what tells the two apart. +`//`. Standing in the family folder already, it does not +nest a second one. `--work-dir` is the override that lands +`/` and makes no folder. The doubled name is on purpose: the +holder keeps its own repository name, and `family.yaml` tells the two apart. `make siblings` in the holder — or `python3 scripts/family.py siblings --family-root `, the same file through the other entry point — clones @@ -910,12 +909,13 @@ inside the holder is pinned and DETACHED, and is what `bootstrap` and **fetched and nothing else** — no checkout, no reset, no pull — and one that is a different repository is reported and skipped. It is idempotent. -**Nothing ever moves a checkout.** When the holder's parent folder is not -named after the family, `make siblings` WARNS and prints the exact `mv` for a -human to run. A tool that relocated the directory would do it out from under -the shell, the editor and every agent lane standing in it, and would break -linked worktrees, whose `.git` files carry absolute paths — an act by -surprise, which is the one thing every refusal here exists to avoid. +**Nothing ever moves a checkout**: a linked worktree's `.git` file carries an +absolute path. The holder's parent folder must be named EXACTLY after its +repository, case included, on every disk. When it is not, and no link of that +name beside it leads to it, `make siblings` WARNS, prints the exact `mv` for a +human to run and clones nothing until it is run, so the next run clones each +member once. For a holder straight in a folder (`~/projects/InkRouter`) the +move makes `~/InkRouter/InkRouter`, beside that folder, not inside it. ### The InkRouter example @@ -944,13 +944,14 @@ branch made first (the tracking branch is refused) and print its push: the gitlink and `members[].pin` move together, the lockstep rule an assembly root applies to its legs, which `scripts/validate-family.py` enforces. Each row records the member's `project.yaml` `id`, so the validator can check -`members/` holds the project the row claims, not merely a repository -at the right commit. **A family pins assembly roots, never legs**: -`-spec` is refused; a leg has no `project.yaml`. - -`make bootstrap` in the holder fetches every member and its legs — resolving -the GitHub App or `SHAPE_LEGS_TOKEN` credential the workflow does, as a member -is a private assembly root mounting two private legs — then runs each member's +`members/` holds the project the row claims. **A family pins assembly +roots, never legs**: `-spec` is refused; a leg has no `project.yaml`. +A holder keeps `members/` in ONE spelling on every disk: `add` refuses +`family-members-dir-variant` (`member-path-taken` for a variant of the member's +own path) until a human renames what sits under another spelling. + +`make bootstrap` in the holder fetches every member and its legs, with the +workflow's GitHub App or `SHAPE_LEGS_TOKEN` credential, then runs each member's own `make bootstrap`; `make validate` runs the family validator, then each member's. A member it cannot fetch is reported and skipped, never a failure. @@ -959,12 +960,11 @@ The holder carries the same COPY pin an assembly root does, so instead of `project.yaml` and must leave `validate-family.py` green. The name is the `family` form in `contracts/repository-naming.yaml`: one -CamelCase token, exactly an assembly root's rule, with **precedence below** -it, and DECLARED-ONLY. `InkRouter` is spelled like an assembly root and +CamelCase token, exactly an assembly root's rule, with **precedence below** it, +and DECLARED-ONLY. `InkRouter` is spelled like an assembly root and `family.yaml` is the only thing that tells them apart, so the classifier reports the holder form only when it is asked for (`--role family`). That is -what let a fifth family be added to a policy two live projects already carry a -copy of without changing any existing name's answer. +what let a fifth family be added without changing any existing name's answer. ## The double pin, and the lockstep invariant diff --git a/docs/handbook.html b/docs/handbook.html index 02c5188e..61cbeffa 100644 --- a/docs/handbook.html +++ b/docs/handbook.html @@ -1282,7 +1282,7 @@

Adopting an existing repository

- +

Reference

Families: a holder for projects that ship separately

@@ -1319,7 +1319,8 @@

The workstation layout: a plain folder, and two copies of every member

IRRS-worktrees/… whatever the member's own work needs session-handoff-*.md the estate's own files live in the folder

scripts/family.py init lands the holder there: --into <dir> - is the PARENT directory (the directory you are standing in, by default), the folder + is the PARENT directory (the directory you are standing in, by default; refused inside a + checkout of this standard, family-landing-in-standard), the folder <Family>/ is created in it, and the holder is cloned INSIDE that at <into>/<Family>/<Family> — the same “lands where you were standing” rule projects have. Standing in the family folder already, it does not @@ -1338,11 +1339,15 @@

The workstation layout: a plain folder, and two copies of every member

follows it. An existing clone is fetched and nothing else — no checkout, no reset, no pull — and one that is a different repository is reported and skipped. It is idempotent.

-

Refuses Nothing ever moves a checkout: when - the holder’s parent folder is not named after the family, make siblings - WARNS and prints the exact mv for a human to run. A tool that relocated the - directory would do it out from under the shell, the editor and every agent lane standing in - it, and would break linked worktrees, whose .git files carry absolute paths +

Refuses Nothing ever moves a checkout: the + holder’s parent folder must be named EXACTLY after its repository, case included, on + every disk. When it is not, and no link of that name beside it leads to it, + make siblings WARNS, prints the exact mv for a human to run and + clones nothing until it is run, so the next run clones each member once. For a holder + straight in a folder (~/projects/InkRouter) the move makes + ~/InkRouter/InkRouter, beside that folder, not inside it. A tool that relocated + the directory would do it out from under the shell, the editor and every agent lane standing + in it, and would break linked worktrees, whose .git files carry absolute paths — an act by surprise, which is the one thing every refusal here exists to avoid.

Ruled by Brett Heap on 2026-09-09 (issue #76): the plain folder, and the holder inside it at <Family>/<Family>.

@@ -1371,6 +1376,10 @@

The InkRouter example

Refuses A family pins assembly roots, never legs: adding <Project>-spec is refused, because a leg has no project.yaml and belongs to its own root.

+

Refuses A holder keeps members/ in ONE spelling on + every disk: add refuses family-members-dir-variant + (member-path-taken for a variant of the member’s own path) until a human + renames what sits under another spelling.

make bootstrap in the holder fetches every member and its legs — resolving the same GitHub App or SHAPE_LEGS_TOKEN credential the workflow does — and then runs each member’s own make bootstrap. diff --git a/templates/family-root/AGENTS-shape.md b/templates/family-root/AGENTS-shape.md index a9e740e4..fbc984d9 100644 --- a/templates/family-root/AGENTS-shape.md +++ b/templates/family-root/AGENTS-shape.md @@ -40,7 +40,10 @@ commit as a pull request: each refuses this holder's tracking branch (`family-on-tracking-branch`) and names the `git switch -c` that makes a branch, as the block's first line does, and ends by printing that branch's `git push`. The member must already be scaffolded or adopted, because a family -cannot make a project out of a repository. +cannot make a project out of a repository. A holder keeps `members/` in ONE +spelling on every disk: `add` refuses `family-members-dir-variant` +(`member-path-taken` for a variant of the member's own path) until a human +renames what sits under another spelling. `bump` takes a repeatable `--trailer ": "`, appended to the commit it writes in the order given. It composes that message itself, so whatever diff --git a/templates/family-root/README.md b/templates/family-root/README.md index e0150475..9bfa6aa9 100644 --- a/templates/family-root/README.md +++ b/templates/family-root/README.md @@ -127,7 +127,10 @@ python3 scripts/family.py remove --family-root --member Each writes ONE commit, with explicit pathspecs, moving the gitlink and the pin together, on the branch the holder stands on: it refuses the tracking branch, so make a branch first, as the block's first line does. Push it as its -`NEXT` line says and land it as a pull request. +`NEXT` line says and land it as a pull request. A holder keeps `members/` in +ONE spelling on every disk: `add` refuses `family-members-dir-variant` +(`member-path-taken` for a variant of the member's own path) until a human +renames what sits under another spelling. ## Private members in CI diff --git a/tests/test_family_docs_one_spelling.py b/tests/test_family_docs_one_spelling.py new file mode 100644 index 00000000..cd35e49b --- /dev/null +++ b/tests/test_family_docs_one_spelling.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The family docs say what `init`, `make siblings` and `members/` do (#695). + +Three sentences had fallen behind the tools, and one rule had no sentence: + +- `init` lands where you stand, and refuses that default inside a checkout of + this standard (`family-landing-in-standard`, #488). README and the + handbook gave the default and not the refusal. +- `make siblings` stops before it clones anything, and a family folder + reached through a link of the family's name passes (#578). README and the + handbook still said it WARNS and prints the `mv`. They now also say the + folder is named EXACTLY after the repository on every disk, and where the + printed move puts the family folder for a holder straight in a folder. +- A holder keeps `members/` in ONE spelling on every disk, and `add` + refuses `family-members-dir-variant` (`member-path-taken` for a variant of + the member's own path) until a human renames the entries (#574). No doc + said so. + +THE SAME WORDS IN FIVE PLACES. The one-spelling sentence is the same text in +AGENTS.md, README.md, the handbook, the holder's README and the holder's +pinned `AGENTS-shape.md`, so a reader meets one rule however they came. The +handbook spells it in HTML, so it is read as the text a browser shows, with +`` as backticks. + +The printed move itself is pinned by `test_family_siblings_placement.py` +(`~/projects/InkRouter` becomes `~/InkRouter/InkRouter`): this file holds the +sentences that describe it. +""" + +from __future__ import annotations + +import ast +import html +import re + +import pytest + +from conftest import REPO + +ONE_SPELLING = ( + "A holder keeps `members/` in ONE spelling on every disk: `add` refuses " + "`family-members-dir-variant` (`member-path-taken` for a variant of the " + "member's own path) until a human renames what sits under another " + "spelling.") + +INTO_REFUSED = ( + "(the directory you are standing in, by default; refused inside a " + "checkout of this standard, `family-landing-in-standard`)") + +STOPS_BEFORE_CLONING = ( + "When it is not, and no link of that name beside it leads to it, " + "`make siblings` WARNS, prints the exact `mv` for a human to run and " + "clones nothing until it is run, so the next run clones each member " + "once.") + +NAMED_EXACTLY = ( + "named EXACTLY after its repository, case included, on every disk") + +STRAIGHT_IN_A_FOLDER = ( + "For a holder straight in a folder (`~/projects/InkRouter`) the move " + "makes `~/InkRouter/InkRouter`, beside that folder, not inside it.") + +#: What README and the handbook said before #578, which `make siblings` no +#: longer does. +BEFORE_578 = "`make siblings` WARNS and prints the exact `mv`" + +HANDBOOK = "docs/handbook.html" +README = "README.md" +THE_FIVE = [ + "AGENTS.md", README, HANDBOOK, "templates/family-root/README.md", + "templates/family-root/AGENTS-shape.md"] +README_AND_HANDBOOK = [README, HANDBOOK] + + +def shown(path: str) -> str: + """The text a reader sees in `path`, on one line. + + A page is read as a browser shows it: `` is a backtick, any other + tag is nothing, entities are their characters and a curly apostrophe is + a straight one. Every other file is its own text. Runs of white space, + and so every wrap, are one space.""" + text = (REPO / path).read_text(encoding="utf-8") + if path == HANDBOOK: + text = re.sub(r"", "`", text) + text = html.unescape(re.sub(r"<[^>]+>", "", text)) + text = text.replace("’", "'") + return " ".join(text.split()) + + +@pytest.mark.parametrize("path", THE_FIVE) +def test_each_doc_names_the_one_spelling_rule_in_the_same_words(path): + assert ONE_SPELLING in shown(path) + + +@pytest.mark.parametrize("code", ["family-members-dir-variant", + "member-path-taken"]) +def test_the_codes_the_rule_names_are_codes_family_py_prints(code): + source = (REPO / "scripts" / "family.py").read_text(encoding="utf-8") + held = {node.value for node in ast.walk(ast.parse(source)) + if isinstance(node, ast.Constant) and isinstance(node.value, str)} + assert code in held + + +@pytest.mark.parametrize("path", README_AND_HANDBOOK) +def test_the_into_sentence_gives_the_refusal_with_the_default(path): + assert INTO_REFUSED in shown(path) + + +@pytest.mark.parametrize("path", README_AND_HANDBOOK) +def test_make_siblings_is_described_as_it_stops_before_cloning(path): + assert STOPS_BEFORE_CLONING in shown(path) + + +@pytest.mark.parametrize("path", README_AND_HANDBOOK) +def test_the_stale_warns_and_prints_sentence_is_gone(path): + assert BEFORE_578 not in shown(path) + + +@pytest.mark.parametrize("path", README_AND_HANDBOOK) +def test_the_folder_is_named_exactly_after_the_repository(path): + assert NAMED_EXACTLY in shown(path) + + +@pytest.mark.parametrize("path", README_AND_HANDBOOK) +def test_the_move_for_a_holder_straight_in_a_folder_is_one_level_up(path): + assert STRAIGHT_IN_A_FOLDER in shown(path) From d91fd15ef9065fda3a35f705f4e1618ac533cb16 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 17:14:35 +0000 Subject: [PATCH 06/15] validate-family.py and validate-pins.py: a gitlink removal staged and not committed says to commit it, `member-gitlink-absent` puts the gitlink and its section back by hand, the variant `mv` is guarded by `-ef` and a `.git` test, and a shape-pin row with no `path:` is `shape-pin-row`, so every remedy types as printed in the state it names and `None` is no path (#681) The holder's validate-family.py (pinned in every holder) and the assembly root's validate-pins.py (pinned in every assembly root); words and one check, no family.py change (Brett Heap, 2026-10-16T14:11:42Z, "take all recommendations for #670 and #681"). - L20. `_staged_removals` names the gitlinks HEAD records where the index holds no entry at the path, at any stage. `member-gitlink-unnamed` for one says the removal is staged and to commit it, and prints no `git rm`, which failed typed again (`pathspec ... did not match any files`, exit 128): a hand mount keeps its branch and `add` lines, a second path's says only to commit. A section `git rm --cached` leaves keeps its own `--unset-all` lines, since no `git rm` above takes it out. An unmerged path is a conflict, not a removal, and keeps its `git rm`. - L2-38. `member-gitlink-absent` with no case variant named `family.py add`, which refuses the row that is there (`member-already-present`); `remove` dies at `git submodule deinit`. It now prints the `update-index --cacheinfo` line that puts the gitlink back at the commit the row pins, and, where HEAD's `.gitmodules` gives no section the path, the section lines `member-gitmodules-absent` prints (`_section_lines`, shared); or the row taken out by hand. - N-1 and N-2. The variant move's `mv` is now `[ -ef ] || [ -e /.git ] || mv -- ...`, typed in a POSIX shell: it moves nothing where the two names are one directory (a case-folding disk, WSL's /mnt/c included) or where the directory is a checkout, whose submodule `.git` file a move leaves pointing nowhere. The text says its commits are kept in `.git/modules/` and to delete a checkout by hand once nothing it holds is needed. It no longer states the condition by system. - L2-3. A `files:` row whose `path:` is absent, empty or no string is `shape-pin-row` in both validators, where it was `shape-copy-missing` for `None`. Tests: tests/test_validate_family_remedies_typed.py (L20, L2-38, N-1, N-2, each typed as printed; the case-folding disk emulated with a `Members` symlink) and tests/test_shape_pin_row_path.py (L2-3, both validators); the pinned texts and typed helpers in test_family_head_lost_gitmodules.py, test_family_bootstrap_gate_order.py, test_validate_family_branch_step.py, test_family_switch_branch_exists.py and test_family.py follow the new words. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- .../assembly-root/scripts/validate-pins.py | 12 +- .../family-root/scripts/validate-family.py | 297 ++++++++--- tests/test_family.py | 2 +- tests/test_family_bootstrap_gate_order.py | 33 +- tests/test_family_head_lost_gitmodules.py | 62 ++- tests/test_family_switch_branch_exists.py | 49 +- tests/test_shape_pin_row_path.py | 67 +++ tests/test_validate_family_branch_step.py | 44 +- tests/test_validate_family_remedies_typed.py | 489 ++++++++++++++++++ 9 files changed, 897 insertions(+), 158 deletions(-) create mode 100644 tests/test_shape_pin_row_path.py create mode 100644 tests/test_validate_family_remedies_typed.py diff --git a/templates/assembly-root/scripts/validate-pins.py b/templates/assembly-root/scripts/validate-pins.py index f4000465..9ba55c04 100755 --- a/templates/assembly-root/scripts/validate-pins.py +++ b/templates/assembly-root/scripts/validate-pins.py @@ -846,12 +846,20 @@ def _check_shape_copy(root: Path, row, rel: str, report: Report) -> None: Split from `_check_shape_pin` for #132 so the loop over the pinned files is a single call per row: is it a mapping, is the file still there, and - does its sha256 still match what the pin recorded. + does its sha256 still match what the pin recorded. A row whose `path:` + is absent, empty or no string is `shape-pin-row`, as `update-shape.py` + refuses it `update-pin-row`: its `path:` was read as a path, so a row + with none was `None is pinned but absent` (#681). """ if not isinstance(row, dict): report.finding("shape-pin-row", f"{rel}: a files row is not a mapping") return - target = root / str(row.get("path")) + path = row.get("path") + if not isinstance(path, str) or not path: + report.finding("shape-pin-row", f"{rel}: a files row's `path:` is " + f"{path!r}, not the path of a file") + return + target = root / path if not target.is_file(): report.finding("shape-copy-missing", f"{rel}: {row.get('path')} is pinned but absent") diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index 67568c71..b3623078 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -66,7 +66,9 @@ it names no command that would un-register the member, and for a member recorded only in the wrong case it names the index move that records it at its row's path, as the row's own refusal does members checked - (#648). + (#648). A gitlink whose removal is staged and not committed, which HEAD + still records, says to commit it, and prints no `git rm`, which would + fail (#681). - that a section of HEAD's `.gitmodules` gives the path of every row's gitlink, the `member-gitmodules-absent` finding (#611, #407's decision 3). A recursive clone reads HEAD's file, so a gitlink no section of it @@ -425,16 +427,14 @@ def _pin_commit(pin: dict, project: str, report: Report) -> str: def _check_member_gitlink(root: Path, path: str, commit: str, project: str, - gitlinks: dict[str, str], report: Report) -> None: + repository, gitlinks: dict[str, str], + report: Report) -> None: """The submodule gitlink this repository records at `path`, against the pin — THE LOCKSTEP RULE. Split from `_check_member` for #137. Raises `Refusal` when there is no gitlink at all: a row naming a member this - repository does not record as a submodule. Its remedy is two lines, the - `git switch -c` that `add` needs first (#539) and the `add`, from a - checkout of openRepoShape and naming this holder by its path, as the - holder's other remedies do: `--family-root .` named the checkout the - line runs from. Where `gitlinks` (`_member_gitlinks`) holds one at a - case variant of `path`, the refusal names that instead (#611).""" + repository does not record as a submodule (`_gitlink_put_back` says + what to do). Where `gitlinks` (`_member_gitlinks`) holds one at a case + variant of `path`, the refusal names that instead (#611).""" gitlink = recorded_gitlink(root, path) if gitlink is None: variants = sorted(other for other in gitlinks if other != path @@ -447,14 +447,7 @@ def _check_member_gitlink(root: Path, path: str, commit: str, project: str, f"no gitlink (mode 160000) recorded at {path!r}. {project} is " f"named in {MANIFEST} but this repository does not record it as a " "submodule.", - "Remediation: from a checkout of openRepoShape, the second " - "command below mounts and pins it in one commit, on a branch " - "other than the holder's tracking branch, which it refuses, so " - "the first makes one; a row with no submodule is a claim about a " - f"tree that is not here. {BRANCH_LEFT}\n" - f" {_switch_line(root, 'add', project)}\n" - f" {PYTHON} scripts/family.py add --family-root " - f"{shlex.quote(str(root))} --member /{shlex.quote(project)}") + _gitlink_put_back(root, path, commit, repository)) if gitlink != commit: report.finding( "member-gitlink-mismatch", @@ -464,6 +457,46 @@ def _check_member_gitlink(root: Path, path: str, commit: str, project: str, report.note(f"{path}: gitlink == pin {commit[:12]}") +def _gitlink_put_back(root: Path, path: str, commit: str, + repository) -> str: + """What `member-gitlink-absent` says where no case variant of `path` + holds the gitlink either: a row whose gitlink was taken out by hand, as + `git rm` takes it out, with its `.gitmodules` section (#681). + + IN WORDS, AS NO TOOL TAKES THIS STATE. It named `family.py add`, which + refuses the row that is there (`member-already-present`), and `family.py + remove` dies at its `git submodule deinit` (`pathspec ... did not + match`), which finds no gitlink, having removed nothing. So a member + that is to stay has its gitlink put back in the index alone, at the + commit the row pins, as `_wrong_case_move` records one, and, where no + section of HEAD's `.gitmodules` gives its path, its section written back + as `member-gitmodules-absent` writes one (`_section_lines`); and a + member that is to go has its row taken out by hand. Either is + committed, and a member put back is checked out as + `member-uninitialized` says. + """ + head = _head_gitmodules(root) + sections = head[1] if head is not None else {} + given = {keys["path"] for keys in sections.values() if "path" in keys} + lines = ("" if path in given + else _section_lines(path, repository, sections)) + written = (", and its .gitmodules section back by hand, in a POSIX " + "shell, as family.py add writes one, with the lines after it, " + "then stage .gitmodules" if lines else "") + return ( + "Remediation: family.py add refuses the row, which is there " + "(member-already-present), and family.py remove cannot take the " + "member out, since its git submodule deinit finds no gitlink. If " + "the member is to stay, put its gitlink back at the commit the row " + "pins, in the index alone, which touches no file, with the first " + f"line below{written}. Commit that, and `git submodule update " + "--init --recursive`, or `make bootstrap`, then checks the member " + f"out at {path}. If it is to go, take its row out of {MANIFEST} by " + "hand and commit that.\n" + " git update-index --add --cacheinfo " + f"{shlex.quote(f'160000,{commit},{path}')}{lines}") + + def _gitlink_in_another_case(path: str, commit: str, project: str, variants: list[str], gitlinks: dict[str, str]) -> Refusal: @@ -491,12 +524,15 @@ def _gitlink_in_another_case(path: str, commit: str, project: str, WHAT IS LEFT ON DISK. Where nothing is checked out at the row's path, as in a fresh clone, `git submodule update --init --recursive` checks - the member out there once the move is committed. On a case-sensitive - disk a directory at a variant (the empty one a fresh clone leaves, or a - checkout) stays behind, untracked, and `family.py add` refuses it as + the member out there once the move is committed. Where the disk holds + a variant apart from the row's path, a directory there (the empty one a + fresh clone leaves, a plain one, or a checkout) stays behind, + untracked, and `family.py add` refuses it as `family-members-dir-variant` (#574), so the text says to move it out of - the holder, by an `mv` with nothing to fill in. On macOS and Windows - that directory is the row's own, and the text says to leave it there. + the holder, by an `mv` with nothing to fill in, which a shell test + skips where the disk folds the two names into one directory, the + row's own, and where the directory is a checkout (#681). A checkout + is deleted by hand once nothing only it holds is needed. """ found = ", ".join(f"{variant} (at {gitlinks[variant]})" for variant in variants) @@ -515,23 +551,51 @@ def _wrong_case_move(path: str, commit: str, variants: list[str]) -> str: """The index move for a member recorded only at `variants`, case variants of its row's `path`, to `path` at the pinned `commit`, with what it leaves on disk: what `_gitlink_in_another_case` says, and - `member-gitlink-unnamed` under `--no-members` (#648), in one text.""" + `member-gitlink-unnamed` under `--no-members` (#648), in one text. + + EACH `mv` IS GUARDED BY TWO TESTS A POSIX SHELL RUNS (#681, rv611c's + N-1 and N-2), not by a system's name: this text is read in CI too, on + another disk than the one it is typed on. `[ -ef ]` is + true where the disk takes the two names for one directory, the row's + own, as a Linux mount that folds case does too (WSL's `/mnt/c`), where + the `mv` moved the member's own checkout out. `[ -e /.git ]` + is true for a checkout, and a submodule's `.git` is a file naming this + holder's `.git/modules/` by a relative path, which a move leaves + pointing nowhere (`fatal: not a git repository`). So the `mv` runs only + for a directory apart from the row's with no `.git` entry, the empty one + a fresh clone leaves or a plain one, and the text says where a + submodule checkout's commits are kept, what deleting it would lose, and + to delete a checkout by hand once nothing it holds is needed. + """ taken_out = " ".join(shlex.quote(variant) for variant in variants) moved = " and ".join( - f"`mv -- {shlex.quote(variant)} " + f"`[ {shlex.quote(variant)} -ef {shlex.quote(path)} ] || " + f"[ -e {shlex.quote(variant + '/.git')} ] || " + f"mv -- {shlex.quote(variant)} " f"{shlex.quote('../' + variant.replace('/', '-'))}`" for variant in variants) + stores = " or ".join(f".git/modules/{variant}" for variant in variants) + status = " or ".join(f"`git -C {shlex.quote(variant)} status`" + for variant in variants) return ( "Move its gitlink to the row's path, at the commit the row pins, in " "the index alone, which touches no file, with the two lines below, " f"and commit that. Where nothing is checked out at {path}, as in a " "fresh clone, `git submodule update --init --recursive` then checks " - "the member out there. On a case-sensitive disk, as Linux's usually " - f"is, a directory left at {' or '.join(variants)} stays there, " + "the member out there. A directory left at " + f"{' or '.join(variants)} is the row's own where the disk folds " + "case, as macOS's and Windows' usually do and a Linux mount can " + "(WSL's /mnt/c). Where the disk holds the two apart it stays there, " "untracked, and family.py add refuses it as " "family-members-dir-variant: move each one that is there out of the " - f"holder, {moved}. On macOS and Windows it is the row's own " - "directory, so leave it. The two lines:\n" + f"holder, {moved}, which moves nothing where the two names are one " + "directory or where it holds a .git entry, a checkout. A checkout " + "git submodule add made there holds a .git file naming this " + "holder's .git/modules/ by a path relative to it, which a move would " + f"leave pointing nowhere; its commits are kept in {stores}, whatever " + "becomes of the directory, and only what is not committed is in the " + f"directory alone ({status} shows it). Delete a checkout by hand " + "once nothing it holds is needed. The two lines:\n" f" git rm -r --cached --ignore-unmatch -- {taken_out}\n" " git update-index --add --cacheinfo " f"{shlex.quote(f'160000,{commit},{path}')}") @@ -637,7 +701,8 @@ def _check_member(root: Path, row: dict, seen: dict, report: Report, # ---- the gitlink ------------------------------------------------------- with _git_runs(f"the gitlink recorded at {path} could not be read"): - _check_member_gitlink(root, path, commit, project, gitlinks, report) + _check_member_gitlink(root, path, commit, project, + row.get("repository"), gitlinks, report) return Gitlinked(row, project, path, pin, commit) @@ -756,11 +821,20 @@ def _check_one_shape_file(root: Path, rel: str, row, report: Report) -> None: """One `files:` row: present, and its digest matches its `sha256`. Split from `_check_shape_pin` for #137 — the per-row body of its loop. + A row that is no mapping, or whose `path:` is absent, empty or no + string, is `shape-pin-row`, as `update-shape.py` refuses it + `update-pin-row`: its `path:` was read as a path, so a row with none + was `None is pinned but absent` (#681). """ if not isinstance(row, dict): report.finding("shape-pin-row", f"{rel}: a files row is not a mapping") return - target = root / str(row.get("path")) + path = row.get("path") + if not isinstance(path, str) or not path: + report.finding("shape-pin-row", f"{rel}: a files row's `path:` is " + f"{path!r}, not the path of a file") + return + target = root / path if not target.is_file(): report.finding("shape-copy-missing", f"{rel}: {row.get('path')} is pinned but absent") @@ -1128,6 +1202,23 @@ def _member_gitlinks(root: Path) -> dict[str, str]: return found +def _staged_removals(root: Path, gitlinks: dict[str, str]) -> frozenset[str]: + """The paths of `gitlinks` (`_member_gitlinks`) whose removal is staged + and not committed (#681): HEAD records the gitlink, and the index holds + no entry at that path, at any stage. A `git rm` of one fails (`pathspec + ... did not match any files`), so a finding about one says to commit + the removal and prints none: it named the same `git rm` until the + commit. A path the index holds unmerged is a conflict, not a removal, + and is not one of these. The whole index, `-z`, as `_member_gitlinks` + lists it.""" + with _git_runs(f"the gitlinks this repository records in " + f"{DEFAULT_MEMBERS_DIR}/ could not be listed"): + listed = git_out(["ls-files", "-z"], cwd=root, binary=True) + indexed = {name.decode("utf-8", "surrogateescape") + for name in listed.split(b"\0")} + return frozenset(set(gitlinks) - indexed) + + class RowPath(NamedTuple): """What a row names by the path it gives: `who` names the row in a finding, `canonical` is the one path its project may have, @@ -1271,7 +1362,8 @@ def _variant_advice(kind: str, path: str, row_path: str, row: RowPath, def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], - named: dict[str, RowPath], report: Report) -> None: + named: dict[str, RowPath], removed: frozenset[str], + report: Report) -> None: """Every gitlink in `members/` is at a path some row names EXACTLY — the `member-gitlink-unnamed` finding (#361). @@ -1306,6 +1398,11 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], A BRANCH FIRST (#539): `add` refuses the holder's tracking branch (#487), so the first of the three lines is the `git switch -c` that makes the branch `add` suggests. + + A REMOVAL STAGED AND NOT COMMITTED (`removed`, `_staged_removals`) is + still a gitlink HEAD records, so it is still named, and its `git rm` + typed again failed: it says to commit the removal instead and prints + no `git rm` (#681). A hand mount's keeps the branch and the `add`. """ holds = set(gitlinks) for path, commit in sorted(gitlinks.items()): @@ -1336,36 +1433,78 @@ def _check_gitlinks_named(root: Path, gitlinks: dict[str, str], continue report.finding( "member-gitlink-unnamed", - f"{said}{_variant_said(row_path, row)} Take it out of the " - "index alone, which touches no file, and commit " - f"that:\n git rm --cached -- {shlex.quote(path)}") + f"{said}{_variant_said(row_path, row)} " + + _second_path_out(path, path in removed)) continue parts = path.split("/") project = parts[1] if len(parts) == 2 and parts[1] else None - member = (PROJECT_PLACEHOLDER if project is None - else shlex.quote(project)) report.finding( "member-gitlink-unnamed", - f"{said} {RESTORE_ROW} If it was mounted by hand, make a branch " - "with the first command below, since family.py add refuses the " - "holder's tracking branch, take it out with the second, which " - f"refuses while its checkout holds changes and takes its " - f"{GITMODULES} section out too, and commit that removal, since " - "family.py add refuses a path under another spelling of members/ " - "that HEAD holds and the index no longer does, then mount it " - "with the third, from a checkout of openRepoShape: family.py add " - "writes its row and its gitlink in one commit, and names the " - "object store git rm leaves under .git/modules if one is still " - f"there. {BRANCH_LEFT}\n" - f" {_switch_line(root, 'add', project)}\n" - f" git rm -- {shlex.quote(path)}\n" - f" {PYTHON} scripts/family.py add --family-root " - f"{shlex.quote(str(root))} --member /{member}") + f"{said} " + _mounted_by_hand(root, path, project, + path in removed)) + + +#: What a gitlink whose removal is staged and not committed says in place of +#: the `git rm` that would take it out, which fails there (#681). +STAGED_REMOVAL = ("HEAD records it and the index holds nothing at that " + "path, so its removal is staged and not committed, and no " + "git rm is printed, since one fails there (pathspec ... " + "did not match any files).") + + +def _second_path_out(path: str, staged: bool) -> str: + """What a `STRAY` gitlink, a second path to a member's own directory, + says to do: take it out of the index alone, or, where that is staged + already (#681), commit it.""" + if staged: + return f"{STAGED_REMOVAL} Commit that removal." + return ("Take it out of the index alone, which touches no file, and " + f"commit that:\n git rm --cached -- {shlex.quote(path)}") + + +def _mounted_by_hand(root: Path, path: str, project: str | None, + staged: bool) -> str: + """What a gitlink no row names, in no case, says to do after what it is: + put its row back, or take a hand mount out, commit that, and mount it + with `family.py add`, on a branch (`_check_gitlinks_named`). Where its + removal is staged already (#681), the same with no `git rm`.""" + member = (PROJECT_PLACEHOLDER if project is None + else shlex.quote(project)) + add = (f" {PYTHON} scripts/family.py add --family-root " + f"{shlex.quote(str(root))} --member /{member}") + if staged: + return ( + f"{STAGED_REMOVAL} Commit that removal before family.py add, " + "which refuses a path under another spelling of members/ that " + "HEAD holds and the index no longer does, and on a branch, since " + "family.py add refuses the holder's tracking branch: where the " + "holder stands on it, the first command below makes one, taking " + "the staged removal with it. If it was mounted by hand, then " + "mount it with the second, from a checkout of " + "openRepoShape: family.py add writes its row and its gitlink in " + "one commit, and names the object store git rm leaves under " + f".git/modules if one is still there. {BRANCH_LEFT}\n" + f" {_switch_line(root, 'add', project)}\n{add}") + return ( + f"{RESTORE_ROW} If it was mounted by hand, make a branch " + "with the first command below, since family.py add refuses the " + "holder's tracking branch, take it out with the second, which " + f"refuses while its checkout holds changes and takes its " + f"{GITMODULES} section out too, and commit that removal, since " + "family.py add refuses a path under another spelling of members/ " + "that HEAD holds and the index no longer does, then mount it " + "with the third, from a checkout of openRepoShape: family.py add " + "writes its row and its gitlink in one commit, and names the " + "object store git rm leaves under .git/modules if one is still " + f"there. {BRANCH_LEFT}\n" + f" {_switch_line(root, 'add', project)}\n" + f" git rm -- {shlex.quote(path)}\n{add}") def _check_gitmodules_named(sections: dict[str, dict[str, str]], gitlinks: dict[str, str], - named: dict[str, RowPath], report: Report) -> None: + named: dict[str, RowPath], removed: frozenset[str], + report: Report) -> None: """Every path a `.gitmodules` section gives is a path some row names EXACTLY — the `gitmodules-path-unnamed` finding (#361). @@ -1386,9 +1525,12 @@ def _check_gitmodules_named(sections: dict[str, dict[str, str]], A section at the path of a gitlink whose finding says to `git rm` it prints no command: that `git rm` takes the section out too, and a - `--unset-all` run after it exits 5, the key being gone. Where two sections give the - path (`gitmodules-path-twice`) it takes out only the one git reads for - it, the last in the file, so each of those keeps its own commands. + `--unset-all` run after it exits 5, the key being gone. A gitlink whose + removal is staged (`removed`, #681) prints no `git rm`, so its section, + which `git rm --cached` leaves, keeps its own commands. Where two + sections give the path (`gitmodules-path-twice`) it takes out only the + one git reads for it, the last in the file, so each of those keeps its + own commands. """ holds = {keys["path"] for keys in sections.values() if "path" in keys} given = Counter(keys["path"] for keys in sections.values() @@ -1418,7 +1560,7 @@ def _check_gitmodules_named(sections: dict[str, dict[str, str]], said += (" If its row was deleted or its path: edited by hand, " "put the row back as git last had it.") lead = "Otherwise take" - if path in gitlinks and given[path] == 1: + if path in gitlinks and path not in removed and given[path] == 1: report.finding( "gitmodules-path-unnamed", f"{said} Otherwise the git rm printed above for the " @@ -1435,6 +1577,28 @@ def _check_gitmodules_named(sections: dict[str, dict[str, str]], f"of them, then commit {GITMODULES}:{removal}") +def _section_lines(path: str, repository, + sections: dict[str, dict[str, str]]) -> str: + """The lines that write the `.gitmodules` section of the gitlink at + `path` as `family.py add` writes it, each on an indented line of its + own: named after its path, with the url it gives a member, + `https://github.com/` and the row's `repository:`, or `` where that + is not `/`. A section of that name in `sections` (HEAD's) + that has a url keeps it: only its path is written. What + `member-gitmodules-absent` prints, and `member-gitlink-absent` (#681). + """ + keys = {"path": shlex.quote(path)} + if "url" not in sections.get(path, {}): + keys["url"] = ( + shlex.quote(f"https://github.com/{repository}.git") + if isinstance(repository, str) + and QUALIFIED_RE.match(repository) else "") + return "".join( + f"\n git config -f {GITMODULES} " + f"{shlex.quote(f'submodule.{path}.{key}')} {value}" + for key, value in keys.items()) + + def _check_gitlinks_registered(root: Path, members: list, gitlinks: dict[str, str], report: Report) -> set[str]: @@ -1462,10 +1626,8 @@ def _check_gitlinks_registered(root: Path, members: list, for nothing until it is committed (the module docstring gives the three cases). - The remedy writes the section as `family.py add` writes it, named after - its path, with the url it gives a member, `https://github.com/` and the - row's `repository:`, or `` where that is not `/`. A - section of that name that has a url keeps it: only its path is written. + The remedy writes the section as `family.py add` writes it + (`_section_lines`). """ head = _head_gitmodules(root) if head is None: @@ -1482,17 +1644,7 @@ def _check_gitlinks_registered(root: Path, members: list, or path not in gitlinks or path in given): continue unregistered.add(path) - repository = row.get("repository") - keys = {"path": shlex.quote(path)} - if "url" not in sections.get(path, {}): - keys["url"] = ( - shlex.quote(f"https://github.com/{repository}.git") - if isinstance(repository, str) - and QUALIFIED_RE.match(repository) else "") - lines = "".join( - f"\n git config -f {GITMODULES} " - f"{shlex.quote(f'submodule.{path}.{key}')} {value}" - for key, value in keys.items()) + lines = _section_lines(path, row.get("repository"), sections) report.finding( "member-gitmodules-absent", f"{path} is a gitlink at {gitlinks[path]} that a row of members: " @@ -1534,8 +1686,9 @@ def _check_members(root: Path, members: list, no_members: bool, _check_gitmodules_paths(sections, report) named = _row_paths(members) gitlinks = _member_gitlinks(root) - _check_gitlinks_named(root, gitlinks, named, report) - _check_gitmodules_named(sections, gitlinks, named, report) + removed = _staged_removals(root, gitlinks) + _check_gitlinks_named(root, gitlinks, named, removed, report) + _check_gitmodules_named(sections, gitlinks, named, removed, report) unregistered = _check_gitlinks_registered(root, members, gitlinks, report) if no_members: diff --git a/tests/test_family.py b/tests/test_family.py index 960daf90..84b2c7d2 100644 --- a/tests/test_family.py +++ b/tests/test_family.py @@ -680,7 +680,7 @@ def test_validate_refuses_a_missing_gitlink(holder): result = validate(holder) assert result.returncode == 2 assert "member-gitlink-absent" in result.stderr - assert "family.py add" in result.stderr + assert "git update-index --add --cacheinfo 160000," in result.stderr def test_validate_finds_a_pin_that_disagrees_with_the_gitlink(holder): diff --git a/tests/test_family_bootstrap_gate_order.py b/tests/test_family_bootstrap_gate_order.py index 9962f731..2598cb94 100644 --- a/tests/test_family_bootstrap_gate_order.py +++ b/tests/test_family_bootstrap_gate_order.py @@ -29,17 +29,16 @@ and commits, never two directories whose names differ only in case, and the holder's own are asked again with `core.ignorecase=true`. A clone's checkout is git's: on a disk that folds case, `Members/

` IS the row's -directory, so the `mv` the text gives for a case-sensitive disk is typed -only where the disk holds the two apart, as the text says. +directory, and the guarded `mv` the text gives, typed in a POSIX shell, +moves nothing there; it moves the directory out only where the disk holds +the two apart (#681). """ from __future__ import annotations import itertools import os -import re import shlex -import subprocess from pathlib import Path import pytest @@ -50,8 +49,8 @@ from test_family_add_refusals import holder, offline # noqa: F401 from test_family_dry_run import isolated from test_family_head_lost_gitmodules import ( - DISKS, assert_passes_in_every_mode, commit_all, head_lost, on_disk, - pinned, recorded_at, variant_refused, + DISKS, GUARDED_MOVE, assert_passes_in_every_mode, commit_all, head_lost, + on_disk, pinned, recorded_at, typed_in_sh, variant_refused, ) from test_family_member_path import MEMBER, OTHER, raw from test_validate_family_unnamed_mounts import ( @@ -165,19 +164,21 @@ def type_line(root: Path, env: dict, line: str) -> None: def moved_out_where_apart(root: Path, remediation: str) -> None: - """Each backticked `mv` the text gives for a case-sensitive disk, typed - as printed where the disk holds its source apart from the row's own - directory, as the text says; on a disk that folds case it is the row's - directory, and the text says to leave it.""" - for move in re.findall(r"`(mv -- [^`]*)`", remediation): - argv = shlex.split(move) + """Each backticked guarded `mv` the text gives, typed as printed in a + POSIX shell for each directory that is there, as the text says: it + moves the directory out where the disk holds it apart from the row's + own, and moves nothing where the disk folds the two names into one + directory, the row's (#681).""" + for move in GUARDED_MOVE.findall(remediation): + argv = shlex.split(move.rsplit(" || ", 1)[1]) source = root / argv[2] own = root / "members" / argv[2].split("/", 1)[1] - if not source.exists() or (own.exists() - and os.path.samefile(source, own)): + if not source.exists(): continue - subprocess.run(argv, cwd=str(root), check=True) - assert not source.exists() + apart = not (own.exists() and os.path.samefile(source, own)) + typed = typed_in_sh(root, move) + assert typed.returncode == 0, typed.stderr + assert source.exists() is not apart # --- (a) bootstrap ----------------------------------------------------------- diff --git a/tests/test_family_head_lost_gitmodules.py b/tests/test_family_head_lost_gitmodules.py index 85f35308..6916e98e 100644 --- a/tests/test_family_head_lost_gitmodules.py +++ b/tests/test_family_head_lost_gitmodules.py @@ -665,8 +665,11 @@ def variant_refused(path: str, commit: str, variants: list[str], project = path.split("/", 1)[1] found = ", ".join(f"{variant} (at {gitlinks[variant]})" for variant in variants) - moved = " and ".join(f"`mv -- {variant} ../{variant.replace('/', '-')}`" + moved = " and ".join(f"`[ {variant} -ef {path} ] || [ -e {variant}/.git ] " + f"|| mv -- {variant} ../{variant.replace('/', '-')}`" for variant in variants) + stores = " or ".join(f".git/modules/{variant}" for variant in variants) + status = " or ".join(f"`git -C {variant} status`" for variant in variants) return [ f"REFUSED member-gitlink-absent: no gitlink (mode 160000) recorded at " f"'{path}'. {project} is named in family.yaml, and this repository " @@ -678,11 +681,19 @@ def variant_refused(path: str, commit: str, variants: list[str], "alone, which touches no file, with the two lines below, and commit " f"that. Where nothing is checked out at {path}, as in a fresh clone, " "`git submodule update --init --recursive` then checks the member " - "out there. On a case-sensitive disk, as Linux's usually is, a " - f"directory left at {' or '.join(variants)} stays there, untracked, " - "and family.py add refuses it as family-members-dir-variant: move " - f"each one that is there out of the holder, {moved}. On macOS and " - "Windows it is the row's own directory, so leave it. The two lines:", + f"out there. A directory left at {' or '.join(variants)} is the " + "row's own where the disk folds case, as macOS's and Windows' " + "usually do and a Linux mount can (WSL's /mnt/c). Where the disk " + "holds the two apart it stays there, untracked, and family.py add " + "refuses it as family-members-dir-variant: move each one that is " + f"there out of the holder, {moved}, which moves nothing where the " + "two names are one directory or where it holds a .git entry, a " + "checkout. A checkout git submodule add made there holds a .git " + "file naming this holder's .git/modules/ by a path relative to it, " + "which a move would leave pointing nowhere; its commits are kept in " + f"{stores}, whatever becomes of the directory, and only what is not " + f"committed is in the directory alone ({status} shows it). Delete a " + "checkout by hand once nothing it holds is needed. The two lines:", " git rm -r --cached --ignore-unmatch -- " + " ".join(variants), f" git update-index --add --cacheinfo 160000,{commit},{path}"] @@ -781,28 +792,43 @@ def case_sensitive_disk(tmp_path): pytest.skip("no mv on PATH to type the printed line with") -def test_a_variant_checkout_left_on_disk_is_moved_out_as_printed( +#: The guarded `mv` the variant refusal gives for each variant, backticked: +#: two shell tests, then the move (#681). +GUARDED_MOVE = re.compile(r"`(\[ [^`]* \|\| mv -- [^`]*)`") + + +def typed_in_sh(root: Path, line: str) -> subprocess.CompletedProcess: + """`line` typed as printed, in a POSIX shell standing in `root`.""" + return subprocess.run(["sh", "-c", line], cwd=str(root), + capture_output=True, check=False) + + +def test_a_variant_directory_left_on_disk_is_moved_out_as_printed( family, holder, offline, tmp_path, case_sensitive_disk): """THE REVIEW'S N2, in its C3 layout: MEMBER's gitlink recorded only at `Members/IRRS`, its checkout still at `members/IRRS`, and a second - checkout at `Members/IRRS`, as a case-sensitive disk can hold. The - index move leaves that directory on disk, untracked, where `family.py - add` refused any other member past it (`family-members-dir-variant`). - Every line the refusal prints, typed as printed: the two lines, their - commit, and the `mv` it gives for a case-sensitive disk. The gate then - passes in every mode, and SPARE is added.""" + directory at `Members/IRRS` holding the member's files, as a + case-sensitive disk can hold. The index move leaves that directory on + disk, untracked, where `family.py add` refused any other member past it + (`family-members-dir-variant`). Every line the refusal prints, typed as + printed: the two lines, their commit, and the guarded `mv`, in a POSIX + shell. A plain directory, apart from the row's and with no `.git` + entry, is moved out (a checkout is not since #681: + `tests/test_validate_family_remedies_typed.py`). The gate then passes + in every mode, and SPARE is added.""" env = isolated(offline, tmp_path) variant = f"Members/{MEMBER}" commit = recorded_at(holder, env, MEMBER, variant) - raw(holder, env, "clone", "-q", "--no-checkout", - str(family["remotes"] / f"{MEMBER}.git"), variant) - raw(holder / variant, env, "checkout", "-q", "--detach", commit) + (holder / variant).mkdir(parents=True) + (holder / variant / "project.yaml").write_bytes( + (holder / "members" / MEMBER / "project.yaml").read_bytes()) lines = refused_at(holder, commit, [variant]) moved_as_printed(holder, env, lines, [variant]) - (move,) = re.findall(r"`(mv [^`]*)`", lines[1]) - subprocess.run(shlex.split(move), cwd=str(holder), check=True) + (move,) = GUARDED_MOVE.findall(lines[1]) + typed = typed_in_sh(holder, move) + assert typed.returncode == 0, typed.stderr assert not (holder / variant).exists() assert (tmp_path / f"Members-{MEMBER}" / "project.yaml").is_file() assert_passes_in_every_mode(holder, 2) diff --git a/tests/test_family_switch_branch_exists.py b/tests/test_family_switch_branch_exists.py index e6490ddc..e8128a96 100644 --- a/tests/test_family_switch_branch_exists.py +++ b/tests/test_family_switch_branch_exists.py @@ -300,24 +300,31 @@ def test_another_name_keeps_a_commit_made_on_a_detached_head(family, on_main): # --- validate-family.py: each remedy that names the branch ------------------ @STATES -def test_a_missing_gitlinks_remedy_typed_as_printed(on_main, left): - """`member-gitlink-absent`: the remedy's prose ends with the sentence, - and its first command, typed as printed, puts the holder on the branch - the `add` under it needs, in either state.""" +def test_a_staged_hand_mount_removals_remedy_typed_as_printed(family, + on_main, left): + """`member-gitlink-unnamed` for a hand mount whose removal is staged and + not committed (#681): the finding's prose ends with the sentence, and + its first command, typed as printed in either state, puts the holder on + the branch `add` needs, with the removal still staged.""" holder = on_main - git("rm", "-r", "-q", "-f", "--", "members/IRRS", cwd=holder) - git("commit", "-qm", "Drop the gitlink but keep the row", cwd=holder) - earlier = leave_branch(holder, "family/add-irrs") if left else None - result = validate(holder) - assert result.returncode == 2, result.stdout + result.stderr - lines = result.stderr.splitlines() - assert lines[0].startswith("REFUSED member-gitlink-absent: "), lines - assert lines[1].endswith( - "a row with no submodule is a claim about a tree that is not here. " - + BRANCH_LEFT), lines[1] - switch = shlex.split(lines[2]) - assert switch[-2:] == ["-c", "family/add-irrs"], lines[2] - type_switch(holder, switch, "family/add-irrs", earlier) + _, tip = spare_commits(family) + path = f"members/{SPARE}" + stage_gitlink(holder, path, tip) + git("commit", "-qm", "A hand mount", cwd=holder) + git("rm", "-q", "--cached", "--", path, cwd=holder) + branch = f"family/add-{SPARE.lower()}" + earlier = leave_branch(holder, branch) if left else None + result = validate(holder, "--no-members") + assert result.returncode == 1, result.stderr + result.stdout + finding = result.stderr.split("FINDING member-gitlink-unnamed: ", 1)[1] + assert finding.splitlines()[0].endswith( + "if one is still there. " + BRANCH_LEFT), finding + assert "git rm" not in "".join(finding.splitlines()[1:]), finding + switch = shlex.split(finding.splitlines()[1]) + assert switch[-2:] == ["-c", branch], switch + type_switch(holder, switch, branch, earlier) + assert git("diff", "--cached", "--name-only", cwd=holder).stdout == ( + f"{path}\n") @STATES @@ -414,10 +421,10 @@ def unguarded(source: str) -> list[int]: def test_every_remedy_that_names_a_switch_line_ends_with_the_sentence(): """THE GUARD (S2). The validator's three remedies that print a `git - switch -c` (`member-gitlink-absent`, `member-duplicate-id` and - `member-gitlink-unnamed`) end their prose with `BRANCH_LEFT`, and so - does any remedy added later: the one that builds it cannot forget, as - #610's `_lockstep` does, without this failing.""" + switch -c` (`member-duplicate-id`, and `member-gitlink-unnamed`'s two + for a hand mount, its removal staged or not, #681) end their prose with + `BRANCH_LEFT`, and so does any remedy added later: the one that builds + it cannot forget, as #610's `_lockstep` does, without this failing.""" source = TEMPLATE.read_text(encoding="utf-8") assert len(switch_remedies(source)) >= 3, switch_remedies(source) assert unguarded(source) == [] diff --git a/tests/test_shape_pin_row_path.py b/tests/test_shape_pin_row_path.py new file mode 100644 index 00000000..e2512f6b --- /dev/null +++ b/tests/test_shape_pin_row_path.py @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: Apache-2.0 +"""A shape pin's `files:` row with no path is refused by name (#681, L2-3). + +`_check_one_shape_file` in the holder's `scripts/validate-family.py`, and +`_check_shape_copy` in the assembly root's `scripts/validate-pins.py`, asked +only whether a row is a mapping and then read `str(row.get("path"))`, so a +row `- nopath: x` was `FINDING shape-copy-missing: contracts/shape-pin.yaml: +None is pinned but absent`: the value `None` read as a path. `path: ''` +and `path: 5` were the same, as ` ` and `5`. `update-shape.py` refuses such +a row by name (`update-pin-row`). Both validators now say `shape-pin-row`, +the code their non-mapping branch says, naming the value, and check every +other row as before. + +THE SAME ROOTS as `tests/test_shape_pin_files_not_a_list.py` (#290): a +clean scaffold's clone, and a holder `family.py init` made, whose fixtures +are imported. The row is written beside the real ones, as BYTES, so the +pin stays LF on every OS and every other row still matches its file. +""" + +from __future__ import annotations + +import pytest + +from conftest import run_script +from test_shape_pin_files_not_a_list import ( # noqa: F401 + SHAPE_PIN, findings, holder, initialised_holder, no_traceback, +) + +#: `(the row written into files:, how the validator shows its path:)`. +ROWS = [ + pytest.param(" - nopath: x\n", "None", id="absent"), + pytest.param(" - path:\n", "None", id="null"), + pytest.param(" - path: ''\n", "''", id="empty"), + pytest.param(" - path: 5\n", "5", id="number"), + pytest.param(" - path: [a]\n", "['a']", id="list"), +] + +#: Each root and the validator it carries. +ROOTS = [ + pytest.param("project", "validate-pins.py", id="assembly-root"), + pytest.param("holder", "validate-family.py", id="family-holder"), +] + + +@pytest.mark.parametrize("fixture, validator", ROOTS) +@pytest.mark.parametrize("row, shown", ROWS) +def test_a_files_row_with_no_path_is_shape_pin_row(request, fixture, + validator, row, shown): + """One finding, `shape-pin-row`, naming the row's `path:` as the + validator names any value it cannot use, and no `shape-copy-missing`: + the value is never read as a path. The real rows are checked as before, + so it is the only finding, exit 1.""" + root = request.getfixturevalue(fixture) + pin = root / SHAPE_PIN + text = pin.read_bytes() + assert text.count(b"\nfiles:\n") == 1, "fixture drift: the pin's files:" + pin.write_bytes(text.replace(b"\nfiles:\n", + b"\nfiles:\n" + row.encode("utf-8"))) + + result = run_script(root / "scripts" / validator, cwd=root) + + no_traceback(result) + assert result.returncode == 1, result.stdout + result.stderr + assert findings(result, "shape-pin-row") == [ + f"FINDING shape-pin-row: {SHAPE_PIN}: a files row's `path:` is " + f"{shown}, not the path of a file"] + assert result.stderr.count("FINDING ") == 1, result.stderr diff --git a/tests/test_validate_family_branch_step.py b/tests/test_validate_family_branch_step.py index 2b611edf..a57e037b 100644 --- a/tests/test_validate_family_branch_step.py +++ b/tests/test_validate_family_branch_step.py @@ -9,11 +9,13 @@ cost one refusal first: `member-duplicate-id`'s `remove`, a hand mount's `member-gitlink-unnamed` `add` (run as printed in `tests/test_validate_family_unnamed_mounts.py`, now from the tracking -branch), and `member-gitlink-absent`'s `add`. Each now names `git -C - switch -c family/-` first, the branch `family.py` -suggests and the holder's docs show, POSIX-quoted as every line this gate -prints is, and the `add` of `member-gitlink-absent` names the holder by its -path, where `--family-root .` named the checkout the line runs from. +branch), and `member-gitlink-absent`'s `add`. Each named `git -C +switch -c family/-` first, the branch `family.py` suggests and +the holder's docs show, POSIX-quoted as every line this gate prints is. +Since #681 `member-gitlink-absent` names no `add`, which refuses the row +that is there, and so no branch: it names the index and `.gitmodules` +lines that put the gitlink back, typed in +`tests/test_validate_family_remedies_typed.py`. THE COUNT. The report ended ` finding(s) in family.yaml.` and THE LOCKSTEP RULE whatever was found; it ends at the rule only where a member's @@ -133,15 +135,13 @@ def test_the_duplicate_id_remedy_makes_the_branch_then_removes(family, assert validate(holder).returncode == 0 -# --- member-gitlink-absent: the branch, then the add, naming the holder ----- +# --- member-gitlink-absent: no add, so no branch (#681) --------------------- -def test_a_missing_gitlink_remedy_makes_the_branch_and_names_the_holder( - on_main): - """The refusal's two lines: the `switch` that makes `family.py`'s - branch for the `add`, and the `add` from a checkout of openRepoShape, - `--family-root` this holder by its path rather than `.`, which names - the checkout the line runs from. The `switch`, run as printed, puts - the holder on that branch.""" +def test_a_missing_gitlink_remedy_names_no_add_and_no_branch(on_main): + """`family.py add` refuses a row that is there + (`member-already-present`), so the refusal names neither it nor the + branch it would need. Every line it prints is a `git` line typed in + the holder.""" holder = on_main git("rm", "-r", "-q", "-f", "--", "members/IRRS", cwd=holder) git("commit", "-qm", "Drop the gitlink but keep the row", cwd=holder) @@ -149,21 +149,9 @@ def test_a_missing_gitlink_remedy_makes_the_branch_and_names_the_holder( assert result.returncode == 2, result.stdout + result.stderr lines = result.stderr.splitlines() assert lines[0].startswith("REFUSED member-gitlink-absent: "), lines - assert lines[1] == ( - "Remediation: from a checkout of openRepoShape, the second command " - "below mounts and pins it in one commit, on a branch other than the " - "holder's tracking branch, which it refuses, so the first makes one; " - "a row with no submodule is a claim about a tree that is not here. " - + BRANCH_LEFT) - switch, add = (line.strip() for line in lines[2:]) - assert lines[2:] == [" " + switch, " " + add], lines - assert shlex.split(switch) == switch_argv(holder, "add", "IRRS") - assert shlex.split(add) == [PYTHON, "scripts/family.py", "add", - "--family-root", str(holder), "--member", - "/IRRS"], add - git(*shlex.split(switch)[1:], cwd=holder.parent) - assert git("symbolic-ref", "--short", "HEAD", - cwd=holder).stdout.strip() == "family/add-irrs" + assert "switch -c" not in result.stderr, result.stderr + assert "scripts/family.py add" not in result.stderr, result.stderr + assert [shlex.split(line)[0] for line in lines[2:]] == ["git"] * 3 # --- the count ends at the rule only where it applies ----------------------- diff --git a/tests/test_validate_family_remedies_typed.py b/tests/test_validate_family_remedies_typed.py new file mode 100644 index 00000000..947a5d77 --- /dev/null +++ b/tests/test_validate_family_remedies_typed.py @@ -0,0 +1,489 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The holder gate's gitlink remedies, typed as printed in the state each +describes (#681). + +THE ISSUE'S FOUR GITLINK LINES, each reproduced on #682's head (8b39cd19) +before this change: + +- L20. `member-gitlink-unnamed` lists a gitlink HEAD records as well as one + the index does, so after its printed `git rm -- members/IRQS`, staged and + not committed, a re-run named the same gitlink and the same `git rm`, + which then died `fatal: pathspec 'members/IRQS' did not match any files`, + exit 128; a second path's `git rm --cached -- Members/IRRS` the same. It + says the removal is staged and to commit it now, and prints no `git rm`. +- L2-38. `member-gitlink-absent` for a row kept with its gitlink and + section taken out by `git rm` named `family.py add`, which refuses the + row (`member-already-present`), and `family.py remove` dies at its `git + submodule deinit`. It names the index line that puts the gitlink back at + the commit the row pins and the lines that write the section back, as + `member-gitmodules-absent` writes one, or the row taken out by hand + (Brett Heap, 2026-10-16T14:11:42Z, "take all recommendations for #670 + and #681": the words-only direction). +- N-1. The variant move's `mv` moved a live submodule checkout out of the + holder, whose `.git` file names `../../.git/modules/Members/

`, so + `git -C ../Members-

status` said `fatal: not a git repository`. The + line moves nothing where the directory holds a `.git` entry, and the text + says where its commits are kept. +- N-2. The `mv`'s condition was stated by system ("On a case-sensitive + disk, as Linux's usually is"), and on a Linux mount that folds case the + `mv` moved the member's own checkout out. The line is guarded by `[ + -ef ]`, which a POSIX shell runs. + +THE SAME FAMILY AND OFFLINE RULES as `test_family_head_lost_gitmodules.py` +(#611), whose helpers are imported. A case-folding disk is EMULATED for +N-2: `Members` a symbolic link to `members`, so the two names are one +directory on this case-sensitive one, which is what `-ef` asks; no +case-folding mount is used. A state that needs a directory at a case +variant is skipped where the disk cannot hold one (`case_sensitive_disk`). +""" + +from __future__ import annotations + +import os +import shlex +from pathlib import Path + +import pytest + +from conftest import git, rmtree, run_script +from test_family import ( # noqa: F401 + FAMILY, MEMBERS, ORG, SPARE, family, validate, +) +from test_family_add_refusals import holder, offline # noqa: F401 +from test_family_bootstrap_gate_order import fetching, type_line +from test_family_dry_run import isolated +from test_family_head_lost_gitmodules import ( # noqa: F401 + DISKS, GUARDED_MOVE, assert_passes_in_every_mode, case_sensitive_disk, + commit_all, github_to, moved_as_printed, on_disk, pinned, + pushed_and_cloned, recorded_at, refused_at, typed_in_sh, +) +from test_family_member_path import raw +from test_family_remove_gitmodules import verb_args +from test_family_tracking_branch import on_main # noqa: F401 +from test_validate_family_branch_step import BRANCH_LEFT +from test_validate_family_unnamed_mounts import ( + GITLINK_REMEDY, GITLINK_SAID, VARIANT, codes, commands, findings, removal, + spare_commits, stage_gitlink, stage_unmerged_gitlink, take_out_as_printed, + with_section, +) + +MEMBER = MEMBERS[0] +HAND = f"members/{SPARE}" + +#: What a gitlink whose removal is staged and not committed says in place of +#: a `git rm`, pinned as text: a holder's CI and a person read it. +STAGED = ("HEAD records it and the index holds nothing at that path, so its " + "removal is staged and not committed, and no git rm is printed, " + "since one fails there (pathspec ... did not match any files).") + +#: What a hand mount's says after it, in place of the `git rm` step. +STAGED_HAND_MOUNT = ( + f" {STAGED} Commit that removal before family.py add, which refuses a " + "path under another spelling of members/ that HEAD holds and the index " + "no longer does, and on a branch, since family.py add refuses the " + "holder's tracking branch: where the holder stands on it, the first " + "command below makes one, taking the staged removal with it. If it was " + "mounted by hand, then mount it with the second, from a checkout of " + "openRepoShape: family.py add writes its row and its gitlink in one " + "commit, and names the object store git rm leaves under .git/modules if " + "one is still there. " + BRANCH_LEFT) + + +def head_gitlink(root: Path, env: dict, path: str) -> str: + return raw(root, env, "rev-parse", f"HEAD:{path}").stdout.decode( + "ascii").strip() + + +def added_as_printed(family, root: Path, env: dict, add: str): + """The printed `family.py add`, `` filled in, from a checkout of + openRepoShape, reading the fixture's local remotes.""" + argv = [arg.replace("", ORG) for arg in shlex.split(add)] + assert argv[1:3] == ["scripts/family.py", "add"], add + return run_script(FAMILY, *argv[2:], "--local-remote-dir", + str(family["remotes"]), env=env) + + +# --- L20: a removal staged and not committed -------------------------------- + +def test_a_hand_mounts_staged_removal_is_committed_and_added_as_printed( + family, holder, offline, tmp_path): + """THE ISSUE'S L20, from a real hand mount. Its printed branch and `git + rm` typed, the re-run before the commit names the gitlink again, as + HEAD still records it, says the removal is staged, and prints the + branch and the `add` with no `git rm`, which typed again fails, as + before the change. Committed as it says, and the `add` typed as + printed (after the object store it names, as before), the holder passes + in every mode with the member mounted.""" + env = fetching(isolated(offline, tmp_path)) + raw(holder, env, "submodule", "add", "-q", "--", + str(family["remotes"] / f"{SPARE}.git"), HAND) + commit_all(holder, env, "Mount IRQS by hand") + at = head_gitlink(holder, env, HAND) + gitlink, _ = findings(validate(holder, "--no-members")) + switch, rm, add = commands(gitlink) + raw(holder.parent, env, *shlex.split(switch)[1:]) + raw(holder, env, *shlex.split(rm)[1:]) + + again = validate(holder, "--no-members") + + assert again.returncode == 1, again.stdout + again.stderr + assert findings(again) == [ + GITLINK_SAID.format(path=HAND, commit=at) + STAGED_HAND_MOUNT + + f"\n {switch}\n {add}"] + retyped = raw(holder, env, *shlex.split(rm)[1:], check=False) + assert retyped.returncode == 128, retyped.stderr + assert b"did not match any files" in retyped.stderr + commit_all(holder, env, "Take the hand mount out, as printed") + refused = added_as_printed(family, holder, env, add) + assert refused.returncode == 2, refused.stdout + refused.stderr + assert "REFUSED member-git-dir-cached: " in refused.stderr + rmtree(holder / ".git" / "modules" / "members" / SPARE) + added = added_as_printed(family, holder, env, add) + assert added.returncode == 0, added.stdout + added.stderr + assert_passes_in_every_mode(holder, 3) + + +def test_a_staged_removal_on_the_tracking_branch_goes_to_the_branch( + family, on_main, offline, tmp_path): + """Staged on the holder's tracking branch, where `add` refuses: the + finding's first command, typed as printed, makes the branch and takes + the staged removal there; committed, the `add` typed as printed mounts + the member on that branch.""" + env = isolated(offline, tmp_path) + _, tip = spare_commits(family) + stage_gitlink(on_main, HAND, tip) + commit_all(on_main, env, "A hand mount on main") + raw(on_main, env, "rm", "-q", "--cached", "--", HAND) + + (finding,) = findings(validate(on_main, "--no-members")) + + switch, add = commands(finding) + raw(on_main.parent, env, *shlex.split(switch)[1:]) + assert raw(on_main, env, "diff", "--cached", "--name-only").stdout == ( + f"{HAND}\n".encode()) + commit_all(on_main, env, "Take the hand mount out, as printed") + added = added_as_printed(family, on_main, env, add) + assert added.returncode == 0, added.stdout + added.stderr + assert raw(on_main, env, "symbolic-ref", "--short", + "HEAD").stdout.strip() == f"family/add-{SPARE.lower()}".encode() + assert_passes_in_every_mode(on_main, 3) + + +@DISKS +def test_a_second_paths_staged_removal_says_commit_it(holder, offline, + tmp_path, ignorecase): + """A second path to MEMBER's directory, `Members/IRRS`, committed: its + printed `git rm --cached` typed, the re-run says the removal is staged + and to commit it, with no command, where it printed the same `git rm + --cached` again. Committed, the holder passes in every mode.""" + env = on_disk(isolated(offline, tmp_path), ignorecase) + at = pinned(holder, MEMBER) + stage_gitlink(holder, f"Members/{MEMBER}", at) + commit_all(holder, env, "A second path to IRRS") + (first,) = findings(validate(holder, "--no-members")) + (rm,) = commands(first) + raw(holder, env, *shlex.split(rm)[1:]) + + (again,) = findings(validate(holder, "--no-members")) + + assert again == (GITLINK_SAID.format(path=f"Members/{MEMBER}", commit=at) + + VARIANT.format(row_path=f"members/{MEMBER}", + project=MEMBER) + + f" {STAGED} Commit that removal.") + commit_all(holder, env, "Commit the removal, as printed") + assert_passes_in_every_mode(holder, 2) + + +def test_a_section_git_rm_cached_leaves_keeps_its_own_commands( + family, holder, tmp_path): + """A hand mount taken out with `git rm --cached`, which leaves its + `.gitmodules` section: the gitlink prints no `git rm`, so its section's + finding prints the lines that take it out, where it said "the git rm + printed above takes this section out too". Typed as printed, the + commit carries both, and the holder passes.""" + _, tip = spare_commits(family) + stage_gitlink(holder, HAND, tip) + with_section(holder, tmp_path, HAND, HAND, f"/nowhere/{SPARE}.git") + git("rm", "-q", "--cached", "--", HAND, cwd=holder) + assert HAND in (holder / ".gitmodules").read_text(encoding="utf-8"), ( + "premise: git rm --cached leaves the section") + + result = validate(holder, "--no-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert codes(result) == ["member-gitlink-unnamed", + "gitmodules-path-unnamed"] + gitlink, section = findings(result) + assert "git rm" not in "".join(gitlink.splitlines()[1:]), gitlink + assert commands(section) == removal(HAND, "path", "url") + assert "takes this section out too" not in result.stderr + take_out_as_printed(holder, section) + assert_passes_in_every_mode(holder, 2) + + +def test_an_unmerged_gitlink_keeps_its_git_rm(family, holder): + """No misfire: a gitlink HEAD records that the index holds at the + conflict stages is a conflict, not a staged removal. It keeps the `git + rm` it had, which takes an unmerged path out, and committed, the holder + passes.""" + older, newer = spare_commits(family) + stage_gitlink(holder, HAND, older) + git("commit", "-qm", "A gitlink no row names", cwd=holder) + stage_unmerged_gitlink(holder, HAND, newer) + + (gitlink,) = findings(validate(holder, "--no-members")) + + assert gitlink.splitlines()[0] == GITLINK_SAID.format( + path=HAND, commit=older) + GITLINK_REMEDY + rm = commands(gitlink)[1] + assert rm == f"git rm -- {HAND}" + git(*shlex.split(rm)[1:], cwd=holder) + git("commit", "-qm", "Take it out, as printed", cwd=holder) + assert_passes_in_every_mode(holder, 2) + + +# --- L2-38: a row whose gitlink was taken out by hand ---------------------- + +def put_back(commit: str, *, section: bool) -> list[str]: + """`member-gitlink-absent`'s lines for MEMBER's row, whose gitlink is + gone and, where `section`, its `.gitmodules` section too.""" + path = f"members/{MEMBER}" + lines = [ + f"REFUSED member-gitlink-absent: no gitlink (mode 160000) recorded " + f"at '{path}'. {MEMBER} is named in family.yaml but this repository " + "does not record it as a submodule.", + "Remediation: family.py add refuses the row, which is there " + "(member-already-present), and family.py remove cannot take the " + "member out, since its git submodule deinit finds no gitlink. If the " + "member is to stay, put its gitlink back at the commit the row pins, " + "in the index alone, which touches no file, with the first line " + "below" + + (", and its .gitmodules section back by hand, in a POSIX shell, as " + "family.py add writes one, with the lines after it, then stage " + ".gitmodules" if section else "") + + ". Commit that, and `git submodule update --init --recursive`, or " + f"`make bootstrap`, then checks the member out at {path}. If it is to " + "go, take its row out of family.yaml by hand and commit that.", + f" git update-index --add --cacheinfo 160000,{commit},{path}"] + if section: + lines += [ + f" git config -f .gitmodules submodule.{path}.path {path}", + f" git config -f .gitmodules submodule.{path}.url " + f"https://github.com/{ORG}/{MEMBER}.git"] + return lines + + +#: How the gitlink was taken out, and whether its section went with it. +TAKEN_OUT = {"git-rm": (("rm", "-r", "-q", "--"), True), + "git-rm-cached": (("rm", "-r", "-q", "--cached", "--"), False)} + + +def taken_out(root: Path, env: dict, how: str) -> str: + """MEMBER's gitlink taken out `how`, committed, its row kept. Returns + the commit the row pins.""" + commit = pinned(root, MEMBER) + raw(root, env, *TAKEN_OUT[how][0], f"members/{MEMBER}") + commit_all(root, env, "Drop the gitlink but keep the row") + return commit + + +def put_back_as_printed(root: Path, env: dict, lines: list[str]) -> None: + """Every command line typed as printed, `.gitmodules` staged where a + line writes it, and the commit the text says to make.""" + for line in lines[2:]: + type_line(root, env, line) + if len(lines) > 3: + raw(root, env, "add", "--", ".gitmodules") + commit_all(root, env, "Put the gitlink back, as printed") + + +@pytest.mark.parametrize("how", TAKEN_OUT) +def test_a_row_whose_gitlink_was_taken_out_is_put_back_as_printed( + family, holder, offline, tmp_path, how): + """THE ISSUE'S L2-38, with the section taken out too (`git rm`) and + left (`git rm --cached`, whose section HEAD's `.gitmodules` still gives, + so no line writes it). The `add` it named is refused, as before. Typed + as printed and committed, `git submodule update --init --recursive` + checks the member out, and the holder passes in every mode.""" + env = fetching(isolated(offline, tmp_path)) + commit = taken_out(holder, env, how) + old_add = run_script(FAMILY, "add", "--family-root", str(holder), + "--member", f"{ORG}/{MEMBER}", "--local-remote-dir", + str(family["remotes"]), env=env) + assert old_add.returncode == 2, old_add.stdout + old_add.stderr + assert "REFUSED member-already-present: " in old_add.stderr + + refused = validate(holder) + + assert refused.returncode == 2, refused.stdout + refused.stderr + lines = refused.stderr.splitlines() + assert lines == put_back(commit, section=TAKEN_OUT[how][1]) + put_back_as_printed(holder, env, lines) + raw(holder, env, "submodule", "update", "--init", "--recursive") + assert_passes_in_every_mode(holder, 2) + + +def test_a_fresh_clone_of_it_is_put_back_and_checks_the_member_out( + family, holder, offline, tmp_path): + """The same state pushed and cloned, as CI checks it out: no object + store of the member's in the clone. Typed as printed there, the update + clones the member from the url the written section gives, and the + clone's gate passes in every mode.""" + env = fetching(isolated(offline, tmp_path)) + commit = taken_out(holder, env, "git-rm") + clone = pushed_and_cloned(holder, env, family, tmp_path, "w681/absent") + assert not (clone / ".git" / "modules" / "members" / MEMBER).exists() + + refused = validate(clone) + + assert refused.returncode == 2, refused.stdout + refused.stderr + lines = refused.stderr.splitlines() + assert lines == put_back(commit, section=True) + put_back_as_printed(clone, env, lines) + raw(clone, env, *github_to(family["remotes"]), "submodule", "update", + "--init", "--recursive") + assert (clone / "members" / MEMBER / "project.yaml").is_file() + assert_passes_in_every_mode(clone, 2) + + +def test_a_row_whose_member_is_to_go_is_taken_out_by_hand(holder, offline, + tmp_path): + """The other way the refusal gives: the row taken out of `family.yaml` + by hand and committed, and the holder passes in every mode with one + member.""" + env = isolated(offline, tmp_path) + taken_out(holder, env, "git-rm") + rows = (holder / "family.yaml").read_bytes() + start = rows.index(f" - project: {MEMBER}\n".encode()) + end = rows.index(f" - project: {MEMBERS[1]}\n".encode()) + (holder / "family.yaml").write_bytes(rows[:start] + rows[end:]) + raw(holder, env, "add", "--", "family.yaml") + commit_all(holder, env, "IRRS is to go: its row taken out by hand") + assert_passes_in_every_mode(holder, 1) + + +# --- N-1 and N-2: what the guarded `mv` moves ------------------------------ + +VARIANT_PATH = f"Members/{MEMBER}" +MOVED_TO = f"Members-{MEMBER}" + + +@pytest.mark.parametrize("layout", ["one-directory", "one-empty-directory", + "apart"]) +def test_the_move_moves_nothing_where_the_two_names_are_one_directory( + holder, offline, tmp_path, case_sensitive_disk, layout): + """THE ISSUE'S N-2. After the two lines and their commit, a directory + at `Members/IRRS` that IS the row's own, as on a disk that folds case + (emulated: `Members` a symbolic link to `members`): `-ef` is true, and + the guarded line moves nothing, where the bare `mv` moved the member's + checkout out of the holder. The row's own directory left empty, as a + fresh clone leaves it before the update (here `git submodule deinit`): + it holds no `.git` entry, so `-ef` alone keeps it in place. One apart + from it, the empty one a fresh clone leaves at `Members/IRRS`: `-ef` is + false, and the line moves it out. The holder passes in every mode + every way.""" + env = fetching(isolated(offline, tmp_path)) + commit = recorded_at(holder, env, MEMBER, VARIANT_PATH) + lines = refused_at(holder, commit, [VARIANT_PATH]) + moved_as_printed(holder, env, lines, [VARIANT_PATH]) + (move,) = GUARDED_MOVE.findall(lines[1]) + one = layout != "apart" + own = holder / "members" / MEMBER + if layout == "one-empty-directory": + raw(holder, env, "submodule", "deinit", "-q", "-f", "--", + f"members/{MEMBER}") + assert own.is_dir() and not any(own.iterdir()), "premise: emptied" + if one: + try: + os.symlink("members", holder / "Members", target_is_directory=True) + except OSError as exc: + pytest.skip(f"no symbolic link to emulate one directory: {exc}") + else: + (holder / VARIANT_PATH).mkdir(parents=True) + checkout = typed_in_sh(holder, f"[ -e {VARIANT_PATH}/.git ]") + + test = typed_in_sh(holder, f"[ {VARIANT_PATH} -ef members/{MEMBER} ]") + typed = typed_in_sh(holder, move) + + assert test.returncode == (0 if one else 1), test.stderr + assert checkout.returncode == (0 if layout == "one-directory" else 1) + assert typed.returncode == 0, typed.stderr + assert own.is_dir() + assert (tmp_path / MOVED_TO).exists() is not one + if one: + (holder / "Members").unlink() + else: + assert not (holder / VARIANT_PATH).exists() + if layout == "one-empty-directory": + raw(holder, env, "submodule", "update", "--init", "--", + f"members/{MEMBER}") + assert (own / "project.yaml").is_file() + assert_passes_in_every_mode(holder, 2) + + +def mounted_at_the_variant(family, root: Path, env: dict, commit: str, + kind: str) -> None: + """A checkout at `Members/IRRS`, at `commit`, recorded there and no + longer at `members/IRRS`, committed: a `git submodule add`, whose `.git` + is a file naming this holder's `.git/modules/Members/IRRS`, the 574/3 + layout, or a plain `git clone`, whose `.git` is its own directory.""" + remote = str(family["remotes"] / f"{MEMBER}.git") + if kind == "submodule": + raw(root, env, "submodule", "add", "-q", "--force", "--", remote, + VARIANT_PATH) + else: + stage_gitlink(root, VARIANT_PATH, commit) + raw(root, env, "clone", "-q", "--no-checkout", remote, VARIANT_PATH) + raw(root / VARIANT_PATH, env, "checkout", "-q", "--detach", commit) + raw(root, env, "add", "--", VARIANT_PATH) + raw(root, env, "rm", "-q", "--cached", "--", f"members/{MEMBER}") + commit_all(root, env, "IRRS recorded at Members/IRRS, checked out there") + + +@pytest.mark.parametrize("kind", ["submodule", "clone"]) +def test_a_checkout_at_the_variant_is_left_in_place_and_deleted_by_hand( + family, holder, offline, tmp_path, case_sensitive_disk, kind): + """THE ISSUE'S N-1. A checkout at `Members/IRRS` holding a commit of + its own: after the two lines and their commit, the guarded line moves + nothing, where the bare `mv` left a submodule's checkout saying `fatal: + not a git repository`. It is still a working checkout, and a + submodule's commits are where the text says, in + `.git/modules/Members/IRRS`, and stay there once the directory is + deleted by hand, as the text says to do when nothing it holds is + needed. Then the holder passes in every mode, and SPARE is added past + it, where `family.py add` refused the directory.""" + env = fetching(isolated(offline, tmp_path)) + commit = pinned(holder, MEMBER) + mounted_at_the_variant(family, holder, env, commit, kind) + checkout = holder / VARIANT_PATH + raw(checkout, env, "commit", "-q", "--allow-empty", "-m", "Work there") + work = raw(checkout, env, "rev-parse", "HEAD").stdout.decode( + "ascii").strip() + store = holder / ".git" / "modules" / "Members" / MEMBER + if kind == "submodule": + assert (checkout / ".git").read_bytes() == ( + b"gitdir: ../../.git/modules/Members/" + MEMBER.encode() + b"\n") + lines = refused_at(holder, commit, [VARIANT_PATH]) + moved_as_printed(holder, env, lines, [VARIANT_PATH]) + (move,) = GUARDED_MOVE.findall(lines[1]) + + typed = typed_in_sh(holder, move) + + assert typed.returncode == 0, typed.stderr + assert not (tmp_path / MOVED_TO).exists() + assert raw(checkout, env, "status", "--porcelain").returncode == 0 + if kind == "submodule": + assert Path(raw(checkout, env, "rev-parse", "--absolute-git-dir") + .stdout.decode().strip()).resolve() == store.resolve() + found = validate(holder) + assert codes(found) == ["gitmodules-path-unnamed"], found.stderr + take_out_as_printed(holder, findings(found)[0]) + rmtree(checkout) + if kind == "submodule": + kept = raw(holder, env, f"--git-dir={store}", f"--work-tree={holder}", + "cat-file", "-e", work, check=False) + assert kept.returncode == 0, kept.stderr + assert_passes_in_every_mode(holder, 2) + added = run_script(FAMILY, *verb_args("add", holder, family), env=env) + assert added.returncode == 0, added.stdout + added.stderr + assert_passes_in_every_mode(holder, 3) From c42286de2b602611fa8e514b804ac334bb3b5ff7 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 18:53:32 +0000 Subject: [PATCH 07/15] README.md and the handbook: for a holder straight in a folder the move makes "the family folder `~/InkRouter`, beside `~/projects`, and the holder `~/InkRouter/InkRouter`, not `~/projects/InkRouter/InkRouter`", so "beside that folder" no longer reads as the holder itself (#695) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review nit N1 on 73fa3ff4. What sits beside `~/projects` is the family folder `~/InkRouter`; the holder is one level inside it. The sentence now says so in README § Families and the handbook, and STRAIGHT_IN_A_FOLDER in tests/test_family_docs_one_spelling.py holds the new words. The README stays under its cap (1458 of 1459): the one added line is paid for by cutting the caption "Eight services, each its own project, one holder:" under "The InkRouter example", which repeats the section's intro and its block. The handbook marker for README § Families is recomputed. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Sonnet 5.5 --- README.md | 5 ++--- docs/handbook.html | 14 ++++++++------ tests/test_family_docs_one_spelling.py | 3 ++- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 80c14dde..1a59c9fb 100644 --- a/README.md +++ b/README.md @@ -915,12 +915,11 @@ repository, case included, on every disk. When it is not, and no link of that name beside it leads to it, `make siblings` WARNS, prints the exact `mv` for a human to run and clones nothing until it is run, so the next run clones each member once. For a holder straight in a folder (`~/projects/InkRouter`) the -move makes `~/InkRouter/InkRouter`, beside that folder, not inside it. +move makes the family folder `~/InkRouter`, beside `~/projects`, and the holder +`~/InkRouter/InkRouter`, not `~/projects/InkRouter/InkRouter`. ### The InkRouter example -Eight services, each its own project, one holder: - ``` InkRouter/ the family FOLDER (see the layout above) InkRouter/ the family holder — family.yaml, members/, utilities diff --git a/docs/handbook.html b/docs/handbook.html index 61cbeffa..cabc7a05 100644 --- a/docs/handbook.html +++ b/docs/handbook.html @@ -1282,7 +1282,7 @@

Adopting an existing repository

- +

Reference

Families: a holder for projects that ship separately

@@ -1344,11 +1344,13 @@

The workstation layout: a plain folder, and two copies of every member

every disk. When it is not, and no link of that name beside it leads to it, make siblings WARNS, prints the exact mv for a human to run and clones nothing until it is run, so the next run clones each member once. For a holder - straight in a folder (~/projects/InkRouter) the move makes - ~/InkRouter/InkRouter, beside that folder, not inside it. A tool that relocated - the directory would do it out from under the shell, the editor and every agent lane standing - in it, and would break linked worktrees, whose .git files carry absolute paths - — an act by surprise, which is the one thing every refusal here exists to avoid.

+ straight in a folder (~/projects/InkRouter) the move makes the family folder + ~/InkRouter, beside ~/projects, and the holder + ~/InkRouter/InkRouter, not ~/projects/InkRouter/InkRouter. A tool + that relocated the directory would do it out from under the shell, the editor and every + agent lane standing in it, and would break linked worktrees, whose .git files + carry absolute paths — an act by surprise, which is the one thing every refusal here + exists to avoid.

Ruled by Brett Heap on 2026-09-09 (issue #76): the plain folder, and the holder inside it at <Family>/<Family>.

diff --git a/tests/test_family_docs_one_spelling.py b/tests/test_family_docs_one_spelling.py index cd35e49b..eced3460 100644 --- a/tests/test_family_docs_one_spelling.py +++ b/tests/test_family_docs_one_spelling.py @@ -58,7 +58,8 @@ STRAIGHT_IN_A_FOLDER = ( "For a holder straight in a folder (`~/projects/InkRouter`) the move " - "makes `~/InkRouter/InkRouter`, beside that folder, not inside it.") + "makes the family folder `~/InkRouter`, beside `~/projects`, and the " + "holder `~/InkRouter/InkRouter`, not `~/projects/InkRouter/InkRouter`.") #: What README and the handbook said before #578, which `make siblings` no #: longer does. From 1a69505d8814ae0cc37e0c4281353c5caaedefaf Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 18:57:59 +0000 Subject: [PATCH 08/15] bootstrap.py: a plain clone's update that dies at a gitlink no section gives a url fails the run naming every row, and a member git exited 0 over is given the update that checks it out, so a stray case variant, a hand mount or a url-less section no longer ends `family bootstrap ok` (#648, rv648) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fix round for rv648's review of 8b39cd19 (1 MUST, 2 SHOULD, 5 NIT). M1. In a plain clone, `git submodule update --init` dies in its init step at the first gitlink, in index order, that no section of `.gitmodules` gives a url, and checks out no member. `unfetchable` asked only each row's own gitlink and section, so a stray `Members/

` beside `members/

` (#361's STRAY), a member mounted by hand with no section, or a row's section that lost its url (rv648's S1, H1 and U1) passed every row as the credential skip, and the run ended ok. `init_stop` now asks the holder's records which gitlink git's init stops at: the index's gitlinks in order, each against the work tree's sections (`gitmodules_sections`, which replaces `gitmodules_paths`) and the git config. Measured before it was taken, rv648's prototype failed three states that are a missing credential: a recursive clone of the stray or of the hand mount with no credential, and a url-less section an earlier `git submodule init` registered. So two of git's rules are kept: where the config sets `submodule.active` (a recursive clone sets `.`), init asks only the gitlinks a section gives, and a section with no url does not stop it where the config holds a url under its name. Where git exited non-zero and init stops at a gitlink, every row not checked out is named `git stopped at , a gitlink no section of .gitmodules gives a url`, and the run fails. What cures it: `validate-family.py --no-members` names a gitlink no row names and the commit that cures it (members checked, it refuses `member-uninitialized` for the first row in these states, a checklist candidate). Where a row names the gitlink, nothing in the validator does, so the line says to write that section back as `family.py add` writes one. Each why now carries its own sentence. S1. For "git exited 0 and checked nothing out there" (a subset clone, an `update = none`), the line printed the validator sentence, whose `member-uninitialized` remedy exits 0 and checks nothing out. It prints `git submodule update --init --checkout --recursive -- ` instead; typed, it checks the member out in both states and the run ends ok. S2. `test_a_missing_credential_is_still_the_skip_it_was` asserts that " every member and leg fetched" is not printed (rv648's mutant R2). NITs. README and the handbook say `make bootstrap` skips a member it lacks a credential for, and the § Families marker is the new segment's blob, b3d19092; README stays at 1459 lines, its cap. The docstrings say "an empty mapping" and "a missing credential exits non-zero too", and `validate-family.py`'s WHAT IS CHECKED says the move under `--no-members` is printed only for a canonical path and a 40-hex pin. Tests: the variant-plain case and the round trip name MEMBER too; new tests for S1 and H1 (typed through `--no-members` and committed, the run checks both out), U1 (the section written back), the two recursive clones with no credential and the registered url (still the skip), and the update that exits 0, by `update = none` and by a subset clone, with its printed line typed. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/handbook.html | 6 +- templates/family-root/scripts/bootstrap.py | 181 ++++++++-- .../family-root/scripts/validate-family.py | 7 +- tests/test_family_bootstrap_gate_order.py | 310 +++++++++++++++--- 5 files changed, 424 insertions(+), 82 deletions(-) diff --git a/README.md b/README.md index 0f17b2b8..4a4e78fa 100644 --- a/README.md +++ b/README.md @@ -952,7 +952,7 @@ at the right commit. **A family pins assembly roots, never legs**: the GitHub App or `SHAPE_LEGS_TOKEN` credential the workflow does, as a member is a private assembly root mounting two private legs — then runs each member's own `make bootstrap`; `make validate` runs the family validator, then each -member's. One it lacks a credential for is reported and skipped, not a failure. +member's. `make bootstrap` reports and skips one it lacks a credential for. The holder carries the same COPY pin an assembly root does, so `update-shape.py` re-syncs it the same way; it mirrors into `family.yaml` diff --git a/docs/handbook.html b/docs/handbook.html index f2e382e2..0c63875c 100644 --- a/docs/handbook.html +++ b/docs/handbook.html @@ -1282,7 +1282,7 @@

Adopting an existing repository

- +

Reference

Families: a holder for projects that ship separately

@@ -1375,8 +1375,8 @@

The InkRouter example

resolving the same GitHub App or SHAPE_LEGS_TOKEN credential the workflow does — and then runs each member’s own make bootstrap. make validate runs the family validator and then each member’s own - make validate. A member it lacks a credential for is reported and skipped, not - a failure. The holder carries the same COPY pin an assembly root does, so + make validate. A member make bootstrap lacks a credential for is + reported and skipped, not a failure. The holder carries the same COPY pin an assembly root does, so update-shape.py re-syncs it the same way; it mirrors into family.yaml instead of project.yaml and must leave validate-family.py green.

diff --git a/templates/family-root/scripts/bootstrap.py b/templates/family-root/scripts/bootstrap.py index e3fb3b10..4e089c99 100755 --- a/templates/family-root/scripts/bootstrap.py +++ b/templates/family-root/scripts/bootstrap.py @@ -31,13 +31,22 @@ every row's member is checked out (#648). A member left NOT CHECKED OUT where no credential would check it out is said here instead, with why, and fails the run (c): this holder records no gitlink at the - row's path (a gitlink recorded only in another case, #363), no + row's path (a gitlink recorded only in another case, #363); no section of `.gitmodules` gives that path (a holder whose HEAD lost the - file, #611), or git exited 0 and checked nothing out there. Git exits - 0 over the second where a recursive clone ran first, and 128 (`No url - found for submodule path`) in a plain clone, as it does for a missing - credential, so what the holder records is the test, never git's - words. The line names `validate-family.py`, which says what to commit. + file, #611); git exited 0 and checked nothing out there; or git + stopped at a gitlink no section gives a url, the row's own or another + (a stray case variant, a member mounted by hand, a section that lost + its url), since git's init dies at the first such gitlink it asks + about, before it checks out any member. Git exits 0 over the second + where a recursive clone ran first, and 128 (`No url found for + submodule path`) in a plain clone; a missing credential exits + non-zero too, so what the holder records is the test, never git's + words. The line says what cures each: `validate-family.py` names the + commit for the first two; for the third it prints the `git submodule + update --init --checkout` that checks the member out whatever the + clone chose; and for the last, `validate-family.py --no-members` + names a gitlink no row names, and where the gitlink is a row's own + the line says to write its section back. (b) RUNS EACH MEMBER'S OWN `make bootstrap`. Each member is a whole project and knows how to bootstrap itself — its legs onto their tracking @@ -60,6 +69,7 @@ import argparse import os +import shlex import shutil import subprocess import sys @@ -74,6 +84,17 @@ MANIFEST = "family.yaml" GITMODULES = ".gitmodules" DEFAULT_MEMBERS_DIR = "members" +#: A submodule's entry in the index, as `ls-files -s` spells its mode. +GITLINK_MODE = b"160000" + +#: Why (a) left a row's member NOT CHECKED OUT where no credential would +#: check it out (#648), in the order `unfetchable` asks. `STOPPED` is +#: filled with the gitlink git's init died at (`init_stop`). +NO_GITLINK = "no gitlink is recorded at that path" +NO_SECTION = f"no section of {GITMODULES} gives that path" +EXITED_0 = "git exited 0 and checked nothing out there" +STOPPED = ("git stopped at {}, a gitlink no section of " + GITMODULES + + " gives a url") #: The two environment variables a leg/member credential may arrive in, in the #: order the `validate` workflow resolves them. An App-minted installation @@ -124,71 +145,159 @@ def fetch_members(root: Path, rows: list[dict]) -> list[str]: "fine-grained PAT with contents:read on them) or export an " "App-minted SHAPE_LEGS_APP_TOKEN, then re-run. What did arrive " "is bootstrapped below.", file=sys.stderr) - left = unfetchable(root, rows, proc.returncode == 0) + left, stopper = unfetchable(root, rows, proc.returncode == 0) if left: print(" NOT CHECKED OUT, and not for want of a credential: " - + ", ".join(f"{spelled(path)} ({why})" for _, path, why in left) - + f". `{PYTHON} scripts/validate-family.py` names the cause and " - "the commit that cures it.", file=sys.stderr) + + ", ".join(f"{spelled(path)} (" + + (STOPPED.format(spelled(stopper)) + if why == STOPPED else why) + ")" + for _, path, why in left) + + ". " + cures(left, stopper, rows), file=sys.stderr) elif proc.returncode == 0: print(" every member and leg fetched") return [f"{spelled(row.get('project') or '?')} (not checked out)" for row, _, _ in left] -def gitmodules_paths(root: Path) -> set[str]: - """Every path a section of `.gitmodules` gives, read as git reads it: - the work tree's file, the one `git submodule update` reads, which a - fresh clone takes from HEAD. None where there is no such file, or one - git cannot read: git has no url for any submodule then either.""" +def cures(left: list[tuple[dict, str, str]], stopper: str | None, + rows: list[dict]) -> str: + """What (a)'s line says cures the rows `left` names, one sentence for + each why among them (#648). `validate-family.py`, which names the + commit, for a gitlink or a section the holder lacks at the row's path. + For the gitlink git stopped at, `validate-family.py --no-members`, which + names a gitlink no row names and the commit that cures it (members + checked, it refuses `member-uninitialized` for the first row there + instead), or, where a row names it, its section written back. For a row + git exited 0 over, the update that checks it out whatever the clone + chose: `--init` with the path registers a member a subset clone left + out, and `--checkout` overrides an `update = none`.""" + whys = {why for _, _, why in left} + said = [] + if whys & {NO_GITLINK, NO_SECTION}: + said.append(f"`{PYTHON} scripts/validate-family.py` names the cause " + "and the commit that cures it.") + if STOPPED in whys and stopper in {row_path(row) for row in rows}: + said.append(f"Write the section of {GITMODULES} for " + f"{spelled(stopper)} back as `family.py add` writes one, " + f"with its path and url, and commit {GITMODULES}.") + elif STOPPED in whys: + said.append(f"`{PYTHON} scripts/validate-family.py --no-members` " + f"names each gitlink in {DEFAULT_MEMBERS_DIR}/ that no " + "row names, and the commit that cures it.") + exited = [path for _, path, why in left if why == EXITED_0] + if exited: + update = shlex.join(["git", "submodule", "update", "--init", + "--checkout", "--recursive", "--", *exited]) + said.append(f"`{spelled(update)}` checks " + f"{'it' if len(exited) == 1 else 'them'} out.") + return " ".join(said) + + +def gitmodules_sections(root: Path) -> dict[str, dict[str, str]]: + """`{name: {key: value}}` of every section of `.gitmodules`, read as git + reads it: the work tree's file, the one `git submodule update` reads, + which a fresh clone takes from HEAD. An empty mapping where there is no + such file, or one git cannot read: git has no url for any submodule + then either.""" if not os.path.lexists(root / GITMODULES): - return set() + return {} try: listed = git_out(["config", "-z", "--file", GITMODULES, "--list"], cwd=root, binary=True) except Refusal: - return set() - paths: set[str] = set() + return {} + sections: dict[str, dict[str, str]] = {} for record in listed.split(b"\0"): key, _, value = record.decode("utf-8", "surrogateescape").partition("\n") - if key.startswith("submodule.") and key.endswith(".path"): - paths.add(value) - return paths + section, _, variable = key.rpartition(".") + if section.startswith("submodule."): + sections.setdefault(section[len("submodule."):], + {})[variable] = value + return sections + + +def init_stop(root: Path, sections: dict[str, dict[str, str]]) -> str | None: + """The gitlink `git submodule update --init` dies at, or None: the first + in the index, in index order, that its init step asks about and finds + no url for (`No url found for submodule path`, exit 128), before it + checks out any member (#648). + + ASKED AS GIT 2.43 ASKS IT, of the index, `sections` and this + repository's git config. Every gitlink the index records is asked, but + where the config sets `submodule.active`, as a recursive clone sets it + (`.`), only those a section gives a path: git counts no other active. + One no section gives stops it; so does one whose section gives no url, + unless the config has one under the section's name, as an earlier init + writes it. So a stray case variant or a member mounted by hand stops a + plain clone's update, and leaves a recursive clone's to the credential. + Read nothing, it answers None.""" + try: + index = git_out(["ls-files", "-s", "-z"], cwd=root, binary=True) + except Refusal: + return None + try: + config = git_out(["config", "-z", "--get-regexp", r"^submodule\."], + cwd=root, binary=True) + except Refusal: + config = b"" + keys = {record.decode("utf-8", "surrogateescape").partition("\n")[0] + for record in config.split(b"\0")} + names: dict[str, list[str]] = {} + for name, held in sections.items(): + if "path" in held: + names.setdefault(held["path"], []).append(name) + for record in index.split(b"\0"): + meta, _, entry = record.partition(b"\t") + if meta.split()[:1] != [GITLINK_MODE]: + continue + path = entry.decode("utf-8", "surrogateescape") + given = names.get(path, []) + if not given and "submodule.active" in keys: + continue + if not any("url" in sections[name] or f"submodule.{name}.url" in keys + for name in given): + return path + return None -def unfetchable(root: Path, rows: list[dict], - exited_0: bool) -> list[tuple[dict, str, str]]: +def unfetchable(root: Path, rows: list[dict], exited_0: bool + ) -> tuple[list[tuple[dict, str, str]], str | None]: """`(row, path, why)` for each row whose member (a) left NOT CHECKED OUT - where no credential would check it out (#648), in row order. + where no credential would check it out (#648), in row order, and the + gitlink git's init died at (`init_stop`), or None. Asked of this holder's own records, never of git's words: a row at whose path the holder records no gitlink (`recorded_gitlink`, as `validate-family.py`'s `member-gitlink-absent` reads one), or whose gitlink no section of `.gitmodules` gives a path (as its `member-gitmodules-absent` reads HEAD's file, which is the work tree's - in a clone); and, where git exited 0, any row at all, since git then - lacked nothing a credential gives. Any other row not checked out is the - missing credential's, and stays the skip it always was. A path two rows - give is said once.""" + in a clone); where git exited 0, any row at all, since git then lacked + nothing a credential gives; and where git's init died at a gitlink no + section gives a url, any row at all, since git then checked out no + member and asked for no credential. Any other row not checked out is + the missing credential's, and stays the skip it always was. A path two + rows give is said once.""" left: dict[str, dict] = {} for row in rows: if not (member_path(root, row) / ".git").exists(): left.setdefault(row_path(row), row) if not left: - return [] - given = gitmodules_paths(root) + return [], None + sections = gitmodules_sections(root) + given = {held["path"] for held in sections.values() if "path" in held} + stopper = None if exited_0 else init_stop(root, sections) found: list[tuple[dict, str, str]] = [] for path, row in left.items(): if recorded_gitlink(root, path) is None: - found.append((row, path, "no gitlink is recorded at that path")) + found.append((row, path, NO_GITLINK)) elif path not in given: - found.append((row, path, f"no section of {GITMODULES} gives that " - "path")) + found.append((row, path, NO_SECTION)) elif exited_0: - found.append((row, path, "git exited 0 and checked nothing out " - "there")) - return found + found.append((row, path, EXITED_0)) + elif stopper is not None: + found.append((row, path, STOPPED)) + return found, stopper def members_of(root: Path) -> tuple[str, list[dict]]: diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index 1e593d20..cee36ce5 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -64,9 +64,10 @@ asks them too. A path a row names in another case is still one of these, and what it says to do depends on which of the two is the member's own: it names no command that would un-register the member, and for a member - recorded only in the wrong case it names the index move that records it - at its row's path, as the row's own refusal does members checked - (#648). + recorded only in the wrong case, where its row's path is its canonical + `members/` and its pin a 40-hex commit, it names the index move + that records it at that path, as the row's own refusal does members + checked (#648). - that a section of HEAD's `.gitmodules` gives the path of every row's gitlink, the `member-gitmodules-absent` finding (#611, #407's decision 3). A recursive clone reads HEAD's file, so a gitlink no section of it diff --git a/tests/test_family_bootstrap_gate_order.py b/tests/test_family_bootstrap_gate_order.py index 9962f731..bbe04750 100644 --- a/tests/test_family_bootstrap_gate_order.py +++ b/tests/test_family_bootstrap_gate_order.py @@ -11,8 +11,13 @@ ok. In a plain clone of that holder, or of #363's state (a member's gitlink recorded only at `Members/

`), git dies `No url found for submodule path` (exit 128), and it still ended ok. It fails now, saying - which members no credential would check out and why, and names - `validate-family.py`. A missing credential is still the skip it was. + which members no credential would check out and why, and what cures + them. So it does where git's init dies at a gitlink no section gives a + url, a stray case variant, a member mounted by hand or a section that + lost its url (rv648's S1, H1 and U1), and where git exits 0 over a member + a subset clone or `update = none` leaves out, whose line prints the + update that checks it out. A missing credential is still the skip it + was, after a recursive clone too. - (b) In a plain clone of #363's state followed by `git submodule update --init --recursive`, as CI and `make bootstrap` run it, the update dies at `Members/

` before it checks out any member, so where that row is not @@ -26,11 +31,13 @@ THE SAME FAMILY, OFFLINE RULES AND STATES as `test_family_head_lost_gitmodules.py` (#611), whose helpers are imported. Every remedy is typed as printed. The case-variant states are index records -and commits, never two directories whose names differ only in case, and -the holder's own are asked again with `core.ignorecase=true`. A clone's -checkout is git's: on a disk that folds case, `Members/

` IS the row's -directory, so the `mv` the text gives for a case-sensitive disk is typed -only where the disk holds the two apart, as the text says. +and commits, and no test makes two directories whose names differ only in +case; the holder's own are asked again with `core.ignorecase=true`. A +clone's checkout is git's: on a disk that folds case, `Members/

` IS the +row's directory, so the `mv` the text gives for a case-sensitive disk is +typed only where the disk holds the two apart, as the text says, and a +clone of a stray (`Members/

` beside `members/

`, rv648's S1) gets its +two empty mounts there as one directory, which no assertion reads. """ from __future__ import annotations @@ -46,26 +53,45 @@ from conftest import clone_just_pushed, git, run_script from repo_shape import PYTHON -from test_family import NAME, family, validate # noqa: F401 +from test_family import NAME, SPARE, family, validate # noqa: F401 from test_family_add_refusals import holder, offline # noqa: F401 from test_family_dry_run import isolated from test_family_head_lost_gitmodules import ( - DISKS, assert_passes_in_every_mode, commit_all, head_lost, on_disk, - pinned, recorded_at, variant_refused, + DISKS, GITMODULES, assert_passes_in_every_mode, commit_all, + committed_gitmodules, head_lost, on_disk, pinned, recorded_at, + variant_refused, ) from test_family_member_path import MEMBER, OTHER, raw from test_validate_family_unnamed_mounts import ( - GITLINK_SAID, VARIANT, WRONG_CASE, commands, findings, stage_gitlink, + GITLINK_SAID, VARIANT, WRONG_CASE, commands, findings, spare_commits, + stage_gitlink, ) #: The line `(a)` prints for the members it left NOT CHECKED OUT where no -#: credential would check them out, and the one the run then ends on. +#: credential would check them out, each with why, then what cures them; +#: the run then ends on `failed_on`. NOT_FETCHED = (" NOT CHECKED OUT, and not for want of a credential: {left}. " - f"`{PYTHON} scripts/validate-family.py` names the cause and " - "the commit that cures it.") + "{cure}") NO_SECTION = "no section of .gitmodules gives that path" NO_GITLINK = "no gitlink is recorded at that path" EXITED_0 = "git exited 0 and checked nothing out there" +STOPPED = ("git stopped at {path}, a gitlink no section of .gitmodules " + "gives a url") + +#: What the line says cures each why: the validator for the first two; for +#: the gitlink git stopped at, the validator's `--no-members`, or, where a +#: row names that gitlink, its section written back; and for a member git +#: exited 0 over, the update that checks it out. +VALIDATOR_CURE = (f"`{PYTHON} scripts/validate-family.py` names the cause " + "and the commit that cures it.") +UNNAMED_CURE = (f"`{PYTHON} scripts/validate-family.py --no-members` names " + "each gitlink in members/ that no row names, and the commit " + "that cures it.") +SECTION_CURE = ("Write the section of .gitmodules for {path} back as " + "`family.py add` writes one, with its path and url, and " + "commit .gitmodules.") +CHECKOUT_CURE = ("`git submodule update --init --checkout --recursive -- " + "{path}` checks it out.") #: Each test pushes a branch of its own into the module's family remote. Not #: `tmp_path.name`: a passed test's directory is removed under @@ -130,10 +156,43 @@ def bootstrap(root: Path, env: dict, *extra: str): env=env) -def not_fetched(*left: tuple[str, str]) -> str: - """`NOT_FETCHED` naming each `(path, why)`.""" +def not_fetched(*left: tuple[str, str], cure: str = VALIDATOR_CURE) -> str: + """`NOT_FETCHED` naming each `(path, why)`, then `cure`.""" return NOT_FETCHED.format( - left=", ".join(f"{path} ({why})" for path, why in left)) + left=", ".join(f"{path} ({why})" for path, why in left), cure=cure) + + +def stray(holder: Path, env: dict, family) -> str: + """#361's STRAY: OTHER's gitlink recorded at `Members/

` too, at its + pin, beside its own, and committed; no section gives the second. Index + order puts it before every member's. Returns its path.""" + stage_gitlink(holder, f"Members/{OTHER}", pinned(holder, OTHER)) + commit_all(holder, env, "A stray case variant of OTHER's gitlink") + return f"Members/{OTHER}" + + +def hand_mounted(holder: Path, env: dict, family) -> str: + """SPARE's gitlink recorded at `members/` by hand, with no row + and no section, and committed. Returns its path.""" + stage_gitlink(holder, f"members/{SPARE}", spare_commits(family)[1]) + commit_all(holder, env, "A member mounted by hand") + return f"members/{SPARE}" + + +#: The two gitlinks no row names at which git's init dies in a plain clone +#: (rv648's S1 and H1). +UNNAMED_STOPS = {"stray": stray, "hand-mount": hand_mounted} + + +def without_url(data: bytes, project: str) -> bytes: + """`.gitmodules` bytes with the `url` line of `project`'s section taken + out, the rest as `family.py add` wrote it (rv648's U1). A checkout's + copy may end its lines CRLF.""" + section = re.compile(rb'(\[submodule "members/' + project.encode() + + rb'"\]\r?\n(?:\t[^\n]*\n)*?)\turl = [^\n]*\n') + edited, count = section.subn(rb"\1", data) + assert count == 1, data + return edited def failed_on(*projects: str) -> str: @@ -182,34 +241,42 @@ def moved_out_where_apart(root: Path, remediation: str) -> None: # --- (a) bootstrap ----------------------------------------------------------- -#: `(set-up, flow, the members (a) names with why, the rows it fails on)`. +#: `(set-up, flow, the members (a) names with why, the rows it fails on, +#: what it says cures them)`. In #363's plain clone git dies at the variant, +#: `Members/

`, before it checks out MEMBER, so MEMBER is named too, and +#: the variant is a gitlink no row names (rv648's M1). UNFETCHED = { "head-lost-recursive": ( head_lost, True, ((f"members/{MEMBER}", NO_SECTION), (f"members/{OTHER}", NO_SECTION)), - (MEMBER, OTHER)), + (MEMBER, OTHER), VALIDATOR_CURE), "head-lost-plain": ( head_lost, False, ((f"members/{MEMBER}", NO_SECTION), (f"members/{OTHER}", NO_SECTION)), - (MEMBER, OTHER)), + (MEMBER, OTHER), VALIDATOR_CURE), "variant-recursive": ( - variant, True, ((f"members/{OTHER}", NO_GITLINK),), (OTHER,)), + variant, True, ((f"members/{OTHER}", NO_GITLINK),), (OTHER,), + VALIDATOR_CURE), "variant-plain": ( - variant, False, ((f"members/{OTHER}", NO_GITLINK),), (OTHER,)), + variant, False, + ((f"members/{MEMBER}", STOPPED.format(path=f"Members/{OTHER}")), + (f"members/{OTHER}", NO_GITLINK)), + (MEMBER, OTHER), f"{VALIDATOR_CURE} {UNNAMED_CURE}"), } -@pytest.mark.parametrize("set_up, recurse, left, projects", +@pytest.mark.parametrize("set_up, recurse, left, projects, cure", UNFETCHED.values(), ids=UNFETCHED) def test_bootstrap_fails_over_members_no_credential_would_check_out( - family, holder, offline, tmp_path, set_up, recurse, left, projects): + family, holder, offline, tmp_path, set_up, recurse, left, projects, + cure): """THE ISSUE'S (a), both of rv611b's states, after a recursive clone and after a plain one. Git exits 0 over a holder whose HEAD lost `.gitmodules` where the clone was recursive, and 128 (`No url found for submodule path`) where it was plain, and 128 over #363's state in a plain clone: the run said ok each time. It names the members no - credential would check out, says why, names `validate-family.py`, and - fails; it never says every member was fetched.""" + credential would check out, says why and what cures them, and fails; + it never says every member was fetched.""" env = isolated(offline, tmp_path) set_up(holder, env) clone = cloned(holder, env, family, tmp_path, @@ -220,8 +287,8 @@ def test_bootstrap_fails_over_members_no_credential_would_check_out( assert result.returncode == 1, result.stdout + result.stderr assert after_first_line(result) == NOT_ALL_FETCHED errors = result.stderr.splitlines() - assert errors[-2:] == [not_fetched(*left), failed_on(*projects)], ( - result.stderr) + assert errors[-2:] == [not_fetched(*left, cure=cure), + failed_on(*projects)], result.stderr if not recurse: assert errors[0].startswith(" FETCH INCOMPLETE (git exit 128) "), ( result.stderr) @@ -259,31 +326,191 @@ def test_a_missing_credential_is_still_the_skip_it_was(family, holder, assert result.returncode == 0, result.stdout + result.stderr assert "FETCH INCOMPLETE" in result.stderr assert "not for want of a credential" not in result.stderr + assert (" every member and leg fetched" + not in result.stdout.splitlines()), result.stdout for project in (MEMBER, OTHER): assert (f" [{project}] {project}: NOT CHECKED OUT; skipped. See the " "fetch above.") in result.stdout.splitlines() assert result.stdout.endswith("\nfamily bootstrap ok\n"), result.stdout +def skip_by_update_none(clone: Path, env: dict) -> dict: + """`submodule..update=none` for OTHER, set for every git the run + starts, as a `.gitmodules` committed with `update = none` sets it.""" + return with_config(env, f"submodule.members/{OTHER}.update", "none") + + +def skip_by_subset(clone: Path, env: dict) -> dict: + """`submodule.active` in the clone's own git config naming MEMBER alone, + as `git clone --recurse-submodules=members/

` stores it.""" + raw(clone, env, "config", "submodule.active", f"members/{MEMBER}") + return env + + +@pytest.mark.parametrize("skip", [skip_by_update_none, skip_by_subset], + ids=["update-none", "subset"]) def test_an_update_that_exits_0_and_skips_a_member_fails(family, holder, - offline, tmp_path): + offline, tmp_path, + skip): """Git exited 0, so it lacked nothing a credential gives, though the - holder's records give OTHER: here `submodule..update=none` set - for the run skips it. (a) does not say every member was fetched, and - the run fails on OTHER alone.""" - env = isolated(offline, tmp_path) + holder's records give OTHER: an `update = none`, or a clone that chose + a subset of members, skips it (rv648's N1 and P1). (a) does not say + every member was fetched, and the run fails on OTHER alone, printing + the update that checks it out: `--init` with its path registers it, and + `--checkout` overrides the `none`. `git submodule update --init + --recursive`, `member-uninitialized`'s remedy, exits 0 there and checks + nothing out. Typed as printed, the line lets the same run end ok.""" + env = fetching(isolated(offline, tmp_path)) clone = cloned(holder, env, family, tmp_path, recurse=False) - skip = with_config(fetching(env), f"submodule.members/{OTHER}.update", - "none") + env = skip(clone, env) - result = bootstrap(clone, skip, "--skip-members") + result = bootstrap(clone, env, "--skip-members") assert result.returncode == 1, result.stdout + result.stderr assert (clone / "members" / MEMBER / "project.yaml").is_file() assert "every member and leg fetched" not in result.stdout assert result.stderr.splitlines() == [ - not_fetched((f"members/{OTHER}", EXITED_0)), failed_on(OTHER)] + not_fetched((f"members/{OTHER}", EXITED_0), + cure=CHECKOUT_CURE.format(path=f"members/{OTHER}")), + failed_on(OTHER)] + raw(clone, env, "submodule", "update", "--init", "--recursive") + assert not (clone / "members" / OTHER / ".git").exists(), ( + "premise: the validator's remedy checks nothing out here") + (line,) = re.findall(r"`(git submodule update [^`]*)`", result.stderr) + type_line(clone, env, line) + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + assert again.stderr == "" + + +@pytest.mark.parametrize("set_up", UNNAMED_STOPS.values(), ids=UNNAMED_STOPS) +def test_a_gitlink_no_row_names_that_stops_a_plain_clones_update_fails( + family, holder, offline, tmp_path, set_up): + """rv648's S1 and H1. In a plain clone of a holder that records a stray + case variant of a member's gitlink, or a member mounted by hand, git's + init dies at that gitlink (`No url found for submodule path`, exit 128) + before it checks out any member, though every row's gitlink and section + are there: the run said ok. It names both rows, says where git stopped, + and names `--no-members`, which names that gitlink and the commit that + cures it. Typed as printed (all but the hand mount's `family.py add`, + which mounts it again) and committed, the same run checks both members + out.""" + env = fetching(isolated(offline, tmp_path)) + stopper = set_up(holder, env, family) + commit = raw(holder, env, "rev-parse", + f"HEAD:{stopper}").stdout.decode("ascii").strip() + clone = cloned(holder, env, family, tmp_path, recurse=False) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + why = STOPPED.format(path=stopper) + assert result.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{MEMBER}", why), (f"members/{OTHER}", why), + cure=UNNAMED_CURE), + failed_on(MEMBER, OTHER)], result.stderr + assert f"No url found for submodule path '{stopper}'" in result.stderr + fallback = validate(clone, "--no-members") + assert fallback.returncode == 1, fallback.stdout + fallback.stderr + (finding,) = findings(fallback) + assert finding.startswith(GITLINK_SAID.format(path=stopper, + commit=commit)), finding + for line in commands(finding): + if not line.startswith(f"{PYTHON} "): + type_line(clone, env, line) + commit_all(clone, env, "Take the gitlink out as printed") + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + + +def test_a_section_with_no_url_that_stops_the_update_says_to_write_it_back( + family, holder, offline, tmp_path): + """rv648's U1. OTHER's section lost its url, committed: in a plain + clone git's init dies at OTHER's own gitlink, after MEMBER's and before + it checks out either. The run said ok, and no mode of the validator + names it. It names both rows and says to write that section back; + written back as `family.py add` wrote it and committed, the same run + checks both members out.""" + env = fetching(isolated(offline, tmp_path)) + data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout + committed_gitmodules(holder, env, without_url(data, OTHER)) + clone = cloned(holder, env, family, tmp_path, recurse=False) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + why = STOPPED.format(path=f"members/{OTHER}") + assert result.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{MEMBER}", why), (f"members/{OTHER}", why), + cure=SECTION_CURE.format(path=f"members/{OTHER}")), + failed_on(MEMBER, OTHER)], result.stderr + assert (f"No url found for submodule path 'members/{OTHER}'" + in result.stderr) + (clone / GITMODULES).write_bytes(data) + raw(clone, env, "add", "--", GITMODULES) + commit_all(clone, env, "OTHER's section written back") + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + + +@pytest.mark.parametrize("set_up", UNNAMED_STOPS.values(), ids=UNNAMED_STOPS) +def test_a_recursive_clone_with_no_credential_is_still_the_skip( + family, holder, offline, tmp_path, set_up): + """No misfire. A recursive clone sets `submodule.active` (`.`), and + git's init then asks only the gitlinks a section gives a path, so the + stray or the hand mount stops nothing there, and git fails on the + members' own fetch, as a private member with no token fails. The + members are the credential's: skipped, and the run ends ok.""" + env = isolated(offline, tmp_path) + set_up(holder, env, family) + remote = family["remotes"] / f"{NAME}.git" + branch = f"w648/{next(BRANCHES)}" + raw(holder, env, "push", "-q", str(remote), f"HEAD:refs/heads/{branch}") + clone = tmp_path / "clone" + # The holder is fetched, as a person with access to it fetches it, and + # its members are not: `user` refuses the file transport to the clones + # git makes for submodules. + made = clone_just_pushed(remote, clone, branch, + config=["-c", "protocol.file.allow=user"]) + assert made.returncode != 0, "premise: the members were not fetched" + assert "transport 'file' not allowed" in made.stderr, made.stderr + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 0, result.stdout + result.stderr + assert "FETCH INCOMPLETE" in result.stderr + assert "not for want of a credential" not in result.stderr + + +def test_a_url_an_earlier_init_registered_is_still_the_skip(family, holder, + offline, + tmp_path): + """No misfire. `git submodule init` wrote each member's url into the + clone's git config, then the work tree's section for OTHER lost its + url: git's init reads the url it wrote and does not stop there, and git + fails on the fetch for want of a credential. The members are skipped and + the run ends ok.""" + env = isolated(offline, tmp_path) + clone = cloned(holder, env, family, tmp_path, recurse=False) + raw(clone, env, "submodule", "init") + data = (clone / GITMODULES).read_bytes() + (clone / GITMODULES).write_bytes(without_url(data, OTHER)) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 0, result.stdout + result.stderr + assert "FETCH INCOMPLETE" in result.stderr + assert "not for want of a credential" not in result.stderr def test_a_row_written_twice_is_named_once(family, holder, offline, @@ -314,7 +541,8 @@ def test_a_plain_clone_of_a_case_variant_is_named_and_mended_as_printed( family, holder, offline, tmp_path): """The round trip a person makes. In a plain clone of #363's state, with the variant at the SECOND row, `make bootstrap`'s script fails naming - OTHER and the validator; the validator, members checked, names the + both rows, since git stopped at the variant before it checked out + MEMBER, and the validator; the validator, members checked, names the variant and the move, where it refused `member-uninitialized` for the first row before. Typed as printed in the clone, with the commit and the `mv` where the disk holds the two apart, the same script checks @@ -328,7 +556,11 @@ def test_a_plain_clone_of_a_case_variant_is_named_and_mended_as_printed( assert booted.returncode == 1, booted.stdout + booted.stderr assert booted.stderr.splitlines()[-2:] == [ - not_fetched((f"members/{OTHER}", NO_GITLINK)), failed_on(OTHER)] + not_fetched((f"members/{MEMBER}", + STOPPED.format(path=f"Members/{OTHER}")), + (f"members/{OTHER}", NO_GITLINK), + cure=f"{VALIDATOR_CURE} {UNNAMED_CURE}"), + failed_on(MEMBER, OTHER)] gate = validate(clone) assert gate.returncode == 2, gate.stdout + gate.stderr lines = gate.stderr.splitlines() From dbca11304a606fff004b165dea8ad28e2cc67dfc Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 19:23:38 +0000 Subject: [PATCH 09/15] validate-family.py: after `git rm --cached` of a hand mount the staged-removal text says to delete the checkout before `family.py add`, and the variant move's text says what a clone's own `.git` holds, where a submodule checkout's store is, and "in a POSIX shell", so each step typed as printed reaches a passing gate (#681, rv681) The fix round for rv681's review of d91fd15e (0 MUST, 2 SHOULD, 3 NIT). Text only, in the pinned holder copy; no guard changes. - S1. A hand mount taken out with `git rm --cached` (as git suggests when the printed `git rm` refuses a checkout with changes) leaves its checkout at the path, and the printed `family.py add` refused it (`member-path-taken`). The staged branch of `_mounted_by_hand` now says to delete it by hand, once nothing only it holds is needed, before the `add`. - S2, Direction B. The `-e` guard stays, as the issue's "no .git entry" that Brett Heap's 2026-10-16T14:11:42Z word took. For a `.git` directory, a clone of its own, the text says its commits are there and nowhere else, names `git -C log --oneline HEAD --branches --not --remotes` for the ones not pushed, and says to move it out whole rather than delete it. - N1. A submodule checkout's commits are "kept under this holder's .git/modules/, in the directory its .git file names", not always `.git/modules/`. The status check no longer promises to work where a plain `mv` brought the checkout to the variant. - N2. The guarded line's sentence says "in a POSIX shell". - N3. A test kills rv681's mutant MO1: the section written back in the work tree alone still prints its lines. Tests: the hand-mount L20 test gains a `git rm --cached` case typed to `add` exiting 0; the checkout-at-the-variant test keeps the clone, types its printed `git log`, moves it out whole, and gains a plain-`mv` kind; a new test for N3; the pinned variant text follows the new words. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- .../family-root/scripts/validate-family.py | 47 +++-- tests/test_family_head_lost_gitmodules.py | 25 ++- tests/test_validate_family_remedies_typed.py | 183 ++++++++++++++---- 3 files changed, 189 insertions(+), 66 deletions(-) diff --git a/templates/family-root/scripts/validate-family.py b/templates/family-root/scripts/validate-family.py index b3623078..0e579d99 100755 --- a/templates/family-root/scripts/validate-family.py +++ b/templates/family-root/scripts/validate-family.py @@ -564,8 +564,14 @@ def _wrong_case_move(path: str, commit: str, variants: list[str]) -> str: pointing nowhere (`fatal: not a git repository`). So the `mv` runs only for a directory apart from the row's with no `.git` entry, the empty one a fresh clone leaves or a plain one, and the text says where a - submodule checkout's commits are kept, what deleting it would lose, and - to delete a checkout by hand once nothing it holds is needed. + submodule checkout's commits are kept, in the store its `.git` file + names (not always `.git/modules/`: a checkout a plain `mv` + brought there names the old one, and git cannot open it), what deleting + it would lose, and to delete it by hand once nothing only it holds is + needed. A clone's own `.git` DIRECTORY holds its commits and nothing + else does, so the text names the `git log` that lists the ones not + pushed and says to move it out whole, not delete it (rv681's S2 and + N1, Direction B: the `-e` guard Brett Heap's word took is kept). """ taken_out = " ".join(shlex.quote(variant) for variant in variants) moved = " and ".join( @@ -574,9 +580,11 @@ def _wrong_case_move(path: str, commit: str, variants: list[str]) -> str: f"mv -- {shlex.quote(variant)} " f"{shlex.quote('../' + variant.replace('/', '-'))}`" for variant in variants) - stores = " or ".join(f".git/modules/{variant}" for variant in variants) status = " or ".join(f"`git -C {shlex.quote(variant)} status`" for variant in variants) + unpushed = " or ".join(f"`git -C {shlex.quote(variant)} log --oneline " + "HEAD --branches --not --remotes`" + for variant in variants) return ( "Move its gitlink to the row's path, at the commit the row pins, in " "the index alone, which touches no file, with the two lines below, " @@ -588,14 +596,20 @@ def _wrong_case_move(path: str, commit: str, variants: list[str]) -> str: "(WSL's /mnt/c). Where the disk holds the two apart it stays there, " "untracked, and family.py add refuses it as " "family-members-dir-variant: move each one that is there out of the " - f"holder, {moved}, which moves nothing where the two names are one " - "directory or where it holds a .git entry, a checkout. A checkout " - "git submodule add made there holds a .git file naming this " - "holder's .git/modules/ by a path relative to it, which a move would " - f"leave pointing nowhere; its commits are kept in {stores}, whatever " - "becomes of the directory, and only what is not committed is in the " - f"directory alone ({status} shows it). Delete a checkout by hand " - "once nothing it holds is needed. The two lines:\n" + f"holder, in a POSIX shell, {moved}, which moves nothing where the " + "two names are one directory or where it holds a .git entry, a " + "checkout. A checkout git submodule add made holds a .git file " + "naming its store by a path relative to it, which a move would " + "leave pointing nowhere. Its commits are kept under this holder's " + ".git/modules/, in the directory its .git file names, whatever " + "becomes of the checkout, and only what is not committed is in the " + f"checkout alone ({status} shows it, except where a plain mv brought " + "the checkout to that path: git cannot open it there, so check its " + "files by hand). Delete such a checkout by hand once nothing only it " + "holds is needed. A checkout whose .git is a directory, a clone of " + "its own, holds its commits there and nowhere else, and " + f"{unpushed} lists the ones not pushed: move it out of the holder " + "whole, by hand, rather than delete it. The two lines:\n" f" git rm -r --cached --ignore-unmatch -- {taken_out}\n" " git update-index --add --cacheinfo " f"{shlex.quote(f'160000,{commit},{path}')}") @@ -1467,7 +1481,9 @@ def _mounted_by_hand(root: Path, path: str, project: str | None, """What a gitlink no row names, in no case, says to do after what it is: put its row back, or take a hand mount out, commit that, and mount it with `family.py add`, on a branch (`_check_gitlinks_named`). Where its - removal is staged already (#681), the same with no `git rm`.""" + removal is staged already (#681), the same with no `git rm`, and the + checkout `git rm --cached` leaves at the path, which `family.py add` + refuses (`member-path-taken`), deleted by hand first (rv681's S1).""" member = (PROJECT_PLACEHOLDER if project is None else shlex.quote(project)) add = (f" {PYTHON} scripts/family.py add --family-root " @@ -1479,8 +1495,11 @@ def _mounted_by_hand(root: Path, path: str, project: str | None, "HEAD holds and the index no longer does, and on a branch, since " "family.py add refuses the holder's tracking branch: where the " "holder stands on it, the first command below makes one, taking " - "the staged removal with it. If it was mounted by hand, then " - "mount it with the second, from a checkout of " + "the staged removal with it. Where git rm --cached took it out, " + f"its checkout is still at {path}, and family.py add refuses that " + "directory (member-path-taken): delete it by hand, once nothing " + "only it holds is needed, before the add. If it was mounted by " + "hand, then mount it with the second, from a checkout of " "openRepoShape: family.py add writes its row and its gitlink in " "one commit, and names the object store git rm leaves under " f".git/modules if one is still there. {BRANCH_LEFT}\n" diff --git a/tests/test_family_head_lost_gitmodules.py b/tests/test_family_head_lost_gitmodules.py index 6916e98e..2aff6080 100644 --- a/tests/test_family_head_lost_gitmodules.py +++ b/tests/test_family_head_lost_gitmodules.py @@ -668,8 +668,9 @@ def variant_refused(path: str, commit: str, variants: list[str], moved = " and ".join(f"`[ {variant} -ef {path} ] || [ -e {variant}/.git ] " f"|| mv -- {variant} ../{variant.replace('/', '-')}`" for variant in variants) - stores = " or ".join(f".git/modules/{variant}" for variant in variants) status = " or ".join(f"`git -C {variant} status`" for variant in variants) + unpushed = " or ".join(f"`git -C {variant} log --oneline HEAD --branches " + "--not --remotes`" for variant in variants) return [ f"REFUSED member-gitlink-absent: no gitlink (mode 160000) recorded at " f"'{path}'. {project} is named in family.yaml, and this repository " @@ -686,14 +687,20 @@ def variant_refused(path: str, commit: str, variants: list[str], "usually do and a Linux mount can (WSL's /mnt/c). Where the disk " "holds the two apart it stays there, untracked, and family.py add " "refuses it as family-members-dir-variant: move each one that is " - f"there out of the holder, {moved}, which moves nothing where the " - "two names are one directory or where it holds a .git entry, a " - "checkout. A checkout git submodule add made there holds a .git " - "file naming this holder's .git/modules/ by a path relative to it, " - "which a move would leave pointing nowhere; its commits are kept in " - f"{stores}, whatever becomes of the directory, and only what is not " - f"committed is in the directory alone ({status} shows it). Delete a " - "checkout by hand once nothing it holds is needed. The two lines:", + f"there out of the holder, in a POSIX shell, {moved}, which moves " + "nothing where the two names are one directory or where it holds a " + ".git entry, a checkout. A checkout git submodule add made holds a " + ".git file naming its store by a path relative to it, which a move " + "would leave pointing nowhere. Its commits are kept under this " + "holder's .git/modules/, in the directory its .git file names, " + "whatever becomes of the checkout, and only what is not committed is " + f"in the checkout alone ({status} shows it, except where a plain mv " + "brought the checkout to that path: git cannot open it there, so " + "check its files by hand). Delete such a checkout by hand once " + "nothing only it holds is needed. A checkout whose .git is a " + "directory, a clone of its own, holds its commits there and nowhere " + f"else, and {unpushed} lists the ones not pushed: move it out of the " + "holder whole, by hand, rather than delete it. The two lines:", " git rm -r --cached --ignore-unmatch -- " + " ".join(variants), f" git update-index --add --cacheinfo 160000,{commit},{path}"] diff --git a/tests/test_validate_family_remedies_typed.py b/tests/test_validate_family_remedies_typed.py index 947a5d77..87582356 100644 --- a/tests/test_validate_family_remedies_typed.py +++ b/tests/test_validate_family_remedies_typed.py @@ -40,6 +40,7 @@ from __future__ import annotations import os +import re import shlex from pathlib import Path @@ -76,13 +77,17 @@ "removal is staged and not committed, and no git rm is printed, " "since one fails there (pathspec ... did not match any files).") -#: What a hand mount's says after it, in place of the `git rm` step. +#: What a hand mount's says after it, in place of the `git rm` step: with +#: the checkout `git rm --cached` leaves deleted by hand first (rv681's S1). STAGED_HAND_MOUNT = ( f" {STAGED} Commit that removal before family.py add, which refuses a " "path under another spelling of members/ that HEAD holds and the index " "no longer does, and on a branch, since family.py add refuses the " "holder's tracking branch: where the holder stands on it, the first " - "command below makes one, taking the staged removal with it. If it was " + "command below makes one, taking the staged removal with it. Where git " + f"rm --cached took it out, its checkout is still at {HAND}, and " + "family.py add refuses that directory (member-path-taken): delete it by " + "hand, once nothing only it holds is needed, before the add. If it was " "mounted by hand, then mount it with the second, from a checkout of " "openRepoShape: family.py add writes its row and its gitlink in one " "commit, and names the object store git rm leaves under .git/modules if " @@ -105,15 +110,21 @@ def added_as_printed(family, root: Path, env: dict, add: str): # --- L20: a removal staged and not committed -------------------------------- +@pytest.mark.parametrize("how", ["git-rm", "git-rm-cached"]) def test_a_hand_mounts_staged_removal_is_committed_and_added_as_printed( - family, holder, offline, tmp_path): - """THE ISSUE'S L20, from a real hand mount. Its printed branch and `git - rm` typed, the re-run before the commit names the gitlink again, as - HEAD still records it, says the removal is staged, and prints the - branch and the `add` with no `git rm`, which typed again fails, as - before the change. Committed as it says, and the `add` typed as - printed (after the object store it names, as before), the holder passes - in every mode with the member mounted.""" + family, holder, offline, tmp_path, how): + """THE ISSUE'S L20, from a real hand mount, taken out with the printed + `git rm`, or with `git rm --cached`, which git itself suggests when + that `git rm` refuses a checkout holding changes (rv681's S1, its probe + P1). The re-run before the commit names the gitlink again, as HEAD + still records it, says the removal is staged, and prints the branch + and the `add` with no `git rm`; the printed one, typed again, fails, as + before the change. After `--cached` the `.gitmodules` section is left, + and its own finding's lines take it out. Committed as it says, the + checkout `--cached` leaves deleted by hand where `add` refuses it + (`member-path-taken`), as the text says, and the `add` typed as printed + (after the object store it names, as before), the holder passes in + every mode with the member mounted.""" env = fetching(isolated(offline, tmp_path)) raw(holder, env, "submodule", "add", "-q", "--", str(family["remotes"] / f"{SPARE}.git"), HAND) @@ -122,18 +133,32 @@ def test_a_hand_mounts_staged_removal_is_committed_and_added_as_printed( gitlink, _ = findings(validate(holder, "--no-members")) switch, rm, add = commands(gitlink) raw(holder.parent, env, *shlex.split(switch)[1:]) - raw(holder, env, *shlex.split(rm)[1:]) + if how == "git-rm": + raw(holder, env, *shlex.split(rm)[1:]) + else: + raw(holder, env, "rm", "-q", "--cached", "--", HAND) again = validate(holder, "--no-members") assert again.returncode == 1, again.stdout + again.stderr - assert findings(again) == [ - GITLINK_SAID.format(path=HAND, commit=at) + STAGED_HAND_MOUNT - + f"\n {switch}\n {add}"] + staged, *section = findings(again) + assert staged == (GITLINK_SAID.format(path=HAND, commit=at) + + STAGED_HAND_MOUNT + f"\n {switch}\n {add}") retyped = raw(holder, env, *shlex.split(rm)[1:], check=False) assert retyped.returncode == 128, retyped.stderr assert b"did not match any files" in retyped.stderr - commit_all(holder, env, "Take the hand mount out, as printed") + if how == "git-rm": + assert section == [] + commit_all(holder, env, "Take the hand mount out, as printed") + else: + (left,) = section + assert commands(left) == removal(HAND, "path", "url") + take_out_as_printed(holder, left) + assert (holder / HAND / ".git").is_file(), "premise: checkout left" + taken = added_as_printed(family, holder, env, add) + assert taken.returncode == 2, taken.stdout + taken.stderr + assert "REFUSED member-path-taken: " in taken.stderr + rmtree(holder / HAND) refused = added_as_printed(family, holder, env, add) assert refused.returncode == 2, refused.stdout + refused.stderr assert "REFUSED member-git-dir-cached: " in refused.stderr @@ -323,6 +348,31 @@ def test_a_row_whose_gitlink_was_taken_out_is_put_back_as_printed( assert_passes_in_every_mode(holder, 2) +def test_a_section_back_in_the_work_tree_alone_still_prints_its_lines( + holder, offline, tmp_path): + """rv681's N3 (its mutant MO1): the section `git rm` took out written + back in the work tree's `.gitmodules` and not committed. HEAD's gives + the path no section, and HEAD is what a clone gets, so the refusal + still prints the section lines; typed as printed over the file, staged + and committed, `git submodule update --init --recursive` checks the + member out, and the holder passes in every mode.""" + env = fetching(isolated(offline, tmp_path)) + commit = taken_out(holder, env, "git-rm") + (holder / ".gitmodules").write_bytes( + raw(holder, env, "show", "HEAD~1:.gitmodules").stdout) + assert raw(holder, env, "diff", "--quiet", "--", ".gitmodules", + check=False).returncode == 1, "premise: the work tree has it" + + refused = validate(holder) + + assert refused.returncode == 2, refused.stdout + refused.stderr + lines = refused.stderr.splitlines() + assert lines == put_back(commit, section=True) + put_back_as_printed(holder, env, lines) + raw(holder, env, "submodule", "update", "--init", "--recursive") + assert_passes_in_every_mode(holder, 2) + + def test_a_fresh_clone_of_it_is_put_back_and_checks_the_member_out( family, holder, offline, tmp_path): """The same state pushed and cloned, as CI checks it out: no object @@ -440,29 +490,59 @@ def mounted_at_the_variant(family, root: Path, env: dict, commit: str, commit_all(root, env, "IRRS recorded at Members/IRRS, checked out there") -@pytest.mark.parametrize("kind", ["submodule", "clone"]) -def test_a_checkout_at_the_variant_is_left_in_place_and_deleted_by_hand( +#: The command the variant refusal names for a clone's commits not pushed. +UNPUSHED = re.compile(r"`(git -C \S+ log --oneline HEAD --branches " + r"--not --remotes)`") + + +@pytest.mark.parametrize("kind", ["submodule", "clone", "moved"]) +def test_a_checkout_at_the_variant_is_kept_and_handled_as_the_text_says( family, holder, offline, tmp_path, case_sensitive_disk, kind): - """THE ISSUE'S N-1. A checkout at `Members/IRRS` holding a commit of - its own: after the two lines and their commit, the guarded line moves - nothing, where the bare `mv` left a submodule's checkout saying `fatal: - not a git repository`. It is still a working checkout, and a - submodule's commits are where the text says, in - `.git/modules/Members/IRRS`, and stay there once the directory is - deleted by hand, as the text says to do when nothing it holds is - needed. Then the holder passes in every mode, and SPARE is added past - it, where `family.py add` refused the directory.""" + """THE ISSUE'S N-1, and rv681's S2 and N1. A checkout at `Members/IRRS` + holding a commit of its own: after the two lines and their commit, the + guarded line moves nothing, where the bare `mv` left a submodule's + checkout saying `fatal: not a git repository`. Then each kind as the + text says: + + - `submodule`, a `git submodule add` there: still a working checkout, + its commits in the store its `.git` file names, + `.git/modules/Members/IRRS`, where they stay once it is deleted by + hand. + - `clone`, its own `.git` directory, the only place its commit is: the + printed `git log` lists that commit, and the clone is moved out of + the holder whole, by hand, with the commit readable there. + - `moved`, the row's own checkout moved there by a plain `mv` (#363's + state reached on Linux): its `.git` file still names + `.git/modules/members/IRRS`, where its commit is, and git cannot open + it there, as the text says. Deleted by hand, the update checks the + member out at its row's path again. + + Then the holder passes in every mode, and SPARE is added past it, where + `family.py add` refused the directory.""" env = fetching(isolated(offline, tmp_path)) commit = pinned(holder, MEMBER) - mounted_at_the_variant(family, holder, env, commit, kind) checkout = holder / VARIANT_PATH - raw(checkout, env, "commit", "-q", "--allow-empty", "-m", "Work there") - work = raw(checkout, env, "rev-parse", "HEAD").stdout.decode( - "ascii").strip() - store = holder / ".git" / "modules" / "Members" / MEMBER - if kind == "submodule": - assert (checkout / ".git").read_bytes() == ( - b"gitdir: ../../.git/modules/Members/" + MEMBER.encode() + b"\n") + if kind == "moved": + own = holder / "members" / MEMBER + raw(own, env, "commit", "-q", "--allow-empty", "-m", "Work there") + work = raw(own, env, "rev-parse", "HEAD").stdout.decode().strip() + (holder / "Members").mkdir() + os.rename(own, checkout) + stage_gitlink(holder, VARIANT_PATH, commit) + raw(holder, env, "rm", "-q", "--cached", "--", f"members/{MEMBER}") + commit_all(holder, env, "IRRS moved to Members/IRRS by a plain mv") + store = holder / ".git" / "modules" / "members" / MEMBER + else: + mounted_at_the_variant(family, holder, env, commit, kind) + raw(checkout, env, "commit", "-q", "--allow-empty", "-m", + "Work there") + work = raw(checkout, env, "rev-parse", "HEAD").stdout.decode().strip() + store = holder / ".git" / "modules" / "Members" / MEMBER + if kind != "clone": + named = (checkout / ".git").read_text(encoding="utf-8") + assert named.startswith("gitdir: "), named + assert (checkout / named[len("gitdir: "):].strip()).resolve() == ( + store.resolve()) lines = refused_at(holder, commit, [VARIANT_PATH]) moved_as_printed(holder, env, lines, [VARIANT_PATH]) (move,) = GUARDED_MOVE.findall(lines[1]) @@ -471,18 +551,35 @@ def test_a_checkout_at_the_variant_is_left_in_place_and_deleted_by_hand( assert typed.returncode == 0, typed.stderr assert not (tmp_path / MOVED_TO).exists() - assert raw(checkout, env, "status", "--porcelain").returncode == 0 - if kind == "submodule": - assert Path(raw(checkout, env, "rev-parse", "--absolute-git-dir") - .stdout.decode().strip()).resolve() == store.resolve() - found = validate(holder) - assert codes(found) == ["gitmodules-path-unnamed"], found.stderr - take_out_as_printed(holder, findings(found)[0]) - rmtree(checkout) - if kind == "submodule": + assert (checkout / ".git").exists() + status = raw(checkout, env, "status", "--porcelain", check=False) + assert (status.returncode == 0) is (kind != "moved"), status.stderr + if kind == "clone": + (log,) = UNPUSHED.findall(lines[1]) + listed = raw(holder, env, *shlex.split(log)[1:]).stdout.decode() + assert [line.split(" ", 1)[1] for line in listed.splitlines()] == [ + "Work there"], listed + assert work.startswith(listed.split(" ", 1)[0]) + assert ("move it out of the holder whole, by hand, rather than " + "delete it") in lines[1] + kept = tmp_path / f"{MEMBER}-clone" + os.rename(checkout, kept) + assert raw(kept, env, "cat-file", "-e", work, + check=False).returncode == 0 + else: + if kind == "submodule": + found = validate(holder) + assert codes(found) == ["gitmodules-path-unnamed"], found.stderr + take_out_as_printed(holder, findings(found)[0]) + else: + assert not (holder / ".git" / "modules" / "Members").exists() + rmtree(checkout) kept = raw(holder, env, f"--git-dir={store}", f"--work-tree={holder}", "cat-file", "-e", work, check=False) assert kept.returncode == 0, kept.stderr + if kind == "moved": + raw(holder, env, "submodule", "update", "--init", "--recursive") + assert (holder / "members" / MEMBER / "project.yaml").is_file() assert_passes_in_every_mode(holder, 2) added = run_script(FAMILY, *verb_args("add", holder, family), env=env) assert added.returncode == 0, added.stdout + added.stderr From 45c512eb797b6e6a8d614e34bb5750f5ccba5221 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Fri, 16 Oct 2026 19:43:52 +0000 Subject: [PATCH 10/15] tests/test_doctor_legs_members_next.py: the bootstrap that cures nothing in #414's url-less member state now fails, as 1a69505d makes it, so the test pins exit 1 and `git stopped at members/Ghost` where it pinned the old `ok` (#648, rv648's U1) `test_414_3_there_the_bootstrap_cures_nothing_and_the_git_line_cures_one` builds rv648's U1 state: a member, Ghost, whose `.gitmodules` section gives its path and no url. Git's init stops the holder's update there, before it checks out the other member. The test asserted bootstrap's exit 0, the false `ok` rv648's M1 asked to end. It asserts exit 1 now, and the line naming where git stopped. Its point is unchanged: the bootstrap cures nothing, and the doctor's git line cures the other member. Found by the targeted run of every test file that drives the family bootstrap (1 failed, 3797 passed, 32 skipped); the file now passes alone, 124 passed. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- tests/test_doctor_legs_members_next.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/test_doctor_legs_members_next.py b/tests/test_doctor_legs_members_next.py index 8c908562..b6429df0 100644 --- a/tests/test_doctor_legs_members_next.py +++ b/tests/test_doctor_legs_members_next.py @@ -1584,7 +1584,10 @@ def test_414_3_there_the_bootstrap_cures_nothing_and_the_git_line_cures_one( f"{module.quote_arg(tried / 'scripts' / 'bootstrap.py')} " f"--root {module.quote_arg(tried)}", tmp_path / "boot") output = bootstrap.stdout + bootstrap.stderr - assert bootstrap.returncode == 0, output + # It cures nothing, and since #648 it says so and fails: git stopped + # at Ghost, whose section gives no url, before it checked out either. + assert bootstrap.returncode == 1, output + assert "git stopped at members/Ghost" in output, output assert "No url found" in output, output assert not any((tried / "members" / FAMILY_MEMBER).iterdir()), output result = paste(row_of(standard, root, "members"), tmp_path / "elsewhere") From 23e11056b320e874a9ecd88fad2ad98fa1ded74f Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Sat, 17 Oct 2026 00:03:06 +0000 Subject: [PATCH 11/15] bootstrap.py: `.gitmodules` is read as git's own parser reads it, a member git has no url for is its own reason, and a gitlink outside members/ gets a cure true for it, so a later duplicate section, a section under a name git ignores or a url git drops no longer ends `family bootstrap ok`, and a subset clone's printed update no longer dies `No url found` (#648, rv648's verify) rv648's verify of 45c512eb (1 MUST, 1 SHOULD, 3 NIT). M1. `init_stop` read `.gitmodules` with `git config --file --list`, which keeps entries git's submodule parser drops or overrides. In three plain clones git's init died (`No url found`, exit 128) and the run still said ok: a second section for a member's path, with no url, after its own (D1); the member's section under the name `../Nova` (Q1); and its url `-nowhere` (O1). `gitmodules_sections` now returns a `Gitmodules` with git 2.43's own reading beside the one the validator's findings use: - a section whose name is empty or has a `..` component, split on `/` and on a backslash, is ignored whole (`check_submodule_name`); - a `path` or `url` that begins with `-` is dropped, and the value before it stands (`looks_like_command_line_option`); - where two sections give one path, the later decides, and a section that gives a path again lets go of the one it gave, even where a later section gave that one too (`cache_put_path`, `cache_remove_path`). `has_url` asks the section git reads for a path, then the config under that section's name. `init_stop` and the reasons ask it. Where more than one section gives the path git has no url for, the line names `validate-family.py --no-members`, whose `gitmodules-path-twice` says to take the others out: a section written back would be one more (D1). S1. In a subset clone of a member whose section has no url (A1), git exits 0 and the line printed the exit-0 skip's update, which dies `No url found` when typed. "git has no url for that path" is now its own reason, asked before "git exited 0", with the write-it-back cure. The stop's sentence says "a gitlink it has no url for", which is true in D1 too, where a section does give the path a url. NIT 1. A gitlink outside members/ that git stopped at (H2) was sent to `--no-members`, which reads only members/ and says `family ok` there. It now gets `git rm --cached -- ` and a commit: measured, that ends ok and the gate passes in every mode, while a section written for it leaves `gitmodules-path-unnamed` in every mode. NIT 2. The `cures` docstring says what members mode refuses for #363's variant (`member-gitlink-absent`, with the move) as well as for a stray or a hand mount (`member-uninitialized`). Tests in tests/test_family_bootstrap_gate_order.py: U1, Q1 (three names) and O1 in one parametrized test, each written back; D1, typed through `gitmodules-path-twice`; D2, a name with dots and a url `-x` after a url, each still the credential's skip and then a true ok; A1, typed through both rounds; and H2, typed and the gate passing in every mode. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- templates/family-root/scripts/bootstrap.py | 270 +++++++++++++------- tests/test_family_bootstrap_gate_order.py | 280 ++++++++++++++++++--- 2 files changed, 436 insertions(+), 114 deletions(-) diff --git a/templates/family-root/scripts/bootstrap.py b/templates/family-root/scripts/bootstrap.py index 4e089c99..25893717 100755 --- a/templates/family-root/scripts/bootstrap.py +++ b/templates/family-root/scripts/bootstrap.py @@ -33,19 +33,25 @@ why, and fails the run (c): this holder records no gitlink at the row's path (a gitlink recorded only in another case, #363); no section of `.gitmodules` gives that path (a holder whose HEAD lost the - file, #611); git exited 0 and checked nothing out there; or git - stopped at a gitlink no section gives a url, the row's own or another - (a stray case variant, a member mounted by hand, a section that lost - its url), since git's init dies at the first such gitlink it asks - about, before it checks out any member. Git exits 0 over the second - where a recursive clone ran first, and 128 (`No url found for - submodule path`) in a plain clone; a missing credential exits - non-zero too, so what the holder records is the test, never git's - words. The line says what cures each: `validate-family.py` names the - commit for the first two; for the third it prints the `git submodule - update --init --checkout` that checks the member out whatever the - clone chose; and for the last, `validate-family.py --no-members` - names a gitlink no row names, and where the gitlink is a row's own + file, #611); git has no url for that path, in the section it reads + for it or in its config (a section that lost its url, or a second + one for the path after the member's); git exited 0 and checked + nothing out there; or git stopped at another gitlink it has no url + for (a stray case variant, a member mounted by hand), since git's + init dies at the first such gitlink it asks about, before it checks + out any member. Which section and which url git reads is its own + parser's answer, not `git config`'s (`gitmodules_sections`). Git + exits 0 over the second where a recursive clone ran first, and 128 + (`No url found for submodule path`) in a plain clone; a missing + credential exits non-zero too, so what the holder records is the + test, never git's words. The line says what cures each (`cures`): + `validate-family.py` names the commit for the first two; for the + fourth it prints the `git submodule update --init --checkout` that + checks the member out whatever the clone chose. For the third and + the last, `validate-family.py --no-members` names a path two + sections give, or a gitlink in members/ that no row names; `git rm + --cached` takes out one outside members/, which the validator does + not read; and where the gitlink is a row's own, with one section, the line says to write its section back. (b) RUNS EACH MEMBER'S OWN `make bootstrap`. Each member is a whole project @@ -73,7 +79,9 @@ import shutil import subprocess import sys +from collections import Counter from pathlib import Path +from typing import NamedTuple sys.path.insert(0, str(Path(__file__).resolve().parent)) from repo_shape import ( # noqa: E402 @@ -92,9 +100,9 @@ #: filled with the gitlink git's init died at (`init_stop`). NO_GITLINK = "no gitlink is recorded at that path" NO_SECTION = f"no section of {GITMODULES} gives that path" +NO_URL = "git has no url for that path" EXITED_0 = "git exited 0 and checked nothing out there" -STOPPED = ("git stopped at {}, a gitlink no section of " + GITMODULES - + " gives a url") +STOPPED = "git stopped at {}, a gitlink it has no url for" #: The two environment variables a leg/member credential may arrive in, in the #: order the `validate` workflow resolves them. An App-minted installation @@ -145,14 +153,14 @@ def fetch_members(root: Path, rows: list[dict]) -> list[str]: "fine-grained PAT with contents:read on them) or export an " "App-minted SHAPE_LEGS_APP_TOKEN, then re-run. What did arrive " "is bootstrapped below.", file=sys.stderr) - left, stopper = unfetchable(root, rows, proc.returncode == 0) + left, stopper, modules = unfetchable(root, rows, proc.returncode == 0) if left: print(" NOT CHECKED OUT, and not for want of a credential: " + ", ".join(f"{spelled(path)} (" + (STOPPED.format(spelled(stopper)) if why == STOPPED else why) + ")" for _, path, why in left) - + ". " + cures(left, stopper, rows), file=sys.stderr) + + ". " + cures(left, stopper, rows, modules), file=sys.stderr) elif proc.returncode == 0: print(" every member and leg fetched") return [f"{spelled(row.get('project') or '?')} (not checked out)" @@ -160,30 +168,63 @@ def fetch_members(root: Path, rows: list[dict]) -> list[str]: def cures(left: list[tuple[dict, str, str]], stopper: str | None, - rows: list[dict]) -> str: + rows: list[dict], modules: Gitmodules) -> str: """What (a)'s line says cures the rows `left` names, one sentence for - each why among them (#648). `validate-family.py`, which names the - commit, for a gitlink or a section the holder lacks at the row's path. - For the gitlink git stopped at, `validate-family.py --no-members`, which - names a gitlink no row names and the commit that cures it (members - checked, it refuses `member-uninitialized` for the first row there - instead), or, where a row names it, its section written back. For a row - git exited 0 over, the update that checks it out whatever the clone - chose: `--init` with the path registers a member a subset clone left - out, and `--checkout` overrides an `update = none`.""" + each kind of cause among them (#648): + + - A gitlink or a section the holder lacks at the row's path: + `validate-family.py`, which names it and the commit that cures it, + `member-gitlink-absent` (with the move, for a gitlink recorded only + in another case, #363) or `member-gitmodules-absent`. + - A path git has no url for (`has_url`), a row's own or the gitlink git + stopped at. Where more than one section gives it, as a merge that + kept both sides leaves them, `validate-family.py --no-members`, whose + `gitmodules-path-twice` says which to keep and to take the others + out: a section written back would be one more. Otherwise a row's own + is its section written back. A gitlink in `members/` no row names is + `--no-members` too, whose `member-gitlink-unnamed` names it and the + commit that cures it: members checked, the validator refuses + `member-uninitialized` for the first row where every row's gitlink + is there, as with a stray or a hand mount, and `member-gitlink-absent` + where a row's is only in another case, which the first sentence + names. A gitlink outside `members/`, which no mode of the validator + asks about, is taken out of the index: no row names it, and with no + url no clone can check it out. + - A row git exited 0 over: the update that checks it out whatever the + clone chose. `--init` with the path registers a member a subset + clone left out, and `--checkout` overrides an `update = none`.""" whys = {why for _, _, why in left} said = [] if whys & {NO_GITLINK, NO_SECTION}: said.append(f"`{PYTHON} scripts/validate-family.py` names the cause " "and the commit that cures it.") - if STOPPED in whys and stopper in {row_path(row) for row in rows}: - said.append(f"Write the section of {GITMODULES} for " - f"{spelled(stopper)} back as `family.py add` writes one, " - f"with its path and url, and commit {GITMODULES}.") - elif STOPPED in whys: + unread = [path for _, path, why in left if why == NO_URL] + if STOPPED in whys and stopper not in {path for _, path, _ in left}: + unread.append(stopper) + named = {row_path(row) for row in rows} + twice = [path for path in unread if modules.given[path] > 1] + own = [path for path in unread if path in named and path not in twice] + if twice: said.append(f"`{PYTHON} scripts/validate-family.py --no-members` " - f"names each gitlink in {DEFAULT_MEMBERS_DIR}/ that no " - "row names, and the commit that cures it.") + f"names each path more than one section of {GITMODULES} " + "gives, and how to take the others out.") + if own: + said.append(f"Write {'the' if len(own) == 1 else 'each'} section of " + f"{GITMODULES} for {', '.join(map(spelled, own))} back " + "as `family.py add` writes one, with its path and url, " + f"and commit {GITMODULES}.") + for path in unread: + if path in named or path in twice: + continue + if path.split("/", 1)[0].casefold() == DEFAULT_MEMBERS_DIR: + said.append(f"`{PYTHON} scripts/validate-family.py --no-members` " + f"names each gitlink in {DEFAULT_MEMBERS_DIR}/ that " + "no row names, and the commit that cures it.") + else: + removal = shlex.join(["git", "rm", "--cached", "--", path]) + said.append(f"No row names {spelled(path)} and no clone can check " + f"it out: `{spelled(removal)}` takes its gitlink " + "out, then commit that.") exited = [path for _, path, why in left if why == EXITED_0] if exited: update = shlex.join(["git", "submodule", "update", "--init", @@ -193,111 +234,170 @@ def cures(left: list[tuple[dict, str, str]], stopper: str | None, return " ".join(said) -def gitmodules_sections(root: Path) -> dict[str, dict[str, str]]: - """`{name: {key: value}}` of every section of `.gitmodules`, read as git - reads it: the work tree's file, the one `git submodule update` reads, - which a fresh clone takes from HEAD. An empty mapping where there is no +class Gitmodules(NamedTuple): + """`.gitmodules` as (a) asks it (#648), read by `gitmodules_sections`. + + `given` counts the sections that give each path, every section kept, as + `validate-family.py` reads them for the findings (a)'s line names: + `member-gitmodules-absent` and `gitmodules-path-twice`. `read` is + `{path: name}`, the section git reads for each path, and `urls` is + `{name: url}`, the url git takes from each, as git's own parser has + them.""" + + given: Counter + read: dict[str, str] + urls: dict[str, str] + + +def gitmodules_sections(root: Path) -> Gitmodules: + """`.gitmodules` as `git submodule update` reads it: the work tree's + file, which a fresh clone takes from HEAD. Nothing where there is no such file, or one git cannot read: git has no url for any submodule - then either.""" + then either. + + `read` and `urls` keep git 2.43's own rules (submodule-config.c), which + drop what `git config --list` keeps. Entry by entry, in file order: + + - a section whose name is empty or has a `..` component, split on `/` + and on a backslash, is ignored whole (git warns `ignoring suspicious + submodule name`); + - a `path` or `url` that begins with `-` is dropped, and the value + before it stands (`may be interpreted as a command-line option`); + - where two sections give one path, the later in the file decides, as + a later value in one section replaces an earlier. A section that + gives a path again lets go of the one it gave, even where a later + section gave that one too: git then reads no section there.""" + nothing = Gitmodules(Counter(), {}, {}) if not os.path.lexists(root / GITMODULES): - return {} + return nothing try: listed = git_out(["config", "-z", "--file", GITMODULES, "--list"], cwd=root, binary=True) except Refusal: - return {} + return nothing sections: dict[str, dict[str, str]] = {} + read: dict[str, str] = {} + held: dict[str, str] = {} + urls: dict[str, str] = {} for record in listed.split(b"\0"): key, _, value = record.decode("utf-8", "surrogateescape").partition("\n") section, _, variable = key.rpartition(".") - if section.startswith("submodule."): - sections.setdefault(section[len("submodule."):], - {})[variable] = value - return sections + if not section.startswith("submodule."): + continue + name = section[len("submodule."):] + sections.setdefault(name, {})[variable] = value + if (not name or ".." in name.replace("\\", "/").split("/") + or value.startswith("-")): + continue + if variable == "path": + if name in held: + read.pop(held[name], None) + held[name] = value + read[value] = name + elif variable == "url": + urls[name] = value + return Gitmodules(Counter(keys["path"] for keys in sections.values() + if "path" in keys), read, urls) + + +def submodule_config(root: Path) -> set[str]: + """Every `submodule.` key this repository's git config sets, in every + scope git reads, as git's init asks them: `submodule.active`, and + `submodule..url` as an earlier init writes it. None where git + lists none.""" + try: + config = git_out(["config", "-z", "--get-regexp", r"^submodule\."], + cwd=root, binary=True) + except Refusal: + return set() + return {record.decode("utf-8", "surrogateescape").partition("\n")[0] + for record in config.split(b"\0")} + +def has_url(modules: Gitmodules, config: set[str], path: str) -> bool: + """Whether git's init finds a url for the gitlink at `path`: in the + section git reads for that path (`Gitmodules.read`), or in the git + config under that section's name, as an earlier init writes it. Where + git reads no section there it has none, whatever the config holds.""" + name = modules.read.get(path) + return name is not None and (name in modules.urls + or f"submodule.{name}.url" in config) -def init_stop(root: Path, sections: dict[str, dict[str, str]]) -> str | None: + +def init_stop(root: Path, modules: Gitmodules, config: set[str] + ) -> str | None: """The gitlink `git submodule update --init` dies at, or None: the first - in the index, in index order, that its init step asks about and finds - no url for (`No url found for submodule path`, exit 128), before it + in the index, in index order, that its init step asks about and has no + url for (`No url found for submodule path`, exit 128), before it checks out any member (#648). - ASKED AS GIT 2.43 ASKS IT, of the index, `sections` and this + ASKED AS GIT 2.43 ASKS IT, of the index, `modules` and `config`, this repository's git config. Every gitlink the index records is asked, but where the config sets `submodule.active`, as a recursive clone sets it - (`.`), only those a section gives a path: git counts no other active. - One no section gives stops it; so does one whose section gives no url, - unless the config has one under the section's name, as an earlier init - writes it. So a stray case variant or a member mounted by hand stops a - plain clone's update, and leaves a recursive clone's to the credential. - Read nothing, it answers None.""" + (`.`), only those a section git reads gives a path: git counts no other + active. A gitlink git has no url for (`has_url`) stops it. So a stray + case variant or a member mounted by hand stops a plain clone's update, + and leaves a recursive clone's to the credential. Read nothing, it + answers None.""" try: index = git_out(["ls-files", "-s", "-z"], cwd=root, binary=True) except Refusal: return None - try: - config = git_out(["config", "-z", "--get-regexp", r"^submodule\."], - cwd=root, binary=True) - except Refusal: - config = b"" - keys = {record.decode("utf-8", "surrogateescape").partition("\n")[0] - for record in config.split(b"\0")} - names: dict[str, list[str]] = {} - for name, held in sections.items(): - if "path" in held: - names.setdefault(held["path"], []).append(name) for record in index.split(b"\0"): meta, _, entry = record.partition(b"\t") if meta.split()[:1] != [GITLINK_MODE]: continue path = entry.decode("utf-8", "surrogateescape") - given = names.get(path, []) - if not given and "submodule.active" in keys: + if path not in modules.read and "submodule.active" in config: continue - if not any("url" in sections[name] or f"submodule.{name}.url" in keys - for name in given): + if not has_url(modules, config, path): return path return None def unfetchable(root: Path, rows: list[dict], exited_0: bool - ) -> tuple[list[tuple[dict, str, str]], str | None]: + ) -> tuple[list[tuple[dict, str, str]], str | None, + Gitmodules]: """`(row, path, why)` for each row whose member (a) left NOT CHECKED OUT - where no credential would check it out (#648), in row order, and the - gitlink git's init died at (`init_stop`), or None. + where no credential would check it out (#648), in row order; the + gitlink git's init died at (`init_stop`), or None; and `.gitmodules` + as it was read, for `cures`. Asked of this holder's own records, never of git's words: a row at whose path the holder records no gitlink (`recorded_gitlink`, as `validate-family.py`'s `member-gitlink-absent` reads one), or whose gitlink no section of `.gitmodules` gives a path (as its `member-gitmodules-absent` reads HEAD's file, which is the work tree's - in a clone); where git exited 0, any row at all, since git then lacked - nothing a credential gives; and where git's init died at a gitlink no - section gives a url, any row at all, since git then checked out no - member and asked for no credential. Any other row not checked out is - the missing credential's, and stays the skip it always was. A path two - rows give is said once.""" + in a clone); one git has no url for (`has_url`), where no credential + helps either, whatever git exited; where git exited 0, any row at all, + since git then lacked nothing a credential gives; and where git's init + died at a gitlink it has no url for, any row at all, since git then + checked out no member and asked for no credential. Any other row not + checked out is the missing credential's, and stays the skip it always + was. A path two rows give is said once.""" left: dict[str, dict] = {} for row in rows: if not (member_path(root, row) / ".git").exists(): left.setdefault(row_path(row), row) if not left: - return [], None - sections = gitmodules_sections(root) - given = {held["path"] for held in sections.values() if "path" in held} - stopper = None if exited_0 else init_stop(root, sections) + return [], None, Gitmodules(Counter(), {}, {}) + modules = gitmodules_sections(root) + config = submodule_config(root) + stopper = None if exited_0 else init_stop(root, modules, config) found: list[tuple[dict, str, str]] = [] for path, row in left.items(): if recorded_gitlink(root, path) is None: found.append((row, path, NO_GITLINK)) - elif path not in given: + elif not modules.given[path]: found.append((row, path, NO_SECTION)) + elif not has_url(modules, config, path): + found.append((row, path, NO_URL)) elif exited_0: found.append((row, path, EXITED_0)) elif stopper is not None: found.append((row, path, STOPPED)) - return found, stopper + return found, stopper, modules def members_of(root: Path) -> tuple[str, list[dict]]: diff --git a/tests/test_family_bootstrap_gate_order.py b/tests/test_family_bootstrap_gate_order.py index bbe04750..06598e5a 100644 --- a/tests/test_family_bootstrap_gate_order.py +++ b/tests/test_family_bootstrap_gate_order.py @@ -12,12 +12,15 @@ gitlink recorded only at `Members/

`), git dies `No url found for submodule path` (exit 128), and it still ended ok. It fails now, saying which members no credential would check out and why, and what cures - them. So it does where git's init dies at a gitlink no section gives a - url, a stray case variant, a member mounted by hand or a section that - lost its url (rv648's S1, H1 and U1), and where git exits 0 over a member - a subset clone or `update = none` leaves out, whose line prints the - update that checks it out. A missing credential is still the skip it - was, after a recursive clone too. + them. So it does where git's init dies at a gitlink it has no url for: + a stray case variant, a member mounted by hand, in `members/` or outside + it, a section that lost its url (rv648's S1, H1, H2 and U1), and, as + git's own parser reads `.gitmodules`, a second section for a member's + path after its own, a section under a name git ignores and a url git + drops (D1, Q1 and O1). So it does where git exits 0 over a member a + subset clone or `update = none` leaves out, whose line prints the update + that checks it out, unless git has no url for it (A1). A missing + credential is still the skip it was, after a recursive clone too. - (b) In a plain clone of #363's state followed by `git submodule update --init --recursive`, as CI and `make bootstrap` run it, the update dies at `Members/

` before it checks out any member, so where that row is not @@ -74,13 +77,15 @@ "{cure}") NO_SECTION = "no section of .gitmodules gives that path" NO_GITLINK = "no gitlink is recorded at that path" +NO_URL = "git has no url for that path" EXITED_0 = "git exited 0 and checked nothing out there" -STOPPED = ("git stopped at {path}, a gitlink no section of .gitmodules " - "gives a url") +STOPPED = "git stopped at {path}, a gitlink it has no url for" #: What the line says cures each why: the validator for the first two; for -#: the gitlink git stopped at, the validator's `--no-members`, or, where a -#: row names that gitlink, its section written back; and for a member git +#: a path git has no url for, a row's own or the gitlink git stopped at, +#: the validator's `--no-members` where two sections give it, a row's own +#: section written back, `--no-members` for a gitlink in members/ no row +#: names and `git rm --cached` for one outside it; and for a member git #: exited 0 over, the update that checks it out. VALIDATOR_CURE = (f"`{PYTHON} scripts/validate-family.py` names the cause " "and the commit that cures it.") @@ -90,6 +95,11 @@ SECTION_CURE = ("Write the section of .gitmodules for {path} back as " "`family.py add` writes one, with its path and url, and " "commit .gitmodules.") +TWICE_CURE = (f"`{PYTHON} scripts/validate-family.py --no-members` names " + "each path more than one section of .gitmodules gives, and how " + "to take the others out.") +OUTSIDE_CURE = ("No row names {path} and no clone can check it out: `git rm " + "--cached -- {path}` takes its gitlink out, then commit that.") CHECKOUT_CURE = ("`git submodule update --init --checkout --recursive -- " "{path}` checks it out.") @@ -184,17 +194,49 @@ def hand_mounted(holder: Path, env: dict, family) -> str: UNNAMED_STOPS = {"stray": stray, "hand-mount": hand_mounted} -def without_url(data: bytes, project: str) -> bytes: - """`.gitmodules` bytes with the `url` line of `project`'s section taken - out, the rest as `family.py add` wrote it (rv648's U1). A checkout's - copy may end its lines CRLF.""" +def url_edited(data: bytes, project: str, edit) -> bytes: + """`.gitmodules` bytes with the `url` line of `project`'s section put + through `edit`, the rest as `family.py add` wrote it. A checkout's copy + may end its lines CRLF.""" section = re.compile(rb'(\[submodule "members/' + project.encode() - + rb'"\]\r?\n(?:\t[^\n]*\n)*?)\turl = [^\n]*\n') - edited, count = section.subn(rb"\1", data) + + rb'"\]\r?\n(?:\t[^\n]*\n)*?)(\turl = [^\n]*\n)') + edited, count = section.subn( + lambda found: found.group(1) + edit(found.group(2)), data) assert count == 1, data return edited +def without_url(data: bytes, project: str) -> bytes: + """`.gitmodules` bytes with the `url` line of `project`'s section taken + out (rv648's U1).""" + return url_edited(data, project, lambda line: b"") + + +def renamed(data: bytes, name: str) -> bytes: + """`.gitmodules` bytes with OTHER's section under the name `name`, its + keys as `family.py add` wrote them, escaped as git config escapes a + subsection name.""" + own = f'[submodule "members/{OTHER}"]'.encode() + assert data.count(own) == 1, data + escaped = name.replace("\\", "\\\\").replace('"', '\\"') + return data.replace(own, f'[submodule "{escaped}"]'.encode("utf-8")) + + +#: A second section, `dup`, that gives OTHER's path and no url, as a merge +#: that kept both sides of `.gitmodules` can leave one (rv648's D1 and D2). +DUPLICATE = f'[submodule "dup"]\n\tpath = members/{OTHER}\n'.encode() + + +def edited_clone(holder: Path, env: dict, family, tmp_path: Path, + edit) -> tuple[Path, bytes]: + """HEAD's `.gitmodules` put through `edit` by hand and committed, then a + plain clone of it (`cloned`). Returns the clone and the bytes `family.py + add` wrote, which writing the file back restores.""" + data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout + committed_gitmodules(holder, env, edit(data)) + return cloned(holder, env, family, tmp_path, recurse=False), data + + def failed_on(*projects: str) -> str: return "family bootstrap FAILED: " + ", ".join( f"{project} (not checked out)" for project in projects) @@ -430,37 +472,217 @@ def test_a_gitlink_no_row_names_that_stops_a_plain_clones_update_fails( assert after_first_line(again) == FETCHED -def test_a_section_with_no_url_that_stops_the_update_says_to_write_it_back( - family, holder, offline, tmp_path): - """rv648's U1. OTHER's section lost its url, committed: in a plain - clone git's init dies at OTHER's own gitlink, after MEMBER's and before - it checks out either. The run said ok, and no mode of the validator - names it. It names both rows and says to write that section back; - written back as `family.py add` wrote it and committed, the same run - checks both members out.""" +#: `(edit, what git warns)`: OTHER's section with no url (rv648's U1), or +#: one git's own parser drops, though `git config --list` keeps it: a name +#: that is empty or has a `..` component, split on `/` or a backslash +#: (Q1), and a url that begins with `-` (O1). +NO_URL_FOR_OTHER = { + "no-url": (lambda data: without_url(data, OTHER), ""), + "dotdot-name": (lambda data: renamed(data, f"../{OTHER}"), + f"ignoring suspicious submodule name: ../{OTHER}\n"), + "empty-name": (lambda data: renamed(data, ""), + "ignoring suspicious submodule name: \n"), + "backslash-name": (lambda data: renamed(data, f"{OTHER}\\..\\x"), + "ignoring suspicious submodule name: " + f"{OTHER}\\..\\x\n"), + "option-url": (lambda data: url_edited(data, OTHER, + lambda line: b"\turl = -nowhere\n"), + "which may be interpreted as a command-line option: " + "-nowhere\n"), +} + + +@pytest.mark.parametrize("edit, warned", NO_URL_FOR_OTHER.values(), + ids=NO_URL_FOR_OTHER) +def test_a_member_git_has_no_url_for_stops_the_update_and_is_written_back( + family, holder, offline, tmp_path, edit, warned): + """rv648's U1, Q1 and O1. OTHER's section lost its url, or is one git + reads no url from, committed: in a plain clone git's init dies at + OTHER's own gitlink, after MEMBER's and before it checks out either. + The run said ok in Q1 and O1, reading the section with `git config`, + which keeps what git drops, and no mode of the validator names any of + the three. It names both rows, OTHER for want of a url and MEMBER + stopped there, and says to write OTHER's section back; written back as + `family.py add` wrote it and committed, the same run checks both + members out.""" env = fetching(isolated(offline, tmp_path)) - data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout - committed_gitmodules(holder, env, without_url(data, OTHER)) - clone = cloned(holder, env, family, tmp_path, recurse=False) + clone, data = edited_clone(holder, env, family, tmp_path, edit) result = bootstrap(clone, env, "--skip-members") assert result.returncode == 1, result.stdout + result.stderr - why = STOPPED.format(path=f"members/{OTHER}") assert result.stderr.splitlines()[-2:] == [ - not_fetched((f"members/{MEMBER}", why), (f"members/{OTHER}", why), + not_fetched((f"members/{MEMBER}", + STOPPED.format(path=f"members/{OTHER}")), + (f"members/{OTHER}", NO_URL), cure=SECTION_CURE.format(path=f"members/{OTHER}")), failed_on(MEMBER, OTHER)], result.stderr assert (f"No url found for submodule path 'members/{OTHER}'" in result.stderr) + assert warned in result.stderr + (clone / GITMODULES).write_bytes(data) + raw(clone, env, "add", "--", GITMODULES) + commit_all(clone, env, "OTHER's section written back") + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + + +def test_a_second_section_git_reads_for_a_row_s_path_names_the_two( + family, holder, offline, tmp_path): + """rv648's D1. A second section, `dup`, gives OTHER's path and no url, + after OTHER's own: git reads the later of two sections for one path, so + in a plain clone its init dies at OTHER's gitlink, after MEMBER's and + before it checks out either. The run said ok, since one section gave + the path a url. It names both rows and `--no-members`, whose + `gitmodules-path-twice` names the two sections and says to take the + other out: a section written back would be a third. Taken out as that + says and committed, the same run checks both members out.""" + env = fetching(isolated(offline, tmp_path)) + clone, _ = edited_clone(holder, env, family, tmp_path, + lambda data: data + DUPLICATE) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert result.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{MEMBER}", + STOPPED.format(path=f"members/{OTHER}")), + (f"members/{OTHER}", NO_URL), cure=TWICE_CURE), + failed_on(MEMBER, OTHER)], result.stderr + assert (f"No url found for submodule path 'members/{OTHER}'" + in result.stderr) + fallback = validate(clone, "--no-members") + assert fallback.returncode == 1, fallback.stdout + fallback.stderr + (finding,) = [found for found in findings(fallback) + if found.startswith("FINDING gitmodules-path-twice: ")] + assert finding.startswith( + f"FINDING gitmodules-path-twice: .gitmodules gives the path " + f"members/{OTHER} to 2 submodules, `members/{OTHER}` (url "), finding + assert "`dup` (no url)" in finding, finding + assert "--unset-all submodule..` for every key" in finding + raw(clone, env, "config", "-f", GITMODULES, "--unset-all", + "submodule.dup.path") + raw(clone, env, "add", "--", GITMODULES) + commit_all(clone, env, "The second section taken out as printed") + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + + +#: `.gitmodules` edits git's own parser reads past: rv648's D2, the second +#: section placed BEFORE OTHER's own, which git then reads; a name whose +#: dots are no `..` component; and a url that begins with `-` after OTHER's +#: own, which git drops, keeping the one before it. +READ_PAST = { + "duplicate-first": lambda data: DUPLICATE + data, + "dots-in-a-name": lambda data: renamed(data, f"..{OTHER}.."), + "option-url-after-a-url": lambda data: url_edited( + data, OTHER, lambda line: line + b"\turl = -x\n"), +} + + +@pytest.mark.parametrize("edit", READ_PAST.values(), ids=READ_PAST) +def test_what_git_reads_past_is_still_the_skip_and_a_true_ok( + family, holder, offline, tmp_path, edit): + """No misfire. Where git's parser still reads a url for OTHER, a plain + clone's update with no credential for the members fails on their fetch, + not in its init, so they are the credential's: skipped, and the run + ends ok. With the credential, the same update checks both members out, + exit 0, and the run is a good holder's.""" + env = isolated(offline, tmp_path) + clone, _ = edited_clone(holder, env, family, tmp_path, edit) + + skipped = bootstrap(clone, env, "--skip-members") + + assert skipped.returncode == 0, skipped.stdout + skipped.stderr + assert "FETCH INCOMPLETE" in skipped.stderr + assert "not for want of a credential" not in skipped.stderr + fetched = bootstrap(clone, fetching(env), "--skip-members") + assert fetched.returncode == 0, fetched.stdout + fetched.stderr + assert after_first_line(fetched) == FETCHED + assert fetched.stderr == "" + + +def test_a_subset_clone_of_a_member_git_has_no_url_for_is_written_back( + family, holder, offline, tmp_path): + """rv648's A1. OTHER's section lost its url, committed, and the clone + chose MEMBER alone: git exits 0, and the line printed the exit-0 skip's + update, which dies `No url found` when typed (exit 128) and checks + nothing out. Git has no url for OTHER, in its section or its config, + so the line says to write the section back. Written back and + committed, the run is the exit-0 skip, and the update it prints, typed + as printed, lets the same run end ok.""" + env = fetching(isolated(offline, tmp_path)) + clone, data = edited_clone(holder, env, family, tmp_path, + lambda data: without_url(data, OTHER)) + env = skip_by_subset(clone, env) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert result.stderr.splitlines() == [ + not_fetched((f"members/{OTHER}", NO_URL), + cure=SECTION_CURE.format(path=f"members/{OTHER}")), + failed_on(OTHER)] (clone / GITMODULES).write_bytes(data) raw(clone, env, "add", "--", GITMODULES) commit_all(clone, env, "OTHER's section written back") + skipped = bootstrap(clone, env, "--skip-members") + assert skipped.returncode == 1, skipped.stdout + skipped.stderr + assert skipped.stderr.splitlines() == [ + not_fetched((f"members/{OTHER}", EXITED_0), + cure=CHECKOUT_CURE.format(path=f"members/{OTHER}")), + failed_on(OTHER)] + (line,) = re.findall(r"`(git submodule update [^`]*)`", skipped.stderr) + type_line(clone, env, line) + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + assert again.stderr == "" + + +def test_a_hand_mount_outside_members_is_taken_out_of_the_index( + family, holder, offline, tmp_path): + """rv648's H2. SPARE's gitlink recorded by hand at `vendor/`, + outside members/, with no row and no section, and committed: in a plain + clone git's init dies there before it checks out any member. No mode + of the validator reads a gitlink outside members/, and `--no-members` + says `family ok`, so the line names the `git rm --cached` that takes + the gitlink out. Typed as printed and committed, the same run checks + both members out, and the gate passes in every mode.""" + env = fetching(isolated(offline, tmp_path)) + stopper = f"vendor/{SPARE}" + stage_gitlink(holder, stopper, spare_commits(family)[1]) + commit_all(holder, env, "A gitlink mounted by hand outside members/") + clone = cloned(holder, env, family, tmp_path, recurse=False) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + why = STOPPED.format(path=stopper) + assert result.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{MEMBER}", why), (f"members/{OTHER}", why), + cure=OUTSIDE_CURE.format(path=stopper)), + failed_on(MEMBER, OTHER)], result.stderr + assert f"No url found for submodule path '{stopper}'" in result.stderr + fallback = validate(clone, "--no-members") + assert fallback.returncode == 0, "premise: it reads only members/" + (line,) = re.findall(r"`(git rm [^`]*)`", result.stderr) + type_line(clone, env, line) + commit_all(clone, env, "The gitlink taken out as printed") again = bootstrap(clone, env, "--skip-members") assert again.returncode == 0, again.stdout + again.stderr assert after_first_line(again) == FETCHED + assert_passes_in_every_mode(clone, 2) @pytest.mark.parametrize("set_up", UNNAMED_STOPS.values(), ids=UNNAMED_STOPS) From f74f640ca69de5dcda0bd84dbbb441f601f26cde Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Sat, 17 Oct 2026 00:29:07 +0000 Subject: [PATCH 12/15] bootstrap.py: a `.gitmodules` a merge left unmerged in the index is its own reason, since git reads no section from it, so a plain clone mid-merge whose update dies `No url found` no longer ends `family bootstrap ok` (#648, rv648's MM1) rv648's verify of 23e11056 (1 MUST). In a plain clone, a merge that conflicts in `.gitmodules`, resolved in the work tree and not added, leaves the index holding the file only at its three conflict stages. Git then reads no `.gitmodules` at all (`repo_read_gitmodules` asks `is_gitmodules_unmerged` first), so its init dies at the first gitlink, exit 128. `gitmodules_sections` read the work tree's file, found every section with its url, and the run ended ok. `gitmodules_sections` now asks `git ls-files -u -z -- .gitmodules` first. Where the index holds the file at a conflict stage, it returns nothing, with `Gitmodules.unmerged` set. `unfetchable` asks that first: every row not checked out gets "git reads no .gitmodules while a merge leaves it unmerged", and the line says to resolve the merge in .gitmodules and `git add .gitmodules`, then run this again. Added, the same run checks both members out. Test: rv648's MM1 in tests/test_family_bootstrap_gate_order.py, the two sides of the merge each giving OTHER's section another `branch`; the cure typed, and the run ends ok. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 --- templates/family-root/scripts/bootstrap.py | 44 +++++++++++---- tests/test_family_bootstrap_gate_order.py | 62 +++++++++++++++++++++- 2 files changed, 95 insertions(+), 11 deletions(-) diff --git a/templates/family-root/scripts/bootstrap.py b/templates/family-root/scripts/bootstrap.py index 25893717..6dce353f 100755 --- a/templates/family-root/scripts/bootstrap.py +++ b/templates/family-root/scripts/bootstrap.py @@ -40,11 +40,14 @@ for (a stray case variant, a member mounted by hand), since git's init dies at the first such gitlink it asks about, before it checks out any member. Which section and which url git reads is its own - parser's answer, not `git config`'s (`gitmodules_sections`). Git - exits 0 over the second where a recursive clone ran first, and 128 - (`No url found for submodule path`) in a plain clone; a missing - credential exits non-zero too, so what the holder records is the - test, never git's words. The line says what cures each (`cures`): + parser's answer, not `git config`'s (`gitmodules_sections`), and + while a merge leaves `.gitmodules` unmerged in the index git reads + none at all: every such member is said to be that, and the cure is + the file resolved and added. Git exits 0 over the second where a + recursive clone ran first, and 128 (`No url found for submodule + path`) in a plain clone; a missing credential exits non-zero too, + so what the holder records is the test, never git's words. The line + says what cures each (`cures`): `validate-family.py` names the commit for the first two; for the fourth it prints the `git submodule update --init --checkout` that checks the member out whatever the clone chose. For the third and @@ -98,6 +101,7 @@ #: Why (a) left a row's member NOT CHECKED OUT where no credential would #: check it out (#648), in the order `unfetchable` asks. `STOPPED` is #: filled with the gitlink git's init died at (`init_stop`). +UNMERGED = f"git reads no {GITMODULES} while a merge leaves it unmerged" NO_GITLINK = "no gitlink is recorded at that path" NO_SECTION = f"no section of {GITMODULES} gives that path" NO_URL = "git has no url for that path" @@ -172,6 +176,8 @@ def cures(left: list[tuple[dict, str, str]], stopper: str | None, """What (a)'s line says cures the rows `left` names, one sentence for each kind of cause among them (#648): + - A `.gitmodules` a merge left unmerged in the index, which git reads + nothing from (`Gitmodules.unmerged`): the file resolved and added. - A gitlink or a section the holder lacks at the row's path: `validate-family.py`, which names it and the commit that cures it, `member-gitlink-absent` (with the move, for a gitlink recorded only @@ -195,6 +201,9 @@ def cures(left: list[tuple[dict, str, str]], stopper: str | None, clone left out, and `--checkout` overrides an `update = none`.""" whys = {why for _, _, why in left} said = [] + if UNMERGED in whys: + said.append(f"Resolve the merge in {GITMODULES} and `git add " + f"{GITMODULES}`, then run this again.") if whys & {NO_GITLINK, NO_SECTION}: said.append(f"`{PYTHON} scripts/validate-family.py` names the cause " "and the commit that cures it.") @@ -242,18 +251,23 @@ class Gitmodules(NamedTuple): `member-gitmodules-absent` and `gitmodules-path-twice`. `read` is `{path: name}`, the section git reads for each path, and `urls` is `{name: url}`, the url git takes from each, as git's own parser has - them.""" + them. `unmerged` says a merge left the file unmerged in the index, and + then git reads nothing from it, and the three are empty.""" given: Counter read: dict[str, str] urls: dict[str, str] + unmerged: bool = False def gitmodules_sections(root: Path) -> Gitmodules: """`.gitmodules` as `git submodule update` reads it: the work tree's file, which a fresh clone takes from HEAD. Nothing where there is no such file, or one git cannot read: git has no url for any submodule - then either. + then either. Nothing, and `unmerged`, where the index holds the file + only at a merge's conflict stages (`git ls-files -u`): git reads no + `.gitmodules` at all then, whatever the work tree holds + (`repo_read_gitmodules`), and its init dies at the first gitlink. `read` and `urls` keep git 2.43's own rules (submodule-config.c), which drop what `git config --list` keeps. Entry by entry, in file order: @@ -268,6 +282,14 @@ def gitmodules_sections(root: Path) -> Gitmodules: gives a path again lets go of the one it gave, even where a later section gave that one too: git then reads no section there.""" nothing = Gitmodules(Counter(), {}, {}) + try: + stages = git_out(["ls-files", "-u", "-z", "--", GITMODULES], + cwd=root, binary=True) + except Refusal: + stages = b"" + if any(record.partition(b"\t")[2] == GITMODULES.encode() + for record in stages.split(b"\0")): + return nothing._replace(unmerged=True) if not os.path.lexists(root / GITMODULES): return nothing try: @@ -364,7 +386,9 @@ def unfetchable(root: Path, rows: list[dict], exited_0: bool gitlink git's init died at (`init_stop`), or None; and `.gitmodules` as it was read, for `cures`. - Asked of this holder's own records, never of git's words: a row at + Asked of this holder's own records, never of git's words: any row at + all while a merge leaves `.gitmodules` unmerged in the index, since git + then reads no section and asks for no credential; a row at whose path the holder records no gitlink (`recorded_gitlink`, as `validate-family.py`'s `member-gitlink-absent` reads one), or whose gitlink no section of `.gitmodules` gives a path (as its @@ -387,7 +411,9 @@ def unfetchable(root: Path, rows: list[dict], exited_0: bool stopper = None if exited_0 else init_stop(root, modules, config) found: list[tuple[dict, str, str]] = [] for path, row in left.items(): - if recorded_gitlink(root, path) is None: + if modules.unmerged: + found.append((row, path, UNMERGED)) + elif recorded_gitlink(root, path) is None: found.append((row, path, NO_GITLINK)) elif not modules.given[path]: found.append((row, path, NO_SECTION)) diff --git a/tests/test_family_bootstrap_gate_order.py b/tests/test_family_bootstrap_gate_order.py index 06598e5a..3bd49d54 100644 --- a/tests/test_family_bootstrap_gate_order.py +++ b/tests/test_family_bootstrap_gate_order.py @@ -19,8 +19,10 @@ path after its own, a section under a name git ignores and a url git drops (D1, Q1 and O1). So it does where git exits 0 over a member a subset clone or `update = none` leaves out, whose line prints the update - that checks it out, unless git has no url for it (A1). A missing - credential is still the skip it was, after a recursive clone too. + that checks it out, unless git has no url for it (A1), and where a + merge leaves `.gitmodules` unmerged, which git then reads nothing from + (MM1). A missing credential is still the skip it was, after a recursive + clone too. - (b) In a plain clone of #363's state followed by `git submodule update --init --recursive`, as CI and `make bootstrap` run it, the update dies at `Members/

` before it checks out any member, so where that row is not @@ -78,6 +80,7 @@ NO_SECTION = "no section of .gitmodules gives that path" NO_GITLINK = "no gitlink is recorded at that path" NO_URL = "git has no url for that path" +UNMERGED = "git reads no .gitmodules while a merge leaves it unmerged" EXITED_0 = "git exited 0 and checked nothing out there" STOPPED = "git stopped at {path}, a gitlink it has no url for" @@ -100,6 +103,8 @@ "to take the others out.") OUTSIDE_CURE = ("No row names {path} and no clone can check it out: `git rm " "--cached -- {path}` takes its gitlink out, then commit that.") +UNMERGED_CURE = ("Resolve the merge in .gitmodules and `git add .gitmodules`, " + "then run this again.") CHECKOUT_CURE = ("`git submodule update --init --checkout --recursive -- " "{path}` checks it out.") @@ -685,6 +690,59 @@ def test_a_hand_mount_outside_members_is_taken_out_of_the_index( assert_passes_in_every_mode(clone, 2) +def with_branch(value: str): + """An edit giving OTHER's section a `branch` key of `value`, after its + url.""" + return lambda data: url_edited( + data, OTHER, lambda line: line + f"\tbranch = {value}\n".encode()) + + +def test_a_gitmodules_a_merge_left_unmerged_says_to_add_it( + family, holder, offline, tmp_path): + """rv648's MM1. In a plain clone, a merge conflicts in `.gitmodules` + (each side gave OTHER's section another `branch`), and the file is + resolved in the work tree but not added: the index holds it only at + the conflict's three stages. Git reads no `.gitmodules` at all then, + so its init dies at MEMBER's gitlink, the first, and the run said ok, + reading the work tree's file. It names both rows for that and says to + add the file once it is resolved; added, the same run checks both + members out.""" + env = fetching(isolated(offline, tmp_path)) + remote = family["remotes"] / f"{NAME}.git" + side = f"w648/{next(BRANCHES)}" + data = raw(holder, env, "show", f"HEAD:{GITMODULES}").stdout + committed_gitmodules(holder, env, with_branch("dev")(data)) + raw(holder, env, "push", "-q", str(remote), f"HEAD:refs/heads/{side}") + raw(holder, env, "update-index", "-q", "--refresh", check=False) + raw(holder, env, "reset", "-q", "--keep", "HEAD~1") + clone, _ = edited_clone(holder, env, family, tmp_path, + with_branch("main")) + raw(clone, env, "fetch", "-q", "origin", side) + merged = raw(clone, env, "merge", "-q", "--no-edit", "FETCH_HEAD", + check=False) + assert merged.returncode == 1, merged.stdout + merged.stderr + stages = raw(clone, env, "ls-files", "-u", "--", GITMODULES).stdout + assert len(stages.splitlines()) == 3, "premise: .gitmodules is unmerged" + (clone / GITMODULES).write_bytes( + raw(clone, env, "show", f":2:{GITMODULES}").stdout) + + result = bootstrap(clone, env, "--skip-members") + + assert result.returncode == 1, result.stdout + result.stderr + assert result.stderr.splitlines()[-2:] == [ + not_fetched((f"members/{MEMBER}", UNMERGED), + (f"members/{OTHER}", UNMERGED), cure=UNMERGED_CURE), + failed_on(MEMBER, OTHER)], result.stderr + assert (f"No url found for submodule path 'members/{MEMBER}'" + in result.stderr) + raw(clone, env, "add", "--", GITMODULES) + + again = bootstrap(clone, env, "--skip-members") + + assert again.returncode == 0, again.stdout + again.stderr + assert after_first_line(again) == FETCHED + + @pytest.mark.parametrize("set_up", UNNAMED_STOPS.values(), ids=UNNAMED_STOPS) def test_a_recursive_clone_with_no_credential_is_still_the_skip( family, holder, offline, tmp_path, set_up): From 1423e5a32eb10c91c7bcc7e0a8045013e785e964 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Sat, 17 Oct 2026 11:29:10 +0000 Subject: [PATCH 13/15] family-root README.md and AGENTS-shape.md, and AGENTS.md: the holder's docs say its parent folder is named EXACTLY after its repository, case included, on every disk, and the never to hand-edit the `members:` block names the row repair `validate-family.py` asks for, whose commit is the person's (#763) Two sets of words now agree. Words only: no tool, no remedy and no --help text changes. The folder rule (N6 of #695's review). The holder's README and its pinned AGENTS-shape.md said the parent folder was "not named after the family", which reads as a match that ignores case. siblings.py compares the name exactly and has a separate branch for a folder that differs only in case. Both now say what the standard's README and the handbook say since #738: the folder must be named EXACTLY after its repository, case included, on every disk. The README is rendered once at `init`, so it reaches new holders only; AGENTS-shape.md is in FAMILY_COPIED_VERBATIM, so it rides in the shape bump as one `upstream-changed` row. The hand repair of a row. AGENTS.md and AGENTS-shape.md said never to hand-edit family.yaml's `members:` block, and validate-family.py tells a person to in two cases: RESTORE_ROW (a row deleted, or its `path:` edited, by hand is put back as git last had it) and `member-gitlink-absent`'s (#681, the row of a member that is to go is taken out). The issue's first option is taken, so no remedy changes: the never stays for the normal path, where `add`, `bump` and `remove` rewrite the block, and each doc says the validator may name a repair of a row and that the commit is then the person's. AGENTS-shape.md, which has the room, names both cases. AGENTS.md is at its cap of 492 lines, so item 3 was reflowed to cost no line: "on a branch you make first (see above): the tracking branch is refused (code)" became "on a branch made first (code refuses the tracking branch)" and "Push the branch" became "Push it". No other sentence of the file moved. tests/test_family_docs_one_spelling.py gains 14 tests: the folder sentence in each holder doc and the old wording gone, each doc's never kept and followed by the validator's repair and whose commit it is, the shape doc's two cases, and the validator's two remedies read from the validator itself (`RESTORE_ROW`, and what `member-gitlink-absent` is given), so a remedy that stops telling a person to edit a row fails there and takes the docs' exception with it. Lands after #738 and #747 (which carries #682), all v1.1.0. Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 12 +- templates/family-root/AGENTS-shape.md | 30 ++--- templates/family-root/README.md | 16 +-- tests/test_family_docs_one_spelling.py | 146 ++++++++++++++++++++++++- 4 files changed, 176 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1b3fb94f..e2d2190e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -392,12 +392,12 @@ python3 scripts/family.py siblings --family-root A holder keeps `members/` in ONE spelling on every disk: `add` refuses `family-members-dir-variant` (`member-path-taken` for a variant of the member's own path) until a human renames what sits under another spelling. -3. **Each of `add`, `bump` and `remove` writes ONE commit** moving the - gitlink and the pin together, on a branch you make first (see above): the - tracking branch is refused (`family-on-tracking-branch`). Push the branch - as its `NEXT` line says and land it as a pull request; never suggest a - push to the default branch. Never hand-edit `family.yaml`'s `members:` - block — the tool rewrites it wholesale. +3. **Each of `add`, `bump` and `remove` writes ONE commit** moving the gitlink + and the pin together, on a branch made first (`family-on-tracking-branch` + refuses the tracking branch). Push it as its `NEXT` line says and land it as + a pull request; never suggest a push to the default branch. Never hand-edit + `family.yaml`'s `members:` block, which the tool rewrites wholesale, but for + a row repair `validate-family.py` names; the commit is then the person's. 4. **The member must be scaffolded or adopted FIRST.** A family cannot make a project out of a repository; it can only pin one that already is. 5. **`make siblings` in the holder — or `family.py siblings --family-root diff --git a/templates/family-root/AGENTS-shape.md b/templates/family-root/AGENTS-shape.md index fbc984d9..e731cb50 100644 --- a/templates/family-root/AGENTS-shape.md +++ b/templates/family-root/AGENTS-shape.md @@ -35,13 +35,17 @@ Each writes ONE commit moving the **gitlink** for `members/` and that row's **`members[].pin`** in `family.yaml` together. `make pins` (`scripts/validate-family.py`) refuses when the two disagree, and it is right to: a gitlink moved alone is a pin that names a commit the tree is not at. Do -not hand-edit the `members:` block — the tool rewrites it wholesale. Land the -commit as a pull request: each refuses this holder's tracking branch -(`family-on-tracking-branch`) and names the `git switch -c` that makes a -branch, as the block's first line does, and ends by printing that branch's -`git push`. The member must already be scaffolded or adopted, because a family -cannot make a project out of a repository. A holder keeps `members/` in ONE -spelling on every disk: `add` refuses `family-members-dir-variant` +not hand-edit the `members:` block — the tool rewrites it wholesale — but for a +row repair that `validate-family.py` names in a holder already in a repair +state: put back, as git last had it, a row deleted or whose `path:` was edited +by hand (`git log -p -- family.yaml` shows the change), or take out the row of +a member that is to go (`member-gitlink-absent`); the commit is then the +person's. Land the commit as a pull request: each refuses this holder's +tracking branch (`family-on-tracking-branch`) and names the `git switch -c` +that makes a branch, as the block's first line does, and ends by printing that +branch's `git push`. The member must already be scaffolded or adopted, because +a family cannot make a project out of a repository. A holder keeps `members/` +in ONE spelling on every disk: `add` refuses `family-members-dir-variant` (`member-path-taken` for a variant of the member's own path) until a human renames what sits under another spelling. @@ -96,12 +100,12 @@ holder (`../`). Two copies of every member is correct and intended — `validate`; the sibling is where a human works, and `family.py bump` is how this family follows it. What it will NOT do, and you must not do either: -* **It moves nothing.** When the parent folder is not named after the family, - and no link of that name beside it leads to it, it WARNS, prints the exact - `mv` for a human to run and stops, exit 1 (`make`: 2), before it clones - anything. Do not run that `mv` on your own initiative: relocating a checkout - breaks an open shell, an editor and any LINKED WORKTREE, whose `.git` file - carries an absolute path. +* **It moves nothing.** The parent folder must be named EXACTLY after its + repository, case included, on every disk. When it is not, and no link of that + name beside it leads to it, it WARNS, prints the exact `mv` for a human to + run and stops, exit 1 (`make`: 2), before it clones anything. Do not run that + `mv` on your own initiative: relocating a checkout breaks an open shell, an + editor and any LINKED WORKTREE, whose `.git` file carries an absolute path. * **It never touches an existing clone beyond `git fetch`.** No checkout, no reset, no pull, no stash, no branch switch — somebody is working in there. A directory that is a clone of a different repository is reported and diff --git a/templates/family-root/README.md b/templates/family-root/README.md index 9bfa6aa9..042c409e 100644 --- a/templates/family-root/README.md +++ b/templates/family-root/README.md @@ -69,14 +69,14 @@ that is the name `make siblings` checks the folder against. The display name in `family.yaml` is only a title and names no folder, even when it is spelled differently. -**Nothing ever moves a checkout.** When this holder's parent folder is not -named after the family, and no link of that name beside it leads to it, -`make siblings` WARNS, prints the exact `mv` for a human to run and stops -before it clones anything, so that the run after the move clones each member -once — it moves nothing itself. Relocating a -directory out from under a shell, an editor or an agent lane breaks linked -worktrees, whose `.git` files carry absolute paths, and this standard does -not act by surprise. +**Nothing ever moves a checkout.** This holder's parent folder must be named +EXACTLY after its repository, case included, on every disk. When it is not, and +no link of that name beside it leads to it, `make siblings` WARNS, prints the +exact `mv` for a human to run and stops before it clones anything, so that the +run after the move clones each member once — it moves nothing itself. +Relocating a directory out from under a shell, an editor or an agent lane +breaks linked worktrees, whose `.git` files carry absolute paths, and this +standard does not act by surprise. From a checkout of `{{SHAPE_REPOSITORY}}` the same utility runs against a holder anywhere on disk, which is what to use before this repository is diff --git a/tests/test_family_docs_one_spelling.py b/tests/test_family_docs_one_spelling.py index eced3460..6ad3b1ea 100644 --- a/tests/test_family_docs_one_spelling.py +++ b/tests/test_family_docs_one_spelling.py @@ -25,17 +25,38 @@ The printed move itself is pinned by `test_family_siblings_placement.py` (`~/projects/InkRouter` becomes `~/InkRouter/InkRouter`): this file holds the sentences that describe it. + +THE HOLDER'S OWN DOCS (#763). The holder's README and its pinned +`AGENTS-shape.md` said the parent folder was "not named after the family", +which reads as a match that ignores case, where `siblings.py` compares the +name exactly and has a separate branch for a folder that differs only in +case. Both now say the standard's words: named EXACTLY after its repository, +case included, on every disk. + +THE HAND REPAIR OF A ROW (#763). `AGENTS.md` and the holder's +`AGENTS-shape.md` say never to hand-edit `family.yaml`'s `members:` block, +and `validate-family.py` has two remedies that tell a person to: `RESTORE_ROW` +(a row deleted, or its `path:` edited, by hand is put back as git last had +it) and `member-gitlink-absent`'s (the row of a member that is to go is taken +out by hand). The never is true of the normal path, where `add`, `bump` and +`remove` rewrite the block; the remedies are for a holder already in a repair +state. So each doc keeps its never and names the validator's repair beside +it, and says whose commit that is; the shape doc, which has the room, names +both cases. The validator's remedies are read from the validator, so a remedy +that stops telling the person to edit a row fails here and takes the docs' +exception with it. """ from __future__ import annotations import ast +import functools import html import re import pytest -from conftest import REPO +from conftest import REPO, load_without_bytecode ONE_SPELLING = ( "A holder keeps `members/` in ONE spelling on every disk: `add` refuses " @@ -72,6 +93,45 @@ "templates/family-root/AGENTS-shape.md"] README_AND_HANDBOOK = [README, HANDBOOK] +HOLDER_README = "templates/family-root/README.md" +SHAPE_DOC = "templates/family-root/AGENTS-shape.md" +HOLDER_DOCS = [HOLDER_README, SHAPE_DOC] + +#: What the holder's docs said of the folder before #763: a match that reads +#: as case-insensitive, where `siblings.py` compares the name exactly. +BEFORE_763 = "not named after the family" + +#: The sentence each doc says the normal path in: the tool rewrites the block. +NEVER_HAND_EDIT = { + "AGENTS.md": "Never hand-edit `family.yaml`'s `members:` block", + SHAPE_DOC: "Do not hand-edit the `members:` block", +} + +#: How both docs end the clause that names the validator's repair. +WHOSE_COMMIT = "the commit is then the person's" + +#: The longest a doc may take between its never and whose commit a repair is +#: for the two to be one passage and not a never with a stray phrase far below. +PASSAGE_LIMIT = 700 + +#: The validator's two remedies that tell a person to edit a row by hand, and +#: the words of each that the shape doc takes its two cases from. `RESTORE_ROW` +#: is printed by `member-gitlink-unnamed`; the other is the remedy of +#: `member-gitlink-absent` (#681), whose code the shape doc names. +RESTORE_ROW_CASE = "as git last had it" +ROW_OUT_CASE = "is to go" +GITLINK_ABSENT = "member-gitlink-absent" +VALIDATOR = (REPO / "templates" / "family-root" / "scripts" / + "validate-family.py") + +#: The sentence of each remedy that tells a person to edit a row by hand, as a +#: pattern. `member-gitlink-absent`'s remedy says "by hand" of a `.gitmodules` +#: section too, so the phrase alone would not show that the row's is there. +REMEDY_SENTENCES = { + "RESTORE_ROW": r"edited by hand, put the row back as git last had it", + GITLINK_ABSENT: r"If it is to go, take its row out of \S+ by hand", +} + def shown(path: str) -> str: """The text a reader sees in `path`, on one line. @@ -125,3 +185,87 @@ def test_the_folder_is_named_exactly_after_the_repository(path): @pytest.mark.parametrize("path", README_AND_HANDBOOK) def test_the_move_for_a_holder_straight_in_a_folder_is_one_level_up(path): assert STRAIGHT_IN_A_FOLDER in shown(path) + + +@pytest.mark.parametrize("path", HOLDER_DOCS) +def test_the_holders_own_docs_say_the_folder_is_named_exactly(path): + assert NAMED_EXACTLY in shown(path) + + +@pytest.mark.parametrize("path", HOLDER_DOCS) +def test_the_holders_own_docs_no_longer_say_not_named_after_the_family(path): + assert BEFORE_763 not in shown(path) + + +@functools.cache +def validator(): + """`validate-family.py` as a holder carries it, loaded with no bytecode.""" + return load_without_bytecode(VALIDATOR, "validate_family_hand_repair_docs") + + +def hand_edit_passage(path: str) -> str: + """What `path` says from its never to whose commit a repair is, on one + line: the passage that has to hold the validator's repair, found by the + words of the never and the words that close the clause.""" + said = shown(path) + start = said.find(NEVER_HAND_EDIT[path]) + assert start >= 0, f"{path} no longer says never to hand-edit `members:`" + end = said.find(WHOSE_COMMIT, start) + assert end >= 0, ( + f"{path} says never to hand-edit `members:` and not whose commit " + f"a repair the validator names is ({WHOSE_COMMIT!r})") + passage = said[start:end + len(WHOSE_COMMIT)] + assert len(passage) <= PASSAGE_LIMIT, ( + f"{path}: the never and {WHOSE_COMMIT!r} are {len(passage)} " + f"characters apart, more than one passage") + return passage + + +@pytest.mark.parametrize("path", NEVER_HAND_EDIT) +def test_each_doc_keeps_its_never_for_the_tool_rewriting_the_block(path): + assert "wholesale" in hand_edit_passage(path) + + +@pytest.mark.parametrize("path", NEVER_HAND_EDIT) +def test_the_never_names_the_repair_the_validator_makes_of_a_row(path): + passage = hand_edit_passage(path) + assert "validate-family.py" in passage + assert "repair" in passage + + +@pytest.mark.parametrize("case", [RESTORE_ROW_CASE, ROW_OUT_CASE, + GITLINK_ABSENT]) +def test_the_shape_doc_names_both_of_the_validators_cases(case): + assert case in hand_edit_passage(SHAPE_DOC) + + +@pytest.fixture +def remedies(tmp_path, monkeypatch) -> dict[str, str]: + """The text of the validator's two remedies, as it prints them: the + constant `RESTORE_ROW`, and what `member-gitlink-absent` is given for a + row whose gitlink is not there (no repository is needed: HEAD's + `.gitmodules` is then unreadable, and the remedy writes the section).""" + monkeypatch.setenv("GIT_CEILING_DIRECTORIES", str(tmp_path.parent)) + module = validator() + put_back = module._gitlink_put_back(tmp_path, "members/IRQS", "0" * 40, + "acme/IRQS") + return {"RESTORE_ROW": module.RESTORE_ROW, + GITLINK_ABSENT: put_back} + + +@pytest.mark.parametrize("remedy, case", [("RESTORE_ROW", RESTORE_ROW_CASE), + (GITLINK_ABSENT, ROW_OUT_CASE)]) +def test_each_remedy_still_tells_the_person_to_edit_a_row_by_hand( + remedies, remedy, case): + text = " ".join(remedies[remedy].split()) + assert re.search(REMEDY_SENTENCES[remedy], text), ( + f"{remedy} no longer tells a person to edit a row by hand: the " + "docs' exception for the validator's repair goes with it") + assert case in text + + +def test_member_gitlink_absent_is_a_code_the_validator_prints(): + source = VALIDATOR.read_text(encoding="utf-8") + held = {node.value for node in ast.walk(ast.parse(source)) + if isinstance(node, ast.Constant) and isinstance(node.value, str)} + assert GITLINK_ABSENT in held From 0e1e0a8da5de7875b7b6f78afba56b27ff829012 Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Sat, 17 Oct 2026 12:49:46 +0000 Subject: [PATCH 14/15] family-root AGENTS-shape.md and tests/test_family_docs_one_spelling.py: the shape doc names the layout's folder after this holder's repository and the rule's subject as "This holder's parent folder", and the hand-edit passages, the folder rule, `member-gitlink-absent`'s raised refusal and `RESTORE_ROW`'s inline copy are read whole, so a one-word edit that puts the contradiction back fails a test (#763) Fix round 1 of #822, on rv763's review (NOT READY: 14 of 19 one-line mutants survived the first tests, two of them putting the issue's contradiction back). The sentences were true; the tests looked for words, not meaning. templates/family-root/AGENTS-shape.md, words only, in the pinned copy that already costs every holder one `upstream-changed` row: - the layout paragraph said "a plain folder named after the family". For `--family Quill --name QuillWorks` that reads as the display name. It now says "a plain folder named after this holder's repository", which the holder's README says is what `{{FAMILY}}` is. - the folder rule said "The parent folder must be named EXACTLY after its repository", where the folder is no repository and "its" has nothing to refer to. It now opens "This holder's parent folder must be named EXACTLY after its repository", as the standard's README says "The holder's parent folder". tests/test_family_docs_one_spelling.py: - each doc's hand-edit passage, from "Never hand-edit" or "Do not hand-edit" through "the commit is then the person's.", is held whole (`PASSAGES`), as `ONE_SPELLING` is. The old tests looked for loose words between the two ends, and "but for a row repair" turned into "not even for a row repair" passed them. - `FOLDER_RULE`, "parent folder must be named EXACTLY after its repository, case included, on every disk", is read in the standard's README, the handbook, the holder's README and AGENTS-shape.md, so a "need not be" or a "should be" fails; the shape doc's layout wording is held. - `member-gitlink-absent` is raised through `_check_member_gitlink` and its code and its remedy are read together, where the old test read a helper and asked whether the code appeared anywhere in the file. That drops the second copy of #738's set of string constants. - the inline copy of `RESTORE_ROW`'s words in `gitmodules-path-unnamed` is read against the constant, so the two cannot say different things. Lands after #738 and #747 (v1.1.0). Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Sonnet 5.5 --- templates/family-root/AGENTS-shape.md | 21 ++-- tests/test_family_docs_one_spelling.py | 161 ++++++++++++++----------- 2 files changed, 102 insertions(+), 80 deletions(-) diff --git a/templates/family-root/AGENTS-shape.md b/templates/family-root/AGENTS-shape.md index e731cb50..ac912591 100644 --- a/templates/family-root/AGENTS-shape.md +++ b/templates/family-root/AGENTS-shape.md @@ -93,19 +93,20 @@ the estate is parked because the command finished. **Nothing under a member's `.git` file carries an absolute path, which is machine plumbing and not the family's business. -`make siblings` places the WORKSTATION layout: a plain folder named after the -family, this holder inside it, and each member's working clone beside the -holder (`../`). Two copies of every member is correct and intended — -`members/` here is pinned and DETACHED, for `bootstrap` and +`make siblings` places the WORKSTATION layout: a plain folder named after this +holder's repository, this holder inside it, and each member's working clone +beside the holder (`../`). Two copies of every member is correct and +intended — `members/` here is pinned and DETACHED, for `bootstrap` and `validate`; the sibling is where a human works, and `family.py bump` is how this family follows it. What it will NOT do, and you must not do either: -* **It moves nothing.** The parent folder must be named EXACTLY after its - repository, case included, on every disk. When it is not, and no link of that - name beside it leads to it, it WARNS, prints the exact `mv` for a human to - run and stops, exit 1 (`make`: 2), before it clones anything. Do not run that - `mv` on your own initiative: relocating a checkout breaks an open shell, an - editor and any LINKED WORKTREE, whose `.git` file carries an absolute path. +* **It moves nothing.** This holder's parent folder must be named EXACTLY after + its repository, case included, on every disk. When it is not, and no link of + that name beside it leads to it, it WARNS, prints the exact `mv` for a human + to run and stops, exit 1 (`make`: 2), before it clones anything. Do not run + that `mv` on your own initiative: relocating a checkout breaks an open shell, + an editor and any LINKED WORKTREE, whose `.git` file carries an absolute + path. * **It never touches an existing clone beyond `git fetch`.** No checkout, no reset, no pull, no stash, no branch switch — somebody is working in there. A directory that is a clone of a different repository is reported and diff --git a/tests/test_family_docs_one_spelling.py b/tests/test_family_docs_one_spelling.py index 6ad3b1ea..c52ac67e 100644 --- a/tests/test_family_docs_one_spelling.py +++ b/tests/test_family_docs_one_spelling.py @@ -42,9 +42,13 @@ `remove` rewrite the block; the remedies are for a holder already in a repair state. So each doc keeps its never and names the validator's repair beside it, and says whose commit that is; the shape doc, which has the room, names -both cases. The validator's remedies are read from the validator, so a remedy -that stops telling the person to edit a row fails here and takes the docs' -exception with it. +both cases. Each passage is held WHOLE (`PASSAGES`), as `ONE_SPELLING` is: the +words a never needs to be an exception, "but for a row repair", are the ones a +one-word edit turns back into the contradiction. The validator's remedies are +read from the validator, `member-gitlink-absent` as the refusal it raises, so +a remedy that stops telling the person to edit a row fails here and takes the +docs' exception with it, and so does a copy of `RESTORE_ROW`'s words that no +longer says what the constant does. """ from __future__ import annotations @@ -97,23 +101,43 @@ SHAPE_DOC = "templates/family-root/AGENTS-shape.md" HOLDER_DOCS = [HOLDER_README, SHAPE_DOC] +#: How the shape doc's layout paragraph names the folder (#763): after the +#: holder's repository, which the holder's README says at `:66-70` is what +#: `{{FAMILY}}` is, and not "the family", which `--name` makes a display name. +SHAPE_LAYOUT = "a plain folder named after this holder's repository" + #: What the holder's docs said of the folder before #763: a match that reads #: as case-insensitive, where `siblings.py` compares the name exactly. BEFORE_763 = "not named after the family" -#: The sentence each doc says the normal path in: the tool rewrites the block. -NEVER_HAND_EDIT = { - "AGENTS.md": "Never hand-edit `family.yaml`'s `members:` block", - SHAPE_DOC: "Do not hand-edit the `members:` block", +#: The folder rule's subject and its "must", with the words every place says +#: it in: the standard's README and the handbook, and the holder's README and +#: `AGENTS-shape.md` (#763). "named EXACTLY ..." alone would pass a doc that +#: said the folder "need not be" named so. +FOLDER_RULE = "parent folder must be " + NAMED_EXACTLY + +#: Each doc's whole passage on hand-editing `members:`, from its never to +#: whose commit a repair is (#763). A WHOLE SENTENCE, as `ONE_SPELLING` is: the +#: never is an exception, for a row repair `validate-family.py` names, and a +#: doc that kept every word of it but turned the "but for" into "not even for", +#: or the repair into another thing, would say what the validator's remedies +#: say not to do. `AGENTS.md` is at its line cap, so its passage is the +#: shorter; the shape doc names both of the validator's cases. +PASSAGES = { + "AGENTS.md": ( + "Never hand-edit `family.yaml`'s `members:` block, which the tool " + "rewrites wholesale, but for a row repair `validate-family.py` " + "names; the commit is then the person's."), + SHAPE_DOC: ( + "Do not hand-edit the `members:` block — the tool rewrites it " + "wholesale — but for a row repair that `validate-family.py` names " + "in a holder already in a repair state: put back, as git last had " + "it, a row deleted or whose `path:` was edited by hand (`git log -p " + "-- family.yaml` shows the change), or take out the row of a member " + "that is to go (`member-gitlink-absent`); the commit is then the " + "person's."), } -#: How both docs end the clause that names the validator's repair. -WHOSE_COMMIT = "the commit is then the person's" - -#: The longest a doc may take between its never and whose commit a repair is -#: for the two to be one passage and not a never with a stray phrase far below. -PASSAGE_LIMIT = 700 - #: The validator's two remedies that tell a person to edit a row by hand, and #: the words of each that the shape doc takes its two cases from. `RESTORE_ROW` #: is printed by `member-gitlink-unnamed`; the other is the remedy of @@ -187,9 +211,9 @@ def test_the_move_for_a_holder_straight_in_a_folder_is_one_level_up(path): assert STRAIGHT_IN_A_FOLDER in shown(path) -@pytest.mark.parametrize("path", HOLDER_DOCS) -def test_the_holders_own_docs_say_the_folder_is_named_exactly(path): - assert NAMED_EXACTLY in shown(path) +@pytest.mark.parametrize("path", README_AND_HANDBOOK + HOLDER_DOCS) +def test_the_four_places_say_the_folder_rule_in_the_same_words(path): + assert FOLDER_RULE in shown(path) @pytest.mark.parametrize("path", HOLDER_DOCS) @@ -197,75 +221,72 @@ def test_the_holders_own_docs_no_longer_say_not_named_after_the_family(path): assert BEFORE_763 not in shown(path) +def test_the_shape_docs_layout_names_the_folder_after_the_repository(): + assert SHAPE_LAYOUT in shown(SHAPE_DOC) + + @functools.cache def validator(): """`validate-family.py` as a holder carries it, loaded with no bytecode.""" return load_without_bytecode(VALIDATOR, "validate_family_hand_repair_docs") -def hand_edit_passage(path: str) -> str: - """What `path` says from its never to whose commit a repair is, on one - line: the passage that has to hold the validator's repair, found by the - words of the never and the words that close the clause.""" - said = shown(path) - start = said.find(NEVER_HAND_EDIT[path]) - assert start >= 0, f"{path} no longer says never to hand-edit `members:`" - end = said.find(WHOSE_COMMIT, start) - assert end >= 0, ( - f"{path} says never to hand-edit `members:` and not whose commit " - f"a repair the validator names is ({WHOSE_COMMIT!r})") - passage = said[start:end + len(WHOSE_COMMIT)] - assert len(passage) <= PASSAGE_LIMIT, ( - f"{path}: the never and {WHOSE_COMMIT!r} are {len(passage)} " - f"characters apart, more than one passage") - return passage - - -@pytest.mark.parametrize("path", NEVER_HAND_EDIT) -def test_each_doc_keeps_its_never_for_the_tool_rewriting_the_block(path): - assert "wholesale" in hand_edit_passage(path) - - -@pytest.mark.parametrize("path", NEVER_HAND_EDIT) -def test_the_never_names_the_repair_the_validator_makes_of_a_row(path): - passage = hand_edit_passage(path) - assert "validate-family.py" in passage - assert "repair" in passage +@pytest.mark.parametrize("path", PASSAGES) +def test_each_doc_says_its_hand_edit_passage_whole(path): + assert PASSAGES[path] in shown(path) @pytest.mark.parametrize("case", [RESTORE_ROW_CASE, ROW_OUT_CASE, GITLINK_ABSENT]) -def test_the_shape_doc_names_both_of_the_validators_cases(case): - assert case in hand_edit_passage(SHAPE_DOC) +def test_the_shape_doc_passage_names_both_of_the_validators_cases(case): + assert case in PASSAGES[SHAPE_DOC] @pytest.fixture -def remedies(tmp_path, monkeypatch) -> dict[str, str]: - """The text of the validator's two remedies, as it prints them: the - constant `RESTORE_ROW`, and what `member-gitlink-absent` is given for a - row whose gitlink is not there (no repository is needed: HEAD's - `.gitmodules` is then unreadable, and the remedy writes the section).""" +def refusal(tmp_path, monkeypatch): + """The refusal the validator raises for a row whose gitlink is not there, + through `_check_member_gitlink`, which gives it its code and its remedy + (no repository is needed: HEAD is then unlistable, `recorded_gitlink` + finds none and the remedy writes the `.gitmodules` section).""" monkeypatch.setenv("GIT_CEILING_DIRECTORIES", str(tmp_path.parent)) module = validator() - put_back = module._gitlink_put_back(tmp_path, "members/IRQS", "0" * 40, - "acme/IRQS") - return {"RESTORE_ROW": module.RESTORE_ROW, - GITLINK_ABSENT: put_back} - - -@pytest.mark.parametrize("remedy, case", [("RESTORE_ROW", RESTORE_ROW_CASE), - (GITLINK_ABSENT, ROW_OUT_CASE)]) -def test_each_remedy_still_tells_the_person_to_edit_a_row_by_hand( - remedies, remedy, case): - text = " ".join(remedies[remedy].split()) - assert re.search(REMEDY_SENTENCES[remedy], text), ( - f"{remedy} no longer tells a person to edit a row by hand: the " + with pytest.raises(module.Refusal) as caught: + module._check_member_gitlink(tmp_path, "members/IRQS", "0" * 40, + "IRQS", "acme/IRQS", {}, None, "rule") + return caught.value + + +def test_member_gitlink_absent_is_raised_telling_the_person_to_edit_the_row( + refusal): + assert refusal.code == GITLINK_ABSENT + text = " ".join(str(refusal).split()) + assert re.search(REMEDY_SENTENCES[GITLINK_ABSENT], text), ( + f"{GITLINK_ABSENT} no longer tells a person to edit a row by hand: " + "the docs' exception for the validator's repair goes with it") + assert ROW_OUT_CASE in text + + +def test_restore_row_still_tells_the_person_to_edit_a_row_by_hand(): + text = " ".join(validator().RESTORE_ROW.split()) + assert re.search(REMEDY_SENTENCES["RESTORE_ROW"], text), ( + "RESTORE_ROW no longer tells a person to edit a row by hand: the " "docs' exception for the validator's repair goes with it") - assert case in text + assert RESTORE_ROW_CASE in text -def test_member_gitlink_absent_is_a_code_the_validator_prints(): +def test_every_inline_copy_of_restore_rows_words_is_what_the_constant_says(): + """`gitmodules-path-unnamed` spells the first words of `RESTORE_ROW` out in + its own text, not through the constant, so a change to one that missed the + other would send a person two ways. Each string the validator holds that + begins as `RESTORE_ROW` does is read against it.""" source = VALIDATOR.read_text(encoding="utf-8") - held = {node.value for node in ast.walk(ast.parse(source)) - if isinstance(node, ast.Constant) and isinstance(node.value, str)} - assert GITLINK_ABSENT in held + constant = " ".join(validator().RESTORE_ROW.split()) + opening = constant[:len("If its row was deleted")] + copies = [" ".join(node.value.split()) + for node in ast.walk(ast.parse(source)) + if isinstance(node, ast.Constant) + and isinstance(node.value, str) + and node.value.strip().startswith(opening)] + assert len(copies) >= 2, "the constant and its inline copy" + for copy in copies: + assert constant.startswith(copy.rstrip(".")) From ea86cb60cf6cfd94ae455024058dd17b38ac630d Mon Sep 17 00:00:00 2001 From: Brett Heap Date: Sat, 17 Oct 2026 13:06:59 +0000 Subject: [PATCH 15/15] tests/test_family_docs_one_spelling.py: the folder rule is read with its owner, "holder's parent folder must be named EXACTLY after its repository, case included, on every disk", in the standard's README, the handbook and both holder docs, so a rule given to "A member's" or to "The parent folder" fails (#763) Fix round 2 of #822, on rv763's NIT 9. `FOLDER_RULE` began at "parent folder must be", so a doc that gave the rule to another subject, "A member's parent folder" or "The parent folder" (which, in a doc about a plain folder that is no repository, leaves "its repository" nothing to refer to), still passed. It now begins at "holder's parent folder must be". All four places carry that text: "The holder's" in README.md, "holder’s" in the handbook (which `shown()` reads as "holder's") and "This holder's" in the holder's README and AGENTS-shape.md. Words in the tests only; no doc, tool or remedy changes. Lands after #738 and #747 (v1.1.0). Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_family_docs_one_spelling.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/tests/test_family_docs_one_spelling.py b/tests/test_family_docs_one_spelling.py index c52ac67e..e69575a9 100644 --- a/tests/test_family_docs_one_spelling.py +++ b/tests/test_family_docs_one_spelling.py @@ -110,11 +110,12 @@ #: as case-insensitive, where `siblings.py` compares the name exactly. BEFORE_763 = "not named after the family" -#: The folder rule's subject and its "must", with the words every place says -#: it in: the standard's README and the handbook, and the holder's README and -#: `AGENTS-shape.md` (#763). "named EXACTLY ..." alone would pass a doc that -#: said the folder "need not be" named so. -FOLDER_RULE = "parent folder must be " + NAMED_EXACTLY +#: The folder rule's owner, its subject and its "must", with the words every +#: place says it in: the standard's README and the handbook, and the holder's +#: README and `AGENTS-shape.md` (#763). "named EXACTLY ..." alone would pass a +#: doc that said the folder "need not be" named so, and "parent folder must be" +#: one that gave it to a member's. +FOLDER_RULE = "holder's parent folder must be " + NAMED_EXACTLY #: Each doc's whole passage on hand-editing `members:`, from its never to #: whose commit a repair is (#763). A WHOLE SENTENCE, as `ONE_SPELLING` is: the