Skip to content

validate-family.py: a gitlink in members/ or a .gitmodules section whose path no row names is a finding in every mode, a hand-edited path: is member-path with nothing read through it and recorded_gitlink reads its path literally, and no git to run refuses git-not-runnable instead of a traceback that exits 1 (#361, #367, #368) - #406

Merged
brettheap merged 3 commits into
mainfrom
fix/validate-family-unnamed-mounts
Oct 12, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Lane: openreposhape-4 (openRepoShape-4)
Claim: #361 (comment)
Claim: #367 (comment)
Claim: #368 (comment)

Three defects in the holder's own gate, templates/family-root/scripts/validate-family.py, fixed in one change because all three are in _check_members and the member checks under it. The third also reaches the shared scripts/repo_shape.py.

#361: a gitlink or a .gitmodules section that no row names

What was wrong. Every member check started from a row. _check_members asked each members: row for its gitlink, digest and identity, and nothing walked back from what the holder records to a row. These all passed the holder's own gate, and the doctor called the holder COMPLIANT:

What it does now. Two findings, each with its own code. Both are made before the --no-members return, as #350's member-duplicate is, because a private family's CI falls back to that mode.

code what it reads
member-gitlink-unnamed every gitlink in members/, the directory in any case (Members/Zed counts, tools/Zed does not). It reads the whole index at stage 0, else HEAD, the order recorded_gitlink reads one in, with no pathspec and -z.
gitmodules-path-unnamed every .gitmodules section whose path no row names. It shares one reading of the file, as git reads it, with #300's gitmodules-path-twice.

"Unnamed" is decided exactly, against the path each row gives: its path: as written, or members/<project>. A path a row names under str.casefold() is still a finding, because macOS and Windows hold the two as one directory. Which of three things it is decides what the finding says to run (fix round 1, below). family.py add's member-path-registered is unchanged (#362).

Remedies. Each command is printed on its own indented line, with no backticks, no ; and no trailing comment.

what was found the remedy printed
an unnamed section taken out by key, #300's removal, one git config -f .gitmodules --unset-all per key it holds. None when its gitlink's git rm below takes it out too (fix round 1).
a gitlink or section that is a SECOND path to a member that stands: the row's path: is its canonical members/<project> and that exact path holds one too git rm --cached -- <path>, which touches no file (a plain git rm would delete the member's own checkout on a disk that folds case), or the --unset-all lines
the member's own path, whose row was hand-edited into a case variant of it no command: the row is the hand edit, so put members/<project> back in its path: (fix round 1)
the only record of a member, at a case variant of its row's path no command: the member is recorded in the wrong case, and the git log -p -- family.yaml restore sentence (fix round 1)
any other unnamed gitlink put the row back from git (git log -p -- family.yaml), or, for a hand mount: git rm -- <path>, then python3 scripts/family.py add --family-root <holder> --member <org>/<Project> from a checkout of openRepoShape, the project shell-quoted as the paths are

This departs from the ruling's git rm --cached. Probed on the issue's repro row (a): after git rm --cached members/Zed, the checkout stays at the mount path, and add refuses REFUSED member-path-taken: members/Zed already exists in the family root but no `members:` row names it. A plain git rm members/Zed works instead:

  • it refuses while the checkout holds changes, and takes the .gitmodules section out with it;
  • add then refuses member-git-dir-cached, naming the object store git rm leaves in .git/modules (the remedy says it will);
  • with that store gone, add mounts and pins the member, and the holder says family ok.

test_a_hand_mount_is_taken_out_and_added_back_as_printed runs that chain as printed.

Smaller decisions.

  • A new refusal, gitlinks-unreadable, for an index git ran and could not list. The gitlink listing is the one check of members/ that --no-members runs, so a failed listing must not pass silently. A HEAD git cannot list records nothing, as recorded_gitlink reads it (a holder before its first commit).
  • The doctor is not changed (ruling 4): its family row reports what this validator finds.

#367: a hand-edited path: was read as a glob

What was wrong. _check_member found the row's path: wrong (member-path), then checked the gitlink, tree and identity at that path anyway. At members/*:

  1. the gitlink listing read the path as a glob and answered with a neighbour's commit, a false member-gitlink-mismatch;
  2. the tree check refused member-uninitialized, whose remedy cannot help;
  3. the refusal dropped both findings unprinted.

What it does now.

  • The row's checks stop once its pin is read; the pin is a fact of the row itself. member-path now names the one path the row may have.
  • recorded_gitlink passes git's global --literal-pathspecs to both of its listings. It never uses :(literal) magic, which a caller's GIT_LITERAL_PATHSPECS=1 turns off.
  • Beyond the ruling: git refuses --literal-pathspecs beside an exported GIT_GLOB_PATHSPECS or GIT_ICASE_PATHSPECS: "global 'literal' pathspec setting is incompatible with all other global pathspec settings", exit 128 (probed on git 2.43). recorded_gitlink reads a refused listing as "no gitlink here", so an operator exporting either variable would have seen every gate refuse a gitlink that is there. Those two variables are now kept out of those two git calls' environment; git_out gained an optional env for this.

One copy of repo_shape.py: scripts/repo_shape.py. The templates carry none. scripts/shape_materialize.py copies it into projects (COPIED_FROM_SHAPE) and holders (FAMILY_COPIED_FROM_SHAPE), run by scaffold-project.py and family.py init, so the fix reaches every caller of recorded_gitlink:

  • validate-family.py;
  • the assembly root's validate-pins.py (187) and bootstrap.py (114);
  • bump-leg.py (740);
  • shape-doctor.py, at three call sites (line numbers as on main 45e01f4, and the same at cb7063f): family_beside_project (1462), check_legs, the legs row (2167), and members_verdict, the members row (3802).

family.py (107) imports recorded_gitlink but never calls it. Every caller passes a declared path, and none relies on globbing. Existing projects and holders get the fix through update-shape.py.

Decision 3 is not taken. A refusal still prints none of the findings gathered before it.

#368: no git on PATH ended in a traceback, exit 1

What was wrong. git_out raises OSError when there is no git to start, and only #317's .gitmodules reader caught it. The default and --pins modes ended in a FileNotFoundError traceback, exit 1, which is a finding's code.

What it does now.

Not fixed here: the assembly root's validate-pins.py ends in the same traceback (the issue's last line). It is #412, fixed in PR #446 with the same git-not-runnable code.

Also

_check_members's docstring said "the two checks that read a file alone" where #317 had made three. It now names the checks that need no member checked out, without a count (the nit from the review of #331, PR #350).

No --help text changed, so docs/cli.md is unchanged.

Fix round 1 (review vf-423fb25)

The review found each issue's repro fixed, the five departures right and the decisive run green (1455 passed, 4 skipped on the scratch merge). It held the head on one remedy that misdirects and left four nits. Head after this round: 369af2e, one commit on 423fb25.

finding what was done
SHOULD-1: the case-variant remedy done: the three cases below, with tests in default mode and --no-members
NIT-1: a hand mount's two findings print overlapping remedies done: the section finding says the git rm above takes it out and prints no --unset-all line
NIT-2: the git-not-runnable wrapper catches more than git not done, as ruled: Follow-ups below
NIT-3: the stage-0 filter in _member_gitlinks is unpinned done: two tests with an unmerged gitlink
NIT-4: PR body accuracy done: the caller list above (the doctor's members row, and the three doctor call sites with lines) and the copier (shape_materialize.py); the --member value in the add remedy is shell-quoted, with shlex.quote, the file's own helper

SHOULD-1. A path that folds to a row's path was always taken for the stray, and the finding printed git rm --cached -- <path> (or the --unset-all lines). That is right in one state. In two others it took the member's own record out of the index (or .gitmodules), after which --no-members, a private family's CI fallback, said family ok. Now _variant_case tells the three apart (_row_paths carries each row's canonical members/<project> for it):

  1. A second path to a member that stands: the row's path: is its canonical path AND that exact path also holds a gitlink (a section). Today's command, unchanged.
  2. The member's own path, the row hand-edited into a case variant of it (path: Members/IRSS over a mount at members/IRSS): the finding says the row is the hand edit and to put members/IRSS back in the row's path:. It prints no command. Under --no-members no member-path finding says so, which is why this one must.
  3. Otherwise (a canonical row whose only gitlink or section is at a case variant): the member is recorded in the wrong case, with the git log -p -- family.yaml restore sentence the unnamed gitlink always had. No command.

Decisions the ruling left open:

  • "Otherwise" also takes a row that is itself hand-edited with a gitlink at a THIRD spelling (Members/IRSS row, gitlink at MEMBERS/IRSS), and a row with no project:. Nothing says either is a second path to a member that stands, so neither is told to remove it. The third case says "could un-register it" for that reason.
  • The section finding's restore sentence carries the git log -p -- family.yaml parenthesis, as the brief names it. The section's non-variant text, which has none, is unchanged.
  • A row's canonical path is members/<project> for a row with a project:, and unknown for one without.

NIT-1. git rm -- <path> takes the .gitmodules section out too, so the section's --unset-all lines exit 5 after it. Where two sections give one path (gitmodules-path-twice), git rm removes only the one git reads for it, the last in the file (probed on git 2.43, and the test asserts one section is left), so each of those keeps its own lines.

NIT-3. A gitlink at stages 1-3 only (made with update-index --index-info bytes) is no gitlink the holder records, and one HEAD records and the index holds only unmerged is named at HEAD's commit, not at a stage's.

How the state tests hold on a case-insensitive disk (macOS, Windows). None puts a second directory on disk. A gitlink is only an index record (update-index --index-info bytes, git rm --cached) and a commit of the index, which reads no directory, so the checkout at members/IRSS stays the one directory it is while the index says members/irss. The .gitmodules edit is hashed into git's objects and the same bytes are written to disk. The one hand edit, family.yaml, is a single file written as bytes. Members checked, state (ii) is the refusal member-gitlink-absent on a git that matches a pathspec exactly, which prints no finding (decision 3 of #367 is not taken), so only --no-members shows its finding. A git that matched the path in any case would find the variant and print the finding, so the test accepts either, and asserts that no command is printed in both. The macOS and Windows legs were not run here: CI is the gate.

Tests

One new file per issue. Every state is built in git's index and objects, with bytes on stdin, except one test that runs the remedy chain on a real git submodule add.

Local targeted run. Linux, git 2.43.0, /usr/bin/python3 -m pytest -q, 16 files: the three new ones, test_validate_family_duplicates.py, test_family.py, test_family_add_refusals.py, test_family_member_path.py, test_family_add_excludes.py, test_family_dry_run.py, test_shape_doctor.py (the doctor calls recorded_gitlink), test_bump_leg.py, test_bump_leg_rollback.py, test_pin_validator.py, test_scaffold_e2e.py (the assembly root's validate-pins.py and bootstrap.py), test_repo_hygiene.py and test_template_values_single_pass.py.

Fix round 1 runs (Linux, git 2.43.0, /usr/bin/python3 -m pytest -q, committed bytes, sha256 recorded first):

  • the six files of the brief's command that exist on this branch, plus test_repo_hygiene.py and test_template_values_single_pass.py: 472 passed;
  • the brief's seven-file command, test_agents_md_codes.py included, on a scratch merge of this head onto main cb7063f (that test landed on main after this branch's base): 305 passed.

CI's three OSes are the gate.

Mutants, each run in a scratch worktree at 5eba7d5, whose code and mutant test files are byte-identical to the pushed head's, against the files named:

mutant tests result
the #361 checks moved after the --no-members return test_validate_family_unnamed_mounts.py killed: 11 failed, 13 passed
--literal-pathspecs removed from both listings test_validate_family_member_path_glob.py killed: 6 failed, 15 passed
the OSError clause in _git_runs removed test_validate_family_without_git.py + #317's no-git test killed: 14 failed, 1 passed
the member-path skip removed (the row goes on at its path) test_validate_family_member_path_glob.py killed: 11 failed, 10 passed
the environment scrub removed test_validate_family_member_path_glob.py -k pathspec_setting killed: 2 failed, 2 passed

Fix round 1 mutants, each run against tests/test_validate_family_unnamed_mounts.py at this head:

mutant result
SHOULD-1 reverted to the old branch (_variant_case always the stray) killed: 8 failed, 31 passed. Both state tests among them: state (i) in all three modes, state (ii) under --no-members (members checked it is a refusal, which prints no finding), the section state and the third spelling.
the stage-0 filter dropped from _member_gitlinks killed: 2 failed, 37 passed, both unmerged-gitlink tests
NIT-1's --unset-all lines printed again killed: 4 failed, 35 passed (the hand mount in three modes, and the real-mount chain)
the row-edited case never detected killed: 3 failed
NIT-1's suppression kept for two sections for one path killed: 1 failed
the --member value unquoted killed: 2 failed
the stray case without the exact path held killed: 4 failed

Follow-ups, not done here

Closes #361
Closes #367
Closes #368

🤖 Generated with Claude Code

Summary by Sourcery

Strengthen family validation by auditing recorded members in reverse, treating member paths literally, and converting unavailable Git into explicit refusals.

New Features:

  • Detect unregistered gitlinks under members/ and .gitmodules sections whose paths are not named by family rows, including case-variant paths, in all validation modes.

Bug Fixes:

  • Prevent hand-edited member paths from being interpreted as glob patterns or triggering misleading downstream checks.
  • Report missing or non-runnable Git as a structured git-not-runnable refusal instead of an uncaught traceback.
  • Preserve correct index-versus-HEAD gitlink resolution and handle unmerged index entries without false findings.

Enhancements:

  • Make recorded gitlink lookups literal and robust against conflicting global Git pathspec environment settings.
  • Provide actionable, state-aware remediation guidance for unnamed gitlinks and submodule sections.
  • Apply the shared gitlink lookup fixes to all copied shape-tool callers.

Tests:

  • Add coverage for unnamed mounts and sections, case-variant paths, literal path handling, unmerged gitlinks, remediation commands, and missing Git across validation modes.

…n whose path no row names is a finding in every mode, a hand-edited `path:` is `member-path` with nothing read through it and `recorded_gitlink` reads its path literally, and no git to run refuses `git-not-runnable` instead of a traceback that exits 1 (#361, #367, #368)

#361. Every member check started from a row: `_check_members` asked each
`members:` row for its gitlink, digest and identity, and nothing walked back
from what the holder records to a row. A member mounted by hand with `git
submodule add`, a row deleted by hand, the section `family.py remove` leaves
of two for one path (#360), and a section for `Members/Atlas` that `add`
commits beside the member's own all passed the holder's own gate, and the
doctor called the holder COMPLIANT. Two findings now, each with its own
code, both made before the `--no-members` return, as #350's
`member-duplicate` is, because a private family's CI falls back to that
mode:
- `member-gitlink-unnamed`: every gitlink in `members/` (the directory in
  any case, so `Members/Zed` counts and `tools/Zed` does not), listed from
  the whole index at stage 0, else HEAD, the order `recorded_gitlink` reads
  one in, with no pathspec and `-z`. A HEAD git cannot list records
  nothing, as `recorded_gitlink` reads it; an index git cannot list is a
  new refusal, `gitlinks-unreadable`, since this is the one check of
  `members/` that `--no-members` runs.
- `gitmodules-path-unnamed`: every section whose path no row names, read
  from `.gitmodules` as git reads it, in one reading shared with #300's
  `gitmodules-path-twice` (`_gitmodules_sections`, which now keeps every
  key a section holds, for the remedy).
"Unnamed" is decided EXACTLY against the path each row gives, its `path:`
as written or `members/<project>`. A path a row names under `str.casefold()`
is still a finding, and says it is the case variant of that row's path,
since macOS and Windows hold the two as one directory. `family.py add`'s
`member-path-registered` is unchanged (#362).

The remedies print each command on its own indented line, with no
backticks, no `;` and no trailing comment. A section is taken out by key,
#300's removal, every key it holds. A gitlink a row names in another case
is taken out with `git rm --cached`, which touches no file, because a plain
`git rm` would take the member's own checkout with it where case is folded.
Any other unnamed gitlink is either a row to put back from git, or a hand
mount taken out with a plain `git rm` and mounted with `family.py add`.
NOT `git rm --cached`, the remedy the ruling named: it leaves the checkout
at the mount path, and `add` then refuses `member-path-taken` (probed).
After a plain `git rm`, which refuses while the checkout holds changes and
takes its section out too, `add` names the object store `git rm` leaves in
`.git/modules` (`member-git-dir-cached`), and once that is gone it mounts
and pins the member. A test runs that chain as printed.

#367. `_check_member` found a hand-edited `path:` wrong, recorded
`member-path`, and then checked the gitlink, tree and identity AT that
path. At `members/*` the gitlink listing read a glob and answered with a
neighbour's commit (a false `member-gitlink-mismatch`), the tree check
refused `member-uninitialized`, whose remedy cannot help, and the refusal
dropped both findings unprinted. Now the row's checks stop once its pin is
read, the pin being a fact of the row itself, and `member-path` names the
one path the row may have. `recorded_gitlink` passes git's global
`--literal-pathspecs` to both of its listings (`GITLINK_LISTINGS`), never
`:(literal)` magic, which a caller's `GIT_LITERAL_PATHSPECS=1` turns off.
git refuses that option beside an exported `GIT_GLOB_PATHSPECS` or
`GIT_ICASE_PATHSPECS` (exit 128), and `recorded_gitlink` reads a refused
listing as "no gitlink here", so those two variables are kept out of those
two calls' environment (`git_out` takes an `env` now), or an operator who
exports one would see every gate refuse a gitlink that is there. There is
ONE `repo_shape.py`, `scripts/repo_shape.py`: the templates carry none, and
`scaffold-project.py` and `family.py init` copy it in (`REPO_SHAPE` in
`COPIED_FROM_SHAPE` and `FAMILY_COPIED_FROM_SHAPE`), so the fix reaches
`validate-family.py`, the assembly root's `validate-pins.py` and
`bootstrap.py`, `bump-leg.py` and `shape-doctor.py`, its callers. The
issue's decision 3, a refusal that prints the findings gathered before it,
is not taken.

#368. `git_out` raises `OSError` when there is no git to start, and only
#317's `.gitmodules` reader caught it, so the default and `--pins` modes
ended in a `FileNotFoundError` traceback, exit 1, a finding's code. The
validator now runs every git-reading check inside `_git_runs`: the
`.gitmodules` reader, #361's gitlink listing, each member's gitlink and
its tree. It refuses ONE new code, `git-not-runnable`, exit 2, naming the
check and that git could not be run. #317's no-git clause moves to that
code too, so `gitmodules-unreadable` means only a file git ran and could
not read. `recorded_gitlink`'s `except Refusal` lets the `OSError` pass,
so the missing git never reads as "no gitlink". `_git_runs` wraps the git
calls alone, not the identity check, which reads a file. `--no-members`
now runs git for the gitlink listing even with no `.gitmodules`, so
without git it refuses where it used to pass. The assembly root's
`validate-pins.py` ends in the same traceback, and is not changed here.

Also: `_check_members`'s docstring said "the two checks that read a file
alone" where #317 had made three; it now names them, without a count.

Tests, one new file per issue, every state built in git's index and
objects with bytes on stdin but the one that runs the remedy chain on a
real mount:
- tests/test_validate_family_unnamed_mounts.py (#361): the issue's rows
  (a), (b) and (c), each a finding in the default mode, `--pins` and
  `--no-members`; a gitlink at a case variant, and its remedy run as
  printed; the index-first, HEAD-second order; `members/` alone in any
  case; a deleted row; each section's removal run as printed; an index
  git cannot list; the hand-mount remedy chain; and the controls, a holder
  whose rows name every gitlink and section, and one `add` grew.
- tests/test_validate_family_member_path_glob.py (#367): six hand-edited
  spellings in both member modes print `member-path` and never end in
  `member-uninitialized`; the other rows are still checked;
  `recorded_gitlink(root, "members/*")` and three other globs are None in a
  holder with members and no file named `*`; and the four pathspec
  variables neither widen nor break it.
- tests/test_validate_family_without_git.py (#368): in-process, git
  failing as a missing (or non-executable) program does, exit 2, the one
  code and no traceback, in the default mode, `--pins` and `--no-members`,
  with and without `.gitmodules`, with git failing only in
  `recorded_gitlink`, and only inside a member.
- tests/test_family_add_refusals.py: #317's no-git test expects the new
  code, and its add-past-a-section control expects the one new finding for
  a section at another path.
- tests/test_validate_family_duplicates.py: five of #331's tests rewrote
  `members:` without the row of a member the holder still mounts, the
  state #361 now finds, so each keeps a row for every mounted member
  (`keeping_mounts`) and still asks only about the names it spells.

Mutants, each run against these files: the #361 checks moved after the
`--no-members` return; `--literal-pathspecs` removed from both listings;
the `OSError` clause in `_git_runs` removed; and also the `member-path`
skip removed, and the environment scrub removed. Each fails at least one
test, listed in the pull request.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR hardens family validation by finding unregistered gitlinks and .gitmodules paths in every mode, preventing malformed member paths from triggering checks through glob-like paths, and replacing missing-git tracebacks with explicit exit-2 refusals; shared literal gitlink lookup behavior is implemented in scripts/repo_shape.py and covered by targeted regression tests.

Flow diagram for family validation findings and refusals

flowchart TD
    Start["validate-family.py"] --> Checks["Run checks before --no-members return"]
    Checks --> Gitmodules["Read .gitmodules sections"]
    Checks --> Gitlinks["List index and HEAD gitlinks"]
    Checks --> Rows["Build paths named by members rows"]
    Gitmodules --> UnnamedSection{"Section path named by a row?"}
    Gitlinks --> UnnamedLink{"Gitlink path named exactly?"}
    Rows --> MemberPath{"Row path is expected?"}
    UnnamedSection -->|No| SectionFinding["gitmodules-path-unnamed"]
    UnnamedLink -->|No| LinkFinding["member-gitlink-unnamed"]
    MemberPath -->|No| PathFinding["member-path; stop row checks"]
    Checks --> GitRunnable{"Git can be started?"}
    GitRunnable -->|No| GitRefusal["git-not-runnable; exit 2"]
    GitRunnable -->|Yes| Continue["Continue validation"]
Loading

File-Level Changes

Change Details Files
Detect repository state that is not represented by family rows and report actionable findings.
  • List gitlinks under any case variant of members/ from the index or HEAD, preferring stage-0 index entries.
  • Parse .gitmodules once and detect sections whose paths no row names, including case-variant paths.
  • Run these checks before --no-members and provide per-finding cleanup or restoration commands.
  • Preserve gitmodules duplicate-path validation over the shared parsed sections.
templates/family-root/scripts/validate-family.py
tests/test_validate_family_unnamed_mounts.py
tests/test_family_add_refusals.py
tests/test_validate_family_duplicates.py
Make member path validation authoritative and make recorded gitlink lookups literal and environment-safe.
  • Stop member gitlink, tree, and identity checks after a noncanonical path produces member-path.
  • Pass --literal-pathspecs to ls-files and ls-tree so edited paths cannot act as globs.
  • Remove incompatible global pathspec environment variables from these git calls.
  • Propagate the shared repo_shape fix to copied project and holder scripts through the canonical implementation.
scripts/repo_shape.py
templates/family-root/scripts/validate-family.py
tests/test_validate_family_member_path_glob.py
Convert missing-git crashes into consistent validator refusals.
  • Add _git_runs around every validator git-reading operation and return git-not-runnable with exit 2 when git cannot start.
  • Keep gitmodules-unreadable for git execution failures after git has started, and preserve gitlinks-unreadable for an unreadable index.
  • Allow OSError to propagate from recorded_gitlink so missing git is not mistaken for an absent gitlink.
  • Exercise default, pins, and no-members modes plus failures in holder and member git operations.
scripts/repo_shape.py
templates/family-root/scripts/validate-family.py
tests/test_validate_family_without_git.py
tests/test_family_add_refusals.py

Assessment against linked issues

Issue Objective Addressed Explanation
#361 Make the holder validator reject gitlinks under members/ that no member row names, and .gitmodules sections whose path no row names, including the issue’s hand-mounted, deleted-row, stale-section, and case-variant states. ✅
#367 Do not run a member’s gitlink, tree, or identity checks through a row path after that path has failed the member-path validation. ✅
#367 Make recorded_gitlink interpret paths literally for every caller, so glob-like row paths cannot match a neighboring gitlink. ✅
#367 Ensure findings gathered before a refusal are not silently discarded, either by printing them or by handling the member’s refusal as a finding. ❌ The PR explicitly leaves this behavior unchanged. Skipping checks for an invalid member path prevents the reported glob-path repro from ending in member-uninitialized, so its member-path finding is printed; however, findings gathered before other refusals can still be dropped.
#368 Make validate-family.py refuse with exit 2, without a traceback or exit 1, when git cannot be started in the default, --pins, and other applicable modes—including when .gitmodules is missing. ✅
#368 Ensure a missing git is not misreported as an absent gitlink or as an unreadable .gitmodules file, and use a consistent refusal code for git that cannot run. ✅
#368 Prevent the same missing-git traceback in the assembly root’s validate-pins.py, which also uses the shared recorded_gitlink helper. ❌ The PR explicitly leaves the assembly root’s validate-pins.py traceback for a follow-up. Its handling of OSError is in validate-family.py, not in the shared helper, so this caller remains affected.

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

…riant of a row's path is told to be taken out (`git rm --cached`, `--unset-all`) only where it is a second path to a member that still stands, so a row hand-edited to `Members/IRSS` is said to be the edit and a member recorded only at `members/irss` is said to be in the wrong case, where the command took the member's own record out and `--no-members` then said `family ok` (#361, #367)

Review vf-423fb25 of PR #406: SHOULD-1 and NITs 1, 3 and 4.

SHOULD-1. The case-variant branch of `_check_gitlinks_named` and
`_check_gitmodules_named` took every path that folds to a row's path for
the stray and printed `git rm --cached -- <path>`, or the `--unset-all`
lines, for it. That is right in one state: the row's `path:` is its
canonical `members/<project>` AND that exact path also holds a gitlink (a
section), so the path found is a second one to a member that stands. In
two others the command took the member's own record out, and the
`--no-members` gate, the run a private family's CI falls back to, then
said `family ok` over a holder with a gitlink fewer than its rows. Probed
by the reviewer, and built here as tests. `_variant_case` tells the three
apart, and `_row_paths` carries each row's canonical path for it:

1. STRAY, the row's path is canonical and that exact path holds a gitlink
   (a section) too: today's `git rm --cached -- <path>` and `--unset-all`
   lines, unchanged.
2. ROW_EDITED, the path found IS the row's canonical `members/<project>`
   and the row spells it in another case: the row is the hand edit, so the
   finding says to put `members/<project>` back in the row's `path:` and
   prints no command. Under `--no-members` no `member-path` finding says
   so, which is why this must.
3. WRONG_CASE, anything else, the row canonical and the only record of the
   member at a case variant: the member is recorded in the wrong case, the
   `git log -p -- family.yaml` restore sentence the unnamed gitlink always
   had, no command. A row that is itself hand-edited and a gitlink at a
   third spelling, and a row with no `project:`, are decided here too
   (nothing says either is a second path to a member that stands), as the
   brief's "otherwise".

NIT-1. A hand mount is two findings, and the gitlink's `git rm -- <path>`
takes the `.gitmodules` section out too, so the section's `--unset-all`
lines exit 5 after it. The section finding now says the `git rm` above
takes it out and prints no line. Where two sections give the path
(`gitmodules-path-twice`) git removes only the one it reads, the last in
the file, so each keeps its own lines (probed on git 2.43; the test
asserts one section is left).

NIT-3. `_member_gitlinks`'s stage-0 filter had no test. A gitlink held at
stages 1-3 only is no gitlink the holder records, and one HEAD records and
the index holds only unmerged is HEAD's. Built with `update-index
--index-info` bytes.

NIT-4. The add remedy quotes its project as the paths beside it are:
`--member <org>/'My Proj'`. `<org>` is the operator's to fill in and stays
outside the quotes. (`shlex.quote` is the file's own helper.) The PR
body's caller list is corrected separately.

NIT-2 is not done: it goes to the coordinator as a follow-up.

Tests: tests/test_validate_family_unnamed_mounts.py 24 -> 39. Every state
is built in git's index and objects (`stage_gitlink`, `git rm --cached`,
`update-index --index-info`, a hashed `.gitmodules`) and never as two
directories, so a case-insensitive disk holds them: the one real
checkout stays the one directory it is.

Mutants, each run against that file at this head: SHOULD-1 reverted to
the old branch (`_variant_case` always STRAY) killed, 8 failed 31 passed,
both state tests among them; the stage-0 filter dropped, 2 failed 37
passed; NIT-1's `--unset-all` lines printed again, 4 failed 35 passed.
Also killed: ROW_EDITED never (3), NIT-1 for two sections (1), the member
unquoted (2), STRAY without the exact path held (4).

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

brettheap pushed a commit that referenced this pull request Oct 12, 2026
…C`) is `git-not-runnable` too, so the rule is any `OSError` whose `filename` is `git`, and the test picks each git call by its subcommand, so #406's `git --literal-pathspecs ls-files` no longer fails eight of its cases on a merge (#412)

Fix round 1 of the review of d4b371f.

MUST-1. `tests/test_validate_pins_without_git.py` chose the call to fail
by `args[0] == "ls-files"` (and `"rev-parse"`, `"ls-tree"`). #406 makes
`recorded_gitlink` run `git --literal-pathspecs ls-files -s -- <path>`,
so on a merge with it the predicate never matched, the "gitlink" cases
failed nothing, and eight of the file's tests went red: whichever of
#406 and this PR landed second would have failed CI. Each predicate now
reads the git SUBCOMMAND, wherever it sits among git's global options
(`subcommand()`, which also skips the value of `-C` and `-c`), and a
test of its own feeds each predicate the argv with options in front of
it, so the file says so on its own head and not only on a merge with
#406.

NIT-1. A `git` on PATH that exists and cannot be executed (a binary for
another architecture, a script with no shebang) fails `ENOEXEC`, a plain
`OSError` whose `filename` is `git`, and ended in the same traceback,
exit 1, that #412 is about. `_git_runs` now counts any `OSError` whose
`filename` is `git`, and a `FileNotFoundError` or `PermissionError`
whose `filename` is None (Windows' `CreateProcess` failures carry no
filename, so None is required there and is kept for those two classes
only). Every exclusion the tests pin stands: an error naming a directory
or another file, and a nameless `OSError` that is neither class, leave
the wrapper unchanged. Tests: `OSError(ENOEXEC, ..., "git")` from the
fake in every mode and at each of the three calls, a nameless
`PermissionError`, and an `ENOEXEC` that names no file or a directory
(not git, so not counted). With a real `git` file with no shebang first
on PATH, `subprocess` raises `OSError: [Errno 8] ... 'git'` and the run
ends in the refusal (checked in scratch, Linux, Python 3.12).

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brettheap pushed a commit that referenced this pull request Oct 12, 2026
…o fix/family-bump-remove-rollback, so CI tests the pair that lands together (#365, #360)

#406 (`validate-family.py` and unnamed mounts) lands before this PR, and
both edit `tests/test_family_add_refusals.py`. Merging it here lets CI
test the two together, so this PR can land right after #406 without
another cycle. The merge was clean: git merged
`tests/test_family_add_refusals.py` with no conflict, and no other file
is touched by both branches.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 12, 2026
…fuses `git-not-runnable`, exit 2, naming the leg's gitlink, pinned commit or tree digest it could not read, instead of a traceback that exits 1 where 1 is a finding, and only a git that will not start counts, so a leg directory nobody may enter is not told to put git on PATH (#412) (#446)

`git_out` raises `FileNotFoundError` when there is no git on PATH,
`PermissionError` when the `git` found there will not run, and a plain
`OSError` (`ENOEXEC`) when it cannot be executed (a binary for another
architecture, a script with no shebang), and nothing in the validator caught
any of them. A scaffolded root on such a machine ended in a traceback out of
`recorded_gitlink`: exit 1, which this script's own exit codes give to "a
FINDING: drift a reviewer can act on", though nothing was found, and a person
or a CI step reading the code took it for drift. The doctor's `pins` row said
so too, in its words: "reports a finding (exit 1): FileNotFoundError: [Errno
2] No such file or directory: 'git'".

The three calls that run git now run inside `_git_runs`: the leg's gitlink
(`recorded_gitlink`), the pinned commit's lookup in the submodule (`git_out`)
and the tree digest (`tree_digest`). It refuses `git-not-runnable`, exit 2,
"the question could not be asked", naming which of the three it was and
saying to put git on PATH and run `scripts/validate-pins.py` again. The code
is the one `validate-family.py` chose for the same fact (#368, #406), so a
project and a holder say the same thing; the wording is this validator's own
and uses no holder words. Three decisions the issue left to its author:

- WHERE: in this validator, as #368's fix did in `validate-family.py`, and not
  in `git_out` or `repo_shape.py`, which every tool shares. The change adds
  nothing to the names this file imports from `repo_shape`, so a root whose
  `repo_shape.py` is older than this copy still imports it.
- ONLY A GIT THAT WILL NOT START COUNTS. #406's wrapper caught every `OSError`
  and told an operator whose directory nobody could read to put git on PATH.
  This one is told apart two ways. The block is around the three git calls
  alone, never a whole check, so the `(submodule / ".git").exists()` before
  them, a pin file or a workflow nobody may read stay what they were. And
  inside a git call the rule is the NAME `subprocess` gives the error: any
  `OSError` whose `filename` is `git` counts (a missing git, one that will
  not run, one that cannot be executed), and so does a `FileNotFoundError` or
  `PermissionError` whose `filename` is None, those two classes only
  (Windows' `CreateProcess` failures carry no filename). `subprocess` raises
  the same two classes when the DIRECTORY it was to run git in cannot be
  entered, and on POSIX names that directory where a program that will not
  start names `git` (probed on Linux, Python 3.12: no git on PATH gives
  `filename 'git'`; a `git` that is not executable gives a PermissionError,
  `'git'`; a `git` with no shebang gives `OSError: [Errno 8]`, `'git'`; a
  missing, unreadable or non-directory `cwd` gives the directory). Any other
  error leaves the wrapper as it entered.
- `recorded_gitlink`'s `except Refusal` does not turn the missing git into
  "no gitlink recorded": an `OSError` is no `Refusal`, passes straight through
  it and reaches `_git_runs`, so there is no false `pin-gitlink-absent`. The
  commit lookup's `_git_runs` is OUTSIDE its `try`, for the same reason one
  step on: inside, its own `except Refusal` would say `pin-commit-
  unresolvable`, which tells the operator to fetch a commit.

`_digest_from_source` (a neutral product's `--pin-source`) keeps its own
`except OSError`: it is reached only after every leg's gitlink was read, so
no git means the refusal above came first.

This file is a pinned copy and reaches existing roots through
`update-shape.py` as `upstream-changed`; nothing more is needed here.

tests/test_validate_pins_without_git.py (58 tests, run in process, the
validator's own `git_out` and the one `recorded_gitlink` and `tree_digest`
call in `repo_shape` patched to raise what a git that will not start raises,
no PATH changed): exit 2, the code and no `Traceback` in each of three runs
(plain, `--quiet`, `--pin-source`) and for each of five errors a git that
will not start raises (missing, not executable, cannot be executed, and two
that name no file); each call named by the check it is; the missing git is
neither `pin-gitlink-absent` nor `pin-commit-unresolvable`, each with its
control where git runs; the same root says `pins ok` through the same
harness; seven errors that are no git that will not start, at each of the
three calls, leave the wrapper unchanged; and one POSIX test with no fake,
PATH an empty directory. The fake picks the call to fail by the git
SUBCOMMAND, wherever it sits among git's global options, and never by
`args[0]`: #406's `recorded_gitlink` runs `git --literal-pathspecs ls-files
...`, which an `args[0]` predicate stops matching, and a test of its own pins
that. 27 fail on main's validator. Mutants, each killed (failing of 58): the
`OSError` clause removed (27), any `OSError` counts (21), the classes narrow
but the directory counts (12), `OSError` broad but a nameless error of any
class counts (9), the refusal exits 1 (29), the commit lookup's wrapper inside
its `try` (4), each of the three wrappers removed (20, 4, 3),
`recorded_gitlink` swallowing `OSError` in `repo_shape.py` (27), the
`ENOEXEC` clause dropped (6), the nameless clause dropped (6), narrowed to
`FileNotFoundError` (3), `ENOEXEC` counted by errno (6), the test's gitlink
predicate back to `args[0]` (4 here, 15 on a merge with #406).

Closes #412

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@brettheap
brettheap merged commit ec8e09a into main Oct 12, 2026
5 checks passed
@brettheap
brettheap deleted the fix/validate-family-unnamed-mounts branch October 12, 2026 20:21
brettheap pushed a commit that referenced this pull request Oct 12, 2026
…t: `check_pin` takes the root its `update-pin-row` remediation reads, so #290's refusal survives #267's split instead of a NameError (#290)

Main 580d7df carries #267 (decff87), which split update-shape.py's
`read_pin(root)` into the read and `check_pin(pin, pin_path)`, so that
shape-doctor.py can check the pins in a root's history in memory. The
`update-pin-row` raises moved into `check_pin`, and on this branch they
call `pin_rows_remediation(root)`, #290's `git -C <root> log` advice,
where `root` is not a name. The textual merge is clean and the tree dies
at run time: `NameError: name 'root' is not defined` for a `files:` of
`5`, `true` or `1.5` in `check`, `apply` and shape-doctor.py, which then
prints no report at all.

This commit takes the planned #267 composition, the PR's two attachments
fix1-compose-267-resolution-v2.patch and root-pin-faults-after-267.patch,
applied unchanged; main moved their hunks by offset only (shape-doctor.py
526 lines, tests/test_shape_doctor.py 521, update-shape.py 1):

- update-shape.py: `check_pin(pin, pin_path, root)`, `read_pin` passes
  its root, and the docstring says `root` is the repository whose history
  a refusal's `git log` reads.
- shape-doctor.py `pin_reads_at` passes `ctx.root`. Its docstring says
  `check_pin` refuses `files: 5` as `update-pin-row`, and `TypeError`
  leaves its `except` tuple: no pin's bytes raise one now, so one means a
  caller out of step with `check_pin`, and the doctor fails loudly rather
  than answer that no commit holds a pin to put back.
- tests/test_shape_doctor.py: #267's ROOT_PIN_FAULTS gains `files: 5`,
  `true` and `1.5`, each `update-pin-row`, so each is INVALID, exit 1,
  with `checkout HEAD` as its NEXT, which run as printed leaves the root
  COMPLIANT.

Main's other changes to this PR's files merge as text away from these
lines: #446 (245d775) in validate-pins.py, #406 (ec8e09a) in
validate-family.py, #379 (87d2eb7) in update-shape.py, and #375 (cd1fc79)
and #357 (5678f68) in shape-doctor.py. Neither module has changed on main
since decff87.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap pushed a commit that referenced this pull request Oct 12, 2026
…landed: `_commit_the_split` keeps #445's reported steps and reads `follow_ups:` through `_plan_follow_ups`

Main moved from dd8d0ea to 580d7df: decff87 (#267), dba5fb0 (#403),
245d775 (#446), 75052cb (#445), ec8e09a (#406) and 580d7df (#416).
Only #445 touches adopt-project.py, and it conflicted in one hunk of
`_commit_the_split`.

#445 wraps that function's body in a `try` that names each step
(cloning the source for the split, mounting the legs, committing the
split, pushing the split branch), so a failure once both legs are pushed
flushes stdout, prints `REFUSED <step>.`, what failed and one notice of
what the run left. This branch reads `follow_ups:` there through
`_plan_follow_ups`, the one read of it (#293). The resolution keeps
#445's `try`, steps and reports whole, and carries this branch's read
into its "committing the split" step: git had kept main's
`[str(f) for f in plan.get("follow_ups", [])]` inside the `try`.

The two contracts meet without overlap. `plan-bad-follow-ups` and
`election-date-malformed` are findings of `_entry_findings`, which
`execute` refuses through `_refuse_what_check_finds`. That runs before
#445's `_refuse_an_occupied_work_dir`, `_refuse_a_split_branch_left_behind`
and `_create_leg_remotes`, so these are refusals before any leg exists:
exit 2, `REFUSED <code>:` and the remediation that says no leg
repository was created and nothing was pushed, as #445's own refusals
before the legs say, and no notice. #445's report covers the failures
once a leg exists, which these values no longer reach. The guard in
`_manifest_values` is unchanged; #445 does not touch it.

#445's comment over `_template_values` in `cmd_execute` named "an
`elected_on` that is not a date" as a refusal the table can raise. With
#294's finding, `_refuse_what_check_finds` refuses such a date first, so
the comment now says the date no longer reaches the table. #445's test
of a table refusal stubs `_template_values` and is unchanged.

tests/test_adopt_execute_refusals.py: the two end-to-end runs through
`main` (#293, #294) now also assert that the refusal says no leg was
created and that #445's notice ("NOTHING has been rolled back") does not
follow it.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap pushed a commit that referenced this pull request Oct 12, 2026
…_spelled` now spells every symlink this branch names, and #283's tracking-branch question is not asked by adopt-project.py, so it meets none of this branch's refusals

Brings in, by sha, main's ten commits since 75052cb: ec8e09a (#406),
580d7df (#416), 1184f67 (#404), d6df36b (#346), 37665ff (#452), 36538fd
(#378, #329), 6c2a371 (#355, #283), 8782333 (#401, #340), ec4a7a0 (#456,
#391) and 437a360 (#343, #290).

No conflict. The tree is `git merge-tree --write-tree 437a360 95f6fc7`'s,
ae7be0a. Three files changed on both sides merged by git alone:

- adopt-project.py: 36538fd rewrites `_spelled`'s docstring and
  `_spelled_character` (a byte that is not UTF-8 is `\udcNN`, a character
  `\xNN`, by `repo_shape.double_quoted`), and two docstrings
  (`_refused_by_policy`, `_index_entries`); 8782333 adds `_leg_path_help`
  and the `help=` of `plan`'s `--spec-path` and `--code-path`. This
  branch's hunks are elsewhere.
- scripts/shape_materialize.py: 6c2a371 adds `import tempfile` and the
  branch-name question (`_ask_git_branch`, `_branch_verdict`,
  `branch_problem`, `refuse_bad_branch`) after `run`. This branch's hunks
  are elsewhere, and it adds no import of `tempfile` to either file
  (adopt-project.py has had one since before 75052cb), so no name is
  bound twice.
- tests/test_adopt_finding_spelling.py: 36538fd's `\udcfe` lines and this
  branch's `follows too many symlinks` line are more than 100 lines apart.

How main's changes meet this branch's code:

- 36538fd: every source path and symlink target this branch's findings
  and the materializer's refusals name is spelled by `_spelled` (`execute`
  hands it to the materializer as `spell`), so they now spell a byte that
  is not UTF-8 as `\udcNN`, as the plan's own `path:` does. One test of this
  branch pinned the old spelling, `nowhere\xff`; the next commit pins
  `nowhere\udcff`. Nothing in this branch has an escape table of its own.
- 6c2a371: `refuse_bad_branch` is called by scaffold-project.py and
  scripts/family.py only. adopt-project.py's `--tracking-branch` is not
  asked of git (6c2a371 leaves it as its own change), so the question
  falls nowhere among `check`'s findings or `execute`'s refusals.
- 8782333: `plan --help` text only; `plan` refuses nothing new, and the
  help names `shape/` as where `execute` writes its copy, which this branch
  does not move.

Checked on the merged tree: no top-level name in adopt-project.py,
scaffold-project.py, scripts/repo_shape.py, scripts/shape_materialize.py
or scripts/family.py is bound twice that was not already bound twice on
main (shape_materialize.py's `YamlError` and `parse_yaml`, as on main), no
name is loaded and bound nowhere, and each side's own change reappears
whole in the merge (no line of either side lost).

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 13, 2026
… is `PRODUCT=PATH` only when its key names a chain link, and a link source that cannot be entered is a WARNING with the system's reason, not a traceback (#338) (#453)

**What was wrong.**

Two faults, one in the parser and one in the lookup.

- **The parser.** `_link_sources_from_args` split every `--link-source` value at its first `=`. A link tree under a directory named `a=b`, passed bare, became a source keyed to a link `a` that no chain names. Nothing asked for it, so the link stayed `declared-unverified`, exit 0. The WARNING then offered `--link-source openXdox=<path>` to the person who had passed exactly that. A mistyped key failed the same silent way.
- **The lookup.** `repo_shape.link_manifest_path` asked `Path.is_dir()` and then `is_file()`. Before Python 3.14 those swallow only ENOENT, ENOTDIR, EBADF and ELOOP. So each of these ended the naming gate in a traceback, exit 1:
  - a keyed source under a parent this user cannot enter;
  - `SHAPE_PIN_SOURCE_<LINK>` pointing there;
  - a checkout `../openXdox` at mode 000, which every `--project` run asks for;
  - a 300-character name.

**What changed.**

**How a value is read** (`LinkSources` in `scripts/validate-repository-naming.py`). This mirrors #312's `PinSources`, with a chain link where that file has a declared pin:
- The text before the first `=` is a LINK only when it casefolds to a link of a chain this run reads. Those are the entries of `--referent-chain`, and of each leg's `naming.referent_chain:` under `--project`. Both are read before any leg is classified.
- The keyed reading wins even over a directory spelled like the whole value. `./` reads that directory.
- Any other value is a bare PATH, taken whole.

**What a lookup says** (`scripts/repo_shape.py`):
- `link_manifest_lookup` asks `stat()` inside a guard, as `_digest_from_source` does in #312. It returns the system's reason when a source cannot answer, or says which of four plain cases it is (`no such file or directory`, `it holds no project.yaml`, `its project.yaml is not a file`, `it is neither a directory nor a file`), and it never raises. It replaces `link_manifest_path`, which is gone: nothing calls it. `link_declarations` gives the reason a manifest could not be read, and `declared_neutral_pins`, which it left with no caller, is removed (fix round 1).
- `resolve_link_source` and `link_pins_from_trees` take a keyword-only `tried`. It records each source asked, in order, with its reason. Nothing in `repo_shape` prints.
- `link_pins_from_trees` also takes a keyword-only `with_reasons` (fix round 2). Asked, it returns a `LinkPins`, a dict equal to the plain mapping that also remembers why a tree it reached could not be read. Only the scaffold asks.

**What the validator prints** (stderr, after each name's WARNINGs):
- `WARNING link-source-unreadable: <path> (<how>) could not answer for <link>: <reason>`. This covers a `--link-source` asked for a link that holds a pin, a `SHAPE_PIN_SOURCE_<LINK>` likewise, and a checkout beside the project that exists and still cannot answer. A keyed value that also names a directory gets a second line saying how to ask for that directory with `./`. A bare value holding `=` says it was read as a PATH.
- `NOTE --link-source <typed>: not used — <why>.` for a value no such link asked for. The why is one of five: a later value replaced it; no name reads that link's own declaration (a keyed value); no name records a chain at all; no name reads any chain link's own declaration (a bare value, where chains exist and none is read); or every link that is read had a more specific source.
- Three kinds of source are never named: a variable or a checkout asked only for a chain's LAST link (no one ever reads that link's own declaration); a variable or a checkout for a chain no classification reads (fix round 1: a spec leg's chain, a name with no referent, a chain whose shape fails; a `--link-source` value for such a chain is the NOTE); and a checkout that is absent or holds no manifest.
- `--quiet` drops both, with the WARNINGs. So `make bootstrap` and the doctor's `naming` row print nothing new.

`--help`, the module docstring and `docs/cli.md` (regenerated) say all of this.

**Fix round 1 (lane 5's review of 40762ca: 0 blockers, 2 should, 4 nits).**

Second commit, 4b21263. Each review finding, in order:

- **A link manifest whose `neutral_product_pins:` is not a list of names is a tree that cannot answer** (SHOULD-1). `link_declarations` iterated `data.get(key) or []` as it stood, against the `link_pins_from_trees` docstring this change wrote ("nothing a source does raises out of here"). `5` ended the naming gate, the scaffold and the project's `validate-manifest.py` in `TypeError: 'int' object is not iterable`. A scalar `openDox` was read as the characters `D`, `e`, `n`, `o`, `p`, `x`, so the chain broke, and under `--project` that is a FINDING. A mapping was read as its keys, so `openDox: yes` DECLARED `openDox`. Each is now `(None, why)`, as `not a mapping` is: declared-unverified, with the reason named ("`neutral_product_pins` is not a list", "holds an entry that is not text"). An absent or null value, an empty list, and a blank entry (`null`, `''`) in a list, declare nothing, as ever. The fix is in the shared function, so the scaffold and `validate-manifest.py` are reached too.
  **Three behaviours change on purpose here, and the tests of fix round 2 pin each:**
  - **A falsy value is no longer an empty list.** `neutral_product_pins: 0`, `false` or `''` was read as no pin at all (`or []`), so a chain through the tree was BROKEN, and under `--project` that is a FINDING. It is now a tree that cannot answer: the link is declared-unverified, a WARNING, exit 0.
  - **An entry that is not text is no longer read as text.** `[openInk, 0]` and `[openInk, false]` dropped the entry as a blank, and `[openInk, true]` read it as the text `True`, so a chain through the list VERIFIED. Each is now declared-unverified, with a WARNING saying the list "holds an entry that is not text". That is stricter than the link tree's own `validate-manifest.py`, which takes `str(pin)` of an entry (`_declared_pins`, `templates/assembly-root/scripts/validate-manifest.py:85`): a chain is better left unverified than verified through a name nobody wrote.
  - **The scaffold no longer refuses `chain-not-declared` over a pins value that is a scalar.** With `--pin-source` a tree whose `neutral_product_pins:` is the scalar `openInk`, the scaffold used to refuse (`REFUSED chain-not-declared: ... openXdox declares I, e, k, n, o, p, not openInk`, exit 2, from the characters). It is now the declared-unverified WARNING and the scaffold goes on, exit 0, which is its documented rule for a tree that cannot answer: "a warning, never a refusal, because the check is offline". The warning says the tree was reached and could not be read (fix round 2, below).
- **The reach wording is corrected** (SHOULD-2): "Who else this reaches" below, and the docstring of the test that made the claim. What stops crashing is the naming and manifest steps. `make validate`, `validate.yml` and `make bootstrap` still end in a traceback on the sibling row before Python 3.14, at a file this change does not touch (below). On 3.14 the sibling is passed over without a word there instead, which is #430.
- **The mutation gaps** (nit). Six of the review's ten mutations survived. Five now die, each by a new test: a `project.yaml` that is a directory; a device or a FIFO as the source; the order A, B, A; a value holding `=` with no chain at all; and every failure of `SHAPE_PIN_SOURCE_<LINK>` for a link that holds a pin (a path that is not there, a directory with no manifest, a file that is no manifest), not only a permission. The sixth, the `typed in self.used` test in `unread_lines`, could not be killed because it was dead: a value is read as one link's keyed source or is the one bare PATH, so every lookup asks it of the same path and it is used or unread, never both. That was asserted over 293 tests and the test is removed.
- **A chain no classification reads** (nit). `classify` consults a link's own declaration only for a name that claims a referent, with a recorded chain whose shape holds. For any other chain (one recorded on a spec leg; one on a name with no `open<Product>` to reach; one that does not begin at a held pin or does not end at a referent of the name) a source was still asked, so a path that was not there was reported as `link-source-unreadable` and a good tree was counted used, each for a question nobody put. `NamingPolicy.reads_link_declarations` now says when `resolve_referent` reads a chain, and `LinkSources.link_pins` asks it first: such a chain asks nothing, and a value for it is the NOTE that no name reads that link's declaration. A test runs `resolve_referent` with a mapping that records whether it was read, over every way a chain is and is not, and holds the new method to it.
- **Dead code and stale sentences** (nit). `declared_neutral_pins` is removed: `git grep` over the tree, every pushed branch and every sibling worktree finds no caller. `link_manifest_lookup`'s docstring no longer says "replaces" over a name that is gone. This description no longer says `tests/test_naming_policy.py` takes `SEARCH_DENIED` from #312: it defines a mark of its own, with the same condition and its own reason text.

**Fix round 2 (lane 5's re-review of 4b21263: 0 blockers, 0 should, 3 nits).**

Third commit, 2ac911c. Each finding, in order:

- **The three behaviours the body did not state are stated and tested** (nit 1). Round 1's not-a-list fix changed three behaviours on purpose, each listed under SHOULD-1 above, and the re-review found two of them had no test. R1 (`if declared is None` read as `if not declared`, which makes `0`, `false` and `''` declare nothing again) and R2 (`if pin is None or pin == ""` read as `if not pin`, which drops a `0` or `false` entry as a blank) both passed all 210 cases of `tests/test_naming_policy.py` at 4b21263. Six rows join `NOT_A_LIST_OF_NAMES`, which runs at the CLI and in process: `0`, `false` and `''` as the value, and `0`, `false` and `true` among the names. R1 now fails 9 cases and R2 fails 6.
- **The scaffold's warning is true for a tree it reached** (nit 1, the third behaviour). Over a `--pin-source` whose `neutral_product_pins:` is not a list, round 1 turned the `chain-not-declared` refusal into a declared-unverified WARNING that says the tree "is not reachable here (a sibling checkout, SHAPE_PIN_SOURCE_<LINK>, or --link-source <LINK>=<path> would let it be read)". `--pin-source` did reach the tree, and only its declaration could not be read, so that blamed a path that was right. Now `link_pins_from_trees` takes a keyword-only `with_reasons`; asked, it returns a `LinkPins`, a dict equal to the plain mapping that also keeps `unreadable`, `{link: why}` for each link whose source answered and whose manifest could not be read. `_chain_link_findings` words such a link "which was reached but could not be read as a declaration (<why>)", in a branch of its own, and only the scaffold asks. The new text is asserted for each of the eleven not-a-list rows, with `scaffold-project.py --dry-run` run as a subprocess.
  - **Every other path's text is the bytes it was.** The sentence for a link no source reached is the same code in the same lines, and four tests hold it: the generic warning for a plain mapping, for an empty `LinkPins` and for an absent tree; the scaffold's own old warning for a clone that holds no `project.yaml`; the naming validator's two lines (its warning, and the `link-source-unreadable` line that names the reason); and `validate-manifest.py`'s. The naming validator and `validate-manifest.py` do not ask for reasons, so they print what they printed.
  - **One more scaffold path changes with it, on purpose.** A tree the scaffold reached and could not read as a declaration for any other reason (a manifest that does not parse, or is not a mapping) used to say "not reachable here" too, and was no truer. It now says what it is, with the system's reason, and a test for each of the two (a manifest that is not a mapping, one that does not parse) holds it, at the scaffold's command line and in process: narrowing the wording to the not-a-list reasons fails both.
  - `_declaration` still returns its four values, so nothing else in `scaffold-project.py` moved.
- **A qualified chain link in the spy test** (nit 2). `_chain_read_cases` had no case whose chain link is qualified, so `reads_link_declarations` could take the chain as typed and not through `_chain_links` (R3) and pass every test. With `--referent-chain testorg/openXdox,openDox` that turns a verified chain into a declared-unverified one. `("codexDox", {"openXdox"}, ("testorg/openXdox", "openDox"), True)` is a case now, and R3 fails it.
- **The description** (nit 3): the ten corrections of the re-review are applied (the reason a lookup returns, the fifth NOTE reason, the third kind of silent source, the SHOULD-1 bullet, "before Python 3.14", the `validate-pins.py` bullet, the 3.14.7 rows, the merge-tree section, the filed numbers, and the project's own `validate-manifest.py`).

The mutation proofs: all three survivors were reproduced first at 4b21263 (210 passed each) and each dies at 2ac911c. Six mutations of the new code each die too: the warning ignoring the reasons (12 cases fail), the lookup recording none (12), the scaffold not asking (11), `with_reasons` on by default (3), an absent tree called reached (3), and the reached sentence used for a tree nothing reached (17).

**Where it departs from #312, and why.**

- **An explicit source that cannot answer stays a WARNING, exit 0.** #312 made an unreadable `--pin-source` a finding. Three reasons for the difference:
  - Exit 1 here means "a name is wrong", and this module has always said an unread link tree is a warning.
  - `SHAPE_PIN_SOURCE_<PRODUCT>` also answers `validate-pins.py`'s digest question, from a checkout that need not hold a manifest.
  - The lookup falls through to the next source, which may still verify the link.
- **The fall-through is kept**, as it always was. With `--pin-source`, a keyed value is final.
- **The lines go to stderr, spelled `WARNING`/`NOTE` as this file spells them.** #312's notes are `  note` lines on stdout, where that file's report lives.

**Who else this reaches.**

The guard is in `repo_shape.py`, so the same `link_pins_from_trees` lookup stops crashing for these callers too:
- **The project's own `validate-manifest.py`.** It reaches the lookup through `SHAPE_PIN_SOURCE_<LINK>` or a sibling checkout, and now gives its usual declared-unverified WARNING instead of the traceback.
- **`scaffold-project.py --referent-chain`.** A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent no longer kills the plan, and `--pin-source` verifies the link.

Neither of these prints the reason a source could not be entered; the scaffold says it only for a tree it reached and could not read (fix round 2, above). A `--pin-source` the scaffold cannot enter is still a traceback: that is #339 (`pin_digest_from_source`), and this PR does not touch it.

**What this does not reach, and what the doctor says.**
- **The project's gates still end in a traceback on the sibling row.** The NAMING and MANIFEST steps of `make validate`, `validate.yml` and `make bootstrap` stop crashing for a checkout `../openXdox` at mode 000. Those targets also run `validate-pins.py`, whose `resolve_neutral_pin_source` asks `(sibling / ".git").exists()` with no guard (`templates/assembly-root/scripts/validate-pins.py:463` on main), so before 3.14 a sibling at mode 000 still ends them in a `PermissionError`. That file is #312's, and `fix/pin-source-env-collisions` (PR #409, since landed), `fix/adopt-pin-writes-pin-file` and open PRs #343 and #419 touched it when this was written, so this PR does not edit it. The same line is listed for Python 3.14 by #430 (lane 2), where the helper that would cure both halves is asked for; the pre-3.14 traceback reproduction is added there as a comment, not filed apart. A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent is different: `validate-pins.py` makes that a `neutral-pin-source-unreadable` FINDING (#312), and that stays.
- **The doctor's `naming` row** runs the STANDARD's copy of the validator, so it stops crashing at once. **Its `manifest` row** runs the project's OWN `validate-manifest.py` on the project's own `repo_shape.py`, so it changes only once the project has applied this change. Over an existing project the doctor says `INVALID (manifest, pins)` until `update-shape.py apply` lands the two `upstream-changed` rows below.
- **The verdict.** For `SHAPE_PIN_SOURCE_<LINK>` under a locked parent, on a project that carries these copies, `INVALID (naming, manifest, pins)` becomes `INVALID (pins)`, and `pins` is #312's finding for that locked variable, which is right. For a sibling at mode 000 the `pins` that remains is the traceback above, not a finding.

The two copied scripts this change edits (`scripts/repo_shape.py` and `scripts/validate-repository-naming.py`) are copied into every project and pinned there. A project takes the change through `update-shape.py` (two `upstream-changed` rows), and a family holder takes the `repo_shape.py` row. Neither file has a second copy in this repository.

**Evidence.**

- **Issue rows, main ab623dd** (`repro338.sh`, a `gh` stub first on PATH). Under 3.10.21 and 3.12.3, rows 4-7 (keyed under a mode-000 parent, `SHAPE_PIN_SOURCE_OPENXDOX` there, a sibling at mode 000, a 300-character name) are a traceback, exit 1. Rows 2-3 (`a=b`, the typo) are a silent declared-unverified, exit 0. Under 3.14.7 row 1 verifies, and rows 2-7 are the silent declared-unverified WARNING, exit 0.
- **Issue rows, this PR.** Under 3.10.21, 3.11.16, 3.12.3, 3.12.14, 3.13.15, 3.14.7 and 3.15.0rc2 (first commit), all 7 rows exit 0 with no traceback, and the output is byte-identical across versions. `a=b` verifies. The other five rows each name their source with its reason. Re-run on the fix-round head, 4b21263, under 3.10.21, 3.12.3 and 3.14.7: byte-identical to the first commit's run. Re-run on the fix-round-2 head, 2ac911c, under the same three: all 7 rows exit 0 with no traceback, the output is byte-identical across the three versions, and it is identical to the re-review's runs of the previous head over main.
- **Reach on main.** Main gives a `PermissionError` traceback, exit 1, for `scaffold --dry-run`, the project's `validate-manifest.py`, `validate-repository-naming.py --project` and its `--quiet` form, each with `SHAPE_PIN_SOURCE_OPENXDOX` under a locked parent. This PR gives exit 0 for all four. `validate-pins.py` on the sibling row is the exception said above.
- **New tests.** 17 cases (11 functions) in the first commit; 16 fail on ab623dd. The fix round adds 31 cases in `tests/test_naming_policy.py`, and 17 of them fail on 40762ca: the five shapes of pins that are not a list (each at the CLI, in-process and, for the scalar, under `--project`), and the chains nobody reads. The other 14 guard what a mutation could break (below) or the side the fix must not move (an absent, null or blank pin list still answers).
- **Targeted run on 4b21263**, files `test_naming_policy`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_cli_reference`, `test_repo_hygiene`, `test_neutral_pin_validator`, `test_scaffold_e2e`, `test_shape_doctor`, `test_git_output_decoding` and `test_pin_validator`: **765 passed, 3 skipped**. The 3 skips are the PowerShell-only rows. (40762ca: 734 passed, 3 skipped.) `render-cli-reference.py --check` is ok.
- **Mutation probes, first commit.** Reverting the stat guard, the parse, keyed-over-directory, the NOTE lines, the chain's-end restriction, the sibling-absence filter, the manifest reason, the `--quiet` drop, the "read as a PATH" sentence, the `./` line, the project-chain read, the WARNING lines or the env reporting each fails at least one new test.
- **Targeted runs on 2ac911c (fix round 2).** Under 3.12.3, `test_naming_policy` (238 cases: 210 and 28 new), `test_cli_reference`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_repo_hygiene` and `test_setup_project_py`: **468 passed**; and `test_scaffold_e2e`, `test_scaffold_pin_and_reuse`, `test_scaffold_leg_path_collisions`, `test_scaffold_pin_collisions`, `test_neutral_pin_validator`, `test_git_output_decoding` and `test_pin_validator`: **312 passed**. Under 3.10.21 and under 3.14.7, `test_naming_policy`, `test_manifest_validator` and `test_scaffold_referent_chain_links`: **321 passed** each. `render-cli-reference.py --check` is ok, and `ast.parse(feature_version=(3, 9))` passes on the three changed files.
- **Mutation probes, fix round.** The review's ten, re-run on 4b21263: MW1, MW2, MW3, MW5 and MW11 now die (they survived), and MW4, MW7, MW12 and MW13 still die; MW6 is the dead test, removed. (Fix round 2 kills the three the re-review found, R1 to R3, and six more: see above.) Nine more for the new code (the list check, the entry check, the null and blank tolerances, `reads_link_declarations` always true, without the shape check, without the held pins, and the validator ignoring it): eight die. The ninth, `reads_link_declarations` ignoring whether the name has a referent, survives as an equivalent mutant: with no referent `_chain_shape_problem`'s last question always reports a problem, so the explicit early return is redundant, and is kept because `resolve_referent` has the same first line. The spy test holds the two together.
- **Windows.** The mode-000 rows skip on Windows and as root, using a `SEARCH_DENIED` mark this file defines, with the same condition as #312's. The new FIFO and `/dev/null` rows skip where those do not exist. `a=b` and the 300-character name run everywhere. On Windows the long-name row asserts the line, not POSIX's reason. None of it had been run on Windows or macOS when this was written; the last paragraph gives the CI result.

**Merge-tree.**

Recomputed at 19:04Z with `git merge-tree --write-tree` of 2ac911c against `origin/main` as it was then, 53bee5b, and against every one of the other 37 pushed branches that touches a file this PR changes (a scan of each head against its merge-base with main):
- **Clean:** origin/main 53bee5b (it carries #314 and #390); #394 `fix/adopt-record-submodule-placement` 563ea16 (`scripts/repo_shape.py` and `validate-manifest.py`) and #406 `fix/validate-family-unnamed-mounts` 369af2e (`scripts/repo_shape.py`); #397 abb27e6, #400 7062a47, #401 409f10b, #416 7cca975, #452 e2df9a8 and #395 1ef3310 (`docs/cli.md`); #355 6fdbe3d, #373 741cbf2 and #395 (`scaffold-project.py`).
- **Conflict, four hunks each, all in `scripts/repo_shape.py`:** `fix/pin-source-env-collisions` (PR #409) 6494714, and `fix/adopt-pin-writes-pin-file` 421e4e5 (no PR). Both also touch `scaffold-project.py`, and that merges clean with this PR's one edit there. Fix round 2's new hunks merge with neither: #409's change to the "not reachable here" sentence (the `offered` text, when a variable is shared) and the reached branch sit in different lines of `_chain_link_findings`. The four hunks, each taking both sides:
  1. the `resolve_link_source` signature: `default_source=None, env_shared: bool = False, *, tried: list | None = None`;
  2. its docstring: both paragraphs;
  3. the `link_pins_from_trees` docstring and the lines after it: both paragraphs, then `found`, this PR's `unreadable` and #409's `links` and `shared`;
  4. the call: `resolve_link_source(name, root, keyed_sources, default_source, pin_product_key(name) in shared, tried=asked)` (the same spelling on both branches), followed by `if tried is not None: tried[name] = asked`.

  Resolved that way in a throwaway and committed there, the merge with 6494714 passed **496 tests** (`test_naming_policy`, `test_pin_source_env_collisions`, `test_neutral_pin_validator`, `test_scaffold_referent_chain_links`, `test_manifest_validator`, `test_cli_reference`), and the merge with 421e4e5 passed **559 tests** (the same files and `test_pin_source_unreadable`). `render-cli-reference.py --check` is ok on both, pyflakes is clean, and the grammar parses as 3.9's. Neither branch carried the resolution when this was measured. #409 has since landed, and the merge of main into this branch (the last paragraph) takes those four hunks both ways. The earlier counts (468 and 434 from the first fix round, 430 from the re-review) are those runs' own, at their heads.

**Left out.**

Each of these was found by lane 5's reviews of this change, and each is filed or folded:
- **A bare `--link-source` answers for every link with no more specific source** (#441). In a chain of three or more links, one bare tree is read as two links' declarations, and a chain that holds is reported broken, a FINDING for a name with no direct pin on its referent. Pre-existing, the same on main; `validate-pins.py`'s `--pin-source` has the same documented rule and was not examined.
- **An empty path after a key (`--link-source openXdox=`) is `Path("")`, the current directory** (#442). Run from a project root, the project's own `project.yaml` is read as the link's declaration. Pre-existing; `validate-pins.py`'s `--pin-source openXdox=` has the same shape (there a finding naming `.`).
- **`validate-pins.py`'s `(sibling / ".git").exists()` is unguarded** (the sibling row above), folded into lane 2's #430, which already covers the Python 3.14 half: the pre-3.14 traceback and its reproduction are added there as a comment. #312's file.
- **The project's OWN manifest** with `neutral_product_pins: 5` ends in a `TypeError` in two gates before either can say anything about it (#443): `_pins_from_project` in the naming gate (`scripts/validate-repository-naming.py:302` on main), and `_declared_pins` in the project's own `validate-manifest.py` (`templates/assembly-root/scripts/validate-manifest.py:85`), which ends it before its own `manifest-neutral-product-pins` finding (`_neutral_product_pins_type_finding`, `:524` on main) can print. Reproduced on 3ef059c. That is the project's own declaration, read by the project's own gates, and not a link source, so it is outside this change; "read by the project's own gates" covers two crashing gates.

Also not done here:
- **Python 3.9 (the floor) was not run** because it is not installed here. The grammar of the three changed files was checked with `ast.parse(feature_version=(3, 9))`. Windows and macOS were not run here; CI ran them (the last paragraph).

Reviewed adversarially at 40762ca, the branch's first commit (CHANGES NEEDED: 0 blockers, 2 should-fixes, 4 nits). The head stopped every crash in the issue's table, and the should-fixes were a docstring claim the code did not keep for a `neutral_product_pins:` that is not a list, and reach claims that overstated what the project's gate does on the sibling row. Fix round 1 (4b21263) resolved every finding, and the re-review of 4b21263 read APPROVE WITH NITS (0 blockers, 0 should-fixes, 3 nits: three behaviour changes this description had not stated, two of them untested; a spy-test case for a qualified chain link; and description corrections). Fix round 2 (2ac911c) resolved those three, and its re-verification of 2ac911c read APPROVE WITH NITS (0 blockers, 0 should-fixes, 2 nits: no test of the scaffold's wider wording for a manifest that is not a mapping or does not parse, now pinned by two tests that each fail when the wording is narrowed, and a cross-reference in this description that pointed the wrong way, now corrected). Main (9736424, #409's squash) was merged into the branch as ced8fac. `scripts/repo_shape.py` conflicted in four hunks; each takes both sides, as the review of 2ac911c resolved it: `resolve_link_source(..., default_source=None, env_shared: bool = False, *, tried=None)`, both docstrings kept, `link_pins_from_trees` opening with `found`, `unreadable`, `links` and `shared`, and the call passing `pin_product_key(name) in shared` with `tried=asked`. Nothing else conflicted. After the merge, `tests/test_pin_source_env_collisions.py` and the targeted set passed (667 and 312), and `render-cli-reference.py --check` printed `[ok]`. CI is green on Linux, macOS and Windows, with SonarCloud passing, at 938a131 (main 48585e6 merged in). Its merge with main 994be39 passed the targeted set (the naming, manifest, scaffold, pin and setup test files): 983 passed.

Closes #338

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant