Skip to content

family.py: add --dry-run makes every refusal add makes before it mounts, reads the member from throwaway clones outside the holder for the row's id and digest, and prints the row, gitlink, .gitmodules section and trailers without mounting, writing or committing anything, and add takes --trailer as bump does, so every subcommand has the dry run the module's EXIT CODES line promises (#326) - #400

Merged
brettheap merged 9 commits into
mainfrom
fix/family-add-dry-run
Oct 12, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Lane: openreposhape-4 (openRepoShape-4)
Claim: #326 (comment)

What

family.py add takes --dry-run. This is the second and last part of #326. Part 1, #345 (3ae0109), gave bump and remove their dry runs and left add out until the changes to cmd_add then in flight had landed. With this PR, every subcommand has the dry run the module's EXIT CODES line promises.

add also takes --trailer, the same repeatable flag bump has. Ruling 4 asks --trailer with --dry-run to print the trailers it would write, and add had no --trailer at all. The flag goes into the real commit too, through _commit_one, as bump's does. Without it, add's commit is byte for byte the one it has always written.

Why

Until now the only way to see what add would pin was to run it: clone the member, git submodule add it, rewrite family.yaml and commit.

The rulings, as implemented

  1. Every refusal before the first write, in the same order, with the same codes, exit 2. cmd_add leaves for _add_dry_run after the probe and before MountRollback is taken or git submodule add runs. So every refusal before that point is the real run's own code, in the real run's order:

    • the holder's: member-malformed, member-already-present, member-path-taken, member-path-unaddable, member-name-taken, member-path-registered, member-git-dir-cached, family-members-unrecognised, member-at-not-a-commit;
    • the member's, asked of the probe's throwaway clone: member-no-tip, member-at-unknown, member-not-a-project, the manifest's own refusals, and an unreadable member (REFUSED reading ... plus the hint).
  2. It reads the member, outside the holder.

    • A real add reads the member twice: the probe's clone, then the mount, whose reading the row records. The dry run has no mount. Its second reading is a second bare clone, made as the probe makes its own: in a temporary directory outside the holder, and deleted whatever the answer, a refusal included.
    • It reads that clone at the commit the probe resolved. The id: comes from _member_id_at, which reads project.yaml with the function the mount's checkout is read with. The digest comes from tree_digest, which reads objects and never a checkout.
    • It never runs git submodule add. It never writes .gitmodules, family.yaml, the index, the work tree, .git/config or .git/modules.
    • The help and the module docstring say it reads the member.
  3. What it prints, in bump --dry-run's layout, exit 0:

    • commit none -> <sha> and the digest, which is the real run's, from the same tree;
    • the gitlink;
    • the .gitmodules section git submodule add writes: its name (the path), its path and its url, and in a new .gitmodules for a family's first member;
    • the registration it writes in .git/config, or the one there whose url it rewrites;
    • the directory and the object store the mount makes, where git keeps the store;
    • the row as family.yaml would carry it, built by _member_row, as the real run's is;
    • the trailers.

    Without --at, a note says the real run pins the tip it finds when it mounts, and prints the --at <sha> that pins the commit shown.

  4. --trailer with --dry-run prints trailers N to append, in order: and the lines, after the row, as part 1 does. With no --trailer it prints trailers none.

  5. The module docstring and the help. EXIT CODES now says every subcommand takes --dry-run. The dry-run paragraph says what add's dry run asks and that it reads the member, and add's --help says so too. docs/cli.md was regenerated in this worktree under GIT_ALLOW_PROTOCOL=file, and render-cli-reference.py --check passes. AGENTS.md is not changed.

  6. Closes #326.

What it names and does not ask:

  • A linked project.yaml. The objects hold only the link's text, and the real run's mount reads through it. The dry run says so in a note and prints id: (read through the link by the real run); it does not guess an id. The real run then writes the rest of the row as printed (test, POSIX only).
  • A .gitmodules the index or HEAD has and the work tree does not. Git 2.43 refuses this case in git submodule add before it writes anything (please make sure that the .gitmodules file is in the working tree). The real run exits 2 there with the holder unchanged. Only git can say that sentence, so the dry run says in a note that the real run will exit 2, and how to put the file back, then exits 0. It never claims a new .gitmodules. Part 1 treats the checkout's own refusal the same way, and the module docstring says so.

Composes with

Each row is git merge-tree --write-tree origin/<branch> e792bd3, at the branch's head after a fetch at 2026-10-12T16:20Z, over the same tree as de628fe. Every one exits 0.

branch PR head files shared with this PR merge
main 1f90a0f none clean
fix/family-add-member-id #381 8c14f79 scripts/family.py (cmd_add) clean
fix/family-member-re #399 37e8f1e scripts/family.py clean
fix/family-init-values #397 abb27e6 scripts/family.py, docs/cli.md clean
fix/trailer-non-ascii-old-git #376 d079893 scripts/family.py clean
fix/ruleset-hint-tracking-branch #373 741cbf2 scripts/family.py clean
fix/scaffold-tracking-branch-ref-format #355 a03abcf scripts/family.py clean
fix/template-quoted-placeholders #395 3894aa8 scripts/family.py clean
fix/adopt-plan-leg-path-help #401 6124cb4 docs/cli.md clean
fix/scaffold-manifest-values #314 a5438a5 docs/cli.md clean

#381 adds member-duplicate-id right after the probe. The dry-run return sits after that point, separated from #381's insertion by the blank line that follows the probe's except. So the merge is textually clean, and the id check comes before the return.

I probed the merged tree offline: a holder pinning IRQS, then add --member InkRouter/IRQS2 (a bare copy with the same id). Real and --dry-run both exit 2 with REFUSED member-duplicate-id: ..., their stdout and stderr byte-identical. The index is unchanged and no temporary clone is left.

Tests

The tests are appended to tests/test_family_dry_run.py; part 1's tests are untouched. They use the same local-bare-remote family, with a gh shim that exits 97 and git's automatic maintenance off. Every "changes nothing" claim is checked as bytes, compared in the order a mount would change them: index, .gitmodules, the .git/modules listing, .git/config, family.yaml, status --porcelain=v1 -z, the work tree, the holder's objects, HEAD. A file that is not there is compared as absent.

  • A real add previewed, for a holder with members and for a family's first member (no .gitmodules: absent stays absent, and the section says in a new .gitmodules). Nothing moved, no gh call, and no temporary clone left. The real add then writes exactly what was printed:

    • the printed row;
    • the printed digest;
    • the gitlink;
    • the section (path and url, read back with git config -f .gitmodules);
    • the registration, the directory and the store;
    • no trailers.

    The holder then validates.

  • 13 refusals before the first write, each run real and --dry-run over the same holder. Both give exit 2 with the same stdout and the same stderr, normalized only for the throwaway clone's random directory name in a failed clone's command line. The holder is unchanged and nothing is left behind. The cases:

  • --trailer with --dry-run prints them, in order, after the row, and the real commit ends with exactly those lines.

  • --at previews that commit, with no without --at note, and the real add pins it.

  • A registration the mount rewrites is named as rewritten, and the real add leaves the url that was printed.

  • The .gitmodules note, for an unstaged and a staged deletion: the dry run exits 0 with the note, and the real add exits 2 on git's line. The holder is unchanged after both.

  • A linked manifest (POSIX only), as above.

  • In process, every program recorded:

    • both clones are under the run's TEMP, and none is in the holder;
    • the holder's git status, asked while the dry run's own clone still exists (at the moment its digest is computed), is the status before the run;
    • no git command run in the holder writes. Git's own add --dry-run, the real run's check of the mount path, is the one exception;
    • no git submodule runs.
  • An id that is not the name lowercased (fix round, S1). Member IRDI records id: routing-core. The dry run prints id: routing-core, and the real add writes exactly the printed row.

  • The dry run's own clone, removed when it fails (fix round, N3). In process, the member's remote is moved away between the probe's clone and the dry run's own, so the second clone fails in git, for real. The dry run gives exit 2 with REFUSED reading, the exact command and the hint. Its clone's directory is gone, TEMP is empty, and the holder is unchanged. The test does not stub _probe_member, whose return family.py: add refuses member-duplicate-id before it mounts anything when the member's project.yaml records an id a row already pins, so adding a repository renamed at the forge no longer commits a second row the holder's own validate-family.py refuses (#330) #381 changes.

  • A holder that is a linked worktree (fix round, N3). The object store the dry run names, under .git/worktrees/<name>/modules/, does not exist before the real add and is the one the real add makes.

Targeted set (Linux, git 2.43.0, CI's git-filter-repo pin): tests/test_family_dry_run.py, test_family.py, test_family_add_refusals.py, test_family_member_path.py and test_cli_reference.py.

  • On head 8bcc57c: 269 passed in 433.53s (0:07:13), 0 skipped.
  • On head de628fe (e792bd3 before its message took these results; the tree is the same, 1ef88b7): 272 passed in 192.45s (0:03:12), 0 skipped.

test_family_add_refusals.py also gains one docstring sentence. It said add had no --dry-run; it now points here.

Mutants

Each mutant was run in a scratch worktree at the head against the test the ruling names (mutants.py, restored after each):

mutant test fails on
the dry-run return moved after git submodule add test_add_dry_run_prints_what_a_real_add_writes_and_changes_nothing[two-members] AssertionError: index changed
same test_add_dry_run_reads_the_member_outside_the_holder_and_writes_nothing AssertionError: the holder's status moved while the member was being read (M .gitmodules ...)
the dry run's temporary clone made inside the holder (mkdtemp(dir=family.root)) test_add_dry_run_reads_the_member_outside_the_holder_and_writes_nothing AssertionError: the holder's status moved while the member was being read (?? openreposhape-member-…/)
the dry run's temporary directory not removed test_add_dry_run_prints_what_a_real_add_writes_and_changes_nothing[two-members] offline.assert_left_nothing(): […/temp/openreposhape-member-yc8fysxa] == []

The second mutant survives the end-state test, and the table shows why it must. A clone made inside the holder and deleted at the end leaves no trace in any end state. That is why the in-process test asks git status while the clone exists.

The fix round's mutants are the reviewer's own M4, M10 and M11, applied by the reviewer's mutants.py in a scratch worktree at e792bd3 (the same tree as de628fe). Each was run over the same selection the review used (pytest tests/test_family_dry_run.py -k add, now 25 tests). All three survived at 8bcc57c:

mutant result at e792bd3 killed by fails on
M4: the id made from the name, project.lower(), for any non-linked manifest 1 failed, 24 passed test_add_dry_run_prints_the_id_the_members_manifest_records At index 1 diff: ' id: irdi' != ' id: routing-core'
M10: the dry run's own clone leaked on its failure path (finally turned into else) 1 failed, 24 passed test_add_dry_run_removes_its_own_clone_when_reading_the_member_fails AssertionError: the dry run's clone was left
M11: the store assumed to be .git/modules/<path> 1 failed, 24 passed test_add_dry_run_names_the_store_git_keeps_for_a_linked_worktree AssertionError: the store printed, …/lane/.git/modules/members/IRQS, is not the one git made

Fix round (review of 8bcc57c)

The review's verdict was MERGEABLE WITH NITS: 0 MUST, 1 SHOULD, 6 NIT. I added one commit, de628fe, on top of 8bcc57c, pushed plainly; nothing on origin was amended or rebased. It changes tests only: scripts/family.py is byte-identical to 8bcc57c's.

Fix round 2 (stacked on #399, main merged, member-duplicate-id parity)

Added on top of de628fe, pushed plainly; nothing on origin was amended or rebased: two merge commits, then one commit of tests/test_family_dry_run.py and a docstring in scripts/family.py.

Tests: ADD_REFUSALS grows from 13 rows to 19. A set-up may return the directory of remotes the add is to read from; the others return None.

Mutants, each run on the parametrized refusal test over the malformed and duplicate-id rows (8 cases):

mutant result fails on
the dry run skips _checked_member_id (if probed_id is not None and not args.dry_run) 1 failed, 7 passed [duplicate-id]: the dry run exits 0 and prints the row for IRRS2, the real run exits 2
the dry run takes --member without _checked_member 7 failed, 1 passed every malformed* row, the five new ones included; [duplicate-id] passes
MEMBER_RE loosened to the pre-#399 class 5 failed, 3 passed exactly the five new rows; malformed, malformed-dot and [duplicate-id] pass

Targeted set: tests/test_family_dry_run.py, test_family_add_refusals.py, test_family_add_member_id.py, test_family_member_name.py, test_family_member_path.py, test_family.py, test_cli_reference.py and test_agents_md_codes.py, on the bytes of head 1d82455 (sha256 of each file recorded first and re-checked): 352 passed in 331.72s (0:05:31), 0 failed, 0 skipped (Linux, git 2.43.0, CI's git-filter-repo pin).

Still open (N6, unchanged by this round): a linked project.yaml whose target records an id a row pins gets exit 0 from the dry run, and the real run refuses member-duplicate-id after the mount and unwinds. That is inside the module docstring's stated limit (what fails only once the mount is made is never reached by the dry run), and the linked note does not yet name this refusal.

Follow-ups (not in this PR)

Closes #326

🤖 Generated with Claude Code

Summary by Sourcery

Enable family.py add to preview its complete pinning operation safely while adding commit trailer support and aligning member-name validation with holder checks.

New Features:

  • Add family.py add --dry-run to preview the member pin, gitlink, .gitmodules entry, registration, storage location, row, and commit trailers without modifying the holder.
  • Add repeatable --trailer support to family.py add and include trailers in dry-run previews and real commits.

Bug Fixes:

  • Reject member names beginning with punctuation that the holder's own validation rules would reject, preventing invalid repository rows from being committed.

Enhancements:

  • Ensure add dry runs preserve pre-mount refusal behavior and read members from temporary clones outside the holder.
  • Document the expanded add command behavior and regenerate the CLI reference.
  • Expand coverage for add previews, refusal parity, trailer handling, linked manifests, temporary-clone cleanup, and linked-worktree storage paths.

Documentation:

  • Update the family CLI and module documentation to describe add dry runs, trailer support, and their limitations.

Tests:

  • Add comprehensive tests covering add dry-run output, no-change guarantees, refusal parity, trailers, pinned commits, linked manifests, cleanup, and storage locations.
  • Add tests enforcing consistent member-name validation across add and the repository validators.

brettheap and others added 2 commits October 12, 2026 15:47
… begin with a letter or a digit as `member-malformed` before the member is read, so `testorg/_Zed`, `testorg/.Zed`, `testorg/-Zed`, `_org/Zed` and `.org/Zed` no longer exit 0 with a commit that the holder's own `validate-family.py` fails as `member-repository` (#366)

`MEMBER_RE` and the validator's `QUALIFIED_RE` shared one character class;
the validator also requires each segment's first character to be a letter or
a digit. `add` writes `--member` verbatim as the row's `repository:`, so a
name the looser pattern admitted was mounted, pinned and committed with
exit 0, and the gate the holder ships then failed the commit `add` had just
reported. Two tools of this standard disagreed about one rule, and the one
that writes the file was the looser.

`MEMBER_RE` now takes `QUALIFIED_RE`'s shape. The rulings:

1. THE `.`/`..` CLAUSE IS KEPT, AND MOVED FIRST. The pattern would refuse
   `<org>/..` too, for beginning with a dot, but in its general sentence;
   what is wrong with `.` and `..` is that they are no name at all, and
   #299's more specific sentence says that. It is read before the pattern, so
   it is still reachable, and only on a name with a `/` (a bare `.` has no
   owner to blame, and the general sentence asks for one). Its remedy no
   longer offers `.github` as the example of a name with a dot, since
   `.github` is refused now, by the validator too; it says `Ze.d`.
2. ONE RULE, PINNED, NOT MOVED. `repo_shape.py` is a copied file: a constant
   moved there would reach an existing holder only through `update-shape.py`,
   and both validators spell `QUALIFIED_RE` themselves and ship into projects
   that run them from their own copy. So `family.py` spells it a third time,
   and `tests/test_family_member_name.py` reads every spelling (the source,
   never running the validators: it finds `MEMBER_RE` and every tracked
   module's `QUALIFIED_RE` outside `tests/`, so a fourth copy is pinned the
   day it is written) and fails when they are not the same pattern. A corpus
   of names says what that pattern means, so a change that keeps the
   spellings equal and moves the rule has to move the corpus too.
3. THE VALIDATOR'S RULE IS UNCHANGED. Which names GitHub accepts is not
   claimed; `testorg/.github` was refused by the validator already, so
   nothing a holder can keep depended on `add` admitting it.

The general sentence now says the rule (each name begins with a letter or a
digit, then letters, digits, `.`, `_`, `-`) and its remedy says why: a
commit that wrote such a name would fail the gate the holder ships.

BEHAVIOUR THAT MOVES BESIDE THE FIVE. `--member=-org/Zed`, the one way a
leading `-` reaches the owner, passed the old pattern and was refused later
by `checked_value` as `unsafe-value`; the pattern is asked first, so it is
`member-malformed` now (exit 2, nothing read, either way). `testorg/.github`
and `testorg/...` are refused. `tests/test_family_member_path.py`'s
`test_add_still_takes_a_name_that_contains_a_dot` asserted `.github` was
admitted and its remedy string offered `.github`: both are corrected, `a.b`
stays.

TESTS, `tests/test_family_member_name.py`, members from local bare remotes
(a bare copy of the fixture's spare member under each name): `add --member`
over the five spellings of the issue plus `-org/Zed` and `.github`, each
`member-malformed`, exit 2, empty stdout, no `gh` call, no throwaway clone,
and the holder byte for byte (HEAD, `ls-files -s -z`, `status
--porcelain=v1 -z`, `family.yaml`, the rest of the snapshot); controls that a
name with an inner `.`, `_` or `-` (and an owner with an inner `.`) still adds
in ONE commit and the holder's `validate-family.py` passes; the
pinned-equality test and the corpus. MUTANT: the old `MEMBER_RE` restored
makes the seven refusal cases and both pin tests fail (9 of 13; the four
controls pass), the refusals with exit 0 and a committed row.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…mounts, reads the member from throwaway clones outside the holder for the row's id and digest, and prints the row, gitlink, `.gitmodules` section and trailers without mounting, writing or committing anything, and `add` takes `--trailer` as `bump` does, so every subcommand has the dry run the module's `EXIT CODES` line promises (#326)

The second part of #326. Part 1 (#345) gave `bump` and `remove` a
`--dry-run` and left `add` out until the changes to `cmd_add` in flight
had landed. Until now the only way to see what `add` would pin was to run
it: clone the member, mount it, rewrite `family.yaml` and commit.

`add --dry-run` leaves `cmd_add` after the probe, which asks the member's
refusals of a throwaway clone, and before `MountRollback` is taken or
`git submodule add` runs. So every refusal `add` makes before its first
write is the real run's own code, asked in the real run's order, with the
same code, the same words and exit 2. That covers the holder's refusals
(`member-malformed`, `member-already-present`, `member-path-taken`,
`member-path-unaddable`, `member-name-taken`, `member-path-registered`,
`member-git-dir-cached`, `family-members-unrecognised`,
`member-at-not-a-commit`) and the member's (`member-no-tip`,
`member-at-unknown`, `member-not-a-project`, the manifest's, and an
unreadable member). Lane 3's #381 adds `member-duplicate-id` right after
the probe; the dry-run return sits after that point, so on a merge it is
asked under `--dry-run` too.

The member is read a second time, as the real run reads it a second
time. The real run reads it from the probe's clone, then from the mount,
whose reading the row records. The dry run has no mount, so it makes a
second bare clone, the way the probe makes its own: in a temporary
directory outside the holder, deleted whatever the answer. It reads that
clone at the commit the probe resolved. The `id:` comes from
`_member_id_at`, through the function the mount's checkout is read with.
The digest comes from `tree_digest`, which reads objects and never a
checkout, so the digest printed is the one the real run computes from the
same tree, never a guess. A `project.yaml` that is a symbolic link is
left to the real run's mount, which reads through it; the dry run says so
and prints no id.

It prints, in `bump --dry-run`'s layout:
- the commit and digest;
- the gitlink;
- the `.gitmodules` section `git submodule add` writes, under its name and
  with its url, and whether the file is new;
- the registration in the holder's `.git/config`, or the one whose url it
  rewrites;
- the directory and object store the mount makes;
- the row as `family.yaml` would carry it;
- the trailers.
Without `--at`, a note says the real run pins the tip it finds when it
mounts, and gives the `--at` that pins the commit printed. The dry run
never runs `git submodule add`. It never writes `.gitmodules`,
`family.yaml`, the index, the work tree, `.git/config` or `.git/modules`.
Exit 0.

`add` had no `--trailer`, so ruling 4 (`--trailer` with `--dry-run` prints
the trailers it would write) needed one. It is `bump`'s flag: repeatable,
kept in order, validated by `trailer_line`, and carried into the one
commit through `_commit_one`, as `bump` carries it. With no `--trailer`,
the commit is the one `add` has always written.

Git's own refusal at the mount is named, not made. With no `.gitmodules`
in the work tree and one in the index or HEAD, `git submodule add`
refuses to write it before writing anything (git 2.43: `please make sure
that the .gitmodules file is in the working tree`), and the real run
exits 2 there with the holder unchanged. Only git can say that sentence.
So the dry run prints a note saying the real run will exit 2 and how to
put the file back, and exits 0. That is how part 1 treats the checkout's
own refusal. The module docstring says so.

The module docstring's `EXIT CODES` line now says every subcommand takes
`--dry-run`. `add`'s help names the dry run and says it reads the member.
`docs/cli.md` is regenerated. AGENTS.md is not changed.

Tests, appended to tests/test_family_dry_run.py over the same
local-bare-remote family:
- a real add previewed, both for a holder with members and for a family's
  first member (no `.gitmodules`: absent stays absent). The dry run leaves
  these byte-identical: HEAD, `ls-files -s -z`, `status --porcelain=v1 -z`,
  `family.yaml`, `.gitmodules`, `.git/config`, the `.git/modules` listing,
  the work tree and the holder's objects. No `gh` call, no temporary clone
  left. The real add then writes exactly the printed row, section url,
  gitlink, registration and directory, and the holder validates;
- 13 refusals before the first write, each giving the same stdout, the
  same stderr and exit 2 from the real run and the dry run, with the
  holder unchanged;
- `--trailer` with `--dry-run` printed, and carried by the real commit;
- `--at` previewed and pinned;
- a registration the mount rewrites;
- the `.gitmodules` note, unstaged and staged deletion: dry run exits 0,
  real run exits 2 on git's line, holder unchanged;
- a linked manifest, POSIX only;
- in process: both clones are under the run's TEMP and none is in the
  holder; the holder's `git status`, asked while the dry run's own clone
  exists, is the status before; no git command in the holder writes
  (git's own `add --dry-run` aside); no `git submodule`.

Targeted set over this tree (Linux, git 2.43.0, CI's git-filter-repo
pin): tests/test_family_dry_run.py, test_family.py,
test_family_add_refusals.py, test_family_member_path.py and
test_cli_reference.py: 269 passed in 433.53s.

Mutants, each run against the test the ruling names, each killed:
- the dry-run return moved after `git submodule add`: `index changed`;
- the dry run's temporary clone made inside the holder: `the holder's
  status moved while the member was being read`. This is asked while the
  clone exists, because an end-state test cannot see a clone deleted
  before the end;
- the temporary directory not removed: the leftover check
  (`offline.assert_left_nothing`).

Closes #326

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 1 hour by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds a fully read-only family.py add --dry-run that performs the same pre-write validation, reads the member from disposable external clones, and prints the exact prospective pin, manifest, gitlink, Git configuration, and trailers. It also adds repeatable --trailer support to real and dry-run adds, with extensive regression coverage and updated CLI documentation.

Sequence diagram for family add dry-run preview

sequenceDiagram
    participant User
    participant Add as family.py add
    participant Holder
    participant Probe as Probe clone
    participant Preview as Preview clone

    User->>Add: add --dry-run [--at] [--trailer]
    Add->>Holder: Validate add target
    Add->>Probe: Clone member and probe refusals
    Probe-->>Add: Resolved commit
    Add->>Preview: Clone member outside holder
    Preview-->>Add: Read project id and tree digest
    Add->>Holder: Check prospective git state
    Add-->>User: Print commit, digest, gitlink, .gitmodules, row, trailers
    Add-->>User: Exit 0 without mounting or writing
Loading

Sequence diagram for family add commit with trailers

sequenceDiagram
    participant User
    participant Add as family.py add
    participant Git
    participant Member
    participant Holder

    User->>Add: add [--at] [--trailer]
    Add->>Git: git submodule add
    Git->>Member: Clone and checkout member
    Member-->>Git: Mounted working tree
    Add->>Holder: Read member id and tree digest
    Add->>Git: _commit_one with row and trailers
    Git-->>Holder: Commit gitlink, .gitmodules, family.yaml
    Add-->>User: Exit 0
Loading

Flow diagram for add dry-run refusal boundary

flowchart TD
    A[family.py add --dry-run] --> B[Validate holder target]
    B --> C[Probe member clone]
    C --> D{Pre-write refusal?}
    D -->|Yes| E[Return same refusal and exit 2]
    D -->|No| F[Second disposable clone outside holder]
    F --> G[Read member id and tree digest]
    G --> H[Print prospective files, git state, row, and trailers]
    H --> I[Exit 0 with holder unchanged]
Loading

File-Level Changes

Change Details Files
Implemented a side-effect-free family.py add --dry-run that mirrors pre-mount validation and previews the member pin and all holder changes.
  • Routes execution to the dry-run path after holder and probe refusals, preserving refusal codes, ordering, and output.
  • Reads the resolved member from a second bare clone in a temporary directory outside the holder, computes its ID and tree digest, and always cleans up.
  • Prints the prospective commit, digest, gitlink, .gitmodules section, registration, module directory/store, manifest row, trailers, and notes for mount-only cases.
  • Adds coverage for successful previews, first-member mounts, all pre-write refusals, --at, registrations, missing work-tree .gitmodules, linked manifests, cleanup, and no-write guarantees.
scripts/family.py
tests/test_family_dry_run.py
Added repeatable commit trailer support to add and propagated trailers through the real mount-and-commit path and dry-run output.
  • Adds --trailer to the add parser using the same validation and ordering behavior as bump.
  • Passes trailers into _mount_and_pin and verifies the real commit ends with the requested lines.
  • Documents trailer behavior in generated CLI help.
scripts/family.py
docs/cli.md
tests/test_family_dry_run.py
Updated family command documentation to state that every subcommand supports dry runs and explain add's read-only behavior and mount-only refusals.
  • Updates the module exit-code and dry-run documentation, including linked manifests and .gitmodules caveats.
  • Regenerates family.py add CLI reference with --trailer and --dry-run details.
  • Updates refusal-test documentation to point to the new add dry-run coverage.
scripts/family.py
docs/cli.md
tests/test_family_add_refusals.py

Assessment against linked issues

Issue Objective Addressed Explanation
#326 Add a bump --dry-run that resolves --to, previews the row and digest it would write, and exits without checking out the member or modifying the holder. ✅
#326 Provide --dry-run for add and remove as well, so those operations can be previewed without modifying the holder. ✅

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

brettheap and others added 2 commits October 12, 2026 16:23
…member's own `project.yaml` records, to deleting its own clone when the member's second reading fails, and to the object store git keeps for a linked-worktree holder, so an invented id, a leaked clone or an assumed `.git/modules` path no longer passes (#326)

The fix round on the review of 8bcc57c (PR #400). Each of the three
behaviours was right in the code, and no test held it.

S1, never an invented id (ruling 3). Every member the family fixture
scaffolds records `id:` as its name lowercased (`irqs`, `irln`). The
reviewer's mutant M4 printed `project.lower()` for every non-linked
manifest, and all 22 `add` tests passed. The new member `IRDI` is built
by `one_commit_remote` with `id: routing-core`. The dry run prints that
id, read from the member, and the real add then writes exactly the
printed row.

N3, the dry run's own clone removed on a refusal (ruling 2, "removed
afterwards, also on a refusal"). Every refusal the tests provoked was
made before that clone exists, so mutant M10 (`finally` turned into
`else`, leaking the clone on its failure path) survived. The new test
runs in process. It moves the member's remote away between the probe's
clone and the dry run's own, so the second clone fails in git, for
real. The dry run must then:
- exit 2;
- print `REFUSED reading`, the exact command and the hint, as every
  failed reading of the member is printed;
- leave its clone's directory gone, TEMP empty and the holder as it was.
The test does not stub `_probe_member`, whose return #381 changes, so
it holds on that merge too.

N3, the store named where git keeps it. Mutant M11 printed
`.git/modules/<path>` and survived. The new test adds from a holder
that is a linked worktree, where `.git` is a file and git keeps a
mount's store under `.git/worktrees/<name>/modules/`. The store the dry
run names must not exist before the real add, and must be the one the
real add then makes.

`add_state` reads `.git/objects` and `.git/modules` as absent where
`.git` is a file, which is what a linked worktree has.

Mutants, the reviewer's own (applied by its `mutants.py` in a scratch
worktree at this tree), each over the review's selection
`tests/test_family_dry_run.py -k add`, 25 tests here. Each survived at
8bcc57c, and each is killed now by its one new test, 24 others passing:
- M4, the id made from the name: `'    id: irdi' != '    id:
  routing-core'`;
- M10, the clone leaked on its failure path: `the dry run's clone was
  left`;
- M11, the store assumed: `the store printed, .../lane/.git/modules/
  members/IRQS, is not the one git made`.

Targeted set over this tree (Linux, git 2.43.0, CI's git-filter-repo
pin): tests/test_family_dry_run.py, test_family.py,
test_family_add_refusals.py, test_family_member_path.py and
test_cli_reference.py: 272 passed in 192.45s, 0 skipped.

Part of #326, which #400 closes.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 12, 2026
…d no others, that they write no plan and rewrite nothing, so a leg path another change refuses under a third code (`null` as `adopt-manifest-invalid`) does not make the closing sentence false (#340)

The previous commit's closing sentence began "A refusal", and a test expected
a YAML word (`null`, `2026`, `true`, `1.5`) to be planned or refused as
`adopt-bad-leg-path`. Composing the head with main, #314, #397 and #400 showed
a third outcome: #314 makes `plan` refuse all four as `adopt-manifest-invalid`
("the manifest this run would write is one the project's own
validate-manifest.py refuses"), under a Remediation of its own. With #266 the
same four are `adopt-bad-leg-path`; without either they are planned (#181). So
the test guessed which of three futures lands, and failed on one of them.

- The closing sentence now begins "Either refusal", so it holds for the two
  codes the help names and claims nothing of a code it does not.
- The YAML test is removed. The REFUSED rows already hold the sentence to the
  tool for every value the help describes: no plan file, and a Remediation that
  begins "nothing here rewrites" or "nothing here picks".
- The docstring says why the help does not say what `plan` accepts, and that a
  value the YAML reader misreads is in none of the clauses.

`docs/cli.md` is regenerated by `scripts/render-cli-reference.py`; `--check`
passes.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brettheap and others added 4 commits October 12, 2026 17:42
…n so CI tests the two stacked (#326)

#399 (`MEMBER_RE`, `_checked_member`, #366) lands before this PR and also edits
`scripts/family.py`. A Windows CI run now takes about 55 minutes, so this branch
is stacked on #399 here: one CI cycle tests main, #399 and #400 together, and
this PR can land right after #399 without a second one. The merge was clean (no
conflict); `add --dry-run` calls `_checked_member` as `add` does, so the new
member-name rule is asked under `--dry-run` through the same function.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…add-dry-run (#326)

Main took #396 (`tests/test_agents_md_codes.py`) after #399's last merge of it, so
it is merged here as well: the branch now holds main, #399 and #400 and CI tests
all three. The merge was clean (no conflict); the one file it adds is the new
test that holds AGENTS.md's code-shaped names to the shipped tools.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…icate-id` and to #399's `member-malformed` names, so a dry run that skipped either check no longer previews a row the real `add` refuses, a copy of a pinned project under a new name or a name beginning with `.`, `_` or `-` included (#326)

Lane 3's #381 landed as f0a8bf6 before this PR: `add` refuses a member whose
`project.yaml` records an `id` a row already pins, as `member-duplicate-id`,
before it mounts anything. The review of 8bcc57c said whichever of the two
landed second must give that refusal a row in `ADD_REFUSALS`, and that is
this PR. #399 (`MEMBER_RE`, #366) also lands first and is merged here, so CI
tests main, #399 and #400 together; both merges were clean.

No behaviour changes. `cmd_add` asks `_checked_member_id` of the probe's
reading just before the dry-run return, so the dry run reaches the check
through the same function as the real run, with the same id, and
`member-duplicate-id` comes out of `--dry-run` with the same stdout, the same
stderr and exit 2. Only the `_add_dry_run` docstring, which listed the
holder's refusals and the member's, now names it.

- `ADD_REFUSALS` gains `duplicate-id`, last, in `add`'s own order: the member
  is `IRRS2`, a bare copy of the pinned `IRRS` made beside the holder from the
  remote its `.gitmodules` names and never in the module's shared remotes,
  as f0a8bf6's own tests build the renamed repository. A set-up may now return
  the directory of remotes the add reads from; the others return None.
- It gains five `malformed-*` rows for #399's rule, after `malformed-dot`:
  a project beginning with `_`, `-` and `.`, and an owner beginning with `_`
  and `.`. `<org>/..` and a bare name, the table's old two, are refused by
  clauses that predate the rule, so none of them covered it. An owner
  beginning with `-` cannot be said as `--member <owner>/<project>` (argparse
  reads it as an option): it stays with the real run's test in
  `test_family_member_name.py`.

Mutants, each run on the parametrized refusal test over the `malformed` and
`duplicate-id` rows:
- the dry run skips `_checked_member_id`: `[duplicate-id]` fails (the dry run
  exits 0 and prints the row for `IRRS2`; the real run exits 2), the other
  seven pass;
- the dry run takes `--member` without `_checked_member`: the seven
  `malformed*` rows fail, `[duplicate-id]` passes;
- `MEMBER_RE` loosened to the pre-#399 class: exactly the five new rows fail.

Targeted set (Linux, git 2.43.0, CI's git-filter-repo pin):
`test_family_dry_run.py`, `test_family_add_refusals.py`,
`test_family_add_member_id.py`, `test_family_member_name.py`,
`test_family_member_path.py`, `test_family.py`, `test_cli_reference.py` and
`test_agents_md_codes.py`: 352 passed.

Not changed: a linked `project.yaml` whose target records an id a row pins
still gets exit 0 from the dry run, and the real run refuses it after the
mount and unwinds, inside the module docstring's stated limit.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

brettheap pushed a commit that referenced this pull request Oct 12, 2026
…-bump-remove-rollback, so this PR is tested on top of #400, which lands first and conflicts with it (#365, #360)

#400 (`add --dry-run`, #326) carries #399 and main 3ef059c, and lands
before this PR. Both rewrote the end of the module docstring's "WHAT ONLY
THE WRITE ITSELF CAN ANSWER IS NOT ASKED" paragraph, and both changed
`tests/test_family_dry_run.py`'s import list from
`test_family_add_refusals`. Resolved as review 365-5d56e75 gives it:

- `scripts/family.py`: #400's `.gitmodules` sentence, then this PR's
  held-`index.lock` sentence, then #400's symbolic-link `project.yaml`
  sentence, and after a blank line this PR's "A REFUSED `bump` OR
  `remove` PUTS THE HOLDER BACK" paragraph, unchanged;
- `tests/test_family_dry_run.py`: the union of the two import lists.

main is not merged separately: #400's head carries it.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettheap
brettheap merged commit a92e60b into main Oct 12, 2026
5 checks passed
@brettheap
brettheap deleted the fix/family-add-dry-run branch October 12, 2026 19:08
brettheap pushed a commit that referenced this pull request Oct 12, 2026
…ly-bump-remove-rollback, so this PR conflicts with main no more (#365, #360)

#400 (`add --dry-run`, #326) landed on main as the squash a92e60b, and
this branch carries #400 as a merge of 7062a47, so main's merge base with
this branch is 3ef059c and the two docstring and import-list hunks of
`scripts/family.py` and `tests/test_family_dry_run.py` conflict again.
Resolved with this branch's side of both files
(`git checkout --ours`), after proving that it loses nothing of #400's:

- for each of #400's four files (`docs/cli.md`, `scripts/family.py`,
  `tests/test_family_add_refusals.py`, `tests/test_family_dry_run.py`),
  `git show a92e60b:<file>` and `git show 7062a47:<file>` are the same
  blob, so the branch's earlier merge of 7062a47 already holds all that the
  squash holds;
- no main commit after a92e60b touches any of the four
  (`git log a92e60b..origin/main -- <files>` is empty);
- both resolved files are byte-identical to 7cca975's, and no other file
  conflicted.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-12T19:54:30Z) A v1.0.0 scope notice. Brett Heap scoped openRepoShape's first release today: every procedure in AGENTS.md works end to end for ordinary repositories on Linux, macOS and Windows. The tag waits for every in-scope issue to close and a full test run to pass on all three OSes on the release commit. This PR closes #326 (P2, parent #459): in scope, in the v1.0.0 milestone. Nothing changes in how it lands. Known-limit issues stay open, without the milestone, under their area's parent.

brettheap pushed a commit that referenced this pull request Oct 12, 2026
…plan-manifest-invalid` joins `_entry_findings` after #255's values, and this branch's `plan-root-symlink` stays after the source's own submodules

origin/main moved 92e29d9 -> dd8d0ea (twelve commits), merged by sha:
- f0a8bf6 (#330 via #381), scripts/family.py;
- 87d2eb7 (#333 via #379), update-shape.py;
- cd1fc79 (#327 via #375), shape-doctor.py;
- 45e01f4 (#348 via #396), tests/test_agents_md_codes.py: every
  code-shaped name AGENTS.md spells is a code a tool prints;
- 3ef059c (#258 and #261 via #314), adopt-project.py, scaffold-project.py
  and scripts/shape_materialize.py: the project's YAML reader and its
  `validate-manifest.py` are asked about the manifest's values before
  anything exists; `check` reports `plan-manifest-invalid` through
  `_manifest_findings` in `_entry_findings`; `_repository_urls` refuses
  `adopt-name-too-long`;
- f7f4b14 (#366 via #399), scripts/family.py;
- cb7063f (#301 and #280 via #390), scripts/repo_shape.py and the
  template's validate-manifest.py: a bracket or brace in a plain flow
  scalar is refused;
- 5678f68 (#278, #297 and #298 via #357), shape-doctor.py;
- 53bee5b (#289 via #352), scripts/bump-leg.py;
- f5cb616 (#369 via #398), scripts/shape_materialize.py: the docstrings
  of `force_stage` and `stage_written`;
- a92e60b (#326 via #400), scripts/family.py;
- dd8d0ea (#292 via #353), adopt-project.py: `_elector` keeps a
  `user.name` byte that is not UTF-8, `_remote_repository` reads an
  origin URL that is not UTF-8 as none, and `adopt-no-org` is asked
  before `checked_value`.

scripts/shape_materialize.py, scripts/repo_shape.py and every other file
merge with no conflict. adopt-project.py conflicts in two hunks of
`_entry_findings`, each resolved by keeping both sides:
1. The docstring. Main's order, with #261's manifest findings after
   #255's values and before #190's relative urls; this branch's symlinks
   stay after the source's own submodules.
2. The return. Main's `_manifest_findings(plan)` call, and the findings
   in main's order: each entry's path (#196), coverage, the legs and what
   the root keeps where the shape writes, the source's own submodules
   (#166), this branch's `_root_symlink_findings`, #255's values, #261's
   manifest findings, and #190's relative urls last. The remediation and
   the NOTEs are main's.

How each of main's commits that reach this branch's code meets it:
- 3ef059c: `_manifest_findings` asks the validator the template ships
  about the values `execute` writes; it reads no symlink and no path
  the shape writes. Its materializer half adds the validator's helpers,
  `refuse_unstorable_names` and `MANIFEST_VALIDATOR`, and holds
  `election_date` to the validator's `DATE_RE`; it leaves `place`,
  `_materialize`, `copy_tree`, `template_files` and `_refuse_unwritable`
  as they were.
  In `execute`, its `adopt-name-too-long` (in `_repository_urls`) comes
  before #255's clone-URL refusal, `_refuse_what_check_finds` and this
  branch's `core.symlinks` probe, all before any leg. Its new paragraph
  of the module docstring follows the one 6b53a6d corrected; both merge
  with no conflict.
- f5cb616: docstrings only.
- dd8d0ea: `_remote_repository`, `_checked_plan_inputs`' organisation and
  `_elector`, none of them this branch's.
- cb7063f: the plans `check` and `execute` read go through this reader.
  The plans `plan` writes for this branch's tests hold their entries as
  block mappings, each path `double_quoted` where it needs to be, and
  the rehearsal below read every one of them.

Checked on the merged tree:
- AST (fin183g/names.py, base 92e29d9): no top-level name is bound twice
  in adopt-project.py, scaffold-project.py or scripts/repo_shape.py, and
  none is loaded but bound nowhere. scripts/shape_materialize.py binds
  `YamlError` and `parse_yaml` twice, as main dd8d0ea does (1f90a0f's
  import and 3ef059c's, the same objects); the merge adds none.
- Both sides intact (fin183g/sides_intact.py): the only lines that differ
  from either side's own change are those of the two hunks.
- A rehearsal of this merge (tree d21f8b4, the same as this one) passed
  tests/test_adopt_root_symlinks.py with main's
  test_adopt_{manifest_values,git_reads,relative_submodule_url}.py,
  test_scaffold_manifest_values.py, test_agents_md_codes.py and
  test_yaml_flow_brackets.py: `440 passed`.

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 12, 2026
… lines and clones nothing, and render-cli-reference.py refuses a tree that is not a checkout, so asking a download for help is no longer a clone of GitHub (#341) (#452)

From anywhere but a checkout of the standard (a downloaded `setup.sh`, `curl … | bash -s -- --help`, a `git archive` extraction), `setup.sh --help` went down section 0 like any other run. It cloned the public upstream before printing a word, died with git's exit 128 offline, and otherwise printed the CLONE's `setup-project.py` usage. `scripts/render-cli-reference.py` inherited all of it in such a tree: every rendering, `--check` included, reached GitHub and then documented upstream's flags under that tree's `setup.sh` heading. The clone attempts were reproduced on 87d2eb7, offline (`GIT_ALLOW_PROTOCOL=file`); what the generator renders when that clone succeeds is read from the code, not reproduced:

| | base 87d2eb7 | this branch |
|---|---|---|
| lone `setup.sh`, `bash setup.sh --help` | `(0) self-bootstrap`, `git clone --quiet --depth 1 -- https://github.com/opensoft/openRepoShape.git <tmp>`, `fatal: transport … not allowed`, exit 128 | the shim's own text, exit 0, no clone (`GIT_TRACE`), no temporary directory |
| same, `OPENREPOSHAPE_REPO=<checkout>` | a local clone, then the clone's `usage: setup-project.py …` | the shim's own text, no clone |
| `bash -s -- -h < setup.sh` | the clone, exit 128 | the shim's own text, exit 0 |
| `render-cli-reference.py --check` in a copy with no `.git` | tried the clone (stopped only by `GIT_ALLOW_PROTOCOL`, surfacing as `setup.sh --help exited 128`) | refuses, naming the tree, exit 2, no clone |

**What changed.**

**`setup.sh`**
- Section 0's one scan of the command line now also notes `-h` and `--help` before the first `--`.
- The shim answers them first: before the `$OPENREPOSHAPE_REPO` guard, before `mktemp`, before `git clone`. That is where `setup-project.py` answers them: before it looks at anything.
- The text says:
  - this is the bash front door to `setup-project.py`;
  - any other run from here first clones the standard into a temporary directory, from `$OPENREPOSHAPE_REPO` or, when that is unset or empty, `https://github.com/opensoft/openRepoShape.git`;
  - the one flag list is `python3 setup-project.py --help`, which answers with no network from a checkout or from that one file downloaded on its own, and `docs/cli.md` in the repository.
- It lists no flag, so #50's one `--help` still holds.
- The default is now one constant, `DEFAULT_SHAPE_REPO` (the name `setup-project.py` gives it), read by both the clone and the help. The help never prints the variable's value.
- Inside a checkout nothing changes: the shim still execs `setup-project.py` before section 0 reads a word.
- The header comment no longer says the shim "parses nothing". It now says the shim parses nothing of the RUN, and names the three things section 0 reads. The comment above the scan no longer says the scan never decides what the run does: the first two reads only decide whether a temporary directory is removed and whether a ref is checked out, and the third decides whether there is a run at all.

**`scripts/render-cli-reference.py`**
- `require_a_checkout()` asks `setup.sh`'s own `is_shape_checkout` question: `git -C <tree> rev-parse --show-toplevel`, and `scaffold-project.py` plus `contracts/repository-naming.yaml` at that top level.
- It also asks that the top level IS the tree (`os.path.samefile`). A copy nested inside another checkout passes the shim's probe, and its `setup.sh --help` would print that checkout's usage.
- It is called in `main()`, before the no-bash logic, and at the top of `render()`, which `tests/test_cli_reference.py` calls directly. The refusal names the tree and says to run it in a `git clone`, exit 2.
- It reads git's answer the way the shim's `$(...)` does: `os.fsdecode(...).rstrip("\r\n")`, so a checkout whose path ends in a space, or is not UTF-8, is a checkout here too.
- Every child now runs under `GIT_ALLOW_PROTOCOL=file`, so a regression fails closed.
- Inside a checkout neither measure changes a byte of a rendered help block. `--check` was `[ok]` before, and `[ok]` after regenerating.

**Docs**
- `README.md`: "`python3 setup-project.py --help` and, from a checkout, `setup.sh --help` list every flag, this one included. From a download, `setup.sh --help` clones nothing and lists none: it names `setup-project.py --help` … and `docs/cli.md`". That is +3 lines against main; the README cap in `tests/test_repo_hygiene.py` goes 1419 -> 1422, with its dated paragraph. `openRepoShape --help` is not the subject any more, because it names eight of `setup-project.py`'s fourteen flags besides `--help`. § What setup.sh does no longer gives the shim a line count ("about seventy lines of code"; it was 93 before this change and is 122 now), here, in the handbook and in `setup-project.py`'s docstring.
- `docs/handbook.html`: the page's copy of that sentence and of the shim's description, the footer's history clause, and the new README blob (`5465d135…`).
- `openRepoShape --help`: its one sentence "`./setup.sh --help` prints setup-project.py's usage, which lists them all" was false from a download. It now names `python3 setup-project.py --help`, and `./setup.sh --help` "in a checkout of this standard".
- `docs/cli.md` is regenerated: a note under `setup.sh` saying what `--help` does outside a checkout, and that `openRepoShape` sentence.
- `tests/test_setup_sh.py`: `test_no_test_here_depends_on_gh_being_authenticated`'s docstring is made true.

**Decisions** (the coordinator's; Brett may reverse).

1. **The shim's answer outside a checkout is option (a), with no flag list.** It is a short text of its own, exit 0, with no clone, no network and no temporary directory. Inside a checkout `--help` is still `setup-project.py`'s.
   - `-h` is treated like `--help`, because `setup-project.py` reads both.
   - Only arguments before `--` count. `setup.sh <Project> -- --help` is still a run, whose scaffold is asked for ITS help.
   - **Known false positives** (written into the shim's comment): runs `setup-project.py` would refuse anyway, such as `--org --help` or `--wat --help`, get the text and exit 0 instead of exit 2. Nothing is created either way.
   - **One known false negative**: `--visibility -- --help`, where `--` is a value, still clones as before. Fixing it would need the value-flag table in bash, which is the second parser #50 removed.
2. **Whose help.** The shim's text is its own, so it names no clone URL or ref. No path that prints a usage is left unattributed. `setup.sh <P> -- --help` prints the clone's scaffold usage under the `(0) self-bootstrap` clone line and `checkout: <tmp>`, which already say whose it is.
3. **The generator gets both measures**: the checkout refusal and `GIT_ALLOW_PROTOCOL=file`.
4. **Docs**: README, the handbook's reading of it, the shim's header, the test docstring and `openRepoShape --help`'s sentence, with `docs/cli.md` regenerated.
5. **The #50 guard is changed on purpose.** `tests/test_setup_project_py.py::test_setup_sh_parses_no_flags_of_its_own` refused any `--help` in the shim's code. It now excuses the spelling in exactly two places: the lines of `shim_help()`, each of which must be one `say "..."` with no `$(`, `${` or backtick in it, and section 0's one `-h|--help)` label. Outside those two, neither `--help` nor a bare `-h` may occur in the code. It still refuses `-y)`, `-h)`, `--yes)` and a second `--help)` label, and since fix round 1 it also refuses a quoted `"--help")`, a `[[ $1 == --help ]]` reader anywhere else (the developer path, the scan), and a reader hidden in a `say` line.

**Evidence.** The runs and the merge-tree below were taken at e2df9a8, the branch's head before the last merge of main (the last paragraph).

- **Targeted runs on e2df9a8** (d15a888, the fix round, merged with `origin/main` 53bee5b; the branch's first commit 968db85 is the writer's 16d1639 re-authored with the same tree 0d3243f), with a stub `gh` first on PATH, `GH_TOKEN` bogus, an empty `GH_CONFIG_DIR`, `GIT_ALLOW_PROTOCOL=file` and a blanked git identity:
  - `tests/test_setup_sh.py` and `tests/test_setup_project_py.py`: **132 passed**, 0 skipped;
  - `tests/test_cli_reference.py`, `tests/test_repo_hygiene.py`, `tests/test_agents_md_codes.py` and `tests/test_openreposhape_command.py`: **164 passed**, 0 skipped;
  - all six files together: **296 passed**, 0 skipped, 0 failed. Before the merge, on a throwaway merge of d15a888 and main 5678f68, the same six files gave 296 passed as well. review341 ran them on 968db85 merged with main 3ef059c: 132 + 162 passed.
- **The new tests against base 87d2eb7's code**: 15 failed. These are every new behaviour test (the shim's help ×4 spellings, its text, no echo, `bash -s`), the five generator refusal tests, `GIT_ALLOW_PROTOCOL` and the shim-probe tie. The 9 that pass are the "unchanged" guards (`-h`/`--help` inside a checkout, `-- --help` passthrough ×2) and pre-existing tests.
- **Mutation proof**: 18 single-hunk mutations in a throwaway worktree, each caught by a named test. They cover:
  - the help exit;
  - `-h`;
  - the `--` stop;
  - help after the env guard;
  - the default not named;
  - a flag line added;
  - the default not shared;
  - a second `--help)` label;
  - `main`'s guard, `render`'s guard, and both;
  - the top-level-is-the-tree check;
  - the files check;
  - `GIT_ALLOW_PROTOCOL`;
  - the files drifting from the shim;
  - the Tool note;
  - the `openRepoShape` sentence;
  - the README sentence.
- **The new tests**:
  - `tests/test_setup_sh.py` (`WINDOWS_SKIP` with the rest of the file):
    - `--help`, `-h`, after a run's arguments, and beside `--keep-shape-checkout --shape-ref`: exit 0, the shim's text, no `(0) self-bootstrap`, no usage, no `git clone` in `GIT_TRACE`, nothing in `TMPDIR`;
    - the text names the variable, its default, `setup-project.py --help` and `docs/cli.md`, and no flag but `--help`;
    - a forged `$OPENREPOSHAPE_REPO` is never echoed;
    - `bash -s -- --help`;
    - `<Project> -- --help` and `-- -h` still clone (locally) and reach the scaffold's usage. This is also the positive control that the trace sees a clone;
    - inside a checkout, `-h` and `--help` are `setup-project.py --help` byte for byte, on stdout and on stderr (stderr was not compared before the fix round; by probe it was already identical).
  - `tests/test_cli_reference.py`:
    - a `git ls-files` copy with no `.git`, one inside an unrelated work tree, and one inside a work tree that looks like a checkout of the standard: `--check` and a plain write both exit 2, naming the tree, with nothing written and no clone;
    - `render()` itself refuses;
    - the checkout refusal comes before the no-bash one;
    - `child_env()` pins `GIT_ALLOW_PROTOCOL=file`;
    - the generator's files are the ones the shim's probe reads;
    - (fix round 1) `--check` in a real `git init` copy whose path ends in a space, and in one whose path is not UTF-8, exits 0 with `[ok]`; each is skipped where the file system cannot hold the name (both on Windows; the second wherever creating the directory fails, which is expected on macOS, where no runner was available here).

**Merge-tree, as of e2df9a8 (re-run after merging `origin/main` 53bee5b into the branch).**

- At e2df9a8, `origin/main` (53bee5b) is an ancestor of the head: the branch carries a merge commit, so it was mergeable with main by construction. `render-cli-reference.py --check` was `[ok]` on it.
- **#352 landed as 53bee5b (#289) and conflicted in `docs/handbook.html`'s footer, as predicted.** The merge keeps both "added (#n)" clauses in landing order after "(#288)": (#289), then (#341), and keeps this branch's `README.md blob 5465d13`, which is `git hash-object README.md` again: main does not touch README.md and its own blob is `e8da9381…`.
- #314 has landed (3ef059c). The other PRs then open that share a file with this branch, re-checked against d15a888 before the merge (heads fetched at that time) and #403 again against e2df9a8:
  - clean against this branch: #394 (`tests/test_repo_hygiene.py`, not the README cap; it conflicts with main alone in `tests/test_manifest_validator.py`, a file this branch does not touch), #395 (one-line `visibility` assertion edits in `tests/test_setup_sh.py` and `tests/test_setup_project_py.py`, independent of this branch; its hunks in `docs/cli.md` are at :246-279), #397, #400 and #401 (each regenerates other sections of `docs/cli.md`);
  - **#403 (head 5b1da66) conflicted in the same footer hunk**: it merged clean with main alone and conflicted with this branch. It has since landed as dba5fb0 (#383, #403), and the merge of main into this branch (the last paragraph) keeps every clause in landing order after "(#289)", that is (#383) and then (#341), and names the blob of the README that lands, `5465d135ca7263010ee354424041f90c4553d2e7`. Main does not change README.md, so that is the blob this branch already named.
- `docs/cli.md`: #395, #397, #400 and #401 all merged clean with d15a888 plus main 5678f68, and the generator's `--check` was `[ok]` on each merged tree (#395's head 1ef3310, #397's abb27e6, #400's 7062a47, #401's 409f10b). Should one ever conflict, regenerating resolves it.

**Left out** (candidate follow-ups, not filed here).

- `setup.sh --preflight` from a download still clones before checking the machine, though `setup-project.py` answers `--preflight` before its own self-bootstrap. The preflight is Python, so the shim cannot answer it itself; it needs its own decision.
- From a download, `setup.sh --help` still needs Python 3.9 and git first: the interpreter and git checks run before the checkout probe, as they always did.
- Windows does not run the shim. The generator refusal tests do run there: they reach the refusal before any bash. The `tests-windows` job was their first run on that platform.

**Fix round 1 (review341's CHANGES NEEDED 0/1/5).**

One commit (d15a888) on top of 968db85, then a merge of `origin/main` 53bee5b (e2df9a8), no force:
- **S1 (should).** The #50 guard excused the spelling `--help` everywhere in the shim, so a second reader in any form but an unquoted label passed it. It now excuses the spelling in `shim_help()`'s `say "..."` lines and the one `-h|--help)` label, and nowhere else (decision 5). Mutation proof in a throwaway worktree: the guard passes on the unmutated shim and fails on each of review341's three variants (a quoted `"--help")` label, a `[[ "${1:-}" == --help ]]` reader in the developer path, a `[[ "$arg" == --help ]]` reader inside the scan) and on four more (a reader as its own line in `shim_help()`, one chained after a `say`, one substituted into a `say`, a bare `-h` reader). The first four and the bare `-h` reader fail only the guard, and the help tests around it stay green; the chained and substituted `say` variants also break the help's own output.
- **N1.** The inside-a-checkout test compares stderr as well as stdout. With the developer-path reader above, it now fails and `test_help_exits_zero` still passes.
- **N2.** The generator reads git's top level with `os.fsdecode(...).rstrip("\r\n")`, so a checkout at a path ending in a space, or not UTF-8, is accepted again, as the shim accepts it. A test for each; on 968db85's generator both fail with exit 2 "not a checkout".
- **N3.** README and the handbook name `python3 setup-project.py --help` as the command that lists every flag; `openRepoShape --help` names eight of fourteen. The handbook's copy, its footer clause and its README blob follow. README stays 1422 lines, so its cap does not move.
- **N4.** "About seventy lines of code" is dropped from README, the handbook and `setup-project.py`'s docstring: the shim is 122 lines of code, 93 before this change, and a count in prose goes stale again.
- **N5.** The comment above section 0's scan says the third read decides whether there is a run at all.
- `docs/cli.md` is unchanged: no `--help` text moved, and `--check` is `[ok]`.

Reviewed adversarially at 968db85, the branch's first commit (CHANGES NEEDED: 0 blockers, 1 should-fix, 5 nits). The shim and the generator behaved as this description says. The should-fix was the #50 guard, which a second `--help` reader slipped past; the nits were a test that compared stdout and not stderr, two real checkouts the generator refused (a path ending in a space, a path that is not UTF-8), and three pieces of stale doc or comment text (the README and handbook sentence that said `openRepoShape --help` lists every flag, the shim's line count, and the shim's own comment). Fix round 1 (d15a888, merged with main 53bee5b at e2df9a8) resolved them, and its re-verification of e2df9a8 read APPROVE WITH NITS (0 blockers, 0 should-fixes, 1 nit: § What setup.sh does does not say that the shim answers `--help` itself outside a checkout, left as it was because README says it a few lines above). Main 75052cb was then merged in at 4c28890: the one conflict, `docs/handbook.html`'s footer, keeps both "added" clauses (#383's, then #341's) and names the merged README's blob (5465d13), and the merged `docs/cli.md` matches its generator. CI is green on Linux, macOS and Windows, with SonarCloud passing, at 4c28890 (main 75052cb merged in).

Closes #341

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 13, 2026
… is `PRODUCT=PATH` only when its key names a chain link, and a link source that cannot be entered is a WARNING with the system's reason, not a traceback (#338) (#453)

**What was wrong.**

Two faults, one in the parser and one in the lookup.

- **The parser.** `_link_sources_from_args` split every `--link-source` value at its first `=`. A link tree under a directory named `a=b`, passed bare, became a source keyed to a link `a` that no chain names. Nothing asked for it, so the link stayed `declared-unverified`, exit 0. The WARNING then offered `--link-source openXdox=<path>` to the person who had passed exactly that. A mistyped key failed the same silent way.
- **The lookup.** `repo_shape.link_manifest_path` asked `Path.is_dir()` and then `is_file()`. Before Python 3.14 those swallow only ENOENT, ENOTDIR, EBADF and ELOOP. So each of these ended the naming gate in a traceback, exit 1:
  - a keyed source under a parent this user cannot enter;
  - `SHAPE_PIN_SOURCE_<LINK>` pointing there;
  - a checkout `../openXdox` at mode 000, which every `--project` run asks for;
  - a 300-character name.

**What changed.**

**How a value is read** (`LinkSources` in `scripts/validate-repository-naming.py`). This mirrors #312's `PinSources`, with a chain link where that file has a declared pin:
- The text before the first `=` is a LINK only when it casefolds to a link of a chain this run reads. Those are the entries of `--referent-chain`, and of each leg's `naming.referent_chain:` under `--project`. Both are read before any leg is classified.
- The keyed reading wins even over a directory spelled like the whole value. `./` reads that directory.
- Any other value is a bare PATH, taken whole.

**What a lookup says** (`scripts/repo_shape.py`):
- `link_manifest_lookup` asks `stat()` inside a guard, as `_digest_from_source` does in #312. It returns the system's reason when a source cannot answer, or says which of four plain cases it is (`no such file or directory`, `it holds no project.yaml`, `its project.yaml is not a file`, `it is neither a directory nor a file`), and it never raises. It replaces `link_manifest_path`, which is gone: nothing calls it. `link_declarations` gives the reason a manifest could not be read, and `declared_neutral_pins`, which it left with no caller, is removed (fix round 1).
- `resolve_link_source` and `link_pins_from_trees` take a keyword-only `tried`. It records each source asked, in order, with its reason. Nothing in `repo_shape` prints.
- `link_pins_from_trees` also takes a keyword-only `with_reasons` (fix round 2). Asked, it returns a `LinkPins`, a dict equal to the plain mapping that also remembers why a tree it reached could not be read. Only the scaffold asks.

**What the validator prints** (stderr, after each name's WARNINGs):
- `WARNING link-source-unreadable: <path> (<how>) could not answer for <link>: <reason>`. This covers a `--link-source` asked for a link that holds a pin, a `SHAPE_PIN_SOURCE_<LINK>` likewise, and a checkout beside the project that exists and still cannot answer. A keyed value that also names a directory gets a second line saying how to ask for that directory with `./`. A bare value holding `=` says it was read as a PATH.
- `NOTE --link-source <typed>: not used — <why>.` for a value no such link asked for. The why is one of five: a later value replaced it; no name reads that link's own declaration (a keyed value); no name records a chain at all; no name reads any chain link's own declaration (a bare value, where chains exist and none is read); or every link that is read had a more specific source.
- Three kinds of source are never named: a variable or a checkout asked only for a chain's LAST link (no one ever reads that link's own declaration); a variable or a checkout for a chain no classification reads (fix round 1: a spec leg's chain, a name with no referent, a chain whose shape fails; a `--link-source` value for such a chain is the NOTE); and a checkout that is absent or holds no manifest.
- `--quiet` drops both, with the WARNINGs. So `make bootstrap` and the doctor's `naming` row print nothing new.

`--help`, the module docstring and `docs/cli.md` (regenerated) say all of this.

**Fix round 1 (lane 5's review of 40762ca: 0 blockers, 2 should, 4 nits).**

Second commit, 4b21263. Each review finding, in order:

- **A link manifest whose `neutral_product_pins:` is not a list of names is a tree that cannot answer** (SHOULD-1). `link_declarations` iterated `data.get(key) or []` as it stood, against the `link_pins_from_trees` docstring this change wrote ("nothing a source does raises out of here"). `5` ended the naming gate, the scaffold and the project's `validate-manifest.py` in `TypeError: 'int' object is not iterable`. A scalar `openDox` was read as the characters `D`, `e`, `n`, `o`, `p`, `x`, so the chain broke, and under `--project` that is a FINDING. A mapping was read as its keys, so `openDox: yes` DECLARED `openDox`. Each is now `(None, why)`, as `not a mapping` is: declared-unverified, with the reason named ("`neutral_product_pins` is not a list", "holds an entry that is not text"). An absent or null value, an empty list, and a blank entry (`null`, `''`) in a list, declare nothing, as ever. The fix is in the shared function, so the scaffold and `validate-manifest.py` are reached too.
  **Three behaviours change on purpose here, and the tests of fix round 2 pin each:**
  - **A falsy value is no longer an empty list.** `neutral_product_pins: 0`, `false` or `''` was read as no pin at all (`or []`), so a chain through the tree was BROKEN, and under `--project` that is a FINDING. It is now a tree that cannot answer: the link is declared-unverified, a WARNING, exit 0.
  - **An entry that is not text is no longer read as text.** `[openInk, 0]` and `[openInk, false]` dropped the entry as a blank, and `[openInk, true]` read it as the text `True`, so a chain through the list VERIFIED. Each is now declared-unverified, with a WARNING saying the list "holds an entry that is not text". That is stricter than the link tree's own `validate-manifest.py`, which takes `str(pin)` of an entry (`_declared_pins`, `templates/assembly-root/scripts/validate-manifest.py:85`): a chain is better left unverified than verified through a name nobody wrote.
  - **The scaffold no longer refuses `chain-not-declared` over a pins value that is a scalar.** With `--pin-source` a tree whose `neutral_product_pins:` is the scalar `openInk`, the scaffold used to refuse (`REFUSED chain-not-declared: ... openXdox declares I, e, k, n, o, p, not openInk`, exit 2, from the characters). It is now the declared-unverified WARNING and the scaffold goes on, exit 0, which is its documented rule for a tree that cannot answer: "a warning, never a refusal, because the check is offline". The warning says the tree was reached and could not be read (fix round 2, below).
- **The reach wording is corrected** (SHOULD-2): "Who else this reaches" below, and the docstring of the test that made the claim. What stops crashing is the naming and manifest steps. `make validate`, `validate.yml` and `make bootstrap` still end in a traceback on the sibling row before Python 3.14, at a file this change does not touch (below). On 3.14 the sibling is passed over without a word there instead, which is #430.
- **The mutation gaps** (nit). Six of the review's ten mutations survived. Five now die, each by a new test: a `project.yaml` that is a directory; a device or a FIFO as the source; the order A, B, A; a value holding `=` with no chain at all; and every failure of `SHAPE_PIN_SOURCE_<LINK>` for a link that holds a pin (a path that is not there, a directory with no manifest, a file that is no manifest), not only a permission. The sixth, the `typed in self.used` test in `unread_lines`, could not be killed because it was dead: a value is read as one link's keyed source or is the one bare PATH, so every lookup asks it of the same path and it is used or unread, never both. That was asserted over 293 tests and the test is removed.
- **A chain no classification reads** (nit). `classify` consults a link's own declaration only for a name that claims a referent, with a recorded chain whose shape holds. For any other chain (one recorded on a spec leg; one on a name with no `open<Product>` to reach; one that does not begin at a held pin or does not end at a referent of the name) a source was still asked, so a path that was not there was reported as `link-source-unreadable` and a good tree was counted used, each for a question nobody put. `NamingPolicy.reads_link_declarations` now says when `resolve_referent` reads a chain, and `LinkSources.link_pins` asks it first: such a chain asks nothing, and a value for it is the NOTE that no name reads that link's declaration. A test runs `resolve_referent` with a mapping that records whether it was read, over every way a chain is and is not, and holds the new method to it.
- **Dead code and stale sentences** (nit). `declared_neutral_pins` is removed: `git grep` over the tree, every pushed branch and every sibling worktree finds no caller. `link_manifest_lookup`'s docstring no longer says "replaces" over a name that is gone. This description no longer says `tests/test_naming_policy.py` takes `SEARCH_DENIED` from #312: it defines a mark of its own, with the same condition and its own reason text.

**Fix round 2 (lane 5's re-review of 4b21263: 0 blockers, 0 should, 3 nits).**

Third commit, 2ac911c. Each finding, in order:

- **The three behaviours the body did not state are stated and tested** (nit 1). Round 1's not-a-list fix changed three behaviours on purpose, each listed under SHOULD-1 above, and the re-review found two of them had no test. R1 (`if declared is None` read as `if not declared`, which makes `0`, `false` and `''` declare nothing again) and R2 (`if pin is None or pin == ""` read as `if not pin`, which drops a `0` or `false` entry as a blank) both passed all 210 cases of `tests/test_naming_policy.py` at 4b21263. Six rows join `NOT_A_LIST_OF_NAMES`, which runs at the CLI and in process: `0`, `false` and `''` as the value, and `0`, `false` and `true` among the names. R1 now fails 9 cases and R2 fails 6.
- **The scaffold's warning is true for a tree it reached** (nit 1, the third behaviour). Over a `--pin-source` whose `neutral_product_pins:` is not a list, round 1 turned the `chain-not-declared` refusal into a declared-unverified WARNING that says the tree "is not reachable here (a sibling checkout, SHAPE_PIN_SOURCE_<LINK>, or --link-source <LINK>=<path> would let it be read)". `--pin-source` did reach the tree, and only its declaration could not be read, so that blamed a path that was right. Now `link_pins_from_trees` takes a keyword-only `with_reasons`; asked, it returns a `LinkPins`, a dict equal to the plain mapping that also keeps `unreadable`, `{link: why}` for each link whose source answered and whose manifest could not be read. `_chain_link_findings` words such a link "which was reached but could not be read as a declaration (<why>)", in a branch of its own, and only the scaffold asks. The new text is asserted for each of the eleven not-a-list rows, with `scaffold-project.py --dry-run` run as a subprocess.
  - **Every other path's text is the bytes it was.** The sentence for a link no source reached is the same code in the same lines, and four tests hold it: the generic warning for a plain mapping, for an empty `LinkPins` and for an absent tree; the scaffold's own old warning for a clone that holds no `project.yaml`; the naming validator's two lines (its warning, and the `link-source-unreadable` line that names the reason); and `validate-manifest.py`'s. The naming validator and `validate-manifest.py` do not ask for reasons, so they print what they printed.
  - **One more scaffold path changes with it, on purpose.** A tree the scaffold reached and could not read as a declaration for any other reason (a manifest that does not parse, or is not a mapping) used to say "not reachable here" too, and was no truer. It now says what it is, with the system's reason, and a test for each of the two (a manifest that is not a mapping, one that does not parse) holds it, at the scaffold's command line and in process: narrowing the wording to the not-a-list reasons fails both.
  - `_declaration` still returns its four values, so nothing else in `scaffold-project.py` moved.
- **A qualified chain link in the spy test** (nit 2). `_chain_read_cases` had no case whose chain link is qualified, so `reads_link_declarations` could take the chain as typed and not through `_chain_links` (R3) and pass every test. With `--referent-chain testorg/openXdox,openDox` that turns a verified chain into a declared-unverified one. `("codexDox", {"openXdox"}, ("testorg/openXdox", "openDox"), True)` is a case now, and R3 fails it.
- **The description** (nit 3): the ten corrections of the re-review are applied (the reason a lookup returns, the fifth NOTE reason, the third kind of silent source, the SHOULD-1 bullet, "before Python 3.14", the `validate-pins.py` bullet, the 3.14.7 rows, the merge-tree section, the filed numbers, and the project's own `validate-manifest.py`).

The mutation proofs: all three survivors were reproduced first at 4b21263 (210 passed each) and each dies at 2ac911c. Six mutations of the new code each die too: the warning ignoring the reasons (12 cases fail), the lookup recording none (12), the scaffold not asking (11), `with_reasons` on by default (3), an absent tree called reached (3), and the reached sentence used for a tree nothing reached (17).

**Where it departs from #312, and why.**

- **An explicit source that cannot answer stays a WARNING, exit 0.** #312 made an unreadable `--pin-source` a finding. Three reasons for the difference:
  - Exit 1 here means "a name is wrong", and this module has always said an unread link tree is a warning.
  - `SHAPE_PIN_SOURCE_<PRODUCT>` also answers `validate-pins.py`'s digest question, from a checkout that need not hold a manifest.
  - The lookup falls through to the next source, which may still verify the link.
- **The fall-through is kept**, as it always was. With `--pin-source`, a keyed value is final.
- **The lines go to stderr, spelled `WARNING`/`NOTE` as this file spells them.** #312's notes are `  note` lines on stdout, where that file's report lives.

**Who else this reaches.**

The guard is in `repo_shape.py`, so the same `link_pins_from_trees` lookup stops crashing for these callers too:
- **The project's own `validate-manifest.py`.** It reaches the lookup through `SHAPE_PIN_SOURCE_<LINK>` or a sibling checkout, and now gives its usual declared-unverified WARNING instead of the traceback.
- **`scaffold-project.py --referent-chain`.** A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent no longer kills the plan, and `--pin-source` verifies the link.

Neither of these prints the reason a source could not be entered; the scaffold says it only for a tree it reached and could not read (fix round 2, above). A `--pin-source` the scaffold cannot enter is still a traceback: that is #339 (`pin_digest_from_source`), and this PR does not touch it.

**What this does not reach, and what the doctor says.**
- **The project's gates still end in a traceback on the sibling row.** The NAMING and MANIFEST steps of `make validate`, `validate.yml` and `make bootstrap` stop crashing for a checkout `../openXdox` at mode 000. Those targets also run `validate-pins.py`, whose `resolve_neutral_pin_source` asks `(sibling / ".git").exists()` with no guard (`templates/assembly-root/scripts/validate-pins.py:463` on main), so before 3.14 a sibling at mode 000 still ends them in a `PermissionError`. That file is #312's, and `fix/pin-source-env-collisions` (PR #409, since landed), `fix/adopt-pin-writes-pin-file` and open PRs #343 and #419 touched it when this was written, so this PR does not edit it. The same line is listed for Python 3.14 by #430 (lane 2), where the helper that would cure both halves is asked for; the pre-3.14 traceback reproduction is added there as a comment, not filed apart. A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent is different: `validate-pins.py` makes that a `neutral-pin-source-unreadable` FINDING (#312), and that stays.
- **The doctor's `naming` row** runs the STANDARD's copy of the validator, so it stops crashing at once. **Its `manifest` row** runs the project's OWN `validate-manifest.py` on the project's own `repo_shape.py`, so it changes only once the project has applied this change. Over an existing project the doctor says `INVALID (manifest, pins)` until `update-shape.py apply` lands the two `upstream-changed` rows below.
- **The verdict.** For `SHAPE_PIN_SOURCE_<LINK>` under a locked parent, on a project that carries these copies, `INVALID (naming, manifest, pins)` becomes `INVALID (pins)`, and `pins` is #312's finding for that locked variable, which is right. For a sibling at mode 000 the `pins` that remains is the traceback above, not a finding.

The two copied scripts this change edits (`scripts/repo_shape.py` and `scripts/validate-repository-naming.py`) are copied into every project and pinned there. A project takes the change through `update-shape.py` (two `upstream-changed` rows), and a family holder takes the `repo_shape.py` row. Neither file has a second copy in this repository.

**Evidence.**

- **Issue rows, main ab623dd** (`repro338.sh`, a `gh` stub first on PATH). Under 3.10.21 and 3.12.3, rows 4-7 (keyed under a mode-000 parent, `SHAPE_PIN_SOURCE_OPENXDOX` there, a sibling at mode 000, a 300-character name) are a traceback, exit 1. Rows 2-3 (`a=b`, the typo) are a silent declared-unverified, exit 0. Under 3.14.7 row 1 verifies, and rows 2-7 are the silent declared-unverified WARNING, exit 0.
- **Issue rows, this PR.** Under 3.10.21, 3.11.16, 3.12.3, 3.12.14, 3.13.15, 3.14.7 and 3.15.0rc2 (first commit), all 7 rows exit 0 with no traceback, and the output is byte-identical across versions. `a=b` verifies. The other five rows each name their source with its reason. Re-run on the fix-round head, 4b21263, under 3.10.21, 3.12.3 and 3.14.7: byte-identical to the first commit's run. Re-run on the fix-round-2 head, 2ac911c, under the same three: all 7 rows exit 0 with no traceback, the output is byte-identical across the three versions, and it is identical to the re-review's runs of the previous head over main.
- **Reach on main.** Main gives a `PermissionError` traceback, exit 1, for `scaffold --dry-run`, the project's `validate-manifest.py`, `validate-repository-naming.py --project` and its `--quiet` form, each with `SHAPE_PIN_SOURCE_OPENXDOX` under a locked parent. This PR gives exit 0 for all four. `validate-pins.py` on the sibling row is the exception said above.
- **New tests.** 17 cases (11 functions) in the first commit; 16 fail on ab623dd. The fix round adds 31 cases in `tests/test_naming_policy.py`, and 17 of them fail on 40762ca: the five shapes of pins that are not a list (each at the CLI, in-process and, for the scalar, under `--project`), and the chains nobody reads. The other 14 guard what a mutation could break (below) or the side the fix must not move (an absent, null or blank pin list still answers).
- **Targeted run on 4b21263**, files `test_naming_policy`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_cli_reference`, `test_repo_hygiene`, `test_neutral_pin_validator`, `test_scaffold_e2e`, `test_shape_doctor`, `test_git_output_decoding` and `test_pin_validator`: **765 passed, 3 skipped**. The 3 skips are the PowerShell-only rows. (40762ca: 734 passed, 3 skipped.) `render-cli-reference.py --check` is ok.
- **Mutation probes, first commit.** Reverting the stat guard, the parse, keyed-over-directory, the NOTE lines, the chain's-end restriction, the sibling-absence filter, the manifest reason, the `--quiet` drop, the "read as a PATH" sentence, the `./` line, the project-chain read, the WARNING lines or the env reporting each fails at least one new test.
- **Targeted runs on 2ac911c (fix round 2).** Under 3.12.3, `test_naming_policy` (238 cases: 210 and 28 new), `test_cli_reference`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_repo_hygiene` and `test_setup_project_py`: **468 passed**; and `test_scaffold_e2e`, `test_scaffold_pin_and_reuse`, `test_scaffold_leg_path_collisions`, `test_scaffold_pin_collisions`, `test_neutral_pin_validator`, `test_git_output_decoding` and `test_pin_validator`: **312 passed**. Under 3.10.21 and under 3.14.7, `test_naming_policy`, `test_manifest_validator` and `test_scaffold_referent_chain_links`: **321 passed** each. `render-cli-reference.py --check` is ok, and `ast.parse(feature_version=(3, 9))` passes on the three changed files.
- **Mutation probes, fix round.** The review's ten, re-run on 4b21263: MW1, MW2, MW3, MW5 and MW11 now die (they survived), and MW4, MW7, MW12 and MW13 still die; MW6 is the dead test, removed. (Fix round 2 kills the three the re-review found, R1 to R3, and six more: see above.) Nine more for the new code (the list check, the entry check, the null and blank tolerances, `reads_link_declarations` always true, without the shape check, without the held pins, and the validator ignoring it): eight die. The ninth, `reads_link_declarations` ignoring whether the name has a referent, survives as an equivalent mutant: with no referent `_chain_shape_problem`'s last question always reports a problem, so the explicit early return is redundant, and is kept because `resolve_referent` has the same first line. The spy test holds the two together.
- **Windows.** The mode-000 rows skip on Windows and as root, using a `SEARCH_DENIED` mark this file defines, with the same condition as #312's. The new FIFO and `/dev/null` rows skip where those do not exist. `a=b` and the 300-character name run everywhere. On Windows the long-name row asserts the line, not POSIX's reason. None of it had been run on Windows or macOS when this was written; the last paragraph gives the CI result.

**Merge-tree.**

Recomputed at 19:04Z with `git merge-tree --write-tree` of 2ac911c against `origin/main` as it was then, 53bee5b, and against every one of the other 37 pushed branches that touches a file this PR changes (a scan of each head against its merge-base with main):
- **Clean:** origin/main 53bee5b (it carries #314 and #390); #394 `fix/adopt-record-submodule-placement` 563ea16 (`scripts/repo_shape.py` and `validate-manifest.py`) and #406 `fix/validate-family-unnamed-mounts` 369af2e (`scripts/repo_shape.py`); #397 abb27e6, #400 7062a47, #401 409f10b, #416 7cca975, #452 e2df9a8 and #395 1ef3310 (`docs/cli.md`); #355 6fdbe3d, #373 741cbf2 and #395 (`scaffold-project.py`).
- **Conflict, four hunks each, all in `scripts/repo_shape.py`:** `fix/pin-source-env-collisions` (PR #409) 6494714, and `fix/adopt-pin-writes-pin-file` 421e4e5 (no PR). Both also touch `scaffold-project.py`, and that merges clean with this PR's one edit there. Fix round 2's new hunks merge with neither: #409's change to the "not reachable here" sentence (the `offered` text, when a variable is shared) and the reached branch sit in different lines of `_chain_link_findings`. The four hunks, each taking both sides:
  1. the `resolve_link_source` signature: `default_source=None, env_shared: bool = False, *, tried: list | None = None`;
  2. its docstring: both paragraphs;
  3. the `link_pins_from_trees` docstring and the lines after it: both paragraphs, then `found`, this PR's `unreadable` and #409's `links` and `shared`;
  4. the call: `resolve_link_source(name, root, keyed_sources, default_source, pin_product_key(name) in shared, tried=asked)` (the same spelling on both branches), followed by `if tried is not None: tried[name] = asked`.

  Resolved that way in a throwaway and committed there, the merge with 6494714 passed **496 tests** (`test_naming_policy`, `test_pin_source_env_collisions`, `test_neutral_pin_validator`, `test_scaffold_referent_chain_links`, `test_manifest_validator`, `test_cli_reference`), and the merge with 421e4e5 passed **559 tests** (the same files and `test_pin_source_unreadable`). `render-cli-reference.py --check` is ok on both, pyflakes is clean, and the grammar parses as 3.9's. Neither branch carried the resolution when this was measured. #409 has since landed, and the merge of main into this branch (the last paragraph) takes those four hunks both ways. The earlier counts (468 and 434 from the first fix round, 430 from the re-review) are those runs' own, at their heads.

**Left out.**

Each of these was found by lane 5's reviews of this change, and each is filed or folded:
- **A bare `--link-source` answers for every link with no more specific source** (#441). In a chain of three or more links, one bare tree is read as two links' declarations, and a chain that holds is reported broken, a FINDING for a name with no direct pin on its referent. Pre-existing, the same on main; `validate-pins.py`'s `--pin-source` has the same documented rule and was not examined.
- **An empty path after a key (`--link-source openXdox=`) is `Path("")`, the current directory** (#442). Run from a project root, the project's own `project.yaml` is read as the link's declaration. Pre-existing; `validate-pins.py`'s `--pin-source openXdox=` has the same shape (there a finding naming `.`).
- **`validate-pins.py`'s `(sibling / ".git").exists()` is unguarded** (the sibling row above), folded into lane 2's #430, which already covers the Python 3.14 half: the pre-3.14 traceback and its reproduction are added there as a comment. #312's file.
- **The project's OWN manifest** with `neutral_product_pins: 5` ends in a `TypeError` in two gates before either can say anything about it (#443): `_pins_from_project` in the naming gate (`scripts/validate-repository-naming.py:302` on main), and `_declared_pins` in the project's own `validate-manifest.py` (`templates/assembly-root/scripts/validate-manifest.py:85`), which ends it before its own `manifest-neutral-product-pins` finding (`_neutral_product_pins_type_finding`, `:524` on main) can print. Reproduced on 3ef059c. That is the project's own declaration, read by the project's own gates, and not a link source, so it is outside this change; "read by the project's own gates" covers two crashing gates.

Also not done here:
- **Python 3.9 (the floor) was not run** because it is not installed here. The grammar of the three changed files was checked with `ast.parse(feature_version=(3, 9))`. Windows and macOS were not run here; CI ran them (the last paragraph).

Reviewed adversarially at 40762ca, the branch's first commit (CHANGES NEEDED: 0 blockers, 2 should-fixes, 4 nits). The head stopped every crash in the issue's table, and the should-fixes were a docstring claim the code did not keep for a `neutral_product_pins:` that is not a list, and reach claims that overstated what the project's gate does on the sibling row. Fix round 1 (4b21263) resolved every finding, and the re-review of 4b21263 read APPROVE WITH NITS (0 blockers, 0 should-fixes, 3 nits: three behaviour changes this description had not stated, two of them untested; a spy-test case for a qualified chain link; and description corrections). Fix round 2 (2ac911c) resolved those three, and its re-verification of 2ac911c read APPROVE WITH NITS (0 blockers, 0 should-fixes, 2 nits: no test of the scaffold's wider wording for a manifest that is not a mapping or does not parse, now pinned by two tests that each fail when the wording is narrowed, and a cross-reference in this description that pointed the wrong way, now corrected). Main (9736424, #409's squash) was merged into the branch as ced8fac. `scripts/repo_shape.py` conflicted in four hunks; each takes both sides, as the review of 2ac911c resolved it: `resolve_link_source(..., default_source=None, env_shared: bool = False, *, tried=None)`, both docstrings kept, `link_pins_from_trees` opening with `found`, `unreadable`, `links` and `shared`, and the call passing `pin_product_key(name) in shared` with `tried=asked`. Nothing else conflicted. After the merge, `tests/test_pin_source_env_collisions.py` and the targeted set passed (667 and 312), and `render-cli-reference.py --check` printed `[ok]`. CI is green on Linux, macOS and Windows, with SonarCloud passing, at 938a131 (main 48585e6 merged in). Its merge with main 994be39 passed the targeted set (the naming, manifest, scaffold, pin and setup test files): 983 passed.

Closes #338

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant