Skip to content

adopt-project.py: a --source or recorded source this user cannot reach is source-unreadable, never read as absent or sent to gh, and an unreadable leg or link place is named: no 3.12 traceback, no 3.14 plan-source-gone (part of #430) - #970

Merged
brettheap merged 4 commits into
mainfrom
fix/430-h-adopt-unsearchable
Oct 19, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Lane: openreposhape-1 (openRepoShape-1)
Claim: #430 (comment)

Slice H of #430's plan (#430 issuecomment-6251069236; lane 1's slices A, H and Z), a v1.2.0 draft on the v1.1.1 release commit 50178305. It changes:

  • adopt-project.py;
  • the three --source help paragraphs in docs/cli.md, which is generated from --help (render-cli-reference.py --check passes).

It also adds tests/test_adopt_unsearchable_source.py.

What

  • A path this user cannot reach is never read as one that is not there. adopt-project.py had no such helper, so this adds _is_a(path, kind) with _absent, _is_there and _os_reason, which keep bump-leg.py's _ABSENT_ERRNOS meaning exactly.
  • --source (_source_spec, _repository_in_url): a refused stat is the new code source-unreadable, exit 2. An org/repo spelling, or the github.com/ one, beside an unreadable ./org is never sent to gh.
  • The plan's recorded source (_recorded_source): a path whose stat is refused (one under a directory this user cannot search) is source-unreadable. It is never plan-source-gone, and never cloned again. A source or clone directory that is itself unreadable is not changed here (adopt-project.py: a --source or recorded clone directory this user cannot enter is cloned again or ends in a PermissionError traceback, and a --source too long to name is offered a URL that cannot clone #1003).
  • A leg under --local-remote-dir whose stat is refused waits, as a leg already on disk does. The leg question then refuses the folder by name, adopt-folder-cannot-be-made. On 3.12 this was a traceback.
  • plan-root-symlink: a place that cannot be read now says "cannot be read (Permission denied), so whether anything is there cannot be told until this user can search every directory above it (chmod u+rx)", not "and nothing is there".
  • The remedies:
    • chmod u+rx on the directory and on every directory above it, or fix who owns them.
    • The org/repo spellings add the https://github.com/<org>/<repo> way.
    • The recorded source adds passing --source again, which is held to the plan's commit:.
    • A cause other than permission gets "put right what it names" in place of chmod.

Before and after

Measured on 3.12.3, and with a shim that answers False to any OSError, as 3.14 does:

case 3.12 before shim before after, both
plan --source <000 dir>/Thing traceback, exit 1 source-unresolvable ... neither a path that exists source-unreadable, exit 2
testorg/Thing beside a 000 ./testorg traceback cloning testorg/Thing, gh asked source-unreadable, gh never asked
check, recorded path under a 000 dir traceback plan-source-gone ... names nothing that exists source-unreadable
the same, recorded as cloned: true traceback "no longer exists ... cloned again", gh asked source-unreadable, nothing cloned
execute --yes --local-remote-dir <000> traceback adopt-folder-cannot-be-made adopt-folder-cannot-be-made
AGENTS.md linked to a file under a 000 dir "and nothing is there" the same "cannot be read (Permission denied) ..."

Tests

All runs used /usr/bin/python3 -m pytest, with a stub gh first on PATH, a bogus GH_TOKEN, an empty GH_CONFIG_DIR, GIT_CONFIG_GLOBAL=/dev/null and GIT_SSH_COMMAND=false.

  • The new file: 27 passed (25, and two from the review's fix round at c5c828c6). With 50178305's adopt-project.py put back, all 27 fail.
  • Every adopt test, in node-id chunks of 30: 3,020 passed and 3 skipped.
    • Two skips are eol-through-links rows that need a root that keeps a link.
    • One is the MedxEHR clone test, which has no clone here.
  • tests/test_shape_doctor.py, in 8 shards: 445 passed. The 3 skips are the PowerShell rows, which run on Windows only.
  • The docs and codes files and test_scaffold_leg_path_collisions.py: 193 and 104 passed.

Notes

Part of #430

🤖 Generated with Claude Code

Summary by Sourcery

Preserve inaccessible filesystem paths as unreadable throughout adoption planning and execution, with clear diagnostics and no unsafe fallback behavior.

New Features:

  • Add explicit source-unreadable handling for inaccessible command-line and recorded source paths, including actionable remediation and safe GitHub URL alternatives.

Bug Fixes:

  • Prevent inaccessible paths from being treated as absent, retried as recorded clones, or sent to GitHub for cloning.
  • Report inaccessible local remote directories and symlink destinations without Python-version-dependent tracebacks or misleading missing-path messages.

Enhancements:

  • Standardize filesystem presence checks so only errors meaning genuine absence are treated as absent.

Documentation:

  • Update CLI reference documentation for inaccessible source behavior and clone prevention.

Tests:

  • Add comprehensive coverage for inaccessible sources, recorded clones, local remotes, symlinks, operating-system errors, and prevention of GitHub calls.

… recorded source this user cannot reach as source-unreadable, and names it at the split-branch and symlink sites; not finished: tests/test_shape_doctor.py and about 2,300 adopt tests not yet run

What changed (adopt-project.py, sites at 5017830):
- A local `_is_a(path, kind)` with bump-leg.py's absent-errno set
  (ENOENT, ENOTDIR, EBADF, ELOOP; Windows 21, 123, 1921) and semantics,
  `_absent(exc)`, `_is_there(path)` and `_os_reason(exc)`.
- :1826 `_source_spec` and :1765 `_repository_in_url`: a refused stat is the
  new code `source-unreadable`, never "neither a path that exists nor an
  `org/repo`", and an `org/repo` spelling is never sent to `gh`; the url is
  offered, as `source-ambiguous` offers it.
- :5735 `_recorded_source`: an unreadable `source.local_path` is
  `source-unreadable`, never `plan-source-gone` and never cloned again.
- :8538 and :11216: a refused stat of a leg under --local-remote-dir defers,
  as a leg on disk does, to the leg question, which names the folder
  (`adopt-folder-cannot-be-made`).
- :9017 `_PlannedRoot._outside`: a refused stat of a symlink's destination
  says "<place>: cannot be read (<reason>)", never "nothing is there"; the
  place is still asked as text (a trailing `/` kept).
- The `--source` help of plan, check and execute names `source-unreadable`;
  docs/cli.md regenerated (`render-cli-reference.py --check` ok).

Measured, Python 3.12.3, and 3.12 with a sitecustomize making Path.exists,
is_file, is_dir and is_symlink answer False to any OSError (as 3.14):
- `plan --source <mode-000 dir>/Thing`: before, a PermissionError traceback
  (exit 1) / `source-unresolvable: ... neither a path that exists`; after,
  on both, `REFUSED source-unreadable: --source '<path>': cannot be read
  (Permission denied), so whether a repository is there cannot be told`.
- `plan --source testorg/Thing` beside a mode-000 ./testorg (and
  github.com/testorg/Thing beside ./github.com): before, traceback /
  `cloning testorg/Thing`, gh asked; after, on both, source-unreadable,
  gh never asked.
- `check` of a plan whose local_path is under a mode-000 directory: before,
  traceback / `plan-source-gone ... names nothing that exists`, or, recorded
  as a clone, "no longer exists ... cloned again" and gh asked; after, on
  both, source-unreadable, nothing cloned.
- `execute --yes --local-remote-dir <mode 000>`: before, traceback at
  :11216 / `adopt-folder-cannot-be-made`; after, on both, the latter.
- `check`, AGENTS.md -> <mode-000 dir>/AGENTS.md: before, on both,
  "and nothing is there"; after, "<place>: cannot be read (Permission
  denied), so whether anything is there cannot be told until this user can
  search every directory above it (`chmod u+rx`)".

Tests: tests/test_adopt_unsearchable_source.py, 25 tests; all 25 fail with
adopt-project.py at 5017830 and pass with this change. Passed so far: 10
test_adopt*.py files (1,039 tests, the new file and
test_adopt_execute_refusals.py among them), test_cli_reference.py,
test_agents_md_codes.py, test_repo_hygiene.py,
test_adopt_plan_leg_path_help.py (193), test_scaffold_leg_path_collisions.py
(104), and 540 node ids of the source, recorded-source, split-branch and
symlink files. Not yet run: tests/test_shape_doctor.py and the rest of the
adopt files (stopped for an account swap).

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettheap brettheap added this to the v1.2.0 milestone Oct 18, 2026
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR makes adopt-project.py preserve the distinction between absent and unreadable paths across Python versions, reporting named source-unreadable refusals and preventing unsafe GitHub fallback or recloning, while improving local-remote and symlink diagnostics and documenting the behavior.

Sequence diagram for unreadable --source handling

sequenceDiagram
    participant CLI
    participant SourceSpec as _source_spec
    participant SourceThere as _source_there
    participant Filesystem
    participant GH

    CLI->>SourceSpec: resolve --source
    SourceSpec->>SourceThere: check candidate path
    SourceThere->>Filesystem: _is_there
    Filesystem-->>SourceThere: PermissionError
    SourceThere-->>SourceSpec: _source_unreadable
    SourceSpec-->>CLI: source-unreadable, exit 2
    Note over SourceSpec,GH: GitHub fallback is not attempted
Loading

Sequence diagram for unreadable recorded source handling

sequenceDiagram
    participant CheckOrExecute
    participant Plan
    participant RecordedSource as _recorded_source
    participant Filesystem
    participant GH

    CheckOrExecute->>Plan: read recorded source
    Plan->>RecordedSource: resolve source.local_path
    RecordedSource->>Filesystem: _is_there
    Filesystem-->>RecordedSource: PermissionError
    RecordedSource-->>Plan: _source_unreadable
    Plan-->>CheckOrExecute: refuse with named path
    Note over Plan,GH: No plan-source-gone and no reclone
Loading

Flow diagram for absent versus unreadable paths

flowchart TD
    A[Check path with _is_a] --> B{stat result}
    B -->|ENOENT or equivalent| C[Treat as absent]
    B -->|Other OSError| D[source-unreadable]
    B -->|Successful stat| E[Use path kind]
    D --> F[Name path and OS reason]
    D --> G[Do not invoke gh or reclone]
    F --> H[Apply permission or cause-specific remedy]
Loading

File-Level Changes

Change Details Files
Introduces consistent filesystem probing that distinguishes missing paths from paths whose metadata cannot be read.
  • Adds _is_a, _is_there, _absent, and _os_reason around stat with stable absent-errno semantics across Python versions.
  • Adds source-unreadable refusals with OS-specific remediation and explicit path naming.
  • Uses the helper for explicit sources, recorded plan sources, and local remote leg detection.
adopt-project.py
Prevents unreadable paths from triggering incorrect source fallback, cloning, or misleading plan-state errors.
  • Refuses unreadable explicit paths before interpreting org/repo or GitHub-shaped spellings and avoids invoking gh.
  • Refuses unreadable recorded sources instead of reporting plan-source-gone or recloning cloned sources.
  • Defers unreadable local-remote directories to the existing folder validation so the failing location is named.
adopt-project.py
tests/test_adopt_unsearchable_source.py
Improves symlink diagnostics when the destination cannot be inspected.
  • Distinguishes an unreadable symlink destination from a confirmed missing destination.
  • Includes the destination and permission guidance in plan-root-symlink findings.
adopt-project.py
tests/test_adopt_unsearchable_source.py
Updates CLI documentation and adds broad regression coverage for cross-version behavior.
  • Updates generated help text and CLI reference for source-unreadable behavior.
  • Tests missing-versus-unreadable errno handling, permission-denied paths, GitHub fallback prevention, recorded sources, local remotes, and symlink findings.
  • Covers both real filesystem permissions and simulated Python 3.14-style exists() behavior.
adopt-project.py
docs/cli.md
tests/test_adopt_unsearchable_source.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-1, 2026-10-19T15:51:35Z) Independent review of #970 (#430 slice H) by r970, at 3ec73e21: READY, no MUST. Lane 1 takes SHOULD-2 and NITs 1 to 3 in a test-only fix round, files SHOULD-1 as a follow-up, and fixes the body (SHOULD-1, NIT-5). It lands only on a re-check of the new head.

Verdict: READY at 3ec73e21. I found no MUST. Every slice-H site is handled, every new sentence I measured is true, the 25 new tests fail without the fix, and 3ec73e21 merges cleanly with main 2a78ae1a and passes there. There are two SHOULDs: one is a PR-body wording fix plus a follow-up issue, the other a missing test. Neither needs a new head unless you choose to add the test now, which would need a re-review. The PR is still a draft.

Findings, most severe first

SHOULD-1: a source or clone directory that is itself unreadable is still cloned again, or ends in a traceback, and the PR body over-claims.

SHOULD-2: the OTHER_REASON branch is untested.

  • Where: adopt-project.py:1832. Mutant M7 (advice = SOURCE_UNREADABLE_PERMISSION for every cause) survives all 25 tests.
  • Fix: add this test to tests/test_adopt_unsearchable_source.py. I ran it from scratch: it passes at the head and fails under M7.
def test_a_stat_refused_for_another_reason_is_not_told_to_chmod(
        adopter, monkeypatch, tmp_path):
    """A `stat` refused for another reason than permission (an I/O error)
    is `source-unreadable` too, told to put right what it names: no `chmod`
    puts that right."""
    def refusing(self, *args, **kwargs):
        raise OSError(errno.EIO, os.strerror(errno.EIO), str(self))

    monkeypatch.setattr(Path, "stat", refusing)
    with pytest.raises(adopter.Refusal) as raised:
        adopter._source_spec(str(tmp_path / PROJECT))

    refusal = raised.value
    assert refusal.code == "source-unreadable"
    assert f"cannot be read ({os.strerror(errno.EIO)})" in refusal.detail
    assert ("which no `chmod` puts right: put right what it names"
            in refusal.remediation)
    assert "chmod u+rx" not in refusal.remediation

NIT-1 (mutant M9 survives). The recorded source's second way out is never asserted. Add after tests/test_adopt_unsearchable_source.py:384:

    assert ("or pass the source again with --source, which is held to the "
            "plan's `commit:`.") in proc.stderr

NIT-2 (mutant M8 survives). The symlink finding's (chmod u+rx) clause is never tested against a cause other than permission. Add a variant of :520 that raises OSError(errno.EIO, …) for the target and asserts "… cannot be read (Input/output error), so whether anything is there cannot be told. Send AGENTS.md" in err. The full text is in suggested/test_r970_suggested.py; it passes at the head and fails under M8.

NIT-3. The split-branch control at tests/test_adopt_unsearchable_source.py:460 asserts any Refusal. It does raise adopt-branch-exists and plan-split-branch-taken (I checked with probe/control_codes.py). Parametrize the expected code next to ask and add assert control.value.code == code.

NIT-4 (ENAMETOOLONG on an org/repo spelling). G1 is plan --source <300 a's>/Thing.

  • Before: 3.12 gave OSError: [Errno 36] File name too long (a traceback); the shim gave cloning … with gh asked.
  • After: source-unreadable, "cannot be read (File name too long)", gh never asked.
  • Two words in it are not quite true. It is not unknown whether the path or the GitHub repository is meant: neither can exist. And the offered https://github.com/<300-byte owner>/Thing can never clone, since owners are at most 39 characters (GITHUB_OWNER_MAX).
  • Optional fix: in _source_there, offer the URL only when exc.errno != errno.ENAMETOOLONG.

NIT-5 (body housekeeping).

Answers to items 1 to 8

1. Sites. All handled:

2. Behaviour (head on 3.12 and under the shim give identical output apart from commit hashes; non-root user; no call reached the stub gh unless noted):

3. The code source-unreadable. The bare code is new in this PR: git grep -P '(?<![-\w])source-unreadable' 50178305 finds nothing. The hits in the brief are other codes that contain it as a substring:

  • pin-source-unreadable
  • neutral-pin-source-unreadable
  • link-source-unreadable
  • adopt's own plan-pin-source-unreadable

So it has one meaning. Its siblings (source-unresolvable, source-ambiguous, source-not-top-level) are listed only in the tool, docs/cli.md, per-release notes and tests, so no list misses it. test_agents_md_codes.py passes (47 tests).

4. Wording. Every sentence is true by probe:

  • The URL remedy works: https://github.com/testorg/Thing beside the unreadable directory reaches gh repo clone.
  • "held to the plan's commit:" is true: check --source <a clone one commit on> gives FINDING plan-stale, and execute gives REFUSED plan-stale.
  • The absent-errno sets are identical to bump-leg.py:409-410.
  • _os_reason matches validate-pins.py's _cannot_read.

5. Tests. 25 pass at the head with no skips. All 25 fail with 50178305's file:

  • the 9 helper rows on AttributeError … _is_there;
  • the 16 behaviour rows on 3.12 tracebacks at the base sites, or "DID NOT RAISE Refusal".

Eight rows need a real mode and skip as root or on Windows. They ran here and run on Linux CI.

7. PR title and body. Every before/after cell matches my own base and shim probe. The counts hold:

  • 25 of 25 fail without the fix.
  • The writer's 101 chunk files sum to 3,020 passed, 3 skipped, 0 failed.
  • test_shape_doctor.py: 445 passed plus 3 PowerShell skips.
  • 193 = 45 + 47 + 23 + 78, and 104 for test_scaffold_leg_path_collisions.py.
  • render-cli-reference.py --check passes.

The exceptions are SHOULD-1 and NIT-5.

8. Style. git diff --check 50178305 3ec73e21 is clean. No added line is over 79 columns; adopt-project.py's maximum is 84 and unchanged. Every added line is ASCII without tabs, and there is no compound assert.

Merges (item 6)

3ec73e21 merged with Result
main 2a78ae1a (now) clean, tree 3b666d1faf44090675a7ac0df986377a6030c518
main c7a3ff16 (holds #964 b0614d51, #963 2771b380, #965 c7a3ff16) clean, tree 22df8d72e2c0387b95297e2934cbbe344af8ca86
c4dd19dc / a19d69cb clean (4ecb4b9c / ed63b82b)
#963 9e6ed9bb / #964 369f2781 / #965 e349d83e clean (78642797 / 27cc6846 / 10a3bc36)
stacked c4dd19d + #963 + #964 + #965 clean (546bf8df); landing order with #966 also clean
  • There are no conflicts, so no resolution is needed.
  • Both real merges hold adopt-project.py, docs/cli.md and the new test file byte for byte as predicted (5f49c67b, 5fbb50b1, 43328cf2).
  • The only callers of _outside use its new signature, and none of the landed PRs adds an existence guard.
  • 2a78ae1a adds only bootstrap.py and its two test files beyond c7a3ff16.

Mutants (each applied by exact substitution, run against the new file, restored with git checkout -- adopt-project.py)

Mutant Result
M1 EACCES added to the absent errnos killed (16)
M2 _source_spec guard back to exists() killed (5)
M3 _repository_in_url guard back to exists() / M3b guard dropped killed (2) / killed (2)
M4 recorded source read as gone, so cloned again killed (4)
M5 --local-remote-dir wait reverted killed (2)
M6 plan-root-symlink wording reverted killed (2)
M7 OTHER_REASON branch dropped survived (SHOULD-2; the suggested test kills it)
M8 chmod clause given to every cause survived (NIT-2; the suggested test kills it)
M9 recorded source's --source way out dropped survived (NIT-1)
M10 URL way out dropped killed (5)
M11 ELOOP no longer absent killed (1)
M12 cloned/not-cloned clause swapped killed (4)
M13 GitHub name ignored killed (5)
M14 only the spec leg asked survived; near-equivalent, since both legs share one directory

Test counts

All with the brief's stub gh and environment, one basetemp, every chunk under 9 minutes.

Tree Files Result
head 3ec73e21 the brief's 15 files 1,213 passed
head test_scaffold_leg_path_collisions.py 104 passed
head test_shape_doctor.py, 8 shards 445 passed, 3 skipped (PowerShell, Windows only, :10489)
head 8 more source and code files 415 passed
head total 2,177 passed, 3 skipped, 0 failed
merge with c7a3ff16 17 adopt and docs files, including #963/#964/#965's new tests 1,203 passed
merge with c7a3ff16 scaffold / doctor 104 / 445 + 3 skipped
merge with 2a78ae1a new file, CLI reference, scaffold 152 passed

CI at the head is green: SonarCloud, tests, tests-git-bash and package.

What I could not verify

  • A real Python 3.14. None is on this host; I used a sitecustomize shim and proved it bites.
  • Windows, macOS, and running as root. The in-process rows cover root.
  • The writer's full 3,020-test adopt run. I checked their outputs and ran my own 1,628 adopt tests on the head.
  • A non-permission cause other than ENAMETOOLONG on a real disk. EIO was tested in-process only.
  • CI on the merged tree.

The worktree is clean at 3ec73e21, and no locked directory remains. The predicted squashes and the merge checks made git objects only (commit-tree, no ref moved).

Everything is in /workspace/projects/openRepoShape-worktrees/_lane1-scratch/r970/:

  • probe/probe.py (and probe2.py): the probe; outputs in probe/head-312.out, probe/head-shim.out, probe/base-312.out, probe/base-shim.out, probe/head-312-d5.out, probe/base-shim-d5.out, probe/stackC-312.out
  • mutants.py: the mutants
  • suggested/test_r970_suggested.py: the suggested tests
  • results/: every test run
  • merge-c7a3/ and merge-2a78/: the merged trees I tested

Lane: openreposhape-1 (openRepoShape-1)

…ver told to chmod; the recorded source's --source way out and each control's code are asserted

The independent review of #970 (issuecomment-6271331621), SHOULD-2 and
NITs 1 to 3, all in tests/test_adopt_unsearchable_source.py. SHOULD-2: a
stat refused with EIO is source-unreadable and told to put right what it
names, never to chmod; the new row kills M7 (every cause told to chmod).
NIT-1: the recorded source's `--source` way out, held to the plan's
`commit:`, is asserted in all three rows of its test; they kill M9 (that
clause dropped). NIT-2: the symlink finding's `(chmod u+rx)` clause is
asserted absent for an EIO on the place; the new row kills M8 (the clause
given to every cause). NIT-3: each split-branch control asserts its own
code, adopt-branch-exists for the left-behind branch and
plan-split-branch-taken for the one in the way. No other file changes.

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-1, 2026-10-19T16:04:49Z) Re-check of #970 by r970, at c5c828c6: READY. The fix round is test-only: SHOULD-2 and NITs 1 to 3 of issuecomment-6271331621. SHOULD-1 and NIT-4 are filed as #1003, and the body is fixed for SHOULD-1 and NIT-5. #970 lands at c5c828c6 once its own checks are green, by lane 3's turn rule.

Verdict: READY at c5c828c6.

The change

  • c5c828c6 is a fast-forward from 3ec73e21 (git merge-base --is-ancestor succeeds). It changes only tests/test_adopt_unsearchable_source.py (+55/−5); adopt-project.py is untouched.
  • The delta is exactly the four fixes I asked for, at the lines you named:
    • SHOULD-2: lines 290-307.
    • NIT-1: lines 405-406.
    • NIT-3: lines 455-460 and 483-486. The new code check comes after the with block, so it actually runs.
    • NIT-2: lines 570-593, with @needs_a_symlink.
  • Style is clean: git diff --check 3ec73e21 c5c828c6 passes, no added line is over 79 columns, and the added lines are plain ASCII with no tabs.

Counts

Run Result
The file at c5c828c6 27 passed, 0 skipped
M7 (OTHER_REASON dropped) killed: 1 failed (the SHOULD-2 test)
M8 (chmod u+rx clause given to every cause) killed: 1 failed (the NIT-2 test)
M9 (the recorded source's --source way out dropped) killed: 3 failed (all three recorded-source rows)
50178305's adopt-project.py put back 27 of 27 fail
c5c828c6 merged with dd2cf2e8, the file plus test_cli_reference.py 50 passed

The number for the PR body is 27 of 27 fail without the fix. The two new rows fail on base for the reason they test:

  • The SHOULD-2 test gets OSError: [Errno 5] Input/output error out of _source_spec instead of a source-unreadable refusal.
  • The NIT-2 test's EIO text is missing, because base says "and nothing is there".

Merge with main

git merge-tree --write-tree c5c828c6 dd2cf2e8 is clean: tree 9241a338f703bdc2ae66200941189c04e46d7340, rc=0, so no resolution is needed.

dd2cf2e8 (#966) changes tests/conftest.py and tests/gh_guard.py, which the new file imports from. That is why I ran the file on the merged tree too.

Still open, none of it blocking and none of it in the head

  • SHOULD-1: narrow the PR body's "an unreadable path … never cloned again" sentence (and the squash message) as worded in my previous report, and file the follow-up issue for a source or clone directory that is itself unreadable.
  • NIT-4: the GitHub URL offered when a name is too long.
  • NIT-5: the body's stale merge line and the head commit's "not finished" text must stay out of the squash message.
  • The PR is still a draft.

The worktree is clean and detached at c5c828c6, and nothing in the scratch area is left locked. The test outputs are in /workspace/projects/openRepoShape-worktrees/_lane1-scratch/r970/results/ (c5-head.out, c5-base.out, and one c5-M*.out per mutant). The merged tree I tested is /workspace/projects/openRepoShape-worktrees/_lane1-scratch/r970/merge-dd2c/.

Lane: openreposhape-1 (openRepoShape-1)

@sonarqubecloud

Copy link
Copy Markdown

@sourcery-ai

sourcery-ai Bot commented Oct 19, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 22 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 41aed9c into main Oct 19, 2026
5 checks passed
@brettheap
brettheap deleted the fix/430-h-adopt-unsearchable branch October 19, 2026 20:53
brettheap added a commit that referenced this pull request Oct 21, 2026
…is_dir/exists guard in a shipped tool fails by file and line unless its `_is_a` or a reasoned allowlist covers it, and a `project.yaml` linked into an unsearchable folder is `yaml-unreadable` (#430) (#1111)

## What

Slice Z of lane 1's plan for #430 (issuecomment-6251069236), the slice that lands last. Slices A to H moved each shipped tool's `Path.is_file()`, `is_dir()`, `exists()` and `is_symlink()` guards onto a local `_is_a` (`is_a` in the doctor). From Python 3.14 those methods answer False to any `OSError`, and `os.path.isfile`, `isdir`, `exists`, `lexists` and `islink` always have. The helper treats ENOENT, ENOTDIR, EBADF and ELOOP (and Windows' 21, 123 and 1921) as absent and raises every other `OSError`, so the caller refuses it by name: `<path>: cannot be read (<reason>)`, exit 2.

| Slice | Files | PR |
| --- | --- | --- |
| A | `scripts/bump-leg.py` | #934 |
| B | `update-shape.py` | #976 |
| C | `shape-doctor.py`, `scripts/shape_advisory.py` | #975 |
| D | the assembly root's `validate-manifest.py` and `bootstrap.py` | #958 |
| E | the family root's `validate-family.py` | #956 |
| F | the family root's `siblings.py` and `bootstrap.py` | #951 |
| G | `scripts/family.py` | #955 |
| H | `adopt-project.py` | #970 |

This PR adds `tests/test_path_guard_hygiene.py`, which stops a new bare guard from bringing the old answer back. It also changes one site the scan found in `shape-doctor.py`.

### The hygiene test

- **Which files.** It reads every shipped tool: the files `tests/test_repo_hygiene.py`'s `SHIPPED` lists, and every `*.py` at the top level, in `scripts/`, in `templates/**/scripts/` and in `src/openreposhape/`. That is 25 files today; the glob adds none that `SHIPPED` lacks.
- **What it finds.** It parses each file with `ast` and finds every call of the four methods and the five `os.path` tests. It catches:
  - a call wrapped over lines;
  - the module reached as `os.path`, `posixpath`, `ntpath` or `genericpath`, or through `import os as o` (`o.path`), `import os.path as p`, `from os import path` or `p = os.path`;
  - a test imported by name (`from os.path import lexists`);
  - a guard passed as a function (`filter(os.path.exists, ...)`, `map(Path.is_dir, ...)`).
- **What it does not catch.** It misses a guard named by a string (`getattr(p, "is_file")`, `operator.methodcaller`), the other questions of the same kind (`os.path.ismount`, `is_fifo`, `os.access` with `F_OK`), and a hand-written `stat` whose `OSError` is False. It also misses an alias bound by tuple unpacking, a walrus, a `for` or `with` target, a default parameter, a class attribute, `importlib` or `sys.modules`, and `o = os; o.path.isfile` or `from os.path import *`. A guard bound once at module level (`EX = os.path.exists`) is flagged at the binding, but its later calls are not. Shipped code has none of these except three Windows-only `getattr(os.path, "isjunction", ...)` lookups: `shape-doctor.py` `unclonable`, and `templates/family-root/scripts/siblings.py` `not_a_clone` and `family_link`. Each sits beside an allowlisted `is_symlink` or `islink`, so each is benign in context. The test's `_os_path_names` docstring still says "in no shipped tool today"; it changes with a follow-up.
- **What fails.** A guard fails unless it is on `BENIGN` or `KNOWN_OPEN`, as often as its function makes it. Those lists key each entry by file, enclosing function and the call's own text, never by line, with a count. The failure names `file:line` and the function, and says what to do: use the file's `_is_a`, or copy validate-pins.py's where the file has none (#370, #610). Where the site is benign, it says to add an entry with a tag and a reason.
- **No guard in a helper.** A guard inside a stat helper fails too: the helper must ask `stat` and nothing else. No entry may name one.
- **Entries must stay true.** An entry whose call has gone, has moved function, or names more calls than its function makes fails as stale. So does an entry with no tag from `TAGS`, or with no reason.
- **The scanner and the counts are checked on their own.**
  - The scanner finds 19 written spellings, and finds no `stat` or `lstat`.
  - An entry for one call does not cover two, and an entry for three, or for a call that has gone, is stale.

### The parity test

- **Where the sets come from.** Each top-level `_is_a` and `is_a` in a shipped tool has its sets read from the AST. That covers the twelve the slices wrote, and any added later. The test follows the helper through every module function it calls (`_absent`, `_mode_of`, `_says_absent`, `refused_read`). It reads each `<exc>.errno in <set>` and each `getattr(<exc>, "winerror", None) in <set>`, through the module constant, as `errno.<NAME>` names and numbers. A set it cannot read that way fails; it is not guessed at.
- **What each helper must hold.** Every set must be {ENOENT, ENOTDIR, EBADF, ELOOP} and {21, 123, 1921}. On the path from its `stat` to its `raise`:
  - it may catch only `OSError` and `ValueError`. A handler for `PermissionError`, `Exception`, or a bare `except:` fails;
  - every `except OSError` must hold a bare `raise`;
  - `errno` and `winerror` may be read only as `in <set>`. An `== errno.EACCES` beside the set fails.
- **Helpers must be plain functions.** A class or an `async def` by a helper's name fails.
- **Each slice's helper must still be there.** That way none is held to nothing because it was renamed.
- **The constants.** Every module constant named `*ABSENT_ERRNOS` or `*ABSENT_WINERRORS` must be the one set. There are 24, two per helper.
- **Results on this head.** All twelve helpers hold the set: `adopt-project.py`, `update-shape.py`, `shape-doctor.py` (`is_a`), `scripts/bump-leg.py`, `scripts/family.py`, `scripts/shape_advisory.py`, and the assembly root's `validate-pins.py`, `validate-manifest.py` and `bootstrap.py`, and the family root's `validate-family.py`, `bootstrap.py` and `siblings.py`.

## What the scan finds

On main `4ed7c7bf` the scan finds 191 bare guards in 18 of the 25 files: the 188 of `16f9a0e8`, `6da757cb` and `823624d9`, and three that #1091 added to `adopt-project.py`. None is inside a helper.

- **1 is the real site below, changed here.** On this head the scan finds 190.
- **179 are on `BENIGN`, in 175 entries.** Each carries the plan's own reason for leaving a site out (lane 1's site table at `5017830`) and a line of its own:

| Tag | Entries (sites) | What it means |
| --- | --- | --- |
| OWN | 57 (58) | the tool's own checkout of the standard, its templates or package, its cache, a temporary or work directory it made, or what this run wrote |
| DELIBERATE | 63 (65) | a deliberate `os.path` call or guard whose False the code relies on and says why: wording, advice, `family.py`'s members-spelling twins, or a check a by-name refusal follows (#608, #609, #798, #735, #784, #793, #764) |
| ASKED | 25 (25) | asked only after the stat helper, a `stat`, `mount_read` or git asked the path or its folder, or inside a `try` that keeps the `OSError` |
| ROOT | 20 (21) | a name directly in the root: asked by `lstat`, or where a link at the name is left out or handled as a link first, in a root the tool refuses by name first where it cannot search it |
| DEST | 9 (9) | the plan's OPEN-DEST, loud: a destination asked before a create or a clone, which then fails by name |
| PINNED | 1 (1) | `scripts/repo_shape.py`'s `find_repo_root`, which the plan left (pinned, low) |

- **#1091's three are `OWN`.** None needs #1110.
  - `_PlannedRoot._no_directory` and `_PlannedRoot._into_git` ask `self._git().is_dir(...)`. `_git()` returns `_CloneGit`, `check`'s model in memory of the names git makes in the clone's own `.git` (#904). Its `is_dir` is a set lookup, not the disk. Measured in a folder with no `.git`, with every `stat`, `lstat`, listing and `os.path` test patched to raise: it answers True, False and None as its sets say, and asks the disk nothing.
  - `_clone_ref_store` asks `(probe / CLONE_GIT_DIR / "reftable").is_dir()` of `git init`'s throwaway, a `mkdtemp` of this user's (0700) in the run's own work directory. A work directory at 0500 or 0600 stops at the `mkdtemp` before the guard is asked, and `ref_store` answers git's defaults. The one way to a refused `stat` there is a git template holding `reftable` as a link into a folder at 000. Then 3.12 raises into `ref_store`, which answers git's defaults, and the shim reads False. Both answer `(False, True)` on git 2.43.0, as the readable control does.
- **11 are on `KNOWN_OPEN`.** These are real sites this slice found and did not change. Each is left to the follow-up issue #1110, which is not in v1.2.0. In each, a name in the root that is a link into a folder this user cannot search is still taken for nothing there. Every "under the shim" result below was measured with a 3.14-like `sitecustomize`.

| Site | What happens |
| --- | --- |
| `scripts/family.py` `_gitmodules_names` | measured in process: on 3.12 a `PermissionError` raised out of `remove`; under the shim `(None, [])`, so `remove --dry-run` says no section is removed. Slice G left it for this slice to flag. A change needs a refusal threaded through `remove` and its rollback. |
| `scripts/family.py` `_sections_giving` | the same file: `[]` under the shim |
| `templates/family-root/scripts/siblings.py` `submodule_urls` | the same, in a pinned copy, which this slice does not edit |
| `shape-doctor.py` `check_agent_files` | an agent file linked that way is `absent` in the `agent files` note, which never moves the verdict |
| `shape-doctor.py` `check_what_is_here` | the same, for the NOT A SHAPE ROOT row's facts |
| `shape-doctor.py` `check_the_way_in` | a `.git` linked that way: under the shim the `way-in` row says there is no repository to adopt, and sends the person to `setup.sh` to scaffold three new ones; on 3.12 the row names `.git: cannot be read` |
| `shape-doctor.py` `check_the_other_way_in` | the same `.git`: the `way-in-scaffold` row is left out |
| `shape-doctor.py` `origin_org` | the same `.git`, measured alone: `None` under the shim, the answer for a folder that is no repository (`'acme'` in the readable control); 3.12 raises `PermissionError` |
| `shape-doctor.py` `repo_local_origin_name` | the same `.git`, measured alone: `None` under the shim (`'Orbit'` in the control), so the `naming` row classifies `Atlas` alone, not `Atlas, Orbit`; 3.12 raises |
| `scripts/shape_advisory.py` `_read` | the same `.git`: `read()` says `not-a-repository` under the shim (`advise` in the control), so the Triad advisory row is left out |
| `scripts/shape_advisory.py` `repository_names` | the same `.git`, measured alone: `['Atlas']` under the shim (`['Atlas', 'Orbit']` in the control); 3.12 raises |

## The site changed here: the doctor's own manifest

`shape-doctor.py`'s `Context.__init__` asked `path.is_file()` of the root's `project.yaml` and `family.yaml`. Slice C's root check in `main` asks only whether the root can be searched, so a `project.yaml` that is a link into a folder this user cannot search got past it.

It is now asked with `is_a`, and a refused `stat` is loaded (`Context._to_load`). `load_yaml` then refuses it as it refuses a file it cannot open, and the `manifest` row names it, as it already did for a `project.yaml` at mode 000 (#205).

Measured on a root holding `project.yaml -> ../locked/project.yaml` with `locked/` at 000, Python 3.12.3, and the 3.14-like shim:

| | Before (`16f9a0e8`) | After |
| --- | --- | --- |
| 3.12 | `PermissionError` traceback at `Context.__init__`, exit 1 | `kind assembly root (project.yaml, which does not load)`, `FINDING manifest ... REFUSED yaml-unreadable: .../project.yaml: cannot be read (Permission denied)`, `INVALID (manifest, manifest-kinds)`, exit 1 |
| 3.14-like | `NOT A SHAPE ROOT`, exit 2, with `adopt-project.py plan --source ...` as the NEXT, over a project | byte for byte the 3.12 report above |

The `manifest kinds` row names the same path with the `chmod` cure, in validate-pins.py's words. Where the `stat` succeeds, `is_a(path, stat.S_ISREG)` answers as `is_file()` did, so every other root reads as it did. An I/O error from the `stat` is loaded and named the same way.

**Out of scope here (r1111's N5).** The row this case now shows carries two older pieces of wording, from #205 and #847, which this PR leaves alone:
- the `manifest` row's WHY for `yaml-unreadable` speaks of "a merge conflict resolved, a hand edit";
- the `chmod` cure says "every directory above it inside the root", where the locked folder lies outside the root.

## Mutants

Each mutant was applied once to a scratch copy of the head (`git archive` of the merge with main, with the new test file laid over it), and the new file was run there. The file was then put back from its bytes and checked by sha256. The worktree was never changed.

- **Round 1: 5 of 5 killed.**

| Mutant | Fails |
| --- | --- |
| M1: `scripts/bump-leg.py` `_validators_readable` back to `path.is_file()` | the hygiene test for that file, naming `scripts/bump-leg.py:1474` |
| M2: `errno.EACCES` added to `validate-pins.py`'s `_ABSENT_ERRNOS` | both parity tests |
| M3: 1921 dropped from `update-shape.py`'s `_ABSENT_WINERRORS` | both parity tests |
| M4: the allowlisted `path.is_file()` in `update-shape.py`'s `_extend_the_eol_block` taken out | the stale-entry test |
| M5: `Context.__init__` back to `path.is_file()` | the doctor tests |

- **Round 2: 17 of 17 killed.**

| Mutant | Fails |
| --- | --- |
| r1111's X3: a guard through `import os as _os` | the hygiene test for that file |
| r1111's X4: `if not path.exists(): return False` inside bump-leg's `_is_a` | the hygiene test (a guard in the helper) |
| r1111's X5: `except PermissionError: return False` before `except OSError` | parity: `catches PermissionError` |
| r1111's X6: `or exc.errno == errno.EACCES` beside the set | parity: `reads exc.errno other than as in <set>` |
| r1111's X15: a class `_is_a` holding a bare guard | hygiene and parity |
| r1111's X18: `except Exception: return False` before `except OSError` | parity: `catches Exception` |
| r1111's T2: the stale check disabled | the count self-test |
| r1111's T3: one entry silences any count | the count self-test |
| round 1's W1 and W4 | hygiene, and the stale-entry test |
| Y1: the handler rule loosened | the parity self-test |
| Y2: an `errno` read other than `in` no longer refused | the parity self-test |
| Y3: `import os as o` no longer followed | the scanner self-test |
| Y4: `p = os.path` no longer followed | the scanner self-test |
| Y5: the helper exemption put back | the scanner self-test |
| Y6: a class or `async def` by a helper's name read as a function | the parity self-test |
| Y7: `except PermissionError` in `_to_load` (r1111's X12) | the doctor test's I/O-error row |

- **Round 3, #1091's sites: 3 of 3 bite.** Each new entry was taken out alone, on the merge with main `4ed7c7bf`. Each run gave 1 failed and 48 passed.

| Entry taken out | Fails |
| --- | --- |
| B1: `_PlannedRoot._no_directory` | the hygiene test for `adopt-project.py`, naming `adopt-project.py:10785` |
| B2: `_PlannedRoot._into_git` | the same test, naming `adopt-project.py:10817` |
| B3: `_clone_ref_store` | the same test, naming `adopt-project.py:11113` |

- **The file's self-tests.** Slice A's helper passes the parity test, and eleven one-line changes to it are each named: three to its sets, the re-raise, a constant's name, three handlers, `==`, an `async def` and a class.

## Tests

- **`tests/test_path_guard_hygiene.py`, new: 49 tests.**
  - 25 hygiene rows, one per tool.
  - The stale-entry, tag, scanner, count and parity tests.
  - Five tests for the doctor change:
    - two in process (`refused_stat` and a refused `read_text`, for a permission and for an I/O error), on every Python and as root;
    - two controls, nothing there and a directory there, which still read as no manifest;
    - one mode case run as a person runs the doctor, skipped on Windows and as root.
- **Against main's tools** (main `4ed7c7bf`, with the new file laid over it), 45 pass and 4 fail. The 4 are the doctor's unlisted `Context.__init__` guard and its three rows: the two in-process rows, the I/O-error row among them, and the mode case. On this head all 49 pass.
- **The gate on head `d286ae9a`.** The head holds main `4ed7c7bf` through the plain merge `8bc4d1b2`, with tree `fc0731ea`; the branch's first merge, `03c16cb9`, took `6da757cb`. Against main, the head changes only `shape-doctor.py` and the new file. The gate ran on the head itself, with Python 3.12.3, git 2.43.0, a stub `gh` first on `PATH`, `GH_TOKEN` bogus, an empty `GH_CONFIG_DIR`, `GIT_CONFIG_GLOBAL=/dev/null` and `GIT_SSH_COMMAND=false`:
  - the new file, `tests/test_repo_hygiene.py`, `tests/test_cli_reference.py`, `tests/test_agents_md_codes.py`, `tests/test_doctor_unreadable_paths.py` and `tests/test_doctor_unreadable_pin.py`: 305 passed;
  - `tests/test_shape_doctor.py`, 448 tests in four shards: 445 passed and 3 skipped (118 passed; 110 passed; 117 passed; 100 passed and 3 skipped);
  - `tests/test_scaffold_leg_path_collisions.py`: 104 passed.
- No `--help` text changes, so `docs/cli.md` is unchanged. No pinned copy changes, so no project or holder gets an `upstream-changed` row.

It lands by fast refresh (Brett Heap, 2026-10-15T22:48:35Z). Its reviewed head `d286ae9a` is green on Linux and SonarCloud. The branch merged main twice and was never rebased; the last merge is update-branch's, clean, giving d286ae9, which is green on SonarCloud and Git Bash. Lane 1's gate passed on its tree: the 8 test files (test_path_guard_hygiene, test_shape_doctor, test_scaffold_leg_path_collisions, test_repo_hygiene, test_cli_reference, test_agents_md_codes, test_doctor_unreadable_paths and test_doctor_unreadable_pin), 854 passed and 3 skipped.

Closes #430

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant