Skip to content

adopt-project.py: under a one-byte locale such as Latin-1, check reads a plan written in UTF-8 whose source is there, asking the plan's UTF-8 bytes where the spelling as written names nothing, rather than refusing plan-source-gone (#1078) - #1106

Merged
brettheap merged 3 commits into
mainfrom
l3-1078-one-byte-locale
Nov 8, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What changed

A follow-up outside v1.2.0 (Brett Heap, 2026-10-21T10:32:04Z). It lands only after the v1.2.0 tag, on a later merge word.

#1078 (found in the review of #1072). adopt-project.py check reads source.local_path from a plan written in UTF-8. #1072 asks the path as the plan's UTF-8 bytes where the locale cannot spell it (C and POSIX with PYTHONUTF8=0). Under a locale of one byte to a character, Latin-1 say, os.fsencode spells the text, so the path was asked only as written: Thing followed by U+00E9 as the one byte 0xE9, where the name plan read is 0xC3 0xA9. check refused plan-source-gone, "names nothing that exists", of a source that is there. Recorded as a clone plan made, it "no longer exists", and gh repo clone was asked.

Following the Direction, a new _recorded_path asks the spelling as written first. Where that names nothing, it asks the plan's UTF-8 bytes too, before anything is refused. _recorded_source and _keep_the_clone_at_the_record both use it, so the two read a record the same way. _utf8_spelling is factored out of _recorded_spelling, which behaves as before.

Decided where the Direction leaves it open:

  • Spelling as written first. A plan written under that one-byte locale records the name as that locale read it, and it reads as it did. Where both spellings name something, the one as written wins.
  • Where neither names anything, the path is the spelling as written, and every refusal says exactly what it said before. Under C and POSIX the lookup is adopt-project.py and family.py: a source this user cannot enter is source-unreadable, a UTF-8 plan reads its source under the C locale, and a member on a branch that is not UTF-8 is put back on it (#918) #1072's, unchanged.
  • The keep-the-clone lookup asks the disk now (_is_there), as _recorded_source does. A stat the operating system refuses leaves the clone on the list to remove, as any OSError there did before.
  • The test builds its own locale. This host has no locales package: no /usr/share/i18n, so no en_US source and no charmaps. localedef -i en_US -f ISO-8859-1 fails because the character map file ISO-8859-1 is not found. The test writes the ISO-8859-1 charmap (each byte is the code point of the same number) and a source with only LC_CTYPE, then builds it with localedef -c in its temporary directory. It needs only localedef and a C library that reads LOCPATH. It skips, quoting localedef's errors, only where a child started in that locale does not report iso8859-1 as both its file name encoding and its preferred encoding.

Out of scope

Measured: the locale, then main and this head

The locale, built with the scratch charmap and source:

$ LOCPATH=<scratch>/locales LC_ALL=en_US.ISO-8859-1 PYTHONUTF8=0 python3 -c 'import sys, locale; print(sys.getfilesystemencoding(), locale.getpreferredencoding(False), sys.flags.utf8_mode)'
iso8859-1 ISO-8859-1 0

The plan was written in UTF-8 mode (LC_ALL=C PYTHONUTF8=1) for a real repository at <scratch>/repro-*/Thing followed by U+00E9. It records local_path with that character as the UTF-8 bytes C3 A9. Then check --plan, with a stub gh that exits 97:

record locale main 2a119f28 this head
given Latin-1 exit 2: REFUSED plan-source-gone: ... names nothing that exists, and the plan does not record it as a clone ... exit 0, plan ok
given C, PYTHONUTF8=0 exit 0, plan ok exit 0, plan ok
given POSIX, PYTHONUTF8=0 exit 0, plan ok exit 0, plan ok
given UTF-8 mode exit 0, plan ok exit 0, plan ok
cloned Latin-1 exit 2: NOTE ... no longer exists, so its source.repository, testorg/Thing, is cloned again from GitHub, then REFUSED source-unclonable (gh asked) exit 0, plan ok
cloned C / POSIX / UTF-8 mode exit 0, plan ok exit 0, plan ok

Tests

Each test file ran on its own, one at a time, under a stub gh that exits 97, with GH_TOKEN=bogus, an empty GH_CONFIG_DIR, GIT_SSH_COMMAND=false, GIT_CONFIG_GLOBAL=/dev/null and EMAIL=tests@openreposhape.invalid. The command was /usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt-<name> -o tmp_path_retention_policy=failed tests/<file> -rfEs, on Python 3.12.3:

  • tests/test_adopt_one_byte_locale.py (new, -rs): 10 passed in 12.65s, none skipped
  • tests/test_adopt_unenterable_source_and_locale.py: 13 passed in 13.96s
  • tests/test_adopt_unsearchable_source.py: 27 passed in 12.45s
  • tests/test_adopt_template_values.py: 78 passed in 89.48s
  • tests/test_adopt_source_clone_record.py: 100 passed in 47.47s
  • tests/test_adopt_temp_work_roots.py: 9 passed in 12.49s
  • tests/test_adopt_clone_claimed.py: 42 passed in 22.69s
  • tests/test_adopt_work_dir_refusal.py: 53 passed in 42.06s
  • tests/test_adopt_plan_escapes.py: 59 passed in 14.54s
  • tests/test_adopt_plan_yaml_quoting.py: 540 passed in 275.81s
  • tests/test_adopt_c_locale_bytes.py: 17 passed in 43.96s
  • tests/test_scaffold_leg_path_collisions.py: 104 passed in 6.14s
  • tests/test_agents_md_codes.py: 47 passed in 2.34s
  • tests/test_cli_reference.py: 23 passed in 34.90s
  • tests/test_update_add_codes_documented.py: 6 passed in 0.49s
  • tests/test_repo_hygiene.py: 100 passed in 2.90s
  • tests/test_no_auto_maintenance.py: 58 passed, 2 skipped in 13.30s. Both skips say "the session began with GIT_CONFIG_GLOBAL set": the runner sets it to /dev/null.
  • tests/test_shape_doctor.py, once: 445 passed, 3 skipped in 793.00s (the three skips: "the PowerShell line is printed, and powershell.exe exists, only on a Windows host"). A first run was cut off at about 16% when the lane's session stopped; this is the rerun.
  • pyflakes 4.0.0: 0 warnings in adopt-project.py at main and at this head; 0 in the new test file.

The new test file:

  • runs check end to end under Latin-1, C and POSIX, for a given record and a cloned one;
  • checks that the refusal's words are unchanged where neither spelling names anything;
  • tabulates _recorded_path under Latin-1 and C, with a directory at the UTF-8 bytes, at both spellings, or at neither;
  • checks _keep_the_clone_at_the_record under Latin-1.

Mutants

Each mutant was applied to adopt-project.py, and the new test file was run. The file was then restored from a saved copy, and cmp found it identical every time.

  • M1, the UTF-8 retry removed (the brief's mutant). KILLED: 4 failed, 6 passed. The failures were test_a_plan_written_in_utf8_is_read_under_a_one_byte_locale[given] and [cloned], test_the_record_is_asked_as_written_then_as_utf8[utf8-latin1], and test_the_clone_at_the_record_is_kept_under_a_one_byte_locale.
  • M2, the UTF-8 bytes asked before the spelling as written. KILLED: 1 failed, 9 passed. The failure was test_the_record_is_asked_as_written_then_as_utf8[both-latin1].
  • M3, _recorded_source back to main's lookup. KILLED: 2 failed, 8 passed. The failures were test_a_plan_written_in_utf8_is_read_under_a_one_byte_locale[given] and [cloned].
  • M4, _keep_the_clone_at_the_record back to main's spelling. KILLED: 1 failed, 9 passed. The failure was test_the_clone_at_the_record_is_kept_under_a_one_byte_locale.

Merges

git merge-tree --write-tree of this head (aea54cc5, based on 2a119f28) was run after a fresh fetch, with origin/main at 16f9a0e8 (#1087):

Fix round 1 (RV1106)

New head: 504a1dd6, one commit on top of aea54cc5.

The MUST: SonarCloud's duplication gate. The gate failed at "4.5% Duplication on New Code (required <= 3%)". The only duplicated block was the Latin-1 keep-the-clone test's body, a copy of the C test's in tests/test_adopt_unenterable_source_and_locale.py. Both tests built the record and a clone elsewhere, ran KEEP_THE_CLONE in a child and parsed its answer.

  • That body is now one helper, kept_clones(tmp_path, env), which returns (spells, left, elsewhere). It sits next to KEEP_THE_CLONE in tests/test_adopt_unenterable_source_and_locale.py, so this PR now touches that file too.
  • Each test keeps its own premise (spells is False under C and True under Latin-1) and asserts left == [elsewhere].
  • The comment on KEEP_THE_CLONE now says which tests run it.
  • The commit changes no line of adopt-project.py.

Tests at 504a1dd6, one file at a time, under the same runner as above:

  • tests/test_adopt_one_byte_locale.py (-rs): 10 passed in 8.51s, none skipped
  • tests/test_adopt_unenterable_source_and_locale.py: 13 passed in 9.53s
  • tests/test_adopt_c_locale_bytes.py: 17 passed in 40.19s
  • pyflakes: 0 warnings on tests/test_adopt_one_byte_locale.py, before and after. tests/test_adopt_unenterable_source_and_locale.py has 7 before and after, all about its adopter fixture import, so nothing is new.

M4 made again (_keep_the_clone_at_the_record back to main's spelling): KILLED, 1 failed, 9 passed. The failure was test_the_clone_at_the_record_is_kept_under_a_one_byte_locale. The file was restored, and cmp found it identical.

Merges of 504a1dd6 after a fresh fetch, with origin/main at 6da757cb. The open PRs on this PR's three files, checked live, are #1105 and #1091:

Closes #1078

Lane: openreposhape-3 (openRepoShape-3)
🤖 Generated with Claude Code

Summary by Sourcery

Fix recorded source-path lookup so UTF-8 plans remain usable under one-byte locales without changing existing refusal behavior.

Bug Fixes:

  • Make adopt-project.py check correctly recognize UTF-8-recorded source paths under one-byte locales, avoiding false plan-source-gone refusals and unnecessary recloning.
  • Use the same locale-aware path resolution when deciding whether a newly created clone matches the recorded source.

Enhancements:

  • Centralize recorded-path resolution while preserving existing spelling and refusal behavior when neither representation exists.

Tests:

  • Add end-to-end coverage for UTF-8 plans checked under Latin-1, C, and POSIX locales, including clone records and unchanged refusal messages.
  • Add tests for path-spelling precedence, clone retention, and locally constructed Latin-1 locale support.

…ads a plan written in UTF-8 whose source is there, asking the plan's UTF-8 bytes where the spelling as written names nothing, rather than refusing `plan-source-gone` (#1078)

Under a locale of one byte to a character, Latin-1 say, with
`PYTHONUTF8=0`, `os.fsencode` spells every character of a plan's
non-ASCII `source.local_path`, so #1072's `_recorded_spelling` asked the
path only as written: the e-acute of `Thing` + U+00E9 as the one byte
0xE9, where the name `plan` read in UTF-8 is 0xC3 0xA9. `check` refused
`plan-source-gone`, "names nothing that exists", of a source that is
there; recorded as a clone `plan` made, it "no longer exists" and `gh
repo clone` was asked.

`_recorded_path` now asks the spelling as written first and, where that
names nothing, the plan's UTF-8 bytes too. `_recorded_source` and
`_keep_the_clone_at_the_record` both ask it, so a clone `check --source`
made at the record is kept under Latin-1 as under C. Under C and POSIX
the lookup is #1072's, and every refusal's words are unchanged: where
neither spelling names anything, the path is the spelling as written.

The new tests build a Latin-1 locale with `localedef` from a charmap and
a locale source they write, and skip with localedef's words where none
can be built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Add _recorded_path to check the spelling recorded in the plan first and retry with its UTF-8 spelling when needed, then use that lookup for source checks and clone retention.

  • adopt-project.py L1921-1924, L1937-1978, L6830-6833 (+4 small edits)

Tests

Add one-byte-locale coverage for the recorded-path fallback and verify that clone retention uses the same lookup, sharing the clone-retention test setup.

  • tests/test_adopt_one_byte_locale.py L1-242
  • tests/test_adopt_unenterable_source_and_locale.py L70-73, L346-351, L367-383

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-21T13:09:46Z) Lane 3's independent review of #1106 (RV1106), head aea54cc: NOT READY, for one MUST. The code change itself is sound.

MUST

  • SonarCloud fails "4.5% Duplication on New Code (required <= 3%)" on one block: tests/test_adopt_one_byte_locale.py:239-252 duplicates tests/test_adopt_unenterable_source_and_locale.py:358-371 (build the record and the elsewhere clone, run KEEP_THE_CLONE in a child, parse the answer). Fix: move that body into a helper kept_clones(tmp_path, env) returning (spells, left, elsewhere), in test_adopt_unenterable_source_and_locale.py just before test_the_clone_at_the_record_is_kept_under_the_c_locale. Each test keeps its call and asserts spells (False under C, True under Latin-1) and left == [elsewhere]. The new file imports kept_clones in place of KEEP_THE_CLONE. Tried in a scratch clone: 28 lines in, 33 out; the new file 10 passed with -rs and none skipped; the unenterable file 13 passed; pyflakes found nothing new; M4 still killed. It removes the only duplicated block, but only the next analysis can confirm the gate.

SHOULD

NIT (no change needed)

  • adopt-project.py:1872 compares text, not bytes: a record holding a surrogate escape is asked twice under Latin-1 where it names nothing. Harmless.
  • The retry can read a directory other than the one a Latin-1 plan meant where that one is gone, with no word. plan-stale (adopt-project.py:12356 and 12537) bounds it to a directory at the plan's own commit.

Checked

Lane 3's plan: the SHOULD is filed as #1112, outside v1.2.0. W1078, the writer, makes the MUST fix as RV1106 tried it (the kept_clones helper), names #1112 in the body's "Out of scope" with no closing keyword, and pushes one commit. RV1106 then checks the new head, and SonarCloud must pass on it. The CI log check for the new file's 10 tests is still owed. #1106 stays a draft with no milestone and lands after the v1.2.0 tag, on a later word.

Lane: openreposhape-3 (openRepoShape-3)

…source_and_locale.py: the clone at the record is asked under the C locale and Latin-1 through one helper, `kept_clones` (#1078)

SonarCloud counted the Latin-1 test's body as a 14-line copy of the C
test's: both built the record and a clone elsewhere, ran
`KEEP_THE_CLONE` in a child and parsed its answer. `kept_clones` does
that once, in the file that holds `KEEP_THE_CLONE`, and returns whether
the locale spells the record, the clones left and the one made
elsewhere. Each test keeps its own premise and its assert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-21T14:57:32Z) Lane 3's independent review of #1106, round 2, head 504a1dd. Verdict: READY.

Lane 3's plan: #1106 is READY at 504a1dd. It stays a draft with no milestone and lands after the v1.2.0 tag, on a later word, by the usual pilot, with an explicit squash message. #1112 (at_the_record) starts once #1106 is on main.

Lane: openreposhape-3 (openRepoShape-3)

This was referenced Oct 21, 2026
@brettheap brettheap added this to the v1.2.1 milestone Oct 23, 2026
brettheap added a commit that referenced this pull request Oct 23, 2026
…py` where GNU make at the root reads a kept `GNUmakefile` or `makefile`, and its NOTE names that file and what it takes for `make bootstrap` to run there (#1047) (#1121)

GNU make reads the first it finds of `GNUmakefile`, `makefile` and `Makefile`, and only that one. Where the source keeps `GNUmakefile` or `makefile` in the root, `make bootstrap` at the root of the clone reads the source's file, not the shape's `Makefile`, and exits 2 with "No rule to make target 'bootstrap'". `execute`'s NEXT printed it there anyway.

This PR also covers the third box of #1048 (see #1048): the `_next_steps` NOTE that said the shape's targets "arrive only once the follow-up merges the two", which is untrue where GNU make reads a kept `GNUmakefile` or `makefile`. #1105 covers that issue's other two boxes, and leaves this one to this PR.

## Measured

GNU Make 4.3 on Linux, a disk that keeps case. One directory per set of files, each file with a `which` target:

| files | `make which` reads | `make -n bootstrap` |
|---|---|---|
| `GNUmakefile` | `GNUmakefile` | exit 2, no rule |
| `makefile` | `makefile` | exit 2 |
| `Makefile` (with a `bootstrap` rule) | `Makefile` | exit 0 |
| `GNUmakefile` + `makefile` | `GNUmakefile` | exit 2 |
| `GNUmakefile` + `Makefile` | `GNUmakefile` | exit 2 |
| `makefile` + `Makefile` | `makefile` | exit 2 |
| all three | `GNUmakefile` | exit 2 |

A `GNUmakefile` that holds `include Makefile` runs the shape's rule (exit 0), and so does `make -f Makefile bootstrap`.

What NEXT printed on main, from a real `plan`, `check` and `execute --local-remote-dir`, with the printed command run in a recursive clone of the split ("S" is the source's own `scripts/bootstrap.py`, kept with `scripts/`):

| the root keeps | NEXT on main | in the clone |
|---|---|---|
| none | `make bootstrap` | exit 0 |
| `Makefile` | `python3 scripts/bootstrap.py`, #480's NOTE | exit 0 |
| `makefile`, `GNUmakefile`, or both | `make bootstrap` | **exit 2, no rule** |
| `Makefile` + `makefile`, `Makefile` + `GNUmakefile`, all three | `python3 scripts/bootstrap.py`, and a NOTE saying the targets arrive once the two `Makefile`s are merged | the command exits 0; `make bootstrap` exits 2, and still does after `cat shape/Makefile >> Makefile` |
| S + `GNUmakefile`, S + `makefile`, S + `Makefile` + `GNUmakefile` | no command; the NOTE says to merge, "and then run `make bootstrap`" | `make bootstrap` exits 2 |

## What changed

`adopt-project.py`, `_next_steps`:
- It takes `surviving=`, the paths the plan's entries send to the root (`paths_for["root"]`, passed by `cmd_execute`).
- It asks them for a kept `GNUmakefile` or `makefile` in GNU make's order, with `_root_keeps`, as `_makefile_follow_up` asks them. So NEXT names the same file as the plan's `CONTRACTS_DIR` follow-up.
- Where it finds one, the command is `python3 scripts/bootstrap.py`. The NOTE names the file in the follow-up's words, and says `make bootstrap` and the shape's other make targets run there only once that file has them, or includes a Makefile that does:
  - the shape's `Makefile` is the root's: "NOTE the shape's Makefile is the root's `Makefile`, but GNU make at the root reads the source's `GNUmakefile`, the first of `GNUmakefile`, `makefile` and `Makefile` it finds, so `make bootstrap` and the shape's other make targets run there only once `GNUmakefile` has them, or includes a Makefile that does; `python3 scripts/bootstrap.py` is what the shape's `make bootstrap` runs."
  - the source also keeps its own `Makefile`: "NOTE the root keeps its own Makefile and the shape's is shape/Makefile, and GNU make at the root reads ..." (the same clause, and the same ending).
  - the source's own script went aside too: the merges first, as before, "and then run `python3 scripts/bootstrap.py`; GNU make at the root reads ...".
- A root that keeps neither name gets the block it got before, byte for byte. That includes `MAKEFILE`, which GNU make never reads where case is kept.

## Tests

`tests/test_adopt_next_make_reads_first.py`, new, 40 tests (38, and the 2 of the fix round below):
- 13 rows word for word: none, `Makefile`, `makefile`, `GNUmakefile`, `makefile`+`GNUmakefile`, `Makefile`+`makefile`, `Makefile`+`GNUmakefile`, all three, S, S+`Makefile`, S+`GNUmakefile`, S+`makefile`, S+`Makefile`+`GNUmakefile`.
- 3 rows that keep `make bootstrap`: `MAKEFILE`; longer names (`makefile.am`, `Makefile.in`, `GNUmakefile.bak`); and the names below the root.
- 13 rows in which NEXT names the same file as `_makefile_follow_up`, for the same root paths.
- 9 end to end rows, one for each row that keeps `GNUmakefile` or `makefile`. Each is a real `execute`, and its clone is made by pasting NEXT's own clone line. In that clone:
  - NEXT's lines are the row's;
  - `make which` prints the file the NOTE names;
  - `make -n bootstrap` has no rule;
  - the printed command exits 0 and ends `bootstrap ok`. For S rows, it does so once the shape's script is merged, as the NOTE says;
  - where a `Makefile` is kept, the follow-up's merge alone still leaves no rule;
  - an `include Makefile` in the file GNU make reads makes `make -n bootstrap` run the shape's recipe.

On main's `adopt-project.py`, the first 38 all fail. The 9 end to end rows fail on what NEXT prints. The unit rows that guard the unchanged cases fail there only for want of the `surviving=` keyword. M1 below shows the same rows passing and every row that keeps `GNUmakefile` or `makefile` failing. On this head, all 40 pass.

Mutants of the change, each restored and checked by sha256. Each run is the new file plus the five `next` tests of `test_adopt_e2e.py` (43 tests). Every mutant is killed:

| mutant | failed |
|---|---|
| M1 `first` never found | 27 |
| M2 `makefile` asked before `GNUmakefile` | 6 |
| M3 the command ignores `first` | 6 |
| M4 the branch for a `Makefile` kept beside `first` dropped | 6 |
| M5 the branch for a script gone aside ignores `first` | 9 |
| M6 the lookup folds case | 7, two of them existing `test_adopt_e2e.py` rows |
| M7 `cmd_execute` passes no paths | 9, the end to end rows |

Gate on this head, named files only, one basetemp per run. Each run had `EMAIL` set, `GIT_CONFIG_GLOBAL` an empty file, `GIT_CONFIG_NOSYSTEM=1`, a stub `gh` first on PATH, `GH_TOKEN=bogus` and an empty `GH_CONFIG_DIR`:
- the new file, `test_adopt_make_reads_first`, `test_adopt_e2e`: 98 passed, with the new file at 38 tests (at `c04bb710`; the fix round's gate is below).
- `test_adopt_github_exits`, `test_adopt_printed_sentences`, `test_repo_hygiene`, `test_readme_windows_quoting_sentence`, `test_cli_reference`, `test_agents_md_codes`: 291 passed.
- `test_scaffold_leg_path_collisions`: 104 passed.
- `test_adopt_contracts_dir`, `test_adopt_verify`, `test_adopt_pins`, `test_adopt_root_manifest`, `test_adopt_ignored_shape_files`, `test_adopt_nested_attributes`, `test_adopt_shape_pin_and_work_dir`: 378 passed.
- `test_shape_doctor.py` in 4 shards: 112 + 112 + 112 + 109 = 445 passed, and 3 skipped (the PowerShell line, which is printed only on a Windows host).

## Fix round 1 (RV1121's SHOULD-1)

RV1121 found the code right and one test missing: nothing held `cmd_execute` to handing `_next_steps(surviving=)` only the root's paths. Its mutant X4, which passes `paths_for["root"] + paths_for["code"] + paths_for["drop"]`, kept all 43 tests green. The case is a real one: the `root-project-makefile` rule sends a kept `GNUmakefile` to the root, and its reason says that moving a Makefile that only builds the code to the code leg "is a one-line edit of the plan" (`contracts/path-classification.yaml`, lines 236-237).

Two end to end rows, `test_next_keeps_make_bootstrap_where_the_plan_sends_the_makefile_elsewhere[code]` and `[drop]`. Each keeps `GNUmakefile` in the source, edits the plan to send it to `leg: code` (then `leg: drop`), runs a real `execute`, and pastes NEXT's own clone line. Each asserts that NEXT's lines after the clone are `    make bootstrap` and nothing else, that the clone's root has no `GNUmakefile`, and that `make -n bootstrap` there exits 0 and runs `python3 scripts/bootstrap.py`.

The `execute`, paste and clone steps of the nine existing end to end rows moved into one helper, `executed_and_cloned`, which the two new rows share, so no block of the test is written twice. Those rows assert what they asserted. The file now holds 40 tests (38 and 2).

- **X4, on this head's `adopt-project.py` with the one line changed:** the new file and the five `next` rows of `test_adopt_e2e.py` (45 selected): 2 failed, 43 passed. Both new rows fail on NEXT's lines, which are `python3 scripts/bootstrap.py` and the NOTE "the shape's Makefile is the root's `Makefile`, but GNU make at the root reads the source's `GNUmakefile`, ...", naming a file the root does not have. The file was restored and its sha256 read `60bad59a...` again.
- **On main's `adopt-project.py`:** the two rows pass, since main printed `make bootstrap` there. They hold the unchanged case against the new lookup, and X4 is the mutant that tells them apart.
- **On this head `1c5173758`**, same environment as above: the new file 40 passed, `test_adopt_e2e.py` 27 passed, `test_repo_hygiene.py` 100 passed.
- **Merge:** `git merge-tree --write-tree` of this head with main at `823624d93`: clean (tree `73ddafb2`).

RV1121's four NITs are not in this PR; the lane files them.

## Merge

- `git merge-tree --write-tree` of this head with main at `6da757cbe`, where the branch starts: clean (tree `21955cc7`). Main has since moved to `7cd66902a`, which changes `scaffold-project.py`, its test and the docs, and none of this branch's files. The merge with it is clean too (tree `ec4e6c8e`).
- With #1105's head `5a4de7f76`: clean (tree `3394d19d`). #1105's `adopt-project.py` change is in `_predict_collisions`, apart from `_next_steps`. On that merge, built in a `git clone --shared` checkout, `test_adopt_next_make_reads_first`, #1105's `test_adopt_collision_make_reads` and `test_pinned_copy_text` gave 152 passed.
- With the other open pull requests that change `adopt-project.py` (#1120 `c55e55e3`, #1119 `bdcf9952`, #1118 `c60066cb`, #1106 `504a1dd6`, #1091 `160de118`): each clean.

## Not in this PR

- Once #1105 lands, the lookup "which of `GNUmakefile` and `makefile` the root keeps first" is written three times: in `_makefile_follow_up`, `_predict_collisions` and `_next_steps`. Each is the same two lines over `GNU_MAKE_READS` and `_root_keeps`. They are kept apart here so this PR and #1105 touch disjoint lines; one helper could serve all three after both land.
- `templates/assembly-root/AGENTS-shape.md` line 19 ("`git clone --recurse-submodules` this repository and run `make bootstrap`") has the same gap in an adopted root that keeps `GNUmakefile` or `makefile`. It is pinned, and #1105 changes that file, so it is left.
- On a disk that ignores case (#1049), a kept `makefile` is reported as a `Makefile` collision, and NEXT's NOTE says "the root keeps its own Makefile". The new clause is true there too, since GNU make reads that file and merging into it brings the targets. The spelling is #1049's.
- A kept directory named `GNUmakefile/` or `makefile/` is named as the source's file. GNU make stops on it with "Is a directory". `_makefile_follow_up` does the same.

Closes #1047

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Nov 8, 2026
…s message spells a path byte that is not UTF-8 as `_shown` does, an escape, and no longer says it names it with a `?` in its place, which #900 replaced (#1018) (#1104)

`env_commit`'s docstring in `scripts/shape_materialize.py` says what the split commit's message does with a `root` or `drop` path that holds a byte that is not UTF-8, as the code does since #900. It said the message names such a path "with a `?` in place of that byte, a path that does not exist". That was true before #900, and it is not true now. A reader of the docstring is told about a `caf?.md` the tool no longer writes.

Found by lane 1's writer on #900 and filed as #1018. Lane 5 took it outside v1.2.0, under Brett Heap's ruling of 2026-10-21T10:32:04Z.

Measured on main `2a119f281` and again on `16f9a0e81` (main after #1087), with a small script that calls: `adopt-project.py`'s own `_split_message` and `_shown`, then `env_commit` and a real `git commit`, read back with `cat -A`.

- The issue's claim held: the docstring (about line 2103 when filed, line 2268 on `2a119f281`, line 2349 on this branch) still said the message names the byte with a `?`.
- The code does not. `_shown("caf\udce9.md")` is `` `caf\udce9.md` ``, and the split message for a `root` and a `drop` entry named that holds no lone surrogate. Through `env_commit` and `git cat-file commit HEAD`, `cat -A`:

  ```
  DROPPED (in no leg and no longer here):$
    `caf\udce9.md`$
  STAYS IN THE ASSEMBLY ROOT:$
    README.md$
    `caf\udce9.md`$
  ```
- What stays true of `env_commit` itself: a message that holds the lone surrogate, handed to it directly, is still written as `?`. `cat -A` reads `Name: caf?.md$`. So the `errors="replace"` paragraph is kept, and what changes is who it describes.
- #1087 (main `16f9a0e81`) changed `scripts/shape_materialize.py` by 128 lines, none of them in `env_commit`, `COMMIT_COMMAND` or the message, and changed nothing in `adopt-project.py` (0 diff lines). The rebase onto it was clean and the docstring needed no other change.

One file, `env_commit`'s docstring, in two paragraphs: the `errors="replace"` paragraph (round 1, 11 lines added and 7 removed) and the sentence above it (round 2, below; 3 added and 2 removed). Together 14 added and 9 removed against main `6da757cb`. The `errors="replace"` paragraph now says:

- encoding, a character UTF-8 cannot carry is written as `?`, and nothing raises after both legs are pushed, as before;
- `adopt-project.py` refuses a lone surrogate in a path it extracts (`entry_path_problem`), and a `root` or `drop` entry that holds one is listed in the split message but never raw: the message names each path as `plan`'s report does (`_shown`, #900), so the byte 0xE9 reads as an escape with no surrogate in it, `` `caf\\udce9.md` ``, the path the source holds;
- written raw it would be a `?`, `caf?.md`, a path that does not exist, which is what `env_commit` still writes for a lone surrogate it is handed.

The backslash is doubled in the source, as the neighbouring docstrings write `\\udcc3`. A single `\u` in a non-raw docstring would be an escape that makes a lone surrogate. Measured: the docstring holds no character from U+D800 to U+DFFF and holds the six characters `\udce9`. The longest docstring line is 78 characters.

**The code is unchanged.** A script compares the AST of `scripts/shape_materialize.py` with every docstring blanked: the branch's file is identical to `origin/main`'s after the rebase onto `16f9a0e81` (round 1), and the merge of this head into `origin/main` `6da757cb` is identical to `6da757cb`'s file (round 2).

**No pin moves.** `scripts/shape_materialize.py` is not a pinned copy. `COPIED_VERBATIM`, `COPIED_FROM_SHAPE`, `FAMILY_COPIED_VERBATIM` and `FAMILY_COPIED_FROM_SHAPE` do not list it, and `templates/assembly-root/scripts/validate-pins.py` says of it "which no project carries". So no `upstream-changed` row, no digest, no assembly copy and no `tests/test_pinned_copy_text.py` entry changes.

**No other text pins these words.** Searched in `tests/`, `README.md`, `docs/`, `docs/handbook.html`, `AGENTS.md` and `templates/`: no test reads `env_commit`'s docstring (`__doc__`, `getdoc`), and none of them holds "in place of that byte" or "a path that does not exist" for this. The other places that name the `?` are about this function's behaviour or about history, and are still true: `_split_message`'s docstring says `caf\\udce9.md` "was `caf?.md`" (before #900), `_refused_by_policy` already says the message "lists it as `_shown` does (#900)", and `tests/test_adopt_spelled_names.py` quotes the message as it was on `c7a3ff16`. The `?` in `argv_text`, `trailer_line` and the commit-trailer text is about a `--trailer`, not the split message.

No rows and no mutants: the change is a docstring, the code is the same AST, and no test pins the docstring. So there is no row to fail on main and nothing to mutate.

All with `$Q/v110/run9.sh`, `EMAIL=service@opensoft.dev`, `GIT_SSH_COMMAND=false`, `GIT_CONFIG_GLOBAL=/dev/null`, the stub `gh`, a fake token, git 2.43.0 (`/usr/bin/git`).

Two shards, run before the rebase, at the `2a119f281` base with this one-file change in the tree. Together they are every `tests/*.py` that names `shape_materialize` or `env_commit` (86 files, including `tests/test_adopt_spelled_names.py` and `tests/test_adopt_entry_paths.py`, which run `env_commit` and the #900 message) and `tests/test_agents_md_codes.py`:

| run | files | result |
|---|---|---|
| shard A | 43 | 2873 passed, 4 skipped in 1707.51s, rc=0 |
| shard B | 44 | 4611 passed, 31 skipped in 2462.01s, rc=0 |

After the rebase onto `16f9a0e81`, on the committed head `e32b2703d733f6c60c1b2cba2d62d164d93d7b9a` (`dirty=0`): the trio, `tests/test_repo_hygiene.py`, `tests/test_agents_md_codes.py` and `tests/test_cli_reference.py`, **170 passed in 55.67s**, rc=0. The docstring-only change needed no new shards: the shards' counts are runs at the `2a119f281` base, and #1087 did not touch `env_commit`.

Round 2, on the committed head `de89aa81b1533e32f1007f991aeae05cd915b36d` (`dirty=0`): the same trio, **170 passed in 61.77s**, rc=0. The round 2 commit is one docstring sentence, so it needed no new shards either.

git 2.47.1: no run. The change adds no test file of its own, so there is none to run on it.

`git merge-tree --write-tree HEAD <other>` with HEAD at `de89aa81b1533e32f1007f991aeae05cd915b36d`, after a re-fetch of `origin main` with `-c gc.auto=0 -c maintenance.auto=false` (round 2, about 14:03Z on 2026-10-21):

| against | result |
|---|---|
| `origin/main` `6da757cbe9a4eb66d60c34f38dcc2b332d175e8b` | clean, rc 0, tree `15c11b6c959fc4c7540105d5d62e3f961c8e27fd` |
| #1105 `fix/1048-makefile-follow-up-gnu-read-order` `5a4de7f763b5a3c32adce9614a3a539508416eb4`, an open PR that touches `scripts/shape_materialize.py` | clean, rc 0, tree `db05aead67052643942e35a9689d9dd4a0873fda` |
| #1118 `fix/1089-named-link-skip-tests` `c60066cbeec18b8b6c926e254ed4bee6b9a486c2`, the other open PR that touches `scripts/shape_materialize.py` | clean, rc 0, tree `a40df3b63c32c2a3d8f4c40587b094940c8d6567` |

Read with `gh pr list --state open --json number,headRefName,headRefOid,files`: 16 open PRs, three of them (this one, #1105 and #1118) touch `scripts/shape_materialize.py`, the only file this change touches. The other 13 (#1120, #1119, #1117, #1111, #1109, #1107, #1106, #1102, #1101, #1098, #1097, #1091, #1088) touch no file this change touches.

The same command with HEAD at the reviewed head `e32b2703` gives rv1104's trees: `de31d857` against `6da757cb` and `195ffe19` against #1105. The round 1 merge against #1090 is moot: #1090 has landed in `6da757cb`.

Held as a draft until the v1.2.0 tag; it is now on the v1.2.1 milestone.

- **NIT 1, `scripts/shape_materialize.py` lines 2340-2343, a line round 1 did not change.** The paragraph above the reworded one said the adoption's split message "names each entry path as the source spells it". Since #900 that is true of a printable path alone: `_split_message` lists any other through `_shown`, as the escape the next paragraph describes. It now says "as the source spells it (a printable one; see below)". One new commit on top of `e32b2703`, `de89aa81b1533e32f1007f991aeae05cd915b36d`, 3 lines added and 2 removed, docstring only, pushed as a fast-forward (no rebase, the PR is open).
  - Measured: the AST with docstrings blanked is identical to `e32b2703`'s and, merged into `6da757cb`, to `6da757cb`'s. The docstring holds no lone surrogate, still holds the six characters `\udce9`, and its longest line is 78 characters.
  - Searched `tests/`, `README.md`, `docs/`, the handbook mirror, `AGENTS.md`, `templates/` and the scripts for "names each entry path", "as the source spells" and the new words: only `tests/test_adopt_entry_paths.py` line 233 matches, a test's docstring about `execute` handing each path on as the source spells it, which is another sentence and pins nothing here.
- **NIT 2, this body.** It named main as `16f9a0e81` and #1090 as the one open PR that touches the file. #1090 has landed. Main is `6da757cb`, and the merge-trees above are redone against it, against #1105 and against #1118, a draft that opened since and also touches the file.

Review: rv1104-1107, lane 5's independent opus reviewer, which wrote none of #1104: READY at `de89aa81`, H0 of this landing (issuecomment-6318111413), with no finding; NIT 1 and NIT 2 of the round before are fixed. CI (Brett Heap's ruling 27 and his fast-refresh ruling of 2026-10-15T22:48:35Z): H0 de89aa8 passed tests, tests-git-bash and SonarCloud; the head 92285d4 is a clean update-branch merge of main b66d8f0, with tests-git-bash and SonarCloud green; and the local gate passed on its predicted squash tree 6bd2128 over main b66d8f0: the repo_hygiene, agents_md_codes, cli_reference sets, 171 passed. Landed on Brett Heap's word of 2026-10-23T13:10:26Z, relayed to every lane by lane 3 (#473 issuecomment-6365697123): "merge v1.2.1 PRs once reviewed and green".

Closes #1018

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

brettheap added a commit that referenced this pull request Nov 8, 2026
…eeps a `GNUmakefile` or a `makefile`, which GNU make reads first, the Makefile collision follow-up no longer says the shape's make targets arrive with the merge, and the pinned file says `make` reads the first of the three and the block reaches it only once that file holds it or includes a Makefile that does (#1048) (#1105)

Two sentences said a kept `Makefile` is what holds the shape's block back. GNU make reads the first of `GNUmakefile`, `makefile` and `Makefile` it finds, and only that one, so where the root also keeps `GNUmakefile` or `makefile`, `make` reads that file, and reaches the merged `Makefile` only where that file includes it.

**Measured** (a real `execute`, then `make -n` in the clone, for each kept name; r1046's NIT-B, repeated; the include rows are RV1105's SHOULD-1):

| the source keeps | file `make` reads | after `cat shape/Makefile >> Makefile`, `make -n bootstrap` |
|---|---|---|
| `Makefile` | `Makefile` | exit 0 |
| `Makefile` + `makefile` | `makefile` | exit 2, "No rule to make target 'bootstrap'" |
| `Makefile` + `GNUmakefile` | `GNUmakefile` | exit 2, same |
| all three | `GNUmakefile` | exit 2, same |
| `Makefile` + a `makefile` or `GNUmakefile` that ends in `include Makefile` | that file | exit 0, runs `python3 scripts/bootstrap.py` (exit 2 before the merge) |
| `makefile` or `GNUmakefile` alone | that file | no merge to make: the shape's `Makefile` is the root's, and `make contracts-dir` has no rule until that file holds the block or includes `Makefile` |

**What changed**
- `scripts/shape_materialize.py`, `collision_follow_up`: takes `reads_first`, the file GNU make reads before the shape's `Makefile`. Where it is set, the Makefile's follow-up says GNU make reads that file, the first of the three it finds, and that the shape's make targets reach `make` there only once that file has them, or includes a Makefile that does; until then `python3 scripts/bootstrap.py` is what the shape's `make bootstrap` runs. Where `scripts/bootstrap.py` went aside too, it says neither command runs the shape's bootstrap until the script is merged, and then `python3 scripts/bootstrap.py` runs it. Where the root keeps only the source's `Makefile`, the sentence is what it was, byte for byte. `GNU_MAKE_READS` joins `SHAPE_MAKE_TARGETS`.
- `adopt-project.py`, `_predict_collisions`: reads the file from the plan's root paths as `_makefile_follow_up` does, so the `CONTRACTS_DIR` line and the collision line of one plan name the same file.
- `templates/assembly-root/AGENTS-shape.md`: says `make contracts-dir` prints the block wherever `make` reads the shape's Makefile, names the three files in the order the code reads them, and says that where a `GNUmakefile` or `makefile` is kept, beside a `Makefile` or instead of one, `make` reads it, and not the shape's Makefile unless that file includes it. It is one verbatim copy: the digest in every project's `contracts/shape-pin.yaml` is rendered from its bytes at adopt and scaffold time, and no digest or `tree_sha256` of it is stored in this repository, so there is no pin to move here; each project re-pins through `update-shape.py` as for any template change.
- `tests/test_pinned_copy_text.py`: `ADOPTED_SENTENCES` holds the new text.
- `tests/test_adopt_collision_make_reads.py` (new, 31 rows): the follow-up per kept set, the two lines of one plan naming one file, no collision line where no `Makefile` is kept, the two name tuples equal, the pinned file naming every file the code reads first, the script-aside tail where `make` reads another file, and, in a real adoption's clone, `make -n` held to both sentences for each kept name and for a kept `GNUmakefile` or `makefile` that ends in `include Makefile`.

**Fix round 1** (RV1105's review, `b28f40de0` on top of `5a4de7f76`)
- **SHOULD-1:** the follow-up said the targets reach `make` "only once `X` holds them, or includes a Makefile that does, and not with that merge alone". The last clause is untrue where `X` already ends in `include Makefile`: the merge alone brings them. The clause is gone, and the docstring paragraph, the `SHAPE_MAKE_TARGETS` comment and the `ADOPTED_SENTENCES` comment now say the merge alone brings the targets only where that file includes `Makefile`.
- **NIT-1 (and RV1121's NIT-3):** the follow-up now uses the words of #1121's NOTE: "has them", and "is what the shape's `make bootstrap` runs". `_next_steps` is not touched.
- **NIT-2:** the script-aside tail, where `reads_first` is set, said "neither `make bootstrap` nor `python3 scripts/bootstrap.py` runs the shape's bootstrap before both merges are made", which reads as if `make bootstrap` runs it after them. It now says neither runs it until the script is merged, and then `python3 scripts/bootstrap.py` runs it. I did not keep "before both merges": where `X` is given the targets itself, as the sentence before invites, `make bootstrap` runs the shape's bootstrap once the script is merged, with no `Makefile` merge at all.
- **NIT-3:** the template's "and never the shape's Makefile" is now ", and not the shape's Makefile unless that file includes it". `ADOPTED_SENTENCES` changes with it. No other copy or regex in `tests/` pins that sentence (`git grep` for "never the shape" and "reads the first of them").
- **NIT-4:** the `collision_follow_up` docstring no longer says `execute` reports through it. It says `plan` predicts the follow-ups and stores them in `follow_ups:`, the split commit carries those, `execute` names its collisions by path only (its `beside` lines and the "WRITTEN BESIDE" list), and `Materialized.follow_ups()` has no caller. The method stays.
- NITs 5 to 8 are left to the lane.

**Rows of this round:** `tests/test_adopt_collision_make_reads.py` and `tests/test_pinned_copy_text.py`, 118 rows. With this round's test files on `5a4de7f76`'s code, 17 fail and 101 pass; on `b28f40de0` all 118 pass. The 17 fail for the right reason:

| rows | why they fail on `5a4de7f76` |
|---|---|
| 4 collision rows (`Makefile+makefile`, `Makefile+GNUmakefile`, all three, `Makefile+GNUmakefile+scripts`) | the line holds "not with that merge alone" |
| 2 new script-tail rows (`GNUmakefile`, `makefile`) | the line holds "before both merges are made" |
| 2 new include rows in a real clone (`GNUmakefile`, `makefile`) | `make -n bootstrap` exits 2 before the merge and 0 after it alone (both asserted first, and passing), then the split message holds "not with that merge alone" |
| 7 clone rows, the pinned-file row and `test_a_scaffolded_root_is_told_its_makefile_exports_none` | the pinned file still says "and never the shape's Makefile" |

**Mutants of this round**, each run against the 118 rows on `b28f40de0`, each file restored and its sha256 checked:

| id | mutant | rows failing | result |
|---|---|---|---|
| MA | ", and not with that merge alone" put back | 6 | killed |
| MB | the script-aside tail put back ("before both merges are made") | 3 | killed |
| MC | the template's "and never the shape's Makefile" put back | 9 | killed |
| MD1 | include rows: `not in message` inverted to `in message` | 2 | killed |
| MD2 | include rows: after the merge, `returncode == 0` inverted | 2 | killed |
| ME | "holds them" put back | 6 | killed |
| MF | "the shape's `bootstrap` target runs" put back | 5 | killed |

**Gate** on `b28f40de0`:

| files | passed | failed | skipped |
|---|---|---|---|
| `test_adopt_collision_make_reads`, `test_pinned_copy_text`, `test_adopt_make_reads_first`, `test_adopt_contracts_dir`, `test_adopt_printed_sentences`, `test_adopt_e2e`, `test_repo_hygiene`, `test_scaffold_leg_path_collisions` | 470 | 0 | 0 |
| `test_shape_doctor`, 8 shards (62 + 56 + 54 + 56 + 60 + 57 + 58 + 45 collected) | 445 | 0 | 3 (PowerShell rows, Windows host only) |
| **total** | **915** | **0** | **3** |

**Merge-tree:** `b28f40de0` against main `4ed7c7bf5` is clean, and so is it against each open PR that touches these files: #1104, #1106, #1118, #1119, #1120, #1121 (`1c5173758`), #1122, #1123 and #1125.

**Round 0** (`5a4de7f76`, main `6da757cbe` merged in): on main's code (plus the constant alone), 12 of the then 27 rows failed and 15 passed; the 30 test files that run the changed code or read the template gave 2109 passed, 0 failed and 4 skipped (3 PowerShell rows in `test_shape_doctor.py`, 1 MedxEHR row in `test_adopt_plan.py` that needs a clone `SHAPE_MEDXEHR_CLONE` names); nine mutants each failed 1 to 11 of the then 114 rows.

**Not here**
- The third box of #1048, the `_next_steps` NOTE in `adopt-project.py`, is covered by the change #1047 makes to it: see #1047 and #1121.
- `templates/assembly-root/AGENTS-shape.md` line 19 and the five-commands table say `make bootstrap` without this caveat. They say nothing about a kept file, so they are outside #1048's two sentences.

Closes #1048

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettheap
brettheap marked this pull request as ready for review November 8, 2026 18:13
@sourcery-ai

sourcery-ai Bot commented Nov 8, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 801a7d6 into main Nov 8, 2026
5 checks passed
@brettheap
brettheap deleted the l3-1078-one-byte-locale branch November 8, 2026 18:14
brettheap pushed a commit that referenced this pull request Nov 8, 2026
#1153 (for #1112) was stacked on PR #1106's branch. #1106 has landed on main as the squash 801a7d6, and its final head bd97901 is 504a1dd with main be21751 merged in by update-branch. This is the first of the estate's three stacked-PR merges: it brings that final head in, so the branch holds everything #1106 held at the end. #1153's own change is not touched.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brettheap pushed a commit that referenced this pull request Nov 8, 2026
…anch's side of #1106's files

#1106 landed on main as the squash 801a7d6. This branch already holds #1106's change, through its final head bd97901, so where a hunk conflicts in adopt-project.py or tests/test_adopt_one_byte_locale.py (two of #1106's three files; the third, tests/test_adopt_unenterable_source_and_locale.py, merged clean) this merge takes this branch's side (-X ours), which is #1106's change plus #1153's. Every other hunk of main comes in as main has it. This is the second of the estate's three stacked-PR merges; it makes the branch a descendant of main again, so the pull request shows only #1153's own change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Nov 8, 2026
…heck` makes again for a plan's clone record that neither spelling names is compared with both, so it is kept and the next run reads it rather than cloning again (#1112) (#1153)

Take a one-byte locale such as Latin-1, a plan written in UTF-8 whose
record is a clone `plan` made, and a work directory outside ASCII. When
the clone is gone, neither spelling of the record names anything, so
`check` clones it again at `<--work-dir>/source-readonly/Thing`, at the
UTF-8 bytes `--work-dir` names, which is the plan's record.
`_recorded_source` then decided whether that clone is the record by
comparing its target with the spelling as written alone, the one byte
0xE9. So the clone was put on the list to remove and was removed when
`check` ended, and every run cloned again. This is #803's bug under a
one-byte locale, which #1106 already fixes in
`_keep_the_clone_at_the_record`.

A new `_recorded_paths(recorded)` lists the paths `_recorded_path` asks,
in its order, with `~` expanded, and `_recorded_path` now asks through
it with its behaviour unchanged. A new
`_at_the_record(target, path, there, recorded)` decides whether a clone
`_recorded_source` makes again is at the record. It compares the
target's real path with every path `_recorded_path` asked: where neither
spelling names anything, each path, and where a spelling is there, that
one and each asked before it, never one asked after it. Where the
spelling as written is there but no longer whole, comparing the UTF-8
bytes too would keep a clone made at them, and the next `check` would
read the broken spelling first and be refused `adopt-work-dir-occupied`,
so that case behaves as it did on #1106's head.

`tests/test_adopt_one_byte_locale.py` has 13 passed, with three new
tests that run `check` twice in a child under the Latin-1 locale the
file builds, with a `gh` stand-in that clones a repository on disk.
`test_a_clone_made_again_at_the_record_is_kept_under_a_one_byte_locale`
has `gh` asked once in all and the clone still there, and
`test_a_clone_made_again_at_the_spelling_asked_first_is_kept` holds the
reverse case, where a file sits at the UTF-8 bytes.
`test_a_clone_made_again_beside_the_spelling_as_written_is_not_kept`
holds the case that must not change. Each of the four mutants run is
killed.

Closes #1112

Lane: openreposhape-3 (openRepoShape-3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Nov 8, 2026
…` before `git filter-repo`, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) (#1251)

**Head `abeac66d` is a fix round for RV1251's review of `825923c7`** (#1251 (comment)): one SHOULD and two NITs, set out under "Review round 1" below. The sections after that describe this head.

`_extract_leg` (`adopt-project.py`) now unsets the leg clone's `remote.origin.url`, in the clone's own config file (`--file <work>/.git/config`), right after its `checkout -B` and before `git filter-repo` runs. filter-repo then prints no url, so a Latin-1 stdout has no U+FFFD to choke on, and it still turns every remote-tracking branch into a branch first, so the legs are the ones main makes.

## The defect, reproduced on main

- **Where:** main `b1f45fa7`, git 2.43.0, Python 3.12.3, git-filter-repo 2.47.0 (`--version` prints `a40bce548d2c`).
- **Launcher:** pip's entry point, `#!.../frvenv/bin/python3`, WITHOUT `-E`. So any `PYTHON*` variable would reach filter-repo, and the reproduction sets none: no `PYTHONIOENCODING`, no `PYTHONUTF8`.
- **Locale:** Latin-1, built as #1214 and #1106 build it (`localedef -c`, `LOCPATH`, `LC_ALL=en_US.ISO-8859-1`).
- **Source:** `make_source_repo` under a directory `caf` plus U+00E9, made as the byte 0xE9. The probe runs `plan`, `check` and `execute --local-remote-dir` as the existing row does, but without its `PYTHONIOENCODING=utf-8:strict`.
- **Result:** `plan` exit 0, `check` exit 0 (`plan ok`), and `execute` exit 2:

```text
creating the leg repositories
  bare  .../remotes/Northwind-spec.git
  bare  .../remotes/Northwind-code.git
REFUSED extracting the spec leg. THE EXACT COMMAND WAS:
    git filter-repo --paths-from-file .../work/spec-paths.txt --force
    (in .../work/Northwind-spec)
    exit 1
--- output ---
Traceback (most recent call last):
  File ".../frvenv/bin/git-filter-repo", line 8, in <module>
    sys.exit(main())
  File ".../git_filter_repo.py", line 4973, in main
    filter.run()
  File ".../git_filter_repo.py", line 4878, in run
    self._migrate_origin_to_heads()
  File ".../git_filter_repo.py", line 4462, in _migrate_origin_to_heads
    print(m)
UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 237: ordinal not in range(256)
--- end output ---
```

  Both bare leg repositories were left behind. So the defect does not need a `-E` launcher: a person in a Latin-1 locale sets no `PYTHONIOENCODING`, and pip's launcher fails as well.

**The existing row, `test_adopt_template_values.py::test_a_utf8_source_directory_still_adopts_with_local_remotes`, under Latin-1 on main:**
- With pip's launcher it PASSES (1 passed). Its `PYTHONIOENCODING=utf-8:strict` reaches filter-repo through `execute`'s environment.
- With a copy of that launcher whose only change is `-E` on the shebang (a model of the pipx launcher #1247 quotes), it FAILS with the traceback above. That matches W1196's log.
- A check that `-E` drops the variable: with `PYTHONIOENCODING=bogus-codec`, pip's launcher dies at start-up, while the `-E` copy prints its version.
- On this head the row passes under Latin-1 with both launchers.

## The fix, and the two candidates measured first

**Candidate 1 in #1247, `git remote remove origin` before filter-repo, changes what the legs hold.** It is rejected.
- `git remote remove` deletes `refs/remotes/origin/*` with the remote. filter-repo then no longer makes the source's other branches into branches, and never rewrites them.
- **Measured** with `execute` on main and on candidate 1, over two ASCII sources with fixed dates. For each leg's bare repository the probe compared: every ref, every commit with its tree, parents and full message, and every reachable object.
  - **The standard source** (`make_source_repo`, one branch): identical.
  - **A source with a `release` branch** whose commit R changes `specs/001-feature/tasks.md`, no tag reaching it, and on `main` a `git cherry-pick -x R`: the spec leg's tip differs, `0316169e` on main against `82176493` on candidate 1, with the same tree `9ef5b676`. On main its message reads `(cherry picked from commit 407d03a0...)`, R as rewritten. Under candidate 1 it reads `(cherry picked from commit 30e9dd30...)`, R's id in the source.
- **It also changes behaviour that `check` describes.** As a mutant, it fails the existing `test_adopt_split_branch.py::test_the_extraction_stops_at_a_source_branch_in_its_way` (both rows, `DID NOT RAISE CommandFailed`). With it, a source branch in git's way no longer stops the extraction.

**Candidate 2, `LC_ALL=C.UTF-8` for the child, depends on the host.**
- **Measured here:** under `python3 -E`, a pipe's encoding is `utf-8` under `LC_ALL=C.UTF-8`. It is also `utf-8` under `C`, `POSIX` and a locale name that does not exist, because CPython falls back to C and turns on its UTF-8 mode.
- **But** it hands the child a locale by name and leans on that interpreter behaviour. It changes the locale of every git command filter-repo starts. And on Windows, `LC_ALL` does not set a pipe's encoding (read, not run).
- No `PYTHON*` variable is a cure, because a `-E` launcher drops it.

**What this PR does: unset the url only (`git config --file <work>/.git/config --unset remote.origin.url`).** It depends on no locale. It names the clone's own config file, so `GIT_CONFIG` in the environment cannot redirect it to a file of the person's (RV1251's SHOULD-1, below). It goes on from a clone that has no url to unset, and from no other failure (NIT-1, below).
- filter-repo's `_migrate_origin_to_heads` (2.47.0) turns `refs/remotes/origin/*` into `refs/heads/*` first. Then it returns when `remote.origin.url` is not in the config, before the NOTICE that printed the url and before its own `git remote rm origin`.
- **The legs are byte-identical to main's** in both probes above, the cherry-pick source included (`diff` of the dumps: empty).
- **What stays in each work clone** (measured on this head, a plain `execute`): the url-less `[remote "origin"]` with its fetch refspec `+refs/heads/*:refs/remotes/origin/*`, and `branch.main.remote = origin` and `branch.main.merge = refs/heads/main`. Nothing reads any of it. `_push_the_leg` pushes to the leg's url, and `_mount_the_legs` mounts the clone by its path. RV1251 measured that the mounted submodules hash the same as main's.
- The change is one `run` in `_extract_leg` (now with a `try`), with a comment. No pinned or template copy of `adopt-project.py` exists, and no document describes this step.

## Review round 1 (RV1251 at `825923c7`)

**SHOULD-1, fixed: the unset named no config file.** `GIT_CONFIG` in the environment (a legacy variable git still reads) names the file a bare `git config` writes.
- **Reproduced** with `GIT_CONFIG` naming a file that holds `[remote "origin"] url = ...` and the standard source, in `execute`:
  - main: exit 0, `adoption verified`, the file unchanged.
  - `825923c7`: exit 2, `REFUSED extracting the spec leg`, both bare legs made and nothing pushed, and the file empty. The first leg's unset took the only key out of it, and filter-repo, reading that empty file as the config, died with a `ValueError` in `get_config_settings`. (RV1251's file ended in an unset exit 5 instead.)
  - this head: exit 0, `adoption verified`, the file byte for byte as it was.
- **Fix:** `run(["git", GIT_CONFIG, "--file", str(work / ".git" / "config"), "--unset", "remote.origin.url"], cwd=work)`. `--local` is no way round it: with `GIT_CONFIG` set, git 2.43 exits 129 (only one config file at a time).
- **`work / ".git"` is a directory wherever `execute` reaches the clone:**
  - the clone is made by one `git clone -q <source> <work>`, and `work` is `work_root / <name>`, where `work_root` is `os.path.realpath(--work-dir)` or a `tempfile.mkdtemp` folder (`_work_root`): an absolute path;
  - an exported `GIT_DIR` does not move a clone (measured: the clone has its own `.git`);
  - an exported `GIT_WORK_TREE` does make the clone another shape (the git dir is `work` itself), but `execute` stops on it first, on main and on this head alike, at the legs' `git init --bare` (`REFUSED a git or gh command failed`);
  - a path whose directory is not there gives exit 255 (`could not lock config file ... No such file or directory`), which is refused.
- **Not changed:** filter-repo's own `git config --list` also reads the file `GIT_CONFIG` names (the `ValueError` above came from it), so under `GIT_CONFIG` it sees that file's settings, as on main. On this head the run ends `adoption verified` with the legs main makes.

**NIT-1, handled: a clone with no `origin` url.** A global `clone.defaultRemoteName = upstream` gives the clone a remote that is not `origin`, so `git config --unset remote.origin.url` has no key and exits 5 (measured, git 2.43). It can happen.
- **Before:** main fails at `mounting the legs` (`No such remote 'origin'`), after both legs are pushed. `825923c7` failed earlier, at the unset (exit 5, an empty output block), with nothing pushed.
- **Now:** `_extract_leg` goes on from exit 5 and from no other exit (`GIT_CONFIG_KEY_ABSENT = 5`). The run fails where main fails, not earlier and not worse. Any other exit of the unset (255 for a file git cannot lock, 129 for a usage it refuses) is refused as any failed command is.
- **Not mended here:** `clone.defaultRemoteName` itself. `execute` fails on it on main too. It is a follow-up for its own issue.
- Exit 5 also covers a config with more than one `remote.origin.url` (measured); that too goes on, as main does.

**NIT-2, corrected in this body:** the claim that the fix "depends on no locale and no variable" (it now names its config file, so `GIT_CONFIG` cannot redirect it); the work clone's leftovers, now listed under "What stays in each work clone" (the url-less `[remote "origin"]`, `branch.main.remote` and `branch.main.merge`); and the note that #1141 conflicts, dropped: it merges clean at `0b8fb320`.

## Rows

`tests/test_adopt_filter_repo_origin_url.py`, new, five rows:
- **`test_filter_repo_starts_with_no_origin_url_and_every_branch`** (any locale). `_extract_leg` itself runs over an ASCII source with a second branch, with `run` spied at the `git filter-repo` call. When filter-repo starts, the clone has no `remote.origin.url` and still has `refs/remotes/origin/main` and `refs/remotes/origin/beside`. filter-repo's output holds neither `Removing 'origin' remote` nor the source path. Afterwards the clone has `refs/heads/beside`, and the leg holds the clone's HEAD.
- **`test_execute_adopts_a_source_named_in_latin1_under_latin1`** (Latin-1). It runs `plan`, `check` and `execute --local-remote-dir` for #1247's source (a directory `caf` plus the byte 0xE9) in a Latin-1 locale the test builds as #1106 builds one. No `PYTHON*` variable is set except `PYTHONUTF8=0`. It asserts no traceback, `check` exit 0, `execute` exit 0 and `adoption verified`.
  - It skips, by name and with `localedef`'s words, where no Latin-1 locale can be built, and on Windows. It also skips where a name that is not UTF-8 cannot be made (`skip_where_a_name_cannot_be_not_utf_8`).
  - It builds its locale whatever the gate's locale is, so it fails on main under a UTF-8 gate too, which is what CI runs.
- **`test_execute_leaves_the_file_git_config_names_alone`** (new, SHOULD-1). `execute` with `GIT_CONFIG` naming a file that holds a `[remote "origin"]` url: `adoption verified`, and the file is byte for byte as it was. The child gets none of the gate's locale variables, so the row runs alike under a UTF-8 and a Latin-1 gate.
- **`test_a_clone_with_no_origin_url_goes_on_to_filter_repo`** (new, NIT-1). `_extract_leg` itself under a global `clone.defaultRemoteName = upstream`: the unset exits 5 (recorded as it is raised), the clone's `remote.upstream.url` is the source, and the leg is pushed.
- **`test_an_unset_that_fails_for_any_other_reason_stops_the_leg`** (new, NIT-1). A `config.lock` in the clone makes the unset exit 255: `CommandFailed`, before `git filter-repo` and with nothing pushed. It needs no git-filter-repo.
- The earlier rows' setup is now two helpers, `extract_the_code_leg` and `adopt_each_step`. The file is ASCII with no backslash escape; each non-ASCII character is built with `chr` or `bytes`.

**Counts** (5 rows; the same file in each tree):

| code | UTF-8 gate, pip and `-E` | Latin-1 gate, pip and `-E` |
|---|---|---|
| main (`_extract_leg` as on `dffc4226`; unchanged in `5926ccca`) | 4 failed, 1 passed (the GIT_CONFIG row passes: main leaves the file alone) | 4 failed, 1 passed |
| `825923c7` (the previous head) | 2 failed, 3 passed (the GIT_CONFIG row and the exit-5 row fail) | 2 failed, 3 passed |
| this head | 5 passed | 5 passed |

The two earlier rows fail on main and pass here. The GIT_CONFIG row fails on `825923c7` and passes here. The exit-5 row fails on `825923c7` and on main (no unset to record) and passes here. The lock row passes on `825923c7`, which refuses every exit; it guards the new `except`.

## Mutants

Each mutant was applied in a worktree at this head, the five rows plus the two `test_the_extraction_stops_at_a_source_branch_in_its_way` rows were run under UTF-8 and under Latin-1, and `adopt-project.py` was restored with its sha256 checked after each one (`RESTORED sha256 ok`, tree as before).

| mutant | UTF-8 | Latin-1 | killed by |
|---|---|---|---|
| M1: drop the unset | 4 failed, 3 passed | same | the two earlier rows, the exit-5 row and the lock row (no unset to see) |
| M2: the unset moved after `git filter-repo` | 2 failed, 5 passed | same | the two earlier rows |
| M3: `git remote remove origin` instead | 5 failed, 2 passed | same | the any-locale row, the exit-5 row, the lock row and both split-branch rows |
| M4: drop `--file <path>` | 1 failed, 6 passed | same | the GIT_CONFIG row |
| M5: `--local` instead of `--file <path>` (exit 129 under `GIT_CONFIG`) | 1 failed, 6 passed | same | the GIT_CONFIG row |
| M6: accept every exit of the unset | 1 failed, 6 passed | same | the lock row |
| M7: accept none (no `try`) | 1 failed, 6 passed | same | the exit-5 row |
| M8: `--file <work>/config`, a path with no file (exit 5, carried on) | 3 failed, 4 passed | same | the any-locale row, the Latin-1 row and the lock row |

## Gates

All runs used `/usr/bin/python3 -m pytest -p no:cacheprovider` with one reused basetemp per run kind and `-o tmp_path_retention_policy=failed`, at most 2 pytest runs at a time. The environment had a stub `gh` first on PATH, `GH_TOKEN=bogus`, an empty `GH_CONFIG_DIR`, `GIT_CONFIG_GLOBAL` an empty file and `GIT_CONFIG_NOSYSTEM=1`, and no `GIT_ALLOW_PROTOCOL`, `GIT_SSH_COMMAND`, `GIT_CONFIG`, `LANES_*` or `PYTHON*` variable. filter-repo was pip's launcher (`a40bce548d2c`) unless a row says otherwise.

**The tree gated is a scratch merge** of this head and `origin/main` `5926ccca` (built with `merge-tree` and `commit-tree`; its diff against main is this PR's two files). One hygiene row caught a path in the first attempt (`/home/someone/...` in the new test, `test_no_committed_file_names_a_host_absolute_path`); the row now names `https://example.invalid/keep-me.git`, and everything below ran on the final head.

**Gate A, UTF-8 (`LC_ALL=C.UTF-8`), one pytest, 23 files:** **1518 passed, 3 skipped** on the scratch merge.
- the new file, `test_adopt_template_values.py`, `test_adopt_split_branch.py` and `test_adopt_e2e.py`;
- the other four files that name `_extract_leg` or `_push_the_leg`: `test_adopt_filter_repo_failure.py`, `test_adopt_history_gitlinks.py`, `test_adopt_late_failure.py` and `test_adopt_plan_escapes.py`;
- `test_adopt_check_throwaway_repos.py`, `test_adopt_partial_clone_source.py`, `test_adopt_pins.py`, `test_adopt_git_rm_bytes.py` and `test_adopt_plan_partial_clone.py` (new on main);
- `test_repo_hygiene.py`, `test_path_guard_hygiene.py`, `test_cli_reference.py` and `test_scaffold_leg_path_collisions.py`;
- the files that scan every test file: `test_no_auto_maintenance.py`, `test_non_utf8_git_failures.py`, `test_git_output_decoding.py`, `test_setup_project_repo_misdrawn.py`, `test_shape_advisory.py` and `test_openreposhape_package.py`.
- **Main `5926ccca`, the same 22 files without the new one:** 1513 passed, 3 skipped. The head's extra 5 are the five rows.
- **Skips (`-rs`):** the same 3 on both, none for lack of git-filter-repo: `test_adopt_check_throwaway_repos.py:482` (a disk that keeps case) and `test_no_auto_maintenance.py:222` x2 (`GIT_CONFIG_GLOBAL` set).

**`tests/test_shape_doctor.py`, UTF-8, 4 shards from a bash script** (449 ids): **Head (scratch merge):** 118, 110, 118 and 100 passed, with 3 skipped in the last shard (the PowerShell line, Windows only): 446 passed, 3 skipped. RV1251 and W1247 measured 446 passed and 3 skipped on main at `11560eb9` and `b1f45fa7`; it was not re-run on `5926ccca`, because this PR touches nothing the doctor reads.

**Latin-1** (the locale built as above, `PYTHONUTF8=0`), the new file and `test_adopt_template_values.py`:

| run | main `5926ccca` | this head (scratch merge) |
|---|---|---|
| new file + `test_adopt_template_values.py`, pip launcher | 4 failed, 69 passed, 5 skipped (template values only) | 4 failed, 74 passed, 5 skipped |
| the same, `-E` launcher | 5 failed, 68 passed, 5 skipped (template values only) | 4 failed, 74 passed, 5 skipped |

- **The 4 failures are the same rows on main and on this head, under both launchers.** All four already fail on main and are #1214's to judge, not this PR's: `test_plan_refuses_a_flag_no_file_can_hold` at `--elected-by` and `--reference` with U+DCE9, `test_plan_and_check_say_the_same_sentence`, and `test_execute_refuses_a_local_source_directory_that_is_not_utf8` (under Latin-1 the byte 0xE9 is valid text).
- The fifth failure on main under `-E` is `test_a_utf8_source_directory_still_adopts_with_local_remotes`, which this head fixes.

## Merge-trees

`git merge-tree --write-tree` of this head against:
- **main:** clean against `5926ccca`, and the merged `adopt-project.py` keeps the `--file` call.
- **#1177 (`5208e874`) and #1225 (`73c9f57e`)**, the only open PRs that touch `_extract_leg`: both clean, each keeps the `--file` call.
- **#1141 (`0b8fb320`):** clean. The earlier note that it conflicts is stale and is dropped.
- RV1251 merge-tree'd the 27 other open PRs that touch `adopt-project.py` against the previous head: 26 clean, and #1237 conflicts in `tests/test_adopt_check_throwaway_repos.py` with main alone. This round re-ran main, #1177 and #1225 (the only PRs that touch `_extract_leg`) and #1141 (the one the earlier body got wrong).

## Not done, and limits

- Windows, another one-byte locale (cp1252 included) and a real pipx install were not run. The `-E` launcher here is a one-line copy of pip's.
- **Read from the code, not run:** filter-repo prints nothing else that holds the url. Its other prints of `origin` are under `--sensitive-data-removal`, which `execute` never passes.

Closes #1247

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant