[brettheap]

Lane: openreposhape-1 (openRepoShape-1)
Claim: #765 (comment)

A v1.1.1 draft. It lands only after the v1.1.0 tag, in the order lane 3 posts.

What

The three tools that read the branch HEAD is on now read it as the bytes git wrote, in one helper each:

Tool	The one read	Its callers
update-shape.py	_symbolic_head	current_branch (commit_on_branch, _awaits_its_pull_request, cmd_apply), and Rollback.new_branch, _undo_branch and _head_is
scripts/family.py	_holder_branch	_checked_branch (the real run of add, bump, remove) and _branch_said (their dry runs)
scripts/bump-leg.py	current_branch	_branch_to_bump_from (both runs)

Each reads git symbolic-ref -q HEAD as bytes, takes off exactly one trailing newline and exactly refs/heads/, and decodes the rest with surrogateescape. None of the three reads has a strip() or an errors="replace" left.

• A branch ending in U+00A0, U+2028 or U+0085 reads as itself, and the push line names it exactly.
• A branch whose name is not UTF-8 is refused by name, before anything is committed or written:
  • update-branch-not-utf8, family-branch-not-utf8 and bump-leg-branch-not-utf8, one per tool, under the prefix each tool already uses.
  • Such a name is one that still holds a surrogate after the decode.
  • The refusal names the branch by its bytes, b'caf\xe9', as trailer-not-utf8 names a trailer (b'Jos\xe9').
  • Its remedy ends with git -C <root> branch -m <a new name>, quoted for a POSIX shell in all three tools. git branch -m renames the branch HEAD is on, so the remedy never has to spell the old name: a pasted b'caf\xe9' would reach git as the text bcaf\xe9 (measured with sh).
  • No push line is printed for a name the tool cannot spell.
• A rollback hands git back the branch it started on as git's own bytes. update-shape.py's Rollback._undo_branch is the one place a branch returned by one of these three reads goes back to git, and it now goes through _as_git_argument, as bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 does for bootstrap.py. Two rollbacks outside this PR hand git a branch they read for themselves: a member's in family.py (_member_head, then _head_back) and a leg's in bump-leg.py (leg_placement, then restore_leg). Both are under "Not in this PR".

Fix round 1 (after r765's review, #849 issuecomment-6242841859)

• Finding 1, CI red. tests/test_readme_windows_quoting_sentence.py failed: _holder_branch's remedy called shell_line for the host, which README's sentence on the lines quoted for PowerShell does not name. The remedy is shell_line([...], "posix") now; the file gave 9 passed then (it holds 40 tests at the head, all passing).
• Finding 2. Under LC_ALL=C PYTHONUTF8=0, a rollback from feat<U+00A0> raised UnicodeEncodeError part way and left the copies written. _as_git_argument fixes it, and a rollback from <U+65E5><U+672C>, which raised on main too, now exits 2 with every byte git holds as it was.
• Finding 3. r765's test_a_rollback_puts_head_back_on_the_branch_it_started_on, in both locales.
• Finding 4. held_by_git also holds every git directory under .git/modules (refs, object count, FETCH_HEAD with its write time), so the two cases' "before the fetch" is asserted. A bump-leg dry run on a bump branch changes the spec entry, and a family bump on a branch changes members/IRRS, so the snapshot sees a fetch.
• Optional point, taken. The refusals name the branch by its bytes.

Merge of main (after r765's PR-READY at d98df568)

The branch conflicted with main, so main d6071c7d is merged into it. The new head is the merge commit da3fe1c8, whose parents are d98df568 and d6071c7d. Nothing was squashed or rebased.

The conflicts. All of them come from main's 5fc049b7 (#766, #814), which names the root and the holder in bump-leg.py's and family.py's refusals as shown_path names a path:

File	Hunk	Resolved
tests/test_bump_leg_branch_and_push_line.py	the module docstring: #849's AND THE BRANCH'S BYTES (#765) paragraph against #766's THE ROOT ABOVE THEM (#766)	both, #765's first; #766's now ends by saying that #765's bump-leg-branch-not-utf8, raised in the same check, names the root the same way
	the conftest imports: #849's NOT_UTF_8_BRANCH_NAMED, held_by_git and on_a_branch_that_is_not_utf_8 against #766's second import line	both
tests/test_family_tracking_branch.py	the conftest imports, the same way	both; #766's line also takes SHOWN_NAMES and named_as, for the new row below

• Every test body on both sides merged without a conflict.
• scripts/bump-leg.py, scripts/family.py and tests/conftest.py merged by themselves.
• Main changed neither update-shape.py nor tests/test_update_shape_apply_lines_and_headers.py. Against main, the +/- lines of those two files and of tests/conftest.py are the ones they had against fc867207.

One change in meaning. The two not-UTF-8 refusals now name the path as shown_path names it.

• bump-leg-branch-not-utf8 is raised from current_branch, which _branch_to_bump_from asks. family-branch-not-utf8 is raised from _holder_branch, which _checked_branch asks.
• Since family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766, every other refusal of those two checks names the root or the holder as shown_path names a path, and their docstrings say so. update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's two named it raw ({root}), so a root under a folder whose name holds an ESC or a line break sent it to the terminal from this one refusal.
• Both details are {shown_path(root)} now. A printable path is named byte for byte, so update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's own cases, whose paths are printable, assert the same lines as before.
• The branch -m remedy is a command to paste and is unchanged. It holds the path raw inside its shell quotes, as the switch -c beside it does (Tracking: setup, scaffold, family and the shared helpers, creating projects and families #459, for family.py).
• update-shape.py names no path through shown_path, on main or here, so update-branch-not-utf8 is unchanged.

Two new rows hold it, each over SHOWN_NAMES (an ESC, a line break then a forged line, and a printable name). Each asserts exit 2, an empty stdout and the refusal's whole detail line:

• tests/test_bump_leg_branch_and_push_line.py: test_a_root_on_a_branch_that_is_not_utf8_is_named_safe_to_show[ESC], [line-break] and [printable];
• tests/test_family_tracking_branch.py: test_a_holder_on_a_branch_that_is_not_utf8_is_named_safe_to_show[ESC], [line-break] and [printable], with remove.

With {root} put back in either tool, in a scratch clone of da3fe1c8, that tool's [ESC] and [line-break] rows fail (2 failed, 4 passed). [printable] passes, as #766's printable rows pass by design.

Revert rows re-run at da3fe1c8, each alone in a scratch clone, against the 12 cases in the table below and the 6 new rows (18 passed with nothing reverted):

• F1, .strip() back in family's _holder_branch: 1 failed, the family no-break space.
• B3, .strip() back in bump-leg's current_branch: 2 failed, the bump-leg no-break space beside a branch and beside the tracking branch.
• U1, .strip() back in update-shape's _symbolic_head: 3 failed, the update-shape no-break space and the rollback in both locales.

Each matches the revert table.

r765's nit is taken too: the 103-character docstring line in tests/test_family_tracking_branch.py is rewrapped.

Gate at da3fe1c8, dirty 0. Each run used /usr/bin/python3 -m pytest -p no:cacheprovider, a stub gh first on PATH, GH_TOKEN=bogus-w765m, an empty GH_CONFIG_DIR, GIT_CONFIG_GLOBAL=/dev/null and GIT_SSH_COMMAND=false. Each mode ran in six shards, with each file in a pytest run of its own. The 66 files:

• every tests/test_family*.py (31 files), tests/test_bump_leg*.py (8) and tests/test_update_shape*.py (18), which take in the PR's three test files;
• tests/test_agents_md_codes.py, test_readme_windows_quoting_sentence.py, test_git_output_decoding.py, test_non_utf8_git_reads.py, test_bootstrap_non_utf8_branch.py and test_no_auto_maintenance.py;
• tests/test_shape_doctor.py, test_scaffold_leg_path_collisions.py and test_cli_reference.py.

Results:

• git-filter-repo on PATH: 3,026 passed, 32 skipped, 3,058 collected. The skips:
  • 19 test_family_init_values.py cases for a template that quotes the value;
  • 4 reftable cases, since git 2.43 cannot make a reftable repository;
  • 3 PowerShell lines that only a Windows host prints, and 3 NTFS junction cases;
  • 2 test_no_auto_maintenance.py cases, because the session began with GIT_CONFIG_GLOBAL set, as the gate sets it;
  • 1 test that needs a disk that ignores case.
• git-filter-repo hidden (PATH=<stub gh>:/usr/local/bin:/usr/bin:/bin, where command -v git-filter-repo finds nothing): 3,007 passed, 51 skipped, 3,058 collected. The 19 extra skips are the tests that need git-filter-repo: 14 in test_update_shape_merged_away.py, 4 in test_shape_doctor.py and 1 in test_update_shape_apply_lines_and_headers.py.

Every shard log names the head, dirty 0, and ends EXIT=0, and every file's run exited 0. The log's filter missed one summary line, 14 skipped for test_update_shape_merged_away.py with git-filter-repo hidden. That line comes from a re-run of the file alone at the same head, which exited 0.

Main moved on to 33419925 after the fetch, and its 75d70502 (#715) touches scripts/bump-leg.py and update-shape.py, in the tracking-branch reads, not the reads this PR changes. A local merge of da3fe1c8 with it, which was not pushed, is clean (Auto-merging scripts/bump-leg.py, Auto-merging update-shape.py; tree c34d4048). On that merge, the PR's three test files, test_readme_windows_quoting_sentence.py and test_agents_md_codes.py give 201 passed, each file EXIT=0.

Why (repro on main)

Measured on main fc867207, where #746, #810 and #813 have all landed, and then on this branch's head before the merge of main. Each run used local bare remotes, a stub gh first on PATH, GH_TOKEN=bogus-w765, an empty GH_CONFIG_DIR, GIT_CONFIG_GLOBAL=/dev/null and git 2.43.0.

Line 1: the branch read strips Unicode whitespace

Each helper, in process, over a repository on feat plus one character, with a branch feat beside it:

HEAD on	update-shape.py current_branch	family.py _holder_branch	bump-leg.py current_branch
feat<U+00A0>	main: feat. Branch: feat<U+00A0>	main: feat. Branch: feat<U+00A0>	main and branch: feat<U+00A0> (#746)
feat<U+2028>	main: feat. Branch: feat<U+2028>	main: feat. Branch: feat<U+2028>	main and branch: feat<U+2028>
feat<U+0085>	main: feat. Branch: feat<U+0085>	main: feat. Branch: feat<U+0085>	main and branch: feat<U+0085>

End to end, with feat<U+00A0> checked out beside feat:

• family.py remove.
  • On main: it committed on feat<U+00A0> and ended NEXT git -C <holder> push -u origin refs/heads/feat:refs/heads/feat. Pasted, that pushed feat at main's commit, not the commit the act made.
  • On the branch: NEXT git -C <holder> push -u origin 'refs/heads/feat<U+00A0>:refs/heads/feat<U+00A0>'. Pasted, origin's feat<U+00A0> is the act's commit, and feat is not pushed.
• update-shape.py apply --branch feat --push.
  • On main: exit 0, committed on feat: ... and pushed feat to origin. But the commit was on feat<U+00A0>, and origin's feat was the commit before the re-pin.
  • On the branch: exit 2, REFUSED update-branch-exists: --branch feat already exists and is not the current branch ('feat\xa0' is checked out), rolled back, with nothing pushed. That is the posture commit_on_branch already documents for a --branch that exists and is not checked out.
• update-shape.py's rollback, from feat<U+00A0> with a pre-commit hook that exits 1:
  • On main: ROLLED BACK: every byte ..., then switched HEAD back to feat, the neighbour.
  • On the branch: HEAD goes back on feat<U+00A0>, in the default locale and under LC_ALL=C PYTHONUTF8=0.
• bump-leg.py has read the name whole since bump-leg.py: the branch is read in full and pushed as refs/heads/<b>:refs/heads/<b>, dirty names are spelled, committed lists what git holds, and the dry run says it runs no validator (#672) #746 (rstrip("\n")). On main and on the branch, its NEXT names feat<U+00A0>, and pasted, it pushes the bump. The new case here is the same read beside the tracking branch main: strip() read main<U+00A0> as main and refused bump-leg-on-tracking-branch.

Line 2: a branch whose name is not UTF-8

HEAD on caf<0xE9>:

• bump-leg.py.
  • On main: exit 0, the bump committed, and NEXT git -C <root> push -u origin 'refs/heads/caf<U+FFFD>:refs/heads/caf<U+FFFD>'. Pasted: exit 1, error: src refspec refs/heads/caf<U+FFFD> does not match any, and nothing was sent.
  • On the branch: exit 2, REFUSED bump-leg-branch-not-utf8, nothing fetched, written or committed. The dry run refuses alike, as it refuses a detached root.
• family.py remove.
  • On main: exit 0, committed on caf<0xE9>. The NEXT line held whatever the stream made of the lone surrogate. Under this machine's POSIX locale (stdout surrogateescape) that was the raw byte 0xE9, which a UTF-8 terminal shows, and copies, as U+FFFD. Under PYTHONIOENCODING=utf-8:strict it was the six characters \udce9, which pasted name another branch.
  • On the branch: exit 2, REFUSED family-branch-not-utf8, for add, bump and remove alike, before anything is fetched or written.
• update-shape.py apply --branch shape/update-partial.
  • On main, with nothing failing: exit 0, and the new branch was committed and pushed. The branch HEAD started on is named nowhere in that run.
  • On main, where a rollback ran (a pre-commit hook that exits 1): ROLLED BACK IN PART, then HEAD is not back on caf<U+FFFD>: git -C <root> checkout 'caf<U+FFFD>'. The branch the run created was left behind, with a by-hand line that names caf<U+FFFD> too. HEAD could not go back, because no branch is called caf<U+FFFD>.
  • On the branch: exit 2, REFUSED update-branch-not-utf8, before the first write, so there is nothing to roll back.
• After git branch -m cafe, each tool runs again as on any branch (measured with bump-leg's dry run: exit 0).

How

• scripts/bump-leg.py, current_branch. The subprocess call drops text=True, encoding and errors. The newline and refs/heads/ come off the bytes, and the rest is decoded with surrogateescape. branch.encode("utf-8") raising is the test for a name that is not UTF-8, and the refusal names the bytes, and the root as shown_path names it (see "Merge of main"). _branch_to_bump_from, the one caller, asks it before the fetch in both runs.
• scripts/family.py, _holder_branch. The same, through the _git_bytes call it already made, and its refusal names the holder as shown_path names it (see "Merge of main"). Its .strip() is gone. Its remedy's branch -m line is quoted for a POSIX shell (shell_line(..., "posix")), as bump-leg's and update-shape's are, so README's sentence on the lines quoted for PowerShell stays true. This is the only family.py hunk.
• update-shape.py.
  • _symbolic_head asks git itself, as bytes, in place of _git_answer's text. It still answers None for no git, or for a git that refuses, as _git_answer did, so nothing raises inside a rollback. refs/heads/ is ASCII, so _branch_of taking it off the decoded text takes off exactly its bytes.
  • current_branch takes the branch from _symbolic_head, so the tool has one read, and refuses a name that is not UTF-8.
  • cmd_apply calls current_branch(root) in its if args.branch: block, beside _refuse_what_head_does_not_commit, so the refusal comes before the first write. This is the one hunk outside a branch-read helper. Without it, commit_on_branch refused only after the copies were written, and rolled them back (revert U3 below).
  • Rollback._undo_branch hands git the branch HEAD started on through _as_git_argument, so subprocess writes the name's own UTF-8 bytes under any locale, as bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 does for bootstrap.py. Under LC_ALL=C PYTHONUTF8=0, a rollback from feat<U+00A0> raised UnicodeEncodeError part way and left the copies written; one from <U+65E5><U+672C> did so on main too. The by-hand lines it prints keep the name as it is.
• Where a read branch goes back to git. family.py hands the branch _holder_branch reads only to printed lines (_checked_branch's refusal, _branch_said, _print_next), and bump-leg.py hands its current_branch only to printed lines (_branch_to_bump_from's refusals, the dry run's hint, the NEXT line). Neither hands it to git, so neither needs _as_git_argument.
• scripts/repo_shape.py is not changed.

Tests

The cases, each in the file where its tool's fixture already lives:

File	Case	New
tests/test_bump_leg_branch_and_push_line.py	test_a_branch_ending_in_a_no_break_space_is_pushed_as_itself[beside-a-branch]	no: #746's case, now one of two parameters
	...[beside-the-tracking-branch]: main<U+00A0> beside main	yes
	test_a_branch_named_with_a_byte_that_is_not_utf8_is_refused_by_name[bump], [dry-run]	yes
tests/test_family_tracking_branch.py	test_a_branch_ending_in_a_no_break_space_is_pushed_as_itself: remove, its NEXT run as printed	yes
	test_a_branch_named_with_a_byte_that_is_not_utf8_is_refused_by_name[add], [bump], [remove]	yes
tests/test_update_shape_apply_lines_and_headers.py	test_a_branch_ending_in_a_no_break_space_is_not_taken_for_its_neighbour	yes
	test_a_branch_named_with_a_byte_that_is_not_utf8_is_refused_first	yes
	test_a_rollback_puts_head_back_on_the_branch_it_started_on[locale], [c-locale] (r765's)	yes

• Every case in the table that is not UTF-8 asserts exit 2 and the code. It asserts the name as the refusal spells it, b'caf\xe9', and the branch -m <a new name> remedy. It also asserts that HEAD, the ref HEAD is attached to, every ref, the index, the status and every git directory under .git/modules are as they were (held_by_git).
• tests/conftest.py gains NOT_UTF_8_BRANCH (caf + byte 0xE9), NOT_UTF_8_BRANCH_NAMED (how a refusal names it), git_answer_bytes, held_by_git and on_a_branch_that_is_not_utf_8. The last skips where skip_where_a_name_cannot_be_not_utf_8 says the name cannot be made: Windows' os.fsencode, and a filesystem that answers EILSEQ.
• own_remote is split out of tagged in the update-shape file, so the new cases push to a bare remote of their own.

Revert table. Each change was reverted alone in a scratch clone of the head before the merge of main, against the 12 cases above and tests/test_readme_windows_quoting_sentence.py (9 tests):

Reverted	What fails
all three tools as main fc867207 has them	10 of the 12 cases. The two [beside-...] bump-leg cases pass, as #746 already fixed that read
U1. update-shape.py _symbolic_head: .strip() back on the decoded ref	3: update-shape no-break space, and the rollback in both locales
U2. update-shape.py current_branch: no refusal	1: update-shape not UTF-8
U3. update-shape.py cmd_apply: no early current_branch(root)	1: update-shape not UTF-8, refused only after the writes, ROLLED BACK
U4. update-shape.py _symbolic_head: errors="replace" for surrogateescape	1: update-shape not UTF-8
UN. update-shape.py: the name as {branch!r}, not its bytes	1: update-shape not UTF-8
R1. Rollback.new_branch: .strip() back	2: the rollback, in both locales
R2. _undo_branch: no _as_git_argument	1: the rollback under LC_ALL=C PYTHONUTF8=0
F1. family.py _holder_branch: .strip() back	1: family no-break space
F2. family.py _holder_branch: no refusal	3: family not UTF-8, add, bump, remove
FA. family.py: the remedy quoted for the host (no "posix")	1: test_readme_windows_quoting_sentence.py [scripts/family.py]
FN. family.py: the name as {branch!r}	3: family not UTF-8
B1. bump-leg.py current_branch: errors="replace" for surrogateescape	2: bump-leg not UTF-8, real and dry run
B2. bump-leg.py current_branch: no refusal	2: bump-leg not UTF-8
B3. bump-leg.py current_branch: .strip() back (the issue's mutant)	2: bump-leg no-break space, beside a branch and beside the tracking branch
BN. bump-leg.py: the name as {branch!r}	2: bump-leg not UTF-8

Taking exactly one newline off, where rstrip("\n") took every trailing one, cannot be told apart by any test: git refuses a newline in a ref name.

Runs. Head d98df568, before the merge of main (the gate at the merge is under "Merge of main"), /usr/bin/python3 -m pytest, a stub gh first on PATH, GH_TOKEN=bogus-w765 and an empty GH_CONFIG_DIR, in six shards per mode. The 62 files:

• the PR's three test files, tests/test_shape_doctor.py, tests/test_scaffold_leg_path_collisions.py and tests/test_cli_reference.py;
• every tests/test_family*.py (27 files), every tests/test_bump_leg*.py (8 files), and tests/test_non_utf8_git_reads.py, test_update_shape_partial_rollback.py, test_update_shape.py, test_update_shape_landing.py, test_update_shape_paste_lines.py, test_update_shape_next_quoting.py, test_next_refspec_branch_length.py, test_tracking_branch_ref.py, test_branch_left_bump_leg_setup.py and test_non_utf8_name_helpers.py;
• the repository-wide guards that read the tools' own source: tests/test_readme_windows_quoting_sentence.py (the one the first gate missed), test_agents_md_codes.py, test_git_output_decoding.py, test_validator_child_encoding.py, test_update_shape_kept_helper.py, test_doctor_next_is_a_command.py, test_openreposhape_package.py, test_setup_project_py.py, test_ruleset_hint_tracking_branch.py, test_template_values_single_pass.py, test_pinned_copy_text.py, test_refusal_default_remediation.py and test_repo_hygiene.py.

Results:

• git-filter-repo on PATH: 3,534 passed, 23 skipped, 1 xfailed, 3,558 collected. The skips: 19 test_family_init_values.py cases for a template that quotes the value, three PowerShell lines that only a Windows host prints, and one test that needs a disk that ignores case.
• git-filter-repo hidden (PATH=<stub gh>:/usr/local/bin:/usr/bin:/bin): 3,523 passed, 34 skipped, 1 xfailed, 3,558 collected. The 11 extra skips are the tests that need git-filter-repo.

Every log names the head, dirty 0, and ends EXIT=0.

On main, too, before the merge. At d98df568, tests/test_bootstrap_non_utf8_branch.py (#824) was not in this branch. On a local merge of that head with main d8b8eb33 (clean: Auto-merging scripts/family.py), that file, main's tests/test_readme_windows_quoting_sentence.py, the PR's three test files, test_update_shape_partial_rollback.py, test_git_output_decoding.py and test_agents_md_codes.py gave 293 passed, EXIT=0. The branch now holds main, and the gate under "Merge of main" runs each of those files.

Windows

CI is Linux only (#597), and none of this was run on Windows. What follows is judged from reading, not measured.

• A branch ending in U+00A0, U+2028 or U+0085. Git for Windows keeps a loose ref as a file and converts its UTF-8 name to UTF-16 for NTFS and back. NTFS takes all three in a name, and Win32 trims only a trailing space or dot. So such a branch is a branch there as on Linux, symbolic-ref prints its UTF-8 bytes, and the three reads keep it.
  • What a console then shows of the push line is the console's business. cp1252 holds U+00A0. Under a redirected cp1252 stdout, U+2028 and U+0085 cannot be written, so escape_what_stdout_cannot_encode writes them as \u2028 and \x85, and so does a code page without U+00A0 for it (\xa0). A pasted escaped name names another branch: the push fails src refspec ... does not match any and sends nothing. POSIX's C locale escapes them the same way.
  • That is already how every name outside the stream's encoding prints, and this PR does not change it.
  • family.py's NEXT line is in PowerShell single quotes there, which take each character literally.
• A branch named with a byte that is not UTF-8. HEAD's own file holds ref: refs/heads/caf<0xE9> as bytes, so symbolic-ref prints the byte however the ref store keeps it, and the three tools refuse it as on Linux.
  • By r765's reading of Git for Windows' path conversion (xutftowcsn), a byte from 0xA0 to 0xFF that is not UTF-8 maps to U+00A0 to U+00FF, so the loose ref is the file caf<U+00E9>. git branch -m then renames the branch.
  • No command line can hand Git for Windows the byte (CreateProcessW carries UTF-16), so such a branch most likely arrives only in a repository copied from another system.
  • _as_git_argument hands the name over as it is on Windows, where git takes UTF-16.
  • The new cases skip on Windows, where os.fsencode makes no byte of the surrogate, and the rollback case is WINDOWS_SKIP.

Overlaps

• Lane 4's family.py: bump fetches every branch of the member's origin whatever the mount's refspec and drops the origin/ refs no ref of origin backs, says a tag of the remote reaches a commit no branch does instead of "a pin the rest of the world cannot fetch", and tells a mount with no origin it has none instead of to fix the network (#686) #795, which edited scripts/family.py's bump fetch, landed as 6f64eea. This PR's only family.py hunk is _holder_branch, so the two touch different lines, and d98df568 merged with main 112eeed2 without a conflict. The conflicts the merge with main d6071c7d resolved came from family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766, in two test files (see "Merge of main").
• The issue also names l3-676-pins-merged-away and l3-683-partial-clone as editing update-shape.py. This PR's update-shape.py hunks are current_branch, _symbolic_head, the new _as_git_argument, one line in Rollback._undo_branch, and one line, with its comment, in cmd_apply's if args.branch: block.
• tests/conftest.py gains three helpers and two constants after skip_where_a_name_cannot_be_not_utf_8. No existing line changes.

Not in this PR

• scripts/repo_shape.py, which is a pinned shape copy. bootstrap.py and repo_shape.py: a leg on a branch whose name is not UTF-8 makes every later make bootstrap end in a UnicodeDecodeError, and a character the filesystem encoding lacks still ends step (a) in a traceback #710's ruling (Option 2, landed in bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 as 0dee22b) keeps git_out's strict decode and its strip(), so family.py's _member_head, which reads a MEMBER's HEAD through it for a rollback, is unchanged; it goes to Tracking: adopt-project.py, converting an existing repository in place #457's checklist.
• bump-leg.py's leg_placement, which reads the LEG's own HEAD for its rollback, still uses errors="replace" and strip(), and restore_leg passes that branch through checked_value, which takes ASCII only. It is not the root's branch, which is what the issue's lines name. It goes to Tracking: adopt-project.py, converting an existing repository in place #457's checklist.
• No document names these codes, and --help is unchanged, so docs/cli.md is unchanged.

Known limits

• update-shape.py now refuses runs that main completed. On main, apply --branch <new> from HEAD on caf<0xE9> committed and pushed the new branch, and one with nothing to do said nothing to do, exit 0, since neither names the branch HEAD started on unless a rollback runs. Every apply --branch run on such a branch is refused now, exit 2, as the coordinator decided for all three tools: refuse a name the tool cannot spell, before anything is written. apply without --branch names no branch and runs as on main. The way out is one git branch -m.
• family.py's dry runs refuse late. They ask _holder_branch last, at their branch line, so a dry run on such a branch prints its plan and then refuses, exit 2 (bump --dry-run has already fetched in the member, as it always does). Their real runs refuse before anything is fetched or written. Keeping the hunk inside the branch read is why the dry run does not report it as a branch line, as it does the tracking branch. No test holds the dry run's refusal.
• U+2028 and U+0085 are printed raw on a UTF-8 stream. A push line that names feat<U+2028> or feat<U+0085> names it exactly, inside single quotes, so the character reaches the terminal as it is, and a viewer that breaks lines there shows the line broken. That was already true of bump-leg's line since bump-leg.py: the branch is read in full and pushed as refs/heads/<b>:refs/heads/<b>, dirty names are spelled, committed lists what git holds, and the dry run says it runs no validator (#672) #746.

Closes #765

🤖 Generated with Claude Code

Summary by Sourcery

Read branch HEAD references as Git-written bytes and refuse branch names that cannot be represented as UTF-8 before modifying repository state.

New Features:

• Preserve Git branch names accurately across update-shape, family, and bump-leg operations, including names containing Unicode whitespace.

Bug Fixes:

• Reject non-UTF-8 branch names before any fetch, write, commit, or push, and provide a safe rename remedy.
• Restore rollback checkouts using the exact branch bytes across locales.

Enhancements:

• Report invalid branch names by their original bytes and safely format affected repository paths.
• Expand regression coverage for Unicode branch names, invalid UTF-8 names, rollback behavior, and refusal-state preservation.

Tests:

• Add end-to-end tests covering branch preservation, invalid-name refusals, safe path rendering, and locale-independent rollback behavior.

[sourcery-ai]

Reviewer's Guide

The PR makes all three tools derive the current branch from Git’s raw symbolic-ref bytes, preserving branch names exactly and refusing names containing undecodable UTF-8 before they can produce invalid push or rollback commands; it also adds early update-shape validation and comprehensive byte-level regression tests.

Sequence diagram for byte-preserving branch validation

sequenceDiagram
    participant Tool
    participant Git
    participant Validator
    participant Workspace

    Tool->>Git: symbolic-ref --quiet HEAD
    Git-->>Tool: raw ref bytes
    Tool->>Validator: decode with surrogateescape
    Validator->>Validator: branch.encode(utf-8)
    alt branch name is valid UTF-8
        Validator-->>Tool: current branch
        Tool->>Workspace: fetch, write, commit, or print push line
    else branch name is not UTF-8
        Validator-->>Tool: Refusal
        Tool-->>Workspace: no fetch, write, or commit
    end

Loading

File-Level Changes

Change	Details	Files
Read symbolic branch refs byte-for-byte and reject branch names that cannot be represented as UTF-8.	Replace text-mode symbolic-ref reads and whitespace stripping with byte reads that remove only Git’s trailing newline and the ASCII refs/heads/ prefix. Decode branch bytes with surrogateescape so Unicode whitespace and other valid branch-name characters are preserved exactly. Raise tool-specific refusals for surrogate-containing names, including a git branch-m rename remediation.	scripts/bump-leg.py scripts/family.py update-shape.py
Ensure update-shape validates the checked-out branch before any apply writes occur.	Validate the current branch at the start of --branch apply runs, preventing partial writes and rollback failures for unspellable names. Preserve rollback-safe behavior when symbolic-ref cannot be read or Git reports a detached/refused HEAD.	update-shape.py
Add regression coverage for exact branch preservation and non-UTF-8 refusal behavior.	Test branches ending in U+00A0 and related whitespace characters through push and branch-selection paths. Test refusal of non-UTF-8 branch names in real and dry-run modes, including exit codes, messages, remediation, and no-mutation guarantees. Add shared fixtures for constructing non-UTF-8 branches and byte-level repository state snapshots; isolate update-shape remotes.	tests/conftest.py tests/test_bump_leg_branch_and_push_line.py tests/test_family_tracking_branch.py tests/test_update_shape_apply_lines_and_headers.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#765	Read the current branch name in update-shape.py, family.py, and bump-leg.py without stripping Unicode whitespace or otherwise changing the branch’s bytes.	✅
#765	Refuse branch names that are not UTF-8, identifying the branch by its Python spelling and doing so before a commit or write can produce a push or recovery line naming a different branch.	✅

Possibly linked issues

• update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read with .strip(), so a branch ending in U+00A0, U+2028 or U+0085 is read as its neighbour, and bump-leg.py (and update-shape.py) decode it with errors="replace", so the push line names a branch that is not there #765: The PR implements the issue’s requested byte-preserving reads and pre-write refusals in update-shape.py, family.py, and bump-leg.py.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

(lane openRepoShape-1, 2026-10-17T23:07:15Z) Lane 1's independent review of #849 at 25692a3: CHANGES NEEDED

Read-only review by r765 (lane openRepoShape-1), against main 677f9bfa.

• The merge: the local-only merge of 25692a37 with that main is clean (Auto-merging scripts/family.py, tree 553730f7).
• Probes: a stub gh first on PATH, GH_TOKEN=bogus-r765, an empty GH_CONFIG_DIR and GIT_CONFIG_GLOBAL=/dev/null, on git 2.43.0 and Python 3.12.3.

Findings, ranked

1. BLOCKING: CI tests is red at the head, because _holder_branch now quotes for the host and README's sentence does not name it

Run 39063562770, job tests: 1 failed, 12262 passed, 145 skipped, 1 xfailed.

FAILED tests/test_readme_windows_quoting_sentence.py::test_the_functions_that_quote_for_the_host_are_the_ones_the_sentence_names[scripts/family.py]
AssertionError: scripts/family.py's functions that call a host-dialect quoter are ['_add_command', '_checked_mount_path', '_holder_branch', '_print_next', '_print_next_steps', '_switch_command'], and the sentence names ['_add_command', '_checked_mount_path', '_print_next', '_print_next_steps', '_switch_command']

• Cause: the family-branch-not-utf8 remedy uses shell_line([...]), which quotes for PowerShell on Windows. README's "Windows, natively" says only family.py's NEXT, switch -c, add and git rm -r --cached lines are quoted that way.
• Why the writer missed it: the writer's 28-file gate did not include this file.
• Repro: pytest tests/test_readme_windows_quoting_sentence.py
  • main: 40 passed;
  • head: 1 failed, 8 passed;
  • merge: 1 failed, 39 passed. Main's newer inventory counts uses and reports {'_holder_branch': 1, ...}.

Fix A (recommended). One line and no README change. All three tools then quote the branch -m remedy for a POSIX shell, as bump-leg's shlex.join and update-shape's _by_hand already do.

-            + shell_line(["git", "-C", root, "branch", "-m"])
+            + shell_line(["git", "-C", root, "branch", "-m"], "posix")

On the merge, the quoting test and test_family_tracking_branch.py then give 61 passed.

Fix B. Keep host quoting. Add branch -m to the sentence in README and in the handbook, and add "_holder_branch": 1 to NAMED. That waits for lane 3's README turn.

2. SHOULD FIX: under LC_ALL=C PYTHONUTF8=0, update-shape's rollback from feat<U+00A0> now raises part way through

Cause.

• Rollback._undo_branch hands start back to git through subprocess. start is now feat<U+00A0>, read as UTF-8.
• subprocess writes each argument with os.fsencode, and an ascii encoding cannot write U+00A0.
• Main read the name as feat, so it did not raise; it went wrong another way (finding 3).
• bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 fixed this same class in bootstrap.py, by decoding so that a name reaches git again as the same bytes.

Repro. HEAD on feat<U+00A0> beside feat, with a .git/hooks/pre-commit that exits 1, then LC_ALL=C PYTHONUTF8=0 update-shape.py apply ... --branch shape/update-partial.

• main: exit 2, every byte put back, status clean, but HEAD on feat.
• merge: exit 1, UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 4 from _git_quietly.
  • HEAD is left on refs/heads/shape/update-partial, and that branch is left.
  • M AGENTS-shape.md, M contracts/shape-pin.yaml, M project.yaml and M scripts/validate-manifest.py are not put back.
• A branch <U+65E5><U+672C> raises the same way on main and on the merge, so that case was already there.

_git_quietly's own contract says: "Never raises ... a rollback that raised part way would leave the tree half put back".

The fix:

def _as_git_argument(name: str) -> str:
    """`name` as `_symbolic_head` decoded it, as the argument that hands
    git the same bytes back under any locale (#710's rule). Windows hands
    git UTF-16, so the name goes as it is there."""
    if os.name == "nt":
        return name
    return os.fsdecode(name.encode("utf-8", "surrogateescape"))

In _undo_branch, the call becomes _git_quietly(["git", "checkout", "-q", _as_git_argument(start), "--"] if start else ...). The printed _by_hand lines keep start as it is.

Verified on the merge.

• Both names exit 2, with HEAD and status as they were, in both locales.
• test_update_shape_partial_rollback.py, test_update_shape_apply_lines_and_headers.py, test_git_output_decoding.py and test_update_shape_kept_helper.py give 237 passed.

If the coordinator rules this out of scope, it needs Known limit text 4 below.

3. TEST GAP: no test holds the Rollback caller of _symbolic_head, which this PR fixes

Same repro as finding 2, in the default locale:

• main: ROLLED BACK: every byte this command wrote has been put back, and the tree is exactly as it was: then switched HEAD back to feat. HEAD went from refs/heads/feat\xc2\xa0 to refs/heads/feat, the neighbour.
• merge: switched HEAD back to feat\xa0, with HEAD's bytes unchanged. U+2028 behaves the same.

Mutant R1 changes Rollback.new_branch to start = _branch_of((_symbolic_head(self.root) or "").strip()). With it, test_update_shape_apply_lines_and_headers.py and test_update_shape_partial_rollback.py give 62 passed: no test fails.

This case holds findings 2 and 3. It belongs in tests/test_update_shape_apply_lines_and_headers.py, which already imports every name it uses:

@WINDOWS_SKIP
@pytest.mark.parametrize("env", [{}, {"LC_ALL": "C", "PYTHONUTF8": "0"}],
                         ids=["locale", "c-locale"])
def test_a_rollback_puts_head_back_on_the_branch_it_started_on(
        standard, root, env):
    git("branch", "feat", cwd=root)
    git("switch", "-q", "-c", "feat" + chr(0xA0), cwd=root)
    hook = root / ".git" / "hooks" / "pre-commit"
    hook.parent.mkdir(exist_ok=True)
    hook.write_bytes(b"#!/bin/sh\nexit 1\n")
    hook.chmod(0o755)
    before = held_by_git(root)

    result = apply(standard, root, "b", "--branch", BRANCH, env=env)

    assert result.returncode == 2, result.stdout + result.stderr
    assert held_by_git(root) == before

Measured:

• on the merge as it is, [c-locale] fails;
• with Fix 2: 2 passed;
• with Fix 2 and R1: 2 failed.

4. MINOR: two docstrings promise more than their asserts hold

• bump-leg's new case says "before the fetch", and family's says "before the member is read".
• But held_by_git snapshots only the root or the holder. A fetch lands under .git/modules/, which neither snapshot shows.
• The code order is right.
• Either snapshot the leg's or member's refs too, or drop those words.

The coordinator's three rulings

1. The early current_branch(root) in cmd_apply stays.
   • It is present at update-shape.py:4612, and How states it ("the one hunk outside a branch-read helper").
   • Row U3, re-run, fails as claimed.
2. update-shape refusing a run that main completed. Known limits states it, but incompletely: a --branch run with nothing to do is refused too.
   • main: exit 0, nothing to do: every copied file matches ...
   • merge: exit 2, REFUSED update-branch-not-utf8
   • apply without --branch: exit 0 on both.
   • Text 3 below covers it.
3. Family's dry runs refuse late. Known limits states this, and it is true, so no change is needed. On the merge:
   • add, bump and remove --dry-run printed 20, 15 and 5 stdout lines;
   • none printed the closing --dry-run: nothing was changed paragraph;
   • each then printed REFUSED family-branch-not-utf8 and exited 2.

Attack points

Line 1

Each of U+00A0, U+2028 and U+0085 beside feat, with each NEXT line run as printed:

• family remove:
  • main prints push -u origin refs/heads/feat:refs/heads/feat for all three. Run, it pushed feat at df059cf9, not the act's commit.
  • The merge prints 'refs/heads/feat<c>:refs/heads/feat<c>'. Run, origin's feat<c> is the holder's HEAD, and feat is absent.
• bump-leg: right on main (bump-leg.py: the branch is read in full and pushed as refs/heads/<b>:refs/heads/<b>, dirty names are spelled, committed lists what git holds, and the dry run says it runs no validator (#672) #746) and on the merge.
• update-shape apply --branch feat --push from feat<c>:
  • main: exit 0, and origin got refs/heads/feat.
  • merge: exit 2, REFUSED update-branch-exists: ... ('feat\u2028' is checked out) (likewise 'feat\xa0' and 'feat\x85'), and origin is empty.

Line 2

• caf<0xE9>: the six new cases hold it.
• git branch -m cafe renames such a branch.

Every caller

• update-shape current_branch: commit_on_branch, cmd_apply and _awaits_its_pull_request.
• update-shape _symbolic_head: current_branch, Rollback.new_branch, _undo_branch and _head_is.
• family _holder_branch: _checked_branch and _branch_said.
• bump-leg current_branch: _branch_to_bump_from.
• Nothing else imports them. Two callers are untested: Rollback (finding 3) and the family dry-run refusal.

Ordinary inputs

• feat, shape/update-x, v1.0.0, a+b, <U+65E5><U+672C>, release/1.2 and a detached HEAD give byte-identical answers from all four reads on main and on the merge.
• There is no new network call.

#824 (0dee22b) does not refuse a non-UTF-8 branch

• bootstrap.py reads HEAD and the upstream as bytes, decodes them with os.fsdecode (UTF-8 on Windows), and hands git the same bytes back, so a leg on a<0xFF>b is moved back.
• Its not-tried line covers a NUL, a lone surrogate, or a character the encoding cannot write. It has no code, no exit 2, and says "Give that one a name git can create".
• So there are no codes to align: bootstrap prints no line a person pastes.
• Two things do carry over:
  • its decode rule, which is finding 2;
  • the sibling code with the same suffix, trailer-not-utf8 (in shape_materialize.py, raised by all three tools). It names its value by its bytes, b'Jos\xe9', where update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849 prints 'caf\udce9'.
• Optional: {branch.encode('utf-8', 'surrogateescape')!r} would print b'caf\xe9' in all three, with the tests changed the same way.

#795 (6f64eea)

The merge is clean, and the 28 family test files pass on it.

AGENTS.md and docs/cli.md

• No test requires the new codes in either.
• test_agents_md_codes.py holds AGENTS.md's codes to the tools, not the reverse.
• AGENTS.md names neither family-holder-detached, bump-leg-root-detached nor update-branch-exists.
• --help is unchanged, and test_cli_reference.py passes.

CI

• SonarCloud: 0 new issues, 0.0% duplication.
• tests-git-bash and package are green.

Gate: local merge 8210508c, dirty 0, git-filter-repo on PATH

Files:

• the PR's three test files;
• test_shape_doctor.py, test_scaffold_leg_path_collisions.py and test_cli_reference.py;
• all 28 test_family*.py;
• test_agents_md_codes.py, test_git_output_decoding.py, test_non_utf8_git_reads.py, test_update_shape_partial_rollback.py and test_bootstrap_non_utf8_branch.py.

Result: 1,985 passed, 22 skipped, 1 xfailed, in six shards, each EXIT=0. Run separately, test_readme_windows_quoting_sentence.py has 1 failure (finding 1).

Revert row, at the head	Result
U3	1 failed: ..._not_utf8_is_refused_first
F1	1 failed: family ..._no_break_space_is_pushed_as_itself
B1	2 failed: bump-leg [bump], [dry-run]
R1 (new)	62 passed (finding 3)

Windows

U+00A0, U+2028 and U+0085.

• NTFS takes them, and Win32 trims only a trailing space or dot, so the loose ref works and the three reads keep it.
• Family's NEXT is in PowerShell single quotes, which take the character literally.
• A redirected cp1252 stdout cannot hold U+2028 or U+0085. escape_what_stdout_cannot_encode then writes \u2028, which, pasted, names another branch: the push fails src refspec ... does not match any and sends nothing. POSIX's C locale does the same.
• The body says this only of U+00A0.

The byte 0xE9 (read, not run).

• Git for Windows' path conversion (xutftowcsn) maps invalid bytes 0xA0-0xFF to U+00A0-U+00FF, so the loose ref is the file caf<U+00E9>.
• HEAD's file keeps the raw 0xE9, which symbolic-ref prints.
• So the tools refuse as on Linux, and branch -m renames the branch.
• The new cases skip on Windows. Their names are NTFS-legal, and no test counts text-mode bytes.

Exact text replacements

1. How, family bullet (with Fix A). Replace Its .strip() is gone. This is the only family.py hunk. with:

   Its .strip() is gone. Its remedy's branch -m line is quoted for a POSIX shell (shell_line(..., "posix")), as bump-leg's and update-shape's are, so README's sentence on the lines quoted for PowerShell stays true. This is the only family.py hunk.

2. How, update-shape (with Fix 2). Add:

   • Rollback._undo_branch hands git the branch HEAD started on through _as_git_argument, so subprocess writes the name's own UTF-8 bytes under any locale, as bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 does for bootstrap.py. Under LC_ALL=C PYTHONUTF8=0, a rollback from feat<U+00A0> raised UnicodeEncodeError part way and left the copies written; one from <U+65E5><U+672C> did so on main too.

3. Known limits, bullet 1. Replace with:

   • update-shape.py now refuses runs that main completed. On main, apply --branch <new> from HEAD on caf<0xE9> committed and pushed the new branch, and one with nothing to do said nothing to do, exit 0, since neither names the branch HEAD started on unless a rollback runs. Every apply --branch run on such a branch is refused now, exit 2, as the coordinator decided for all three tools: refuse a name the tool cannot spell, before anything is written. apply without --branch names no branch and runs as on main. The way out is one git branch -m.

4. Known limits (only if Fix 2 is not taken). Add:

   • Under LC_ALL=C PYTHONUTF8=0, update-shape.py's rollback raises on a branch name that is not ASCII, which it hands back to git as UTF-8 text. From feat<U+00A0> it now raises UnicodeEncodeError part way, leaving the new branch and the copies; main read the name as feat and put HEAD back there. <U+65E5><U+672C> raised on main too.

5. Overlaps, bullet 1. Replace with:

   • Lane 4's family.py: bump fetches every branch of the member's origin whatever the mount's refspec and drops the origin/ refs no ref of origin backs, says a tag of the remote reaches a commit no branch does instead of "a pin the rest of the world cannot fetch", and tells a mount with no origin it has none instead of to fix the network (#686) #795, which edited scripts/family.py's bump fetch, landed as 6f64eea. This PR's only family.py hunk is _holder_branch, so the two touch different lines, and the head merges with main 677f9bfa without a conflict.

6. Not in this PR, bullet 1. Replace with:

   • scripts/repo_shape.py, which is a pinned shape copy. bootstrap.py and repo_shape.py: a leg on a branch whose name is not UTF-8 makes every later make bootstrap end in a UnicodeDecodeError, and a character the filesystem encoding lacks still ends step (a) in a traceback #710's ruling (Option 2, landed in bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 as 0dee22b) keeps git_out's strict decode and its strip(), so family.py's _member_head, which reads a MEMBER's HEAD through it for a rollback, is unchanged; it goes to Tracking: adopt-project.py, converting an existing repository in place #457's checklist.

7. Tests.
   • Add tests/test_readme_windows_quoting_sentence.py to the guards.
   • Add the rollback case to the case table, and R1 to the revert table.
   • Refresh both modes' counts.

For the checklist

• family _member_head reads a member's HEAD for PinRollback through git_out's strict decode and strip(). Read from the source, not measured:
  • a member on feat<U+00A0> beside feat at the same commit is put back on feat;
  • a member on a non-UTF-8 branch raises UnicodeDecodeError as PinRollback starts, since only Refusal is caught.
• bump-leg restore_leg passes the leg's branch through checked_value (SAFE_ARG_RE, ASCII only), so a leg on <U+65E5><U+672C> is refused unsafe-value inside the rollback. That is besides leg_placement's errors="replace" and strip(). Read from the source.
• update-shape under LC_ALL=C PYTHONUTF8=0: any non-ASCII branch reaches finding 2's raise, on main too.
• No test holds the family dry run's late refusal (_branch_said).

Lane 1 takes a fix round on the same branch now: Fix A for finding 1, Fix 2 with the rollback test for findings 2 and 3, and the text replacements. The same reviewer re-verifies.

Lane: openreposhape-1 (openRepoShape-1)

[brettheap]

(lane openRepoShape-1, 2026-10-18T00:48:37Z) Lane 1's re-verification of #849 at d98df56: PR-READY

Read-only re-verification by r765 (lane openRepoShape-1) of w765's fix round 1 (25692a37..d98df568). The local-only merge with main 112eeed2 is clean (Auto-merging scripts/family.py, Auto-merging tests/conftest.py; merge 1e6fecb6, tree 78e8ec83). Probes ran as before: a stub gh, GH_TOKEN=bogus-r765, an empty GH_CONFIG_DIR and GIT_CONFIG_GLOBAL=/dev/null. No code change is needed. It is PR-READY once the two false body sentences under "Texts" are corrected.

The four findings

1. CI red: fixed (Fix A). _holder_branch's remedy is now shell_line([...], "posix").
   • test_readme_windows_quoting_sentence.py on the merge: 40 passed. Before the fix it was 1 failed, 39 passed.
   • CI at d98df568:
     • tests: pass, 12896 passed, 145 skipped, 2 xfailed.
     • SonarCloud: Quality Gate passed, 0 new issues, 0.0% duplication.
     • package and tests-git-bash: pass.
2. C-locale rollback: fixed (Fix 2). Run on the merge under LC_ALL=C PYTHONUTF8=0, with a pre-commit hook that exits 1, as apply ... --branch shape/update-partial:
   • From feat<U+00A0>:
     • exit 2;
     • HEAD is refs/heads/feat\xc2\xa0 before and after, and held_by_git is equal;
     • the rollback prints switched HEAD back to feat\xa0;
     • before the fix it was exit 1 with UnicodeEncodeError, and the copies were left written.
   • From <U+65E5><U+672C>:
     • exit 2;
     • HEAD is refs/heads/\xe6\x97\xa5\xe6\x9c\xac before and after, and held_by_git is equal;
     • on main it was exit 1 with a traceback.
3. Rollback test gap: closed. test_a_rollback_puts_head_back_on_the_branch_it_started_on[locale] and [c-locale] are in. Rows R1, R2 and U1 fail them (below).
4. Snapshots: they now hold what the docstrings say. For each git directory under .git/modules, held_by_git now adds its refs, count-objects -v, and FETCH_HEAD's bytes and write time.
   • Measured on the merge, on an ordinary branch feat:
     • bump-leg --dry-run: keys attached, code, head, index, refs, spec, status; changed ['spec'];
     • family bump --dry-run: changed ['members/IRRS'].
   • So a refusal moved after the fetch would fail both cases.
   • Family's claim that "add prints not even the line that says the member is being read" holds by stdout == "": checking ... before mounting it is printed and flushed before _probe_member.

The bytes form

• Consistency with trailer-not-utf8:
  • All three refusals now read ... is on the branch b'caf\xe9', whose name is not UTF-8 .... That is the same b'...' form as trailer-not-utf8 (b'Jos\xe9').
  • The bytes come from raw in bump-leg and family, and from branch.encode("utf-8", "surrogateescape") in update-shape. Both are the exact inverse of the decode.
  • trailer-not-utf8 uses os.fsencode, which gives the same bytes on POSIX.
• The remedy, pasted, still works in all three:
  • each tool's remedy line was run through sh -c, with <a new name> as cafe: rc 0, and HEAD is refs/heads/cafe;
  • bump-leg --dry-run and family remove --dry-run then exit 0.

The writer's claim: only _undo_branch hands a branch it read back to git

This is true of the branches that this PR's three reads return.

• What update-shape does with the answers of current_branch and _symbolic_head:
  • compares them (commit_on_branch, _awaits_its_pull_request, _head_is);
  • stores them (new_branch);
  • prints them (_by_hand, the refusals);
  • checks them out in _undo_branch, now through _as_git_argument.
• update-shape's other ref read, for-each-ref --points-at HEAD in the landing failure, is only compared.
• The answers of family's _holder_branch and bump-leg's current_branch are only printed.

It is not true of the tools as a whole, which is how the body's What section states it. Two rollbacks hand git a branch they read themselves:

• scripts/family.py: _member_head reads through git_out (strict UTF-8 and strip()). The branch goes to PinRollback, then _head_back, then _offline_git(["checkout", "-q", *_head_target(...)]), which runs checkout <branch> -- in the member.
• scripts/bump-leg.py: leg_placement reads with errors="replace" and strip(). The branch goes to restore_leg, which runs run(["git", "checkout", "-q", branch]) in the leg, after checked_value, which takes ASCII only.

Both read a member's or a leg's HEAD, not the root's, so both are outside #765. Both are already under "Not in this PR" and in the checklist. Only the What sentence needs fixing (text 1). I read this from the source and did not measure it.

Revert rows re-run at d98df56

Each change was reverted alone in a scratch clone.

Row	Result
FA: host quoting back	1 failed: the quoting test, [scripts/family.py]
R2: no _as_git_argument	1 failed: rollback [c-locale]
U1: _symbolic_head .strip() back	3 failed: update-shape no-break space, rollback [locale] and [c-locale]
BN: bump-leg {branch!r} back	2 failed: bump-leg not UTF-8, [bump] and [dry-run]

With nothing reverted, the same selection gives 30 passed. All four rows match w765's table.

Gate: merge 1e6fecb6, dirty 0, git-filter-repo on PATH

The files:

• the PR's three test files;
• test_shape_doctor.py, test_scaffold_leg_path_collisions.py and test_cli_reference.py;
• all 29 test_family*.py;
• test_agents_md_codes.py, test_git_output_decoding.py, test_non_utf8_git_reads.py, test_update_shape_partial_rollback.py, test_bootstrap_non_utf8_branch.py and test_readme_windows_quoting_sentence.py;
• w765's guards: test_validator_child_encoding.py, test_update_shape_kept_helper.py, test_doctor_next_is_a_command.py, test_openreposhape_package.py, test_setup_project_py.py, test_ruleset_hint_tracking_branch.py, test_template_values_single_pass.py, test_pinned_copy_text.py, test_refusal_default_remediation.py and test_repo_hygiene.py.

Result: 2,999 passed, 26 skipped, 1 xfailed, in seven shards, each EXIT=0.

Texts

The body:

• is 21,095 characters by len();
• has the Lane, Claim and v1.1.1 lines, Closes #765 and the robot line;
• holds no non-ASCII character except the robot face.

Every other sentence I checked is true at the head, including the Known limits for the rulings, the revert table, and the Why and Windows sections. Two sentences are false:

1. What, last bullet. Replace

   update-shape.py's Rollback._undo_branch is the one place any of the three tools hands a branch it READ back to git, and it now goes through _as_git_argument, as bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 does for bootstrap.py.

   with

   update-shape.py's Rollback._undo_branch is the one place a branch returned by one of these three reads goes back to git, and it now goes through _as_git_argument, as bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 does for bootstrap.py. Two rollbacks outside this PR hand git a branch they read for themselves: a member's in family.py (_member_head, then _head_back) and a leg's in bump-leg.py (leg_placement, then restore_leg). Both are under "Not in this PR".

2. Overlaps, bullet 3. Replace gains four helpers and two constants with gains three helpers and two constants.

   • The helpers are git_answer_bytes, held_by_git and on_a_branch_that_is_not_utf_8.
   • The constants are NOT_UTF_8_BRANCH and NOT_UTF_8_BRANCH_NAMED.

3. Optional, Overlaps bullet 1. merges with main d8b8eb33 without a conflict is still true. The merge with main 112eeed2 is clean too (measured), if you want the newer sha.

A nit that does not block: one docstring line in tests/test_family_tracking_branch.py is 103 characters long (escape, which name another branch. Refused by name now, by its bytes, before anything is fetched or). Rewrap it the next time the file is touched.

For the checklist

• family _member_head -> _head_back. Besides the strip() and the strict decode, a member on a non-ASCII UTF-8 branch is handed back to git as text under LC_ALL=C PYTHONUTF8=0. That is the class Fix 2 closed in update-shape. Read from the source.
• bump-leg restore_leg. checked_value takes ASCII only, so a non-ASCII leg branch is refused unsafe-value inside the rollback.
• No test holds the family dry run's late refusal (_branch_said).

Lane 1 applies texts 1 and 2 to the body now, and names the newer main. The docstring nit is non-blocking. #849 joins lane 1's landing turns.

Lane: openreposhape-1 (openRepoShape-1)

[brettheap]

(lane openRepoShape-1, 2026-10-18T05:24:59Z) # r765m: re-check of the merge round of PR #849 (issue #765) at da3fe1c8

Verdict: PR-READY. No code change is needed.

• Optional wording fix: one body sentence, in "Fix round 1", is now out of date. See below.
• CI: the tests job is still running. It is the only check not green. The earlier run at d98df568 took 64 minutes and passed. This one started at 04:50:23Z, so it should finish around 05:55Z.

1. git show --remerge-diff da3fe1c8

• The merge has parents d98df568 and d6071c7d (git log -1 --format=%P).
• The diff touches four files: scripts/bump-leg.py, scripts/family.py and the two test files.
• Three conflict hunks, all from family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766 (5fc049b7):
  • tests/test_bump_leg_branch_and_push_line.py, module docstring:
    • Both paragraphs are kept verbatim, update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's first.
    • One clause is added to family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766's: "and so does update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read with .strip(), so a branch ending in U+00A0, U+2028 or U+0085 is read as its neighbour, and bump-leg.py (and update-shape.py) decode it with errors="replace", so the push line names a branch that is not there #765's bump-leg-branch-not-utf8, raised in the same check". It is true.
    • "THE ROOT ABOVE THEM" still reads correctly after update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's paragraph.
  • Same file, conftest imports:
    • update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's import line is kept whole, and family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766's second import line is kept.
    • Main's first line (FILE_PROTOCOL, PROJECT, REPO, cloned_just_pushed, git) is dropped. Every name in it is in update-shape.py, family.py and bump-leg.py: the branch HEAD is on is read as the bytes git wrote, so a branch ending in U+00A0 is no longer read as its neighbour, and a name that is not UTF-8 is refused by name before anything is written (#765) #849's line.
  • tests/test_family_tracking_branch.py, conftest imports:
    • Both kept. SHOWN_NAMES and named_as are added to family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766's line for the new row.
• Imports: an AST check finds no unused conftest import in either file.
• Docstring rewrap in the family test: the words are identical (a.split()==b.split() is True), and the longest line went from 103 to 74 characters.
• Nothing else changed meaning. The remerge diff has no other hunk. tests/conftest.py is not in it. The two scripts show only the shown_path change and its comment.
• No new long lines. scripts/bump-leg.py has 13 lines over 79 and scripts/family.py has 20, the same counts as main d6071c7d.

2. The shown_path change

• Applied in both tools:
  • bump-leg: current_branch raises bump-leg-branch-not-utf8, and _branch_to_bump_from is its only caller. The detail is now {shown_path(root)}, as the refusals beside it have it.
  • family: _holder_branch raises family-branch-not-utf8. It is called by _checked_branch (real run) and _branch_said (dry run), so both get the change.
  • shown_path is imported in both files (bump-leg.py:124, family.py:204).
• update-shape.py needs no change:
  • It has no shown_path anywhere, and family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766 did not touch it.
  • It names the root raw elsewhere too (root {root}, update-root-missing).
  • It is byte-identical across the merge: git diff d98df568 da3fe1c8 -- update-shape.py is empty.
• The branch -m remedy is right for a POSIX shell:
  • It keeps the path raw inside shell quotes, as _switch_command does. Both docstrings say that is the design.
  • I put a temporary probe test in my scratch copy, for ESC, line break and printable roots.
  • Each remedy was cut from stderr, <a new name> replaced with cafe, and run with sh -c.
  • Each ran with exit 0 and HEAD became refs/heads/cafe. The refusal was gone on the next run.
  • Result: 78 passed for the two tools together. The probe files are removed.
• Revert proof, in a scratch worktree at da3fe1c8. The two new tests are the ones named ..._on_a_branch_that_is_not_utf8_is_named_safe_to_show:

Reverted	Result
nothing	6 passed
both tools back to {root}	4 failed, 2 passed: [ESC] and [line-break] fail in both
bump-leg only	2 failed, 4 passed ([ESC], [line-break])
family only	2 failed, 4 passed ([ESC], [line-break])
restored	6 passed; git diff empty, git status --porcelain empty

• Bytes-level check of the new tests:
  • They assert exit 2, an empty stdout, and the whole detail line.
  • They compare against spelt, which is repr(str(path)), not against shown_path, so the oracle is independent.
• Writer's three revert rows re-run at da3fe1c8 on the selection of 12 old cases plus 6 new rows:

Row	Result	Writer's table
none	18 passed	18
F1: .strip() back in _holder_branch	1 failed (family no-break space)	1
B3: .strip() back in bump-leg current_branch	2 failed (beside a branch, beside the tracking branch)	2
U1: .strip() back in _symbolic_head	3 failed (no-break space, rollback [locale], [c-locale])	3

3. Targeted gate on a local-only merge

• Merge: da3fe1c8 with origin/main 33419925, the current main when I fetched.
  • Merge commit 246eb9ed, tree c34d4048, the same tree the writer reported.
  • merge-tree --messages printed Auto-merging scripts/bump-leg.py and Auto-merging update-shape.py, with no conflict.
• Worktree and files: the worktree was under _lane1-scratch/r765m/ and is removed. The 44 files:
  • every test_family*.py (31) and test_bump_leg*.py (8);
  • the PR's three test files, which are among those;
  • test_update_shape_apply_lines_and_headers.py, test_agents_md_codes.py, test_readme_windows_quoting_sentence.py, test_shape_doctor.py and test_cli_reference.py.
• How: each file in its own pytest run, 5 shards, /usr/bin/python3 -m pytest -p no:cacheprovider, stub gh first on PATH, GH_TOKEN=bogus-r765m, an empty GH_CONFIG_DIR, GIT_CONFIG_GLOBAL=/dev/null, GIT_SSH_COMMAND=false. GIT_ALLOW_PROTOCOL was never set.

Mode	Result	Shard EXIT	File exits
git-filter-repo on PATH (/home/brett/.local/bin/git-filter-repo)	2,048 passed, 27 skipped, 0 failed	all 0	all 0
hidden (PATH=<stub gh>:/usr/local/bin:/usr/bin:/bin, command -v git-filter-repo rc 1)	2,043 passed, 32 skipped, 0 failed	all 0	all 0

• Hidden mode: the 5 extra skips are 4 in test_shape_doctor.py and 1 in test_update_shape_apply_lines_and_headers.py. These match the body's "14 + 4 + 1" split. The 14 is test_update_shape_merged_away.py, which is outside my 44 files.
• Dirty state: every shard log records DIRTY=0 at HEAD=246eb9ed.
• Match with the writer: the per-file counts of all 44 files are identical to the writer's logs at da3fe1c8 in both modes. So main 33419925 changed no result in these files.
• The body's "201 passed": the five files named in the body give 44 + 28 + 42 + 40 + 47 = 201 passed. This agrees with the body.
• Main moved again. It is now 3bf2d3c9 (5c54af20, 3bf2d3c9).
  • Neither commit touches a PR file.
  • merge-tree with it is clean (Auto-merging scripts/bump-leg.py, Auto-merging update-shape.py).
  • I did not re-run the gate on it.
  • The shape-doctor.py change in 3bf2d3c9 is main's own business.

4. CI at da3fe1c8

gh pr view 849 reads head da3fe1c8, draft, merge state UNKNOWN at the last read. The checks:

Check	State
SonarCloud Code Analysis	pass (1m11s)
package	pass (32s)
tests-git-bash	pass (2m14s)
tests	pending

• SonarCloud:
  • The issues API (pullRequest=849&resolved=false) returned "total":0,"issues":[].
  • The quality gate is OK: new_duplicated_lines_density is 0.0, the ratings are 1, and security hotspots reviewed are 100.0.
• tests:
  • Run 39098477406 has headSha da3fe1c8. The job was in step 7, "tests", at 05:23:54Z.
  • It had run 33 minutes at my last read, and I did not wait for it.
  • The earlier run 39080013572 at d98df568 took 64m25s and passed.

5. The body

The posted body equals pr765-body.md: 28,135 characters, and the only non-ASCII character is the robot face. Everything in "Merge of main", and every line it dates "before the merge", checks out against the head:

• Merge facts: da3fe1c8, parents d98df568 and d6071c7d, nothing squashed or rebased. Three hunks in two files, all from family.py, scaffold-project.py and bump-leg.py: init's plan rows and seven refusals name a holder, a folder, a lock, a remote path or a root as it is, so an ESC, a line break or U+202E in a path reaches the terminal as #692's refusals no longer let it #766 (family.py, scaffold-project.py and bump-leg.py: init's plan rows, seven refusals and bump-leg's root name each path as shown_path names it, so an ESC or a line break in a holder, folder, lock, local remote or root no longer reaches the terminal from those plan rows or those refusals' details #814). The table rows and the "both" resolutions match the remerge diff.
• Unchanged files:
  • Main d6071c7d against fc867207 leaves update-shape.py and tests/test_update_shape_apply_lines_and_headers.py unchanged.
  • The +/- lines of those two files and of conftest.py have identical md5s across d98df568–fc867207 and da3fe1c8–d6071c7d.
  • scripts/bump-leg.py, scripts/family.py and tests/conftest.py auto-merged, so the body is right that they needed no hand-merge.
• The change in meaning:
  • The refusals' {root} was raw, and their docstrings say the others use shown_path. I read both docstrings.
  • Printable paths are unchanged.
  • The branch -m remedy is unchanged and keeps the path raw, as switch -c does.
  • update-shape.py uses no shown_path.
• New tests and reverts:
  • The two new test names match.
  • "2 failed, 4 passed" holds, as measured above.
  • F1, B3 and U1 match.
  • "12 cases and 6 new rows (18 passed)" holds.
• Gate counts:
  • The family, bump-leg and update-shape counts (31, 8 and 18) are right at da3fe1c8 (d98df568 had 27 family files).
  • The 66-file list is 31+8+18+9, as stated.
  • From the writer's logs: mode 1 is 3,026 passed and 32 skipped, and mode 2 is 3,007 passed with 37 logged skips plus the 14 from the re-run (14 skipped, in m2-rerun-merged-away.out) = 51.
  • Every shard logs HEAD=da3fe1c8, DIRTY=0 and EXIT=0.
  • The skip breakdown matches the log's SKIPPED lines: 19, 4, 3, 3, 2 and 1.
• Main moved: 75d70502 (repo_shape.py's YAML reader: Unicode spaces are stripped from text YAML keeps, the empty block key and k: {?: 1} still read, and three texts about the reader are one word out of date (what #635 left) #715) changes only Project.tracking in bump-leg and tracking_branch_of in update-shape, as the body says. My merge reproduces tree c34d4048 and the two Auto-merging lines.
• Other statements:
  • "This is the only family.py hunk" holds (1 hunk against main).
  • conftest gains 58 lines, 0 deleted, three helpers and two constants.
  • update-shape has 4 hunks, unchanged by the merge.
  • No document names the new codes (git grep outside tests and tools finds none).

One out-of-date sentence, optional. "Fix round 1", line 29, ends: "The remedy is shell_line([...], "posix") now; the file gives 9 passed."

• That was true at d98df568. At the head tests/test_readme_windows_quoting_sentence.py gives 40 passed, which is its count in my gate.
• The "(9 tests)" in the revert-table line is already dated "before the merge", so it is fine.
• It is not a false claim about the PR, only a stale number in a history section. If you edit the body, replace the file gives 9 passed. with the file gave 9 passed then (it holds 40 tests at the head, all passing).

Checks I could not make

• I did not run the full suite or any file outside the 44. Only tests in CI can speak for the rest.
• I measured nothing on Windows.

Files:

• /workspace/projects/openRepoShape-worktrees/_lane1-scratch/r765m/logs/ holds the gate logs (m1-s*.log, m2-s*.log).
• /workspace/projects/openRepoShape-worktrees/_lane1-scratch/r765m/gate.sh and pt.sh are the scripts I used.
• Both scratch worktrees are removed with git worktree remove --force. Nothing was pushed, posted or committed to a shared branch. The shared checkout still shows only ?? handoffs.

Next (lane 1's coordinator): the optional body fix is applied ("the file gave 9 passed then (it holds 40 tests at the head, all passing)."). #849 takes lane 1's next turn, after lane 4's #862, on condition that CI tests at da3fe1c8 finishes green. Its gate g849r is running on the predicted squash over main plus #771 and #862.

Lane: openreposhape-1 (openRepoShape-1)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 13 hours by commenting @sourcery-ai review. Upgrade to get a review now.