[brettheap]

Lane: openreposhape-3 (openRepoShape-3)
Claim: #732 (comment)

Closes #732.

Lands after the v1.1.0 tag. #700 (#668), which the issue says it lands after, is on main as 55c6ba7d.

What was wrong (re-measured on main e3137010, git 2.43.0)

I built a superproject with a real submodule NAMED members/caf\xe9 (Latin-1) at members/IRRS. I made a commit on its detached HEAD, then ran git submodule deinit -f.

• The issue's claim holds. Under LC_ALL=C.UTF-8 and under LC_ALL=C, checkout_reach gives ('checkout', None), so the update line is handed over with no keep line. The ASCII control (members/cafe) gives the keep tuple, and kept_branch_line prints git --git-dir=…/.git/modules/members/cafe branch NAME <commit>.
• configured_url misses the same name. A section that kept its path and lost its url, with the url in the root's config, gives submodule_url → ('members/caf\ufffd', None). The mount is called url-less.
• New: a traceback under LC_ALL=C PYTHONUTF8=0. Python's filesystem encoding is ASCII there, and it is the encoding subprocess uses for a str argument.
  • Any section name that is not ASCII raises UnicodeEncodeError out of subprocess in configured_url or kept_gitdir. That includes valid UTF-8 (caf\xc3\xa9).
  • Main's shape-doctor.py --root <holder> --json then exits 1 with an empty stdout and a traceback, measured on holders this PR's tests build.

What changed, in shape-doctor.py

• gitmodules_listing (:5019-5047) decodes git config -z --list with surrogateescape rather than the replacement, so nothing git listed is lost.
• submodule_keys (:4986-5016) keeps each section NAME as git's own bytes, the surrogateescape. It still gives each VALUE with the replacement (shape-doctor.py submodule_config reads a family holder's .gitmodules through git_out, which decodes strictly, inside except (Refusal, OSError), so a member url that is not UTF-8 raises a UnicodeDecodeError that run_checks does not catch and shape-doctor.py --root ends in a traceback, exit 1 (also the code of a finding), with no verdict, where validate-family.py and siblings.py read the same file as bytes or with errors="replace" #432, scripts/shape_advisory.py repository_names and shape-doctor.py origin_url and origin_name read git remote get-url origin through git_out and catch Refusal and OSError only, so an origin URL that is not UTF-8 makes read answer not-answered (a single repository loses the Triad advisory, a leg clone its pointer to the assembly root) and ends shape-doctor.py --root in a UnicodeDecodeError traceback, exit 1, where an ASCII origin gives NOT A SHAPE ROOT, exit 2 #385): a url is compared with origin_url's reading and printed, so mounted_from and the members row read urls exactly as before. The name is never printed; it is only handed back to git.
• git_bytes (:5080-5094, new) turns such a name back into the argument that hands git the same bytes. On POSIX that is bytes, so neither a byte that is not UTF-8 nor an ASCII filesystem encoding changes it. On Windows, whose command line is text, it is the text itself.
• configured_url (:5050-5077) asks git config --get for submodule.<name>.url through git_bytes.
• kept_gitdir (:5116-5148) asks git rev-parse --git-path modules/<name> through git_bytes, and reads git's answer as a path (os.fsdecode). kept_head and the keep line hand that path back to git unchanged.
  • It used git_out, whose strict decode failed on that path.
  • The keep line prints the path through ascii_text, as every row does, so caf\xe9 shows as caf? (quoted).
• submodule_config's docstring (:7791-7843) now says where each part is decoded.

Tests

• tests/test_doctor_legs_members_next.py:1807-1878. member_named gives the holder fixture's member a submodule NAMED with bytes: its .gitmodules header (committed), its section of the holder's config, the repository git keeps and the mount's .git file. The names are members/caf\xe9 (Latin-1) and members/caf\xc3\xa9 (UTF-8). Each runs under three locales: a UTF-8 locale locale -a lists, LC_ALL=C, and LC_ALL=C PYTHONUTF8=0. The tests skip on Windows.
  • test_732_a_member_named_in_bytes_gets_its_keep_line: an emptied member's detached commit gets branch NAME <commit> in the members row. The ASCII case is test_413a_an_emptied_members_kept_repository_names_its_detached_commit.
  • test_732_a_member_named_in_bytes_whose_url_the_config_keeps_has_one: a member whose section lost its url is not called url-less. The ASCII case is test_414_3_a_member_url_the_holders_config_keeps_still_gets_the_line.
• Against main's shape-doctor.py, 8 of these 12 fail.
  • The Latin-1 name fails under all three locales, in both tests.
  • The UTF-8 name fails under LC_ALL=C PYTHONUTF8=0, where the doctor printed no JSON.
  • The 4 that pass are the UTF-8 name under a UTF-8 locale and under LC_ALL=C, which main already handled.
• tests/test_doctor_gitmodules_not_utf8.py:312 (shape-doctor.py: submodule_config and gitmodules_entry read a holder's .gitmodules through one git config -z --list, so a section name holding a space or a byte that is not UTF-8, or a url holding NEL or U+2028, gives the doctor the mount url git and make siblings use, under any locale (#668) #700's test). assert_read_as_git_reads now expects the section name from gitmodules_entry decoded with surrogateescape.

Commands on head 311ad7d9, each run with a stub gh (exits 97) first on PATH, GH_TOKEN=bogus and an empty GH_CONFIG_DIR, from tests/:

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt -o tmp_path_retention_policy=failed test_doctor_gitmodules_not_utf8.py test_doctor_legs_members_next.py test_doctor_members_spelled.py test_doctor_gitlink_repairs.py test_repo_hygiene.py test_cli_reference.py
293 passed in 217.04s (0:03:37)

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt-others -o tmp_path_retention_policy=failed test_doctor_next_is_a_command.py test_doctor_validator_rows.py test_windows_paths.py
152 passed in 99.46s (0:01:39)

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt-sd -o tmp_path_retention_policy=failed test_shape_doctor.py
436 passed, 3 skipped, 1 xfailed in 363.34s (0:06:03)

The skips and the xfail are in test_shape_doctor.py, which this PR does not change.

Pinned shape copies

None. shape-doctor.py is not a pinned copy.

Neighbours

#733's PR is stacked on this one: it changes submodule_keys again. This diff stays in the .gitmodules readers that #700 added or changed, plus configured_url and kept_gitdir. It does not touch pin_restore (#736), the partial-clone row (#754) or the lockstep NEXT (#750).

Round 2 (head 93b9bac3)

RV-c's review (issuecomment-6229515744) is READY. This round takes SHOULD 2 alone, as a docstring:

• git_bytes (shape-doctor.py:5080-5102) now says that on Windows a section name that is not UTF-8 is still not found. Git for Windows' xwcstoutf turns the lone surrogate into ED B3 A9, not E9. A UTF-8 name is found there.
• It names the lookup that would find it. Read the root's config with git config -z --list and match the key's bytes in Python, then build the kept repository's path from git rev-parse --git-common-dir, modules/ and the name's bytes. That is not built here.

SHOULD 1 (the doctor prints a non-ASCII path as ?) is lane 3's own issue.

On 93b9bac3:

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt -o tmp_path_retention_policy=failed test_repo_hygiene.py test_cli_reference.py test_doctor_gitmodules_not_utf8.py test_doctor_legs_members_next.py -k "hygiene or cli or not_utf8 or 732 or reference or repo"
147 passed, 114 deselected in 55.13s

Round 3: main bad05d21 merged (#748's import block)

Main moved to bad05d21. Its merge with 93b9bac3 conflicted in one hunk of one file, the import block of tests/test_doctor_legs_members_next.py, since #748 (431c8cf8): main splits the conftest import over two lines to add submodule_update_through_git, and this PR's 311ad7d9 adds from test_doctor_gitmodules_not_utf8 import listed_locale just below it. No other file conflicted. Nothing else changes: this PR is still a draft and still lands after the v1.1.0 tag.

• The resolution keeps main's two-line conftest import, then the listed_locale line. The merge commit is 012a7726 (parents 93b9bac3 and bad05d21), the head. Its combined diff against both parents:

@@@ -48,8 -48,8 +48,9 @@@ from pathlib import Path, PurePosixPath
  import pytest

- from conftest import WINDOWS_SKIP, cloned_just_pushed, git, rmtree
+ from conftest import (WINDOWS_SKIP, cloned_just_pushed, git, rmtree,
+                       submodule_update_through_git)
 +from test_doctor_gitmodules_not_utf8 import listed_locale
  from test_shape_doctor import (  # noqa: F401

On 012a7726, under a stub gh (exits 97), GH_TOKEN=bogus and an empty GH_CONFIG_DIR:

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt -o tmp_path_retention_policy=failed tests/test_doctor_gitmodules_not_utf8.py tests/test_doctor_legs_members_next.py
162 passed in 261.35s (0:04:21)   [23 tests in the first file, 139 in the second; none skipped on this git]

/usr/bin/python3 -m pytest -q -p no:cacheprovider --basetemp=<scratch>/bt -o tmp_path_retention_policy=failed tests/test_shape_doctor.py
436 passed, 3 skipped, 1 xfailed in 635.33s (0:10:35)   [run once, because this PR changes `shape-doctor.py`]

#775, stacked on this PR, takes this head and main in its own round.

🤖 Generated with Claude Code

Summary by Sourcery

Preserve Git’s original bytes for .gitmodules section names so submodule URL and repository lookups remain reliable across locales and non-UTF-8 names.

Bug Fixes:

• Preserve non-ASCII .gitmodules section names when resolving configured URLs and retained submodule repositories, preventing missing keep lines and locale-dependent UnicodeEncodeError failures.

Enhancements:

• Keep Git-reported section names as their original bytes while continuing to decode values for comparison and display.
• Add locale-independent coverage for Latin-1 and UTF-8 submodule names across UTF-8 and C locales.

Tests:

• Update Git configuration parsing expectations and add coverage for byte-preserving submodule name handling.

[sourcery-ai]

Reviewer's Guide

Preserve Git’s original section-name bytes throughout .gitmodules parsing and Git lookups, preventing missing submodule keep lines, incorrect URL-less diagnostics, and locale-dependent UnicodeEncodeError failures; add comprehensive cross-locale regression tests.

Sequence diagram for byte-preserving submodule lookup

sequenceDiagram
    participant Doctor as shape-doctor.py
    participant Git as Git
    participant Config as Repository config

    Doctor->>Git: git config -z --file .gitmodules --list
    Git-->>Doctor: Listing decoded with surrogateescape
    Doctor->>Doctor: submodule_keys preserves NAME bytes
    Doctor->>Git: git config --get submodule.NAME.url via git_bytes
    Git->>Config: Read configured URL
    Config-->>Git: URL
    Git-->>Doctor: URL text
    Doctor->>Git: git rev-parse --git-path modules/NAME via git_bytes
    Git-->>Doctor: Kept repository path
    Doctor->>Git: branch NAME COMMIT
    Git-->>Doctor: Keep line for detached commit

Loading

File-Level Changes

Change	Details	Files
Preserve Git’s original bytes for .gitmodules section names while continuing to decode displayed values as replacement-decoded text.	Decode git config -z --list with surrogateescape. Retain section names as surrogateescaped strings and convert them back to POSIX bytes when invoking Git. Document the distinct decoding and Windows limitations.	shape-doctor.py
Use byte-preserving names when resolving configured URLs and retained submodule repositories.	Pass byte-preserving submodule.<name>.url keys to git config --get. Pass byte-preserving modules/<name> paths to git rev-parse --git-path. Read returned repository paths with filesystem decoding so detached commits produce keep lines across locales.	shape-doctor.py
Add cross-locale regression coverage for non-ASCII and non-UTF-8 submodule names.	Build fixtures with Latin-1 and UTF-8 section names across UTF-8, C, and ASCII filesystem locales. Verify detached commits receive keep lines and retained URLs are recognized. Update existing non-UTF-8 expectations to require surrogateescaped section names.	tests/test_doctor_legs_members_next.py tests/test_doctor_gitmodules_not_utf8.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#732	Preserve .gitmodules section-name bytes when reading and passing names to Git, so configured_url and kept_gitdir can find non-UTF-8-named members under different locales.	✅
#732	Ensure a detached commit in an emptied member with a non-UTF-8 section name receives its keep line, and pin the behavior with Latin-1 and UTF-8 names across UTF-8, C, and ASCII-filesystem locales.	✅
#732	Avoid Unicode encoding tracebacks when passing non-ASCII section names to Git under LC_ALL=C PYTHONUTF8=0.	✅

Possibly linked issues

• shape-doctor.py: a .gitmodules section name that is not UTF-8 is looked up by its decoded form, so configured_url and kept_gitdir miss a Latin-1 member and the update line is handed over with no keep line #732: The PR implements the issue’s requested byte-preserving lookups and adds cross-locale tests confirming detached commits receive keep lines.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

(lane openRepoShape-3, 2026-10-17T04:14:52Z) Independent review of #774 by RV-c (opus), on head 311ad7d9: READY. No MUST. Two SHOULDs: one goes to a follow-up issue, the other is a docstring line. It lands only after the v1.1.0 tag.

• Claims: main's doctor fails 8 of the 12 new test_732_* cases, exactly as the body says, plus shape-doctor.py: submodule_config and gitmodules_entry read a holder's .gitmodules through one git config -z --list, so a section name holding a space or a byte that is not UTF-8, or a url holding NEL or U+2028, gives the doctor the mount url git and make siblings use, under any locale (#668) #700's two Latin-1 cases whose expectation this PR changes.
• A real holder with a member named caf\xe9 or caf\xc3\xa9, under each of the three locales: the table and --json both name the detached commit in the keep line, with no traceback, in all six runs.
• POSIX: a surrogateescape round trip is lossless and injective, so no name is lost or doubled. git_bytes (:5080-5094) hands git bytes, and kept_gitdir reads the answer with os.fsdecode (:5142), its exact inverse.
• Windows (by reading only): a UTF-8 name is found. A name that is not UTF-8 is still not found, because Git for Windows' xwcstoutf turns the lone surrogate into ED B3 A9, not E9. Nothing crashes or doubles; shape-doctor.py: a .gitmodules section name that is not UTF-8 is looked up by its decoded form, so configured_url and kept_gitdir miss a Latin-1 member and the update line is handed over with no keep line #732 simply stays open on Windows, and the tests skip there.
• CI: RV-c agrees the failure is the tests: the holder fixtures copy the family fixture with shutil.copytree while git's background gc or maintenance can still be packing it, so a test errors at setup when a loose object vanishes mid-copy (#440's race, through a file copy) #645 copytree flake: an ERROR at setup in test_doctor_partial_clone.py, which this PR does not touch. shape-doctor.py and siblings.py: a .gitmodules section git ignores ([submodule ""], a .. component), a path or url beginning with -, a section with no path, a file git refuses for a bare url, and an unmerged .gitmodules name no mount url, NEXT line or clone, where git says No url found and exits 128 (#733) #775's run carries this commit and passed 11558.

SHOULD 1 (follow-up issue): the keep line names a path that does not exist.

• In all six runs it printed …/.git/modules/members/caf? (or caf??), in the table and in --json. The doctor flattens every non-ASCII character to ?, even café under a UTF-8 locale, so the pasted git --git-dir='…/caf?' branch … fails.
• This is doctor-wide and already on main for UTF-8 names. Lane 3 files it.

SHOULD 2: say in git_bytes' docstring (or the body) that a name that is not UTF-8 is still not found on Windows. A fix there would read the root's config with git config -z --list, match the key's bytes in Python, and build the modules path from git rev-parse --git-common-dir + modules/ + the name's bytes.

Mutants:

mutant	killed by
decode with replace again	8
drop git_bytes	2 (the UTF-8 name with Python's UTF-8 mode off)
strict decode in kept_gitdir	4

Tests:

• main + shape-doctor.py: a .gitmodules section name keeps git's own bytes and is handed back to git as them, so configured_url and kept_gitdir find a member named caf\xe9, its detached commit gets its keep line under any locale, and a non-ASCII name no longer ends --root in a UnicodeEncodeError traceback under LC_ALL=C PYTHONUTF8=0 (#732) #774, 8 files: 399 passed;
• the 732 and not_utf8 subsets on the head: 35 passed.

Merges: clean with bf1f099b and every open lane-3 v1.1.1 PR. Hygiene and Sonar are clean, and no docs or pinned copies are affected.

Lane: openreposhape-3 (openRepoShape-3)

[brettheap]

(lane openRepoShape-3, 2026-10-17T04:46:59Z) RV-c's re-verification of #774 round 2 (93b9bac3): READY. SHOULD 2 is done: git_bytes' docstring (shape-doctor.py:5080-5102) names the Windows limit (Git for Windows turns the lone surrogate into ED B3 A9) and the name-free lookup that would fix it.

• Tests: main + shape-doctor.py: a .gitmodules section name keeps git's own bytes and is handed back to git as them, so configured_url and kept_gitdir find a member named caf\xe9, its detached commit gets its keep line under any locale, and a non-ASCII name no longer ends --root in a UnicodeEncodeError traceback under LC_ALL=C PYTHONUTF8=0 (#732) #774: 134 passed.
• Merges: clean with bf1f099b, every open lane-3 PR, and lane 4's siblings.py (make siblings): the printed mv goes astray over an existing .holder/.folder or a dangling link, a lone surrogate or NUL in project: is a traceback, a url with NEL or U+2028 is cut, two warnings print paths raw #690 tip (38a40abf).
• CI: Linux tests was pending at 04:46Z.

SHOULD 1, the doctor printing a non-ASCII path as ?, is filed as #785.

Lane: openreposhape-3 (openRepoShape-3)

[brettheap]

(lane openRepoShape-3, 2026-10-17T19:23:37Z) RV775r4's check of the main merge into #774 (head 012a7726): READY, with nothing left.

• 012a7726 is 93b9bac3 + main bad05d21. git's own merge conflicts only in tests/test_doctor_legs_members_next.py, and 012a7726 differs from that conflicted result in exactly the one hunk: main's two-line conftest import, then the listed_locale line. The two doctor files give 162 passed, and F774b's test_shape_doctor.py run gave 436 passed, 3 skipped, 1 xfailed.
• Nothing else entered the PR: its own change against main is identical before and after the merge (diff exit 0).
• Merges: clean against main fc867207, and against every open lane-3 PR that touches siblings.py or shape-doctor.py.
• CI: SonarCloud's gate is OK, with 0.0% new duplication; tests was still running at 19:23Z.

This lands after the v1.1.0 tag, under Brett Heap's word "merge lane 3's v1.1.1 drafts after the v1.1.0 tag".

Lane: openreposhape-3 (openRepoShape-3)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="shape-doctor.py" line_range="5253" />
<code_context>
+    `modules/` and the name's bytes.
+    """
+    if os.name == "nt":
+        return text
+    return text.encode("utf-8", "surrogateescape")
+
</code_context>
<issue_to_address>
**Configured URLs or keep lines go missing**

When the doctor runs on Windows and a `.gitmodules` section name contains bytes that are not valid UTF-8, `git_bytes` returns surrogate-containing text, which Git for Windows converts to different bytes. `configured_url` and `kept_gitdir` then fail to find the member, so its URL or keep line is missing.

Have `git_bytes` preserve the original invalid UTF-8 bytes on Windows instead of returning surrogate-containing text.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: shape-doctor.py:5253

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[sourcery-ai]

🟡 Medium · Configured URLs or keep lines go missing

When the doctor runs on Windows and a .gitmodules section name contains bytes that are not valid UTF-8, git_bytes returns surrogate-containing text, which Git for Windows converts to different bytes. configured_url and kept_gitdir then fail to find the member, so its URL or keep line is missing.

Have git_bytes preserve the original invalid UTF-8 bytes on Windows instead of returning surrogate-containing text.

Prompt for AI agents

In `shape-doctor.py` at line 5253:

**Configured URLs or keep lines go missing**

When the doctor runs on Windows and a `.gitmodules` section name contains bytes that are not valid UTF-8, `git_bytes` returns surrogate-containing text, which Git for Windows converts to different bytes. `configured_url` and `kept_gitdir` then fail to find the member, so its URL or keep line is missing.

Have `git_bytes` preserve the original invalid UTF-8 bytes on Windows instead of returning surrogate-containing text.