Repository navigation
adopt-project.py: under LC_ALL=C PYTHONUTF8=0, plan, check and execute hand git and gh each name, url and branch and the leg description as the UTF-8 bytes it spells, so none of them ends in a UnicodeEncodeError before any leg exists (#653) - #727
Merged
Merged
Conversation
…e, a url and the leg description as the UTF-8 bytes each spells, so under `LC_ALL=C PYTHONUTF8=0` `gh repo create`'s em dash, `plan`'s `git grep` and `check`'s submodule url no longer end in a `UnicodeEncodeError` traceback before any leg exists (#653) On POSIX `subprocess` encodes a `str` argument with the locale's codec when Python is not in UTF-8 mode, and under `LC_ALL=C PYTHONUTF8=0` that codec is ASCII. #614 (#527) made `execute` hand `git rm` the bytes the tree holds; three places still handed git or gh text, and each ended in a traceback, exit 1, before any leg existed. This carries out the 2026-10-12 ruling for #527, "pass git the exact bytes the tree holds, under any locale", and Brett Heap's 2026-10-16 answer to #457's triage, "In scope, file D4". - L1-33: `gh repo create --description` carries the tool's own em dash, so every GitHub-mode `execute` stopped at the first leg, an all-ASCII source included. The `gh pr create --body` after it is the split's message, which names the moved paths. - L1-34: `plan`'s `git grep` (`cross_leg_references`) was handed a spec directory or a code entry outside ASCII as text, and `plan` wrote no plan. - L1-35: `_resolved` handed `git config` a relative submodule url outside ASCII, or a source remote outside ASCII, as text, and `check` and `execute` died. So did `_default_remote`, asking `git config` for the url of a remote named outside ASCII. Past them, `same_as` asked `realpath` of such a url, and #418's `git ls-remote` was handed it, and both died the same way. The change: - `_run_exact(command, cwd)` is `run`, with each argument handed as `_path_argument` (#614's helper) makes it: UTF-8 bytes on POSIX, text on Windows. Wherever the locale's codec already gives those bytes, which is every UTF-8 locale, it calls `run` itself as the call site did, so nothing changes there, the tests' stand-ins for `run` included. `gh repo create`, `gh pr create` and `_resolved`'s two `git config` calls use it. - `cross_leg_references` hands `git grep` the spec directory and the code paths through `_path_argument`, and so do `_config_value` for its key, `_answers` for its two `git ls-remote` calls, and `same_as` for `realpath`. - `_git_spelling(path)` spells a path read from the disk as `_git_path` spells one git printed, the same bytes. `_default_remote` spells the source's directory that way (the base, or the working tree a relative answer is joined to), and `_relative_url_findings` the leg's remote, so that each goes to git as the bytes the disk holds under any locale, a Latin-1 one included, and is named as UTF-8 mode names it (`Ü`, not `\udcc3\udc9c`). - The description and the body are handed to gh as they were written. Tests: tests/test_adopt_c_locale_bytes.py adds 5 tests (9 cases), all failing on main at a524542, each in a child under the C locale and skipped on Windows and where that locale still encodes file names as UTF-8. A GitHub-mode `execute` (tests/test_adopt_github_exits.py's harness, with a `gh` stand-in that keeps each call's arguments as bytes) hands `gh repo create` the description and `gh pr create` the split's message as UTF-8, byte for byte the commit's; `plan` writes the plan UTF-8 mode writes and finds both code paths that name `Ärger/`; `check` says what UTF-8 mode says for a url, a remote, a remote's name, a source directory and a working tree outside ASCII; and `execute --local-remote-dir` refuses as UTF-8 mode does and adopts `../Ünterlagen`. Each change reverted alone fails at least one of them. A `_run_exact` that never calls `run` passes them, and fails 8 existing tests (and errors 10) that stand in for `run` to catch a `gh` call. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nch as the bytes git keeps it, so under `LC_ALL=C PYTHONUTF8=0` a source on `café` is refused `unsafe-value`, or adopted past it with `--tracking-branch`, and `plan`, `check` and `execute` no longer end in a `UnicodeEncodeError` before anything is said of the branch (#653) w425's probe, forwarded by the coordinator: `Source.open` read the source's default branch with `_default_branch` and handed it to `git rev-parse` through `git_out` as text. Under the C locale a UTF-8 name such as `café` raised `UnicodeEncodeError` there, in `plan`, `check` and `execute` alike, whatever `--tracking-branch` said. - `_git_out_exact(args, cwd)` is to `git_out` what `_run_exact` is to `run`: `git_out` itself wherever the locale hands every argument alike, which is every UTF-8 locale; elsewhere git is run with `_path_argument` of each argument, and a git that refuses is refused as `git_out` refuses it, `git-failed`, naming the command as the person reads it. `git_out` is `scripts/repo_shape.py`'s, a pinned shape copy, and it cannot take a bytes argument: its refusal joins the arguments as text. - `Source.open` asks `git rev-parse` through it. It holds whichever of #425 and #653 lands first. #425 reads the name as bytes and keeps a byte that is not UTF-8 as a lone surrogate, which every locale's codec writes back as that byte; a UTF-8 name is still text, which only this change hands git as its bytes. Tests: tests/test_adopt_c_locale_bytes.py adds one test, 3 cases, each failing on 9b8e176: under the C locale `plan` of a source on `café` is refused `unsafe-value` as in UTF-8 mode; with `--tracking-branch main` it writes the plan UTF-8 mode writes, byte for byte, and `check` names `café` as UTF-8 mode does; and where git refuses `git rev-parse café`, the refusal is UTF-8 mode's `git-failed`. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideMakes adopt-project.py robust under non-UTF-8 POSIX locales by preserving the exact UTF-8 bytes for Git/GitHub arguments and Git-derived paths, while retaining existing command/error behavior; adds regression tests covering plan, check, and execute workflows. Sequence diagram for locale-safe Git and GitHub executionsequenceDiagram
participant AdoptProject
participant ExactRunner
participant GitOrGh
AdoptProject->>ExactRunner: _run_exact(command, cwd)
ExactRunner->>ExactRunner: _the_locale_hands_it_alike(argument)
alt Locale already preserves arguments
ExactRunner->>GitOrGh: run(command)
else Non-UTF-8 locale
ExactRunner->>GitOrGh: subprocess.run(_path_argument(argument))
end
GitOrGh-->>ExactRunner: UTF-8 stdout or failure
ExactRunner-->>AdoptProject: Result or CommandFailed
Sequence diagram for locale-safe source branch resolutionsequenceDiagram
participant SourceOpen
participant ExactGitRunner
participant Git
SourceOpen->>SourceOpen: _default_branch(path)
SourceOpen->>ExactGitRunner: _git_out_exact([rev-parse, branch], path)
ExactGitRunner->>Git: git rev-parse _path_argument(branch)
Git-->>ExactGitRunner: Commit or git failure
ExactGitRunner-->>SourceOpen: Commit or git-failed refusal
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
…s asked of a UTF-8 branch name and of one holding a byte that is not UTF-8, present and missing, under `LC_ALL=C PYTHONUTF8=0`, so a strict encode or a bytes argument in `git_out`'s refusal would fail a test (#653) #425's review named two risks for 04c6843's `_git_out_exact`. Both hold on 04c6843, and this test now holds them. - The name must be encoded with `surrogateescape`: #425 keeps a branch byte that is not UTF-8 as a lone surrogate (`caf\udce9`), in every locale. `_git_out_exact` hands bytes only through `_path_argument` (UTF-8, `surrogateescape`), and where it hands `git_out` text, `subprocess` encodes it with `os.fsencode`, which uses `surrogateescape` too. - No bytes may reach a refusal that joins the arguments as text, as `git_out`'s does. `_git_out_exact` joins the arguments as the text they were given, and hands `git_out` only text. The test runs `_git_out_exact(["rev-parse", <name>], ...)` in a child under the C locale, for `café` and `caf\udce9`, each present and missing. A present branch gives back its commit. A missing one is refused `git-failed`, naming `git rev-parse <name>`. With the helper encoding strictly, the two `caf\udce9` cases fail with `UnicodeEncodeError`. With it handing `git_out` bytes, the two missing cases fail with `TypeError: sequence item 0: expected str instance, bytes found`. adopt-project.py is unchanged. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…own words since #634, the folder named with `shown_path` and git's message `spelled` as one line, so under `LC_ALL=C PYTHONUTF8=0` a refused `git rev-parse café` no longer reads otherwise than in UTF-8 mode, nor begins a line from a folder's name (#653) r653's review of cc652b1, F1. #634 (#610, 30676c9) changed the refusal of `git_out` in scripts/repo_shape.py: it now names the folder with `shown_path` and spells what git printed as one line. `_git_out_exact` copies that refusal for the path it runs itself, which is the C locale with a name outside ASCII, and it still built the old text: the folder and git's lines raw. On any merge with main, `test_a_default_branch_outside_ascii_reads_as_in_utf_8_mode[gone]` failed in both modes. And under the C locale, a folder named with a line break began a line the tool never printed, the one #610 closed for `git_out`. - `shown_path` and `spelled` (as `spelled_as_git_out`, because adopt-project.py has local variables named `spelled`) are imported from the pinned module. `_git_out_exact`'s refusal uses them exactly as `git_out` does. The docstring says the two must stay in step. - `test_git_out_exact_refuses_in_git_out_s_own_words` asks `git_out` and `_git_out_exact` the same failing command in a folder whose name `shown_path` spells (U+2028), with git's message three lines long. The two refusals must be equal, on one line. With `shown_path` reverted alone, it fails. With `spelled` reverted alone, it fails, and so does `[gone]`. Main (b618ec1) was merged into the branch first, so this head holds #634. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap
marked this pull request as ready for review
October 16, 2026 18:23
|
Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 3 hours and 50 minutes by commenting |
|
11 of 15 tasks
brettheap
pushed a commit
that referenced
this pull request
Oct 17, 2026
main gained 25 pull requests since de92614: #634, #699, #671, #719, #701, #717, #700, #702, #722, #720, #725, #721, #728, #705, #727, #718, #744, #743, #749, #740, #737, #734, #742, #751 and #736. No conflict: AGENTS.md, adopt-project.py and docs/cli.md merged as text, and README.md and docs/handbook.html are unchanged by the merge. README.md is 1473 lines, its cap; AGENTS.md 494, its cap; every handbook marker is current; `render-cli-reference.py --check` is ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap
pushed a commit
that referenced
this pull request
Oct 17, 2026
…es every reason THIS checkout cannot answer, and the notes are current to main bf1f099 with the 25 pull requests since de92614 (#563, #683) - W683 and RV-f's N4 on #742: README's exit-3 list (Keeping a project's shape current, :1208-1220) and its handbook twin (`doctor`) named only a pin whose commit this checkout does not carry. They now say, as shape-doctor.py's EXIT CODES does: a pin's commit it lacks, a partial clone missing what is compared (#683), a read that fails (`update-upstream-unreadable`), a crashed validator of its own (#278, #410), or its own naming policy not loading (#678). +1 line, with tests/readme-cap.d/683.txt; README.md is 1474, its cap (1422 + 2 + 5 + 12 + 22 + 10 + 1). The two `Keeping a project's shape current` markers move: `doctor` text and marker, `current` marker only. - The notes, "What changed since v1.0.1", one entry per pull request from `git log --first-parent de92614..bf1f099`, each from its squash commit and the issues GitHub says it closes: - adopt-project.py: #699, #719, #749, #727, #717, #728 (with #621's hint), #743 + #720 + #718, #722, #721; - update-shape.py, bump-leg.py and a project's own scripts: #702 (with the pinned validators), #634; - scaffold, setup, family and the shared readers: #725, #671, #744; - shape-doctor.py: #742, #701, #737, #751, #736, #734, #700 + #705, #740; - install, CI and the docs: the exit-3 sentence above. - Counts (99 pull requests since bfed143; 109 of 135 milestone issues closed at 10:28Z), the "current to main" sha and the open list (26) are at main bf1f099, still placeholders for tag time. Left for round B: lane 5's eight lines (#630 issuecomment-6227129879), whose pull requests #726, #747, #682, #724, #746, #745 and #738 are open drafts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap
pushed a commit
that referenced
this pull request
Oct 17, 2026
…d A3 (three SHOULDs, four nits), and notes entries for the seven pull requests landed since bf1f099, current to main 2000488 (#563, #683) Round A4, from lane 2's review (#630 issuecomment-6233033833, READY): - SHOULD 1: the doctor's exit-3 list names "a validator of its own that crashed or could not be run", so the case of one that cannot be given its `PYTHONPATH` (shape-doctor.py's EXIT CODES, #278) is listed too: README.md:1208-1222, its handbook twin (`doctor`) and the notes' docs line. The paragraph re-wraps to one more line: tests/readme-cap.d/683.txt is +2, and its text now says which causes the EXIT CODES block names and that #742 added the partial clone and #701 the naming policy. README.md is 1475 lines, its cap (1422 + 2 + 5 + 12 + 22 + 10 + 2). Both `Keeping a project's shape current` markers move to c728c970: `doctor` text and marker, `current` marker only. - SHOULD 2: #727's entry names the leg description, whose em dash stopped every `execute` against GitHub at the first leg. - SHOULD 3: #749's entry names `source-ambiguous`, for an `org/repo` that is also a directory here. - Nits: the doctor section's three near-duplicate pairs are merged (#636 with #737, #632 with #700 and #705, #586 with #736); #718 has its own bullet; #634's entry names `git_out`'s spelling for every caller and #744's the gate's `--link-source-label` and `--link-remedy`; the header comment says 2026-10-17 at main 2000488. Wave B1, `git log --first-parent bf1f099..2000488`: #706 (#664), #703 (#696), #670 (#646), #682 (#648), #726 (#667), #730 (#666) and #724 (#665), each from its squash commit; #726's and #682's from lane 5's lines (#630 issuecomment-6227129879), #682's with the unmerged `.gitmodules` its last round added, and #724's with lane 5's note that v1.0.0's test-hygiene line on the generator's `.pyc` files closes. The counts (106 pull requests; 116 of 135 milestone issues closed at 11:53Z), "current to main" and the open list (19) are at main 2000488, still placeholders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Lane: openreposhape-1 (openRepoShape-1)
Claim: #653 (comment)
What
Under
LC_ALL=C PYTHONUTF8=0,adopt-project.pynow hands git and gh every name, url and branch it read from git, and the leg description it writes, as the UTF-8 bytes each spells, as #614 did forgit rm(#527). Each place below used to end in aUnicodeEncodeErrortraceback, exit 1, before any leg existed.gh repo create --descriptioncarries the tool's own em dash (Northwind — spec leg, extracted from testorg/Northwind with history), so everyexecuteagainst GitHub stopped at the first leg, an all-ASCII source included. Both legs are now created. The description and thegh pr create --bodyafter it (the split's message, which names every moved path) reach gh as written, byte for byte.plan. Thegit grepthat finds code naming a spec directory (cross_leg_references) was handedÄrger/, or a code path such asWerkzeuge-Ü/, as text, andplanwrote no plan. It now writes the plan UTF-8 mode writes, byte for byte.checkandexecute._resolvedhandedgit configa relative submodule url (../Ünterlagen), or a source remote under a directory outside ASCII, as text. So did_default_remotefor a remote whose name isÜrsprung. Past those,same_asaskedrealpathof such a url, and adopt-project.py saysadoption verifiedwhen a submodule the plan keeps in a leg has a url that names no repository (a relative url that resolves alike from the source and the leg, or an absolute one), so the recursive clone itsNEXT:line prints fails; nothing asks whether the url answers #418'sgit ls-remotewas handed it. Each now goes as bytes, andcheckandexecutesay what UTF-8 mode says.Source.open, forwarded from w425's probe. It handedgit rev-parsea UTF-8 default branch such ascaféas text, soplan,checkandexecutedied before anything was said of the branch. Now a source oncaféis refusedunsafe-valuefor the tracking branch, as in UTF-8 mode, or adopted past it with--tracking-branch main.The ruling is Brett Heap's of 2026-10-12, for #527: "pass git the exact bytes the tree holds, under any locale". His 2026-10-16 answer to #457's triage, "In scope, file D4 (Recommended)", puts this locale in scope here.
Why (repro on main)
Probed on main at a524542 (the review found the new tests failing the same way, at the same functions, on 30676c9), with a stub
ghfirst onPATHand a throwaway git config that sendshttps://github.com/to a local directory (no network):LC_ALL=C PYTHONUTF8=0executeof an all-ASCII sourceUnicodeEncodeError('—') in_create_leg_remote, exit 1, no repository madeadoption verified, exit 0Überblick.mdmoving to the spec legplanof a source whoseÄrger/(spec) is named byWerkzeuge-Ü/run.pyandÜbung.py(code)UnicodeEncodeErrorincross_leg_references, exit 1, no plancheck,../Ünterlagenkept in the code leg, source on GitHubUnicodeEncodeErrorin_resolved, exit 1plan okand UTF-8 mode's NOTEs, exit 0check, a source whoseoriginis<dir>/Ünter/testorg/Thing.gitUnicodeEncodeErrorin_resolved, exit 1plan-submodule-relative-urlfinding, exit 1check, a source whose branch tracks a remote namedÜrsprungUnicodeEncodeErrorin_config_value, exit 1Ürsprung, exit 1check, a source with no remote in<dir>/Ünter/\udcc3\udc9cnterÜnter, as UTF-8 mode doesexecute --local-remote-dir,../Ünterlagennaming one repository from the source and from the legUnicodeEncodeErrorin_resolved, exit 1adoption verified, exit 0planof a source oncaféUnicodeEncodeErrorinSource.open, exit 1REFUSED unsafe-value, exit 2, as in UTF-8 mode; with--tracking-branch main, the plan UTF-8 mode writesHow
_run_exact(command, cwd)isrunwith each argument handed as_path_argument(adopt-project.py:executehandsgit rmeach path the split moves or drops as the bytes the tree holds, so a name outside ASCII underLC_ALL=C PYTHONUTF8=0no longer stops the run withUnicodeEncodeErrorafter both legs are pushed (#527) #614's helper) makes it: UTF-8 bytes on POSIX, text on Windows. Where the locale's codec already gives those bytes, which is every UTF-8 locale and any all-ASCII command, it callsrunitself, exactly as the call site did. So nothing changes there, and the tests that stand in forrunto catch aghcall still catch it. Elsewhere it runs the command itself and refuses asrundoes:CommandFailed, naming the command as the person reads it.gh repo create,gh pr createand_resolved's twogit configcalls use it._git_out_exact(args, cwd)is the same forgit_out, which isscripts/repo_shape.py's, a pinned shape copy.git_outcannot take a bytes argument, because its refusal joins the arguments as text. Where it is not called, a git that refuses is refused asgit_outrefuses it on main since Pinned shape copies: one bump, so AGENTS-shape.md is true of every root, the scaffold's chain WARNING offers only its own sources and spells--pin-source, the root's not-used note spells it too, and THE LOCKSTEP RULE'sbumpruns (#610) #634 (Pinned shape copies: AGENTS-shape.md'sCONTRACTS_DIRsentence is false for every scaffolded root, and the chain WARNING offers remedies the scaffold lacks (one shape bump before the v1.1.0 tag) #610):git-failed, the directory named byshown_pathand what git saidspelledas one line. Its docstring says the two must stay in step, and a test compares them.Source.open'sgit rev-parseuses it._path_argumentdirectly where the call was already a plainsubprocess.run:cross_leg_references'sgit greppattern and pathspecs,_config_value's key,_answers' twogit ls-remoteurls, andsame_as'srealpath._git_spelling(path)spells a path read from the disk as_git_pathspells one git printed: the same bytes, decoded as UTF-8 withsurrogateescape._default_remotespells the source's directory that way (as the base, or as the working tree a relative answer is joined to), and so does_relative_url_findingsfor the leg's remote under--local-remote-dir. Two things follow.os.fsdecode's text to_path_argumentwould double-encode it.Tests
tests/test_adopt_c_locale_bytes.py: 8 tests, 17 cases. Every case fails on main at a524542, and again at b618ec1, and passes here, on this head, which holds main b618ec1. All but one runadopt-project.py, or the helper itself, in a child underLC_ALL=C PYTHONUTF8=0. Each of those skips on Windows, and where that locale still encodes file names as UTF-8 (#614'sc_locale_or_skip); the one that runs in this process runs everywhere. What a run says is compared with what the same run says in UTF-8 mode (LC_ALL=C PYTHONUTF8=1), once the C locale's\xdc-style escapes are undone, and the comparison requires at least one REFUSED, FINDING or NOTE line.execute, on tests/test_adopt_github_exits.py's harness. Itsghstand-in gets one more line, which keeps each call's arguments, NUL-separated, as bytes.gh repo createis handed each leg's description as its UTF-8 bytes.gh pr createis handed the split commit's message byte for byte,Überblick.mdin it.planwrites the plan UTF-8 mode writes, and its follow-ups name both code paths that nameÄrger/.checksays what UTF-8 mode says for five sources: a url, a remote, a remote's name, a source directory and a working tree outside ASCII. Each locale writes its own plan.execute --local-remote-dirrefuses with UTF-8 mode's finding where the legs would sit in<dir>/Ünter, and adopts../Ünterlagenwhere they sit beside the dependency.Source.open(3).caféis refusedunsafe-value.--tracking-branch main, the source writes UTF-8 mode's plan, andchecknamescaféas UTF-8 mode does.git rev-parse café(a clone whose owncaféwas deleted), the refusal is UTF-8 mode'sgit-failed.Source.open's helper, from adopt-project.py_default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425's review (4)._git_out_exact(["rev-parse", <name>])runs in a child, forcaféand forcaf\udce9(a byte that is not UTF-8, as adopt-project.py_default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425 keeps it), each present and missing. The commit comes back, or the refusal isgit-failed, naminggit rev-parse <name>as text.git_out's, from the review (1). In this process, with no locale needed,git_outand_git_out_exact(made to take its own path, as it does under the C locale) are asked the same failinggit rev-parse. They run in a folder whose nameshown_pathspells (U+2028), and git's message is three lines. The two refusals must be equal, and one line.Revert table. Each change reverted alone in the final file, run over the new file (17 cases; 17 pass with the change). The first 13 rows were measured before the two helper tests were added; those call only
_git_out_exact, which none of the 13 touches. The last two rows were measured on this head:gh repo createthroughrungh pr createthroughrungit grephanded textplancase_resolved's url throughruncheckurl, and theexecutecase_resolved's base throughruncheckoriginandown-directory, and theexecutecase_git_spellingedcheckown-directory(the finding said\udcc3\udc9c)_git_spellingedcheckworking-treesame_as'srealpathof textexecutecase_answers'git ls-remote --get-urlof textexecutecase_answers'git ls-remoteof textexecutecase_git_spellingedexecutecase (the refusal named<dir>/\udcc3\udc9cnter)_config_value's key handed textcheckremote-nameSource.open'sgit rev-parsethroughgit_outcafécase_run_exactnever callingrunrunto catch aghcall fail, and 10 error: test_adopt_filter_repo_failure.py, test_adopt_late_failure.py and test_adopt_split_branch.py_git_out_exactnever callinggit_outgit_outonSource.open's path. The hand-off keeps every UTF-8 run ongit_outitself_git_out_exactencoding strictly, never throughgit_out(#425's first risk)caf\udce9cases, withUnicodeEncodeError_git_out_exacthandinggit_outbytes (#425's second risk)TypeError: sequence item 0: expected str instance, bytes foundfromgit_out's refusal_git_out_exact's refusal withoutspelled(git's lines raw, the review's F1)gonecase and the refusal comparison_git_out_exact's refusal withoutshown_path(the folder raw)Focused runs. Run as the coordinator asked after a session exit voided the full runs, and again after the review, with
/usr/bin/python3 -m pytest, a stubghfirst onPATH,GH_TOKENa dummy, an emptyGH_CONFIG_DIRand noGIT_ALLOW_PROTOCOL. Each log names its head and dirty 0, at its start and its end.git filter-repoonPATH: 258 passed, 1 skipped, 0 failed. With it hidden: 252 passed, 7 skipped, 0 failed.run.git filter-repoonPATH: 931 passed, 4 skipped, 0 failed. With it hidden (PATH=/usr/local/bin:/usr/bin:/binbehind the stubgh): 877 passed, 58 skipped, 0 failed._default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425's tests/test_adopt_git_reads.py were run: 37 passed withgit filter-repo, and 31 passed, 6 skipped hidden.git filter-repohidden: 6,059 passed, 360 skipped.Main, and #425. Main at b618ec1 is merged into this branch (217a8f7, clean, no conflict). The trial merges below were made with
git merge-treeandcommit-treein a scratch clone, and nothing of them was pushed:legs.code_pathandlegs.spec_pathand no path, and thecontracts/rule's reason names none, so the hand edit of a leg's mount the tool asks for leaves no plan-time path incheck, the split commit or the pull request (#600) #628 (2476992, the rewrite ofcross_leg_references): the new file and adopt-project.py:executehandsgit rmeach path the split moves or drops as the bytes the tree holds, so a name outside ASCII underLC_ALL=C PYTHONUTF8=0no longer stops the run withUnicodeEncodeErrorafter both legs are pushed (#527) #614's passed, 16 tests, before the helper tests were added. With main at 30676c9, which holds Pinned shape copies: one bump, so AGENTS-shape.md is true of every root, the scaffold's chain WARNING offers only its own sources and spells--pin-source, the root's not-used note spells it too, and THE LOCKSTEP RULE'sbumpruns (#610) #634'sgit_outwording, the merge was clean. On ittest_a_default_branch_outside_ascii_reads_as_in_utf_8_mode[gone]failed in both modes, until_git_out_exactspelled its refusal asgit_outdoes. That is 0274b68, on main b618ec1, where it passes, with adopt-project.py_default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425's head as well._default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425's head 44e9711, which reads_default_branchas bytes withsurrogateescape: the three-way merge of main, adopt-project.py_default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425 and this head (416ffd0f) gives, on the new file and adopt-project.py_default_branchreads the source's branch name throughgit_out, which decodes strictly, so a source whose HEAD (ororigin/HEAD) names a branch that is not UTF-8 endsplan,checkandexecutein a UnicodeDecodeError traceback, exit 1, fromSource.open, with no plan written and no refusal naming the branch #425's tests/test_adopt_git_reads.py, 48 passed withgit filter-repo, and 39 passed, 9 skipped hidden.The fix holds whichever of #425 and #653 lands first. #425 keeps a branch byte that is not UTF-8 as a lone surrogate, which
_path_argumentandos.fsencodeboth write back as that byte, throughsurrogateescape. A UTF-8 name such ascaféis still text after #425, and only this change hands it to git as its bytes. On the merge,plan --tracking-branch mainunder the C locale works for both acaf\xe9and acafédefault branch.Windows
CI is Linux only since #597. On Windows,
_path_argument,_git_spelling,_run_exactand_git_out_exacteach leave the text as it is and call what was called before, so nothing changes there. The C-locale tests skip on Windows: a C locale is a POSIX idea, and Windows hands a command line toCreateProcessWas UTF-16. The comparison of the two refusals runs everywhere; on Windows_git_out_exactalways callsgit_out, so the two are one. Every name the tests make (Ünter,Ünterlagen,Ürsprung,Ärger,Werkzeuge-Ü,Übung.py,Überblick.md,café,line\u2028separator) is legal on NTFS. They compare names as the bytes git records, and never count a\nin a file a helper wrote.Not in this PR
plantable's padding under the C locale._print_entries(adopt-project.py:4212 at this head) pads_shown(entry.path)to 34 characters before stdout escapes it, socafé.md(caf\xe9.md, 4 characters wider) shifts thelegcolumn. It is a display fix, not this one, and stays on Tracking: adopt-project.py, converting an existing repository in place #457._git_spellingturned into such text first, so a Latin-1 locale gets the bytes the disk or the tree holds.…/Ünter/Thing. Under the C localePath.exists()cannot hold that name and says False, soSource.opentakes it fororg/repoand dies inSource.open'sgh repo clone(adopt-project.py:1561 at this head) withUnicodeEncodeError. A plan written under the C locale records the bytes as surrogates, and reads back fine in either locale. This is a new line for Tracking: adopt-project.py, converting an existing repository in place #457's checklist.Known limits
_run_exactand_git_out_exactcallrunandgit_outonly where the locale hands every argument alike. A test that stands in for either under the C locale therefore does not see these calls. No test does so today.plan,checkandexecutemake still hands text, as read on this head, because nothing outside ASCII reaches it, or only a path the disk gave does. Such text is: nameschecked_valueadmits (organisation, repository, tracking and split branches, leg mounts), the shape's own file names, object ids, and pathsos.fsdecodemade, whichsubprocesshands back as the same bytes in any locale.Closes #653
🤖 Generated with Claude Code
Summary by Sourcery
Preserve UTF-8 byte spelling across Git and GitHub operations so adoption workflows behave consistently under non-UTF-8 locales.
Bug Fixes:
Enhancements:
Tests: