[brettheap]

Lane: openreposhape-5 (openRepoShape-5)
Claim: #214 (comment)

Closes #214.

adopt-project.py plan wrote the plan, then raised UnicodeEncodeError in its terminal report for a name its stdout could not encode, and exited 1 before plan written to and the questions it exists to ask. On Windows that happens on any stdout that is not the console itself: a file, a pipe to tee, CI, an agent running the tool. Python gives such a stdout the ANSI code page (cp1252 on a Western-European install), which has no 日本. It also gives it the surrogateescape error handler, which CPython uses for EVERY Windows stdout by default. config_get_stdio_errors() in Python/initconfig.c ends /* On Windows, always use surrogateescape by default */ return L"surrogateescape"; in 3.9, 3.12 and 3.13, and Doc/c-api/init_config.rst says the same ("Default error handler: On Windows: use "surrogateescape""). That handler covers only U+DC80..U+DCFF, so it raises on 日本 just as strict does.

What was wrong

Reproduced on origin/main 20e13f0 with the issue's method, PYTHONIOENCODING=cp1252:strict. Under utf-8:strict every run exits 0, except the doctor's NOT A SHAPE ROOT, which exits 2 as designed. Under cp1252 the raise comes at the first plain print of a path or a person's text, in six tools and the root bootstrap shim:

tool	what holds the name	line on 20e13f0
adopt-project.py plan	a source file 日本.md; the --source path; the --out path	1588, 1497, plan written to
adopt-project.py check	the source	2141
shape-doctor.py	the --root path	3690
update-shape.py check	the --root path	1245
scaffold-project.py	--name, --elected-by; a --work-dir or --local-remote-dir path	602, 620
scripts/family.py init	--name, --created-by; an --into or --local-remote-dir path	563, 567
scripts/bump-leg.py (--dry-run)	the project's display name	706
bootstrap (root shim)	the directory it is run from	templates/assembly-root/scripts/bootstrap.py:400

Under cp1252:surrogateescape, the Windows default, the same prints raise. This was measured on this branch's first commit 3006303, whose helper changed only strict and so left that case exactly as on main.

The fix

shape_materialize.escape_what_stdout_cannot_encode() is the first statement of each of the six tools' main(), and the bootstrap shim calls it before it hands over to the canonical bootstrap with runpy. It changes two error handlers and no others, and it never changes the encoding:

• strict becomes backslashreplace, the handler Python already gives stderr. strict is what a POSIX locale gives stdout when it is not C or POSIX and not one Python coerces C to (en_US.UTF-8, but not C.UTF-8, the usual container locale, which gives surrogateescape), and what PYTHONIOENCODING=<codec> gives when it names no handler.
• surrogateescape becomes openreposhape.surrogateescape-then-backslashreplace, a handler registered with codecs.register_error by the first call that needs it and by no other. It writes a surrogate-escaped byte back as that byte, which is surrogateescape's own rule, so a name that is not UTF-8 still comes out as its own bytes. Anything else the encoding cannot hold it writes in codecs.backslashreplace_errors' own spelling. surrogateescape is the Windows default, and UTF-8 mode and the C, POSIX and C.UTF-8 locales give it too.
• Every other handler (replace, backslashreplace, ignore, xmlcharrefreplace, namereplace, surrogatepass, or one an operator set) is left as it was.

Each replacement writes exactly the bytes the handler it replaces wrote, and differs only where that handler raised. So nothing that printed before prints differently. On a cp1252 stdout 日本.md prints as \u65e5\u672c.md under either handler, and plan exits 0 with its whole report. stderr is untouched. The helper lives in shape_materialize.py, not repo_shape.py, because repo_shape.py is a digested shape copy, and a helper no project calls would put an upstream-changed row in front of every project. The bootstrap shim at the repository root is not a shape copy: no copy list and no shape-pin.yaml names it, and projects hold templates/*/scripts/bootstrap.py instead.

Why this and not flattening each name where it is printed (the issue's other candidate):

• It keeps every run that worked byte-identical, because each new handler differs from the old one only where the old one raised. The advisory's printable (ASCII, ?) would print ??.md on a UTF-8 terminal that shows 日本.md today.
• It is one helper and one call per tool, against nine raising sites in six files and the next print somebody adds.
• It composes with adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199, whose _shown() rewrites the prints in _print_entries and the question list.

Evidence

• tests/test_stdout_encoding.py, 48 tests.

  • Every run is made under cp1252:surrogateescape (the Windows default), cp1252:strict and utf-8:strict. Output is read as bytes and decoded strictly in the encoding the run was given.
  • plan runs on a source holding 日本.md (answered, in the table) and 東京.ipynb (asked, in the question list).
  • check, the doctor, update-shape.py check, bump-leg.py --dry-run, the scaffold's and family.py init's dry runs, and the bootstrap shim each exit alike under all three, with the name shown.
  • plan --out writes into a directory whose name holds byte 0xff, under utf-8:surrogateescape and cp1252:surrogateescape. The raw byte comes out raw, and in the cp1252 run 日本.md is escaped in the same report. This one runs on POSIX filesystems only, and skips where a name that is not UTF-8 cannot be made.
  • In process: a surrogateescape stream in utf-8 and in cp1252 keeps the raw byte, escapes the rest and keeps its encoding, and a second call changes nothing. The handler is registered once, under a name starting openreposhape.. Six other handlers are left alone. A reconfigure that raises a plain ValueError or a plain OSError, a closed stream, a stream with no reconfigure, and no stdout at all each fail nothing.
  • The handler's boundaries. 17 in-process cases, in cp1252 and in utf-8, each through a TextIOWrapper and through str.encode: a raw byte before and after 日 in one run, the lowest and highest raw bytes, U+DC7F, U+DC00 and U+DC41 under the range, U+DD00 over it, a high surrogate, and one run holding every kind.
  • No PYTHONIOENCODING at all. Two tests give the child none, and take what Python itself chooses for a redirected stdout. A first child reports that stdout's codec and handler. The test skips, with the reason, when the handler is not surrogateescape or the code page holds 日本, so it never fails on a platform it does not test. Then plan runs on 日本.md and 東京.ipynb, its stdout decoded strictly in the probed codec, and exits 0 with the name escaped.
    • POSIX: LC_ALL=C PYTHONUTF8=0, which gives ascii:surrogateescape (PYTHONUTF8=0 because the C locale turns UTF-8 mode on, and 3.15 does everywhere). It runs on Linux, and on macOS when its C locale gives the same.
    • Windows: PYTHONUTF8=0 (the job sets PYTHONUTF8=1), which gives the ANSI code page with surrogateescape. It has not been run on Windows by the author: only this PR's CI runs it. A Windows whose code page holds 日本 (cp932 does) skips it.
  • The tests run on Windows CI too: 日本.md is a legal NTFS name, and PYTHONIOENCODING overrides the job's PYTHONUTF8=1.

• They fail where the bug is. On 3006303 the 29 tests of round 1 give 12 failed, 17 passed. The failures are every cp1252:surrogateescape run (7 tool runs, plan, and the raw-byte cp1252 case), each with UnicodeEncodeError, plus the three in-process tests that need the new handler. The file as it stands gives 30 failed, 17 passed, 1 skipped there: those 12, the POSIX default test (exit 1, UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1), and the 17 handler cases, which fail there for want of the handler and so say nothing about their boundaries. The mutants below do. The first test file, 11 tests, failed 10 of 11 on 20e13f0; the utf-8 plan control passed.

• Mutations. 14 mutants were run against the test file, and 13 were killed:

  • catching only AttributeError, dropping OSError, and dropping ValueError;
  • mapping surrogateescape to plain backslashreplace;
  • leaving surrogateescape alone;
  • registering on every call;
  • not splitting a run of escaped bytes from the characters after it;
  • writing escaped bytes as \udcff;
  • no call in the shim;
  • forcing the encoding to UTF-8;
  • the re-review's three on the handler's boundaries: the backslash stretch running to the end of the codec's span, so a raw byte after an unencodable character in one run comes out as \udcff and not as the byte; U+DC80 left out of the range; and U+DC00..U+DC7F let into it. The first 29 tests let all three through, and the new cases kill them (4, 5 and 8 failures, the other 39 to 43 tests still passing).

  The survivor moves the doctor's call after parse_args, and it is not observable. Before the first report line, only argparse writes to stdout, and only for --help. All 17 --help outputs (7 tools, 10 subcommands) are pure ASCII: all exit 0 under ascii:strict, with no byte above 0x7f. The doctor's prog is fixed, and its help interpolates nothing from the environment. A usage error goes to stderr, which is already backslashreplace.

• Same bytes where it worked. 17 commands, each run through a main checkout (20e13f0) and through 20e13f0 with this branch's diff applied. Both checkouts were equally dirty, and each checkout's own path was normalised. The commands: plan on four sources, check on three, plan with an --out path holding byte 0xff, adopt-project.py --help, the doctor on a non-shape root, on a project and with --json, update-shape.py check, the scaffold's and family.py init's dry runs, bump-leg.py --dry-run and the bootstrap shim. Each ran under five environments: utf-8:strict, the container's inherited locale (utf-8:surrogateescape), PYTHONUTF8=1, cp1252:strict and cp1252:surrogateescape, for 85 pairs.

  • Every pair in which main did not raise, 64 of them, is byte-identical in exit code, stdout, stderr and plan file.
  • In the 16 pairs where only main raised (UnicodeEncodeError every time), main's partial stdout is a byte prefix of the head's. Those 16 are 8 under cp1252:strict, 7 under cp1252:surrogateescape, and the 0xff --out path under utf-8:strict.
  • In the other 5, a source file named caf\xff.md makes both sides raise in write_lf writing the plan file (see Left out). Stdout and the plan are identical, and stderr is identical once traceback line numbers are masked.

• Targeted runs. At round 1's head 4d4e844: tests/test_shape_doctor.py 181 passed. In the second stream, 392 passed and 1 skipped: test_stdout_encoding 29, test_adopt_plan 56 (+1 skip, which needs a MedxEHR clone), test_shape_advisory 40, test_update_shape 61, test_family 76, test_bump_leg 23, test_scaffold_e2e 44, test_repo_hygiene 50 and test_cli_reference 13 (no --help text changed). In all, 573 passed, 1 skipped, 0 failed. At round 2's head 2d60c3e, which changes only the test file and the helper's docstring, test_stdout_encoding (48 tests), test_repo_hygiene and test_cli_reference give 110 passed, 1 skipped (the Windows-only test); the other files were not re-run. git diff --check is clean, and no added line holds a host-absolute path or a non-stdlib import.
  At the merge head 164fa16 (main 4835fb4 merged in), git-filter-repo 2.47.0 on PATH, the quick check over test_stdout_encoding (48), test_adopt_plan_escapes (59), test_yaml_flow_quotes (17), test_adopt_plan (57), test_repo_hygiene (50) and test_cli_reference (13) gives 242 passed, 2 skipped (the Windows-only test in test_stdout_encoding, and the MedxEHR-clone test in test_adopt_plan). Then test_shape_doctor gives 181 passed, and test_bump_leg (23), test_update_shape (77), test_family (76) and test_scaffold_e2e (44) give 220 passed. In all 643 passed, 2 skipped, 0 failed. git diff --check HEAD^1 HEAD is clean.
  At the merge head 657abb1 (main 39ac59c merged in), git-filter-repo 2.47.0 on PATH, the quick check over test_stdout_encoding (48), test_update_shape (77), test_update_shape_add (17), test_scaffold_e2e (44), test_repo_hygiene (50), test_cli_reference (13) and test_adopt_plan_escapes (59) gives 307 passed, 1 skipped (the Windows-only test in test_stdout_encoding). Then test_shape_doctor gives 181 passed, and test_bump_leg (23), test_family (76), test_scaffold_leg_path_collisions (104) and test_scaffold_mount_ignored (42) give 245 passed. In all 733 passed, 1 skipped, 0 failed over those 12 files. test_update_shape_add_ignored (update-shape.py apply --add stages the new file with a plain git add, so a path the operator's excludes ignore fails after the copies, the re-pin and the branch are written, nothing is rolled back, and check then exits 0 over a re-pin nobody committed #202's new file, 26 tests) and test_scaffold_ignored_shape_files (15 tests) also pass on 657abb1, 41 passed. git diff --check HEAD^1 HEAD is clean.

Left out

• A real bump-leg.py run, and update-shape.py apply, still fail under a code page that cannot hold the project's display name. Their child validate-manifest.py prints manifest ok: <name> with its own default handler. It is a shape copy and does not call the helper. update-shape.py's child inherits the tool's stdout, and bump-leg.py captures the child's output. Either way the child takes the stdout's configuration from the environment and not the parent's reconfigure. The fix belongs in the two tools' run_validator, the way shape-doctor.py:823 already sets PYTHONIOENCODING=utf-8 for the same validators. But shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230 (fix/decode-git-output-utf8) rewrites bump-leg.py's run_validator call, with the opposite decoding choice: the locale codec with errors="replace". A trial git merge-tree of the review's proposed change against shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230's head conflicts in scripts/bump-leg.py. So it is a follow-up that lands with or after shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230. Filed as bump-leg.py (a real bump) and update-shape.py apply run the project's own validators as children that cannot print its display name on a cp1252 stdout, so a project named 日本 is refused and rolled back as bump-leg-validators-red or update-validators-red under PYTHONIOENCODING=cp1252:surrogateescape, and the refusal blames the validators #254.
• scripts/validate-repository-naming.py, the project validators and templates/*/scripts/bootstrap.py are shape copies and were not changed. setup-project.py already sets its own streams, and its scaffold child now calls the helper itself (by reading, not run on Windows).
• An escaped name is wider than the name, and the report pads on the string's length, so \u65e5\u672c.md pushes the columns after it out of line. Cosmetic.
• backslashreplace does not escape a backslash. A name that literally holds the six characters \u65e5 therefore prints the same as a name holding 日. Unlikely, and it comes with any backslash escaping.
• A plan follow-up naming a code file with a non-ASCII name prints git's quoted octal (code/"src/\346\227\245\346\234\254.py", from git grep -l under core.quotepath). That is a separate defect. It is covered by adopt-project.py plan's cross_leg_references passes a spec directory's name to git grep -F with no -e and the code entries as glob pathspecs, so a spec directory named -notes or a code entry named :cache silently drops the follow-up, a code entry named * names files that are not in the code leg, a name outside ASCII is printed in git's quoted spelling, and a spec directory named -O<cmd> is run as git grep's pager when a tracked file holds the planned commit's hash #216 and PR adopt-project.py: plan's follow-up git grep takes a spec directory as its pattern and each code entry as a literal path, so a spec directory named -notes/ or a code entry named :cache/ no longer loses its follow-up, a code entry named */ no longer names a spec-leg file, a name outside ASCII is spelled instead of C-quoted, and a spec directory named -O<cmd>/ no longer runs a command (#216) #228, which reads git grep -z.
• A source file whose name is not UTF-8 makes plan raise in write_lf while writing the plan file, the same on main and on this head. adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199 fixes that. On the trial merge with adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199's a02240a, plan over caf\xff.md exits 0 under utf-8:strict, cp1252:strict and cp1252:surrogateescape. The plan holds path: "caf\udcff.md", and the table shows `caf\xff.md`.

Composition

• origin/main 39ac59c (scaffold-project.py accepts a leg path that collides with a path the shape writes (--spec-path contracts, --code-path project.yaml), creates all three repositories and pushes both legs, then dies: #172's shape half on the scaffold side #197 via scaffold-project.py: a leg path that is, holds or lies inside a path the shape writes into the assembly root, a --pin's contracts/<product>-pin.yaml included, is refused before anything is created, under --dry-run alike, through the one definition adopt-project.py asks, so --spec-path contracts no longer creates three repositories and pushes both legs to die at git submodule add (#197) #234, update-shape.py apply --add stages the new file with a plain git add, so a path the operator's excludes ignore fails after the copies, the re-pin and the branch are written, nothing is rolled back, and check then exits 0 over a re-pin nobody committed #202 via update-shape.py: apply --add force-stages its additions past the operator's own excludes, and a branch, stage or commit that fails before the commit lands now puts back the copies, the pin, the README line, the index and the branch and names the step, so check no longer exits 0 over an uncommitted re-pin (#202) #233, scaffold-project.py _build_assembly_root runs git submodule add without --force, so a leg path an ignore rule matches (--code-path __pycache__ or .DS_Store under the shape's own .gitignore, or the default spec under an operator's global excludes) passes --dry-run, then dies at the mount AFTER all three repositories exist and both legs are pushed #229 via scaffold-project.py: the leg mount passes --force to git submodule add, so a leg path an ignore rule matches (__pycache__, .DS_Store, or the default spec under an operator's own excludes) scaffolds instead of dying at the mount after all three repositories exist and both legs are pushed (#229) #235; before them adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199 via adopt-project.py: the plan writer escapes every character that is not printable from the one table repo_shape's reader resolves, so a source holding the macOS Icon\r no longer writes a plan check refuses, and a non-UTF-8 name no longer empties the plan (#199) #247, scripts/repo_shape.py _split_flow ignores a backslash inside a double-quoted flow item, so pins: ["x\",y"] reads back as two items x\ and y" where the same text as a block sequence item reads as the one item x",y #213 via repo_shape.py: _split_flow and _split_key_flow take the character after a backslash inside a double-quoted flow item, so pins: ["x\",y"] reads as the one item x",y its block twin reads, not as x\ and y" (#213) #244, update-shape.py apply rewrites the manifest's shape: block with fixed two- and four-space indents, so a project.yaml indented four spaces, which the project's own reader and both validators accept, is rewritten into YAML that reader rejects, and the rolled-back refusal relays the validators' lockstep remedy #208). This branch carries main 39ac59c, as merge commit 657abb1 (parents 164fa16 and 39ac59c), so the PR no longer conflicts. 164fa16 had carried main 4835fb4 the same way. Each merge had exactly one conflict and both sides are kept whole. At 164fa16 it was the add/add hunk in scripts/shape_materialize.py right after write_lf: adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199's write_paths_listing and this PR's escape_what_stdout_cannot_encode, with its handler, sit side by side. At 657abb1 it is the from shape_materialize import (...) list in update-shape.py: this PR adds escape_what_stdout_cannot_encode and update-shape.py apply --add stages the new file with a plain git add, so a path the operator's excludes ignore fails after the copies, the re-pin and the branch are written, nothing is rolled back, and check then exits 0 over a re-pin nobody committed #202 added force_stage, so the list holds both, in its existing order, re-wrapped to 79 columns like its neighbours. No other file conflicted at either merge, and no code was written for them. adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOS Icon\r file), so check refuses the whole plan as yaml-unparsable with no entry to edit #199's own tests, tests/test_adopt_plan_escapes.py, and main's tests/test_update_shape.py, tests/test_update_shape_add.py, tests/test_scaffold_leg_path_collisions.py and tests/test_scaffold_mount_ignored.py pass together with this PR's tests/test_stdout_encoding.py on the merge (see Targeted runs).
• shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230 (fbdf8db). git merge-tree --write-tree is clean at 2d60c3e, tree 7718902ebc43. On the merged tree at 4d4e844 (tree 823450fb125a), shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230's tests/test_git_output_decoding.py (its guard against text-mode calls with no codec) and tests/test_stdout_encoding.py gave 32 passed, not re-run at 2d60c3e. This change adds no subprocess call.

🤖 Generated with Claude Code

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[brettheap]

(lane openRepoShape-5, 2026-10-11T20:54Z) @codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[sourcery-ai]

Reviewer's Guide

This PR prevents affected commands from failing after producing output when stdout cannot encode paths or user-provided names by safely adapting only strict and surrogateescape error handling, while preserving the stream encoding and adding comprehensive cross-platform regression tests.

Sequence diagram for safe stdout encoding in CLI tools

sequenceDiagram
    participant User
    participant ToolMain
    participant StdoutHelper as shape_materialize.escape_what_stdout_cannot_encode
    participant Stdout
    participant Report

    User->>ToolMain: main(argv)
    ToolMain->>StdoutHelper: escape_what_stdout_cannot_encode()
    StdoutHelper->>Stdout: inspect errors
    alt errors is strict
        StdoutHelper->>Stdout: reconfigure(errors=backslashreplace)
    else errors is surrogateescape
        StdoutHelper->>Stdout: reconfigure(errors=openreposhape.surrogateescape-then-backslashreplace)
    else other handler or unavailable stream
        StdoutHelper-->>ToolMain: leave stdout unchanged
    end
    ToolMain->>Report: print path or user-provided name
    Report->>Stdout: encode using original stream encoding
    Stdout-->>User: encoded name or escaped name, command continues

Loading

File-Level Changes

Change	Details	Files
Added centralized stdout reconfiguration that preserves the selected encoding while preventing unrepresentable names from terminating commands.	Map strict output to backslashreplace. Register a repository-scoped handler that preserves surrogateescaped bytes and backslash-escapes other unencodable characters. Ignore unsupported, closed, or unavailable stdout streams and leave all other error handlers unchanged.	scripts/shape_materialize.py
Enabled the stdout-safety helper at the start of each affected command and the repository bootstrap shim.	Install the handling before argument parsing and reporting, including help paths. Cover adopt-project, shape-doctor, update-shape, scaffold-project, family, bump-leg, and bootstrap execution.	adopt-project.py shape-doctor.py update-shape.py scaffold-project.py scripts/family.py scripts/bump-leg.py bootstrap
Added regression coverage for encoding failures, byte preservation, handler behavior, and command-wide compatibility.	Exercise cp1252 strict and surrogateescape alongside UTF-8 across all affected tools. Verify plans and reports complete with escaped Unicode names and preserve raw surrogateescaped filesystem bytes. Verify registration is idempotent, unrelated handlers are unchanged, reconfiguration failures are harmless, and previously successful output remains compatible.	tests/test_stdout_encoding.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#214	Ensure adopt-project.py plan completes successfully after writing the plan when report names cannot be encoded by stdout, displaying those names escaped and still printing the complete report, including plan written to and follow-up questions.	✅
#214	Apply a robust stdout encoding fallback across the relevant tools so unencodable names do not cause UnicodeEncodeError, while preserving the configured stdout encoding and existing behavior for handlers that already work.	✅
#214	Add regression coverage for strict and Windows-like surrogateescape stdout behavior, including the affected plan report and related tools.	✅

Possibly linked issues

• adopt-project.py plan raises UnicodeEncodeError in its terminal report, after the plan file is written, for a printable name stdout cannot encode (日本.md under cp1252, as on a redirected Windows stdout), so plan exits 1 before plan written to #214: The PR directly fixes issue 214 by escaping unencodable stdout characters and ensuring plan completes successfully.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

(lane openRepoShape-5, 2026-10-11T21:07Z) Review trail for this PR.

• 3006303: adversarial review, CHANGES NEEDED. B1 was the blocker: Windows' default stdio handler is surrogateescape, not strict, so the strict-only guard left a redirected cp1252 stdout unfixed.
• 4d4e844: fix round 1 adds a handler registered as openreposhape.surrogateescape-then-backslashreplace, which keeps surrogate-escaped bytes raw and backslash-escapes everything else. S1 is scoped out because it collides with shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230's run_validator; a follow-up is being filed. Re-review: APPROVE WITH NITS. B1 is fixed, and the handler checks pass on CPython 3.10 to 3.15rc2.
• 2d60c3e: fix round 2 answers R1-R4:
  • 17 handler-boundary cases kill the three surviving mutants;
  • a test with no PYTHONIOENCODING at all (POSIX: LC_ALL=C PYTHONUTF8=0), which fails on 3006303 and passes now;
  • a Windows-only no-PYTHONIOENCODING test, which only this PR's tests-windows job runs;
  • wording fixes.
    tests/test_stdout_encoding.py: 47 passed, 1 skipped (the Windows test) here.

This PR conflicts with #247 (#199) in scripts/shape_materialize.py, where both add a function after write_lf; whichever lands second keeps both. It merges cleanly with main and with #230. Every report is in opensoft/brett-wip handoffs/openRepoShape/attachments/lane-openRepoShape-5/ (review214-report.md, fix214-report.md, rereview214-report.md; fix214b-report.md to follow).

[brettheap]

(lane openRepoShape-5, 2026-10-11T21:07Z) @codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[brettheap]

(lane openRepoShape-5, 2026-10-11T21:34Z) @codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 164fa16c89

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T21:39:22.684514Z	164fa16	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[brettheap]

(lane openRepoShape-4, 2026-10-11T22:05:55Z) Landing note: our #233 (#202) landed as 150c41d, and main is now 39ac59c. This PR now conflicts with main in one hunk only: update-shape.py's from shape_materialize import (...) list. Main added force_stage and this PR adds escape_what_stdout_cannot_encode. The resolution is the union. Replace everything between the markers with:

    commit_trailers, escape_what_stdout_cannot_encode, force_stage,
    lane_trailer_argument, readme_shape_lines,

Leave the next line, readme_trailing_shape_line, run, trailer_line,, as it is. Everything else merges by itself. Each tool's main() still calls escape_what_stdout_cannot_encode() first, so the new prints from #233 and #234 are covered. --help is unchanged (render-cli-reference.py --check reports ok).

We applied this in scratch on 39ac59c + 164fa16. tests/test_stdout_encoding.py tests/test_update_shape.py tests/test_update_shape_add_ignored.py tests/test_cli_reference.py: 163 passed, 1 skipped. The patch against this branch is on Eagle (py-bench) at /tmp/claude-1000/-workspace-projects-openRepoShape/83338f75-6338-4d51-9749-89179ecaf55f/scratchpad/l4cf/resolved-246-vs-pr.patch.