[brettheap]

setup-project.py and the assembly root's bootstrap.py: every Python child is told the encoding its words are read in, so a cp1252 stdout gets one encoding from setup, and make bootstrap passes a project whose name the code page cannot hold

Lane: openreposhape-2 (openRepoShape-2)
Claim: #836 (comment)

Stacked on

• Lane 5's tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 (tests: the quoting-sentence inventory misses a quoter passed as a value and pins functions where it means calls, and the launcher guard has no row for src/openreposhape/cli.py or setup-project.py #759), fix/759-quoting-launcher-guards, head 08a6ed39634f9cf34edcd2a453adde1bd682fb23 (checked on origin before the merge). Merged with --no-ff as db693114: "Merge openRepoShape-5's tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 head 08a6ed3 (fix/759-quoting-launcher-guards) as the stacked base of setup-project.py's run and the assembly root's bootstrap.py _run start Python children with no environment, so a cp1252 stdout gets two encodings from setup and a name it cannot hold ends make bootstrap in validate-manifest.py #836". No conflict: tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 touches only tests/test_validator_child_encoding.py and tests/test_readme_windows_quoting_sentence.py.

• tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 has landed as 677f9bfa, the squash of that same head: git diff 08a6ed39 677f9bfa over its two files is empty. Merging 08a6ed39 again was "Already up to date". The branch then merged that squash, then main:

  • 61eceb41: "Merge lane 5's tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 squash 677f9bf (its final head 08a6ed3 is merged at db69311)". Its one conflict was tests/test_validator_child_encoding.py. The squash's side of it is 08a6ed39's file byte for byte, and the branch's side is that file plus this PR's change, so the branch's side was kept. git diff 90cb256b 61eceb41 over tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808's two files is empty. The merge also brought main from d66a9054 to 677f9bfa, including bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 (below).
  • dad55d2d: "Merge origin/main (c100843) into fix/836-child-environment after tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 landed as 677f9bf", with no conflict. It brought shape-doctor.py: a conflicted shape pin's NEXT takes the side whose pin reads, and names no line where no side's pin reads, so run as printed it no longer leaves the doctor naming the same NEXT again (#752) #779 (shape-doctor.py and tests/test_doctor_pin_restore_merges.py), which touches none of this PR's files.

• Review the commit after db693114: 90cb256b. git diff origin/main dad55d2d is that commit's change. It covers the same three files, with the same 470 changed lines at -U0: the two -U0 line lists diff empty, and the -U0 patch-ids are equal. Only the context in templates/assembly-root/scripts/bootstrap.py differs: bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 added lines above this PR's hunks, which now sit 6 and 10 lines lower.

• Overlaps, each git merge-tree --write-tree of the head dad55d2d with theirs, all clean:

  • lane 5's bootstrap.py: a validator's exit 2 ends bootstrap REFUSED, exit 2, an unreadable upstream path, cache or pin is no answer and a reftable leg's cherry-pick or revert is read from its ref store, so no traceback or ended pick follows (#764) #815 (bootstrap.py (the assembly root's): a SHAPE_UPSTREAM_PATH directory nobody may enter ends in a traceback, a validator's refusal is reported as FAILED, exit 1, a reftable leg's cherry-pick goes unseen, and docs/cli.md says bootstrap places each leg AT the pin with no condition #764, now 05a1ebb3, was f6f318d1), which edits the assembly bootstrap.py;
  • repo_shape.py, setup-project.py, shape-doctor.py, README and the handbook: under Git Bash (MSYSTEM set) the lines that follow the host are quoted for sh, through one question, host_shell_dialect, so the doctor's lines, setup's family lines and family.py's NEXT, switch -c, add and cd lines paste whole in the shell that printed them, and README's Windows sentence says so #821 (7581ffb9), setup-project.py: a refused value names the check it failed, a project name over 95 characters is refused before step (4), and the prompt takes the spaces off a typed name, so Atlas no longer hides in step (4) (#777) #811 (050abea6) and setup-project.py: a project name typed at the prompt is checked as --project is and an empty line says no name was typed, so ../x, a/b, .. and --help are refused by name, not by an --id hint, a help page or a probe write (#729) #756 (now 892ffd97, was b3c77cc2), which edit setup-project.py far from run.

  Lane 4's bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 (bootstrap.py and repo_shape.py: a leg on a branch whose name is not UTF-8 makes every later make bootstrap end in a UnicodeDecodeError, and a character the filesystem encoding lacks still ends step (a) in a traceback #710) has landed as 0dee22b3, and this branch holds it through 677f9bfa. Its bootstrap.py merged without a conflict. _run is still that file's only Python child (templates/assembly-root/scripts/bootstrap.py:160); bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824's git children go through _git_says, and PYTHONIOENCODING does not reach git. Whichever of this PR and bootstrap.py: a validator's exit 2 ends bootstrap REFUSED, exit 2, an unreadable upstream path, cache or pin is no answer and a reftable leg's cherry-pick or revert is read from its ref store, so no traceback or ended pick follows (#764) #815 lands second re-gates against the other. The helper's first placement, directly after VALIDATORS, conflicted with bootstrap.py: a validator's exit 2 ends bootstrap REFUSED, exit 2, an unreadable upstream path, cache or pin is no answer and a reftable leg's cherry-pick or revert is read from its ref store, so no traceback or ended pick follows (#764) #815's VALIDATOR_REFUSED = 2 at the same spot. It sits below _run instead, directly above bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824's _try_git.

What changed

1. setup-project.py: every child of run is told what its words are read in. main reconfigures stdout and stderr to UTF-8, and a child does not inherit a reconfigure: a Python child takes its streams' encoding from its own environment. So under PYTHONIOENCODING=cp1252:strict or cp1252:surrogateescape (Windows' ANSI code page on a stdout that is not the console), five children wrote cp1252 into the UTF-8 stream, and the sixth, the plan run again to read why it was refused, wrote cp1252 into a pipe run decodes as UTF-8.

• child_environment(capture) is bump-leg.py (a real bump) and update-shape.py apply run the project's own validators as children that cannot print its display name on a cp1252 stdout, so a project named 日本 is refused and rolled back as bump-leg-validators-red or update-validators-red under PYTHONIOENCODING=cp1252:surrogateescape, and the refusal blames the validators #254's rule, as scripts/shape_materialize.py writes it in inherited_child_environment and captured_child_environment: an inherited child is told the stdout's own encoding (UTF-8 after main) with its handler where that handler never raises, and backslashreplace where it does; a captured child is told CAPTURED_CHILD_IOENCODING (utf-8:backslashreplace), what run decodes. No usable stdout leaves the environment as it is.
• run passes env=child_environment(capture) in its own subprocess.run, so all six Python children get it: self_bootstrap (the checkout's own copy of the file), check_names (the naming validator), plan_and_confirm (the plan), why_the_plan_did_not_go_through (the plan again, captured), scaffold and clone_and_bootstrap (scripts/bootstrap.py). The next child added goes through run too. git and gh get it as well; PYTHONIOENCODING is read by Python alone.
• A copy, not an import: the file runs before any checkout of the standard exists, and run is what clones one. main's comment and the module docstring said a child's output "travels through this process's streams"; an inherited child's does not, and both now say so.

2. The assembly root's scripts/bootstrap.py: the validators are told what bootstrap's stdout is. _run passes env=inherited_child_environment(), a copy of the library's: a project does not carry shape_materialize.py, and a pinned script that imported a new name out of repo_shape.py would be refused at update-shape.py apply (update-accept-local-missing-name, #536, tests/test_update_shape_kept_helper.py), so the copy keeps repo_shape.py untouched, as the validators keep their own _shown. This is a shape bump: every project re-pins scripts/bootstrap.py. No digest is committed anywhere in the repository; a scaffold computes contracts/shape-pin.yaml from the bytes it copies.

3. tests/test_validator_child_encoding.py. The setup-project.py row leaves NOT_YET (which is now empty, its comment kept), LAUNCHERS gains templates/assembly-root/scripts/bootstrap.py: inherited_child_environment, and the tests below are new.

No message's wording changes. The bytes do, measured by a probe (setup-project.py --project Sample --yes --org demoorg --local-remote-dir <d> --into <d>, a gh stub that exits 97 first on PATH, GH_TOKEN=bogus, an empty GH_CONFIG_DIR):

setup-project.py, PYTHONIOENCODING=cp1252:strict (cp1252:surrogateescape: the same)
  before (db693114): exit 0, 129 stdout lines, 9 not UTF-8 (24 31 38 45 64 79 80 100 102)
    line 24: ...at the root of the repository \x97 `kind: family-manifest`...     (U+2014 in cp1252)
    line 45: shape        opensoft/openRepoShape @ db693114a950 (tree e67601b2f778\x85)
  after:            exit 0, 129 stdout lines, 0 not UTF-8
    line 24: ...at the root of the repository \xe2\x80\x94 `kind: family-manifest`...
    line 45: shape        opensoft/openRepoShape @ db693114a950 (tree e67601b2f778\xe2\x80\xa6)
  utf-8:strict, before and after: exit 0, 129 lines, 0 not UTF-8

scripts/bootstrap.py of a scaffolded project with `name: "日本"`, PYTHONIOENCODING=cp1252:strict
  before: bootstrap: 日本 (<root>)
          ...
            --- manifest ---
          Traceback (most recent call last):
            ...
          UnicodeEncodeError: 'charmap' codec can't encode characters in position 13-14: character maps to <undefined>
          bootstrap FAILED: manifest (exit 1)                                     exit 1
  after:  bootstrap: 日本 (<root>)
          ...
            --- manifest ---
          manifest ok: 日本 (sample), 3 legs
          ...
          bootstrap ok                                                            exit 0
  cp1252:surrogateescape: as cp1252:strict. cp1252:replace: `manifest ok: ?? (sample), 3 legs`,
  exit 0, before and after. utf-8:strict: `manifest ok: 日本 (sample), 3 legs`, exit 0, before and after.

A new bootstrap.py copied by hand into a project scaffolded before it is refused shape-copy-drift by that project's own pin, as it should be: the change reaches a project through update-shape.py apply.

Tests

file	test	what it pins
tests/test_validator_child_encoding.py	test_each_copy_tells_an_inherited_child_what_the_library_does (2 copies x 14 INHERITED streams)	each copy (COPIES: setup-project.py's child_environment, the bootstrap's inherited_child_environment) builds the very environment shape_materialize.inherited_child_environment builds, read from the stdout it runs under
	test_each_copy_leaves_the_environment_alone_with_no_stdout (2 x 3)	no stdout, no encoding, an encoding no codec answers to: PYTHONIOENCODING left as it was
	test_each_copy_keeps_the_handlers_the_library_keeps (2)	HANDLERS_THAT_NEVER_RAISE equals the library's _HANDLERS_THAT_NEVER_RAISE
	test_setup_projects_copy_tells_a_captured_child_what_the_library_does	child_environment(capture=True) equals captured_child_environment(), under an operator's cp1252:strict
	test_setup_projects_run_tells_each_child_what_its_words_are_read_in (3)	run itself: an inherited child is told utf-8:replace (the stdout main leaves) or cp1252:backslashreplace (one it could not reconfigure); a captured one utf-8:backslashreplace, and run reads manifest ok: 日本 back
	test_setup_writes_one_encoding_whatever_its_stdout_is (cp1252:strict, cp1252:surrogateescape)	the issue's first line, #808's probe: exit 0, every stdout and stderr line UTF-8, U+2014 and U+2026 in it as UTF-8
	test_bootstrap_runs_the_validators_of_a_project_named_in_japanese (5 STDOUTS)	the issue's second line: exit 0, bootstrap ok, and bootstrap: <name> and manifest ok: <name> written alike (escaped under cp1252, ?? under replace, as is under cp932 and UTF-8)
	test_make_bootstrap_on_a_project_named_in_japanese_under_cp1252	the same through make bootstrap
	test_a_tool_whose_child_inherits_never_changes_its_stdouts_encoding (INHERITING: siblings.py)	the issue's third line for siblings.py: the guard reads its run, the run hands no env=, and nothing in the file reconfigures a stream's encoding
	test_the_encoding_reader_sees_the_tool_that_changes_it	encoding_changes finds setup-project.py's one reconfigure(encoding=...), so the empty answer above is a real one
	LAUNCHERS rows [templates/assembly-root/scripts/bootstrap.py] (2)	the guard reads the bootstrap's run, and it hands inherited_child_environment()
	LAUNCHERS row [setup-project.py]	was a strict xfail; passes now, out of NOT_YET

On the base. The new file over db693114's tools: 46 of the 52 new cases fail (10 failed, 36 errors where the copies do not exist yet), and the setup-project.py row, a strict xfail there, fails outright. The 6 that pass there are the bootstrap run's three control encodings (cp1252:replace, cp932:strict, utf-8:strict), the siblings.py pin and the encoding reader (both pin what is already true), and the guard reading the bootstrap's run.

Mutants, one line each, each reverted at once; every one fails at least one new test:

mutant	killed by
S1 env=child_environment(capture) if capture else None (the captured child only)	test_setup_writes_one_encoding... x2, test_setup_projects_run...[inherited] and [inherited-not-reconfigured]. The LAUNCHERS row passes it (see Not changed)
S2 env= dropped from run	the setup-project.py row, the end-to-end run x2, run x3
S3 env=child_environment() (the capture flag ignored)	test_setup_projects_run...[captured]
S4 / B2 encoding = sys.stdout.encoding (no canonical name)	INHERITED rows ANSI_X3.4-1968 and latin-1, and two no-stdout cases, for that copy
S5 / B3 a raising handler passed on (errors = errors)	7 INHERITED rows for that copy. The end-to-end runs do not see B3: escape_unencodable_stdout has already made the bootstrap's stdout backslashreplace
S6 "replace" dropped from the handler set	test_each_copy_keeps_the_handlers...[setup-project.py], 2 INHERITED rows
S7 CAPTURED_CHILD_IOENCODING = "utf-8:replace"	the captured copy test, run[captured]
S8 TypeError no longer caught	no-encoding
B1 env= dropped from _run	the bootstrap row, the bootstrap end-to-end runs under both cp1252 handlers, make bootstrap
B4 the bootstrap's child told utf-8:backslashreplace (the captured rule)	12 INHERITED rows, 4 end-to-end runs, make bootstrap
I1 env=dict(os.environ) on siblings.py's run	test_a_tool_whose_child_inherits...[siblings.py]
I2 sys.stdout.reconfigure(encoding='utf-8') in siblings.py's main	the same
G1 the setup-project.py row put back in NOT_YET	the row (XPASS(strict))

Runs (/usr/bin/python3 -m pytest -q -p no:cacheprovider -o tmp_path_retention_policy=failed, GH_TOKEN unset, an empty GH_CONFIG_DIR), the same 95 files in 8 batches of 12: every file git grep -l -E '_run\(|child_environment|child_env|PYTHONIOENCODING|LAUNCHERS|NOT_YET' -- tests names, every test file naming setup-project.py or the assembly bootstrap.py, the issue's Verify files, test_pinned_copy_text.py, test_family_printed_quoting.py, test_repo_hygiene.py, test_cli_reference.py and test_agents_md_codes.py:

• before, on db693114 (a scratch clone): 6540 passed, 5 skipped, 2 xfailed, nothing red;
• after, on 90cb256b: batches 1 to 7 (84 files): 5932 passed, 5 skipped, 1 xfailed, the same counts as the base's batches 1 to 7, nothing red. Batch 8 (the last 11 files: the six test_update_shape*.py, test_validate_pins_lockstep_lines.py, test_validate_pins_without_git.py, test_validator_child_encoding.py, test_windows_paths.py, test_workflow_concurrency.py): 661 passed, nothing red; its base run was 608 passed, 1 xfailed, so the setup-project.py xfail is now a pass and the 52 new cases are the rest. All 8 batches, 95 files: 6593 passed, 5 skipped, 1 xfailed (the one xfail left is batch 7's, as on the base). test_validator_child_encoding.py was run on its own on these bytes: 163 passed.
• after the restack (the same command, a narrower set; the whole-suite gate covers the rest):
  • on 61eceb41 (tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808's squash and bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 merged), 18 files in 2 batches: test_validator_child_encoding.py, test_readme_windows_quoting_sentence.py, test_repo_hygiene.py, test_cli_reference.py, test_agents_md_codes.py, test_git_output_decoding.py (it reads every tool's subprocess calls, these two files' among them), test_openreposhape_package.py (it names a child_environment, the package's own), the issue's other Verify files (test_setup_project_py.py, test_setup_sh.py, test_bootstrap_without_git.py, test_bootstrap_after_bump.py), four more test_bootstrap_*.py (bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824's two among them), test_update_shape_kept_helper.py, test_pinned_copy_text.py and test_stdout_encoding.py: 1421 passed, 1 skipped (test_stdout_encoding.py:286, Windows only), nothing red;
  • on dad55d2d (shape-doctor.py: a conflicted shape pin's NEXT takes the side whose pin reads, and names no line where no side's pin reads, so run as printed it no longer leaves the doctor naming the same NEXT again (#752) #779 merged): test_validator_child_encoding.py (still 163 cases), test_readme_windows_quoting_sentence.py, test_repo_hygiene.py, test_cli_reference.py, test_agents_md_codes.py and shape-doctor.py: a conflicted shape pin's NEXT takes the side whose pin reads, and names no line where no side's pin reads, so run as printed it no longer leaves the doctor naming the same NEXT again (#752) #779's test_doctor_pin_restore_merges.py: 376 passed, nothing red.

tests/test_validator_child_encoding.py alone: 111 cases before (110 passed, 1 xfailed), 163 after (163 passed).

Not changed

• templates/family-root/scripts/siblings.py: measured, no defect, no change. run_member_bootstrap (:567) run on a member under cp1252:strict, cp1252:surrogateescape and utf-8:strict, with a member carrying the old bootstrap.py and then the new: every line of the run is in the stdout's own encoding, the parent and the member both at <encoding>:backslashreplace. The child is given the very environment the parent's stdout was configured from, and escape_unencodable_stdout changes only the handler, which the member's bootstrap changes the same way. The only failure, with the old member, is that member's own validator: this PR's second line. It has no LAUNCHERS row: a row there names a helper the environment must name, and this child is right because it is handed none. So it is in INHERITING, whose test reads the file's AST and not its bytes, so the next edit to the pinned copy is no re-pin of the test.
• tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808's guard logic. interpreter_runs, environment_of and the row test are as tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 left them. One limit, measured: the row reads the helper's name in the source text, so it passes env=child_environment(capture) if capture else None (S1), the captured-child-only fix lane 2's rv808 SHOULD warned of, with five children bare. The end-to-end run and the run test kill it. A guard that reads both branches of a conditional env= is tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808's to decide.
• scripts/repo_shape.py: no helper moved there (the skew rule, and repo_shape.py, shape_materialize.py and adopt-project.py: a value the YAML reader refuses for opening with an indicator is named by its line and says to double-quote it or reword it, plan --org @x names --org again under the reader's own remedy, and check names one bad tracking_branch once (#708) #817, repo_shape.py, setup-project.py, shape-doctor.py, README and the handbook: under Git Bash (MSYSTEM set) the lines that follow the host are quoted for sh, through one question, host_shell_dialect, so the doctor's lines, setup's family lines and family.py's NEXT, switch -c, add and cd lines paste whole in the shell that printed them, and README's Windows sentence says so #821, repo_shape.py strips only YAML's space and tab, so a value a Unicode space opens reads as PyYAML reads it, and validate-manifest, bootstrap, bump-leg and update-shape refuse a branch holding a character that does not print or is white space (#715) #823 and validate-pins.py and repo_shape.py: a pin, the manifest or a shape copy the gate cannot reach or read is refused by name, relative to the root, once and with its cause's advice, and every name and path it prints is spelled (#712) #827 edit it).
• templates/family-root/scripts/bootstrap.py starts make in members (:517), not the interpreter, and never changes its stdout's encoding either.
• No --help text, README, handbook, AGENTS.md, AGENTS-shape.md or docs/cli.md changes.

Closes #836

🤖 Generated with Claude Code

Summary by Sourcery

Propagate each parent process's effective output encoding to Python children so setup and bootstrap remain reliable across non-UTF-8 environments.

Bug Fixes:

• Ensure setup-project.py and assembly-root bootstrap validators use consistent child-process encodings, preventing mixed-encoding output and failures for project names unsupported by the active code page.

Enhancements:

• Add encoding-aware child environment handling for inherited and captured subprocess output in setup-project.py and for validator subprocesses in the assembly bootstrap script.
• Expand validator encoding coverage with unit, integration, make-bootstrap, and regression tests, including projects named with Japanese characters.

Tests:

• Add comprehensive tests validating copied encoding rules, subprocess environments, UTF-8 setup output, bootstrap behavior, and unchanged inheriting-child behavior.

[sourcery-ai]

Reviewer's Guide

The PR makes child-process output encoding explicit across setup-project.py and the assembly-root bootstrap, preventing mixed UTF-8/cp1252 output and validator failures for unrepresentable project names, while adding comprehensive behavioral and static-analysis tests and extending the Windows quoting guard.

Sequence diagram for consistent child-process encoding

sequenceDiagram
    participant Setup as setup-project.py
    participant Child as Python child
    participant Stream as stdout or captured pipe

    Setup->>Setup: main reconfigures stdout and stderr to UTF-8
    Setup->>Setup: child_environment(capture)
    Setup->>Child: subprocess.run(env=child_environment(capture))
    Child->>Stream: write using PYTHONIOENCODING
    alt captured child
        Setup->>Stream: decode as UTF-8 with errors=replace
    else inherited child
        Child->>Stream: use stdout encoding and safe error handler
    end

Loading

Sequence diagram for encoding-safe assembly bootstrap validators

sequenceDiagram
    participant Bootstrap as scripts/bootstrap.py
    participant Validator as Validator
    participant Stdout as Bootstrap stdout

    Bootstrap->>Bootstrap: inherited_child_environment()
    Bootstrap->>Validator: subprocess.run(env=inherited_child_environment())
    Validator->>Stdout: write using bootstrap stdout encoding
    Stdout-->>Bootstrap: consistent validator output
    Bootstrap-->>Stdout: bootstrap status and validation messages

Loading

Flow diagram for captured and inherited child environment selection

flowchart TD
    Start[run child process] --> Capture{capture output?}
    Capture -->|yes| Captured[Set PYTHONIOENCODING=utf-8:backslashreplace]
    Capture -->|no| Encoding[Read sys.stdout.encoding]
    Encoding --> Usable{Usable encoding?}
    Usable -->|no| Original[Keep environment unchanged]
    Usable -->|yes| Handler{Handler never raises?}
    Handler -->|yes| Preserve[Keep stream handler]
    Handler -->|no| Safe[Use backslashreplace]
    Preserve --> Inherited[Set encoding and handler in child environment]
    Safe --> Inherited
    Captured --> Launch[Launch subprocess]
    Original --> Launch
    Inherited --> Launch

Loading

File-Level Changes

Change	Details	Files
Propagate the effective output encoding to every child process launched by the project setup script.	Added a standalone child-environment helper that mirrors the library’s inherited and captured I/O encoding rules, including codec normalization and non-raising error-handler fallback. Updated the common subprocess wrapper to pass the environment to all children, including the captured plan diagnostic run. Clarified documentation describing why parent stream reconfiguration is not inherited by Python children.	setup-project.py
Make the assembly-root bootstrap configure validator children consistently with its own stdout.	Added a pinned standalone copy of the inherited-child environment logic without importing newer library code. Passed that environment to validator subprocesses so non-ASCII project names succeed under incompatible code pages. Established the corresponding shape bump for scaffolded projects.	templates/assembly-root/scripts/bootstrap.py
Expand encoding and launcher validation to cover the new copied helpers and subprocess paths.	Added unit, subprocess, end-to-end, and make-bootstrap coverage for inherited and captured encodings, absent streams, handler behavior, cp1252, and Japanese project names. Added checks that each helper copy matches the library rule and that launcher analysis recognizes setup-project and assembly bootstrap child-process spellings. Removed the setup-project launcher from the pending-failure list and added the assembly bootstrap helper row.	tests/test_validator_child_encoding.py
Strengthen the Windows quoting guard’s static analysis and account for use counts.	Recognized annotated aliases and host-quoting functions passed as values, while excluding POSIX-directed uses and unsupported alias forms. Tracked host-dialect use counts per function and updated the expected setup-project and family-root inventories.	tests/test_readme_windows_quoting_sentence.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#836	Make every Python child started through setup-project.py's run use an encoding compatible with the parent’s output: UTF-8 for inherited output after reconfiguration, and UTF-8 with backslashreplace for captured output decoded as UTF-8.	✅
#836	Have the assembly-root bootstrap.py pass its validators an environment matching the bootstrap’s stdout encoding and non-raising error handler, so names unrepresentable in the code page do not make validation fail.	✅
#836	Update the child-encoding guard and tests to cover the fixed launchers and verify that the family-root siblings.py, which intentionally inherits its environment without changing stdout encoding, remains consistent.	✅

Possibly linked issues

• setup-project.py's run and the assembly root's bootstrap.py _run start Python children with no environment, so a cp1252 stdout gets two encodings from setup and a name it cannot hold ends make bootstrap in validate-manifest.py #836: The PR implements both requested environment fixes, removes setup-project.py’s NOT_YET guard, and adds regression tests.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

(lane openRepoShape-2, 2026-10-17T21:26:10Z) Independent review of #851 (#836) at 90cb256b, for lane openRepoShape-2: READY

I found nothing at MUST or SHOULD level. Every finding below is a NIT, plus one hand-off to lane 5 about #808's guard.

Findings

• NIT 1: the PR body and the commit message (lines 27-28) say more about the skew rule than it does.
  • They say tests/test_pinned_copy_text.py::test_no_pinned_script_imports_what_an_older_library_lacks stops a pinned script from importing a new name.
  • That test (tests/test_pinned_copy_text.py:774-790) reads only validate-pins.py, validate-repository-naming.py and validate-family.py, and it checks only shown_path. It never reads bootstrap.py.
  • The skew is actually enforced by update-shape.py apply, which refuses with update-accept-local-missing-name (update-shape.py apply --accept-local scripts/repo_shape.py can keep an old helper that a newly copied file imports from, and make bootstrap then fails with ImportError #536, tests/test_update_shape_kept_helper.py).
  • A copy is still the right choice:
    • the validators already keep their own _shown this way;
    • repo_shape.py does not have to be re-pinned;
    • it avoids colliding with repo_shape.py, shape_materialize.py and adopt-project.py: a value the YAML reader refuses for opening with an indicator is named by its line and says to double-quote it or reword it, plan --org @x names --org again under the reader's own remedy, and check names one bad tracking_branch once (#708) #817, repo_shape.py, setup-project.py, shape-doctor.py, README and the handbook: under Git Bash (MSYSTEM set) the lines that follow the host are quoted for sh, through one question, host_shell_dialect, so the doctor's lines, setup's family lines and family.py's NEXT, switch -c, add and cd lines paste whole in the shell that printed them, and README's Windows sentence says so #821, repo_shape.py strips only YAML's space and tab, so a value a Unicode space opens reads as PyYAML reads it, and validate-manifest, bootstrap, bump-leg and update-shape refuse a branch holding a character that does not print or is white space (#715) #823 and validate-pins.py and repo_shape.py: a pin, the manifest or a shape copy the gate cannot reach or read is refused by name, relative to the root, once and with its cause's advice, and every name and path it prints is spelled (#712) #827.
  • Fix: reword the body and the squash message. No code change.
• NIT 2: INHERITING's reasoning (tests/test_validator_child_encoding.py:1340-1350) covers the environment, but not interpreter options.
  • -X utf8 is not passed down to a child. Measured with a synthetic pair under LC_ALL=C PYTHONUTF8=0: python3 -X utf8 reports utf-8 1, and its sys.executable child, in the same environment, reports ascii 0.
  • So py -X utf8 scripts\siblings.py > log on Windows would write the holder's lines in UTF-8 and the member bootstrap's lines in cp1252.
  • This is not setup-project.py's run and the assembly root's bootstrap.py _run start Python children with no environment, so a cp1252 stdout gets two encodings from setup and a name it cannot hold ends make bootstrap in validate-manifest.py #836's defect, and I did not run it on a real family.
  • Fix: name the limit in the comment, or file a Tracking: setup, scaffold, family and the shared helpers, creating projects and families #459 checklist line.
• NIT 3: one test mutant survives (G4).
  • Dropping and any(keyword.arg == "encoding" ...) from encoding_changes (:1358-1361) still passes all 163 cases.
  • In practice it changes nothing: siblings.py has no reconfigure, and setup-project.py's one call names encoding. It also makes the reader stricter, not looser.
  • Optional fix: add a source-string case where sys.stdout.reconfigure(errors="backslashreplace") gives [].
• NIT 4: the fixture setup_module (:1151-1153) has the name of pytest's xunit module-setup hook.
  • It works only because pytest skips functions marked as fixtures (_get_first_non_fixture_func; CI pins pytest 8.3.4).
  • Fix: rename it, for example to setup_tool.
• For lane 5 (tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808), not this PR: the guard only checks that the helper's name appears.
  • environment_of returns the env= expression as source text. A row passes if the helper's name appears anywhere in it.
  • So two conditional mutants pass the LAUNCHERS rows:
    • S1, in setup-project.py: env=child_environment(capture) if capture else None;
    • B2, in the bootstrap: env=inherited_child_environment() if not args else None.
  • This PR's end-to-end and run tests catch both (S1 with 4 failures, B2 with 3), so setup-project.py and the assembly root's bootstrap.py: every Python child is told the encoding its words are read in, so a cp1252 stdout gets one encoding from setup, and make bootstrap passes a project whose name the code page cannot hold #851 itself is covered.
  • The fix is small. When env= is an ast.IfExp, read each branch, and call a run bare if any branch lacks the helper.
  • Measured as an in-memory patch: S1 gives 6 bare runs, the inherited-only conditional 6, B2 1. All six shipped rows stay at 0.
  • It needs about 10 lines in environment_of or runs_without_the_environment, plus one ENVIRONMENTS case. It belongs in tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808, or in a Tracking: the test suite, CI and the docs #461 checklist line if tests/: the quoting-sentence inventory sees a quoter handed on as a value and pins each function's host-dialect calls, and the launcher guard reads run_child and run([PYTHON, ...]), with rows for cli.py and setup-project.py (#759) #808 lands first.
• Out of scope, a Tracking: setup, scaffold, family and the shared helpers, creating projects and families #459 candidate: running the validator directly still raises. On the same project named 日本 under cp1252:strict, make validate and scripts/validate-manifest.py run alone still end in UnicodeEncodeError at manifest ok: (measured at head). The issue covers only setup and bootstrap.
• For the release notes (lane 3): the v1.1.1 notes should say that projects take the bootstrap fix with update-shape.py apply. v1.1.0's "A project's pin" bullet does the same.

Ruling on the shape bump: it is acceptable in v1.1.1.

• Nothing in the repository forbids changing a template in a patch release. The README's "Keeping a project's shape current" names update-shape.py as the way an upstream fix reaches a project.
• The v1.1.0 notes, now on main, list "The pinned shape copies, in one bump" as ordinary release content.
• The issue itself calls this a shape bump and sits in v1.1.1.
• fix/763-holder-docs-agree (templates/family-root README.md and AGENTS-shape.md, AGENTS.md and validate-family.py: the holder's own docs say its parent folder is "named after the family" and omit "EXACTLY, case included", and "never hand-edit the members: block" is contradicted by the validator's repair remedies #763, v1.1.1 by lane 3's list) already edits templates/assembly-root/scripts/validate-pins.py and three family-root files.
• No digest of bootstrap.py is committed anywhere: a grep for its sha256 and its blob id finds nothing.
• Every test that reads the template passes: test_repo_hygiene, test_pinned_copy_text, test_update_shape_kept_helper, test_template_text_placeholders and the lockstep tests.

Checked and true

• Both copies match bump-leg.py (a real bump) and update-shape.py apply run the project's own validators as children that cannot print its display name on a cp1252 stdout, so a project named 日本 is refused and rolled back as bump-leg-validators-red or update-validators-red under PYTHONIOENCODING=cp1252:surrogateescape, and the refusal blames the validators #254's rule exactly. The rule is scripts/shape_materialize.py:256-325. My parity script compared the library, the setup-project.py copy and the bootstrap.py copy over 207 streams (17 encodings × 12 handlers, plus 3 odd streams): 0 differences. The captured child is equal too.
• No Python child is left out.
  • Every child in setup-project.py starts through run, whose one subprocess.run is at :372. That covers all six children the writer names.
  • The bootstrap's only interpreter start is _run at :150.
• Issue line 1, measured.
  • Head: 129 lines and 0 that are not UTF-8, under both cp1252:strict and cp1252:surrogateescape.
  • Base db693114: 9 such lines (24 31 38 45 64 79 80 100 102).
  • With --into …/日本work, head writes 0 non-UTF-8 lines and 日本 five times as UTF-8. Base writes 9 non-UTF-8 lines and one \u65e5.
• Issue line 2, a project named 日本.
  • Head exits 0 under cp1252:strict, cp1252:surrogateescape, cp1252:replace, cp932, utf-8 and an unset variable.
  • Base exits 1 under both cp1252 handlers with bootstrap FAILED: manifest (exit 1).
  • Head also passes, and base fails, under:
    • LC_ALL=C PYTHONUTF8=0;
    • PYTHONUTF8=1 with cp1252:strict;
    • an empty PYTHONIOENCODING;
    • :strict.
  • cp65001, latin-1:namereplace and cp1252:xmlcharrefreplace behave the same on head and base.
  • A name holding a lone surrogate gives the same result on head and base.
  • A non-UTF-8 byte in --into shows the same way on head and base.
• The new tests on the base's two tools: 11 failed and 36 errors (46 new cases plus the setup-project.py row), 116 passed. This matches the PR body.
• Mutants: 31 one-line mutants, 30 killed, G4 survives (NIT 3).
• Sonar:
  • Cognitive complexity is 3 for child_environment and 2 for inherited_child_environment.
  • There are no composite asserts and no string literal used 3 or more times.
  • The repository has no sonar-project.properties and no Sonar step in a workflow, so templates are not excluded. The duplicated part of the rule is about 10 lines (≈60 tokens by hand count), below Sonar's 100-token minimum for a duplicate.
• Docs: no --help, README, handbook, AGENTS.md or docs/cli.md text changes. test_cli_reference, test_agents_md_codes and test_repo_hygiene pass.
• Merges: all clean (details in Part 2).

Could not verify

• No run on Windows: a real cp1252 pipe or console, a cp65001 console, or the py launcher.
• siblings.py was not run under -X utf8 on a real family.
• The milestones of templates/family-root README.md and AGENTS-shape.md, AGENTS.md and validate-family.py: the holder's own docs say its parent folder is "named after the family" and omit "EXACTLY, case included", and "never hand-edit the members: block" is contradicted by the validator's repair remedies #763, bootstrap.py: a validator's exit 2 ends bootstrap REFUSED, exit 2, an unreadable upstream path, cache or pin is no answer and a reftable leg's cherry-pick or revert is read from its ref store, so no traceback or ended pick follows (#764) #815 and bootstrap.py: step (a) decodes git's output as the system decodes a name it is given, so a branch whose upstream is not UTF-8 is moved back, and a good branch this system's encoding cannot write gets a line of its own, not a traceback (#710) #824 were not read from GitHub (no gh).
• SonarCloud's own analysis (the MCP server is down).

Lane: openreposhape-2 (openRepoShape-2)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 16 hours by commenting @sourcery-ai review. Upgrade to get a review now.