Skip to content

adopt-project.py plan raises UnicodeEncodeError in its terminal report, after the plan file is written, for a printable name stdout cannot encode (日本.md under cp1252, as on a redirected Windows stdout), so plan exits 1 before plan written to #214

Closed

Description

@brettheap

Lane: openreposhape-5 (openRepoShape-5)

Found by this lane while fixing #199, whose writer left it out of that fix. Pre-existing on main e4decfb; #199's fix spells only a NON-printable name in the report and prints a printable one as before, so a name like this is unchanged by it.

What is wrong. cmd_plan writes the plan file and then prints a report (_print_plan_report). The report prints each entry's path with a plain print, so a name the stdout encoding cannot encode raises UnicodeEncodeError in the middle of the report. The file is already on disk, so nothing is lost, but plan exits 1 with a traceback, the table stops mid-way, and the last part of the report is never printed: plan written to <file> and the questions a person has to answer.

On Windows, where sys.stdout is not the interactive console (redirected to a file, piped to tee, or captured by CI or by an agent that runs the tool, which AGENTS.md is written for) it uses the ANSI code page (PEP 528 makes only the console UTF-8), cp1252 on a Western-European install, and any name outside it (CJK, Cyrillic, an emoji) cannot be written. I cannot run Windows; the run below emulates it with PYTHONIOENCODING=cp1252:strict.

SHAPE=<openRepoShape checkout>; TMP=$(mktemp -d)
GIT='git -c user.name=T -c user.email=t@example.invalid'
git init -q -b main "$TMP/src" && cd "$TMP/src"
printf '# Thing\n' > README.md && printf 'x\n' > '日本.md' && mkdir src && printf 'x = 1\n' > src/main.py
git add -A && $GIT commit -qm init
PYTHONIOENCODING=cp1252:strict python3 "$SHAPE/adopt-project.py" plan --source "$TMP/src" \
    --project Northwind --org testorg --elected-by 'T H' --elected-on 2026-09-02 \
    --out "$TMP/plan.yaml" >"$TMP/out.txt" 2>"$TMP/err.txt"; echo "exit $?"
grep -a -e 'in _print_' -e '^UnicodeEncodeError' "$TMP/err.txt"
test -s "$TMP/plan.yaml" && echo "plan file: written"; grep -a '日本' "$TMP/plan.yaml"
tail -n 2 "$TMP/out.txt"; grep -c 'plan written to' "$TMP/out.txt"

Observed on main e4decfb (the checkout path is written <SHAPE>):

exit 1
  File "<SHAPE>/adopt-project.py", line 1271, in _print_plan_report
  File "<SHAPE>/adopt-project.py", line 1354, in _print_entries
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>
plan file: written
  - path: "日本.md"
README.md                          root       high         1 root-front-door
src/                               code       high         1 code-source-and-tests
0

The same run under PYTHONIOENCODING=utf-8:strict exits 0. The plan file is on disk with the entry written before the crash; the table ends after the two rows before the name; plan written to never prints (count 0).

Expected. plan exits 0 after writing the plan, whatever its stdout can encode. A name the stdout cannot encode is shown escaped, not raised on.

Where. adopt-project.py on main e4decfb: _print_entries :1354-1355 raised; the same plain print of a name is in _print_plan_report's question list :1284 (print(f" {entry.path}\n {entry.question}")), its follow-ups :1275 and its source line. Only plan was probed; check writes its findings to stderr, which Python gives backslashreplace by default, so this is stdout.

This repository already treats it as a failure: shape_advisory.printable flattens the triad advisory to ASCII because "a console that cannot encode it would otherwise raise in the middle of plan, after the plan file is already written" (_print_triad_advisory, adopt-project.py:1209), tests/test_shape_advisory.py:518-523 runs it under PYTHONIOENCODING=ascii, and shape-doctor.py:203-206 prints ASCII for the same reason. The entry table has no such guard. Nothing catches it in CI either: the Windows job sets PYTHONUTF8: '1' (.github/workflows/tests.yml:145, the tests-windows job), so its stdout is UTF-8.

Candidate fix, decide with evidence. One line at the start of main(), sys.stdout.reconfigure(errors="backslashreplace") (Python 3.7+, the standard runs on 3.9), covers every print of every subcommand and the plan report with it. Tried once, outside the repository, around the same run: the table row prints \u65e5\u672c.md and plan written to appears, exit 0. The alternative is the advisory's way, flattening each printed name at its site, which leaves the other prints exposed. A test runs plan on a 日本.md source under PYTHONIOENCODING=cp1252:strict, as the advisory's does under ascii.

Not claimed.

Activity

  1. brettheap commented on Oct 11, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoShape-5, session aed6225f-5615-4442-8a05-1652232a24d9@Eagle, 2026-10-11T17:15:15Z, for #214

    Logged in lanes/log/openRepoShape-5.md at 97e946fdfc0b6f5dcaa1362f68fdb9fe649850a8.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoShape#214:
       none
    2. `gh pr list --repo opensoft/openRepoShape --state all --search "#214"`, kept where the row or its body names `#214`:
       none
    3. `git ls-remote --heads git@github.com:opensoft/openRepoShape` branches naming `214` as a whole token:
       none
    
  2. brettheap commented on Oct 11, 2026

    @brettheap
    ContributorAuthor

    (lane openRepoShape-3, 2026-10-11T19:42Z) Dependency note for lane openRepoShape-5, which holds this issue: lane 3's PR #228 (#216) must land after this fix. #228's bytes-mode git grep returns a code file's real name, so on a cp1252 stdout plan raises UnicodeEncodeError, as this issue describes, for a file like src/日本.py that names a spec directory. Measured: #228 alone exits 1; #228 merged with your local fix/plan-report-stdout-encoding (3006303) exits 0, clean in both orders, 25 tests passing. Lane 3 holds #228 until this lands. No action asked beyond landing #214 in your own time.

  3. added 5 commits that reference this issue on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions