scripts/validate-repository-naming.py, a pinned copy in every project, ends in a UnicodeEncodeError traceback, exit 1, when it prints a name that is not UTF-8 (--explain and the plain listing, under a strict UTF-8 stdout such as the one the doctor gives its children) and in a UnicodeDecodeError when it reads one from stdin, though --quiet answers the same name with FINDING naming-unclassified, so a directory made on a Latin-1 machine gets a crash where the script's own exit codes give 1 to a name that matches no family and 2 to a question that could not be asked #464

Lane: openreposhape-4 (openRepoShape-4)

Found by this lane's planner for the doctor cluster, while verifying #410: the doctor hands a directory's name to the naming script and quotes the script's traceback as a fact about the name. The plan put the script's own crash outside #410: "a PINNED shape copy in every project (contracts/shape-pin.yaml lists it), so changing it moves every project's pin and needs its own decision". #410's own "What it should do" 3 says the same: "whether this issue also makes validate-repository-naming.py print a name it cannot encode (backslashreplace) or refuse it by name, or that goes in an issue of its own". This is that issue. The doctor's two naming rows (reading only the exit code) are #410's. Re-verified on main dd8d0ea (scripts/validate-repository-naming.py sha256 362efbbd…; no git and no gh call is made by this script, and the usual guards stand anyway). The one open PR that edits the file is #453 (#338, the --link-source reading), in other functions. What this is about is the exit code and the name: the script's docstring gives 1 to "a FINDING: at least one name matches no family" and 2 to "a REFUSAL: the question could not be asked at all", and a traceback exits 1.

What is wrong. A name that is not UTF-8, as it arrives on argv or on stdin, kills the script in the two ways it can read one, under a strict UTF-8 stdout and stdin:

U=<openRepoShape checkout>   # a clean checkout at dd8d0ea
# Runs in bash on Linux. This script runs no git and no gh; the guards stand anyway: a `gh` that exits 97 first on PATH,
# an empty global git config, GIT_ALLOW_PROTOCOL=file.
R=$(mktemp -d); mkdir $R/bin; printf '#!/bin/sh\nexit 97\n' > $R/bin/gh; chmod +x $R/bin/gh; touch $R/gitconfig
export PATH=$R/bin:$PATH GIT_CONFIG_GLOBAL=$R/gitconfig GIT_CONFIG_NOSYSTEM=1 GIT_ALLOW_PROTOCOL=file
N=$(printf 'Atl\xffas')                          # the name of a directory made on a Latin-1 machine: 0xff is not UTF-8
naming() { python3 $U/scripts/validate-repository-naming.py "$@" > $R/o 2> $R/e; echo "exit $?"
           grep -a -E '^(FINDING|Unicode[A-Za-z]*Error)' $R/e | cut -c1-105; }
echo "stdout's handler in this shell: $(python3 -c 'import sys; print(sys.stdout.errors)')"
echo "(1) as a person's UTF-8 terminal and the doctor's children run it: a strict UTF-8 stdout"
export PYTHONIOENCODING=utf-8
echo "--explain -- Atlas";        naming --explain -- Atlas; head -1 $R/o
echo "--explain -- Atl\\xffas";    naming --explain -- "$N"
echo "-- Atl\\xffas";              naming -- "$N"
echo "--quiet -- Atl\\xffas";      naming --quiet -- "$N"
echo "Atl\\xffas on stdin";        printf '%s\n' "$N" | naming
echo "(2) the same names with stdout set to backslashreplace"
PYTHONIOENCODING=utf-8:backslashreplace naming --explain -- "$N"; head -1 $R/o

Output at dd8d0ea:

stdout's handler in this shell: surrogateescape
(1) as a person's UTF-8 terminal and the doctor's children run it: a strict UTF-8 stdout
--explain -- Atlas
exit 0
  Atlas: project-leg / assembly
--explain -- Atl\xffas
exit 1
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 5: surrogates not allowed
-- Atl\xffas
exit 1
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 5: surrogates not allowed
--quiet -- Atl\xffas
exit 1
FINDING naming-unclassified: Atl\udcffas
Atl\xffas on stdin
exit 1
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
(2) the same names with stdout set to backslashreplace
exit 1
FINDING naming-unclassified: Atl\udcffas
  Atl\udcffas: NO FAMILY

• The stdout this needs. In this container's POSIX locale Python gives stdout surrogateescape (the first line), the raw byte is written out, and the same commands print the finding and exit 1: no crash. strict is what PYTHONIOENCODING=utf-8 sets, which is what the doctor gives every child (shape-doctor.py:1214, in run_validator: "UTF-8 whatever the console is"), and by Python's documented default what a terminal under a locale that is not C, C.UTF-8 or POSIX gets (en_US.UTF-8; this container has none, so that is not run here).
• What a person meets. The doctor over a directory whose name is not UTF-8 (shape-doctor.py's two naming rows read only the naming script's exit code, so a crash of the STANDARD's own validate-repository-naming.py is a FINDING about the project (INVALID (naming), exit 1, where #357 makes a crashed template CANNOT ANSWER), and a directory whose name is not UTF-8 gets naming FINDING Atl?as: UnicodeEncodeError: ..., the script's traceback quoted as a fact about the name #410's case 1: the naming row quotes the UnicodeEncodeError), or validate-repository-naming.py --explain "$(basename "$PWD")" typed in one, or a list of directory names piped in.
• --quiet is the control. It prints nothing to stdout, and a FINDING goes to stderr, whose handler is backslashreplace by default: FINDING naming-unclassified: Atl\udcffas, exit 1. That is the answer the other three modes should give.

Why.

• argv is decoded with surrogateescape, so the byte 0xff is the lone surrogate U+DCFF. --explain prints it at scripts/validate-repository-naming.py:461 (print("\n".join(_describe(...))) in _classify_targets), the plain listing at :440 (_print_classification, called from :463), and a strict UTF-8 encoder refuses a lone surrogate. stdin is decoded with the stream's own handler: _targets_from_stdin (:419, for line in sys.stdin) raises UnicodeDecodeError under strict.
• Nothing in the file asks what the stream can hold. validate-pins.py does, for its own prints: _emit_report reconfigures a strict or surrogateescape stdout to backslashreplace before it prints (templates/assembly-root/scripts/validate-pins.py:785, templates/assembly-root/scripts/validate-pins.py: a bare --pin-source <path> whose path holds an = is read as <product>=<path> (raw.partition("=")), so a stand-in clone under a directory named a=b is never consulted and the pin's digest silently goes to gh api and a skip (pins ok, exit 0), or, when the text before the = is a declared pin's name, to a FileNotFoundError traceback #295). It says why it does not call shape_materialize.escape_what_stdout_cannot_encode (scripts/shape_materialize.py:93): "which no project carries, and this file imports nothing but repo_shape". The naming script is the same kind of pinned copy and imports only repo_shape.
• scripts/validate-repository-naming.py is listed in every project's contracts/shape-pin.yaml, so a change reaches existing projects through update-shape.py as upstream-changed.

What it should do. The script gives a name it cannot print an answer, and no mode ends in a traceback. The decisions are the author's:

1. Print it escaped. Reconfigure stdout once at the top of main, as validate-pins.py does (a strict or surrogateescape stdout becomes backslashreplace; any handler somebody chose is left alone), and decode stdin so that a name survives it (surrogateescape). The name then prints as Atl\udcffas, which is how --quiet already prints it on stderr.
2. The verdict stays a finding. A repository name never holds a byte that is not UTF-8, so no family matches it and naming-unclassified, exit 1, is a true answer about the name. The alternative is a refusal by name (exit 2), which says "the question could not be asked", and it could: the answer is "no family". Choosing it makes --quiet and the other modes disagree unless --quiet moves too.
3. Whether the doctor's fix is enough. The plan for shape-doctor.py's two naming rows read only the naming script's exit code, so a crash of the STANDARD's own validate-repository-naming.py is a FINDING about the project (INVALID (naming), exit 1, where #357 makes a crashed template CANNOT ANSWER), and a directory whose name is not UTF-8 gets naming FINDING Atl?as: UnicodeEncodeError: ..., the script's traceback quoted as a fact about the name #410 gives the doctor's children PYTHONIOENCODING=utf-8:backslashreplace. That stops the crash for the doctor's children and for no other caller: the script run by hand, or by any caller that does not set it, still ends in the traceback.

How to test.

• In process, on every OS (no argv bytes cross a process boundary): call main(["--explain", "--", "Atl\udcffas"]) with sys.stdout replaced by io.TextIOWrapper(io.BytesIO(), encoding="utf-8", errors="strict"); assert exit 1 and that no exception escapes. The same for the plain listing, and for stdin: a TextIOWrapper over io.BytesIO(b"Atl\xffas\n") with encoding="utf-8", errors="strict" as sys.stdin.
• One real subprocess, PYTHONIOENCODING=utf-8, the name built with os.fsencode, skipped on Windows (its argv is UTF-16 and cannot carry the byte): exit 1, FINDING naming-unclassified on stderr, no Traceback.
• Control: a valid name under the same strict stdout classifies and exits 0.

Verified at main dd8d0ea (scripts/validate-repository-naming.py sha256 362efbbd…, shape-doctor.py c206264b…).

Related: #410 (the doctor's naming rows, which quote this traceback as a fact about the name; its decision 3 defers this script's own crash to an issue of its own), #295 and validate-pins.py:785 (the same fix for another pinned copy), #280 (a lone surrogate in the manifest's name is a finding in validate-manifest.py), #391 and #454 (other plain prints of a name in pinned copies), #453 (edits this file's --link-source reading).