[brettheap]

Closes #1214

Goal

#1214 counts 50 tests in 12 files that fail under a Latin-1 locale before and after #1209, and asks of each whether it is a test that cannot judge under a one-byte locale, or a tool that misbehaves there. All 50 are settled here:

• 3 are a tool defect, fixed: a family holder's scripts/validate-family.py ended in a UnicodeEncodeError traceback, exit 1, on any stdout that cannot hold U+2026, when the holder passed.
• 45 cannot judge under Latin-1, and now skip there on their own premise, which the skip reason states. 18 fail in the test process itself, before the tool runs. In the other 27 the tool answers rightly for the bytes Latin-1 gives, and the row expects UTF-8's.
• 2 judge under Latin-1 as they are written, and carry no guard: test_a_utf8_source_directory_still_adopts_with_local_remotes (it passes with the pip-installed git-filter-repo, and fails with a python -E one on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's tool defect; see "Left") and test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 (its if asks the one thing its assertion needs). Both came out of RV1258's review of the first commit.

Under Latin-1 with the pip-installed git-filter-repo, none of the 50 fails on this head. Under UTF-8 every row keeps its outcome, and the rows added pass.

The classification is W1214's: lane 4's first writer of #1214, who read each test's code and stopped at a checkpoint with half the guards written and nothing run. This branch finishes the guards, runs every gate below, and measures each row. RV1258 reviewed the first commit; the second answers it (next section).

The second commit

RV1258 reviewed the first commit, c8641c691 (NOT READY: two SHOULDs, three NITs, no MUST; #1258 (comment)). The second commit, cd9f3cfd1, answers it on top of the first, with no rebase. It changes four test files and no tool:

• S1, a premise guard on a row that passes: the guard on test_a_utf8_source_directory_still_adopts_with_local_remotes is gone, and so is the A_NAME_HELD_AS_UTF_8 import it alone used in tests/test_adopt_template_values.py.
  • The guard skipped a row that passes: on main under Latin-1, with CI's pip-installed git-filter-repo, the row passes. Its skip reason (the tool "names it as" a name that is not UTF-8) was false for it, because execute adopts the source.
  • The row now runs under Latin-1 and passes with the pip launcher. With a launcher started as python -E it fails on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's tool defect (git-filter-repo's UnicodeEncodeError on U+FFFD) until adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 lands. That is a failure of the tool, and tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214's rule sends such a row to an issue of its own, not to a skip.
  • validate-family.py: the holder's gate escapes what its stdout cannot encode, so it passes under Latin-1 where it ended in a traceback, and 45 rows in 11 test files that cannot judge under a one-byte locale skip there on their own premise (#1214) #1258 and adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 are now independent of each other. This PR carries no guard for adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 to drop, so the earlier plan, that whichever of the two lands second drops this PR's guard, is void. Neither waits for the other.
• S2, nothing CI runs held the helper's answers: tests/test_non_utf8_name_helpers.py gets 20 rows, written as that file writes its Windows rows: os.fsencode, os.fsdecode and sys.getfilesystemencoding are replaced for one test by UTF-8, Latin-1 and ASCII as POSIX has them.
  • 15 rows (five texts, three locales) hold which question skips the text (none, "cannot spell", "writes" or "reads"), and that the reason names the encoding, ends with the row's premise, and is ASCII.
  • 5 rows hold the emulation to this run's real locale: the helper answers the same with the real os as with the emulation of its file system encoding.
  • RV1258's mutant A5 (UTF-8 bytes built with surrogatepass instead of surrogateescape) now fails 3 rows under CI's UTF-8 locale; before, it made 11 rows skip silently. The mutants that survived, A1, A2 and A4, now fail 5, 1 and 6 rows (see the mutants below).
• N1: the first-statement row of tests/test_bootstrap_spelling.py also reads templates/family-root/scripts/validate-family.py, so RV1258's mutant D3 (the escape_unencodable_stdout() call moved after parse_args) fails it.
• N3: test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 no longer asks the helper. Its if also asks os.fsdecode(b"\xe9") == chr(0xDCE9), so under Latin-1 the row runs its first assertion and passes where it skipped, and under UTF-8 and ASCII it asserts as before.
• N2, known and left: under emulated Windows codecs the helper skips the U+DCE9 rows that carry no Windows mark, with a premise that is false on Windows. That is reasoned and emulated, not run on Windows, and CI runs Linux only (ci: run the test suite on Linux only #597).

The 50, by file

Kinds:

• a1: the test process itself cannot make or pass the name under Latin-1, so it raises UnicodeEncodeError before the tool runs. Guard: the existing skip_where_the_environment_cannot_carry.
• a2: the tool runs and answers rightly for the bytes Latin-1 gives, and the row expects UTF-8's reading or spelling. Guard: the new skip_where_the_locale_does_not_spell_it_as_utf_8(text, premise), asked only of the text the row's premise is about.
• b: a tool defect.
• c: the row judges under Latin-1 as it is written, and runs there with no guard.

test_stdout_encoding.py comes first, as #1214 asks. 18 a1 + 27 a2 + 3 b + 2 c = 50.

File	Rows	Kind	Why it fails under Latin-1	Guard
tests/test_stdout_encoding.py	10: test_plan_reports_every_name_whatever_its_stdout_can_encode (3), test_plan_on_the_posix_default_stdout, test_every_tool_exits_alike_whatever_its_stdout_can_encode (6)	a1	The test makes a file or directory named U+65E5 U+672C, or passes that name on a command line, and Latin-1 cannot encode it.	skip_where_the_environment_cannot_carry, in plan_arguments and in each of the six builders. (bump-leg-dry-run writes the name as bytes and passes, so it has none.)
tests/test_adopt_pins.py	5 [zwj] rows	a1	U+200D cannot go on the command line, or through copytree.	skip_where_the_environment_cannot_carry(name)
tests/test_adopt_pins.py	4 [not-utf-8...] rows	a2	The byte 0xFF is read as U+00FF, printable text the tool takes, so plan goes on where UTF-8 gives U+DCFF and a refusal.	A_BYTE_READ_AS_A_SURROGATE, where the name holds U+DCFF
tests/test_adopt_pins.py	5 rows that compare a pin or write line	a2	A Latin-1 stdout has no U+2026, so the tool prints the digest's ellipsis as \u2026, as #214 designed.	PRINTED_AS_ITSELF, just before the line is compared
tests/test_pin_source_unreadable.py	3 [not-ascii-*]	a1	The path U+65E5 U+672C -missing cannot go on the command line.	skip_where_the_environment_cannot_carry
tests/test_pin_source_unreadable.py	3 [not-utf-8-*]	a2	miss<0xFF>ing reaches the tool as text, and is printed as itself, not as \udcff.	A_BYTE_READ_AS_A_SURROGATE
tests/test_family_tracking_branch.py	test_each_act_refuses_the_tracking_branch_then_lands_on_a_branch [add], [bump], [remove]	b	Each act passes until validate(holder). The holder's validate-family.py prints members/X: tree digest recomputes (<12 hex>...), ending in U+2026, and never escaped its stdout, so it ended in a traceback, exit 1.	none: the tool is changed
tests/test_family_tracking_branch.py	test_a_branch_ending_in_a_no_break_space_is_pushed_as_itself	a2	git switch -c feat<U+00A0> writes the single byte 0xA0, so the branch is not UTF-8, and the tool rightly refuses it (#765).	A_NAME_HELD_AS_UTF_8
tests/test_adopt_template_values.py	the --elected-by and --reference rows of test_plan_refuses_a_flag_no_file_can_hold whose value holds U+DCE9, and test_plan_and_check_say_the_same_sentence	a2	0xE9 is read as U+00E9, text plan takes.	A_BYTE_READ_AS_A_SURROGATE (in the flag test, only where the value holds U+DCE9)
tests/test_adopt_template_values.py	test_a_utf8_source_directory_still_adopts_with_local_remotes	c	With the pip launcher it does not fail under Latin-1 on this head or on main: its caf<U+00E9> directory is the single byte 0xE9, and execute adopts the source all the same. It fails only where git-filter-repo is started as python -E, on #1247's tool defect (see "Left"), which is an issue of its own and not a premise to skip on.	none (a guard was here in the first commit, and RV1258's S1 took it out)
tests/test_clone_url_bytes.py	the 4 [bidi-override-*] rows	a2	The directory is made from U+202E's UTF-8 bytes E2 80 AE. Latin-1 reads them as three characters, one of them the C1 control U+0080, so the tool rightly says "a control character".	A_NAME_READ_AS_UTF_8, in skip_where_this_system_cannot_name
tests/test_bump_leg_branch_and_push_line.py	2	a2	The branches feat<U+00A0> and main<U+00A0> hold the single byte 0xA0.	A_NAME_HELD_AS_UTF_8
tests/test_bootstrap_leg_moves.py	2	a2	bootstrap.py prints U+2014 escaped on a Latin-1 stdout.	PRINTED_AS_ITSELF, just before the assert
tests/test_update_shape_apply_lines_and_headers.py	1	a2	The branch feat<U+00A0>.	A_NAME_HELD_AS_UTF_8
tests/test_adopt_spelled_names.py	1	a2	The test writes caf<U+00E9>.md as 0xE9, so git holds a name that is not UTF-8, and the C-locale child prints caf\udce9.md, not caf\xe9.md.	A_NAME_HELD_AS_UTF_8, after its C-locale probe
tests/test_family_member_branch_not_utf8.py	[trailing-no-break-space]	a2	_member_head reads the byte 0xA0 as U+DCA0, which is the branch git wrote.	A_NAME_HELD_AS_UTF_8, only where the branch is not NOT_UTF_8_BRANCH: [not-utf8] passes under Latin-1 and still runs there
tests/test_family_head_by_hand_bytes.py	1	c	Its if, os.fsencode(chr(0xDCE9)) == b"\xe9", holds under Latin-1 too, and its assertion, os.fsdecode(LATIN_1) == branch, holds only where the locale reads 0xE9 as U+DCE9 (UTF-8, ASCII). Its first assertion holds in every locale.	none (RV1258's N3): the if also asks os.fsdecode(b"\xe9") == chr(0xDCE9), so under Latin-1 the row runs its first assertion and passes

What changed, file by file

18 files: one tool, tests/conftest.py, and 16 test files.

• templates/family-root/scripts/validate-family.py (the tool):
  • main's first statement is now escape_unencodable_stdout(), imported from repo_shape, as in siblings.py and both roots' bootstrap.py (templates/assembly-root/scripts/bootstrap.py and templates/family-root/scripts/bootstrap.py: main prints the manifest's name with a plain print before any step runs, so name: "a\ud83db" (a lone surrogate) ends make bootstrap in a UnicodeEncodeError traceback, exit 1 (make: 2), and the validator that names the problem is never reached; a valid name the stdout cannot encode ends the same way #391, family-root siblings.py: make siblings, park and resume print the family's name with a plain print, so name: "a\ud83db" ends each in a UnicodeEncodeError, and the literal dash ends every name under PYTHONIOENCODING=ascii #454).
  • That call adds no version-skew risk. The function shipped in the same commit as spelled (ec4a7a0), which the file already imports, and it is not in test_pinned_copy_text.py's NEWER_THAN_A_KEPT_LIBRARY.
  • The holder's shape-pin digests are written by init, so there is no static digest to move.
• tests/test_family.py: one new row, test_the_gate_passes_on_a_stdout_that_cannot_hold_its_notes. It runs the holder's gate with PYTHONIOENCODING=latin-1 and expects exit 0, no traceback, and the two notes ending in \u2026).
• tests/conftest.py:
  • The new helper, skip_where_the_locale_does_not_spell_it_as_utf_8(text, premise), asks three questions, each through os at call time: whether os.fsencode can spell the text, whether it gives UTF-8's bytes, and whether os.fsdecode reads those bytes back as the text.
  • Four premise sentences go with it: A_BYTE_READ_AS_A_SURROGATE, A_NAME_HELD_AS_UTF_8, PRINTED_AS_ITSELF and A_NAME_READ_AS_UTF_8. Every skip reason is ASCII.
  • It is never asked inside skip_where_a_name_cannot_be_not_utf_8. A branch whose bytes are not UTF-8 is one in every locale, so those rows still judge under Latin-1, and this takes none of tests/conftest.py: follow-up of PR #1209 (#1196) from its review: the helper's locale skip also catches 80 rows that can judge the tool under a one-byte locale, and four smaller gaps (a Windows failure, an unheld order, a triple predicate, a docstring) #1224's N1 over-skip.
• The 11 files with guards are the files of the table above that name a guard. Each guard sits just before the first line that needs its premise. tests/test_family_head_by_hand_bytes.py has none: its if is narrowed (N3).
• tests/test_non_utf8_name_helpers.py (the second commit): 20 rows hold skip_where_the_locale_does_not_spell_it_as_utf_8 under emulated UTF-8, Latin-1 and ASCII (S2, above).
• tests/test_bootstrap_spelling.py (the second commit): the first-statement row takes validate-family.py as a third file (N1).
• tests/test_non_utf8_name_callers.py: windows_names_it now emulates Windows' os.fsdecode too (UTF-8 with surrogatepass, windows_fsdecode).
  • Without it, the bidi guard in skip_where_this_system_cannot_name asked this run's real Latin-1 os.fsdecode under an emulated Windows.
  • Measured: test_where_windows_holds_a_bidi_override_the_name_check_does_not_skip then failed under Latin-1 (1 failed, 35 passed), where it passes on main. With the emulation, the file gives 36 passed in both locales.
  • Real Windows reads UTF-8 in every locale, which is what that row's own reasoning ("NTFS holds U+202E, so the bidi-override row ... runs on Windows as it does here") assumes.

Runs

Environment. One runner for every run: /usr/bin/python3 -m pytest -p no:cacheprovider (Python 3.12.3).

• A stub gh first on PATH, a bogus GH_TOKEN and an empty GH_CONFIG_DIR.
• GIT_CONFIG_GLOBAL an empty file, GIT_CONFIG_NOSYSTEM=1, the test identity, and EMAIL set.
• GIT_ALLOW_PROTOCOL, GIT_SSH_COMMAND and the LANES_* variables unset.
• Every run counted here had git-filter-repo 2.47.0 on PATH (git filter-repo --version: a40bce548d2c, CI's pin, pip-installed in a venv). An earlier pair of head runs without it skipped 58 adoption rows, and is not counted.
• That git-filter-repo's launcher is the pip one, with no python -E, so a test's own PYTHONIOENCODING reaches it. The -E comparison is the last paragraph of this section.

Locales.

• UTF-8: LC_ALL=C.UTF-8.
• Latin-1: built with localedef -c, as adopt-project.py: under a one-byte locale such as Latin-1, a clone check makes again for a plan's clone record that neither spelling names is compared with both, so it is kept and the next run reads it rather than cloning again (#1112) #1153's test_adopt_one_byte_locale.py builds one; then LOCPATH, LC_ALL=en_US.ISO-8859-1, PYTHONUTF8=0. Python reports iso8859-1 and reads 0xFF as U+00FF.
• ASCII: LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0.

Round one's gate (the first commit, c8641c691, on main bd3e9c148, 27 files, F1214's runs): UTF-8 1520 passed and 1 skipped on main, 1521 and 1 on the head; Latin-1 123 failed, 1373 passed, 25 skipped on main, and 44 failed, 1404 passed, 74 skipped on the head. RV1258 re-ran it on 35 files (main 5926ccca, merge 103ada82) and got the same transitions. The gate below replaces it.

The gate: 29 files, on a scratch merge. The merge is this head (cd9f3cfd1) into origin/main 46fe6ac19 (local, never pushed, clean); the baseline is 46fe6ac19 itself. Both with the pip launcher.

• The 27 files of round one: the 23 files on main that use skip_where_a_name_cannot_be_not_utf_8 or on_a_branch_that_is_not_utf_8, with test_repo_hygiene.py (W1196's 24, less the two files tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 adds, plus test_family_leg_path_by_hand_bytes.py), and four more that run the changed code: test_family.py, test_pinned_copy_text.py, test_update_shape_kept_helper.py and test_local_remote_dir_urls.py. They include test_non_utf8_name_helpers.py.
• Two more for the second commit: test_path_guard_hygiene.py (the second hygiene file) and test_bootstrap_spelling.py.

Run	main 46fe6ac19	scratch merge with this head
29 files, UTF-8	1873 passed, 1 skipped	1895 passed, 1 skipped
29 files, Latin-1	128 failed, 1721 passed, 25 skipped	49 failed, 1775 passed, 72 skipped
the 50, UTF-8	50 passed	50 passed
the 50, Latin-1	49 failed, 1 passed	0 failed, 5 passed, 45 skipped
the 50, Latin-1, -E launcher	50 failed (on bd3e9c148, round one)	1 failed, 4 passed, 45 skipped

The 5 that pass under Latin-1 are the 3 tracking-branch rows (the tool fix), test_a_utf8_source_directory_still_adopts_with_local_remotes and test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9.

Main moved twice while this was gated, none of its commits touching a file of the gate or of this PR. The merge was run again on each: on 9729315f1 the same 29 files, with identical outcomes by test id in both locales (1895 passed, 1 skipped; 49 failed, 1775 passed, 72 skipped); on 801a7d6d0 the 50 and the four files the second commit changes (test_non_utf8_name_helpers.py, test_bootstrap_spelling.py, test_family_head_by_hand_bytes.py, test_adopt_template_values.py). The 50 give the same as above in all three modes (UTF-8 50 passed; Latin-1 pip 0 failed, 5 passed, 45 skipped; Latin-1 -E 1 failed, 4 passed, 45 skipped), the four files give 385 passed under UTF-8, and 6 failed, 371 passed, 8 skipped under Latin-1: the 5 test_bootstrap_spelling.py rows and test_execute_refuses_a_local_source_directory_that_is_not_utf8, all of which fail on main too.

By test id, from main to the scratch merge:

• UTF-8: 22 new rows, all passed: the 20 of test_non_utf8_name_helpers.py, the [validate-family] row of the first-statement test, and the first commit's test_the_gate_passes_on_a_stdout_that_cannot_hold_its_notes. Nothing else changes: no new skip, no row from passed to failed. The one skip is test_stdout_encoding.py's Windows default-stdout row, on both.
• Latin-1:
  • 0 rows from passed to failed.
  • 0 from passed to skipped. (The first commit had one, the S1 row; it passes now.)
  • 32 from failed to passed: the 3 tracking-branch rows and test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 of the 50, and 28 other rows of test_family.py whose holder's gate ended in the same traceback.
  • 47 from failed to skipped: 45 of the 50, and the 2 bidi rows of test_local_remote_dir_urls.py, which share BAD_DIRECTORIES.
  • 22 new rows, passed.
  • Each of the 49 failures fails on main too: the rows tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 settles, the rows under "Left", and 5 rows of test_bootstrap_spelling.py (see "Left").

On a scratch merge with #1209 (main 46fe6ac19, #1209 at 13a2c58cb, then this head; local, never pushed, both merges clean). #1209 shares four files with this PR: tests/conftest.py, tests/test_non_utf8_name_callers.py, tests/test_non_utf8_name_helpers.py and tests/test_stdout_encoding.py. Run on 8 files: the three test files among those four, #1209's test_adopt_plan_paths_spelled.py and test_run_script_encoding.py, and test_adopt_template_values.py, test_family_head_by_hand_bytes.py and test_bootstrap_spelling.py. Against main plus #1209 alone:

• UTF-8: 483 passed and 1 skipped, then 504 passed and 1 skipped: the 21 new rows (the 20, and the [validate-family] row), nothing else.
• Latin-1: 19 failed, 411 passed, 54 skipped, then 5 failed, 433 passed, 67 skipped: 21 new rows passed, 1 failed to passed, 13 failed to skipped. The 5 failures are the test_bootstrap_spelling.py rows, which fail on both. Of the 50, the 15 in these files give 13 skipped and 2 passed, none failed.

The other 59 test files that name validate-family (round one, on the first commit, UTF-8): 3374 passed, 28 skipped, 0 failed. The second commit changes no tool.

The same launcher started as python -E (W1247's copy of it with -E added, used read-only), Latin-1, the 50: on main bd3e9c148, 50 failed, the 50 exactly as #1214 lists them. On the first commit, 0 failed. On this head, 1 failed, 4 passed, 45 skipped: the S1 row, test_a_utf8_source_directory_still_adopts_with_local_remotes, on #1247's traceback (UnicodeEncodeError on U+FFFD in git_filter_repo.py's _migrate_origin_to_heads, then REFUSED extracting the spec leg). It stays red with that launcher until #1251 lands, and the other 49 rows are as with the pip launcher.

The new rows, and mutants

Each mutant was made in a scratch worktree at the head and restored afterwards, with each file's sha256 checked. "UTF-8" and "Latin-1" name the locale of the run.

The second commit's rows. The helper's own rows run in CI's UTF-8 locale, with the three locales emulated.

Mutant	Run	Result
none (the head), test_non_utf8_name_helpers.py	UTF-8	39 passed (19 before, 20 new)
the new rows against main's tests/conftest.py, which has no helper	UTF-8	the file does not import: cannot import name 'skip_where_the_locale_does_not_spell_it_as_utf_8'
A5: the helper's UTF-8 bytes built with surrogatepass	UTF-8	3 failed: the byte-e9 row in each of the three locales. Before, the same mutant made 11 rows skip silently under UTF-8 and failed none (RV1258).
A1: the first question's skip taken as UTF-8's bytes	UTF-8	5 failed: iso8859-1 and ascii for bidi-override and ellipsis, and ascii-e-acute
A2: the second question dropped	UTF-8	1 failed: iso8859-1-e-acute (it reads the bytes back and says "reads", not "writes")
A4: only the third question	UTF-8	6 failed: the rows A1 and A2 fail, together
A3: the third question dropped	UTF-8	1 failed: iso8859-1-byte-e9
A6: the third question inverted	UTF-8	9 failed
the encoding in the reason named utf-8 always	UTF-8	7 failed
the row's premise left out of the first question's reason	UTF-8	5 failed
the UTF-8 bytes built with strict	UTF-8	4 failed: the three byte-e9 rows, and the row that holds the emulation to the real locale
the second and third questions swapped	UTF-8	1 failed
the first question catching any exception	UTF-8	survives, and is equivalent: os.fsencode of a str raises only UnicodeEncodeError
D3: validate-family.py's escape_unencodable_stdout() moved after parse_args	UTF-8	1 failed: the [validate-family] row
main's validate-family.py under the head's tests	UTF-8	2 failed: the [validate-family] row and test_the_gate_passes_on_a_stdout_that_cannot_hold_its_notes
N3's narrowed if taken out (main's if, with no helper call)	Latin-1	1 failed: test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 (it passes under UTF-8)

On the 50 under Latin-1 the helper's mutants are as in round one, re-measured on this head: A3 gives 10 failed (it was 11; the N3 row no longer asks the helper), the first question unguarded gives 11 failed, and A1, A2 and A4 survive (5 passed, 45 skipped), because on Linux the third question alone decides there, as RV1258 found. Only the new rows tell them apart.

The first commit's rows (mutants made on c8641c691, which the second commit does not change):

Mutant	Run	Result
none (the head), the new row of test_family.py	UTF-8	1 passed
main's validate-family.py under the head's tests (the new row on main)	UTF-8	1 failed: the new row
validate-family.py's main without the escape_unencodable_stdout() call (import kept)	Latin-1	4 failed: the new row, and the [add], [bump], [remove] rows of test_each_act_refuses_the_tracking_branch_then_lands_on_a_branch
the bidi guard taken out of skip_where_this_system_cannot_name	Latin-1	6 failed: the 4 bidi rows of test_clone_url_bytes.py and the 2 of test_local_remote_dir_urls.py, the same BAD_DIRECTORIES
windows_names_it without the os.fsdecode emulation	Latin-1	1 failed: test_where_windows_holds_a_bidi_override_the_name_check_does_not_skip
the guard of test_the_member_head_is_the_branch_git_wrote asked of every branch (its branch != NOT_UTF_8_BRANCH dropped)	Latin-1	[not-utf8] goes from passed (the head) to skipped; no row fails. A skip fails no run, so only a comparison of outcomes sees this one.

Merge-trees

git merge-tree --write-tree of the head (cd9f3cfd1):

• against origin/main 46fe6ac19: clean;
• against tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 13a2c58cb, tests/test_adopt_plan_paths_spelled.py: seven mutants of the plan written to and cloning ... into lines that no row killed now fail one, so a printable non-ASCII path prints as typed, ESC, CR and a non-UTF-8 --work-dir are spelled, a relative --out is not made absolute, an --out that ends in a space or is wrapped in backticks is spelled, and execute spells the clone line as plan does (#1167) #1185 d193617c7, adopt-project.py: plan written to and cloning <repository> (read only) into name the caller's --out and --work-dir with _shown, as the other lines of the report name a path, so a bidi control, a line feed or a tab in either no longer reaches the terminal as it was typed (#1115) #1154 8a7828332, adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 abeac66de and adopt-project.py: _pin_row_problem says a pin field whose value is not text as _read_as says what the plan's reader read, "records commit: as the number 7", where it spelled it as Python does, commit: 7 and True, so check and execute name it in the words the plan's other refusals use under the same plan-pin-malformed (#1114) #1198 d38e4e36d: each clean;
• against the other open PRs that touch any of these 18 files, at their heads: validate-family.py: a member whose git directory's HEAD this user cannot open, under .git/modules/<path> or in a .git directory, is refused member-unreadable by name, where it ended member-commit-unresolvable with a fetch (#1256) #1263 9ef8f0ab5, validate-family.py: a member whose .git file this user cannot open, or whose git directory (its own .git, or the one its gitfile names) this user cannot search, is refused member-unreadable naming that part, and no longer member-commit-unresolvable, which said the pinned commit was missing and prescribed a fetch (#1221) #1246 0deac3a65, update-shape.py: a line with U+00A0 or U+3000 in front of shape: or files: is no block header, as it is none to the reader (#1220) #1231 aca00dfd9, validate-family.py: the check for values git dies on reads the index's .gitmodules beside the work tree's and its stage remedies end git add -- .gitmodules, an index file git cannot parse is refused, and an unmerged one is read as none (#1150) #1227 2d56c1906, adopt-project.py: plan's table pads each name by the columns a terminal gives it, not the code points it holds, so a wide or combining name's leg column lines up under leg with every other row's (#1034) #1178 c4beba193, bootstrap.py, validate-family.py and siblings.py: a root or holder whose .git this user cannot read is refused git-repo-unreadable, exit 2, not a traceback or not-a-git-repo, and the doctor's members row names it once, never drift (#1148) #1172 2eabec8e0 and adopt-project.py: check and execute ask a plan's org: and id: right after its three repository names, so a bad one beside an unanswered entry, a moved source, a refused name or another repository is that one refusal in both, and a pins: entry that is no pin is said as _read_as says it (#1020) #1125 9549006fe are clean. family.py: the add, bump and remove lines its remedies name begin with the interpreter and the script's path, quote the row's name, fill in the family's org, say the branch they run on, and member-already-present prints its remove (#1127) #1201 eac177af9 conflicts at docs/handbook.html, which this branch does not touch, and conflicts there against origin/main alone too.

#1209 changes skip_where_a_name_cannot_be_not_utf_8 and the callers fixture a_system_that_carries_the_byte. The new helper sits more than 100 lines from the first, and windows_names_it 65 lines from the second. #1209 also edits tests/test_non_utf8_name_helpers.py, where the second commit adds its rows at the end of the file; the two merge without a conflict, and the merged file's rows pass (see "Runs").

Left, and outside this issue

• W1214's filter-repo defect is filed separately, as adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247, and adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 (a draft for it) is independent of this PR.
  • What W1214 found: adopt-project.py execute fails for a Latin-1 user whose source path holds U+00E9. The leg clone's remote.origin.url holds the byte 0xE9, git-filter-repo decodes it with errors='replace' and prints U+FFFD to a Latin-1 stdout, and execute stops at REFUSED extracting the spec leg.
  • Which launcher runs git-filter-repo decides it. One started as python -E ignores the PYTHONIOENCODING the tool gives its children; a pip-installed one, as CI installs it, is not started that way. Measured here (see "Runs"): test_a_utf8_source_directory_still_adopts_with_local_remotes passes on main under Latin-1 with the pip launcher, and fails there with a -E copy of the same launcher. W1196's list of 50 has this row failing, which this bench gives only with the -E launcher.
  • Independent of adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251. This PR no longer guards that row (RV1258's S1), so it carries nothing for adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 to drop, and the earlier plan, that whichever of the two PRs lands second drops this PR's guard, is void. adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 changes _extract_leg in adopt-project.py and adds tests/test_adopt_filter_repo_origin_url.py, and touches none of these 18 files (git merge-tree against its head abeac66de is clean). The row passes under Latin-1 with the pip launcher whichever lands first. With a -E launcher it fails on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's traceback until adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 lands, which is the tool's failure showing, not a premise this PR can judge.
• N2 (RV1258), known and left: under emulated Windows codecs, the helper skips the U+DCE9 rows that carry no Windows mark, with a premise that is false on Windows. It is reasoned and emulated, not run on Windows; CI runs Linux only (ci: run the test suite on Linux only #597).
• 5 rows of tests/test_bootstrap_spelling.py fail under Latin-1 on main, on this head and on the merge with tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 alike. It is a file tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 does not list, read from the failure messages only: four rows end in a UnicodeEncodeError on U+2028, which Latin-1 cannot encode, and test_a_branch_name_is_spelled_in_every_line_that_names_it fails its assertion. They are left for an issue of their own.
• 8 rows in two files tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 does not list fail under Latin-1 on main, on this head and on the merge with tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 alike:
  • 6 rows of test_pinned_copy_text.py: the tool prints U+2014 or U+2026 escaped on a Latin-1 stdout, and the row compares the character.
  • test_family.py's test_init_writes_the_holders_agent_files: it reads the UTF-8 file it checks in the locale's encoding.
  • test_family.py's test_commit_one_refuses_a_trailer_that_is_no_text_before_it_stages: DID NOT RAISE Refusal, a byte Latin-1 reads as text.
  • These are read from their failure messages only, not classified per test. They are left for an issue of their own.
• The fourth row of tests/test_adopt_template_values.py that fails under Latin-1 on main is test_execute_refuses_a_local_source_directory_that_is_not_utf8. It is not one of the 50, and this PR leaves it alone.
  • It asks skip_where_a_name_cannot_be_not_utf_8, and tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 gives that helper the locale question. So it skips at tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209's head, where the 50 were counted, and it skips on the scratch merge with tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209.
  • It fails on main and on this head alike, so it is tests/test_adopt_plan_paths_spelled.py: follow-up of PR #1185 (#1167) from its review: three of its rows fail under a one-byte locale where the tool is right (the 0xff --work-dir and the two U+00E9 paths), and its new helper builds the child's environment without blank_unnamed_pin_sources or the git identity defaults #1196's (tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209's) row, not tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214's.
  • The other three rows of that file that fail on main are among the 50: the two U+DCE9 flag rows and test_plan_and_check_say_the_same_sentence. Each skips here.
• tests/conftest.py: follow-up of PR #1209 (#1196) from its review: the helper's locale skip also catches 80 rows that can judge the tool under a one-byte locale, and four smaller gaps (a Windows failure, an unheld order, a triple predicate, a docstring) #1224 is untouched. This helper takes none of N1's over-skip; N2 to N5 stay tests/conftest.py: follow-up of PR #1209 (#1196) from its review: the helper's locale skip also catches 80 rows that can judge the tool under a one-byte locale, and four smaller gaps (a Windows failure, an unheld order, a triple predicate, a docstring) #1224's.

🤖 Generated with Claude Code
Lane: openreposhape-4 (openRepoShape-4)

Summary by Sourcery

Make family validation robust across non-UTF-8 stdout encodings and ensure locale-dependent tests skip only when their UTF-8-specific premises cannot be evaluated.

Bug Fixes:

• Make family validation succeed when stdout cannot encode its diagnostic characters instead of ending in an encoding traceback.
• Correct Windows filename emulation so locale-sensitive tests reflect Windows decoding behavior.

Enhancements:

• Add premise-aware locale guards so tests that specifically require UTF-8 behavior skip themselves under incompatible one-byte locales while retaining coverage for locale-independent cases.

Tests:

• Add coverage for validator output under Latin-1 and comprehensive tests for the new locale guard behavior and test classifications.

[sourcery-ai]

Reviewer's Guide

Implementation

Escape unencodable stdout in the family holder’s validator before argument parsing, matching the other entry points so a passing holder does not end in a UnicodeEncodeError traceback.

• templates/family-root/scripts/validate-family.py L188-190, L4297-4301

Tests

Add coverage for the validator’s Latin-1 stdout behavior and locale-aware premise guards, then apply those guards to rows that cannot judge UTF-8-specific behavior under a one-byte locale; emulate Windows filesystem decoding and test the helper’s UTF-8, Latin-1, and ASCII answers.

• tests/conftest.py L1285-1364
• tests/test_adopt_pins.py L49-56, L1088-1091 (+7 small edits)
• tests/test_adopt_spelled_names.py L45-48, L522-523
• tests/test_adopt_template_values.py L52-56, L209-211, L461-462
• tests/test_bootstrap_leg_moves.py L58-60, L532-533
• tests/test_bootstrap_spelling.py L72-75, L929-935
• tests/test_bump_leg_branch_and_push_line.py L60-64, L267-268
• tests/test_clone_url_bytes.py L47-50, L166-170, L178-183 (+1 small edits)
• tests/test_family.py L672-689
• tests/test_family_head_by_hand_bytes.py L279-283
• tests/test_family_member_branch_not_utf8.py L29-31, L118-120
• tests/test_family_tracking_branch.py L45-47, L219-220
• tests/test_non_utf8_name_callers.py L195-212
• tests/test_non_utf8_name_helpers.py L21-26, L41, L197-338
• tests/test_pin_source_unreadable.py L47-51, L314-317
• tests/test_stdout_encoding.py L149-152 (+7 small edits)
• tests/test_update_shape_apply_lines_and_headers.py L62-65, L518-519

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

RV1258: independent review of #1258 at c8641c69: NOT READY

Lane 4's independent adversarial review (task rv1258). The tool change is right and proved, and 46 of the 47 premise guards hold. Two SHOULDs stand between this head and READY: one guard skips a row that judges a real tool defect (#1247) under a false reason, and nothing CI runs holds the new helper's own answers.

Bench

• Own clone; head c8641c69; scratch merge 103ada82 of current origin/main 5926ccca and the head (local, never pushed). git merge-tree --write-tree is clean, and the merged tree differs from main only in the PR's 16 files.
• Runner /usr/bin/python3 -m pytest -p no:cacheprovider (3.12.3), git 2.43.0, the lane's gate environment (stub gh, GH_TOKEN=bogus, empty GH_CONFIG_DIR, empty GIT_CONFIG_GLOBAL, GIT_CONFIG_NOSYSTEM=1, test identity, EMAIL set, GIT_ALLOW_PROTOCOL/GIT_SSH_COMMAND/LANES_* unset).
• git-filter-repo 2.47.0 (--version a40bce548d2c) on PATH in every run: CI's pip launcher, and W1247's copy with -E (read-only) where marked -E. -rs shows no row skipped for lack of it in any gate run.
• Locales: UTF-8 LC_ALL=C.UTF-8. Latin-1 my own, built with localedef -c from a 256-entry charmap (Python reports iso8859-1, os.fsdecode(b"\xff") == "\xff"), LOCPATH, LC_ALL=en_US.ISO-8859-1, PYTHONUTF8=0. ASCII LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0.

1. The fix is real

• Head's new row against MAIN's validate-family.py (the file swapped into the merge tree, then restored, sha256 checked): UTF-8 1 failed, Latin-1 4 failed (the new row and the [add], [bump], [remove] tracking-branch rows). The decisive line: the holder's scripts/validate-family.py, line 4340 print(note), UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026'. On the merge all 4 pass in both locales.
• Dropping the call (import kept, mutant D1): UTF-8 1 failed (the new row; it forces PYTHONIOENCODING=latin-1, so CI's locale sees it), Latin-1 4 more failed than the baseline (the same 4).
• Shipped beside the validator: repo_shape.py is not in templates/family-root/scripts/; init copies scripts/repo_shape.py there, a pinned row of the holder's shape-pin.yaml (- path: scripts/repo_shape.py in the fixture holder), and that copy defines escape_unencodable_stdout (1 match), which the holder's siblings.py and bootstrap.py already import.
• The row runs the holder's COPY (fam/InkRouter/scripts/validate-family.py), so a holder init makes takes the change; an existing holder takes it when update-shape.py apply re-copies the file and moves its shape-pin.yaml row, as that file's own header says. No static digest exists: templates/family-root/contracts/shape-pin.yaml carries {{SHAPE_FILES}}, and git grep finds neither the old blob id nor the file's sha256 anywhere.
• Skew: git log -S gives ec4a7a0e5 as the commit that added BOTH def escape_unencodable_stdout and def spelled to scripts/repo_shape.py, and validate-family.py already imports spelled, so no kept library that serves today's validator lacks the new name. Neither is in test_pinned_copy_text.py's NEWER_THAN_A_KEPT_LIBRARY; test_no_pinned_script_imports_what_an_older_library_lacks[validate-family] and test_update_shape_kept_helper.py pass on the merge in both locales.

2. Over-skip: each of the 47, by premise

Read from each row's code and from its failure on main under Latin-1 with the -E launcher, where all 50 fail (exactly #1214's list):

Premise	Rows	Main's Latin-1 failure (sample)	Premise false under Latin-1?
skip_where_the_environment_cannot_carry (a1)	18: test_stdout_encoding.py 10, test_adopt_pins.py 5 [zwj], test_pin_source_unreadable.py 3 [not-ascii-*]	UnicodeEncodeError: 'latin-1' codec can't encode U+65E5 U+672C or U+200D, raised in the test process	yes: no Latin-1 byte exists for them
A_BYTE_READ_AS_A_SURROGATE	11: test_adopt_pins.py 4, test_pin_source_unreadable.py 3, test_adopt_template_values.py 3, test_family_head_by_hand_bytes.py 1	REFUSED pin-source-unreadable: .../miss<0xFF>ing, adopt-not-the-source, legs created for --elected-by Jos<0xE9>, assert 'caf\xe9' == 'caf\udce9'	yes: 0xFF and 0xE9 are text there
A_NAME_HELD_AS_UTF_8	7: tracking 1, bump-leg 2, update-shape 1, member 1, spelled-names 1, test_a_utf8_source_directory_still_adopts_with_local_remotes 1	REFUSED family-branch-not-utf8 ... b'feat\xa0', bump-leg-branch-not-utf8, update-branch-not-utf8, 'feat\udca0' != 'feat\xa0', `caf\udce9.md`	yes for 6; not for the source-directory row (S1)
PRINTED_AS_ITSELF	7: test_adopt_pins.py 5, test_bootstrap_leg_moves.py 2	the line ends \u2026 or holds \u2014 escaped (#214)	yes
A_NAME_READ_AS_UTF_8	4 (and the 2 test_local_remote_dir_urls.py bidi rows)	- ... holds a bidirectional control / + ... holds a control character.	yes: E2 80 AE reads as three characters, one U+0080

• Would a defect hide? I scanned each of the 50 failure texts on main (-E) for a tool traceback. There are only two kinds: the 3 tracking-branch rows (the validator, fixed here) and test_a_utf8_source_directory_still_adopts_with_local_remotes (filter-repo, UnicodeEncodeError on U+FFFD: adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247). Every other guarded row fails with no tool traceback.
• Under UTF-8 every row still runs: see 3. The helper, asked through skip_message with emulated codecs, answers "runs" for every premise's text under UTF-8; under ASCII it runs a surrogate byte and skips U+00A0, U+2026, U+2014 and U+202E.

3. Counts by test id

The gate, 35 files (the body's 27, test_path_guard_hygiene.py, and the family-root copy and stdout tests test_bootstrap_spelling.py, test_validator_child_encoding.py, test_siblings_family_name.py, test_family_docs_one_spelling.py, test_validate_family_git_not_runnable.py, test_family_init_values.py, test_update_shape.py), pip launcher, run two at a time:

	main 5926ccca	merge 103ada82
UTF-8	2529 passed, 20 skipped	2530 passed, 20 skipped
Latin-1	135 failed, 2370 passed, 44 skipped	56 failed, 2401 passed, 93 skipped

• UTF-8, by id: one change, the new row (absent to passed). The same 20 skips on both: 19 test_family_init_values.py template-design rows and the Windows default-stdout row. No new UTF-8 skip.
• Latin-1, by id:
  • 0 rows go from passed to failed, and 0 from skipped to failed. Every one of the merge's 56 failures fails on main.
  • 31 go from failed to passed: the 3 tracking-branch rows and 28 test_family.py rows, each a UnicodeEncodeError on U+2026 in the holder's gate on main.
  • 48 go from failed to skipped: 46 of the 50, and the 2 test_local_remote_dir_urls.py bidi rows.
  • 1 goes from passed to skipped: test_a_utf8_source_directory_still_adopts_with_local_remotes (S1).
  • 1 new row passes.
• The 50, Latin-1: main -E 50 failed. Head c8641c69: pip 3 passed and 47 skipped, -E the same. Merge: pip and -E the same, 0 failed. UTF-8 on the head: 50 passed.
• The PR's 14 test files, Latin-1, -E: main 99 failed, 656 passed, 15 skipped; merge 21 failed, 688 passed, 62 skipped. By id: 0 from passed to failed; 31 from failed to passed; 47 from failed to skipped, the source-directory row among them, as it fails on main with -E; and the new row. Each of the 21 fails on main. On main, the two launchers differ on one row of these 770: that one.
• ASCII, the 14 files: main 96 failed, 649 passed, 25 skipped; merge 41 failed, 672 passed, 58 skipped. By id: 0 from passed to failed or skipped; 22 from failed to passed (19 test_family.py, the 3 tracking-branch rows); 33 from failed to skipped; and the new row. The 11 A_BYTE_READ_AS_A_SURROGATE rows still run and pass there, as the helper means. Two of the 50 still fail under ASCII, on main and here alike, before any guard: test_a_clone_the_plan_records_is_printed_as_how_it_was_read[no-break-space] (UnicodeEncodeError on U+00A0 in the test) and test_check_passes_the_clone_a_plan_records_and_writes_no_path (check exits non-zero). tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 asks ASCII only to keep its counts, which it does; this is noted, not graded.

Main moved while the gate ran, to 12122952 (2 commits: adopt-project.py, scripts/shape_materialize.py, templates/assembly-root/AGENTS-shape.md and three test files, test_pinned_copy_text.py among them; none of the PR's 16). git merge-tree against it is clean and differs from it only in the 16 files. On a second scratch merge 3137e45c, the 19 files most exposed (the PR's 14 test files, test_pinned_copy_text.py, test_adopt_git_reads.py, test_adopt_plan_escapes.py, test_repo_hygiene.py, test_path_guard_hygiene.py): UTF-8 1175 passed and 1 skipped; Latin-1 29 failed, 1080 passed, 67 skipped. By id, no outcome differs from the first merge in either locale; the 50 give 50 passed under UTF-8, and 3 passed and 47 skipped under Latin-1. Main then moved once more, to 46fe6ac1 (one commit: scripts/shape_advisory.py, shape-doctor.py and a doctor test, none of them a gate file). git merge-tree is clean, and the result differs from main only in the 16 files. On that merge 62b3b153, test_stdout_encoding.py, test_family.py and test_doctor_validator_verdicts.py, the gate files that run shape-doctor.py and the holder's gate most: UTF-8 199 passed and 1 skipped; Latin-1 4 failed (the two --out rows #1209 settles and the two test_family.py rows under the body's "Left", each failing on main), 185 passed, 11 skipped. No outcome differs from the first merge by id.

4. windows_names_it

Right. Windows' os.fsdecode is UTF-8 with surrogatepass in every locale (PEP 529), and windows_fsdecode passes a str through and takes os.fspath first, as the real one does. My mutant E1 (the os.fsdecode line dropped from windows_names_it): Latin-1 1 failed, test_where_windows_holds_a_bidi_override_the_name_check_does_not_skip; UTF-8 36 passed. So the row asked the host's codec before, and asks Windows' now. With it the callers file gives 36 passed in both locales, and its other three Windows rows still skip for the reasons they name (os.fsencode, "control character").

5. Mutants (mine; each applied to a worktree of the merge, restored with git checkout, sha256 of all five files checked after every one)

Set: the 50, the 2 test_local_remote_dir_urls.py bidi rows, the 4 test_clone_url_bytes.py [not-utf8] rows, member [not-utf8], the new row, and all of test_non_utf8_name_callers.py (94 rows). Baseline: UTF-8 94 passed; Latin-1 4 failed (the 4 [not-utf8] rows, which fail on main too: #1209's), 41 passed, 49 skipped.

Mutant	Latin-1	UTF-8 (CI's locale)
A1 the first question takes an unspellable text as UTF-8's bytes	survives (as baseline)	-
A2 the second question dropped	survives	-
A3 the third question dropped	killed: +11 failed, the 11 A_BYTE_READ_AS_A_SURROGATE rows	-
A4 only the third question left (first two deleted)	survives; under ASCII survives too	-
A5 the second question's UTF-8 bytes made with surrogatepass	survives	survives: 83 passed, 11 skipped, 0 failed (the 11 A_BYTE rows skip silently)
A6 the third question inverted	killed: +16 failed	killed: 5 callers rows fail
B1 the bidi guard dropped	killed: +6 failed (4 + 2 bidi rows)	-
B2 the locale question also asked of a name that is not UTF-8	killed: 4 callers rows fail (and the 4 [not-utf8] rows skip)	-
C1 the member guard asked of every branch	survives: [not-utf8] passed to skipped, nothing fails	-
C2 the member guard inverted	killed: [trailing-no-break-space] fails	-
D1 the escape_unencodable_stdout() call dropped	killed: +4 failed	killed: the new row
D2 the call moved to after the notes print	killed: +4 failed	killed: the new row
D3 the call moved to after parse_args	survives (equivalent today: the parser's text is ASCII)	survives
E1 windows_names_it without the os.fsdecode emulation (callers file alone)	killed: 1 failed	survives: 36 passed

A1, A2 and A4 show that in the three locales measured the third question alone decides; the first two act only under an emulated Windows (N2). A5 is S2.

6. Merge-trees

git merge-tree --write-tree of the head, each clean: against origin/main 5926ccca; #1209 13a2c58c; #1185 d193617c; #1154 8a782833; #1251 825923c7; #1198 d38e4e36; and the scratch merge against each of those five, clean too. Main moved 9 commits since the body's bd3e9c14, one of them in validate-family.py; in the merged file main's first statement is still the call. Main + this head + #1209 (local 80e44af7), the five files both touch or share (test_non_utf8_name_callers.py, test_non_utf8_name_helpers.py, test_stdout_encoding.py, test_clone_url_bytes.py, test_adopt_pins.py): UTF-8 287 passed and 1 skipped; Latin-1 221 passed, 67 skipped, 0 failed.

Findings

S1 (SHOULD): the guard on test_a_utf8_source_directory_still_adopts_with_local_remotes hides #1247 under a false reason, and should not stay.

• On main under Latin-1 with CI's pip launcher the row PASSES: execute adopts the source under caf<0xE9>. The skip now given says "held as other bytes it is a name that is not UTF-8, and the tool names it as one". The tool names nothing; it adopts. It is the only row this PR takes from passed to skipped.
• With the -E launcher it is the one guarded row whose failure on main is a tool traceback: filter-repo's UnicodeEncodeError on U+FFFD, adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247. tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 sends such a row to "a defect, and gets its own issue", which adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247 is; it does not ask for a skip.
• The body plans to drop the guard when adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 lands ("whichever of the two PRs lands second drops this PR's premise guard"). A guard planned to go when a tool fix lands is not a premise guard; and the plan lives only in two PR bodies, while adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 does not touch this file.
• Change: drop the guard and the A_NAME_HELD_AS_UTF_8 import it alone uses in tests/test_adopt_template_values.py. Under the pip launcher the row passes under Latin-1, so the 50 still give 0 failed. Under -E it fails on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247 alone until adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 lands, and the two PRs no longer need to know about each other.

S2 (SHOULD): no row CI runs holds the new helper's answers, so a plausible slip skips 11 rows in CI and nothing goes red.

• Mutant A5 (text.encode("utf-8", "surrogatepass") for the second question's bytes) makes every A_BYTE_READ_AS_A_SURROGATE row skip under UTF-8: 83 passed, 11 skipped, 0 failed, against 94 passed. conftest.py's own skip_message docstring names this hazard: "a helper that skipped where it should not would pass".
• Only the names skip_where_this_system_cannot_name asks it of (ASCII controls and U+202E) have CI-run witnesses, the callers rows that kill A6. No row witnesses A_BYTE_READ_AS_A_SURROGATE, A_NAME_HELD_AS_UTF_8 or PRINTED_AS_ITSELF text.
• Change: add rows for skip_where_the_locale_does_not_spell_it_as_utf_8 beside the other helpers' (tests/test_non_utf8_name_helpers.py), each through skip_message with os.fsencode and os.fsdecode emulated:
  • UTF-8 returns None for 'gl\udcffass', 'feat\xa0', '\u2026' and '\u202eevil';
  • Latin-1 skips with the premise in the reason, once by each question that can answer there;
  • ASCII returns None for '\udcff' and skips '\u2026'.
    A5 must then fail in CI's locale.

N1 (NIT): nothing holds that the call is main's FIRST statement, as the comment says (D3 survives). test_bootstrap_spelling.py::test_main_escapes_stdout_before_it_prints_or_parses_anything holds it for both roots' bootstrap.py. Change: add templates/family-root/scripts/validate-family.py to its parametrization.

N2 (NIT, reasoned, not run on Windows): under Windows' codecs the helper's second question skips a surrogate byte, with a reason ("here it is a character, and the tool takes it as one") that is false there. A Windows command line is UTF-16 and hands U+DCE9 to the child as itself. Measured by emulation (skip_message, with os.fsencode/os.fsdecode patched to Windows' codec; P stands for the premise):

--- windows (utf-8 surrogatepass)
  A_BYTE   'Jos\udce9'        -> SKIP: this locale's encoding (utf-8) writes 'Jos\udce9' as b'Jos\xed\xb3\xa9', not as UTF-8's b'Jos\xe9': P

The rows with no POSIX-only mark are test_plan_refuses_a_flag_no_file_can_hold's two U+DCE9 rows and test_plan_and_check_say_the_same_sentence. The third question skips there too (Windows' os.fsdecode raises on the byte 0xE9, read as None), so no reordering of the three cures it. Change: ask the locale question only off Windows, where a command line is bytes, or say in the docstring and the reason what Windows answers.

N3 (NIT): test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 now ends SKIPPED under Latin-1, though its main claim, Python's own surrogateescape read, ran and passed first. Change: narrow its existing if to the case its docstring names, by adding and os.fsdecode(b"\xe9") == chr(0xDCE9) after the os.fsencode half (which keeps Windows, where that os.fsdecode raises, from asking), and drop the guard; the row then passes under Latin-1.

Lane: openreposhape-4 (openRepoShape-4)

[brettheap]

F1258 fix round on this PR for RV1258's NOT READY review (#1258 (comment)).

New head: cd9f3cfd1137229e1b8e23b2d82376833c536f5e. One new commit on top of c8641c691, pushed without force, no rebase, no amend. It changes four test files and no tool. Please review the DELTA only (git diff c8641c691 cd9f3cfd1); the first commit has been reviewed.

Findings and fixes

• S1 (guard on a row that passes): fixed. The guard on test_a_utf8_source_directory_still_adopts_with_local_remotes and the A_NAME_HELD_AS_UTF_8 import it alone used in tests/test_adopt_template_values.py are gone. The row runs under Latin-1 and passes with the pip launcher. With the -E launcher it fails on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's traceback (UnicodeEncodeError on U+FFFD, then REFUSED extracting the spec leg) until adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 lands, as expected. The PR body now says plainly that validate-family.py: the holder's gate escapes what its stdout cannot encode, so it passes under Latin-1 where it ended in a traceback, and 45 rows in 11 test files that cannot judge under a one-byte locale skip there on their own premise (#1214) #1258 and adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 are independent of each other and that the "whichever lands second drops the guard" plan is void. (adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251's own body still carries that sentence; I left it alone.)
• S2 (nothing CI runs held the helper's answers): fixed. tests/test_non_utf8_name_helpers.py gets 20 rows, built as that file builds its Windows rows (os.fsencode, os.fsdecode and sys.getfilesystemencoding replaced for one test): 15 hold which question skips each of five texts in UTF-8, Latin-1 and ASCII (none, "cannot spell", "writes", "reads") plus the reason's encoding, ending and ASCII; 5 hold the emulation to the run's real locale.
• N1: fixed. The first-statement row of tests/test_bootstrap_spelling.py also reads templates/family-root/scripts/validate-family.py.
• N3: fixed. test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9 no longer asks the helper; its if also asks os.fsdecode(b"\xe9") == chr(0xDCE9). It passes under Latin-1 where it skipped, and asserts as before under UTF-8 and ASCII (run in all three).
• N2: known, left, named in the body (emulated Windows only, not run on Windows).

Counts (scratch merge of this head with origin/main 46fe6ac19, pip launcher; baseline is that main; by test id)

• 29 files (the body's 27, test_path_guard_hygiene.py and test_bootstrap_spelling.py; both hygiene files and test_non_utf8_name_helpers.py included):
  • UTF-8: main 1873 passed, 1 skipped; merge 1895 passed, 1 skipped. The only change is 22 new rows, all passed. No new skip, nothing passed to failed.
  • Latin-1: main 128 failed, 1721 passed, 25 skipped; merge 49 failed, 1775 passed, 72 skipped. 0 passed to failed, 0 passed to skipped (the S1 row used to be the one), 32 failed to passed, 47 failed to skipped, 22 new rows passed; each of the 49 failures also fails on main.
• The 50, Latin-1, pip launcher: main 49 failed, 1 passed; merge 0 failed, 5 passed (the S1 row and the N3 row among them), 45 skipped. UTF-8: 50 passed on both.
• The 50, Latin-1, -E launcher: 1 failed (the S1 row, on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's traceback), 4 passed, 45 skipped.
• Scratch merge with tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 (8 shared and related files): UTF-8 483 -> 504 passed, 1 skipped; Latin-1 19 -> 5 failed (the 5 test_bootstrap_spelling.py rows that fail on main too), 13 failed to skipped, 1 failed to passed.

Main moved while this was gated (9729315f1, then 801a7d6d0, neither touching a gated file): the merge re-run on 9729315f1 gave identical outcomes by test id on the same 29 files in both locales, and on 801a7d6d0 the 50 and the four files the commit changes gave the same results.

Mutants (each restored, sha256 checked)

• A5 (surrogatepass): was 11 rows skipping silently under UTF-8; now 3 failed (the byte-e9 rows).
• RV1258's survivors A1, A2, A4: now 5, 1 and 6 failed (UTF-8, test_non_utf8_name_helpers.py). A3 (1 failed) and A6 (9 failed) still die. They still survive on the 50 under Latin-1, as you found; only the new rows tell them apart.
• D3 (call moved after parse_args): 1 failed, the [validate-family] row.
• My own on the new rows: encoding named utf-8 always 7 failed; premise left out of a reason 5; UTF-8 bytes built strict 4; questions 2 and 3 swapped 1; N3's narrowed if taken out, 1 failed under Latin-1. One survives and is equivalent (the first question catching any exception).
• The new rows fail on main's code: the file does not import without the helper, and the [validate-family] row and the first commit's test_family.py row fail on main's validate-family.py.

Merge-trees (git merge-tree --write-tree, head against each): clean against origin/main, #1209, #1185, #1154, #1251 and #1198, and against #1263, #1246, #1231, #1227, #1178, #1172 and #1125; #1201 conflicts at docs/handbook.html, which this branch does not touch, and conflicts there against main alone too.

Also found, outside this issue (left, named in the body): 5 rows of tests/test_bootstrap_spelling.py fail under Latin-1 on main and here (four on U+2028 in a name Latin-1 cannot encode, one on an assertion).

The body and title are updated: the counts (45 rows in 11 files skip, 2 run), the table, the second commit, the runs, and "Left".

Lane: openreposhape-4 (openRepoShape-4)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[brettheap]

RD1258: independent delta review of #1258 at cd9f3cfd: READY

Scope: git diff c8641c69 cd9f3cfd only (four test files, no tool), plus the PR body and title. No MUST and no SHOULD. Three NITs. RV1258's two SHOULDs and its N1 and N3 are each settled, and I could not make any of them come back.

Everything below was run with CI's git-filter-repo pin (2.47.0, the pip launcher with no -E) unless it says -E, on a scratch merge e4c6d2bce of cd9f3cfd1 into the current origin/main 51a9c5f10 (local, never pushed). The merged tree differs from main in exactly the PR's 18 files. Latin-1 is a localedef locale (LC_ALL=en_US.ISO-8859-1 PYTHONUTF8=0), ASCII is LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0.

What I measured

S1 (the guard on the source-directory row is gone). The guard and the A_NAME_HELD_AS_UTF_8 import are gone from tests/test_adopt_template_values.py, and pyflakes is clean on all four files. test_a_utf8_source_directory_still_adopts_with_local_remotes:

• UTF-8: passes (it runs, it is not skipped).
• Latin-1, pip launcher: passes on the head, and also passes on main.
• Latin-1, -E launcher: fails on the head, and only on adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247's traceback. The message is UnicodeEncodeError: 'latin-1' codec can't encode character U+FFFD in git_filter_repo.py _migrate_origin_to_heads, then REFUSED extracting the spec leg, and assert_no_traceback trips on it.
• Latin-1, -E, with adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 merged on top of the head, and with adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 on main alone: the row passes in both.
• ASCII: skipped by skip_where_the_environment_cannot_carry, as before.

S2 (the 20 new rows).

• I held fsencode_in and fsdecode_in to the real os.fsencode and os.fsdecode in each real locale, over 2833 texts (every code point below U+0300, every surrogate, the characters the rows use) and 266 byte strings (all 256 single bytes, multi-byte and invalid runs), by result or by exception type, and with a str, bytes, a path-like and a non-path. UTF-8, Latin-1 and ASCII each: 0 differences. They match CPython's _fscodec (the codec with surrogateescape, os.fspath first). The encoding names the rows give (utf-8, iso8859-1, ascii) are the ones sys.getfilesystemencoding() returns in those three runs.
• The helpers file gives 39 passed in each of UTF-8, Latin-1 and ASCII (19 before, 20 new, no skip).
• The 15 emulated rows are independent of the run's locale. The 5 rows that hold the emulation to the real locale do hold it: mutant T1 (the emulated fsencode using surrogatepass) fails the [byte-e9] real-locale row as well as the three emulated byte-e9 rows.
• RV1258's survivors now die under CI's UTF-8 locale (table below): A5 3 failed (it was 11 silent skips), A1 5, A2 1, A4 6.

N1. test_main_escapes_stdout_before_it_prints_or_parses_anything[validate-family] passes on the head under UTF-8 and Latin-1. Mutant D3 (the call moved after parse_args) fails it, and so does D1 (the call dropped).

N3. test_surrogateescape_reads_the_issues_branch_as_caf_and_u_dce9: 1 passed in UTF-8, Latin-1 and ASCII on the head. On the first commit it was skipped under Latin-1, and on main it fails there. The assertion still runs where it ran: replacing it with its inverse fails the row under UTF-8 and under ASCII and passes under Latin-1 (where the if is false). The old if put back fails the row under Latin-1 only.

Mutants

Each was made in a scratch worktree at the head and restored afterwards, and every file's sha256 was checked equal to the head's. Run: UTF-8, tests/test_non_utf8_name_helpers.py (the head gives 39 passed).

Mutant	Result
A5: UTF-8 bytes built with surrogatepass	3 failed (the byte-e9 rows)
A1: the first question's skip taken as UTF-8's bytes	5 failed
A2: the second question dropped	1 failed
A4: only the third question	6 failed
A3: the third question dropped	1 failed
A6: the third question inverted	9 failed
O2: the reason's encoding read from sys.stdout.encoding	7 failed
O3: os.fsencode captured at import, not asked at call time	6 failed
O4: UTF-8 bytes built with backslashreplace	3 failed
O9, O5, O10: the premise dropped from the first, second, third question's reason	5, 1, 1 failed
O6: the third question's reason built with repr (not ASCII)	1 failed
O11: UTF-8 bytes built strict	4 failed
O12: the second and third questions swapped	1 failed
O14: the first question's reason shows UTF-8's bytes in place of the text	5 failed
T1, T2, T3: the emulation itself (surrogatepass; ASCII fsdecode via Latin-1; encoding name fixed at utf-8)	4, 1, 7 failed
O1: the second question's reason shows UTF-8's bytes in place of the bytes spelled	survives (NIT 1)
O13: the third question's reason shows the text in place of what was read	survives (NIT 1)
O7: the second question compares lengths only	survives; equivalent for these three codecs
O8: ASCII text returns before any question	survives; equivalent for any ASCII-compatible locale

The same helper mutants over the 50 under Latin-1 (pip): A1, A2, A4 and A5 give 5 passed, 45 skipped (they survive there, as RV1258 found, and only the new rows tell them apart); A3 and A6 give 10 failed, 5 passed, 35 skipped.
For the validator: main's validate-family.py under the head's tests gives 2 failed (UTF-8: the new test_family.py row and the [validate-family] row); the escape call dropped gives 4 failed under Latin-1 (the [add], [bump] and [remove] rows and the test_family.py row).

No new over-skip (29-file gate, by test id)

The 29 files are the body's 27 plus test_path_guard_hygiene.py and test_bootstrap_spelling.py (both hygiene files and the helpers file included). Main is 51a9c5f10.

Run	main	scratch merge
UTF-8	1873 passed, 1 skipped	1895 passed, 1 skipped
Latin-1	128 failed, 1721 passed, 25 skipped	49 failed, 1775 passed, 72 skipped

• UTF-8: the only change is 22 new rows, all passed. No new skip, nothing from passed to failed.
• Latin-1: 0 rows go from passed to failed, and 0 from passed to skipped. 32 go from failed to passed (4 of the 50, and 28 rows of test_family.py whose holder gate ended in the same traceback), 47 from failed to skipped (45 of the 50 and the 2 bidi rows of test_local_remote_dir_urls.py), and 22 are new rows that pass. Each of the 49 merge failures also fails on main. No row skipped for lack of git-filter-repo.
• The four changed files under ASCII, main vs merge: 353 passed, 4 failed, 7 skipped, then 374 passed, 4 failed, 7 skipped. The 21 new rows are the only change, and the same 4 test_bootstrap_spelling.py rows fail on both.
• 8 related files on a scratch merge with tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 (13a2c58cb), against main plus tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 alone: UTF-8 483 passed, 1 skipped, then 504 passed, 1 skipped. Latin-1 19 failed, 411 passed, 54 skipped, then 5 failed, 433 passed, 67 skipped: 21 new rows pass, 1 failed to passed, 13 failed to skipped, and the 5 failures are the test_bootstrap_spelling.py rows that fail on both.

The 50, by test id (the ids F1258 used, the same 50 as #1214):

Run	main	scratch merge
UTF-8, pip	50 passed	50 passed
Latin-1, pip	49 failed, 1 passed	0 failed, 5 passed, 45 skipped
Latin-1, -E	50 failed	1 failed (the S1 row, #1247), 4 passed, 45 skipped

The body and the title

• The split is true. The 5 that pass under Latin-1 are the 3 test_each_act_refuses_the_tracking_branch_then_lands_on_a_branch rows (the tool change), the S1 row and the N3 row. The 45 that skip are in 11 files (14 test_adopt_pins, 10 test_stdout_encoding, 6 test_pin_source_unreadable, 4 test_clone_url_bytes, 3 test_adopt_template_values, 2 each in test_bootstrap_leg_moves and test_bump_leg_branch_and_push_line, 1 each in test_adopt_spelled_names, test_family_member_branch_not_utf8, test_family_tracking_branch and test_update_shape_apply_lines_and_headers). By their skip reasons 18 are skip_where_the_environment_cannot_carry (a1) and 27 are the new helper (a2): 18 + 27 + 3 + 2 = 50. The title's "45 rows in 11 test files" is true. Every count in the Runs section that I re-measured (both gates, the 50 in three modes, the tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 run) matches to the row, and main has moved since to 51a9c5f10 without touching a gated file or a file of this PR.
• "validate-family.py: the holder's gate escapes what its stdout cannot encode, so it passes under Latin-1 where it ended in a traceback, and 45 rows in 11 test files that cannot judge under a one-byte locale skip there on their own premise (#1214) #1258 and adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 are independent" is true. Their file lists are disjoint (adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251: adopt-project.py and tests/test_adopt_filter_repo_origin_url.py), the merge-tree is clean, and with adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 merged on top of the head the 50 give 0 failed under Latin-1 with either launcher (5 passed, 45 skipped each). Neither needs the other.
• Closing line: the body names tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 as the issue it closes exactly once, closingIssuesReferences is [1214], the PR is a draft, and the delta's commit message has no closing keyword.

Merge-trees

git merge-tree --write-tree of cd9f3cfd1 against origin/main 51a9c5f10, #1209 13a2c58cb, #1185 d193617c7, #1154 8a7828332, #1251 abeac66de and #1198 d38e4e36d: all six clean. File overlap with the head: #1209 shares conftest.py, test_non_utf8_name_callers.py, test_non_utf8_name_helpers.py and test_stdout_encoding.py; #1185 and #1154 share the last two of those; #1198 shares test_adopt_pins.py; #1251 shares nothing.

Findings

• NIT 1: the tails of two reasons are not held. Only the opening of a "writes" or "reads" reason is compared (what_the_reason_says), plus its ending and ASCII. Mutants O1 and O13 change what the middle shows (the bytes spelled, the text read) and survive. Change: build the whole expected reason from the case table, or assert repr(spelled) and ascii(read) are in it. The reason is informative only, so this is a follow-up.
• NIT 2: adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251's body still carries the void plan. Its "Overlap with tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214" paragraph says a tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 guard exists and that whichever of tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214 and adopt-project.py: under a one-byte locale such as Latin-1, execute stops extracting the spec leg of a source whose path holds U+00E9, because git filter-repo prints the clone's origin url as U+FFFD to a stdout that cannot hold it #1247 lands second drops it. After this delta validate-family.py: the holder's gate escapes what its stdout cannot encode, so it passes under Latin-1 where it ended in a traceback, and 45 rows in 11 test files that cannot judge under a one-byte locale skip there on their own premise (#1214) #1258 carries no such guard. F1258 left it alone and said so. Change: edit that paragraph of adopt-project.py: execute unsets the leg clone's remote.origin.url before git filter-repo, so under a Latin-1 locale a source under a directory named with the byte 0xE9 is extracted, where filter-repo died printing that url as U+FFFD, and every branch of the source is still rewritten as before (#1247) #1251 to say validate-family.py: the holder's gate escapes what its stdout cannot encode, so it passes under Latin-1 where it ended in a traceback, and 45 rows in 11 test files that cannot judge under a one-byte locale skip there on their own premise (#1214) #1258 holds no guard for it to drop (not part of this PR).
• NIT 3: N2 stays known and open. Under emulated Windows codecs the helper skips the unmarked U+DCE9 rows with a premise that is false on Windows. Reasoned and emulated, never run on Windows, and CI is Linux only (ci: run the test suite on Linux only #597). Already named in the body.

Not graded: 4 to 5 test_bootstrap_spelling.py rows fail under Latin-1 and ASCII on main and on this head alike (named in the body's "Left"). At the time I looked, the tests job on the head was still pending; the SonarCloud, package and tests-git-bash checks had passed.

Lane: openreposhape-4 (openRepoShape-4)