Skip to content

install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR the bytes refuse is read again in the locale's own charset, so a GB18030, GBK or Shift_JIS name that holds no control character and nothing drawn as another is placed, and where nothing can read it the bytes decide as before - #867

Merged
brettheap merged 19 commits into
mainfrom
fix/858-bin-dir-bytes
Oct 18, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Lane: openreposhape-2 (openRepoShape-2)
Claim: #858 (comment)

Stacked on

Fix round 1, on rv858's review (READY, 2 SHOULDs, 5 NITs)

Two commits on top of 2d1ff8a1: 18ae33b1 (the fixes) and 8bb85cbf (a test row the full mutant table asked for: F07, below). Each mutant named here is rv858's, or new with the fix, and is killed in the full table that follows.

Finding What changed Mutant now killed, and by what
SHOULD 1: no test had a multibyte charset read a name whose reading still holds a C1 control or one of the 41 two rows through both files: a hanzi of the C1 bytes beside U+009B in GB18030's own form (0x81 0x30 0x83 0x37), refused as a control character, and beside U+202E (0x81 0x36 0xA7 0x31), refused naming (U+202E); install.sh before any request R1 (printf '%s' "$reading" → printf '%s' x, both files): gb18030-reads-u+009b, gb18030-reads-u+202e
SHOULD 2: the stub locale answered one charmap whatever it was asked under the stub answers as glibc's does, from the first of $LC_ALL, $LC_CTYPE and $LANG set (a marker xx_XX.<charmap>, C.UTF-8, else ANSI_X3.4-1968); the marker cases run with LANG=xx_XX.<charmap> and LC_ALL, LC_CTYPE empty R2 (LC_ALL=C locale charmap, both files): every placed case
NIT 3: nothing pinned "only a value the bytes refuse is read again" café under GB18030, with an iconv that leaves a mark if it runs: placed, no mark, both files R3 (the launcher's trigger → if :; then): no-byte-refused-never-read-again; and its install.sh twin, I12
NIT 4: an iconv that exits 0 and prints nothing passed the count the reading must write back into the charset as the value, byte for byte: [ "$(printf '%s' "$reading" | iconv -f UTF-8 -t "$charmap" 2>/dev/null)" = "$1" ], in both copies (the identity test still passes). A test with such an iconv: today's refusal. The rows where iconv must not be asked now use the marking iconv, not one that reads anything, which the round trip would refuse anyway G01 (no round trip): an-iconv-that-prints-nothing; G02 (back into UTF-8) and G03 (!=): every placed case. The round trip made F07 (the first iconv's || return 1 dropped) survive at 18ae33b1, since it also refuses what a failing iconv printed in part; 8bb85cbf adds an-iconv-that-reads-then-fails (this machine's iconv run in full, then exit 1), which only that exit check refuses
NIT 5: the comments did not state the trade one sentence each in locale_reading's comment, both callers' and export_path_line's (which now says the C1 controls and the 41 are refused as the locale reads the name) (comments)
NIT 6: the body cited the brief "Line 1" now cites the issue's Direction: "for another multibyte charset, either refuse nothing the locale reads as a letter" (body)
NIT 7: line 2's reason was incomplete the launcher's comment and "Line 2" name the ISO 8859 case as a candidate, left: the charset is known and a<0x9B>b is placed (comment, body)

C and C.UTF-8, kept where they still mean something. bash cannot run in a locale that is not installed, so under the marker it runs in C, and C.UTF-8 beside a GB18030 LANG is a pair no person has. So:

  • as the person's own locale (LC_ALL), they stay, in the new ..._under_c_or_c_utf_8 tests: three names, refused as the bytes say, iconv never asked;
  • as bash's own locale, they stay in test_locale_reading_counts_bytes_whatever_bash_itself_reads, locale_reading lifted out of each file with a locale function: there C.UTF-8 stands in for the multibyte locale bash runs in where GB18030 is installed, and it kills F10 (the count not under LC_ALL=C), which the marker cases, run in C, no longer can;
  • the placed and the other refused cases run once, under the marker.

Runs at 8bb85cbf (Linux, bash 5.2.21, Python 3.12.3, glibc iconv; as under "Tests"):

  • the eight-file set: 800 passed, 159.89s (737 + 63 new: 8 placed and 15 refused names, each through both files; 3 names under C and C.UTF-8 through both files; the lifted test, 2 files × 2 locales; the identity test). At 18ae33b1 it gave 798 passed (one row fewer, in two files);
  • test_shape_doctor.py: 439 passed, 3 skipped, 753.91s;
  • render-cli-reference.py --check: ok; git merge-tree with main d7befb91: clean, not merged.

The full mutant table at 8bb85cbf. Each mutant is one edit in a git archive copy of 8bb85cbf, run against -k "multibyte_locale or reading_say or bytes_say or locale_reading or every_locale or unprinted_letter or link_in_it or symlink_check_names" (299 cases: the new ones and #804's and #837's directory cases) after an unmutated control (299 passed). A mutant of locale_reading is made in both files alike, so the identity test is not what kills it; a caller's is made in one file (X the launcher, I install.sh). 46 mutants: 44 killed, 2 survived (X06, I06), named below. The runner, the mutants, the run log and the kill table are in this PR's attachments (mutants/run-fix1.log, mutants/kills-fix1.txt).

Mutant Killed by
B00: 2d1ff8a1's shell files, before fix round 1 an-iconv-that-prints-nothing, both files (no round trip there)
B01: the stacked base's shell files (1ba81b07) 21 cases: every placed case but café, the identity test, the lifted test, the U+202E reading row
F01: locale failing ignored a-locale-that-fails
F02 to F06: each answer the first case stops at dropped (empty, UTF-8, utf8, ANSI_X3.4-1968, *ASCII*) iconv-unasked-under-<that answer> (the marking iconv runs); F03 and F05 also by ..._under_c_or_c_utf_8 under C.UTF-8 and C
F07: the first iconv's || return 1 dropped an-iconv-that-reads-then-fails (8bb85cbf)
F08, F09, F15: read as UTF-8; into UTF-16; the bytes printed for the reading every placed case but café
F10: the count not under LC_ALL=C the lifted test under bash's C.UTF-8, both files
F11, F12: -le for -lt; no count a-single-byte-charset; the lifted test
F13, F14: the continuing bytes from 0x81, or to 0xBE shift_jis-0xc2-0x81-0x58, shift_jis-0xc2-0x81-0x51; the lifted test
R1: printf '%s' "$reading" → printf '%s' x gb18030-reads-u+009b, gb18030-reads-u+202e; the lifted test
R2: LC_ALL=C locale charmap every placed case but café, and gb18030-reads-u+202e (refused, but as a control character)
G01: no round trip an-iconv-that-prints-nothing
G02, G03: the round trip back into UTF-8; != for = every placed case but café, gb18030-reads-u+202e and the lifted test (G03 also an-iconv-that-prints-nothing)
X01, I01, X10, I10: no C1 condition; && for || the GB18030 and Shift_JIS placed cases
X02, I02: no misdrawn condition the GBK and Shift_JIS U+2028 placed cases
X03, X04, I03, I04: the condition's C1 range from 0x81, or to 0x9E gb18030-0xc2-0x80, gb18030-0xc2-0x9f
X05, I05: no fall back to the bytes 122 cases each, #804's and #837's among them
X07, I07: no [[:cntrl:]] of the bytes #837's a-tab-then-u+202e, #804's carriage-return link case
X08, X09, I08, I09: the C1 bytes, or misdrawn_characters, asked of the bytes, not the reading the C1, or the misdrawn, placed cases
X11, I11: seen never set NOT_CONTROLS (placed no more under set -u)
X12, I12 (rv858's R3, and its install.sh twin): every value read again no-byte-refused-never-read-again (the marking iconv runs)
X06, I06: [[:cntrl:]] asked of the reading, not the bytes. Survived. Equivalent since the round trip, in every charset a locale uses. A reading is used only where it writes back into the value byte for byte, and in an ASCII-compatible charset a C0 byte or DEL reads as itself, so the reading holds exactly the value's controls. A 7-bit stateful charset (ISO-2022) cannot read the bytes that start a reading. The two differ only under a charmap that is not ASCII-compatible, such as UTF-16LE, which no locale has. At 2d1ff8a1, without the round trip, a-newline-at-the-end killed them: $(…) dropped the newline from the reading. Now the round trip refuses that reading, and the bytes decide. The check stays on the bytes as typed, so it does not lean on the round trip

Decisions

Line 1 (a GBK or GB18030 name refused as a control character): taken, as the Direction says, and for both arms alike. Where the bytes refuse a directory, the person's locale is asked how it reads them, and a multibyte charset's reading has the last word.

Line 2 (a lone 0x9B placed): left, as the documented limit.

  • A terminal that reads UTF-8 does not act on a lone 0x9B: it is not UTF-8, and a decoder draws it as a character it cannot read. The C1 controls such a terminal acts on are U+0080 to U+009F, two bytes each, which the C1 arm refuses. Only a terminal set to an 8-bit charset with C1 controls (ISO 8859) acts on the one byte. (Not measured on a terminal: none here.)
  • A rule needs to know the byte is not inside a UTF-8 sequence (316 code points below U+3000 hold a 0x9B byte, U+011B among them), and bash has no validity builtin; iconv -f UTF-8 -t UTF-8 may be missing. Under C, where the charset is unknown, refusing a lone byte would refuse GBK and Shift_JIS names (every JIS level-1 kanji starts with a byte from 0x88 to 0x98). And 0x9B is not alone: 0x85 (NEL) ends a line and 0x9D (OSC) starts a terminal command, so the rule would be the whole lone C1 range.
  • A candidate, left: under an ISO 8859 charmap, the locale an 8-bit terminal goes with, the charset is known: printf 'a\x9bb' | iconv -f ISO-8859-1 -t UTF-8 gives 61 c2 9b 62. But the name holds no byte the check refuses, so nothing reads it again, and under an ISO-8859-1 locale a<0x9B>b is placed (rv858's measurement). Reading every non-ASCII name again under a known 8-bit charset would close it, at the cost of a locale call for every such name.
  • The person set the variable themselves, and only a terminal in 8-bit mode acts on the byte. The launcher's comment now says so, names the ISO 8859 case as a candidate, and so does this body.

What changed

Measured by a probe (probes/probe.sh in this PR's attachments): a locale first on $PATH that answers charmap as shown, this machine's iconv, a gh stub that exits 97, GH_TOKEN=bogus, an empty GH_CONFIG_DIR, a curl stub that logs and exits 22, bash's own LC_ALL=C.UTF-8, <H> for the probe's directory, and OPENREPOSHAPE_BIN_DIR=<H>/a + the bytes + b:

bytes             locale says   stacked base 1ba81b07               this change
0xC2 0x9B         GB18030       both refuse: control character      the command: exit 0, placed; install.sh: makes its request
0xC2 0x9B         UTF-8         both refuse: control character      both refuse: control character (the same words)
0xC2 0x9B         fails         both refuse: control character      both refuse: control character
0xC2 0x9B         not there     both refuse: control character      both refuse: control character
0xC2 0xAD         GBK           both refuse: (U+00AD)               the command: exit 0, placed; install.sh: makes its request
0xC2 0xAD         UTF-8         both refuse: (U+00AD)               both refuse: (U+00AD)
0xE2 0x80 0xA8    SHIFT_JIS     both refuse: (U+2028)               the command: exit 0, placed; install.sh: makes its request
0xE2 0x80 0xA8    UTF-8         both refuse: (U+2028)               both refuse: (U+2028)
0xC2 0x9B, 0xFF   GB18030       both refuse: control character      both refuse: control character (GB18030 cannot read 0xFF)
0xEF 0xBB 0xBF    ISO-8859-1    both refuse: (U+FEFF)               both refuse: (U+FEFF) (single-byte: the bytes decide)
0xC2 0x9B, LF     GB18030       both refuse: control character      both refuse: control character (the newline, in the bytes)

Where it is placed, the command's lines name the directory as typed, as for any other name:

openRepoShape: installed at <H>/a<0xC2 0x9B>b/openRepoShape
openRepoShape: 1 of 1 placed in <H>/a<0xC2 0x9B>b

and the refusals, before and after, are the words #804 and #837 gave them, for example:

REFUSED: --install would place this command in a directory whose name holds a control character, such as a newline, a carriage return, a tab or an escape, and no line about $PATH can print such a name as it is: a terminal acts on it. Nothing was placed. Set $OPENREPOSHAPE_BIN_DIR to a directory without one, then run this again.
REFUSED: the directory to install into holds a character a terminal draws as something other than itself (U+00AD): one that is invisible, is drawn as another character, ends a line or reorders the text around it, so no line about $PATH can print the name as it reads. Nothing was installed. Set $OPENREPOSHAPE_BIN_DIR to a directory without one, then run this line again.

Tests

No GB18030, GBK or Shift_JIS locale is installed here or on CI, and none can be built. So put_a_locale puts a locale first on $PATH that answers locale charmap as glibc's does, from the first of $LC_ALL, $LC_CTYPE and $LANG that is set (fix round 1): a marker locale xx_XX.<charmap> gives that charmap, C.UTF-8 gives UTF-8, anything else ANSI_X3.4-1968. It can also fail after answering, or exit 127 as a missing program does. Beside it is this machine's iconv, or one that exits 127, one that prints nothing and exits 0, one that runs this machine's and then exits 1, or one that leaves a mark if it runs. The person's locale is LANG=xx_XX.<charmap> with LC_ALL and LC_CTYPE empty; bash says nothing of a LANG that is not installed, and runs in C.

File Test What it pins
tests/test_openreposhape_command.py test_a_bin_dir_a_multibyte_locale_reads_as_text_is_placed (over PLACED_AS_THE_LOCALE_READS) the Direction's case: 0xC2 0x80, 0xC2 0x9B and 0xC2 0x9F under GB18030 (the first, the issue's and the last of the 32 hanzi); 0xC2 0xAD under GBK; U+2028's bytes under Shift_JIS; two Shift_JIS straddles (0xC2 0x81 0x58, 0xC2 0x81 0x51) whose reading holds a 0x80 and a 0xBF, the edges of the bytes that continue a character; and café, which the bytes pass, under GB18030 with an iconv that leaves a mark if it runs: exit 0, the command placed, no mark
tests/test_install_sh.py test_the_installer_places_a_directory_a_multibyte_locale_reads_as_text the same eight through install.sh, which reads them as the command does, and the command it fetched places itself there with the release recorded
tests/test_openreposhape_command.py test_a_bin_dir_is_refused_as_its_bytes_or_its_reading_say (over REFUSED_AS_THE_BYTES_SAY) the reading checked itself: a hanzi of the C1 bytes beside U+009B (0x81 0x30 0x83 0x37) or U+202E (0x81 0x36 0xA7 0x31) in GB18030's own form, refused as a control character or naming U+202E; and the bytes deciding: a locale that fails, or is not there; no iconv; an iconv that prints nothing, or prints the whole reading and then exits 1; a byte GB18030 cannot read; Latin-1 and U+FEFF's bytes (single-byte); a newline at the end; and a marking iconv never asked where locale answers nothing, UTF-8, utf8, ANSI_X3.4-1968 or US-ASCII: exit 2, the words, nothing placed, never echoed, no mark
tests/test_install_sh.py test_the_installer_refuses_as_the_bytes_or_the_reading_say the same fifteen through install.sh, before any request
tests/test_openreposhape_command.py test_a_bin_dir_is_refused_as_its_bytes_say_under_c_or_c_utf_8 (over REFUSED_UNDER_C_OR_UTF_8) the person's own locale LC_ALL=C or C.UTF-8 (ANSI_X3.4-1968 or UTF-8 from the stub): 0xC2 0x9B, 0xC2 0xAD and U+2028's bytes refused in #804's and #837's words, iconv never asked
tests/test_install_sh.py test_the_installer_refuses_as_the_bytes_say_under_c_or_c_utf_8 the same six through install.sh, before any request
tests/test_openreposhape_command.py test_locale_reading_counts_bytes_whatever_bash_itself_reads[openRepoShape, install.sh] locale_reading lifted out of each file, with a locale function, under bash's own C and C.UTF-8 (standing in for the multibyte locale bash itself runs in where GB18030 is installed): each placed name's reading is Python's codec's, and Latin-1's U+FEFF bytes and a byte GB18030 cannot read give none
tests/test_openreposhape_command.py test_install_sh_carries_the_launchers_locale_reading_as_it_is install.sh's locale_reading is the launcher's, byte for byte (through shell_function, which misdrawn_check now uses too)

At 2d1ff8a1, before fix round 1: the new cases over the stacked base's two shell files (1ba81b07's install.sh and openRepoShape, these tests): 29 failed (the 28 placed cases and the identity test), and the 56 refused cases pass, as they should: they hold that nothing else changed.

Mutants at 2d1ff8a1, before fix round 1. Each one is a single edit in a git archive copy of 2d1ff8a1, run against the new cases with #804's and #837's directory cases (-k "multibyte_locale or nothing_reads or locale_reading or every_locale or unprinted_letter or link_in_it or symlink_check_names": 321 cases) after an unmutated control (321 passed). A mutant of locale_reading is made in both files alike, so the identity test is not what kills it; a mutant of a caller is made in one file (X the launcher, I install.sh). 37 mutants, 37 killed, no survivor. The runner, the mutants, the run log and the kill table are in this PR's attachments, under mutants/.

Mutant Killed by
locale failing ignored (F01) a-locale-that-fails, both files
each answer the first case stops at dropped: empty, UTF-8, utf8, ANSI_X3.4-1968, *ASCII* (F02 to F06) iconv-unasked-under-<that answer>: the iconv that would read anything is asked, and the name is placed
iconv failing ignored (F07) no-iconv, a-byte-gb18030-cannot-read
read as UTF-8, not the charset (F08); read into UTF-16 (F09); the bytes printed for the reading (F15) every placed case
the count not under LC_ALL=C (F10) the placed cases under bash's C.UTF-8
-le for -lt (F11); no count (F12) a-single-byte-charset (Latin-1 would let U+FEFF's bytes through)
the continuing bytes from 0x81 (F13), or to 0xBE (F14) shift_jis-0xc2-0x81-0x58 (a 0x80 in the reading), shift_jis-0xc2-0x81-0x51 (a 0xBF)
no C1 condition, or && for `
no misdrawn condition (X02, I02) the GBK and Shift_JIS U+2028 placed cases
the condition's C1 range from 0x81, or to 0x9E (X03, X04, I03, I04) gb18030-0xc2-0x80, gb18030-0xc2-0x9f
no fall back to the bytes when nothing reads them (X05, I05) 129 cases each, #804's and #837's among them (the real locale answers UTF-8 under C.UTF-8)
[[:cntrl:]] asked of the reading (X06, I06) a-newline-at-the-end
no [[:cntrl:]] of the bytes (X07, I07) a-newline-at-the-end, #837's a-tab-then-u+202e, #804's carriage-return link case
the C1 bytes, or misdrawn_characters, asked of the bytes and not the reading (X08, X09, I08, I09) the C1, or the misdrawn, placed cases
seen never set when nothing is read (X11, I11) NOT_CONTROLS, placed under set -u no more
the stacked base's two shell files (B00) the 28 placed cases and the identity test

Runs (Linux, bash 5.2.21, Python 3.12.3, glibc iconv; env -u GH_TOKEN -u GITHUB_TOKEN GH_CONFIG_DIR=<empty> /usr/bin/python3 -m pytest -q -p no:cacheprovider, one basetemp, df -i checked first):

Set Stacked base 1ba81b07 Head 2d1ff8a1 (restacked on main ca47b71d)
test_openreposhape_command.py, test_install_sh.py, test_install_sh_filter_repo.py, test_openreposhape_package.py, test_repo_hygiene.py, test_cli_reference.py, test_agents_md_codes.py, test_readme_windows_quoting_sentence.py 737 passed, 190.49s 822 passed, 157.14s (737 + 85 new)
test_shape_doctor.py (it runs the launcher's --doctor) 439 passed, 3 skipped, 629.75s 439 passed, 3 skipped, 665.31s

render-cli-reference.py --check: ok. shellcheck 0.9.0: the same 8 findings before and after, none in the new lines.

Not changed

Closes #858

🤖 Generated with Claude Code

Summary by Sourcery

Make installation directory validation aware of how the user's multibyte locale interprets refused byte sequences while preserving safe fallback checks.

Bug Fixes:

  • Allow directories containing byte sequences that a configured multibyte locale reads as ordinary text to pass installation safety checks, while retaining the existing byte-based refusal behavior when locale-based validation is unavailable or inconclusive.

Enhancements:

  • Apply consistent locale-aware directory validation in both the installer and launcher, including round-trip validation and checks for controls or visually misleading characters in the locale's interpretation.
  • Expand coverage for GB18030, GBK, Shift_JIS, UTF-8, ASCII, single-byte locales, locale/iconv failures, and installer/launcher behavior parity.

Tests:

  • Add comprehensive tests for locale-aware directory placement and refusal behavior, including stubbed locale and iconv scenarios.

Brett Heap and others added 16 commits October 16, 2026 14:07
…IR` is respelled with `cygpath -u` or refused before install.sh's fetch, and the PATH note single-quotes a directory bash would still read, and `--install` refuses one holding `:` or a newline, so install.sh names `/d/tools/bin` as the command does and the `export PATH` line does what it says (#667)

#649's two leftovers, both from #656's review.

install.sh read `$OPENREPOSHAPE_BIN_DIR` as typed, so with `D:\tools\bin`
its `preflight:` and `done:` lines (and `install:` for an old release) said
`D:\tools\bin/openRepoShape` while the command's own lines said
`/d/tools/bin/openRepoShape`, and where no `cygpath` answers the command's
`--install` refused (exit 2) only after install.sh had fetched it. install.sh
now carries a copy of the command's rule where it reads the variable, before
the symlink check and the fetch: a value that starts `[A-Za-z]:/` or
`[A-Za-z]:\` is spelled with `cygpath -u --`, and is refused with exit 2 where
no `cygpath` answers, fails, or answers a drive spelling again. The refusal
does not echo the value.

The command's PATH note printed `export PATH="$dir:$PATH"` with the directory
between double quotes as it was, so `a"b$HOME c` printed a line `eval` could
not parse, and a `$`, a backquote or a `\` in a name was read, not kept.
`export_path_line` keeps today's line byte for byte for a directory that
double quotes hold (a space and an apostrophe included) and, for one holding
a `"`, `$`, backquote, `\` or `!` (bash reads that at a prompt as history),
prints it between single quotes with `'\''` for an apostrophe. `--install`
also refuses a directory holding a `:` or a newline, which no `$PATH` line can
name, before anything is placed; the newline's refusal does not echo it.

Tests feed one value through install.sh and through the command, in each
spelling and each state of `cygpath`, and pin every `install:`, `preflight:`
and `done:` line; the PATH note is checked as printed, by `eval` and by
`bash -i`.

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…holding a control character before they name it, and install.sh refuses a `:` before its fetch, so a carriage return or an escape is no longer echoed as typed (#760)

#667's two checklist candidates, with rv667's NIT 2.

`--install` refused a newline in `$OPENREPOSHAPE_BIN_DIR` and nothing else
that is a control character: a carriage return, an escape, a tab or DEL
placed the command and was printed as typed (`/tmp/x^MFORGED is not on
$PATH`; an escape sequence reaches the terminal the same way). The newline
arm is now `*[[:cntrl:]]*`, first in `install_commands`, ahead of the
Windows spelling and the colon, and its refusal still says nothing of the
value. It is `[[:cntrl:]]` and not `[![:print:]]`, so a directory with a
letter the locale does not print (`café` under LC_ALL=C) is placed. Which
C1 controls match is the locale's, measured with bash 5.2.21: C.UTF-8
matches U+0080 to U+009F (and U+2028, U+2029), C and POSIX match none, and
the ASCII controls and DEL match in all three.

install.sh learnt of a `:` or a newline only after its fetch: it fetched the
launcher, ran `--install`, and said only that `--install` refused (exit 2); a
release older than #667 placed the command under the directory. It now
carries both arms of the command's `case`, copied as #667 copied the
drive-letter one: the control-character arm first, before any line names the
directory, and the colon arm after the Windows respelling, both before the
symlink check and any request. Where the control-character refusal says
nothing of the value, the colon refusal names the directory, as the
command's does.

Tests feed each value through install.sh and through the command (a colon,
two directories, a drive-relative name, a newline, a carriage return, an
escape, a tab, DEL, a drive spelling with a newline or an escape, a colon
with an escape): exit 2, no request, nothing placed, neither stream holds a
forged tail. A link in a directory holding a carriage return, a drive
spelling, a colon with an escape, a letter under LC_ALL=C and LC_ALL=C.UTF-8,
a C1 control under a UTF-8 locale, and the `$HOME` default pin the order and
the width of the arms.

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…`LC_ALL=C` and names a C1 control's two bytes, so U+0080 to U+009F is refused in every locale and whatever else the value holds, where a UTF-8 locale let one through beside a byte that is not UTF-8 (#760)

Lane 1's review of #804 (finding 1): `*[[:cntrl:]]*` matches a C1 control only
in a value that is valid UTF-8. One byte that is not (a lone 0xE9) makes bash
match byte by byte, and then U+009B's two bytes (0xC2 0x9B) are not controls,
as under `C`. Measured on 7569ffa with bash 5.2.21: under `LC_ALL=C.UTF-8`
and `$OPENREPOSHAPE_BIN_DIR` set to `<dir>/caf`, 0xE9, U+009B, `31mFORGED/bin`,
`openRepoShape --install` exited 0, placed the command and printed the value
raw in four lines; install.sh, given the same value after a `:`, exited 2 with
the colon refusal naming it, the C1 control in the line.

Both files now run the check as `(LC_ALL=C; case ... *[[:cntrl:]]* |
*$'\302'[$'\200'-$'\237']* ...)`: the ASCII controls and DEL byte by byte, and
the two bytes of each C1 control named, so the answer is the same in every
locale. Every value measured on 7569ffa and refused only under C.UTF-8, or
not at all, is refused with exit 2, nothing placed, nothing echoed, and
install.sh makes no request. `café` and U+00A0 are still placed. U+2028 and
U+2029, which only a UTF-8 locale's `[[:cntrl:]]` matched, are now placed in
every locale, and a test says so. Both comments say what the check does, and
no longer say the C1 controls are the locale's.

Tests: the C1 case, which skipped where no UTF-8 locale read U+009B as a
control, is now five values (U+009B, the first and the last of the range, a
0xE9 byte before U+009B, and a colon beside that) in each of `C` and
`C.UTF-8`, through the command and through install.sh. The placed-letter test
gains U+00A0 (the byte after the range's last), U+2028 and U+2029. 18 of the
36 fail on 7569ffa's code and the other 18 are guards; 18 mutants of the two
code hunks are caught.

Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	install.sh
#	openRepoShape
#	tests/test_install_sh.py
#	tests/test_openreposhape_command.py
…holding a bidi control, a character of no width, a soft hyphen or U+2028 before any request and name it by its code point, so no line about `$PATH` shows the directory as another (#837)

#804's check refuses the C0 and C1 controls and DEL, and passed every
character a terminal draws as something other than itself, under `C` and
`C.UTF-8` alike: the command placed itself under `<dir>/a` U+202E
`forged/bin` and printed the value in four lines, and install.sh made its
request. Laid out as a terminal lays out a line (fribidi), `~/` U+202E
`.local/bin` reads `~/nib/lacol.`, `/opt/tools-1` U+200F `/2/bin` reads
`/opt/tools-12//bin`, and `~/.local` U+200B `/bin` reads `~/.local/bin`.

Both files now refuse, right after the control check, before the drive
spelling, the colon, the symlink check and any request, a directory
holding one of 29 code points, `MISDRAWN_CHARACTERS` in the launcher:
U+00AD; the twelve bidirectional controls (U+061C, U+200E, U+200F,
U+202A to U+202E, U+2066 to U+2069), as `repo_shape.spelled` escapes
them (#391); U+2028 and U+2029, which end a line for every reader that
keeps Unicode's line breaks, and which #804 left to a ruling; and U+200B,
U+FEFF and U+2060 to U+206F, of no width, the whole block
default-ignorable. U+200C and U+200D stay allowed: Persian, the Indic
scripts and emoji sequences need them. So does every space.

`misdrawn_characters` builds each one's UTF-8 from its code point with
printf's octal escapes and looks for the bytes under `LC_ALL=C`, in the
subshell #804's check uses, so one list says what is refused and what
the refusal names, and no byte is read back as a number. The refusal
says "holds a character a terminal draws as something other than itself
(U+202E): one that is invisible, ends a line or reorders the text around
it, so no line about $PATH can print the name as it reads", names every
one found, and nothing else of the value. install.sh carries the list and
the function as the launcher has them, and a test holds the two to one
text.

Tests: #804's C1 locale test in each file now runs `MISDRAWN_BIN_DIRS`
too (each code point, two at once, after a byte that is not UTF-8, beside
a colon, in a drive spelling), under `C` and `C.UTF-8`, through the
command and install.sh, with no request; the link test takes U+202E; the
lifted function names exactly its 29 over every code point that shares a
first byte with them (896, on stdin, so it runs under Git Bash);
`NOT_CONTROLS` drops U+2028 and U+2029 and gains a Persian word with
U+200C, an emoji sequence with U+200D, U+2026 and U+FF21.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion and the redirect's location are refused unechoed for a misdrawn character too, the set takes in U+FFF9 to U+FFFB, U+1D173 to U+1D17A and U+E0001, and the refusal says a character may be drawn as another (#837)

Fix round 1 on rv841's review of #841.

The set (SHOULD 1). `MISDRAWN_CHARACTERS` gains FFF9-FFFB (the
interlinear annotation characters), 1D173-1D17A (the musical beam, tie,
slur and phrase controls) and E0001 (the deprecated language tag): format
characters of no width (glibc `wcwidth` 0) that nothing needs, which the
command placed and printed in four lines. The encoder builds the
four-byte form too, with bash 3.2's arithmetic and `%b`'s `\0NNN`. Every
other format character is left out for a reason the comment now gives:
U+200C, U+200D and U+180E, which Persian, the Indic scripts, emoji and
Mongolian need; the Egyptian hieroglyph and shorthand format controls,
with which those scripts build their signs; the tags, the variation
selectors, U+034F and U+1160, drawn as part of the character before them;
the signs drawn over the digits after them, which take a cell; and every
space and the blank Hangul fillers.

The repository and the ref (SHOULD 2). #837's line named the
`[![:print:]]` checks of both files: a UTF-8 locale's U+202E, U+200B,
U+FEFF or U+00AD passes them, and the refusal after them echoed the value.
Each now asks `misdrawn_characters` first, and refuses by the code point,
saying nothing else of the value: `checked_repo`, `checked_version` (the
version named and the newest release's tag), the option check and
`latest_from_redirect`'s location in install.sh; `shape_checkout` and
`ref_fault` (`$OPENREPOSHAPE_REF` and the record) in the command. First,
because `[![:print:]]` is the locale's, so the words do not change with it.

The words (NIT 3): "one that is invisible, is drawn as another character,
ends a line or reorders the text around it", because U+00AD is drawn as a
hyphen. The order (NIT 5): a tab before U+202E is refused as a control
character, and a row holds it.

Tests: `MISDRAWN` and the locale table take the twelve new code points and
the tab row; the lifted sweep takes U+1D100 to U+1D1FF, U+E0000 to U+E007F
and a sample of the planes past U+FFFF; `NOT_CONTROLS` takes a subdivision
flag (tags U+E0020 to U+E007F); a test of each file refuses a value holding
U+202E at every new site under `C` and `C.UTF-8`, through both entry
points, with no request, nothing placed and neither stream holding it.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fuse is read again in the locale's own charset, so a GB18030, GBK or Shift_JIS name that holds no control character and nothing drawn as another is placed, and where nothing can read it the bytes decide as before (#858)

#804's control check and #837's `misdrawn_characters` look for UTF-8's
bytes under `LC_ALL=C`, and a multibyte charset that is not UTF-8 reads
some of those bytes as text: GBK, GB18030 and CP936 read `0xC2 0x80` to
`0xC2 0x9F` as 32 hanzi; GBK, Big5, EUC-JP and EUC-KR read U+00AD's two
bytes as one character; and in Shift_JIS a kanji whose second byte is
`0xC2` before a common kanji puts a C1 control's two bytes side by side
(390 in 100,000 random pairs of common kanji, measured with Python's
codecs). Such a directory, the default `$HOME/.local/bin` under a user
name among them, was refused as holding a control character or a
character drawn as another, in a locale where it holds neither.

Both files now carry `locale_reading` (install.sh as a copy, held to one
text by a test). Where the bytes refuse a directory, it asks `locale
charmap` under the person's own locale, and where the answer is neither
empty, UTF-8 nor ASCII, `iconv` reads the whole value from that charset
into UTF-8; the reading is used only where it has fewer characters than
the value has bytes, so a single-byte charset keeps the byte rule, as the
issue's Direction says. The C1 bytes and `misdrawn_characters` are then
asked of the reading, and `[[:cntrl:]]` of the bytes as typed. Only a
value the bytes refuse is read again, so no name placed before is
refused; with no `locale` or `iconv`, a charset either does not know, or
a value the charset cannot read, the bytes decide, in #804's and #837's
words.

A lone `0x9B` stays placed, the limit #804 states: a terminal that reads
UTF-8 draws it as a character it cannot read, only one set to an 8-bit
charset acts on it, and a rule needs a UTF-8 validity step bash does not
have. The launcher's comment says so.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…above)

# Conflicts:
#	install.sh
#	openRepoShape
#	tests/test_install_sh.py
#	tests/test_openreposhape_command.py
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR updates both the launcher and installer to reinterpret only previously rejected path bytes through the user’s supported multibyte locale, allowing names that locale reads as text while preserving byte-based safety checks and fallback behavior; extensive tests cover locale/iconv failure modes, charset-specific cases, and implementation parity.

Sequence diagram for locale-aware directory installation

sequenceDiagram
    participant User
    participant Launcher as openRepoShape
    participant Locale
    participant Iconv
    participant Filesystem

    User->>Launcher: --install with bin_dir
    Launcher->>Launcher: misdrawn_characters(bin_dir)
    alt bytes are refused
        Launcher->>Locale: locale charmap
        Locale-->>Launcher: charmap
        Launcher->>Iconv: iconv -f charmap -t UTF-8
        Iconv-->>Launcher: locale reading
        Launcher->>Launcher: Check byte controls and misdrawn_characters(reading)
    else no refused bytes
        Launcher->>Launcher: Check bin_dir bytes
    end
    alt safety check refuses
        Launcher-->>User: Existing refusal message
    else safe directory
        Launcher->>Filesystem: Place command in bin_dir
        Filesystem-->>User: Installation result
    end
Loading

Flow diagram for locale-aware bin directory validation

flowchart TD
    A[bin_dir as typed] --> B{C1 byte pair or misdrawn character?}
    B -->|No| C[Use bytes as seen]
    B -->|Yes| D[locale_reading]
    D --> E{locale and iconv succeed?
Multibyte reading shortens value?}
    E -->|No| C
    E -->|Yes| F[Use locale reading as seen]
    C --> G[Check control bytes and misdrawn characters]
    F --> G
    G --> H{Refused?}
    H -->|Yes| I[Print existing refusal]
    H -->|No| J[Install or place command]
Loading

File-Level Changes

Change Details Files
Add locale-aware fallback decoding for directory safety checks in both installation paths.
  • Introduce matching locale_reading functions that use locale charmap and iconv only for byte patterns already rejected by the existing checks.
  • Accept a path when a supported multibyte locale decodes the rejected bytes as ordinary text, while retaining byte-based checks for controls and falling back on all locale/iconv failures or unsupported charsets.
  • Apply control and misdrawn-character validation to the locale-decoded value where available, preserving existing refusal messages and behavior otherwise.
install.sh
openRepoShape
Expand regression coverage for multibyte locale behavior and launcher/installer parity.
  • Add stubbed locale and iconv test infrastructure covering GB18030, GBK, Shift_JIS, UTF-8, ASCII, single-byte, missing, failing, and undecodable cases.
  • Verify accepted paths are installed by both the command and installer, while refused paths retain the prior diagnostics, avoid network activity, and are not echoed or created.
  • Assert the copied locale_reading implementation remains byte-for-byte identical between install.sh and openRepoShape.
tests/test_openreposhape_command.py
tests/test_install_sh.py

Assessment against linked issues

Issue Objective Addressed Explanation
#858 Avoid refusing a directory name as a control character when a GBK, GB18030, or other multibyte locale reads the matching UTF-8 byte sequence as text, or clearly document that limitation. ✅
#858 Prevent a lone C1 byte such as 0x9B from being placed, or document it as a known limit in both the refusal wording and the issue. ❌ The PR leaves lone C1 bytes placeable and documents that limit in a code comment and the PR body, but does not add it to the refusal wording or update the issue as requested.

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-2, 2026-10-18T02:55:11Z) Independent review of #867 (#858) at 2d1ff8a1, for lane openRepoShape-2: READY

The change is sound as written. Only a value the bytes already refuse is read again. Every fallback I probed refuses in #804's or #837's words. Where the locale's reading still holds a C1 control or a misdrawn character, it is refused and named, in install.sh before any request.

The cost is a non-UTF-8 multibyte locale in a terminal that reads UTF-8. That is what the issue's Direction allows ("refuse nothing the locale reads as a letter"), and the PR body states it.

The two SHOULDs below are test gaps. The code is right today, but two plausible regressions would ship green.

SHOULD 1: no test pins that the locale's reading is itself checked.

  • Where: tests/test_openreposhape_command.py:1010 (REFUSED_AS_THE_BYTES_SAY), covering openRepoShape:578 / install.sh:421, openRepoShape:627-629 and :638.
  • What is wrong: every refused case is a fallback. No case has a multibyte charset read the name while the reading still holds U+0080 to U+009F or one of the 41.
  • The mutant: printf '%s' "$reading" → printf '%s' x in both files. It survives all 321 selected cases.
  • What it would place: the head refuses both of these, and the mutant places both (exit 0):
    • a<C2 9B><81 30 83 37>b under a GB18030 locale: U+009B in GB18030's four-byte form. The head refuses it as a control character.
    • a<C2 9B><81 36 A7 31>b: U+202E. The head refuses it naming (U+202E), and install.sh refuses it before any request.
  • Fix: add these two rows through both files, under C and C.UTF-8:
    • ("GB18030", None, A_HANZI_OF_C1_BYTES + "\udc810\udc837", HOLDS_A_CONTROL_CHARACTER);
    • ("GB18030", None, A_HANZI_OF_C1_BYTES + "\udc816\udca71", bin_dir_drawn_as_another("U+202E")).

SHOULD 2: no test pins that locale charmap is asked under the person's own locale.

NIT 3: no test pins "only a value the bytes refuse is read again".

  • Where: openRepoShape:623-626, install.sh:489-492.
  • The mutant: the launcher's trigger replaced by if :; then. It survives all 321 cases.
  • Fix: add a case with no trigger bytes under a GB18030 locale, with an iconv that leaves a mark if it runs. Expect the name placed and the mark absent. café, or U+202E in GB18030's own form, would do.

NIT 4: a reading shorter than the value is taken as it is.

  • Where: openRepoShape:576-578, install.sh:419-421.
  • What is wrong: the count letters < bytes is the only check on the reading. An iconv that exits 0 and prints nothing passes it.
  • Repro: under a GB18030 locale, such an iconv lets a<C2 9B>b and a<U+202E bytes>b be placed.
  • Why it is a NIT: glibc's iconv exits 1 on bytes it cannot read (measured), so only a broken iconv reaches this.
  • Fix: require the reading to round-trip: [ "$(printf '%s' "$reading" | iconv -f UTF-8 -t "$charmap" 2>/dev/null)" = "$1" ]. At least, refuse an empty reading.

NIT 5: the code comments do not state the trade the PR body states.

  • Where: openRepoShape:611-621, install.sh:481-487, locale_reading's comment at openRepoShape:540-564, and export_path_line's comment at openRepoShape:405-418.

  • What the trade is: under a GB18030 locale, two names are placed:

    • a<C2 9B>2Jb;
    • a<C2 9D>0;TITLE<C2 9C>b.

    Four stdout lines then carry CSI 2J (erase the screen) or OSC 0 (set the title) as UTF-8. A terminal that reads UTF-8 acts on them.

  • What the comments say: locale_reading's comment names only the Latin-1 half of the mismatch. export_path_line's comment still says install_commands refuses a control character before it is asked. That is now true only of the locale's reading.

  • Fix: add one sentence in each place, as the body's "What it costs" says it.

NIT 6: the PR body cites a brief nobody can see.

NIT 7: line 2's reason is true but incomplete.

Checked and true

  • The restack:

    • git diff --name-only ca47b71d 2d1ff8a1 names the four files only;
    • its -U0 changed lines equal git diff 1ba81b07 e3baca96 (333 lines, cmp silent);
    • 1ba81b07's four files equal ca47b71d's and f2483cf1's (empty diffs);
    • merge-tree with main 5cbec222 is clean.
  • The two copies: install.sh's locale_reading is the launcher's, byte for byte, and the identity test fails on the base.

  • On the base: the new tests over 1ba81b07's shell files give 29 failed (the 28 placed cases and the identity test) and 56 passed, as the body says.

  • Every fallback refuses in today's words:

    • a locale that fails;
    • a locale that is missing;
    • the real glibc locale under an uninstalled zh_CN.GB18030 (it answers ANSI_X3.4-1968);
    • no iconv;
    • an iconv that prints part of a reading and exits 1;
    • 0xFF under GB18030;
    • ISO-8859-1;
    • a lying UTF-16LE charmap;
    • a newline at the end.

    A lowercase gb18030 answer is read like GB18030.

  • Checks that stay on the bytes as typed: [[:cntrl:]], the colon, the Windows spelling and the symlink check. No trailing byte in GBK, GB18030, Big5 or Shift_JIS is below 0x30, so a name the locale reads as text loses nothing to [[:cntrl:]].

  • The documented limits hold as stated:

    • GB18030's four-byte U+202E on its own is placed;
    • Shift_JIS half-width katakana alone (C2 AD) is refused. The count runs over the whole value, so the same pair is placed when a kanji is elsewhere in the path. It fails closed either way;
    • a lone 0x9B under UTF-8 is placed, as before.
  • The comment's codec claims check out with Python's codecs and glibc's iconv:

    • U+00AD's two bytes are one character in each charset the comment names: U+9885 in GBK and GB18030, U+7C2B in Big5, U+8DB3 in EUC-JP, U+C9EF in EUC-KR;
    • GB18030 reads all 32 pairs as one hanzi each;
    • the CP932 example 93 C2 90 ED is two kanji.
  • install.sh:

  • stdin: neither locale nor iconv reads the script's stdin, so curl … | bash is unaffected.

  • Code quality:

    • shellcheck 0.9.0 gives the same 8 findings before and after, none in the new lines;
    • the new tests have no composite assert;
    • their decorator stacks repeat the pattern of the existing paired tests.
  • Docs: no sentence in README, the handbook, docs/cli.md, AGENTS.md or templates/** states these bytes.

  • Tests:

    • the four Verify files, the trio and test_readme_windows_quoting_sentence.py: 822 passed at the head and 822 passed on its merge with main 5cbec222;
    • test_shape_doctor.py: 439 passed (3 skipped) at the head, and 440 passed on the merge.
  • Mutants: nine of the writer's mutants, re-run, match their table exactly, failed counts included.

Could not verify

  • bash 3.2 (none here). By reading, the new code uses nothing newer than bash 3.x.
  • Git Bash: whether locale.exe and iconv.exe ship there and answer locale charmap. The new cases skip on Windows. Under the default UTF-8 charmap the function stops at its first case, and a missing program fails closed.
  • Charset names that macOS's locale and iconv use (eucJP, SJIS), and musl. By my reading, musl only answers UTF-8 or ASCII, which stop at the first case. A name iconv does not know fails closed.
  • A real GBK, GB18030 or Shift_JIS locale, and a real terminal's handling of the payloads in NIT 5.
  • SonarCloud: its MCP server would not connect.

Lane: openreposhape-2 (openRepoShape-2)

…_BIN_DIR` must write back into the value byte for byte, and tests hold that the reading is itself checked, that `locale charmap` is asked under the person's own locale and that a name the bytes pass is never read again (#858)

Fix round 1 on rv858's review of #867 (READY, 2 SHOULDs, 5 NITs).

SHOULD 1: no test had a multibyte charset read a name whose reading still
holds a C1 control or a character drawn as another, so `locale_reading`
printing anything at all (rv858's R1) shipped green. Two rows now do,
through both files: a GB18030 hanzi of the C1 bytes beside U+009B in
GB18030's own four-byte form (`0x81 0x30 0x83 0x37`), refused as a control
character, and beside U+202E (`0x81 0x36 0xA7 0x31`), refused naming
U+202E, install.sh before any request.

SHOULD 2: the stub `locale` answered one charmap whatever it was asked
under, so `LC_ALL=C locale charmap` (R2) shipped green. `put_a_locale`'s
stub now answers as glibc's does, from the first of `$LC_ALL`,
`$LC_CTYPE` and `$LANG` set: a marker `xx_XX.<charmap>` gives that
charmap, `C.UTF-8` gives UTF-8, anything else `ANSI_X3.4-1968`; the cases
run with `LANG` the marker and `LC_ALL` and `LC_CTYPE` empty, so bash runs
in `C` (it says nothing of a `LANG` that is not installed). `C` and
`C.UTF-8` stay where they mean something: as the person's own locale, in
a test of three names refused there with `iconv` never asked; and as
bash's own locale, in a test of `locale_reading` lifted out of each file
with a `locale` function, `C.UTF-8` standing in for a multibyte locale of
bash's own, where the readings are Python's codecs' and the count stays
in bytes.

NIT 3: a name the bytes pass (`café`) under a GB18030 locale, with an
`iconv` that leaves a mark if it runs: placed, no mark (R3, the trigger
replaced by `if :; then`).

NIT 4: an `iconv` that exits 0 and prints nothing passed the count. The
reading must now write back into the charset as the value, byte for
byte (`iconv -f UTF-8 -t <charmap>`), in both copies; a broken `iconv`
falls back to today's refusal, and a test holds it. The rows where
`iconv` must not be asked use the marking `iconv` instead of one that
reads anything, which the round trip would refuse anyway.

NIT 5: `locale_reading`'s comment, both callers' and `export_path_line`'s
say the trade: a multibyte locale that is not UTF-8, in a terminal that
reads UTF-8, is handed C1 bytes it reads as letters. NIT 7: the
launcher's comment names the ISO 8859 case (the charset is known and
`a`, `0x9B`, `b` is placed) as a candidate, left. NIT 6 is the PR body's.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…BIN_DIR` and then fails gives no reading, so the first `iconv`'s exit is pinned beside the round trip (#858)

Fix round 1 on rv858's review of #867, its last step. With the round trip
in `locale_reading`, dropping the first `iconv`'s `|| return 1` (F07)
survived the mutant table: a failing `iconv` that printed part of a
reading, or nothing, is refused by the round trip anyway. One that prints
the whole reading and then exits 1 is not: the round trip passes. A row
of `REFUSED_AS_THE_BYTES_SAY` now has such an `iconv` (`READS_THEN_FAILS`,
this machine's run in full, then `exit 1`) read a hanzi of the C1 bytes
under GB18030, through both files: refused as a control character, as
the bytes say, and install.sh before any request.

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-2, 2026-10-18T04:51:55Z) Re-verification of #867 (#858) at 8bb85cbf, for lane openRepoShape-2: READY

The seven findings of the first review

Finding State at 8bb85cbf Measured
SHOULD 1: the locale's reading is never itself checked Fixed. Two new rows: GB18030's own U+009B and U+202E, each beside a hanzi made of C1 bytes, through both files. R1 (printf '%s' x for the reading) killed by 8 cases.
SHOULD 2: locale charmap not pinned to the person's own locale Fixed. The stub answers from LC_ALL, then LC_CTYPE, then LANG, the way glibc does. The placed and refused cases set LANG=xx_XX.<charmap> and leave LC_ALL and LC_CTYPE empty. R2 (LC_ALL=C locale charmap) killed by 16 cases, and R2b (LC_ALL=C.UTF-8) by 16.
NIT 3: "only a value the bytes refuse is read again" unpinned Fixed. café under GB18030, with an iconv that leaves a mark if it runs. R3 (the launcher's trigger replaced by if :) killed by 2 cases, and its install.sh twin by 1.
NIT 4: a short or empty reading accepted Fixed by the round trip in both copies, plus rows for an iconv that prints nothing and one that prints the whole reading and then fails. The writer's G01, G02 and G03 fail 2, 20 and 22 cases; F07 2; R7 (round-trip $1, not the reading) 20; R8 (compare with the reading, not $1) 22.
NIT 5: comments silent on the trade Fixed in locale_reading's comment, in both callers' comments and in export_path_line's comment. Read.
NIT 6: the PR body cited a brief Fixed. "Line 1" now quotes the issue's Direction, and no "brief" is left in the body. grep.
NIT 7: line 2's reason incomplete Fixed in the launcher's comment and the body: the ISO 8859 case is named as a candidate and left. The Goal's "on the issue" line is still to be posted on #858 when it closes. Read.

NIT 8 (new): the round trip refuses names a correct iconv reads but writes back differently. Neither the comment nor the body says so.

  • Where: openRepoShape:558 (the comment), with the check at openRepoShape:583 and install.sh:417.

  • What happens: the comment says the round trip catches "an iconv that drops or changes a byte". But glibc's own iconv writes back other bytes for some duplicate codes: 398 one- and two-byte sequences in CP932, 10 in BIG5 and 8 in BIG5-HKSCS (measured through iconv(3)). These are NEC row 13 and NEC-selected IBM extensions in CP932, and box drawing and two hanzi in Big5. GB18030, GBK, CP936, SHIFT_JIS, EUC-JP, EUC-KR, CP949, EUC-TW and EUC-CN have none.

  • Repro: with a stub locale answering CP932:

    • a<93 C2 90 ED><87 90>b (a kanji pair that straddles the C1 bytes, then the CP932 code 87 90) was placed at 2d1ff8a1 and is refused at 8bb85cbf;
    • the same with ED 40 instead of 87 90 behaves the same;
    • with FA 5C it is placed by both heads.

    Under BIG5, a<C2 AD><A2 CC>b was placed and is now refused.

  • Why it is only a NIT: this fails closed, and main refuses these names too, so nothing regresses against main. Names that Windows writes in CP932 use the codes that round-trip.

  • Fix: add one clause to the comment and to the body's "Not changed": "a name holding one of CP932's or Big5's duplicate codes is refused as the bytes say, as before".

  • Taken into the body: lane 2 adds the clause to the PR body's "Not changed". The code comment stays as reviewed, so the gate and CI at 8bb85cbf still hold.

Checked and true

Could not verify

  • bash 3.2; Git Bash's locale and iconv; macOS and musl iconv; a real GB18030, GBK or Shift_JIS locale or terminal; SonarCloud.
  • Whether Cygwin writes CP932's NEC duplicate codes in any real file name. This is reasoned from Windows' mapping tables, not measured.

Lane: openreposhape-2 (openRepoShape-2)

@sonarqubecloud

Copy link
Copy Markdown

@brettheap
brettheap marked this pull request as ready for review October 18, 2026 05:04
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 13 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 5c54af2 into main Oct 18, 2026
5 checks passed
@brettheap
brettheap deleted the fix/858-bin-dir-bytes branch October 18, 2026 05:04
brettheap pushed a commit that referenced this pull request Oct 18, 2026
… as 5c54af2) as the stacked base of #874

Lane: openreposhape-2 (openRepoShape-2)
brettheap pushed a commit that referenced this pull request Oct 18, 2026
… its tree a4b0a79 is #867's final head 0ccda48's, merged above

Lane: openreposhape-2 (openRepoShape-2)
brettheap pushed a commit that referenced this pull request Oct 18, 2026
…ter past ASCII is no control character under `C` either, as #760's byte rule decides in all seven checks that used `[![:print:]]`, and every control character is still refused (#940)

#872 checked `$OPENREPOSHAPE_REPO` and `$OPENREPOSHAPE_REF` before every
run but `--help`, with `[![:print:]]`, which under `C` and POSIX matches
every byte past ASCII: a mirror at `/srv/café/openRepoShape.git`, or a ref
`café`, was refused as "carrying a control character" by `--version`, a
scaffold run, `--install` and `--preflight`, where `C.UTF-8` took it.
`setup.sh`'s two guards and `install.sh`'s `checked_repo`,
`checked_version` and option check held the same class.

All seven now ask #760's rule, as `install_commands` holds the directory to
it: under `LC_ALL=C`, in a subshell, `[[:cntrl:]]` or U+0080 to U+009F's
two bytes, so one value gets one answer in every locale, and the launcher,
`setup.sh`, `install.sh` and the package agree. The leading `-` arms stay,
`misdrawn_characters` stays first, and no refusal's words change. A lone
`0x9B` byte that is not UTF-8 now passes in both locales, the trade #867
took for the directory, which the comment at each place states. `setup.sh`,
which has no copy of `misdrawn_characters`, also refuses U+2028 and U+2029
by their bytes, as it refused both in every locale before. The redirect's
location in `install.sh` keeps `[![:print:]]`: it is the network's, and
that class prints no lone C1 byte from it.

One table, `CONTROL_RULE` in `tests/conftest.py`, holds the seven to one
answer under `C` and `C.UTF-8`: `café`, a lone `0x9B` and a lone `0xE9`
taken; an escape, U+009B, DEL, a tab, U+0085, U+0080, U+009F, a newline at
the end and a lone `0xE9` then U+009B refused; U+202E, U+200B, U+2028 and
U+2029 refused by code point. The three test files read it, and a
repository or ref with `café` goes through every launcher run in both
locales. Every arm of the rule dropped from any place fails a test, the
seven the issue found surviving included.

Closes #940

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 20, 2026
…ter past ASCII is no control character under `C` either, as #760's byte rule decides in all seven checks that used `[![:print:]]`, and every control character is still refused (#940) (#986)

#872 checked `$OPENREPOSHAPE_REPO` and `$OPENREPOSHAPE_REF` before every
run but `--help`, with `[![:print:]]`, which under `C` and POSIX matches
every byte past ASCII: a mirror at `/srv/café/openRepoShape.git`, or a ref
`café`, was refused as "carrying a control character" by `--version`, a
scaffold run, `--install` and `--preflight`, where `C.UTF-8` took it.
`setup.sh`'s two guards and `install.sh`'s `checked_repo`,
`checked_version` and option check held the same class.

All seven now ask #760's rule, as `install_commands` holds the directory to
it: under `LC_ALL=C`, in a subshell, `[[:cntrl:]]` or U+0080 to U+009F's
two bytes, so one value gets one answer in every locale, and the launcher,
`setup.sh`, `install.sh` and the package agree. The leading `-` arms stay,
`misdrawn_characters` stays first, and no refusal's words change. A byte
that is not UTF-8, as a lone `0x9B` or an overlong form of a control
(`0xC0 0x9B`), now passes in both locales, the trade #867 took for the
directory, which the comment at each place states. `setup.sh`, which has
no copy of `misdrawn_characters`, also refuses U+2028 and U+2029 by their
bytes, as it refused both in every locale before.

The network's values get no such trade. `install.sh` still names the
redirect's location only where the locale's `[![:print:]]` finds nothing
in it, and it now takes the tag in a `…/releases/tag/<tag>` location only
when that tag is printable ASCII under `LC_ALL=C`; any other tag names no
release, in every locale, and `gh` is asked next. Without that, a tag
holding a lone `0x9B` would pass `checked_version` by the trade, and its
shape refusal would print the byte as it came from the network. A release
tag is ASCII, so this refuses no real one.

One table, `CONTROL_RULE` in `tests/conftest.py`, holds the seven to one
answer under `C` and `C.UTF-8`: `café`, a lone `0x9B`, a lone `0xE9` and an
overlong escape taken; an escape, U+009B, DEL, a tab, U+0085, U+0080,
U+009F, a newline at the end and a lone `0xE9` then U+009B refused; U+202E,
U+200B, U+2028 and U+2029 refused by code point. The three test files read
it, and a repository or ref with `café` goes through every launcher run in
both locales. A test holds the seven lines to one text, so a check without
`LC_ALL=C`, which no locale installed here answers differently, fails it,
and the redirect's test gives a tag with a lone `0x9B` or a letter past
ASCII no answer but "names no release". Every arm of the rule dropped from
any place fails a test, the seven the issue found surviving included.

Independent review READY at H0 d4bba68 (#986 6277825631), with H0's tests and SonarCloud green. Refreshed by update-branch onto main af0e60f as 2a6554c, a merge whose tree 3e049cd lane 2's gate ran on (8 files: 165 passed; 961 passed; 612 passed), with SonarCloud green there. Landed on Brett Heap's word "merge v1.2.0 PRs once reviewed and green" (2026-10-18T14:45:28Z, #473 6251255596).

Closes #940

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant