Skip to content

openRepoShape, setup.sh and install.sh: a URL, path or ref with a letter past ASCII is no control character under C either, as #760's byte rule decides in all seven checks that used [![:print:]], and every control character is still refused - #986

Merged
brettheap merged 8 commits into
mainfrom
fix/940-c-locale-byte-rule
Oct 20, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Lane: openreposhape-2 (openRepoShape-2)
Claim: #940 (comment)

Milestone v1.2.0. Brett Heap's v1.2.0 merge word, "merge v1.2.0 PRs once reviewed and green" (2026-10-18T14:45:28Z, relayed on #473 issuecomment-6251255596), lands it once an independent review is READY at the landing head and tests and SonarCloud are green. It stays a DRAFT until then. Cut at a4de0aae, with origin/main merged in at ebbd9fa4 and at 2a78ae1a (a4c39592); nothing stacked. Fix round 1 (d4bba685b) answers rv986's review of 9ab5874b (READY, with SHOULD-1 and NIT-1 to NIT-3): see Fix round 1, below.

What changed

openRepoShape, setup.sh, install.sh, tests/conftest.py and the three test files.

#872 (v1.1.1) checked $OPENREPOSHAPE_REPO and $OPENREPOSHAPE_REF before every run but --help, with [![:print:]]. That class is the locale's: under C and POSIX it matches every byte past ASCII, so a mirror at /srv/café/openRepoShape.git, or a ref café, was refused as "carrying a control character" by --version, a scaffold run, --install and --preflight, where C.UTF-8 took it and v1.1.0 ran it. setup.sh's two guards and install.sh's three checks held the same class.

All seven now ask #760's rule, the one install_commands holds $OPENREPOSHAPE_BIN_DIR to, without locale_reading: under LC_ALL=C, in a subshell, [[:cntrl:]] or U+0080 to U+009F's two bytes. Each case became the same three lines:

	if (LC_ALL=C; case "$ref" in *[[:cntrl:]]* | *$'\302'[$'\200'-$'\237']*) exit 0 ;; *) exit 1 ;; esac); then
		printf '%s' "$what carries a control character, which is not a branch, tag or commit this tool will pass to \`git checkout\`."
	fi
  • the launcher: checked_repo (with its leading - arm, over $url$REPO as before) and ref_character_fault;
  • setup.sh: the guard on $SHAPE_REPO (with its leading - arm) and the one on --shape-ref, each with one more arm, U+2028 and U+2029 by their bytes (Decisions, 3);
  • install.sh: checked_repo, checked_version and the option check in main.

misdrawn_characters stays first where it was. No refusal's words changed, and "carries a control character" is now true of what each names:

OPENREPOSHAPE_REPO is not a value this tool will put on a `git clone` command line: it starts with '-' or carries a control character. Set it to a URL or a path.
$OPENREPOSHAPE_REF carries a control character, which is not a branch, tag or commit this tool will pass to `git checkout`.
--shape-ref carries a control character, which is not a branch, tag or commit this tool will pass to `git checkout`.
$OPENREPOSHAPE_REPO carries a control character; it must be an owner/name on github.com, such as opensoft/openRepoShape.
the version carries a control character, which is not a release tag, branch or commit.
an option carries a control character. The one argument is a version, such as v1.0.1; with none, the newest release.

The comments that said the repository and the ref are held to [![:print:]] say what holds them now (openRepoShape in install_commands, above checked_repo, above ref_fault and above ref_character_fault; install.sh above checked_repo, checked_version, in latest_from_redirect and in main), each place's comment states the trade (a byte that is not UTF-8, as a lone 0x9B or an overlong form of a control), and the docstring of test_version_prints_a_path_a_utf_8_locale_prints says the path is no fault in either locale (the test now runs under both).

install.sh's latest_from_redirect takes the tag in a …/releases/tag/<tag> location only when it is printable ASCII, asked under LC_ALL=C in a subshell so every locale answers alike; any other tag names no release ("it redirects to a location that names no release"), and gh is asked next (Decisions, 4).

Measured, before (a4de0aae) and after

C.UTF-8 gave the same answers before and after in every cell; "before" names C and C.UTF-8 where they differed. "taken" means the check passed the value on: --version printed it, setup.sh's guard went on to git clone, and otherwise the next check answered as it did under C.UTF-8 (the ref's shape refusal, install.sh's owner/name refusal or "unknown option"). "ctrl" is the refusal in the words above; U+XXXX is misdrawn_characters's refusal by code point. Each value ends a repository (/srv/<x>, or acme/r<x> for install.sh), a ref (v1<x>) or an option (-x<x>); a lone byte is that byte and no UTF-8.

value launcher repo launcher ref setup.sh repo setup.sh --shape-ref install.sh repo install.sh version install.sh option
cafe ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken
lone-9B ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
lone-85 ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
lone-E9 ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
overlong-ESC (C0 9B)¹ ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
ESC ctrl ctrl ctrl ctrl ctrl ctrl ctrl
U+009B ctrl ctrl ctrl ctrl ctrl ctrl ctrl
DEL ctrl ctrl ctrl ctrl ctrl ctrl ctrl
TAB ctrl ctrl ctrl ctrl ctrl ctrl ctrl
U+0085 ctrl ctrl ctrl ctrl ctrl ctrl ctrl
U+0080 ctrl ctrl ctrl ctrl ctrl ctrl ctrl
U+009F ctrl ctrl ctrl ctrl ctrl ctrl ctrl
trailing-newline ctrl ctrl ctrl ctrl ctrl ctrl ctrl
lone-E9-then-U+009B ctrl ctrl ctrl ctrl ctrl ctrl ctrl
U+202E U+202E U+202E ctrl / taken → taken ctrl / taken → taken U+202E U+202E U+202E
U+200B U+200B U+200B ctrl / taken → taken ctrl / taken → taken U+200B U+200B U+200B
U+00A0 ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken ctrl / taken → taken
U+2028 U+2028 U+2028 ctrl ctrl U+2028 U+2028 U+2028
U+2029 U+2029 U+2029 ctrl ctrl U+2029 U+2029 U+2029
U+0378 ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
U+FFFE ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken ctrl → taken
leading - ctrl ctrl

¹ From rv986's probe of 9ab5874b in C, C.UTF-8 and POSIX, which found C0 80, C1 BF, E0 80 9B, E0 82 9B and F0 80 82 9B taken the same way in all seven places, and each refused at the base. CONTROL_RULE's an-overlong-escape row pins C0 9B in every place.

A real bare repository at a café path, and a ref café with an ASCII repository (stub gh and curl that exit 97, so a fetch fails as offline):

Run before, C before, C.UTF-8, and after in both
--version exit 2, a control character prints it, exit 0 (both values)
scaffold run Atlas --org acme exit 2, no request the two requests for setup.sh, then exit 2 "could not fetch setup.sh" (both values)
--install from a file exit 2, nothing placed installed, exit 0 (both values)
--doctor <dir> beside a checkout exit 2, the doctor never ran the doctor ran (both values)
--doctor <dir>, no checkout exit 2, a control character repository: cloned, the doctor ran; ref: exit 2, the shape refusal (#942 moves it before the clone)
--preflight, $OPENREPOSHAPE_SETUP_SH in a checkout exit 2 repository: the preflight ran; ref: setup-project.py's shape refusal
--preflight, a setup.sh outside any checkout exit 2 repository: cloned, the preflight ran; ref: setup.sh's shape refusal

The package, from ebbd9fa4 (after #947), python3 -m openreposhape --version over the same values: the repository's answers are the launcher's in every row and both locales (taken: café, the three lone bytes, U+00A0, U+0378, U+FFFE; control: the nine; code point: U+202E, U+200B, U+2028, U+2029). Its ref gets the same control answer in every row; where the control check takes a ref, the package's --version goes on to the ref's shape, which the launcher's --version does not ask (#866), so there it is refused by shape (Not changed).

Decisions

  1. The rule is install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR holding : or a newline is refused by install.sh only after its fetch, and one holding a carriage return or an escape is placed and echoed as typed, so both refuse every control character before they fetch or print anything #760's bytes, in all seven places, under LC_ALL=C in a subshell, so each answers one value one way in every locale. Each if is the three lines the case it replaces was. The one answer still holds after standard.py: the package's openRepoShape refuses a repository or ref holding U+202E, U+200B or another character the launcher refuses, by code point, before --version or any git call names it, and refuses a.lock/b, a/.b, a//b, a/b/ and a. before it clones, in the launcher's words (#939, #941) #947 (5f62ea48, now on main), which made the package refuse install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR holding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837's characters by code point before its CONTROL_RE, as the launcher does: probed from ebbd9fa4, the package's --version gives a repository the launcher's answer in every row of the table and every row above, in both locales, and gives a ref the same control answer (Measured). tests/test_openreposhape_package.py now runs this table's rows (through UNPRINTABLE_VALUES) against the package too.
  2. A byte that is not UTF-8, as a lone 0x9B or an overlong form of a control (C0 9B, an escape in two bytes): the trade is taken, as install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR the bytes refuse is read again in the locale's own charset, so a GB18030, GBK or Shift_JIS name that holds no control character and nothing drawn as another is placed, and where nothing can read it the bytes decide as before #867 took it for the directory. Both are the same class: bytes no UTF-8 decoder reads as a control. Each passes in both locales now, where [![:print:]] refused it in both (rv986 measured the overlong forms in all seven places); the package's --version already passes both (measured: /srv/a, 0x9B, b prints as a?b, exit 0, under C and C.UTF-8; rv986: /srv/a, C0 9B, b prints as a??b, exit 0). A terminal that reads UTF-8 draws it as a character it cannot read, and only one set to an 8-bit charset acts on it. Where it passes, the line after it prints it: --version, setup.sh's clone line, and the shape or owner/name refusals that name a value. The comment above each place says so, as install_commands's does (openRepoShape:633-635 after this change).
  3. setup.sh keeps refusing U+2028 and U+2029, by their three bytes (\342\200, then \250 or \251), one arm beyond install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR holding : or a newline is refused by install.sh only after its fetch, and one holding a carriage return or an escape is placed and echoed as typed, so both refuse every control character before they fetch or print anything #760's rule in both guards. This is lane 2's ruling: keeping setup.sh's existing refusal of the two separators is not install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR holding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837's candidate. Probed at the base, setup.sh refused both in both locales: C's [![:print:]] with every byte past ASCII, and C.UTF-8's because glibc counts both among its controls. The bare rule would take both in both locales, and the clone line and the --shape-ref refusal would then print a line separator they never printed. The launcher, install.sh and the package refuse both first by code point (install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR holding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837), so they need no arm; the arm keeps setup.sh as it was, in the same words, and the four still refuse both.
  4. The network's values get no trade. install.sh's redirect location keeps the locale's [![:print:]] for naming it: the fault names it only to say more, and that class matches a lone 0x9B and every overlong form in C, C.UTF-8 and POSIX (measured), so no such byte from the network is printed; the byte rule's trade would print one. The cost: under C a location with a letter past ASCII goes unnamed ("it redirects to a location that names no release"). The tag in a …/releases/tag/<tag> location is taken only when it is printable ASCII, under LC_ALL=C (fix round 1, rv986's SHOULD-1): at 9ab5874b the arm took any tag, checked_version passed a lone 0x9B in it by the trade, and the shape refusal printed the byte raw. Now such a tag names no release, in every locale, and gh is asked next. A release tag is ASCII by the release scheme, so this refuses no real one, and a tag with a letter past ASCII gets one answer in every locale, where a UTF-8 locale's [:print:] would have passed it on. The tag gh answers is unchanged: gh decodes the JSON with Go's encoding/json, which reads a byte that is not UTF-8 as U+FFFD, so a lone 0x9B cannot come from it, and a C1 control comes as its two bytes, which the rule refuses (read from Go's documentation, not measured: no real gh runs here). The comments in latest_from_redirect and above checked_version say so, and the redirect test pins both forms.
  5. A ref café under --doctor with no checkout is not tested here. Whether checked_ref refuses it before or after the clone is openRepoShape: --doctor's shape_checkout runs git clone before checked_ref, so a ref it refuses (a.lock/b, a//b, one over 255 characters) costs a whole clone first, and where the clone fails the ref's refusal is never reached #942's (shape_checkout), which lane 2's w942 changes in the same file; this PR's ref test of --doctor runs it beside a checkout, where the ref names nothing to check out.

Tests

One table, CONTROL_RULE in tests/conftest.py, appended after REF_PARTS's helpers, read by all three files, as #874's REF_PARTS is. Each row is how a value ends and what the checks say of it: taken (café, a lone 0x9B, a lone 0xE9, an overlong escape C0 9B), a control character (an escape, U+009B, DEL, a tab, U+0085, U+0080, U+009F, a newline at the end, and a lone 0xE9 then U+009B), or a code point (U+202E, U+200B, U+2028, U+2029). the_rules_answer reads a run's answer and assert_a_copy_answers_as_the_rule_does holds it to the row. A lone byte is a surrogate that surrogateescape hands a child as the byte, and run_cmd, run_bare_setup and Net.run read output that way where a test asks.

File Test Cases What it pins
tests/conftest.py CONTROL_RULE, SEPARATORS, A_LETTER_PAST_ASCII, REPO_WITH_A_LEADING_DASH, the_rules_answer, assert_a_copy_answers_as_the_rule_does the one table and how a run's answer is read and held to it
tests/test_openreposhape_command.py test_version_answers_a_repository_or_ref_as_the_control_rule_does (new) 136 each of the 17 rows at the end of a path, a URL, an scp address and a ref, under C and C.UTF-8: taken printed as typed, exit 0; the rest refused in the launcher's words, none of it echoed
test_a_letter_past_ascii_goes_through_every_run_in_either_locale (new) 32 café in each form, in both locales: a scaffold run fetches setup.sh once with it, --install from a file places the command, --doctor hands a fake git the URL (repository) or runs beside a checkout (ref), --preflight hands setup.sh the repository or --shape-ref
test_a_repository_or_ref_no_line_can_print_is_refused_before_any_request (rows now from the table) 180 (was 72) every refused row in both variables, the leading -, the three trailing-newline forms, through --version, a scaffold run and --install from stdin: exit 2, the one refusal, no request. tests/test_openreposhape_package.py runs the same rows through the package's --version (70 cases, was 34)
test_version_prints_a_path_a_utf_8_locale_prints (now both locales) 14 (was 7) NOT_CONTROLS paths printed under C too
tests/test_setup_sh.py test_the_repository_guard_answers_as_the_launchers_rule_does (new) 32 15 rows (the table, with the separators as control characters) and a leading -, both locales: taken goes to git clone and the clone line names it; the rest exit 2 with the guard's words, no clone, nothing echoed. The file's first C1 and leading-- cases
test_the_shape_ref_guard_answers_as_the_launchers_rule_does (new) 30 the 15 rows as --shape-ref v1<x>: taken meets the shape refusal in C.UTF-8's words; the rest exit 2 with the guard's words. The file's first C1 case for the ref
tests/test_install_sh.py test_the_installer_answers_each_value_as_the_launchers_rule_does (new) 102 the 17 rows in $OPENREPOSHAPE_REPO, the version and an option, both locales: exit 2, nothing placed, no request, in each check's words (taken: the owner/name refusal, the shape refusal, "unknown option"). checked_repo's first control-character test, and the first C1 case of all three
test_the_redirect_names_no_location_holding_a_control_character (new) 26 Decisions, 4: a location ending in each control row, or in a byte that is not UTF-8 (the lone 0x9B, the lone 0xE9, the overlong escape), is never named; a tag holding a lone 0x9B, or a letter past ASCII, is not taken and names no release; in either locale
test_the_installer_refuses_a_value_drawn_as_another_unechoed (the_newest_tag now from gh) 10 the newest release's tag holding U+202E comes from gh, since the redirect's tag is no longer passed on (Decisions, 4)
tests/test_openreposhape_command.py test_the_seven_control_checks_carry_one_rule_as_it_is (new) 3 the seven if (LC_ALL=C; case … lines are one text in each file (2, 2 and 3), apart from the value, the leading - arm and setup.sh's separators arm, as test_install_sh_carries_the_launchers_misdrawn_check_as_it_is holds misdrawn_characters
tests/test_setup_sh.py test_the_shim_takes_a_shape_ref_as_git_does_before_any_clone (#874's) as before calls run_shim_under, now (tmp_path, repo, *args, locale=""), in place of the two blocks it shared with it; its assertions are unchanged

Helpers: run_cmd, run_bare_setup and Net.run take errors; cloned_for takes env; the fake curl writes headers with surrogateescape, so a lone byte reaches the installer.

Mutants (S/w940/mutants.py, and S/w940/fix1-out/mutants2.py for fix round 1, one at a time in a scratch copy of the tree, the targeted tests of the file each mutant is in):

Place No C1 arm No [[:cntrl:]] arm No leading-- arm No separators arm Back to [![:print:]] in the locale No LC_ALL=C
launcher checked_repo killed (60) killed (66) killed (6) killed (34) killed (1)²
launcher ref_character_fault killed (40) killed (32) killed (9) killed (1)²
setup.sh $SHAPE_REPO killed (10) killed (8) killed (2) killed (4) killed (5) killed (1)²
setup.sh --shape-ref killed (10) killed (8) killed (4) killed (5) killed (1)²
install.sh checked_repo killed (10) killed (8) killed (5) killed (1)²
install.sh checked_version killed (10) killed (8) killed (5) killed (1)²
install.sh option check killed (10) killed (8) killed (5) killed (1)²

The numbers are the failing cases. The seven the issue found surviving (setup.sh guard 1 without the C1 or the - arm, guard 2 without the C1 arm, install.sh's checked_repo without either arm, checked_version and the option check without the C1 arm) are killed. The redirect's location moved to the byte rule is killed by 6 cases (the lone 0x9B, the lone 0xE9 and the overlong escape, both locales).

² By test_the_seven_control_checks_carry_one_rule_as_it_is, the case for that file (fix round 1, NIT-2). The behaviour tests cannot kill these: no locale installed here answers a row differently without LC_ALL=C (Not changed).

Fix round 1's other mutants, all killed:

Mutant Killed by Failing cases
the option check's C1 range ending at \236 the spelling test 1
the redirect's tag arm without LC_ALL=C the redirect test 1 (café tag, C.UTF-8)
the redirect's tag arm taking every tag (as 9ab5874b did) the redirect test 4 (both tag rows, both locales)
the redirect's tag arm on the byte rule the redirect test 4

And 9ab5874b's install.sh against this round's tests (-k "names_no_location or drawn_as_another_unechoed"): 4 failed, the two tag rows in both locales, with REFUSED: the newest release's tag is 'v1<0x9B>31mforged', … where the test asks for "names no release"; 32 passed.

Not changed

Overlaps

Fix round 1

rv986's review of 9ab5874b was READY with one SHOULD and three NITs; d4bba685b answers all four, on the merge of 2a78ae1a (a4c39592).

  • SHOULD-1 (install.sh's redirect tag): latest_from_redirect takes the tag only when it is printable ASCII under LC_ALL=C, else "it redirects to a location that names no release" and gh is asked; the comments there and above checked_version say so. The redirect test gains a tag with a lone 0x9B (rv986's repro) and a tag with café, and the_newest_tag now comes from gh. Decisions, 4.
  • NIT-1 (overlong forms): the comment at each of the seven rule lines names the trade as "a byte that is not UTF-8, as a lone 0x9B or an overlong form of a control", CONTROL_RULE gains an-overlong-escape (a, C0 9B, b, taken), and the redirect test takes the table's bytes that are not UTF-8 as rows. Decisions, 2.
  • NIT-2 (the LC_ALL=C survivors): test_the_seven_control_checks_carry_one_rule_as_it_is holds the seven lines to one text; it kills the seven LC_ALL=C mutants and a range-end mutant (Mutants).
  • Runs on this tree (one pytest process at a time, EMAIL set): the eight files (the three test files, tests/test_openreposhape_package.py, the trio and tests/test_install_sh_filter_repo.py), 1738 passed in 305.51s; this round's targeted tests, 377 passed.
  • NIT-3 (the repeated blocks in tests/test_setup_sh.py): run_shim_under(tmp_path, repo, *args, locale="") moved above setup.sh, install.sh and the openRepoShape command: each keeps its own copy of the old ref rule (*..* | *.lock), so a ref git refuses part by part (a.lock/b, a/.b, a//b) clones the standard before it fails #874's test, which calls it; the callers pass --preflight themselves.

Closes #940

🤖 Generated with Claude Code

Summary by Sourcery

Use a consistent locale-independent control-character policy across repository, ref, version, option, and release-redirect validation while preserving safe handling of actual controls and line separators.

Bug Fixes:

  • Align repository and ref control-character validation across the launcher, setup.sh, and install.sh so non-ASCII letters and invalid UTF-8 bytes are no longer misclassified while actual control characters remain rejected.
  • Prevent non-printable or non-ASCII release tags from redirect responses from being echoed or treated as releases.

Enhancements:

  • Standardize the seven shell validation checks on a locale-independent byte-based control-character rule, while preserving setup.sh handling for Unicode line and paragraph separators.

Tests:

  • Add shared control-character cases covering ASCII controls, C1 controls, Unicode separators, non-ASCII text, invalid UTF-8 bytes, and overlong encodings across all shell entry points.
  • Expand integration coverage to verify non-ASCII repository paths and refs work consistently under C and UTF-8 locales, and validate redirect safety and rule consistency.

Brett Heap and others added 3 commits October 18, 2026 18:48
…a letter past ASCII is no control character under `C` either, as #760's byte rule now decides in all seven checks that held one to `[![:print:]]`, and every control character is still refused in both locales (#940)

#872 checked `$OPENREPOSHAPE_REPO` and `$OPENREPOSHAPE_REF` before every
run but `--help`, with `[![:print:]]`, which under `C` and POSIX matches
every byte past ASCII: a mirror at `/srv/café/openRepoShape.git`, or a ref
`café`, was refused as "carrying a control character" by `--version`, a
scaffold run, `--install` and `--preflight`, where `C.UTF-8` took it.
`setup.sh`'s two guards and `install.sh`'s `checked_repo`,
`checked_version` and option check held the same class.

All seven now ask #760's rule, as `install_commands` holds the directory to
it: under `LC_ALL=C`, in a subshell, `[[:cntrl:]]` or U+0080 to U+009F's
two bytes, so one value gets one answer in every locale, and the launcher,
`setup.sh`, `install.sh` and the package agree. The leading `-` arms stay,
`misdrawn_characters` stays first, and no refusal's words change. A lone
`0x9B` byte that is not UTF-8 now passes in both locales, the trade #867
took for the directory, which the comment at each place states. `setup.sh`,
which has no copy of `misdrawn_characters`, also refuses U+2028 and U+2029
by their bytes, as it refused both in every locale before. The redirect's
location in `install.sh` keeps `[![:print:]]`: it is the network's, and
that class prints no lone C1 byte from it.

One table, `CONTROL_RULE` in `tests/conftest.py`, holds the seven to one
answer under `C` and `C.UTF-8`: `café`, a lone `0x9B` and a lone `0xE9`
taken; an escape, U+009B, DEL, a tab, U+0085, U+0080, U+009F, a newline at
the end and a lone `0xE9` then U+009B refused; U+202E, U+200B, U+2028 and
U+2029 refused by code point. The three test files read it, and a
repository or ref with `café` goes through every launcher run in both
locales. Every arm of the rule dropped from any place fails a test, the
seven the issue found surviving included.

Closes #940

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter past ASCII is no control character under `C` either, as #760's byte rule decides in all seven checks that used `[![:print:]]`, and every control character is still refused (#940)

#872 checked `$OPENREPOSHAPE_REPO` and `$OPENREPOSHAPE_REF` before every
run but `--help`, with `[![:print:]]`, which under `C` and POSIX matches
every byte past ASCII: a mirror at `/srv/café/openRepoShape.git`, or a ref
`café`, was refused as "carrying a control character" by `--version`, a
scaffold run, `--install` and `--preflight`, where `C.UTF-8` took it.
`setup.sh`'s two guards and `install.sh`'s `checked_repo`,
`checked_version` and option check held the same class.

All seven now ask #760's rule, as `install_commands` holds the directory to
it: under `LC_ALL=C`, in a subshell, `[[:cntrl:]]` or U+0080 to U+009F's
two bytes, so one value gets one answer in every locale, and the launcher,
`setup.sh`, `install.sh` and the package agree. The leading `-` arms stay,
`misdrawn_characters` stays first, and no refusal's words change. A lone
`0x9B` byte that is not UTF-8 now passes in both locales, the trade #867
took for the directory, which the comment at each place states. `setup.sh`,
which has no copy of `misdrawn_characters`, also refuses U+2028 and U+2029
by their bytes, as it refused both in every locale before. The redirect's
location in `install.sh` keeps `[![:print:]]`: it is the network's, and
that class prints no lone C1 byte from it.

One table, `CONTROL_RULE` in `tests/conftest.py`, holds the seven to one
answer under `C` and `C.UTF-8`: `café`, a lone `0x9B` and a lone `0xE9`
taken; an escape, U+009B, DEL, a tab, U+0085, U+0080, U+009F, a newline at
the end and a lone `0xE9` then U+009B refused; U+202E, U+200B, U+2028 and
U+2029 refused by code point. The three test files read it, and a
repository or ref with `café` goes through every launcher run in both
locales. Every arm of the rule dropped from any place fails a test, the
seven the issue found surviving included.

Closes #940

Lane: openreposhape-2 (openRepoShape-2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR standardizes seven shell-side validation checks on #760’s locale-independent byte rule so non-ASCII letters and selected non-control bytes are accepted consistently under C and UTF-8 locales, while control characters remain rejected; setup.sh retains its separator-specific safeguards, and extensive shared tests validate behavior, propagation, output safety, and the intentional network-redirect exception.

Sequence diagram for validating a non-ASCII repository or ref

sequenceDiagram
    participant User
    participant Launcher as openRepoShape
    participant Script as setup.sh_or_install.sh
    participant Git
    User->>Launcher: Provide repository, ref, version or option
    Launcher->>Launcher: misdrawn_characters
    Launcher->>Launcher: LC_ALL=C byte-rule check
    alt control character
        Launcher-->>User: Refuse with control-character message
    else non-control value
        Launcher->>Script: Pass value to downstream check
        Script->>Script: LC_ALL=C byte-rule check
        alt setup separator U+2028 or U+2029
            Script-->>User: Refuse with control-character message
        else accepted by byte rule
            Script->>Git: Continue clone or checkout validation
            Git-->>User: Operation result
        end
    end
Loading

Flow diagram for locale-independent input validation

flowchart TD
    A[Input value] --> B[misdrawn_characters]
    B -->|visual hazard| C[Reject with code-point message]
    B -->|safe| D[LC_ALL=C byte-rule check]
    D -->|control byte| E[Reject as control character]
    D -->|non-control byte or UTF-8 letter| F[Continue to existing validation]
    F --> G[Accept or apply existing shape or owner/name rules]
Loading

File-Level Changes

Change Details Files
Replace locale-dependent printable-character validation with a consistent byte-oriented control-character rule for user-supplied repositories, refs, versions, and options.
  • Use an LC_ALL=C subshell to reject POSIX control bytes and UTF-8 encodings of U+0080–U+009F.
  • Preserve leading-dash protection and setup.sh-specific U+2028/U+2029 rejection.
  • Keep network redirect validation on [![:print:]] to avoid echoing unsafe network-provided locations.
openRepoShape
setup.sh
install.sh
Add shared control-rule fixtures and comprehensive cross-script tests covering locale behavior, byte edge cases, propagation, and output safety.
  • Define shared values and expected outcomes for taken values, controls, code points, separators, and leading-dash repositories.
  • Test launcher, setup, installer, package, and redirect behavior under C and C.UTF-8, including surrogate-escaped lone bytes.
  • Extend subprocess helpers to preserve non-UTF-8 output and verify control-character refusals do not echo input.
tests/conftest.py
tests/test_openreposhape_command.py
tests/test_setup_sh.py
tests/test_install_sh.py

Assessment against linked issues

Issue Objective Addressed Explanation
#940 Make the launcher's repository and ref checks, both setup.sh guards, and install.sh's three checks use a locale-independent control-character rule, so letters past ASCII are not rejected as controls under C while control characters remain refused. ✅
#940 Keep the package and shell commands consistent in their treatment of repository values, including the agreed behavior for lone non-UTF-8 C1 bytes. ✅
#940 Add coverage for the changed behavior across the relevant scripts and locales, including preserved refusals for control characters and setup.sh's existing treatment of U+2028 and U+2029. ✅

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-2, 2026-10-19T14:14:44Z) Independent review of #986 (#940) at 9ab5874b, for lane openRepoShape-2: READY

The PR does what #940 asks. All seven places now use one byte rule and give one answer under C, C.UTF-8 and POSIX. A URL, path or ref with é, U+00A0 or CJK goes through every run. Every C0 control, DEL, a trailing newline and all 32 C1 controls are still refused everywhere. There is one SHOULD, with low practical risk, and three NITs.

SHOULD-1. install.sh:615 with :231-232 and :192: a lone 0x9B byte that comes from the network is now printed, which Decision 4 and the comment at install.sh:238-244 say cannot happen.

  • What happens: latest_from_redirect takes the tag out of a …/releases/tag/<tag> location as FOUND (:231-232). main then runs it through checked_version "$version" "the newest release's tag" (:615).
  • checked_version now passes a lone 0x9B byte. That is the trade, and it was meant for values a person types. The shape refusal at :192 then prints the tag as it came.
  • Repro: a stub curl answers -fsSI with location: https://github.com/opensoft/openRepoShape/releases/tag/v1, then the byte 0x9B, then 31mforged. The probe ran bash install.sh with no arguments.
    • At head, under C, C.UTF-8 and POSIX: exit 2, and the output holds REFUSED: the newest release's tag is 'v1<0x9B>31mforged', which is not a release tag, ... with the raw byte. A terminal set to an 8-bit charset reads that byte as CSI, so the rest of the line turns red.
    • At the base a4de0aae, in all three locales: exit 2, the newest release's tag carries a control character, ..., and nothing of the tag is printed.
    • The same byte in a location without /releases/tag/ stays unnamed at head. So Decision 4 holds only for that form.
  • Why it is not a MUST: the value comes from github.com, over HTTPS only (--proto-redir '=https'). GitHub percent-encodes a location, and the repository's owner can already ship the command it installs.
  • Gap in the tests: test_the_redirect_names_no_location_holding_a_control_character tries only locations without /releases/tag/.
  • Fix: in the */releases/tag/?*) arm, take FOUND only when the tag holds nothing [![:print:]] matches. Otherwise set FAULT="it redirects to a location that names no release" and return 1. Then add a row to the redirect test: /releases/tag/v1, then a lone 0x9B, then b. The other fix is to narrow what Decision 4 and the comment claim.

NIT-1. Each rule line (openRepoShape:827, :937, setup.sh:233, :270, install.sh:149, :188, :476) and its comment: overlong encodings of a control now pass, and nothing names them.

  • C0 9B (an overlong ESC), C0 80, C1 BF, E0 80 9B, E0 82 9B and F0 80 82 9B are taken in all seven places, in all three locales. The base refused each one.
  • This is the same class as the lone 0x9B trade (bytes that are not UTF-8), and the package takes them too (/srv/a, then C0 9B, then b prints as a??b, exit 0).
  • However, the comments name only "a lone C1 byte that is not UTF-8", and CONTROL_RULE has no row for them.
  • Fix: say "a byte that is not UTF-8, as a lone 0x9B or an overlong form of a control" in the comments, and add a taken row such as a, C0 9B, b.

NIT-2. The seven "no LC_ALL=C" mutants survive. They are not equivalent in principle, and no test holds the seven rule lines to one spelling.

NIT-3. tests/test_setup_sh.py:1067-1070 and :1075-1078 repeat :951-954 and :962-965.

Checked and true

  • Every control is refused in every locale. A probe of all seven places under C, C.UTF-8 and POSIX: 111 values, 2331 runs at head and 2331 at base. It used a logging git, and gh and curl stubs that log and exit 97.
    • At head and at base, all 31 C0 controls (TAB and ESC among them), DEL, a trailing newline, all 32 C1 code points as UTF-8, and a lone 0xE9 followed by U+009B are refused as control characters, in every cell.
    • At head, é, U+00A0 and 漢字 are taken in every cell. The base refused them under C and POSIX.
    • No value gets a different answer in different locales at head. No run outside setup.sh's clone sent a request.
    • A NUL cannot reach any of the checks: $'a\0b' is a in bash, and the environment and argv refuse NUL.
  • The curl … | bash paths: piped setup.sh and install.sh behave as they do from a file. café in setup.sh's repository reaches git clone, and ESC and U+009B are refused.
  • The shells: the LC_ALL=C set in the subshell does not leak. After the if, LC_ALL and ${#v} are unchanged (4 for café under C.UTF-8), and a child process sees the caller's locale. set -e does not fire on the subshell's exit 1. Measured in bash 5.2.21.
  • The tests pin it.
    • The new tests against the base's three scripts gave 68 failed, 530 passed. Every failure is a taken row: é and NOT_CONTROLS under C, the lone 0x9B and the lone 0xE9 in both locales, and the C-only letter-past-ASCII runs. No control row failed.
    • The case counts in the body are right: 128, 32, 180, 14, 30, 28, 96, 18, and 70 in the package.
  • My 14 mutants: 13 killed, 1 survived (NIT-2).
    • Killed:
      • three ranges narrowed by one, at each end (2, 2 and 12 failing cases);
      • [[:cntrl:]] swapped for [![:print:]] in three places (14, 6, 6);
      • each -* arm dropped (6, 2);
      • the option check without its C1 arm (10);
      • the separators arm without U+2029 (2);
      • the launcher checking $url and not $url$REPO (24);
      • the redirect's class moved to the locale's [[:cntrl:]] (8);
      • checked_version without its control arm (8).
    • This agrees with the writer's 34 mutants: 27 killed, and the seven LC_ALL=C ones survived.
  • Decisions:
  • Merges:
    • git merge-tree --write-tree of 9ab5874b is clean with c4dd19dc, cf66102f and 4f83eaf5, the three together, and with main c7a3ff16 (tree ad3bdfe6).
    • On the merge with c7a3ff16, the four test files gave 1521 passed.
    • On the merge with cf66102f, the eight files (the three test files, the package's, the trio and test_install_sh_filter_repo.py) gave 1693 passed.
  • Sonar: the diff has no composite asserts. The new helpers stay well under complexity 15 (the_rules_answer has three ifs). I found no duplicate of 4 lines or more across the three test files (NIT-3 is within one file).
  • Docs: git grep 'print:]' over the three scripts, README.md, AGENTS.md, docs/cli.md, docs/handbook.html, templates/** and src/** finds no sentence still saying the repository or ref is held to [![:print:]].

Could not verify

Lane 2's decision: SHOULD-1 and the three NITs go back to the writer for one fix round before #986 lands. This reviewer then re-checks the new head, so the landing head has its own READY.

Lane: openreposhape-2 (openRepoShape-2)

Brett Heap added 2 commits October 19, 2026 14:16
…leases/tag/ tag is taken only in printable ASCII, the same in every locale, so a lone 0x9B from the network is never printed (SHOULD-1); the seven rule comments name an overlong form of a control beside the lone 0x9B, and CONTROL_RULE gains that row (NIT-1); a test holds the seven rule lines to one spelling (NIT-2); #874's test calls run_shim_under (NIT-3)

Tested by w940: the eight files, 1738 passed; the new tag rows fail against 9ab5874's install.sh; NIT-2's eight mutants and three on the tag check are killed.

Lane: openreposhape-2 (openRepoShape-2)
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-2, 2026-10-19T21:33:31Z) Independent review of #986 (#940) at d4bba685, for lane openRepoShape-2: READY

This re-checks the PR after its fix commit. d4bba685b is the old head 9ab5874b, then the merge of main 2a78ae1a (a4c39592), then one fix commit for round 1's four findings. All four are fixed. One new NIT, on the landing text, is left.

Fixed since 9ab5874b

  • SHOULD-1 is fixed (install.sh:231-249). A tag taken from a …/releases/tag/<tag> location is now accepted only when it is printable ASCII, checked under LC_ALL=C. Any other tag counts as a location that names no release: nothing of it is printed, and gh is asked next.
    • Reran the redirect probe at d4bba685b, under C, C.UTF-8 and POSIX. The tag was v1 followed by each of: a lone 0x9B, an overlong ESC (C0 9B), U+009B, café, ESC, or U+202E.
      • Every run exits 2 with could not find the newest release …. gh is asked, and nothing of the tag reaches either output stream.
      • At 9ab5874b, the lone 0x9B and the overlong ESC were printed raw by the shape refusal.
    • A real ASCII tag still works: v1.2.3 is taken (release: v1.2.3, the newest release of …, then the curl -fsSL fetch). v1..2 still gets the shape refusal.
    • The new rows a-tag-with-a-lone-0x9b-byte and a-tag-with-an-e-acute in test_the_redirect_names_no_location_holding_a_control_character pin this behaviour. Dropping LC_ALL=C from the tag check is killed (1 failing case).
  • NIT-1 is fixed. The comments at the rule lines now name "a byte that is not UTF-8, as a lone 0x9B or an overlong form of a control". CONTROL_RULE gains the taken row an-overlong-escape (a, C0 9B, b), which every file that reads the table now runs.
  • NIT-2 is fixed. test_the_seven_control_checks_carry_one_rule_as_it_is (tests/test_openreposhape_command.py:927) holds the seven lines to one text. It allows only three differences: the value checked, the leading - arm, and setup.sh's separators arm.
    • Dropping LC_ALL=C; from each of the seven places, one at a time: 7 of 7 killed by that test, each with 1 failing case.
    • The test also fails if an eighth one-line check appears, or one goes away (counts 2, 2 and 3).
  • NIT-3 is fixed. run_shim_under (tests/test_setup_sh.py:934) is now shared, and setup.sh, install.sh and the openRepoShape command: each keeps its own copy of the old ref rule (*..* | *.lock), so a ref git refuses part by part (a.lock/b, a/.b, a//b) clones the standard before it fails #874's test calls it, so the two 4-line copies are gone.
  • No code changed in the other two scripts: the fix commit's changes to openRepoShape and setup.sh are comment lines only.

NIT-1 (new). The landing text does not yet describe the fix.

  • The squash message I could read is the one in lane 2's attachments folder, impl940-c-locale-byte-rule/squash-message.md, dated 2026-10-18T20:15Z. It names neither the tag rule nor the spelling test.
  • The PR body's Decision 4 also speaks only of the location.
  • Fix: before landing, add one sentence to each. For example: "a tag the redirect names is taken only in printable ASCII, so no byte that is not UTF-8 from the network is printed, and a test holds the seven rule lines to one text."

Checked and true

  • On the merge with main 3864b18cc: git merge-tree --write-tree is clean (tree c498a570). There, tests/test_openreposhape_command.py, tests/test_setup_sh.py, tests/test_install_sh.py and tests/test_openreposhape_package.py gave 1550 passed, exit 0.
    • That is 29 cases more than at 9ab5874b merged with c7a3ff16 (1521). The new rows and the new test account for exactly 29: 8 + 4 + 6 + 8 + 3.
  • Still true from round 1 at 9ab5874b, and the fix touches none of it except comments:
    • in all seven places and all three locales, every C0 control, DEL, a trailing newline and all 32 C1 controls are refused, and é, U+00A0 and CJK are taken (2331 probe runs);
    • the new tests fail against the base's scripts only in the rows the change should take (68 failures, no control row among them);
    • 13 of my 14 mutants were killed (the one survivor was an LC_ALL=C drop, which the new spelling test now kills, above);
    • the LC_ALL=C set in a subshell does not leak to the caller;
    • the curl | bash paths behave as runs from a file;
    • no doc still says the repository or the ref is held to [![:print:]].
  • Rulings on the decisions: the lone 0x9B trade for values a person sets, setup.sh's U+2028 and U+2029 arm, the location keeping [![:print:]] (now the tag too), and the --doctor ref left to openRepoShape: --doctor's shape_checkout runs git clone before checked_ref, so a ref it refuses (a.lock/b, a//b, one over 255 characters) costs a whole clone first, and where the clone fails the ref's refusal is never reached #942 (landed as ae63d450) are all accepted, as in round 1.

Could not verify

Lane 2's note on the new NIT: the reviewer read the 2026-10-18 copy of the squash message. The writer's updated one already says the tag in a …/releases/tag/<tag> location is taken only when it is printable ASCII under LC_ALL=C. Lane 2 adds the sentence on the spelling test before landing, and #986 lands only once its tests job is green at d4bba685b.

Lane: openreposhape-2 (openRepoShape-2)

@sonarqubecloud

Copy link
Copy Markdown

@brettheap
brettheap marked this pull request as ready for review October 20, 2026 03:19
@sourcery-ai

sourcery-ai Bot commented Oct 20, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 21 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 64bd58d into main Oct 20, 2026
5 checks passed
@brettheap
brettheap deleted the fix/940-c-locale-byte-rule branch October 20, 2026 03:19
brettheap pushed a commit that referenced this pull request Oct 20, 2026
…lict, so #1025 is tested against main as it stands, with #983's change to validate-family.py and #1026's fix of main's two red rows (#936)

Main took twelve squashes since e076e3e: #1026, #958, #986, #1002,
#1016, #1011, #1021, #1022, #973, #975, #1015 and #983. Only #983
touches validate-family.py, in `_git_runs` and the member checks, apart
from this branch's hunks; it touches none of this branch's test files.

Lane: openreposhape-4 (openRepoShape-4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant