Repository navigation
install.sh and openRepoShape: a $OPENREPOSHAPE_BIN_DIR, the repository or a ref holding a bidi control, a character of no width, a soft hyphen or U+2028 is refused by its code point before any request and never echoed, so no line shows the value as another - #841
Merged
Merged
Conversation
…IR` is respelled with `cygpath -u` or refused before install.sh's fetch, and the PATH note single-quotes a directory bash would still read, and `--install` refuses one holding `:` or a newline, so install.sh names `/d/tools/bin` as the command does and the `export PATH` line does what it says (#667) #649's two leftovers, both from #656's review. install.sh read `$OPENREPOSHAPE_BIN_DIR` as typed, so with `D:\tools\bin` its `preflight:` and `done:` lines (and `install:` for an old release) said `D:\tools\bin/openRepoShape` while the command's own lines said `/d/tools/bin/openRepoShape`, and where no `cygpath` answers the command's `--install` refused (exit 2) only after install.sh had fetched it. install.sh now carries a copy of the command's rule where it reads the variable, before the symlink check and the fetch: a value that starts `[A-Za-z]:/` or `[A-Za-z]:\` is spelled with `cygpath -u --`, and is refused with exit 2 where no `cygpath` answers, fails, or answers a drive spelling again. The refusal does not echo the value. The command's PATH note printed `export PATH="$dir:$PATH"` with the directory between double quotes as it was, so `a"b$HOME c` printed a line `eval` could not parse, and a `$`, a backquote or a `\` in a name was read, not kept. `export_path_line` keeps today's line byte for byte for a directory that double quotes hold (a space and an apostrophe included) and, for one holding a `"`, `$`, backquote, `\` or `!` (bash reads that at a prompt as history), prints it between single quotes with `'\''` for an apostrophe. `--install` also refuses a directory holding a `:` or a newline, which no `$PATH` line can name, before anything is placed; the newline's refusal does not echo it. Tests feed one value through install.sh and through the command, in each spelling and each state of `cygpath`, and pin every `install:`, `preflight:` and `done:` line; the PATH note is checked as printed, by `eval` and by `bash -i`. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…holding a control character before they name it, and install.sh refuses a `:` before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #667's two checklist candidates, with rv667's NIT 2. `--install` refused a newline in `$OPENREPOSHAPE_BIN_DIR` and nothing else that is a control character: a carriage return, an escape, a tab or DEL placed the command and was printed as typed (`/tmp/x^MFORGED is not on $PATH`; an escape sequence reaches the terminal the same way). The newline arm is now `*[[:cntrl:]]*`, first in `install_commands`, ahead of the Windows spelling and the colon, and its refusal still says nothing of the value. It is `[[:cntrl:]]` and not `[![:print:]]`, so a directory with a letter the locale does not print (`café` under LC_ALL=C) is placed. Which C1 controls match is the locale's, measured with bash 5.2.21: C.UTF-8 matches U+0080 to U+009F (and U+2028, U+2029), C and POSIX match none, and the ASCII controls and DEL match in all three. install.sh learnt of a `:` or a newline only after its fetch: it fetched the launcher, ran `--install`, and said only that `--install` refused (exit 2); a release older than #667 placed the command under the directory. It now carries both arms of the command's `case`, copied as #667 copied the drive-letter one: the control-character arm first, before any line names the directory, and the colon arm after the Windows respelling, both before the symlink check and any request. Where the control-character refusal says nothing of the value, the colon refusal names the directory, as the command's does. Tests feed each value through install.sh and through the command (a colon, two directories, a drive-relative name, a newline, a carriage return, an escape, a tab, DEL, a drive spelling with a newline or an escape, a colon with an escape): exit 2, no request, nothing placed, neither stream holds a forged tail. A link in a directory holding a carriage return, a drive spelling, a colon with an escape, a letter under LC_ALL=C and LC_ALL=C.UTF-8, a C1 control under a UTF-8 locale, and the `$HOME` default pin the order and the width of the arms. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…`LC_ALL=C` and names a C1 control's two bytes, so U+0080 to U+009F is refused in every locale and whatever else the value holds, where a UTF-8 locale let one through beside a byte that is not UTF-8 (#760) Lane 1's review of #804 (finding 1): `*[[:cntrl:]]*` matches a C1 control only in a value that is valid UTF-8. One byte that is not (a lone 0xE9) makes bash match byte by byte, and then U+009B's two bytes (0xC2 0x9B) are not controls, as under `C`. Measured on 7569ffa with bash 5.2.21: under `LC_ALL=C.UTF-8` and `$OPENREPOSHAPE_BIN_DIR` set to `<dir>/caf`, 0xE9, U+009B, `31mFORGED/bin`, `openRepoShape --install` exited 0, placed the command and printed the value raw in four lines; install.sh, given the same value after a `:`, exited 2 with the colon refusal naming it, the C1 control in the line. Both files now run the check as `(LC_ALL=C; case ... *[[:cntrl:]]* | *$'\302'[$'\200'-$'\237']* ...)`: the ASCII controls and DEL byte by byte, and the two bytes of each C1 control named, so the answer is the same in every locale. Every value measured on 7569ffa and refused only under C.UTF-8, or not at all, is refused with exit 2, nothing placed, nothing echoed, and install.sh makes no request. `café` and U+00A0 are still placed. U+2028 and U+2029, which only a UTF-8 locale's `[[:cntrl:]]` matched, are now placed in every locale, and a test says so. Both comments say what the check does, and no longer say the C1 controls are the locale's. Tests: the C1 case, which skipped where no UTF-8 locale read U+009B as a control, is now five values (U+009B, the first and the last of the range, a 0xE9 byte before U+009B, and a colon beside that) in each of `C` and `C.UTF-8`, through the command and through install.sh. The placed-letter test gains U+00A0 (the byte after the range's last), U+2028 and U+2029. 18 of the 36 fail on 7569ffa's code and the other 18 are guards; 18 mutants of the two code hunks are caught. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he stacked base of #837
…holding a bidi control, a character of no width, a soft hyphen or U+2028 before any request and name it by its code point, so no line about `$PATH` shows the directory as another (#837) #804's check refuses the C0 and C1 controls and DEL, and passed every character a terminal draws as something other than itself, under `C` and `C.UTF-8` alike: the command placed itself under `<dir>/a` U+202E `forged/bin` and printed the value in four lines, and install.sh made its request. Laid out as a terminal lays out a line (fribidi), `~/` U+202E `.local/bin` reads `~/nib/lacol.`, `/opt/tools-1` U+200F `/2/bin` reads `/opt/tools-12//bin`, and `~/.local` U+200B `/bin` reads `~/.local/bin`. Both files now refuse, right after the control check, before the drive spelling, the colon, the symlink check and any request, a directory holding one of 29 code points, `MISDRAWN_CHARACTERS` in the launcher: U+00AD; the twelve bidirectional controls (U+061C, U+200E, U+200F, U+202A to U+202E, U+2066 to U+2069), as `repo_shape.spelled` escapes them (#391); U+2028 and U+2029, which end a line for every reader that keeps Unicode's line breaks, and which #804 left to a ruling; and U+200B, U+FEFF and U+2060 to U+206F, of no width, the whole block default-ignorable. U+200C and U+200D stay allowed: Persian, the Indic scripts and emoji sequences need them. So does every space. `misdrawn_characters` builds each one's UTF-8 from its code point with printf's octal escapes and looks for the bytes under `LC_ALL=C`, in the subshell #804's check uses, so one list says what is refused and what the refusal names, and no byte is read back as a number. The refusal says "holds a character a terminal draws as something other than itself (U+202E): one that is invisible, ends a line or reorders the text around it, so no line about $PATH can print the name as it reads", names every one found, and nothing else of the value. install.sh carries the list and the function as the launcher has them, and a test holds the two to one text. Tests: #804's C1 locale test in each file now runs `MISDRAWN_BIN_DIRS` too (each code point, two at once, after a byte that is not UTF-8, beside a colon, in a drive spelling), under `C` and `C.UTF-8`, through the command and install.sh, with no request; the link test takes U+202E; the lifted function names exactly its 29 over every code point that shares a first byte with them (896, on stdin, so it runs under Git Bash); `NOT_CONTROLS` drops U+2028 and U+2029 and gains a Persian word with U+200C, an emoji sequence with U+200D, U+2026 and U+FF21. Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideThe PR extends both the launcher and installer with an identical, Bash-compatible UTF-8 byte detector for 41 Unicode characters that terminals may render invisibly, misleadingly, or as line breaks, refusing affected directories and repository/ref-related inputs before requests or output can expose them. Tests comprehensively verify code-point coverage, supplementary-plane encoding, refusal ordering, locale independence, no-echo and no-side-effect guarantees, allowed Unicode exceptions, and installer/launcher parity. Sequence diagram for preflight refusal of misleading Unicode inputsequenceDiagram
participant User
participant EntryPoint
participant Detector as misdrawn_characters
participant Network
participant Filesystem
User->>EntryPoint: invoke installer or launcher
EntryPoint->>Detector: inspect repository, ref, option, or bin directory
Detector-->>EntryPoint: matching code points or empty result
alt misdrawn character found
EntryPoint-->>User: refuse with U+XXXX code point
Note over EntryPoint,User: no value echo, request, checkout, placement, or PATH output
else input is safe
EntryPoint->>Network: request release or repository data
EntryPoint->>Filesystem: checkout or install command
end
Flow diagram for Unicode-safe installation preflightflowchart TD
Start["Receive installation input"] --> Control["Check control characters"]
Control -->|control found| ControlRefuse["Refuse as control character"]
Control -->|no control| Misdrawn["Run misdrawn_characters under LC_ALL=C"]
Misdrawn -->|character found| MisdrawnRefuse["Refuse and report U+XXXX only"]
Misdrawn -->|none found| Shape["Run existing value and path validation"]
Shape -->|valid| Proceed["Make request or install"]
Shape -->|invalid| ShapeRefuse["Refuse before naming the value"]
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
Author
|
(lane openRepoShape-2, 2026-10-17T19:16:51Z) Independent review of #841 (#837) at I reviewed the one commit after the stacked merge: Findings
Checked and true
Could not verify
Lane: openreposhape-2 (openRepoShape-2) |
…tion and the redirect's location are refused unechoed for a misdrawn character too, the set takes in U+FFF9 to U+FFFB, U+1D173 to U+1D17A and U+E0001, and the refusal says a character may be drawn as another (#837) Fix round 1 on rv841's review of #841. The set (SHOULD 1). `MISDRAWN_CHARACTERS` gains FFF9-FFFB (the interlinear annotation characters), 1D173-1D17A (the musical beam, tie, slur and phrase controls) and E0001 (the deprecated language tag): format characters of no width (glibc `wcwidth` 0) that nothing needs, which the command placed and printed in four lines. The encoder builds the four-byte form too, with bash 3.2's arithmetic and `%b`'s `\0NNN`. Every other format character is left out for a reason the comment now gives: U+200C, U+200D and U+180E, which Persian, the Indic scripts, emoji and Mongolian need; the Egyptian hieroglyph and shorthand format controls, with which those scripts build their signs; the tags, the variation selectors, U+034F and U+1160, drawn as part of the character before them; the signs drawn over the digits after them, which take a cell; and every space and the blank Hangul fillers. The repository and the ref (SHOULD 2). #837's line named the `[![:print:]]` checks of both files: a UTF-8 locale's U+202E, U+200B, U+FEFF or U+00AD passes them, and the refusal after them echoed the value. Each now asks `misdrawn_characters` first, and refuses by the code point, saying nothing else of the value: `checked_repo`, `checked_version` (the version named and the newest release's tag), the option check and `latest_from_redirect`'s location in install.sh; `shape_checkout` and `ref_fault` (`$OPENREPOSHAPE_REF` and the record) in the command. First, because `[![:print:]]` is the locale's, so the words do not change with it. The words (NIT 3): "one that is invisible, is drawn as another character, ends a line or reorders the text around it", because U+00AD is drawn as a hyphen. The order (NIT 5): a tab before U+202E is refused as a control character, and a row holds it. Tests: `MISDRAWN` and the locale table take the twelve new code points and the tab row; the lifted sweep takes U+1D100 to U+1D1FF, U+E0000 to U+E007F and a sample of the planes past U+FFFF; `NOT_CONTROLS` takes a subdivision flag (tags U+E0020 to U+E007F); a test of each file refuses a value holding U+202E at every new site under `C` and `C.UTF-8`, through both entry points, with no request, nothing placed and neither stream holding it. Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
(lane openRepoShape-2, 2026-10-17T20:42:17Z) Fix round 1 on #841 at
Lane: openreposhape-2 (openRepoShape-2) |
brettheap
changed the title
$OPENREPOSHAPE_BIN_DIR holding a bidi control, a character of no width, a soft hyphen or U+2028 before any request and name it by its code point, so no line about $PATH shows the directory as another$OPENREPOSHAPE_BIN_DIR, the repository or a ref holding a bidi control, a character of no width, a soft hyphen or U+2028 is refused by its code point before any request and never echoed, so no line shows the value as another
Contributor
Author
|
(lane openRepoShape-2, 2026-10-17T21:03:57Z) Re-verification of #841 (#837) at I reviewed Findings from the first review
New
Ruling on the gaps the writer left alone: a follow-up issue, not #841.
Checked and true
Could not verify
Lane: openreposhape-2 (openRepoShape-2) |
3 tasks
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="install.sh" line_range="369" />
<code_context>
+# `main` asks it of the directory, of an option, `checked_repo` of the
+# repository, `checked_version` of a version and `latest_from_redirect`
+# of the location it reads, each before any line prints the value.
+MISDRAWN_CHARACTERS=(00AD 061C 200B 200E-200F 2028-202E 2060-206F FEFF FFF9-FFFB 1D173-1D17A E0001)
+
+misdrawn_characters() {
</code_context>
<issue_to_address>
**Stray emoji tags remain invisible**
When a binary directory contains emoji tag characters outside a valid subdivision-flag sequence, the byte check excludes the emoji tag characters everywhere, not just inside a valid subdivision-flag sequence. A directory name with tag characters inserted between ordinary letters is therefore accepted even though terminals that ignore those tags display a different name, so the PATH note can still identify the wrong directory.
Recognize and allow tags only as part of a valid subdivision-flag sequence, and refuse tag characters elsewhere.
</issue_to_address>
### Comment 2
<location path="install.sh" line_range="390" />
<code_context>
+ fi
+ printf -v bytes '%b' "$bytes"
+ case "$value" in
+ *"$bytes"*)
+ printf -v name 'U+%04X' "$code"
+ found="${found:+$found, }$name"
</code_context>
<issue_to_address>
**Valid legacy-encoding names are refused**
When the directory name is encoded in a multibyte non-UTF-8 locale and its bytes overlap a listed character’s UTF-8 encoding, `misdrawn_characters` matches UTF-8 byte sequences without decoding the input, so a valid character in a multibyte non-UTF-8 locale whose encoding overlaps one of those sequences is reported as a forbidden code point and the installer refuses that directory. The launcher carries the same byte-matching logic, so it refuses the same names.
Decode and classify the value according to an explicitly supported input encoding, or limit this byte-sequence refusal to UTF-8 inputs and document the behavior for other encodings.
</issue_to_address>Sourcery assessment
Approval pending. 2 findings to address first.
Blocking findings: install.sh:369, install.sh:390
| # `main` asks it of the directory, of an option, `checked_repo` of the | ||
| # repository, `checked_version` of a version and `latest_from_redirect` | ||
| # of the location it reads, each before any line prints the value. | ||
| MISDRAWN_CHARACTERS=(00AD 061C 200B 200E-200F 2028-202E 2060-206F FEFF FFF9-FFFB 1D173-1D17A E0001) |
There was a problem hiding this comment.
🟡 Medium · Stray emoji tags remain invisible
When a binary directory contains emoji tag characters outside a valid subdivision-flag sequence, the byte check excludes the emoji tag characters everywhere, not just inside a valid subdivision-flag sequence. A directory name with tag characters inserted between ordinary letters is therefore accepted even though terminals that ignore those tags display a different name, so the PATH note can still identify the wrong directory.
Recognize and allow tags only as part of a valid subdivision-flag sequence, and refuse tag characters elsewhere.
Prompt for AI agents
In `install.sh` at line 369:
**Stray emoji tags remain invisible**
When a binary directory contains emoji tag characters outside a valid subdivision-flag sequence, the byte check excludes the emoji tag characters everywhere, not just inside a valid subdivision-flag sequence. A directory name with tag characters inserted between ordinary letters is therefore accepted even though terminals that ignore those tags display a different name, so the PATH note can still identify the wrong directory.
Recognize and allow tags only as part of a valid subdivision-flag sequence, and refuse tag characters elsewhere.| fi | ||
| printf -v bytes '%b' "$bytes" | ||
| case "$value" in | ||
| *"$bytes"*) |
There was a problem hiding this comment.
🟡 Medium · Valid legacy-encoding names are refused
When the directory name is encoded in a multibyte non-UTF-8 locale and its bytes overlap a listed character’s UTF-8 encoding, misdrawn_characters matches UTF-8 byte sequences without decoding the input, so a valid character in a multibyte non-UTF-8 locale whose encoding overlaps one of those sequences is reported as a forbidden code point and the installer refuses that directory. The launcher carries the same byte-matching logic, so it refuses the same names.
Decode and classify the value according to an explicitly supported input encoding, or limit this byte-sequence refusal to UTF-8 inputs and document the behavior for other encodings.
Prompt for AI agents
In `install.sh` at line 390:
**Valid legacy-encoding names are refused**
When the directory name is encoded in a multibyte non-UTF-8 locale and its bytes overlap a listed character’s UTF-8 encoding, `misdrawn_characters` matches UTF-8 byte sequences without decoding the input, so a valid character in a multibyte non-UTF-8 locale whose encoding overlaps one of those sequences is reported as a forbidden code point and the installer refuses that directory. The launcher carries the same byte-matching logic, so it refuses the same names.
Decode and classify the value according to an explicitly supported input encoding, or limit this byte-sequence refusal to UTF-8 inputs and document the behavior for other encodings.
This was referenced Oct 18, 2026
brettheap
pushed a commit
that referenced
this pull request
Oct 18, 2026
…trap_after_bump.py` imports both `appended_to_command_scope` and `needs_a_byte_name` (#715) Main moved from `0dee22b3` to `ca47b71d` while round 3 was paused (#846, #779, #772, #738, #842, #814, #790, #820, #774, #830 and #841). `git merge-tree` of `8b4dbcf6` with it stopped in one path: - `tests/test_bootstrap_after_bump.py`, its imports. Main's `aec3b38f` (tests append what they tell git to the command scope) added `appended_to_command_scope` to the `from conftest import (...)` list. This branch added `from test_bootstrap_spelling import needs_a_byte_name` below it (#715, round 1). Both are kept. `scaffold-project.py`, `scripts/bump-leg.py` and `scripts/shape_materialize.py` merged without a conflict. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 18, 2026
brettheap
added a commit
that referenced
this pull request
Oct 18, 2026
…cked once, before `--version` answers and before any fetch, so a control character or a character a terminal draws as another in either is refused, never printed or sent (#866) (#872) `--version` printed both values as typed and exited 0. A scaffold run sent its fetch with them, and so did `--install` from stdin, and each named them three times in its refusal. So an escape, a C1 control, U+202E or U+200B in either value reached the terminal as it was typed. One function, `refuse_bad_values`, now runs once the arguments are read, and inside the `--version` case, which answers within the argument loop. It refuses the broken record first, as before. Then it checks `$OPENREPOSHAPE_REPO` with `checked_repo`, `shape_checkout`'s guard moved whole: #841's misdrawn characters, then a leading `-` or `[![:print:]]` on the clone URL. Then it checks `$OPENREPOSHAPE_REF` with `ref_character_fault`, `ref_fault`'s misdrawn and control-character checks, moved whole. `ref_fault` calls them in the same place. Every refusal keeps the words it had. The misdrawn characters are looked for in `$REPO` as typed, and the control characters in it and in the clone URL, because `$(shape_clone_url)` drops a trailing newline: a URL, an scp address or a path ending in one passed, was printed by `--version` and was sent by a scaffold run and by `--install` from stdin (rv866's SHOULD-1). The leading `-` is still looked for in the URL's first character, so an owner/name such as `-acme/r` passes as before. A test holds the order the checks state: the record, then the repository, then the ref. The ref's length and shape stay where a ref is used: `checked_ref` when `--install` records, `shape_checkout` before `git checkout`, and setup.sh's `--shape-ref` checks. So `--version` still prints any printable ref in force, and an install that records no ref still takes one that names nothing, as workBenches sets it. `--version` refuses with exit 2, as a broken record already makes it do, and does not respell the value. install.sh's `shown=` parse is unchanged for every value it can hand the command: an owner/name that its own `checked_repo` passed, and the tag that its `checked_version` passed. Both pass this check, and `--version` prints them byte for byte. `--help` still answers first. It names the variables, never their values. Closes #866 Lane: openreposhape-2 (openRepoShape-2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lane: openreposhape-2 (openRepoShape-2)
Claim: #837 (comment)
Stacked on
$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 (install.sh and openRepoShape: a$OPENREPOSHAPE_BIN_DIRholding:or a newline is refused byinstall.shonly after its fetch, and one holding a carriage return or an escape is placed and echoed as typed, so both refuse every control character before they fetch or print anything #760),fix/760-bin-dir-control-chars, has landed as2e476a77. Its final head7e4181c0fb2bb6b2e6161ee704139e9645297a74was squash-merged to main at 2026-10-17T22:24:53Z, andgit diff 7e4181c0 2e476a77over install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's four files (install.sh,openRepoShape,tests/test_install_sh.py,tests/test_openreposhape_command.py) is empty.merge --no-ffonto origin/maind66a9054as8a559cb5, "Merge lane 5's install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 head 7e4181c (fix/760-bin-dir-control-chars) as the stacked base of install.sh and openRepoShape: a$OPENREPOSHAPE_BIN_DIRholding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837". No conflict; the merge brings in install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's four files and nothing else.$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 landed, the branch merged the squash, then main:eef556b4, "Merge lane 5's install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 squash 2e476a7 (its final head 7e4181c is merged at 8a559cb)". It conflicted in install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's four files and nowhere else. Each was taken as this branch's side, because install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's side there is7e4181c0's text, which8a559cb5already holds.f2483cf1, "Merge origin/main (677f9bf) into fix/837-bin-dir-format-chars after install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 landed as 2e476a7". No conflict.git diff 4123b9e5 f2483cf1over them is empty.git diff origin/main f2483cf1names only the four, and it is byte for bytegit diff 8a559cb5 4123b9e5.8a559cb5:cc5ae88e(the change) and4123b9e5(fix round 1, on rv841's review). The two merges after them are the restack.f2483cf1:test_install_sh.py,test_openreposhape_command.py,test_install_sh_filter_repo.py,test_openreposhape_package.py,test_repo_hygiene.py,test_cli_reference.py,test_agents_md_codes.pyandtest_readme_windows_quoting_sentence.pygive 737 passed.test_shape_doctor.pygives 436 passed, 3 skipped, 1 xfailed.render-cli-reference.py --checkis ok.Fix round 1, on rv841's review (READY, 2 SHOULDs, 4 NITs)
MISDRAWN. The encoder builds the four-byte form as well (lead byte F0 to F4, then three continuation bytes, with bash 3.2's arithmetic and%b's\0NNN). The comment says which forms it builds. I checked every other format character by the same rule, and the comment gives each one left out its reason (table below)[![:print:]]check in both files now asksmisdrawn_charactersfirst and refuses by the code point, without echoing the value. In install.sh that ischecked_repo,checked_version(the version named, and the newest release's tag), the option check andlatest_from_redirect's location. In the command it isshape_checkout(the repository) andref_fault($OPENREPOSHAPE_REFand the record). Tested underCandC.UTF-8, through both entry pointsWhat changed
The goal is #837's first line: refuse, before any request and before any line names it, a
$OPENREPOSHAPE_BIN_DIRholding a character a terminal draws as something other than itself. #804's check (*[[:cntrl:]]*and the C1 bytes, underLC_ALL=C) passed every such character underCandC.UTF-8alike. The command then placed itself under<dir>/aU+202Eforged/bin, exit 0, and printed the value in four lines (installed at,placed in, the PATH note, theexport PATHline), and install.sh made its request.#837's own line also named
checked_repoand the other[![:print:]]checks of both files (w760's report): a UTF-8 locale's U+202E passes[![:print:]], and the refusal after it echoed the value. Fix round 1 closes those too, soCloses #837closes the whole record.Refused, not printed with
LC_ALL=C printf '%q'. UnderLC_ALL=C,%qprints every byte past ASCII as octal, so a directory calledcafé, or one named in Persian or Hindi, would print as$'caf\303\251'. It is bash's own spelling, and bash 3.2's is unmeasured. Theexport PATHline would carry$'…', which no POSIXshreads. And every line naming the directory would change, for every non-ASCII name. The value is the person's own variable, so a refusal costs them one rename, and it is what #804 does for a control character.What a terminal draws (fribidi,
libfribidi.so.0through ctypes, each name in an LTR paragraph as a terminal line is, the zero-width characters dropped):The set: 41 code points,
MISDRAWN_CHARACTERS=(00AD 061C 200B 200E-200F 2028-202E 2060-206F FEFF FFF9-FFFB 1D173-1D17A E0001)in the launcher, copied as it is into install.sh.wcwidthbelow is glibc's, underC.UTF-8.repo_shape.spelled(#391) escapes them, andunwritable_reason(#255) refuses themwcwidth0), and as strong directional characters they reorder the neutral characters beside them:/opt/tools-12//binabove. #391 escapes all twelve bidi controls for this reason; #255 refuses only the ninestr.splitlinesamong them, andwcwidthgives -1 for both. #391 and #255 both treat them as line breaks; #804 left them "refused in none", open to a ruling.NOT_CONTROLSloses its two rows for themwcwidth0, or -1 for the unassigned U+2065):~/.local<U+200B>/binis drawn as the name of another directory. perl's\p{Default_Ignorable_Code_Point}holds for every code point U+2060 to U+206F, so the range has no gapwcwidth0 that nothing needs; the command placed and printed them beforewcwidth1) and as nothing elsewhere, never as what it isNOT_CONTROLS(the first two)wcwidth0, but those scripts build their signs with themNOT_CONTROLS), a variation selector chooses how that character is drawn, U+034F keeps a grapheme whole, and U+1160 forms a syllable block with the consonant before itwcwidth1, drawn as themselveswcwidth1 or 2)How it matches.
misdrawn_charactersbuilds each character's UTF-8 from its code point with printf's octal escapes (printf -v bytes '\\0%o…', then'%b', which POSIX defines): two bytes below U+0800, three below U+10000, four above. It looks for those bytes in the value underLC_ALL=C, in the subshell #804's check uses, and names every one it finds asU+XXXX, in the list's order. So one list says what is refused and what the refusal names, and no byte is read back as a number. Nothing in it is newer than bash 3.1 by the manual (printf -vcame in 3.1); bash 3.2 itself was not run. Swept against Python at4123b9e5, the function lifted from the file,/tmp/a+ the character +b:C, 0 underC.UTF-8, and each of the 41 is still named behind a lone 0xE9.Where.
$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's control check. That is before the drive spelling, the colon (whose refusal names the directory), the symlink check (whose refusal names the link) and any request.[![:print:]]check of each, because that class is the locale's. UnderCit matches every byte past ASCII, and under a UTF-8 locale none of these characters, so after it the words would change with the locale. Every one of these refusals comes before any line that prints the value.The refusals, measured by a probe (
<H>stands for the probe's directory;ghstub first onPATH,GH_TOKEN=bogus, emptyGH_CONFIG_DIR, acurlstub that logs and exits 22,LC_ALL=C.UTF-8, U+202E shown as<U+202E>):Before (the stacked base),
OPENREPOSHAPE_BIN_DIR=<H>/a<U+202E>forged/bin openRepoShape --install, exit 0, placed:and
install.sh v9.9.9made one request (exit 2 only because the stubcurlfailed it). Now, exit 2, nothing placed, nothing on stdout, no request:The repository and the ref, at
cc5ae88e(each exit 2, the value echoed) and at4123b9e5(exit 2, not echoed):The command's
--doctorrefuses such a repository with "OPENREPOSHAPE_REPO holds a character … around it. This tool will not put it on agit clonecommand line. Set it to a URL or a path.", before anygitruns. A redirect to a location holding one gives "it redirects to a location that names no release". Two characters at once are named(U+200B, U+202E). UnderCevery one of these words is the same. A Persian name with U+200C, an emoji sequence with U+200D and a subdivision flag (tags) pass through both files.Also: install.sh's header (exit status 2) names #837. #804's comment in both files no longer says U+2028 and U+2029 "are refused in none".
export_path_line's comment names all three refusals.Tests
tests/test_openreposhape_command.pytest_a_bin_dir_is_refused_in_every_locale_whatever_else_it_holds(was #804's C1 test; now overREFUSED_IN_EVERY_LOCALE):, one in a drive spelling; a tab before U+202E, refused as a control character (NIT 5); and #804's five C1 rows. Each underCandC.UTF-8: exit 2, the words with the code points, nothing placed, noforgedtail and no character ofMISDRAWNin either streamtests/test_install_sh.pytest_the_installer_refuses_in_every_locale_before_any_request(was #804's C1 test)curlandghrecord no call, nothing placed,$TMPDIRemptytests/test_openreposhape_command.pytest_a_ref_or_repository_drawn_as_another_is_refused_unechoed$OPENREPOSHAPE_REFfor--installto record, the record beside an installed command (--version),$OPENREPOSHAPE_REPOfor--doctorto clone (agitthat leaves a mark if it runs): underCandC.UTF-8, exit 2, the words, nothing placed, no clone, neither stream holds the valuetests/test_install_sh.pytest_the_installer_refuses_a_value_drawn_as_another_unechoedCandC.UTF-8, exit 2, nothing placed, nothing fetched, no request at all where the value is not the network's, neither stream holds the valuetests/test_openreposhape_command.pytest_misdrawn_characters_names_its_own_and_nothing_else[openRepoShape, install.sh]LC_ALL=C, over every code point that shares a first byte with one of the 41 (U+0080 to U+00BF, U+0600 to U+063F, U+2000 to U+20FF, U+FE00 to U+FFFF, U+1D100 to U+1D1FF, U+E0000 to U+E007F) and a sample of the planes past U+FFFF (the Kaithi, Egyptian and shorthand format characters among it): it names exactly the 41, and nothing else. The names go on stdin as bytes, so this one also runs under Git Bashtests/test_openreposhape_command.pytest_install_sh_carries_the_launchers_misdrawn_check_as_it_istest_a_bin_dir_is_refused_before_a_link_in_it_names_it,test_a_bin_dir_is_refused_before_the_symlink_check_names_it(#804's, now overLINKED_BIN_DIRS)test_a_bin_dir_with_an_unprinted_letter_is_placed,test_the_installer_places_a_directory_with_an_unprinted_letter(#804's, overNOT_CONTROLS)NOT_CONTROLSdrops U+2028 and U+2029, and gains a Persian word with U+200C, an emoji sequence with U+200D, U+2026 (first byte0xE2), U+FF21 (first byte0xEF) and England's subdivision flag (tags, first bytes0xF3 0xA0, as U+E0001's): placed, under both locales, through both filestests/test_openreposhape_command.pyassert_refused_saying(shared by every refusal test above)MISDRAWNand noforgedtail in either streamThe new and changed cases (
-k "every_locale or misdrawn or unprinted_letter or link_in_it or symlink_check_names or drawn_as_another or record_that_is_not") give 260 passed at4123b9e5.cc5ae88e's two shell files: 199 failed, 61 passed.Cpasses, becausecc5ae88e's[![:print:]]already hid it.cc5ae88erefused them already, without "is drawn as another character".cc5ae88e's own tests gave 137 failed, 46 passed.Mutants. Each one is a single edit in a
git archivecopy of the head, run against the selected cases after an unmutated control. List and function mutants are applied to both files alike, so the identity test is not what kills them. The runner, the mutants, the run logs and the kill tables are in this PR's attachments, undermutants/; the probes are underprobes/.Fix round 1, 32 mutants, all killed (control: 260 passed):
0xE0, lead>> 17, second byte& 31, last byte& 31(B01-B05)the_repository(after the control case: underC, where[![:print:]]answers in other words)a_version,the_newest_tagan_option;a_locationunderC.UTF-8a_repository_to_clonereturn 0dropped so the shape refusal follows (S11, S12)a_ref_to_record,a_recorded_refa-tab-then-u+202e, both localescc5ae88e(W01)cc5ae88e's 50 mutants: 48 killed, 2 survived. The survivors areLC_ALL=Cdropped from the new call, in either file. That is equivalent inCandC.UTF-8, and in the other locales rv841 tried. It would differ only in a multibyte locale that is not UTF-8, and none could be built here. Their table ismutants/kills.txt.Runs (Linux, bash 5.2.21, Python 3.12.3;
env -u GH_TOKEN -u GITHUB_TOKEN GH_CONFIG_DIR=<empty> PATH=<a gh stub that exits 97>:… /usr/bin/python3 -m pytest -q -p no:cacheprovider, one basetemp,df -ichecked first):8a559cb5cc5ae88e4123b9e5test_openreposhape_command.py,test_install_sh.py,test_install_sh_filter_repo.py,test_openreposhape_package.pyf2483cf1(#804's squash2e476a77and main677f9bfamerged; the four files are4123b9e5's, byte for byte)test_repo_hygiene.py,test_cli_reference.py,test_agents_md_codes.pytest_shape_doctor.py(it runs the launcher's--doctor),test_release_workflow.py,test_doctor_validator_rows.py,test_doctor_machine_row.pytests-git-bashruns, everyPath.write_textwriting CRLF (a plugin; a probe test passes with it and fails without it)render-cli-reference.py --check: ok.Not changed
$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's control-character refusal and its words, and its order. A value holding a control character and U+202E is refused as a control character, and a test now holds that.--help, README, the handbook,AGENTS.md,docs/cli.md: no sentence states these refusals.$REPOand$REF(a candidate; not install.sh and openRepoShape: a$OPENREPOSHAPE_BIN_DIRholding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837's record, which names the[![:print:]]checks).openRepoShape --versionprints both with no check at all, and so do two fetch refusals:--version(openRepoShape:1003at4123b9e5): measured withOPENREPOSHAPE_REF=v1ESC[31mX, it printsv1 033 [ 3 1 m Xand exits 0. U+202E and an ESC in$OPENREPOSHAPE_REPObehave the same.fetch_or_die(:235) andcollect_commands(:301), which name$REPOand$REFwhen a fetch fails.checked_reforshape_checkout.$OPENREPOSHAPE_BIN_DIRholding a bidi override, U+200B, U+2028 or U+FEFF passes #804's control-character check and is echoed as typed, so the PATH note can show a directory as another #837's. Closing it means checking both values up front, before--versionanswers.$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804 found for the C1 bytes, a byte string that is one of the 41 in UTF-8 can be other text in such a locale, and it is refused all the same. Not measured: no such locale can be built here.$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's do, but the lifted test does not), any real terminal emulator.$OPENREPOSHAPE_BIN_DIRis respelled in install.sh's own lines and refused before its fetch, and the PATH note quotes the directory it prints (#649's two leftovers) #667's drivecaseand install.sh and openRepoShape: both refuse a$OPENREPOSHAPE_BIN_DIRholding a control character before they name it, and install.sh refuses a:before its fetch, so a carriage return or an escape is no longer echoed as typed (#760) #804's controlcaseare. If the new-code duplication gate counts shell, those lines are what it counts.Closes #837
🤖 Generated with Claude Code
Summary by Sourcery
Refuse terminal-misdrawn Unicode characters in installer inputs and paths before they can trigger requests, filesystem changes, or misleading output.
Bug Fixes:
Enhancements:
Tests: