Skip to content

tests/test_bom_tool_io.py: the byte-order-mark rows hold each tool's own read and write and that only U+FEFF is cut, so a utf-8-sig read that drops the mark and a cut that also takes spaces no longer survive (#1176) - #1188

Merged
brettheap merged 3 commits into
mainfrom
l3-1176-bom-rows-hold-the-tools
Oct 23, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changed

A follow-up outside v1.2.0 (Brett Heap, 2026-10-21T10:32:04Z). It lands only after the v1.2.0 tag, on a later merge word.

One new file, tests/test_bom_tool_io.py (126 rows), and no tool code. #1050's byte-order-mark rows (tests/test_bom_rewrites.py) hold what each line rewriter returns. They do not hold the tools' own read and write, nor that the cut in front of line 1 takes U+FEFF and nothing else, so rv1050's five mutants each passed every file that reads a pin (#1176).

  • The tools' own read and write. update-shape.py's _repin (the shape pin and the manifest, through the Rollback ledger) and bump-leg.py's _leg_pin_text, rewrite_pin and write_lf (in the order its main calls them) run over a file that starts with 0, 1 or 2 marks. The file written starts with exactly the marks it was read with, and every byte after them is the rewrite. The marks are bytes (codecs.BOM_UTF8), never typed.
  • Each tool end to end. update-shape.py apply (a local --upstream only, never the default GitHub one) on a root whose pin and manifest are committed with the marks, and bump-leg.py on a leg pin with the marks: both write the marks back, and the project's own gate is green after.
  • Only U+FEFF is cut, and only in front of line 1. Another zero-width or format (Cf) character as the second character (ZWSP, ZWNJ, ZWJ, word joiner, soft hyphen, LRM) stays: in front of the key the file is refused unchanged in each tool's words, and in front of a key of its own it is written back where it was. A space or a tab in front of the key (with no mark, behind a mark and in front of one) and a file indented two spaces on every line are refused as before update-shape.py and scripts/bump-leg.py: the three line rewriters find a first-line key after a byte-order mark, as every reader does, and write the mark back as it was (#933) #1050. A mark that is the second character of a comment is text and is written back.
  • No fault found. Every row passes at main, so no tool changes. The issue's own measurement holds: the tools keep the mark today, and only a test was missing.

Measured at main (961640c6)

  • Before: tests/test_bom_rewrites.py + tests/test_bom_reads.py = 37 passed. rv1050's five mutants (utf-8-sig at _repin's two reads and at _leg_pin_text's read; an lstrip that also cuts spaces and line feeds in _marks_off and in rewrite_pin) each pass the six files of the issue (272 passed under each).
  • After: tests/test_bom_tool_io.py = 126 passed in about 13 s. Each of the five fails 5 to 20 of its rows (table below), and every row passes without them.

Mutants

Each is one exact-once edit, run against the 3 or 6 existing files (the BOM files, plus the files that read a pin for M1 to M5 and M19 to M22) and against the new file. The tool file is restored from a saved copy after each and checked with cmp (22 of 22 cmp exit 0; git diff of the tools against HEAD is empty). Runner: W1176's own mutants.py, in the lane's scratch.

# Mutant Existing files New rows
M1 _repin reads the shape pin as utf-8-sig survives (272 passed) KILLED (20 failed)
M2 _repin reads the manifest as utf-8-sig survives (272 passed) KILLED (20 failed)
M3 _leg_pin_text reads the leg pin as utf-8-sig survives (272 passed) KILLED (20 failed)
M4 _marks_off: lstrip also cuts spaces and line feeds survives (272 passed) KILLED (10 failed)
M5 rewrite_pin: lstrip also cuts spaces and line feeds survives (272 passed) KILLED (5 failed)
M6 _marks_off: cut widened to every Cf character survives (152 passed) KILLED (24 failed)
M7 rewrite_pin: cut widened to every Cf character survives (152 passed) KILLED (12 failed)
M8 _marks_off: cut left out killed (4 failed) KILLED (6 failed)
M9 rewrite_pin: cut left out killed (2 failed) KILLED (4 failed)
M10 _marks_off: cut also applied to the second character survives (152 passed) KILLED (6 failed)
M11 rewrite_pin: cut also applied to the second character survives (152 passed) KILLED (3 failed)
M12 _marks_off: cut takes one mark only killed (2 failed) KILLED (3 failed)
M13 rewrite_pin: cut takes one mark only killed (1 failed) KILLED (2 failed)
M14 _marks_off: cut takes every mark anywhere killed (2 failed) KILLED (6 failed)
M15 rewrite_pin: cut takes every mark anywhere killed (1 failed) KILLED (3 failed)
M16 _repin writes the shape pin as utf-8-sig survives (152 passed) KILLED (22 failed)
M17 _repin writes the manifest as utf-8-sig survives (152 passed) KILLED (22 failed)
M18 write_lf (bump-leg.py's write) as utf-8-sig killed (1 failed) KILLED (22 failed)
M19 _marks_off: a space or a tab in front of the marks is cut first (lstrip(" \t"), then the marks) survives (272 passed) KILLED (10 failed)
M20 rewrite_pin: the same cut, in front of the marks, first survives (272 passed) KILLED (5 failed)
M21 _marks_off: a space or a tab is cut only where a mark follows it survives (272 passed) KILLED (4 failed, the rows with the lead in front of a mark)
M22 rewrite_pin: the same, only where a mark follows survives (272 passed) KILLED (2 failed, the same rows)

Tests

Under a stub gh (exit 97), GH_TOKEN=bogus, an empty GH_CONFIG_DIR, GIT_SSH_COMMAND=false, a scratch GIT_CONFIG_GLOBAL and EMAIL=tests@openreposhape.invalid, one pytest at a time. The two lines below are the first two heads', with 120 rows; the third head's gates, with 126, are in Fix round 2:

  • tests/test_bom_tool_io.py tests/test_bom_rewrites.py tests/test_bom_reads.py tests/test_pin_rewrite_layout.py tests/test_repo_hygiene.py tests/test_path_guard_hygiene.py tests/test_no_auto_maintenance.py tests/test_agents_md_codes.py tests/test_cli_reference.py: 550 passed, 2 skipped in 87.26s.
  • tests/test_update_shape.py tests/test_bump_leg.py tests/test_update_shape_apply_lines_and_headers.py tests/test_update_shape_header_comments.py tests/test_bump_leg_dry_run_truth.py (the helpers the new file imports, and the files the issue names): 264 passed, 1 skipped in 200.18s.
  • tests/test_shape_doctor.py is not run: no shape-doctor.py or helper changes.

Merges

Re-listed live on 2026-10-23 at about 07:52Z and again at 08:21Z (the same eleven PRs at the same heads), and run at d65340e5b: git merge-tree --write-tree of this head is clean (exit 0) with origin/main (961640c6, frozen until the v1.2.0 tag) and with each of the eleven open PRs that touch update-shape.py, scripts/bump-leg.py, scripts/shape_materialize.py or scripts/repo_shape.py:

#1178 no longer touches these files. No open PR touches this file or tests/test_update_shape.py, tests/test_bump_leg.py, tests/test_bom_rewrites.py, tests/test_pin_rewrite_layout.py or tests/conftest.py, which it imports. The rows were not run on a merged tree: a clean merge is a clean merge of text only.

Fix round

CI's full suite on the first head (aa0275be, tests.yml run 39861510476) gave 47 setup errors, all in tests/test_bump_leg.py. tests-git-bash, package and SonarCloud passed.

Fix round 2

RV1188's review of 024084b9 (issuecomment-6360127603): READY, one SHOULD and three NITs. One new commit, tests/test_bom_tool_io.py only; the three tool files are byte-identical to HEAD (cmp against saved copies, exit 0).

  • SHOULD-1, SonarCloud python:S3415 ("Unify assertion argument order in this file"). The one == assert with a tuple on its left, assert (pin.read_bytes(), manifest.read_bytes()) == before, is now now = pin.read_bytes(), manifest.read_bytes() and assert now == before, the shape tests/test_naming_policy.py: bind the compared tuple to a name before asserting it, so Sonar's assertion-order rule stops misreading the literal-looking display as the expected side (#130) #131 (d682127d6) fixed. The name is now, not after, which is a function in this file. A scan of every assert in the file by syntax tree found no other == with the expected value on the left: the in asserts keep the text looked for on the left, as in reads. The local analyzer does not reproduce the rule, so the result is read on SonarCloud once CI has run on this head.
  • NIT-1. A space or a tab in FRONT of a mark is now a row for each of the three files (LEAD_PLACES: no mark, behind a mark, in front of one). Each tool refuses it unchanged by its own code (update-pin-unrecognised, update-manifest-no-shape, bump-leg-pin-unrecognised), as the reader does. Six rows more: 126 in all. The module docstring, the row's docstring and the third bullet above say what the rows hold.
  • NIT-2. The Merges list above is re-listed live, with scripts/repo_shape.py added to the files watched.
  • NIT-3. The two end-to-end refusal rows use FINDERS["shape-pin-commit"].refused and FINDERS["leg-pin-commit"].refused in place of the retyped words.
  • Combined runs, tests/test_bom_tool_io.py, tests/test_bump_leg.py and tests/test_update_shape.py in ONE pytest, stub gh, EMAIL=tests@openreposhape.invalid: 270 passed, 1 skipped in 129.58s in CI's order and 270 passed, 1 skipped in 137.65s reversed. With the three other files the issue names (tests/test_bump_leg_dry_run_truth.py, tests/test_update_shape_apply_lines_and_headers.py, tests/test_update_shape_header_comments.py), six files in ONE pytest: 390 passed, 1 skipped in 218.42s and 390 passed, 1 skipped in 195.71s reversed.
  • Gates, one pytest at a time: the new file with tests/test_bom_rewrites.py, tests/test_bom_reads.py and tests/test_pin_rewrite_layout.py: 278 passed in 21.85s. The hygiene files, each alone: tests/test_repo_hygiene.py 101 passed, tests/test_path_guard_hygiene.py 49 passed, tests/test_no_auto_maintenance.py 58 passed and 2 skipped, tests/test_agents_md_codes.py 47 passed, tests/test_cli_reference.py 23 passed.
  • Mutants, all 18 of mutants.py re-run at this head, each to its end, and four more for the new rows (M19 to M22): 22 of 22 new rows: KILLED, 22 of 22 cmp exit 0, runner exit 0. M4 and M5 fail two and one rows more than before. M21 and M22 cut a space or a tab only where a mark follows it, so only the new rows in front of a mark kill them; the six existing files all pass under both.

Left out of scope

docs/release-notes/v1.2.0.md (it records v1.2.0's state, known limits included, and is not edited). No milestone.

Closes #1176

Lane: openreposhape-3 (openRepoShape-3)

🤖 Generated with Claude Code

…own read and write and that only U+FEFF is cut, so a `utf-8-sig` read that drops the mark and a cut that also takes spaces no longer survive (#1176)

#1050's rows (tests/test_bom_rewrites.py) hold what the three line
rewriters return. They do not hold the tools' own read and write, nor
that the cut in front of line 1 takes U+FEFF and nothing else, so
rv1050's five mutants (`utf-8-sig` at `_repin`'s two reads and at
`_leg_pin_text`'s read, and an `lstrip` that also cuts spaces and line
feeds in `_marks_off` and in `rewrite_pin`) each passed the whole file
set that reads a pin.

The new file runs update-shape.py's `_repin` and bump-leg.py's
`_leg_pin_text`, `rewrite_pin` and `write_lf` over a file that starts
with 0, 1 or 2 marks, and each tool end to end (`update-shape.py apply`
against a local `--upstream`, and `bump-leg.py`). It asserts that the
marks are written back exactly and that every byte after them is the
rewrite, that another zero-width or format character as the second
character stays (refused in front of the key, written back in front of
a key of its own), that a space, a tab or a whole file indented two
spaces is refused as it was before #1050, and that a mark that is the
second character of a comment is text. No tool code changes: no row
showed a fault.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds 120 regression tests covering the complete BOM lifecycle in update-shape.py and bump-leg.py, including byte-preserving reads and writes, narrowly scoped U+FEFF stripping, refusal behavior, and end-to-end command validation; no tool implementation changes are included.

File-Level Changes

Change Details Files
Added comprehensive byte-order-mark I/O coverage for both pin-manipulation tools.
  • Tests preservation of zero, one, and two UTF-8 BOM byte sequences through each tool’s actual read and write path.
  • Covers update-shape.py repinning of the shape pin and manifest, plus bump-leg.py pin rewriting and writing.
  • Verifies end-to-end CLI behavior preserves BOMs and leaves project validation green.
tests/test_bom_tool_io.py
Added regression coverage defining the exact scope and position of BOM removal.
  • Confirms only leading U+FEFF bytes on the first line are removed before key matching.
  • Ensures spaces, tabs, indentation, other Unicode format characters, and later BOMs are preserved or rejected without modifying files.
  • Checks refusals retain original bytes and expected tool-specific error codes.
tests/test_bom_tool_io.py
Strengthened mutation-detection coverage for BOM reads, writes, and stripping behavior without changing production code.
  • Exercises mutants involving utf-8-sig, widened or misplaced stripping, incomplete mark removal, and unconditional BOM insertion.
  • Uses byte-level assertions to distinguish marks from typed U+FEFF text and verify rewritten content exactly.
tests/test_bom_tool_io.py

Assessment against linked issues

Issue Objective Addressed Explanation
#1176 Test each tool’s real read-and-write path to ensure shape pins, manifests, and leg pins retain exactly the one or two UTF-8 BOMs they were read with. ✅
#1176 Add regression coverage that only U+FEFF is removed at the start of line 1, while spaces, tabs, other format characters, and marks elsewhere remain intact or cause the same refusal as before. ✅

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

… the spec leg on a branch of their own, so the full suite no longer pushes `bump-leg-advance` twice and tests/test_bump_leg.py's 47 setup errors are gone (#1176)

CI's full suite on #1188 (tests.yml run 39861510476) gave 47 setup
errors in tests/test_bump_leg.py. This file imported test_bump_leg.py's
module-scoped fixture `advanced`, and a module-scoped fixture runs once
for every module that names it. Both runs push HEAD to the branch
`bump-leg-advance` of the session's one shared remote, so the second
push was refused as non-fast-forward: this file sorts first, and
test_bump_leg.py's own `advanced` was the one refused. The gates run one
file at a time and could not see it.

The file now has its own module-scoped `advanced`, which pushes to
`bom-tool-io-advance`, as the other modules that advance the spec leg do
(test_bump_leg_unreadable.py, test_bump_leg_unopenable.py and others).
Nothing else the file imports writes shared state: `upstream_and_project`
builds in its own tmp paths and `tools` only loads two modules. All 120
rows are kept and the tools are untouched.

tests/test_bom_tool_io.py, tests/test_bump_leg.py and
tests/test_update_shape.py in ONE pytest: 218 passed, 47 errors at
aa0275b, 264 passed, 1 skipped with this change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-23T07:48:07Z) Independent review (RV1188, opus, ruling 8) of #1188 at 024084b98ed8a77ae7ec6fe5513b3032f86dd461: READY. One SHOULD and three NITs; none blocks landing after the v1.2.0 tag.

SHOULD-1, tests/test_bom_tool_io.py:446.

NIT-1, tests/test_bom_tool_io.py:38 and :254-256, and the body's third bullet: "before a mark or after one".

  • The rows hold "no mark" and "after one mark"; no row puts the space or tab in front of a mark.
  • Measured: that case is refused unchanged today by all three tools (update-pin-unrecognised, update-manifest-no-shape, bump-leg-pin-unrecognised), and the reader refuses it too.
  • Add a before-the-mark row, or reword.

NIT-2, the body's Merges.

NIT-3, :445 and :462 retype "has 0 commit line(s)". FINDERS["shape-pin-commit"].refused and FINDERS["leg-pin-commit"].refused hold the same words. Optional.

Observation, nothing for this PR:

  • update-shape.py's _is_block_header (:1695-1698) uses str.strip(), so a no-break space (U+00A0) or an ideographic space (U+3000) in front of shape: is taken as the header. A direct call to _repin wrote such a manifest, while parse_yaml reads that line as another key.
  • This is already on main, is not the byte-order-mark cut, and no row here pins it. It was not run through apply end to end.
  • A follow-up issue is the lane's call.

Checked:

  • Tools: byte-identical at main and the head; only the test file changes.
  • Own probe: the three files with 0 to 3 marks are written back with exactly those marks. A mark followed by a space, tab, zero-width space, word joiner, right-to-left mark or U+FFFE is refused unchanged. A mark inside a comment is written back.
  • The fixture fix: at aa0275be a narrow selection gives 2 passed and 3 errors (rejected HEAD -> bump-leg-advance); at the head it gives 5 passed. bom-tool-io-advance is unique among the 14 ADVANCE_BRANCH names.
  • Five modules together (bom_rewrites, bom_tool_io, bump_leg, pin_rewrite_layout, update_shape): 391 passed, 1 skipped, in CI order and reversed.
  • The new file with the three hygiene files: 328 passed, 2 skipped.
  • On a merge of this PR with all eleven open PRs touching the tools or repo_shape.py: the BOM files give 157 passed.
  • Own mutants, each restored with cmp 0, all KILLED, and all surviving the three existing files:
    • a utf-8-sig read in _repin (20 failed);
    • an lstrip that cuts spaces (8);
    • the cut widened to every Cf (24);
    • a second-character mark cut (6);
    • the shape pin written with a mark (22);
    • a new one, a utf-8-sig write at _apply_rewrite (2, the end-to-end bump-leg rows).
  • CI: tests, tests-git-bash, package and SonarCloud all SUCCESS. The Sonar gate is OK, with 0.0% duplication.
  • Body and commits: as required.
  • Not re-made: M2, M3, M5, M7-M9, M11-M15, M17 and M18. The writer's log shows all 18 killed.

Lane: openreposhape-3 (openRepoShape-3)

…used in each tool's words as one behind a mark is, the compared pair of files is named before it is asserted, and the retyped refusal words are the finders' own (#1176)

RV1188's review of #1188, from the head 024084b:

SHOULD-1 (SonarCloud python:S3415, "Unify assertion argument order in this
file"): the one assert with a bracketed tuple on the left was
`assert (pin.read_bytes(), manifest.read_bytes()) == before`, the shape
#131 (d682127) fixed. The pair is now bound to `now` and compared as
`assert now == before`. Not `after`, which is a function in this file.
Every other `==` assert has the value read on the left; the `in` asserts
keep the text looked for on the left, as `in` reads.

NIT-1: the module docstring, the row's docstring and the PR body said a
space or a tab in front of the key was held "before a mark or after one",
and the rows held no mark and one mark in front of it. A space or a tab in
FRONT of a mark is now a row for each of the three files: refused
unchanged by each tool's own code (update-pin-unrecognised,
update-manifest-no-shape, bump-leg-pin-unrecognised), as the reader refuses
it. Six rows more (126 in all), and the places are named in LEAD_PLACES.

NIT-3: the two end-to-end refusal rows retyped "has 0 `commit` line(s)";
they now use FINDERS["shape-pin-commit"].refused and
FINDERS["leg-pin-commit"].refused, as the in-process rows do.

The tools are untouched.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-23T08:30:48Z) Delta review (RV1188, opus) of #1188's second fix round: head d65340e5, on 024084b9.

Verdict: READY. All four findings from the review of 024084b9 are fixed, and nothing new is wrong. Land only once CI's tests on d65340e5 is green; it was still running at 08:29:55Z (started 08:20:58Z).

Lane: openreposhape-3 (openRepoShape-3)

@sourcery-ai

sourcery-ai Bot commented Oct 23, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 10 hours and 26 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 423e667 into main Oct 23, 2026
5 checks passed
@brettheap
brettheap deleted the l3-1176-bom-rows-hold-the-tools branch October 23, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant