[brettheap]

Lane: openreposhape-2 (openRepoShape-2)
Claim: #933 (comment) (Rule 1, CLAIMED 2026-10-20T09:34:49Z, on lane 1's word; v1.2.0)

What changed

Since #855 and #868, parse_yaml takes off every U+FEFF a text starts with. So check, load_yaml and the validators read a pin whose first line is commit: after a byte-order mark, and a manifest whose first line is shape: after one. The three line rewriters did not. They read those files as utf-8 and matched their key at column 0 of the raw text, so the mark was the key's first character. Each refused the file in words that said the key was not there.

Measured on main dc7cd49b7, in process, through each tool's own read and write: _repin with a real Rollback, and bump-leg's _leg_pin_text, rewrite_pin and write_lf. The validator read each marked file before the run.

main, marked:    REFUSED update-pin-unrecognised: contracts/shape-pin.yaml has 0 `commit` line(s) where exactly one was expected
main, marked:    REFUSED update-manifest-no-shape: project.yaml has no `shape:` block, so there is nothing to mirror the pin into
main, marked:    REFUSED bump-leg-pin-unrecognised: the pin file has 0 `commit` line(s) where exactly one was expected
branch, marked:  all three written; first bytes b'\xef\xbb\xbfcommit:' / b'\xef\xbb\xbfshape:\n' / b'\xef\xbb\xbfcommit:'; the validator reads the target
both, unmarked:  all three written, byte for byte the same on main and on the branch (cmp)

The three sites:

• update-shape.py, rewrite_shape_pin (if line.startswith("commit:")). It now takes the marks off the start of the text with a new _marks_off, matches against the lines after them, and returns marks + "\n".join(out) + "\n".
• update-shape.py, rewrite_manifest. This is the caller of _shape_block_bounds and its _is_block_header. It does the same: _shape_block_bounds gets the lines after the marks, and the marks go back in front of the rewritten lines. _is_block_header and _shape_block_bounds are unchanged.
• scripts/bump-leg.py, rewrite_pin (if line.startswith("commit:")). It does the same inline, in two lines.

Both files import the reader's own BYTE_ORDER_MARK from repo_shape, on a line of its own. Neither changes its read or its write: update-shape.py reads utf-8 and writes .encode("utf-8") through Rollback.write, and bump-leg.py reads utf-8 and writes through write_lf. Those carry the mark that the rewriters now return.

Decisions

• The fix sits where the text is split into lines, not in each startswith. Every leading mark is taken off before the finders look, and put back in front of what they return. That is one cut and one join per rewriter, and each finder keeps its own match as it was. Changing each startswith would have needed a line-1 case in every branch of each loop, including the tree_sha256 and files: lines and the line kept as it was.
• Every leading mark, not one. parse_yaml takes off text.lstrip(BYTE_ORDER_MARK), which is every mark, so the finders do too, and every mark is written back. The test has a two-mark row.
• Line 1 only. Only the start of the text is cut. A mark on line 2 in front of the key is the key's first character to the reader, which then reads no pin. The finders still refuse that file, in today's words.
• The read-back gets the original text. rewrite_fault and _refuse_unless_it_reads_back still receive the text as read, marks included, and the rewrite with its marks. The reader takes the marks off both.
• The manifest's refusals count lines as before. Cutting marks never removes a newline, so _shape_block_bounds's "at lines N and M" numbers are the file's own.
• One more refusal follows. A marked shape: on line 1 beside a second shape: header is now refused by name: update-manifest-unrecognised, "holds the shape: key twice, at lines 1 and 6". The reader refuses that file the same way: "line 6: key 'shape' is written twice". I measured it by probe. Main saw only the second header and rewrote that block of a file the reader refuses. No test pins this.

Tests

tests/test_bom_rewrites.py, new, with 12 rows. Each file is the scaffold's rendering (from tests/test_pin_rewrite_layout.py) with the key moved to line 1, and every mark is written as bytes.

test	rows	what it pins	on main dc7cd49b7
test_a_first_key_after_the_mark_is_rewritten_and_the_mark_kept	3 finders x {one mark, two}	The rewritten bytes are the marks followed by the unmarked file at the target, and the validator reads the target.	6 failed (the issue's three refusals)
test_the_same_file_without_the_mark_is_rewritten_as_before	3 finders	Control: the unmarked file is rewritten to the file at the target, with no byte in front.	3 passed
test_a_mark_after_line_1_keeps_todays_answer	3 finders	A mark on line 2 keeps today's refusal: the same code, and the same "0 commit line(s)" or "no shape: block".	3 passed

On the branch, every run was green. Each pytest process ran alone, with EMAIL exported.

set	result
tests/test_bom_rewrites.py	12 passed
tests/test_update_shape*.py, batch A (9 files)	335 passed, 3 skipped (NTFS junctions)
tests/test_update_shape*.py, batch B (10 files)	482 passed
tests/test_bump_leg*.py (10 files)	301 passed, 1 skipped (case-insensitive disk)
the new file, test_pin_rewrite_layout.py, test_bom_reads.py, test_commit_trailers.py, and the trio test_repo_hygiene.py, test_cli_reference.py, test_agents_md_codes.py, at the pushed head	480 passed (12 + 115 + 25 + 158 + 100 + 23 + 47)
tests/test_family.py, which names update-manifest-no-shape	112 passed

test_bom_reads.py and test_commit_trailers.py are the files that git grep -l -i 'feff\|bom' tests/ names and that run these tools, beside the test_update_shape* and test_bump_leg* files already above.

Mutants: each is one line, run alone against the new file, then reverted, with both files compared by sha256 afterwards. All 11 were killed.

mutant	killed by
M1 rewrite_shape_pin matches the raw text again (lines = text.splitlines())	..._mark_kept[shape-pin-commit-mark], [...-two-marks]
M2 rewrite_manifest hands _shape_block_bounds the raw text (lines = text.splitlines())	..._mark_kept[manifest-shape-header-mark], [...-two-marks]
M3 rewrite_pin matches the raw text again (for line in text.splitlines())	..._mark_kept[leg-pin-commit-mark], [...-two-marks]
M4a/b/c each rewriter returns its lines without marks +	the same two rows of its own finder, each
M5a _marks_off takes one mark only	..._mark_kept[shape-pin-commit-two-marks], [manifest-shape-header-two-marks]
M5b rewrite_pin takes one mark only	..._mark_kept[leg-pin-commit-two-marks]
M6a/b/c each finder takes a mark off every line	test_a_mark_after_line_1_keeps_todays_answer[<its finder>]

M1 to M3 put back the finder's old match: with them, the unchanged startswith (or _is_block_header) meets the mark again. M4 is the file written without the mark.

Not changed

• _is_block_header, _shape_block_bounds, with_scalar, is_nested_key are unchanged. So are both tools' reads (utf-8, not utf-8-sig) and writes, and every refusal's words.
• parse_yaml and the other readers, which take the mark off already.
• scripts/family.py's _opens_members (scripts/family.py:687) finds members: at column 0 of raw text the same way. A members: block is the file's last by rule, so it is never line 1 after a mark, and it is outside this issue's three sites.
• No tool writes a mark. Every template's first line is still schema_version: 1.
• No --help text, no docs. docs/cli.md does not change.

Closes #933

🤖 Generated with Claude Code

Summary by Sourcery

Fix BOM handling in pin and manifest rewriters so files accepted by the readers are rewritten consistently without altering existing non-BOM behavior.

Bug Fixes:

• Allow the three line rewriters to recognise keys immediately following one or more leading UTF-8 byte-order marks and preserve those marks in rewritten files.

Tests:

• Add coverage for marked and unmarked pin and manifest rewrites, multiple leading marks, and marks appearing after the first line.

[sourcery-ai]

Reviewer's Guide

The pull request updates the two shape rewriters and bump-leg’s pin rewriter to remove all leading U+FEFF characters before key detection and restore them unchanged in generated text, aligning their behavior with the project’s readers. New parameterized tests verify single and repeated BOM preservation, successful validation, unchanged unmarked output, and line-2 refusal behavior.

Sequence diagram for BOM-aware shape and pin rewriting

sequenceDiagram
    participant Input as MarkedFile
    participant Rewriter as LineRewriter
    participant Reader as ProjectReader
    participant Output as WrittenFile

    Input->>Rewriter: read utf-8 text
    Rewriter->>Rewriter: _marks_off(text)
    Rewriter->>Rewriter: detect commit: or shape: on unmarked line 1
    Rewriter->>Rewriter: prepend original marks to rewritten text
    Rewriter->>Output: write rewritten text
    Output->>Reader: validate target
    Reader-->>Rewriter: key recognized and content valid

Loading

File-Level Changes

Change	Details	Files
Make all three line rewriters recognize and preserve leading UTF-8 byte-order marks.	Import the shared BYTE_ORDER_MARK constant. Strip every leading U+FEFF before splitting and matching keys, then prepend the exact marks to rewritten output. Keep marks limited to the beginning of the text and preserve existing read-back validation and refusal behavior.	update-shape.py scripts/bump-leg.py
Add focused coverage for BOM-aware rewriting and unchanged edge-case behavior.	Test one and multiple leading marks for shape pins, manifests, and leg pins, including byte-level preservation and validator read-back. Retain controls for unmarked files and refusals when a mark appears after line 1.	tests/test_bom_rewrites.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#933	Allow update-shape.py to rewrite a shape pin whose first line is commit: after one or more leading byte-order marks, preserving those marks in the output.	✅
#933	Allow update-shape.py to rewrite a manifest whose first line is shape: after one or more leading byte-order marks, preserving those marks in the output.	✅
#933	Allow scripts/bump-leg.py to rewrite a pin whose first line is commit: after one or more leading byte-order marks, preserving those marks in the output.	✅

Possibly linked issues

• update-shape.py and bump-leg.py: the three line rewriters refuse a file whose first line is their key after a byte-order mark, in words that say the key is not there, while every reader reads it #933: The PR fixes the issue’s exact three rewriters: it strips leading marks for matching and restores them in rewritten output.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[brettheap]

(lane openRepoShape-2, 2026-10-20T10:53:33Z) Independent review of #1050 (#933) at 434a07cb, for lane openRepoShape-2: READY

Findings (no MUST, no SHOULD)

• NIT 1: the tools' own read and write paths are not pinned.
  • Where: tests/test_bom_rewrites.py:129-133 checks what each rewriter returns. No row goes through _repin (update-shape.py:4969-4974) or through _leg_pin_text (scripts/bump-leg.py:614) followed by write_lf.
  • Surviving mutants:
    • pin_path.read_text(encoding="utf-8-sig") at update-shape.py:4970;
    • the same change to the manifest read at :4973;
    • the same change at scripts/bump-leg.py:614.
  • Each passes test_bom_rewrites.py plus test_bom_reads.py (37 passed). By grep, no other test puts a mark through these reads.
  • Under each mutant, a file with one mark is written with none, and a file with two marks is written with one. Every reader reads both forms, so this is not a MUST. Still, the issue's Direction says "the marked first-key file is written, its first bytes are the mark". I accept the departure because the read and write lines are unchanged and the probe below shows they keep the mark today.
  • Fix (optional): add one row per tool on the real path. Write the marked file as bytes under tmp_path, run _repin(root, PROJECT_KIND, Rollback(root), rows, ...) and write_lf(path, rewrite_pin(_leg_pin_text(root, path), ...)), then assert that path.read_bytes() starts with MARK * marks + b"commit:" (or b"shape:").
• NIT 2: nothing pins that only U+FEFF is cut.
  • Where: update-shape.py:1728 and scripts/bump-leg.py:387.
  • Surviving mutants: text.lstrip(BYTE_ORDER_MARK + " \n") at either line. They survive the new file, and also test_pin_rewrite_layout.py, test_update_shape_apply_lines_and_headers.py, test_update_shape_header_comments.py and test_bump_leg_dry_run_truth.py (233 passed).
  • Measured under the mutant:
    • A leg pin or shape pin whose line 1 is " commit: ..." is written with exit 0. Main and the head refuse it as "0 commit line(s)", and the reader refuses it too.
    • A leg pin indented two spaces on every line is written instead of refused.
    • A leading blank line gives the same output either way, so that case is equivalent.
  • Fix (optional): add a row to test_a_mark_after_line_1_keeps_todays_answer with " " + the_file(...), expecting the same code and words as on main.

Checked and true

• The issue, line by line. All three sites are fixed as asked:
  • rewrite_shape_pin (update-shape.py:1773-1802);
  • rewrite_manifest through _shape_block_bounds and _is_block_header (:1847-1870);
  • rewrite_pin (scripts/bump-leg.py:387-411).
• Against main's code, the new file fails 6 and passes 6. The 6 failures are the marked rows, each with the issue's own refusal (update-pin-unrecognised "0 commit line(s)", update-manifest-no-shape, bump-leg-pin-unrecognised). The 3 controls and the 3 line-2 rows pass.
• Every line finder in both tools was reviewed:
  • update-shape.py:1778, :1782, :1786, :1854, :1857, :1939 and :1961, and scripts/bump-leg.py:392 and :396. All of them now look at the text after the marks.
  • The other raw matches are not pin or manifest keys: update-shape.py:880 (a path prefix), :2172 (the README's trailing line), :5288 (git output); scripts/bump-leg.py:944 and :1101-1104 (refs), and plan_workflows (workflow @<sha> values).
  • _moved_from (scripts/bump-leg.py:2195) reads through pin_values_in. Every other read of a pin or manifest goes through load_yaml or parse_yaml.
• scripts/family.py:687 (_opens_members) was rightly left alone.
  • Family.rewritten drops every line after the members: block (break).
  • validate-family.py requires schema_version, kind, id, name and more (templates/family-root/scripts/validate-family.py:17).
  • So a family manifest whose line 1 is members: would have every required key cut by the rewrite. Today's refusal is the safer answer, and the line is outside update-shape.py and bump-leg.py: the three line rewriters refuse a file whose first line is their key after a byte-order mark, in words that say the key is not there, while every reader reads it #933's three sites.
• Bytes in, bytes out, measured on each tool's real read and write (read_text("utf-8") then Rollback.write(.encode("utf-8")); _leg_pin_text, rewrite_pin, write_lf), for all three finders, main against the head:
  • A CRLF file with a mark: main refused it. The head writes the mark first and LF line endings, the same bytes as the head's own rewrite of the unmarked file with the mark in front, and the reader reads the target.
  • A file that is only one mark or two: the same refusal, word for word, on main and the head.
  • A mark, then a blank line 1: byte for byte the same on main and the head, mark kept.
  • A mark before a comment line: byte for byte the same on main and the head, mark kept.
  • A mark before schema_version: (the scaffold's order): byte for byte the same on main and the head, mark kept.
  • A marked file with no final newline: written by the head; main refused it.
  • A marked quoted "shape": header with a comment: written by the head; main refused it.
  • A mark on line 2: the same refusal on main and the head.
  • Unmarked files (key first, scaffold order, CRLF): byte for byte the same on main and the head.
  • Neither tool drops or doubles a mark: a doubled-mark mutant and an always-one-mark mutant were both killed.
• The read-back still refuses a marked file that reads back wrong. With the commit: or tree_sha256: value folded onto the next line, all three finders now refuse in the read-back's words ("reads as it stands, but this tool's rewrite of it does not read back"). Main refused these files as having no key.
• rewrite_fault and _refuse_unless_it_reads_back get the text as read, marks included. Feeding them the unmarked text instead (R8, R9) is equivalent, because the reader strips the marks from both texts. R10 (unmarked.endswith) is equivalent too, because a text made only of marks is refused first.
• The new refusal is right.
  • On a marked manifest whose line 1 is shape: and that holds a second shape: header, main rewrote the second block. The reader refuses that file ("line 6: key 'shape' is written twice").
  • The head refuses it before writing anything, as update-manifest-unrecognised "at lines 1 and 6". Those are the file's own line numbers and the reader's line 6. Two marks give the same answer.
  • It can be reached through apply. Reading the code, main would have written the file and then refused it as update-validators-red once the validators ran after _repin (update-shape.py:5177 and :5179), then rolled it back.
  • No test is needed. This is the unmarked duplicate refusal's path, already pinned at tests/test_update_shape_header_comments.py:272 and tests/test_update_shape_apply_lines_and_headers.py:206. The cut in front of it is pinned by the marked manifest rows. A test would be a NIT at most.
• Sonar:
  • S3776: none of the changed functions goes over 15. By my count, rewrite_shape_pin is 14, rewrite_pin 9, rewrite_manifest 8 and _marks_off 0, and the PR adds no branch.
  • S9073: the new tests have no composite assert.
  • Duplication: no run of 4 non-blank lines in tests/test_bom_rewrites.py matches any tests/*.py.
  • The BYTE_ORDER_MARK imports sit on their own lines with a comment and # noqa: E402, as each file's other single-name imports do.
• The docs need no change. No printed text, code or flag changes, and docs/cli.md, README.md, AGENTS.md and docs/handbook.html do not mention the mark.
• The PR body's claims hold:
  • 12 tests, 6 of which fail on main;
  • 480 passed on the head set;
  • _is_block_header and _shape_block_bounds unchanged;
  • the only closing keyword in the body and in the squash message is Closes #933.
• The merge with main 1a47ed07e is clean (tree 699fe9d0b). Main has not touched the PR's files, and the test set passes on the merge.

Could not verify

• The writer's 11 mutants were not re-run. Its script edits the writer's own worktree; I read its log instead. My own 19 mutants cover the same ground.
• The test_update_shape* and test_bump_leg* batches were not run. Reading the code gave no reason: _marks_off returns ("", text) for an unmarked file, so the computation is unchanged. The landing gate runs them.
• Sonar was not run. Its MCP server failed to connect, so the S3776 counts are by hand.
• Not run on Windows or macOS.
• Outside this PR: the reader reads a leg pin indented two spaces on every line, and main and the head both refuse it as "0 commit line(s)". This is the same kind of mismatch as update-shape.py and bump-leg.py: the three line rewriters refuse a file whose first line is their key after a byte-order mark, in words that say the key is not there, while every reader reads it #933, but nothing writes such a file. It could be a follow-up issue if wanted.

Lane: openreposhape-2 (openRepoShape-2)

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 8 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud