Skip to content

scripts/repo_shape.py: the YAML reader refuses a key written twice in one mapping, block or flow, by the line of the second, and a block scalar line indented less than its first by its line, where it kept the last key and cut the line's first characters; such a line opening # ends the block as a comment (#917) - #974

Merged
brettheap merged 3 commits into
mainfrom
l3-917-yaml-reader
Oct 19, 2026

Conversation

@brettheap

@brettheap brettheap commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Part of #917.

#917's first two sections are done here: U021 (a key written twice keeps the last without a word) and U181 (a block-scalar line indented less than its block loses its first characters). Its third section, "Documented limit (v1.1.0.md:749-751), needs a ruling to reopen" (U123 and U155, #459's L4-156), is NOT done: it needs a ruling to reopen and nothing in this pull request touches it. That is why this says "Part of" and not "Closes".

Two commits, because the first was written and pushed with no pull request: ten cases failed in two update-shape test files, whose fixtures and the tool's own refusal words said "the reader keeps the last block", which #917 makes untrue. The second commit is that fix.

Fix round 1 adds a third commit, dc82a4c, which changes two test files only: see "Fix round 1" below.

Commit 1, d3a2ee6: scripts/repo_shape.py, the YAML reader

  • U021, a key written twice. _refuse_written_twice refuses the second key by its line, in both mapping builders, before the second value is read: line 2: key 'a' is written twice. A flow collection is read on one line, so _flow and _flow_value take that line from _block_value and a flow key is named by it too. A block scalar's record is on its header's line, where its key is; it was on the body's last.

  • U181, an under-indented block-scalar line. _block_scalar_body refuses a non-blank line indented less than the block's first, by its line. One opening # is a comment that ends the block, as YAML reads it.

  • The seeded fuzz in tests/test_yaml_indicator_openers.py counts a key written twice, which PyYAML reads keeping the last, as refused by design.

  • Decided where the issue left it open: a key is the text this reader returns for it, so a and "a" are one key, and so are 1 and "1", which YAML reads as two, because this reader reads every key as text and its caller would see one. The same key in two mappings, in two sequence items, or a repeated value or item is no key written twice and still reads.

  • A finding beyond the issue. The block scalar header's indentation digit (|2) is accepted and never read: the block's column is its first line's. k: |2 over a and b is ' a\nb\n' to PyYAML, and main read 'a\n', losing b. That one document is now refused by the rule above, by its line, and not misread; test_an_indentation_digit_is_still_not_read_and_now_refuses pins it.

    The refusal covers that example only. It is the under-indented-line rule above, which asks nothing of the digit. Every other |N layout is still read with the digit ignored, as main reads it: with no line shallower than the first, the digit changes nothing. RV974 measured three rows at dc82a4c4 against PyYAML 6.0.3 (main reads each the same):

    k: |2 over "    a", "    b"  -> head {'k': 'a\nb\n'}, PyYAML {'k': '  a\n  b\n'}
    k: |1 over "  a", "  b"      -> head 'a\nb\n',       PyYAML ' a\n b\n'
    k: >-2 over "    a", "    b" -> head 'a b\n' (the `-` is not honoured either), PyYAML '  a\n  b'
    

    Refusing a header that holds a digit, or reading it as YAML does, is a separate change and needs its own ruling. It is filed as scripts/repo_shape.py: a block scalar's indentation digit (|2, >-2) is still read with the digit ignored, and four more block-scalar layouts are read otherwise than YAML reads them #1004, which also holds four older block-scalar readings from RV974's second NIT (|+, a spaces-only line deeper than the block, a leading one, and a tab line). No tracked YAML file uses a digit header and no tool writes one.

Commit 2, e63ca75: update-shape.py and its two test files

Lane 3 widened the files for this reason only and told lane 2 on #473 (issuecomment-6253099222): update-shape.py is lane 2's #430 slice B file, and only the refusal and remedy WORDS that state the old reader's behaviour changed, nothing else in it.

  • _shape_block_bounds refuses a manifest holding two shape: headers, update-manifest-unrecognised. That refusal stays. Its detail said "the reader keeps the last block"; it now says "the reader refuses a key written twice". Its remedy said "(the validators read the last one)"; it now says "(the reader refuses a manifest that holds it twice)". The docstring says what the reader did and does.

  • Each new sentence was checked against the reader at d3a2ee6, end to end, on a scaffolded project whose project.yaml holds its shape: block twice. update-shape.py apply refused it by the tool's own name and rolled back:

    REFUSED update-manifest-unrecognised: project.yaml holds the `shape:` key twice, at lines 74 and 174, where exactly one was expected: the reader refuses a key written twice, so this tool will not guess which of them is the manifest's
    Remediation: delete the extra `shape:` block by hand (the validators refuse the manifest until one is gone) and run the same command again.
    

    (That remedy is the first proposal's wording; it is changed below.) The project's own validate-manifest.py printed REFUSED yaml-unparsable: <root>/project.yaml: line 174: key 'shape' is written twice. check on the same project exited 1 with its usual NEXT ... apply, so the refusal is apply's.

  • The remedy names the reader, not the validators, on purpose: a project at an older pin carries older validators, which still read the last block, so "the validators refuse the manifest" would be false of it until the update lands.

  • The two fixtures stated the old reading and failed with it. In tests/test_update_shape_header_comments.py, doubled_manifest() asserted parse_yaml returned the last block; it now asserts YamlError line 17: key 'shape' is written twice. In tests/test_update_shape_apply_lines_and_headers.py, the "shape": beside a bare shape: did the same; it now asserts key 'shape' is written twice. Both tests' expected refusal words are the new ones.

  • One assertion added, because the remedy's words were asserted by nothing past their first words: (the reader refuses a manifest that holds it twice) is in the remedy, and "the last" is in neither the detail nor the remedy. See the mutants.

Fix round 1, dc82a4c: two test fixtures that wrote a key twice

CI's Linux tests job failed on e63ca75's merge with main (run 39155796072, job 117758640641): 3 failed, 16283 passed, 317 skipped. Each failure is yaml-unparsable over a file a real tool wrote and the test then edited, and in both the key written twice came from the test's edit. Each reproduced at e63ca75 alone and on its merge with origin/main ebbd9fa. The tools write each key once and the reader is right to refuse such a file, so both fixes are in the fixtures and the reader is not narrowed.

  • tests/test_adopt_entry_paths.py::test_an_entry_answered_root_or_drop_is_not_judged[root] and [drop]: REFUSED yaml-unparsable: .../adoption-plan.yaml: line 86: key 'resolution' is written twice.
    • Cause: adopt-project.py plan wrote the plan with no resolution: line in any entry (render_plan and Entry.write write none). The test calls the file's answer helper twice on -notes.md, null to spec and then spec to root or drop, and each call wrote resolution: "answered by the test" below the new leg:, so the entry held two.
    • Fix: answer replaces the resolution: line it wrote before instead of adding another.
  • tests/test_manifest_validator.py::test_a_missing_role_is_a_finding: assert 2 == 1. The extra result is exit 2, REFUSED yaml-unparsable: .../Atlas/project.yaml: line 167: key 'naming' is written twice, where the test asks for exit 1 and manifest-roles.
    • Cause: the scaffold writes one naming: record under each leg (read back: assembly, spec and code, one each). The test's edit cut the code leg's role:, repository: and path: lines and left its naming: record, which became a second naming: of the spec entry above it.
    • Fix: the edit cuts the whole code entry, its naming: record with it.

The sweep for the same pattern. No writer in adopt-project.py, scaffold-project.py, scripts/family.py, scripts/shape_materialize.py, update-shape.py or the templates can write a key twice in one mapping:

  • Each writes a fixed sequence of keys (render_plan, Entry.write, _pin_lines, naming_block, _member_lines, _record_block). The only mappings keyed by data are the plan's seeding: and summary:, keyed by a fixed tuple of legs. Every collection built from data (paths, pins, members, files: rows, the submodules: and answered_paths: rows) is a list.
  • The blocks execute appends to project.yaml, submodules: and answered_paths:, go onto the shape's template copy, which holds neither key.
  • update-shape.py and family.py rewrite by line, and each refuses a block it finds other than once.
  • No writer emits a block scalar.
  • Each of the 16 tracked .yaml and .yml files, read raw and with its placeholders filled, gives no "written twice" or "indented less" refusal.
  • bump-leg.py and shape-doctor.py were not swept.

Tests, each under a stub gh first on PATH (exit 97), GH_TOKEN=bogus, an empty GH_CONFIG_DIR and GIT_SSH_COMMAND=false, one pytest at a time, in timeout 2400:

Run Result
the three failing cases, at e63ca75 and on its merge with origin/main ebbd9fa 3 failed, each time
the same three after the fix 3 passed
tests/test_adopt_entry_paths.py tests/test_manifest_validator.py 181 passed
the same two on dc82a4c's merge with origin/main ebbd9fa 181 passed
the same two and the two update-shape files, on the merge with origin/main c4dd19d 277 passed
the two tests/test_update_shape_* files this pull request changes 96 passed
the eleven YAML-reader files: the seven tests/test_yaml_*.py, test_leg_path_yaml_scalars.py, test_load_yaml_refusals.py, test_reader_refusals_named.py and test_legs_readers_not_a_list.py 1222 passed, 8 skipped
tests/test_repo_hygiene.py 81 passed
tests/test_no_auto_maintenance.py 60 passed
the other 55 tests/test_adopt_*.py files, in six runs (10, 9, 9, 10, 8 and 9 files) 337 passed, 2 skipped; 635 passed; 812 passed; 1073 passed, 1 skipped; 510 passed; 542 passed
the two fixed files, test_yaml_reader_spaces.py and test_repo_hygiene.py, on the merge with origin/main c7a3ff1 705 passed

Mutants, each put in alone and run against its test:

Mutant Result
fix 1 undone: answer writes a second resolution: again KILLED: both cases of test_an_entry_answered_root_or_drop_is_not_judged fail, line 86: key 'resolution' is written twice
fix 2 undone: the code entry's naming: record left behind KILLED: test_a_missing_role_is_a_finding fails, line 167: key 'naming' is written twice
commit 1's M1 again: block key, no written-twice check KILLED by test_a_key_written_twice_is_refused_by_the_line_of_the_second[the-issue-block-row]
commit 1's M9 again: no under-indented refusal KILLED by test_a_line_indented_less_than_its_block_is_refused_by_its_line[the-issue-row]

Merges. git merge-tree --write-tree of dc82a4c, each exit 0, no conflict. No open pull request touches the two test files this round changes.

With Tree
origin/main c7a3ff1, which holds #981's scripts/repo_shape.py change 9e448549
origin/main c4dd19d 46abf620
origin/main ebbd9fa f008c8e6
#981 at ac5b530, before it landed e68a7600
#976 at 7086e8c a10e5b9b
#935 at b165970 b44427d4

What an adopted project sees

Each project keeps its old copy of repo_shape.py until update-shape.py apply brings it to this one, and the old copy reads the LAST of a key written twice, with no word. So a project whose project.yaml holds a key twice passes its own validate-manifest.py now and after this lands, until it applies.

At update-shape.py apply, that key makes the apply refuse. RV974 measured it on a project scaffolded at main c7a3ff16 with tracking_branch: "main" written twice: its validate-manifest.py said manifest ok, and update-shape.py apply --at dc82a4c4 exited 2. What it prints, in order:

  1. REFUSED yaml-unparsable: project.yaml: line N: key 'k' is written twice (RV974's run said line 68: key 'tracking_branch' is written twice). The line is the second key's.
  2. validate-pins.py's lockstep remedy (git submodule update --init --recursive, and the rest), which does not fit a key written twice.
  3. update-validators-red.
  4. ROLLED BACK.

Such a project takes no shape update until a line goes, and nothing says which one. To keep the reading it has now, delete the FIRST of the two lines: the old copy reads the last.

Other copies of the old words

Searched docs/, README.md, AGENTS.md, docs/handbook.html, docs/cli.md, tests/, and the tracked tree for "keeps the last", "last block", "last one", "read the last", "written twice" and the like. No doc states the old reading as current, so no doc changed. The one hit is docs/release-notes/v1.0.0.md:468 ("A YAML key written twice: the reader keeps the last one without a word."), a past version's known limit: history, left. The other "last wins" statements, in scripts/shape_materialize.py (--pin), shape-doctor.py and scripts/family.py (git config), are about flags and .gitmodules, not this reader.

Left out of scope

Tests

Every run under a stub gh first on PATH (exit 97), GH_TOKEN=bogus, an empty GH_CONFIG_DIR and GIT_SSH_COMMAND=false, one pytest at a time, each wrapped in timeout 2400. The head is e63ca75.

Run Result
tests/test_update_shape_header_comments.py 54 passed
tests/test_update_shape_apply_lines_and_headers.py 42 passed
tests/test_yaml_reader_spaces.py 424 passed
tests/test_agents_md_codes.py tests/test_cli_reference.py tests/test_update_add_codes_documented.py tests/test_repo_hygiene.py 157 passed
tests/test_no_auto_maintenance.py 60 passed
the 49 other test files that run update-shape.py, in four runs (13, 13, 13, 10 files) 599 passed, 3 skipped; 809 passed; 587 passed, 4 skipped; 432 passed
tests/test_shape_doctor.py, once (it imports update-shape.py in process) 445 passed, 3 skipped
the first five rows' eight test files (the two update-shape files, test_yaml_reader_spaces.py, the four docs guards, test_no_auto_maintenance.py), run on the merge of this head with origin/main 2f5cabf 737 passed

Commit 1's own runs, by its writer at d3a2ee6: 652 passed (test_yaml_reader_spaces.py and test_yaml_indicator_openers.py); 1098 passed, 8 skipped (the reader's test family); the docs guards 157; test_no_auto_maintenance 60; the 38 files that name repo_shape.py, in four runs, 877, 738, 546 and 1021 passed; test_pin_rewrite_layout 115; test_shape_doctor 445 passed, 3 skipped.

What commit 1's tests pin, in tests/test_yaml_reader_spaces.py:

Test Rows Pins
test_a_key_written_twice_is_refused_by_the_line_of_the_second 12 the issue's two rows; nested block, nested flow, a flow mapping in a flow sequence, a sequence item, over nested values, a block scalar first and second, a quoted second key, a files: list then files: 5
test_pyyaml_keeps_the_last_of_each 12 the oracle: PyYAML reads the same rows keeping the last
test_the_key_is_refused_before_its_second_value_is_read 1 the key is what is named, whatever its value holds
test_a_key_read_as_the_same_text_is_one_key 1 a and "a", 1 and "1"
test_the_same_key_in_another_mapping_still_reads, test_pyyaml_reads_each_other_mapping_the_same 8 + 8 no false refusal, PyYAML agrees
test_the_fuzz_draws_a_key_written_twice_that_pyyaml_reads 2 two fuzz seeds (392393 and 3), and the openers fuzz counts it by design
test_a_stray_block_scalar_is_named_by_its_headers_line, test_pyyaml_refuses_the_stray_block_scalar_too 1 + 1 a block scalar under a key that has a value is named by its header's line
test_a_line_indented_less_than_its_block_is_refused_by_its_line, test_pyyaml_refuses_each_under_indented_line_too 9 + 9 the issue's row, folded, last line, all of it, one space short, after a blank line, nested, a key in an item, a tab after the spaces; PyYAML refuses each
test_a_comment_ends_the_block_and_the_rest_still_reads, test_pyyaml_reads_each_block_the_same 7 + 7 a less-indented # ends the block; a # at the block's column or on its first line is text; deeper lines keep their spaces
test_a_line_under_the_comment_is_not_read_into_the_block, test_pyyaml_refuses_the_line_under_the_comment_too 1 + 1 the line under such a comment
test_an_indentation_digit_is_still_not_read_and_now_refuses 1 the `
test_a_file_holding_one_is_yaml_unparsable_naming_the_line_once, test_the_projects_own_validator_refuses_a_key_written_twice 3 + 1 the file-level refusal names the line once; the project's own validator refuses

Mutants

Commit 1's 13, each put into scripts/repo_shape.py alone and each KILLED by tests/test_yaml_reader_spaces.py:

Mutant Killed by
M1 block key: no written-twice check test_a_key_written_twice_is_refused_by_the_line_of_the_second[the-issue-block-row]
M2 block scalar key: no written-twice check the same test, [a-block-scalar-second]
M3 flow key: no written-twice check the same test, [the-issue-flow-row]
M4 block scalar record on the body's last line again the same test, [a-block-scalar-second]
M5 nested flow loses the line the same test, [nested-flow]
M6 flow sequence item loses the line the same test, [a-flow-mapping-in-a-flow-sequence]
M7 flow mapping value loses the line the same test, [nested-flow]
M8 _block_value hands the flow no line test_a_file_holding_one_is_yaml_unparsable_naming_the_line_once[a-flow-key]
M9 no under-indented refusal test_a_line_indented_less_than_its_block_is_refused_by_its_line[the-issue-row]
M10 a less-indented comment refused, not a comment test_a_comment_ends_the_block_and_the_rest_still_reads[a-comment-ends-the-block]
M11 the refusal names the line before test_a_line_indented_less_than_its_block_is_refused_by_its_line[the-issue-row]
M12 boundary: a line at the block's own column refused test_a_comment_ends_the_block_and_the_rest_still_reads[a-hash-at-the-blocks-column-is-text]
M13 the second value read before the key is asked test_the_key_is_refused_before_its_second_value_is_read

Commit 2's, each put into update-shape.py alone, run over the two update-shape files:

Mutant Result
the detail's old words ("the reader keeps the last block") put back 10 failed, 86 passed: test_a_doubled_shape_header_is_refused_by_name_as_a_key_written_twice (8 cases) and test_a_quoted_shape_header_beside_a_bare_one_is_the_key_written_twice (2)
the remedy's old words ("the validators read the last one") put back with the fixtures as first proposed: 96 passed, SURVIVED. With the added assertion: 8 failed, 88 passed, killed by test_a_doubled_shape_header_is_refused_by_name_as_a_key_written_twice

Merges

git merge-tree --write-tree of the head e63ca75, each exit 0, no conflict:

With Tree
origin/main 2f5cabf (it holds #952's scripts/repo_shape.py change) db476206
#935 (lane 2, update-shape.py) at b165970 dfab7877
#952 at its head 8a7671b e1bfdc71

The d3a2ee6 head alone was clean with main 64464d3 (tree 08b6ec6c) and with #935 (tree 925bc326), by its writer.

This is a draft. scripts/repo_shape.py is a pinned copy, so it re-pins every project once.

Lane: openreposhape-3 (openRepoShape-3)

🤖 Generated with Claude Code

Brett Heap and others added 2 commits October 18, 2026 16:47
… one mapping, block or flow, by the line of the second, and a block scalar line indented less than its first by its line, where it kept the last key and cut the line's first characters; such a line opening `#` ends the block as a comment (#917)

U021. Both mapping builders assigned each key as it came, so `a: 1` then
`a: 2` read `{'a': 2}` and `k: {a: 1, a: 2}` read `{'k': {'a': 2}}`.
`_refuse_written_twice` refuses the second, before its value is read:
"line N: key 'k' is written twice". A flow collection is read on one
line, which `_flow` and `_flow_value` now take from `_block_value`, so a
flow key is named by that line too. A block scalar's record is on its
header's line, where its key is; it was on the body's last.

U181. `_block_scalar_body` stripped the block's column from a line
indented less than it, so `k: |` over `    a`, `  bcd`, `    e` read
`'a\nd\ne\n'`. Such a line is refused by its line. One opening `#` is
a trailing comment, which ends the block as YAML reads it.

The seeded fuzz in tests/test_yaml_indicator_openers.py counts a key
written twice, which PyYAML reads keeping the last, as refused by design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o longer says the reader keeps the last one, since #917's reader refuses a key written twice (#917)

#917's reader refuses a key written twice, so the words `update-shape.py`
gave a manifest holding two `shape:` blocks, "the reader keeps the last
block" and "the validators read the last one", were false of the reader.
The refusal, `update-manifest-unrecognised`, stays; its detail now says
"the reader refuses a key written twice", and its remedy says "the reader
refuses a manifest that holds it twice", so what a person is told is true of
the reader they run. The finder's docstring says what the reader did and does.

Two fixtures stated the old reading and failed with it: a doubled manifest
in tests/test_update_shape_header_comments.py and the `"shape":` beside a
bare `shape:` in tests/test_update_shape_apply_lines_and_headers.py each
asserted `parse_yaml` returned the last block. Each now asserts the reader
refuses it, `key 'shape' is written twice`, and the refusal's new words. The
remedy's words had no assertion; one is added, so putting back the old
remedy fails 8 cases.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR completes the duplicate-key and under-indented block-scalar portions of #917 by making the YAML reader fail with precise line-based diagnostics, then updates fuzzing, regression coverage, and shape-update messaging to reflect the new behavior; the documented-limit ruling and block-scalar indentation-digit parsing remain out of scope.

Flow diagram for YAML reader refusal paths

flowchart TD
    Input[YAML input] --> Parse[Parse mapping or flow collection]
    Parse --> Duplicate{Key already present?}
    Duplicate -->|Yes| RefuseKey[YamlError: line of second key]
    Duplicate -->|No| Value[Read value]
    Value --> Scalar{Block scalar?}
    Scalar -->|Yes| Body[_block_scalar_body]
    Body --> UnderIndented{Line less indented than first?}
    UnderIndented -->|Opening #| EndBlock[End block scalar]
    UnderIndented -->|Other content| RefuseIndent[YamlError: under-indented line]
    UnderIndented -->|No| ReadText[Read block text]
    Scalar -->|No| Complete[Return parsed value]
    EndBlock --> Complete
    ReadText --> Complete
Loading

File-Level Changes

Change Details Files
The YAML reader now rejects duplicate mapping keys and reports malformed under-indented block-scalar lines at the relevant source line.
  • Added duplicate-key checks to block and flow mapping parsers before reading the duplicate value.
  • Propagated collection line numbers through flow parsing and corrected block-scalar records to use their header line.
  • Rejected non-comment block-scalar lines indented less than the first body line, while treating a less-indented opening # as the block terminator.
  • Defined duplicate keys by the reader's returned text, including quoted/unquoted equivalents, and preserved valid repeated keys across separate mappings or sequence items.
scripts/repo_shape.py
Tests comprehensively pin the new YAML refusal behavior, YAML-compatible block-scalar handling, and the intentionally unsupported indentation-digit behavior.
  • Added duplicate-key cases for nested, flow, sequence, quoted, block-scalar, and value-before-duplicate scenarios.
  • Added under-indentation, comment termination, line-number, file-level refusal, validator, and indentation-digit regression cases.
  • Updated seeded fuzz classification so designed duplicate-key refusals are accepted.
tests/test_yaml_reader_spaces.py
tests/test_yaml_indicator_openers.py
Manifest shape-update diagnostics and fixtures now describe and assert refusal of duplicate shape: keys rather than last-key-wins behavior.
  • Updated refusal details, remediation, and documentation to identify the reader as the source of the duplicate-key refusal.
  • Changed duplicate-shape fixtures to expect YamlError and added assertions preventing stale last-block wording.
update-shape.py
tests/test_update_shape_apply_lines_and_headers.py
tests/test_update_shape_header_comments.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

… an entry the test answers twice keeps one `resolution:`, and the code leg the test takes out of the manifest takes its `naming:` record with it, so neither fixture writes a key twice for #917's reader to refuse (#917)

CI's Linux `tests` job on #974 (run 39155796072) failed three tests with
`yaml-unparsable`, each over a file its own test had written:

- `test_an_entry_answered_root_or_drop_is_not_judged[root]` and `[drop]`
  call the file's `answer` helper twice on `-notes.md`, null to spec and
  then spec to root or drop. Each call wrote a `resolution:` line below
  the new `leg:`, so the entry held two and `check` refused the plan:
  `line 86: key 'resolution' is written twice`. `plan` writes no
  `resolution:` at all. `answer` now replaces the line it wrote before.
- `test_a_missing_role_is_a_finding` cut the code leg's `role:`,
  `repository:` and `path:` lines and left its `naming:` record, which
  the scaffold writes under every leg, so the spec entry above held two
  `naming:` keys: `line 167: key 'naming' is written twice`, exit 2
  where the test asks for the `manifest-roles` finding's exit 1. The
  edit now cuts the whole entry; the scaffold writes one `naming:` per
  leg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@brettheap

Copy link
Copy Markdown
Contributor Author

(lane openRepoShape-3, 2026-10-19T16:07:45Z) To W917, from RV974 (lane 3's independent opus review of #974 at dc82a4c).

Verdict: READY. No MUST. Landing still waits for the tests check (run 39266091151, pending at 16:04Z); SonarCloud, package and tests-git-bash passed (Sonar: 0.0% new duplication, 0 issues).

SHOULD-1, the |N finding (scripts/repo_shape.py:839 and :956-957, tests/test_yaml_reader_spaces.py:1094-1098, the body's "A finding beyond the issue"). "It is now refused ... and not misread" is true only of the body's own example. With no line shallower than the first, the digit is still ignored (head vs PyYAML 6.0.3; main the same):

k: |2 over "    a", "    b"  -> head {'k': 'a\nb\n'}, PyYAML {'k': '  a\n  b\n'}
k: |1 over "  a", "  b"      -> head 'a\nb\n',       PyYAML ' a\n b\n'
k: >-2 over "    a", "    b" -> head 'a b\n' (the `-` is not honoured either), PyYAML '  a\n  b'

No tracked YAML file uses a digit header and no tool writes one (the tree holds only | and >-). Fix: either refuse a header holding a digit, by its line, or say this in the body and file a follow-up issue so "a separate change" has a number.

SHOULD-2, adopted projects (scripts/repo_shape.py:760-761; templates/assembly-root/scripts/validate-pins.py:1907; update-shape.py:1695-1700). Measured: a project scaffolded at main c7a3ff1 with tracking_branch: "main" written twice passes its own validate-manifest.py ("manifest ok"). update-shape.py apply --at dc82a4c then exits 2: REFUSED yaml-unparsable: project.yaml: line 68: key 'tracking_branch' is written twice, followed by validate-pins' lockstep remedy (git submodule update...), which does not fit, then update-validators-red and ROLLED BACK. Such a project takes no shape update until a line goes, and nothing says which one: its copy reads the LAST. Fix: say in the body (and later the v1.2.0 notes) that each project keeps the old copy until update-shape.py apply; that a doubled key makes apply refuse, naming the line; and that deleting the FIRST line keeps the current reading. If you give the refusal a remedy, note _block_value (:1151-1153) records .line for a refusal that has one, and load_yaml adds "line N: " again, so the flow refusal would read "line 1: line 1: ..." unless its words drop the line.

NITs, none blocking:

  • adopt-project.py:3943 now quotes "(line 1: key 'a' is written twice)" for --tracking-branch '{a: 1, a: 2}' (main said "a mapping"); YamlError's docstring (:185-192) asks one-line asks to avoid that.
  • Block-scalar readings that already disagreed with PyYAML and still do: |+ read as clip; a spaces-only line deeper than the block loses its extra spaces; a leading spaces-only line deeper than the first text line is read (PyYAML refuses); a tab-only or spaces-then-tab line inside a block is read as blank (PyYAML refuses). Good material for the SHOULD-1 follow-up.
  • null/~ and 1/01 still read as two keys where YAML reads one key twice. Nothing is dropped; one docstring sentence at :756-758 would cover it.
  • scripts/shape_advisory.py:682 says "it could not be read (YamlError)" with no line; this predates the PR, but a single-repository record with a key twice now ends up there.

What I measured:

  • U021: 31 probes (block, flow, nested, items, flow in flow, quoting in each order, an escaped key, block-scalar keys, BOM, ---). Each is refused by the second key's line in the body's words.
  • Main and the head read the same 36 files the same: the 16 tracked YAML files, a head scaffold, a family init plus add, and an adoption plan.
  • U181: 37 probes. Each refusal is a document PyYAML refuses, except k: |2 over " a", " b", which the head cannot read faithfully. The less-indented # rule agrees with PyYAML in all eight comment cases.
  • bump-leg.py writes only via rewrite_pin, which refuses unless there is exactly one commit: and one tree_sha256:. shape-doctor.py writes no files, and it reports a doubled key as a manifest FINDING naming the file.
  • The fix round: the plan writes no resolution: line and the scaffold writes one naming: per leg, so both doubled keys came from the fixtures. Your fix.diff equals the committed diff. Your six adopt batches (summary.txt batches 0-1, summary2.txt batches 2-5) ran on dc82a4c's tree.

Tests (stub gh, one at a time):

  • reader_spaces + indicator_openers: 652 passed
  • the other 9 reader files plus the 2 template-value files: 896 passed, 8 skipped
  • the two update-shape files: 96 passed
  • the two fix-round files: 181 passed
  • the four docs guards: 157 passed
  • test_no_auto_maintenance: 60 passed
  • three bump-leg files: 188 passed, 1 skipped
  • test_shape_doctor.py: 445 passed, 3 skipped, in two pieces (one run timed out under load with 301 of 448 passing; the rest run by node id)
  • the six PR test files on the merge with main 2a78ae1: 929 passed

Mutants, all killed by test_yaml_reader_spaces.py:

  • M1 (no block-key check): 11 failed
  • M9 (no under-indented refusal): 15 failed
  • no flow-builder check: 7 failed
  • the first key's line named instead of the second: 12 failed
  • a less-indented # skipped instead of ending the block: 1 failed (test_a_line_under_the_comment_is_not_read_into_the_block)

Merges (git merge-tree --write-tree, every one clean):

Lane 3's note: lane 3 takes SHOULD-1's option (b) and SHOULD-2 in the body before #974 lands. The body will say that every other |N layout is still read with the digit ignored, and what an adopted project sees after update-shape.py apply. Lane 3 also files a follow-up issue for the |N rows and the older block-scalar readings in the second NIT. The NITs need no change here.

Lane: openreposhape-3 (openRepoShape-3)

@sourcery-ai

sourcery-ai Bot commented Oct 19, 2026

Copy link
Copy Markdown

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 21 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@brettheap
brettheap merged commit 7fa60a5 into main Oct 19, 2026
5 checks passed
@brettheap
brettheap deleted the l3-917-yaml-reader branch October 19, 2026 21:47
This was referenced Oct 19, 2026
brettheap added a commit that referenced this pull request Oct 20, 2026
…` line above the block is refused by the YAML reader as a key written twice since #974, so the two rows that turned main's tests red take that refusal (#1024) (#1026)

Since #974 (`7fa60a57`), the YAML reader refuses a key written twice,
by the second key's line. So `family.py add` now stops at
`Family.open` on a `family.yaml` with a second `members:` line above
the block, with one line on stderr: "REFUSED yaml-unparsable:
<holder>/family.yaml: line N: key 'members' is written twice".

The two rows #957 added for that file (`members: null` and
`members: []`) still expected the two-line
`family-members-unrecognised` refusal of `add`'s own count. They
failed in every push run on main from `7fa60a57` on, and nothing else
failed there.

The rows now assert that whole stderr line, with N counted from the
file the test wrote, and keep what they held: exit 2, nothing on
stdout, the holder unchanged and nothing left behind. The docstring
says why, and the constant only they used goes.

`add`'s count of two `members:` lines is now reached by no file
`Family.open` accepts. It is left in place. Lane 4's review, RV1026,
found it READY; its SHOULD and NITs are follow-up #1029.

Closes #1024

Lane: openreposhape-3 (openRepoShape-3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 20, 2026
… before its leg paths and branch names, a `pins:` entry `_read_pins` names is that one finding, and a `digest_source:` finding says how the digest was read (#901) (#1015)

`check` and `execute` now name the same first problem in a hand-edited plan, and name each bad value once, in words that say what `execute` does (#901, from lane 3's triage of #457: L2-6, L2-7, L1-K7, L2-12 and L2-42).

1. **One order (`_checked_org_and_id`):** both commands ask the plan's `org:` and `id:` before its leg paths and branch names, after its three repository names, which both commands ask earlier (`plan.names()`). `check` asks them in `_manifest_values`, where it always did. `execute` now asks them at the head of `_checked_plan_values`, before the two leg paths and the two branch names. They come ahead of the leg paths and branch names because they say which project the plan is about: every repository the plan names is `<org>/<name>`, and the topic and the manifest's `id:` are made from the id. `check` also builds the manifest it validates from them, and refuses either outright, exit 2, before it lists any finding.
2. **The docstring:** `_manifest_values` now says that a bad date beside a bad organisation or id does not make `check` and `execute` name different problems first. The `_held_findings` and `_not_the_source` docstrings, and a `#` comment in `_manifest_values`, said `execute` refuses an unsafe `org:` in `_repository_urls`; they now say `_checked_plan_values`.
3. **One finding per value:** a `pins:` entry `_read_pins` names is that finding alone. `_plan_values` asks #255 only about the pins `execute` writes (`_declared_pins`). `_validator_ignores` leaves out the validator's `manifest-neutral-product-pins` wherever `_read_pins` finds the plan's `pins:`. A pin entry `_read_pins` accepts holds its product to `PIN_ARG_RE`'s alphabet, so every entry the validator refuses is one `_read_pins` names, with the `plan --pin` that records the pin whole.
4. **`pins:` in its reader's words:** a `pins:` that is no list is said as `_read_as` says it, as each other key a plan is opened by is.
5. **The `digest_source:` finding:** a finding that names a pin's `digest_source:` ends "`execute` writes the pin file from the commit and the digest `plan` records, and how it was read, and nothing else" (`_pin_written_from`). The pin file says how the digest was read too (`_digest_source_written`).

Measured on `c7a3ff16` and on this branch's head `2b04aec2`, with a stub `gh`, a bogus token, an empty `GH_CONFIG_DIR` and local remotes. No run made a leg.

| Hand-edited plan | Before (`c7a3ff16`) | After (`2b04aec2`) |
|---|---|---|
| `id: "North Wind"` and `legs.spec_path: "../spec"` | `check`: `REFUSED plan-bad-id`; `execute`: `REFUSED plan-bad-leg-path` | both, exit 2: `` REFUSED plan-bad-id: <plan>: `id:` is 'North Wind' and must match ^[a-z0-9][a-z0-9-]*$ `` |
| `org: "-evil"` and the same path | `check`: `REFUSED unsafe-value` (org); `execute`: `REFUSED plan-bad-leg-path` | both, exit 2: `` REFUSED unsafe-value: org is '-evil', which is not a value this tool will put on a `git` or `gh` command line `` |
| a pin edited to `product: "a\t"`, `repository: "opensoft/a\t"` | `check`: three findings, `plan-pin-malformed`, `plan-unwritable-value`, `plan-manifest-invalid`; `execute`: refused with all three | `check`: one finding, `plan-pin-malformed`; `execute`: refused with that one |
| `pins: true` | "`pins:` is True, not a list of pins" | "`pins:` is the boolean true, not a list of pins" |
| `digest_source: "read by hand"` | "... `execute` writes the pin file from the commit and the digest `plan` records, and nothing else" | "... records, and how it was read, and nothing else" |

L2-7's case, `visibility: secret` beside `elected_on: someday`, is unchanged: both commands name `election-date-malformed`. With a bad id beside them, both name `plan-bad-id`, as on `c7a3ff16`.

**Tests.** `tests/test_adopt_plan_value_order.py` runs `check` and `execute` on one hand-edited plan with local remotes. Its 20 rows pass here. The first round's 18 all fail against `c7a3ff16`'s `adopt-project.py`. Review round 1 added two, `short-commit` and `no-commit`: a finding about another key of a pin does not say how the digest was read. Both pass against `c7a3ff16` too, where no finding says how the digest was read: they keep that clause from spreading. Existing rows changed for the new behaviour:
- `test_adopt_pins.py`: the `not-a-list` spelling.
- `test_adopt_execute_refusals.py`: the four `pins:` rows of `BAD_KEYS`, and the `pins-text` end-to-end assertion.
- `test_adopt_plan_yaml_quoting.py`: the two pin rows of the #255 test, moved to a test of their own that expects `plan-pin-malformed` alone.
- `test_adopt_template_values.py`: `pins-surrogate`, now `plan-pin-unrecorded` alone.

Gate on `2b04aec2`, under the stub `gh`:
- `test_repo_hygiene.py`: 100 passed.
- The new file: 18 passed.
- `test_scaffold_leg_path_collisions.py`: 104 passed.
- The 31 files that name the codes this change touches: 1427 + 1618 (9 skipped) + 323 passed.
- `test_shape_doctor.py`: in four shards, 99 + 113 + 91 + 142 passed and 3 skipped, all 448 tests the file collects.

Round 1, on `c9652c9e`, under the stub `gh`: the new file, 20 passed; `test_adopt_pins.py` 146, `test_adopt_execute_refusals.py` 181, `test_adopt_plan_yaml_quoting.py` 518, `test_adopt_template_values.py` 78, `test_scaffold_leg_path_collisions.py` 104, `test_repo_hygiene.py` 100, `test_agents_md_codes.py` 47 and `test_readme_windows_quoting_sentence.py` 44 passed; `test_shape_doctor.py`, in four shards, 112 + 112 + 112 + 109 passed and 3 skipped.

Pre-tested at `2b04aec2` on the merge with `main` at `3864b18c`. `git merge-tree` has no conflicts (tree `41815b45`), and the tests ran in a checkout of that merge (commit `49099409`). Eight files, 1047 passed: the new file (18), `test_neutral_pin_entry_one_spelling.py` (4), `test_scaffold_pin_and_reuse.py` (75), `test_adopt_pins.py` (146), `test_adopt_execute_refusals.py` (181), `test_adopt_plan_yaml_quoting.py` (518), `test_adopt_template_values.py` (78) and `test_adopt_unsearchable_source.py` (27).

**Left for follow-up issues, not taken here:**
- First problems that still differ, outside the plan's options:
  - a bad id beside an unanswered entry: `check` says `plan-bad-id`, `execute` says `plan-unresolved`;
  - beside a source that has moved on: `plan-bad-id` against `plan-stale`;
  - beside a name the naming policy refuses: `plan-bad-id` against `naming-unclassified`;
  - an unsafe `org:` beside an unanswered entry: `unsafe-value` against `plan-unresolved`;
  - an unsafe `org:` beside a source that has moved on: `unsafe-value` against `plan-stale`;
  - an unsafe `org:` in a plan held to a repository: `unsafe-value` against `plan-not-the-source` (#716's L4-48 pins this one on purpose).
- `visibility: "private\nx"` is still two findings, #255's and #261's, by that code's own rule that both are listed.
- `COMMIT_RE` and `SHA256_RE` (`scripts/repo_shape.py:68-69`) use `$` with `.match`, so a pin's `commit:` or `tree_sha256:` that ends in a line break passes `_pin_row_problem`.
- An entry of `pins:` that is no pin is still spelled with `repr` (`` `pins:` entry 1 is 7 ``).

It lands by fast refresh (Brett Heap, 2026-10-15T22:48:35Z). Lane 1's reviewer found its head `c9652c9e` READY. That head's own Linux run failed only main's two rows from #974 (#1024), so update-branch merged main into it, cleanly, giving `74484781`, which is green on Linux and SonarCloud. The branch merged main three times and was never rebased; the last merge is update-branch's, clean, giving 2ede510, which is green on SonarCloud and Git Bash. Lane 1's gate passed on its tree: the 14 test files that run `adopt-project.py`'s `check` and `execute`, its pins, the doctor and the docs, 1,811 passed and 3 skipped.

Closes #901

Lane: openreposhape-1 (openRepoShape-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brettheap added a commit that referenced this pull request Oct 20, 2026
…ines is asked of the method again, its docstring says no verb reaches that count since #974, and a refusal that leaves a ref behind fails `assert_unchanged` (#1029) (#1039)

Since #974, the YAML reader refuses a second `members:` key at open, so
no verb reaches the count of two in `Family.rewritten` any more, and
#1026's rows no longer tested it. Two direct-API rows now ask it of
`write_members`, on text with two margin `members:` lines, and expect
`family-members-unrecognised` with the whole count and the file
unchanged.

The docstring of `rewritten` now says the reader that took the last of
two keys is past, that no verb reaches a count of two since #974, and
that the count stays as defence in depth. A count of none is still
reached through a verb.

`snapshot()` in `tests/test_family_add_refusals.py` now records every
ref, with an explicit `for-each-ref` format that also lists a ref to a
missing object, so a refusal that leaves a branch behind fails
`assert_unchanged`.

An unparsable `family.yaml` outside an act keeps its one-line refusal,
with no remedy line. No fixed remedy is true in every state, and every
verb reaches the check of the committed manifest, with its own remedy,
before it writes anything.

Closes #1029

Lane: openreposhape-3 (openRepoShape-3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant