[brettheap]

Lane: openreposhape-5 (openRepoShape-5)
Claim: #248 (comment)

What

The YAML reader kept a block scalar (key: |, key: >) as one _Rec whose text was "\x00BLOCK\x00" + key + "\x00" + body. _parse_mapping took it apart with split("\x00", 3), and both _parse_mapping and _parse_sequence recognised it by that text prefix. Now:

• _Rec has a block slot holding (key, body). It is set only on the record a block scalar's header makes, and that record's text is empty.
• _tokenise builds the record with the pair. _parse_mapping and _parse_sequence test rec.block, not a text prefix.
• _BLOCK_SENTINEL is gone. Its comment, which named the collision as a known limit, becomes a _Rec docstring saying why nothing can collide now.

No other function in the file changes, and no writer changes.

Why

Since #199 the reader resolves \x00, \u0000 and \U00000000 in a double-quoted key, which is valid YAML. A key holding a NUL was cut short at the NUL, and the rest of it read into the body. A raw NUL did the same before #199. The marker was also only a text prefix, so an ordinary line beginning with a raw NUL + BLOCK + NUL was taken for a block record:

document	main (150c41d)	this branch
"a\x00b": |	{'a': 'b\x00text\n'}	{'a\x00b': 'text\n'} (PyYAML agrees)
"a\x00b": text (control)	{'a\x00b': 'text'}	same
<NUL>BLOCK<NUL>k<NUL>v: 1	{'k': 'v: 1'}	{'\x00BLOCK\x00k\x00v': 1}
- <NUL>BLOCK<NUL>x	bare ValueError out of load_yaml	['\x00BLOCK\x00x']
k:\n <NUL>BLOCK<NUL>x<NUL>y	{'k': {'x': 'y'}}	YamlError, as k:\n xyz is

A structure that is not a string has no character to collide with, so this removes the class and not just \x00. A different separator only moves the collision, and refusing a NUL in a block key leaves the prefix misreads in place (a \x01 separator and a NUL refusal were each tried as mutants, and both are caught by the new tests).

A NUL in a key is read, not refused. YAML allows it escaped. PyYAML 6.0.3 reads every escaped row the way this branch does. The reader already read the same key over a plain value, and double_quoted writes a NUL as \x00. A raw NUL, which YAML forbids, is read the way the reader reads a raw NUL in any other key (a<NUL>b: text reads as {'a\x00b': 'text'} on main). Refusing raw control characters is a reader-wide decision and is left out (see below).

Issue row 2 ("a\^@b": >) does not reproduce, not even on a02240a: \^@ is no escape the reader (or YAML) resolves, so the key keeps its backslash on main and here. It is pinned as a control.

Evidence

• tests/test_yaml_block_scalar_keys.py (new, 51 tests). On 150c41d: 41 fail and 10 pass, and the 10 that pass are controls. On this branch: 51 pass.

• Targeted runs on 9be6287:

  • the new file plus test_yaml_flow_quotes.py, test_load_yaml_refusals.py, test_adopt_plan_escapes.py, test_manifest_validator.py, test_pin_validator.py, test_neutral_pin_validator.py, test_repo_hygiene.py, test_cli_reference.py and test_adopt_plan.py: 326 passed, 1 skipped (SHAPE_MEDXEHR_CLONE unset);
  • test_shape_doctor.py: 181 passed.

• Mutation: each fix hunk reverted on its own (_tokenise, _parse_sequence, _parse_mapping, _Rec) fails 46, 4, 48 and 51 tests. Of the narrow alternatives, refusing a NUL in a block key fails 41, including every marker-prefixed-line test. A \x01 separator fails the every-character round-trip test. Dropping the sequence check fails the placement test, and taking the pair reversed fails 46.

• Every code point in a written block key, double_quoted("a" + chr(c) + "b") under | and >: all 2,228,224 documents read back, and PyYAML agrees on every one. Main misreads exactly 2 of them, both NUL.

• A base-against-head fuzz of 200,000 documents built around block scalars finds 0 differences outside the class. All 23,298 documents with no NUL in any spelling read the same. Over the same corpus, main lets 4,250 bare ValueErrors out of parse_yaml, and this branch none.

• Reach: all 14 tracked YAML files read the same on main and here, rendered templates too. Block scalars in tests.yml, path-classification.yaml and both validate.yml templates read unchanged. No writer in this repository emits the construct:

  • no shipped Python writes a block-scalar header;
  • adopt-project.py's keys are literals, and its one variable key comes from summary_of's fixed names;
  • y never writes |, > or a raw NUL plain;
  • the doctor's placement plan uses y/emit, and family.py, update-shape.py and bump-leg.py write literal keys.

  So the defect shows only on a file a person edits or another tool writes. Every YAML file every tool reads goes through this reader.

Composition

git merge-tree --write-tree from 9be6287:

• origin/main 60d08a1 (with adopt-project.py plan raises UnicodeEncodeError in its terminal report, after the plan file is written, for a printable name stdout cannot encode (日本.md under cp1252, as on a redirected Windows stdout), so plan exits 1 before plan written to #214 and shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230): clean.
• fix/yaml-quote-boundaries (scripts/repo_shape.py treats a quote character in the MIDDLE of a plain scalar as opening a quote, so k: [it's, a] reads ["it's, a"] where its block twin reads ["it's", "a"], k: it's fine # note keeps the comment in the value, and #244 turns k: [a"b\", c] from right (by accident) into one item #249/scripts/repo_shape.py _scalar takes any text that starts and ends with a quote for a quoted scalar and anything else for plain text, so an unterminated "abc reads as "abc, - "a\" reads as a\, and "a" b "c" reads as a" b "c, where PyYAML refuses each #250) at 21fc213: clean; the review also passed 363 tests on this branch merged with it and scripts/repo_shape.py reads ? a: b in a flow mapping as the literal key ? a, where the block reader refuses explicit keys, and refuses {a: } and {a:} as not a flow collection: '' instead of reading {'a': None} #252 together.
• fix/yaml-flow-mapping-keys (scripts/repo_shape.py reads ? a: b in a flow mapping as the literal key ? a, where the block reader refuses explicit keys, and refuses {a: } and {a:} as not a flow collection: '' instead of reading {'a': None} #252) at 2438190: clean. The merged tree passes this file, test_yaml_flow_quotes.py and test_yaml_flow_mappings.py (114 passed).
• fix/scaffold-referent-chain-links (scaffold-project.py --referent-chain writes each middle link into project.yaml as typed (naming_block, scripts/shape_materialize.py:1010-1011) and checks none, so a link holding # scaffolds all three repositories, exits 0, and pushes a manifest the project's own validators refuse as yaml-unparsable, and a link holding ' or ] is recorded as a different chain #251) at 114bec9: clean.

Left out

• Refusing the raw control characters YAML forbids (PyYAML: unacceptable character #x0000). The reader takes them everywhere today, and double_quoted's docstring says so. A refusal belongs to the whole reader, not to one path. The plain-twin tests here say those rows become refusals together if that changes. A follow-up issue is proposed.
• Honouring + chomping and the indentation digit is documented as accepted, not honoured, and is unrelated.

Closes #248

🤖 Generated with Claude Code

Summary by Sourcery

Make block-scalar parsing preserve arbitrary keys and distinguish genuine block records from ordinary YAML lines.

Bug Fixes:

• Preserve NUL-containing YAML keys in block scalars instead of splitting them into the key and body.
• Prevent ordinary lines beginning with the former block-scalar marker from being misclassified as block scalars.

Enhancements:

• Represent block-scalar records with structured key and body data, eliminating sentinel-based parsing collisions.

Tests:

• Add comprehensive coverage for escaped and raw NUL keys, marker-prefixed lines, nested and sequence contexts, character round trips, and unchanged block bodies.

[brettheap]

(lane openRepoShape-5, 2026-10-11T22:46Z) @codex review

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 23 hours and 56 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T23:08:07.259146Z	9be6287	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[sourcery-ai]

Reviewer's Guide

The PR eliminates the collision-prone NUL-delimited representation of block scalars by storing their key and body as a structured pair on _Rec, updates parsers to use that metadata, and adds comprehensive regression tests covering NUL keys, former sentinel-like lines, nesting, sequences, and body preservation.

Sequence diagram for parsing a YAML block scalar with a NUL key

sequenceDiagram
    participant Input as YAML input
    participant Tokenise as _tokenise
    participant Rec as _Rec
    participant Mapping as _parse_mapping

    Input->>Tokenise: _tokenise(text)
    Tokenise->>Tokenise: _opens_block_scalar(kv)
    Tokenise->>Tokenise: _block_scalar_body(raw_lines, i, indent)
    Tokenise->>Tokenise: _block_scalar_text(kv[1], body)
    Tokenise->>Rec: create with block=(kv[0], body)
    Note over Rec: text is empty, key and body remain separate
    Tokenise-->>Mapping: list[_Rec]
    Mapping->>Rec: read rec.block
    Rec-->>Mapping: key, body
    Mapping-->>Input: {key: body}

Loading

Flow diagram for avoiding block scalar marker collisions

flowchart LR
    A[YAML block scalar header] --> B[_tokenise]
    B --> C[_Rec.block = key, body]
    C --> D{rec.block is not None}
    D -->|yes| E[_parse_mapping uses key and body]
    D -->|no| F[Parse ordinary text normally]
    G[Key or line contains NUL or marker-like text] --> F

Loading

File-Level Changes

Change	Details	Files
Replace the NUL-delimited block-scalar encoding with explicit structured metadata on each token record.	Add an optional (key, body) block field to _Rec. Have _tokenise store block-scalar keys and processed bodies as a pair while leaving record text empty. Have mapping and sequence parsing detect block records through the structured field and unpack the pair directly. Remove the _BLOCK_SENTINEL constant and document the collision-avoidance rationale on _Rec.	scripts/repo_shape.py
Add regression coverage for NUL-containing keys and lines that resemble the former block marker.	Verify escaped and raw NUL keys round-trip for literal and folded block scalars, including placement in sequences and nested mappings. Verify marker-prefixed ordinary keys and sequence items are no longer interpreted as block records. Check block bodies, chomping behavior, plain-value twins, broad character coverage, and file-based load_yaml reads.	tests/test_yaml_block_scalar_keys.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#248	Prevent block-scalar records from misparsing keys containing NUL characters, including escaped NULs introduced by #247, so that documents such as `"a\x00b":	` preserve the complete key and correct body.	✅
#248	Replace the NUL-delimited block-scalar sentinel representation with a collision-safe structured representation, and update mapping and sequence parsing accordingly.	✅
#248	Prevent ordinary input beginning with the former block-scalar sentinel from being mistaken for a block scalar, while adding regression coverage for the affected key, sequence, nesting, and marker-prefix cases.	✅

Possibly linked issues

• scripts/repo_shape.py: the double-quoted reader that #247 adds (fix/adopt-plan-linebreak-names, not yet on main) resolves \x00 inside a block-scalar KEY, and _parse_mapping splits the NUL-joined sentinel record, so "a\x00b": | reads as the key a with the body b<NUL>text where main reads the literal key a\x00b #248: PR directly fixes issue scripts/repo_shape.py: the double-quoted reader that #247 adds (fix/adopt-plan-linebreak-names, not yet on main) resolves \x00 inside a block-scalar KEY, and _parse_mapping splits the NUL-joined sentinel record, so "a\x00b": | reads as the key a with the body b<NUL>text where main reads the literal key a\x00b #248 by replacing NUL-joined block records with separate key/body fields and adding regression tests.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 9be6287e8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".