repo_shape.py: a flow collection with an unmatched ] or } is read, not refused, so k: {a"]:} reads {'a"]': None} since #284 (a quote toggle used to swallow the :}) and k: [a]] reads ['a]'], where PyYAML refuses both #301

Lane: openreposhape-5 (openRepoShape-5)

Found by this lane's re-verification of PR #284 (fix/yaml-quote-boundaries at b72ab0f, merged as b30595c) as NIT-B, which the review ruled acceptable as a follow-up and not a should-fix. #284's own body lists the class under "Left out" ("A plain flow item followed by a nested flow, and unbalanced brackets"). The class is old, and #284 made part of it newly reachable. Every result below is on main b30595c, and the contrast is its parent c3a9645.

What is wrong. scripts/repo_shape.py _flow checks only that a flow collection opens with [ or { and ends with ] or }. _split_flow counts [ and { up and ] and } down and never asks whether the count went below zero. So a body that closes a bracket it never opened is split at its top-level commas and read, with the stray closer kept inside an item or a key. A plain item that holds a [ is the same defect one level in: _split_flow('x [a, b]') returns ['x [a, b]'], and _scalar reads that as text.

Until #284, one slice of this was refused only by accident. The quote toggle took a " in the middle of a plain item for the start of a quoted scalar and swallowed what followed, so k: {a"]:} ended in flow mapping entry without a key: 'a"]:'. #284 reads a quote only where a scalar begins, as YAML does, so nothing hides the ] any more, and k: {a"]:} reads as its quote-free twin k: {a]x:} already did.

cd <an openRepoShape checkout at b30595c>
python3 - <<'EOF'
import sys

sys.path.insert(0, "scripts")  # run from the repository root
from repo_shape import YamlError, parse_yaml

try:
    import yaml  # PyYAML, as the oracle only
except ImportError:
    yaml = None

DOCS = [
    "pins: [a, b]]",
    'k: {a"]:}',
    'k: {a]x:}',
    'k: {]:}',
    "k: [a]]",
    "k: [a}]",
    "k: [x [a, b]]",
    "k: [a[]",
    "k: {a:}#}: b}",
    'k: {}#":}',
]


def ours(doc):
    try:
        return repr(parse_yaml(doc + "\n"))
    except YamlError as e:
        return f"refused: {e}"


def pyyaml(doc):
    try:
        return repr(yaml.safe_load(doc + "\n"))
    except yaml.YAMLError as e:
        return f"refused: {type(e).__name__}"


for doc in DOCS:
    print(repr(doc))
    print("    repo_shape:", ours(doc))
    if yaml:
        print("    PyYAML:    ", pyyaml(doc))
EOF

Observed on b30595c (Python 3.12.3, PyYAML 6.0.3, Linux):

'pins: [a, b]]'
    repo_shape: {'pins': ['a', 'b]']}
    PyYAML:     refused: ParserError
'k: {a"]:}'
    repo_shape: {'k': {'a"]': None}}
    PyYAML:     refused: ParserError
'k: {a]x:}'
    repo_shape: {'k': {'a]x': None}}
    PyYAML:     refused: ParserError
'k: {]:}'
    repo_shape: {'k': {']': None}}
    PyYAML:     refused: ParserError
'k: [a]]'
    repo_shape: {'k': ['a]']}
    PyYAML:     refused: ParserError
'k: [a}]'
    repo_shape: {'k': ['a}']}
    PyYAML:     refused: ParserError
'k: [x [a, b]]'
    repo_shape: {'k': ['x [a, b]']}
    PyYAML:     refused: ParserError
'k: [a[]'
    repo_shape: {'k': ['a[']}
    PyYAML:     refused: ParserError
'k: {a:}#}: b}'
    repo_shape: {'k': {'a:}#}': 'b'}}
    PyYAML:     {'k': {'a': None}}
'k: {}#":}'
    repo_shape: {'k': {'}#"': None}}
    PyYAML:     {'k': {}}

On c3a9645, the commit before #284, the same script prints the same for eight of the ten documents. The two that differ are refused: 'k: {a"]:}' with flow mapping entry without a key: 'a"]:', and 'k: {}#":}' with flow mapping entry without a key: '}#":'.

PyYAML refuses eight of the ten. It reads k: {a:}#}: b} as {'k': {'a': None}} and k: {}#":} as {'k': {}}, only because it accepts a # with no space before it as a comment, which YAML forbids. This reader's reads of those two are neither. (#284's body says PyYAML refuses k: {a:}#}: b}. It reads it.)

On seed 2490250 (785,646 documents: every string of up to 4 characters drawn from a , : space " ' [ ] \ { } # -, plus 100,000 random strings of up to 14, each placed in six contexts: a flow sequence, a flow mapping, a value, an item, a key and a nested item; PyYAML 6.0.3 as the oracle only), b30595c reads 775 flow-mapping documents that PyYAML refuses and c3a9645 refused. It also reads 13 more that PyYAML reads and c3a9645 refused, like k: {}#":}. The review measured this against b72ab0f, and it was re-run for this issue against b30595c with the same counts. The sketch below refuses 617 of the 775. The other 158 are a different class and are not claimed here: 145 of them open the body, after any spaces, with :, , or # (k: {:":}, k: {,a":}, k: {#":}), which the review counted with a separate follow-up, a key that begins with an indicator character.

Why it matters. YamlError's docstring is the reader's contract: "Raising rather than guessing is the whole point: a validator that silently mis-reads its own contract file is worse than one that will not start." A manifest, pin, plan or family file with one stray ] or } typed into a flow list or mapping reads as data instead (pins: [a, b]] reads {'pins': ['a', 'b]']}), where PyYAML refuses the file. The project's validators and a stock YAML parser then disagree about what the file says.

No valid YAML document is misread by this. Every document above is one PyYAML refuses, except the two it reads through its # leniency. No writer in this tree emits a stray closer, so only a hand edit reaches it, which is why the review ruled it a follow-up and not a blocker. What #284 changed is the reach: on seed 2490250, 775 more flow-mapping documents are read where main refused them, and 617 of those are this class.

Expected. Refuse a flow body that closes a bracket it never opened, and a plain flow item that holds a [ or {. In YAML a plain scalar inside a flow collection ends at any of [ ] { } ,. The review tested a two-line sketch for the first half, in _split_flow:

        elif ch in "]}":
            depth -= 1
            if depth < 0:
                raise YamlError(f"a flow collection closes a bracket it never opened: {body!r}")

On the ten documents above it refuses eight, all but k: [x [a, b]] and k: [a[]. As measured by the review on seed 2490250 against b72ab0f (the first four lines were re-run for this issue against b30595c with the same counts):

• Flow-mapping documents newly read: 775 to 158. The 158 are the other class above.
• Documents PyYAML reads that this reader reads and c3a9645 refused: 13 to 0.
• Invalid documents c3a9645 read that are now refused: 73,380 to 118,578, which is 45,198 more.
• Valid documents regressed: the same 768 as on b30595c, every one a refusal and every one a bracket in a key, which repo_shape.py: a quote opens a quoted scalar only where one begins and must close there, so k: [it's, a] reads two items, k: it's fine # note drops its comment, and an unclosed "abc or a mid-key [ is refused, not misread (#249, #250) #284 refuses on purpose. Fixed: the same 13,752. Crashes: 0.
• The targeted reader test files passed: 359 passed, 1 skipped (the environment skip in test_adopt_plan_escapes), 3 deselected.

The sketch does not touch a plain item that holds an opener (k: [x [a, b]], k: [a[]). Five of the 158 documents it leaves read end their body one bracket level deep, an opener never closed (k: {t-'': x{} reads {"t-''": 'x{'}). That needs its own rule, and no fix for it was measured. scripts/repo_shape.py is the digest-pinned copy in every scaffolded root, so a fix reaches projects through update-shape.py. Add a row for each document above, with the two PyYAML reads refused by design as k: "a"#c is.

NIT-A, a test docstring. The module docstring of tests/test_yaml_quote_boundaries.py (lines 30-32) says every refused document is one PyYAML refuses, "except k: "a"#c". There are eight exceptions: that one, and seven BRACKET_IN_A_KEY rows (- a[b]: c, a[b]: c, k:\n - a{b}: c, - note "[1]": x and three more) that PyYAML reads and the subset refuses on purpose. Whoever fixes the class should correct that sentence.

Found by. handoffs/openRepoShape/attachments/lane-openRepoShape-5/reviewA-report.md in opensoft/brett-wip (NIT-A and NIT-B of its "Re-verification of b72ab0f"). The ten-document run on b30595c and c3a9645, the seed 2490250 counts for b30595c and the eight-disagreement count for NIT-A were re-run for this issue.

Not claimed.

🤖 Generated with Claude Code