fix(md-exec): heredoc body containing a ``` line is mistaken for the end of the code block - #259
batuhankocyigit wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Disclosure: I'm not affiliated with Circle — an external community contributor, not a maintainer. I have no write access to this repository, so any review state I set (approval or change request) carries no merge authority and is advisory only. Please treat this as one contributor's technical assessment, and defer to Circle maintainers for the binding review.
Full executable verification this time (Python is in my environment, unlike the Rust/shell gaps I've disclosed elsewhere) — every claim reproduces:
- Bug confirmed on
main: your exact repro file fails with the exact warning quoted (here-document at line 3 delimited by end-of-file (wanted 'EOF')), 0 passed 1 failed. - Fix confirmed on your branch: same repro passes; the parser now correctly carries the full heredoc body — including the embedded
```line — into the command. - Self-test suite: 29/29 on main, 30/30 on your branch — and this time the pre-existing count in the PR body is right too.
- Real-world exposure confirmed:
docs/running-an-arc-node.mdhas 3 heredocs on main, andmake test-unit-contractdoes runmd-exec.pyovercrates/quake/tests(Makefile line 262), so the parser is live test infrastructure, not just self-documentation. - Scope check: the diff is exactly the one condition (
and heredoc_delimiter is None) plus an explanatory comment and the regression case inmd-exec.md. The state machine already clearsheredoc_delimiteron the delimiter line before the next fence check, so the fix composes correctly with the existing tracking — no new state introduced.
One edge case worth a sentence in the comment, though I don't consider it blocking: if a code block contains a heredoc whose delimiter line never appears (malformed markdown), the parser now also skips the real closing fence and silently swallows the rest of the file into that command. Pre-fix, the same malformed input produced a confusing-but-visible failure. That's a strictly-broken-input scenario and the old behavior wasn't good either, so I'd merge as-is — but if you want to harden it, resetting heredoc_delimiter with a warning when EOF is reached inside an open heredoc would make the failure mode loud again.
Also appreciated the restraint in the notes: not "fixing" the 4-dot ellipsis ambiguity you found while fuzzing, because the spec only defines ..., is the right call — a behavior-defining discussion first is exactly how that should go.
LGTM. Small, correctly-scoped, honestly-documented, and verified end-to-end.
fix(md-exec): heredoc body containing a ``` line is mistaken for the end of the code block
The bug
scripts/md-exec.py'sMarkdownParser.parse()detects code-fenceboundaries by checking whether a line, stripped, starts with
```.This check runs unconditionally on every line inside a code block —
including lines that are part of an open heredoc body being fed to a
shell command. If a heredoc body contains a line starting with
```(for example, a heredoc that writes out a markdown file, or a snippet that
itself documents markdown syntax), the parser treats that line as the end
of the code fence, cutting the command short mid-heredoc. The shell then
receives an unterminated heredoc.
Reproduction — this file:
fails today with:
heredoc_delimiteris already tracked by the parser (set when a<< DELIMcommand starts, cleared when the delimiter line is seen), but thefence-detection check doesn't consult it.
This isn't just a contrived case:
docs/running-an-arc-node.mdalreadyuses heredocs to write out full files (
cat << "EOF" > ~/.arc_env, twosudo tee ... <<EOFsystemd unit blocks), andmake test-unit-contractruns
python3 scripts/md-exec.py crates/quake/testsas part of the testsuite — so this is live infrastructure, not just the tool's own
self-documentation.
The fix
One-line change: only treat a
```-prefixed line as a fence boundarywhen we're not currently inside an open heredoc body.
Testing
scripts/md-exec.md(the tool's ownself-test suite, run via
python3 md-exec.py md-exec.md) covering aheredoc body with an embedded ``` line.
git stashthe.pychange, rerun — reproduces the exact bash heredoc warning above) and
passes with it.
regressions.
python3 -m py_compile scripts/md-exec.py— compiles cleanly.Notes for reviewers
touched.
check_match) functionseparately looking for bugs there; found one genuinely ambiguous edge
case (4+ consecutive literal dots, e.g.
foo....bar), but the docs onlydefine behavior for exactly
..., so I didn't treat that as a bug orinclude a "fix" for it here — happy to open a separate discussion/PR if
you'd like that behavior formally defined.