👌 Fix quadratic inline tokenization on runs of special characters

[chrisjsewell]

Summary

Long runs of characters that begin an inline rule but never complete a construct (bare &, <, ~, {, incomplete entities, …) were tokenised in O(n²) time, so a few hundred KB of such input stalls render() for many seconds. Output was always correct; the cost is pure CPU. This is the same class as #367 and #389.

Supersedes #411. The root causes and the core fix were independently identified by @hdimer there, and the first commit here is code-identical to that PR (credited as co-author, thank you!). The second commit closes a gap found while auditing it that made the fix ineffective for a common plugin configuration, and adds a few hardening changes.

Root causes

Two independent quadratic factors, each hit once per character in a run:

1. state.pending accumulation. The inline tokenizer's fallback path did state.pending += state.src[state.pos] one character at a time. Because pending is an attribute, str += ch can't use CPython's in-place concatenation optimisation (the attribute holds a second reference), so every append copies the whole accumulated string.
2. src[pos:] slicing in entity / html_inline. Both matched ^-anchored regexes against state.src[pos:], copying the remainder of the source on every & / <. On Python 3.10 this carried a second quadratic factor inside the regex engine: before 3.11, search() on a ^-anchored pattern retries at every offset.

Neither is quadratic in the JavaScript markdown-it (rope-backed concat, sticky regex matching), so this is Python-port-specific.

Fix

Commit 1 (as in #411):

• StateInline.pending accumulates through a lazily-materialised list buffer via a new append_pending() method, giving amortised O(1) appends. It's still exposed as a plain str property, so existing readers and plugins doing state.pending += x keep working.
• entity and html_inline anchor with .match(state.src, pos) instead of slicing; the leading ^ is dropped since .match anchors at pos. Neither pattern uses \b or lookbehind, so this is exactly equivalent.

Commit 2 (new):

• The buffered getter materialised with self._pending += joined, itself an attribute concat. Any rule that reads state.pending before its cheap character check therefore re-introduced the quadratic on every character. The mdit-py-plugins attrs rule does exactly that, and myst-parser's attrs_inline extension enables it: with it, "~a~" * 400_000 still took 34 s (5.7× per doubling) after commit 1. The getter now moves the string into a local and drops the instance's reference before concatenating, so CPython resizes it in place: 3.0 s, 2.0× per doubling, verified on CPython 3.10–3.13.
• The setter no longer depends on __init__ having run (a subclass assigning pending before super().__init__() raised AttributeError).
• copy.copy(state) no longer shares the mutable buffer with the original.

Results

"&" * 400_000, before → after:

interpreter	before	after
CPython 3.10	653 s	1.8 s
CPython 3.11	6.8 s	1.3 s
CPython 3.13	6.6 s	1.1 s
PyPy 7.3	145 s	0.2 s

All affected inputs now grow ~2.0× per doubling out to 640k characters. Runs of [ were also reported as superlinear, but once the pending quadratic is removed they measure a flat 2.0× per doubling: that path was already linear (bounded by the skipToken cache), just with a large constant.

Trade-off: the property indirection costs ~2% on ordinary Markdown (up to ~4.7% on inline-dense files) — measured over 55 inputs × 3 presets, 8 interleaved rounds. The regex half is a pure win at every size. Memory is unchanged on realistic input. If wanted, most of the 2% can be recovered by having push()/pushPending() read the private fields directly; happy to do that as a follow-up once benchmarked properly.

Verification

• Output unchanged: 47,936-case differential (repo fixtures, CommonMark spec, targeted constructs, 6k fuzzed inputs × 7 presets) comparing HTML, renderInline and full token streams byte-for-byte: 0 differences.
• Downstream: mdit-py-plugins (511), myst-parser (1245), mdformat (4282), rich and mdformat-gfm test suites give identical results against this branch and master; a 61-document differential through a full plugin stack shows 0 differing renders.
• Full suite passes on CPython 3.10, 3.11, 3.12, 3.13 and PyPy. Pinned ruff and strict mypy clean.
• New tests: test_long_special_char_runs_are_linear, test_state_inline_pending_buffer_semantics, test_pending_reader_rule_stays_linear. Each linearity test exceeds the global 10 s timeout on the pre-fix code.

Notes for reviewers

• HTML_TAG_RE, DIGITAL_RE and NAMED_RE lose their ^ anchor (they're now applied with .match(src, pos)). No external consumer was found in any scanned package or via code search, but any third-party .search() on them would now be unanchored. Probably worth a changelog line. HTML_OPEN_CLOSE_TAG_RE is untouched.
• mdformat-gfm ships its own replacement text rule that still does state.pending +=; it keeps its quadratic and gains nothing from this fix.
• Not covered here, follow-up PR: with html=True (the commonmark and gfm-like presets), runs of <![CDATA[, <!--, <? and <!a in inline context are still quadratic because the regex sub-patterns rescan to end-of-input on every attempt (<![CDATA[ × 40k takes ~245 s). This is inherited from upstream (it's quadratic in markdown-it JS too), pre-existing, and a different code path, so it's kept separate. A validated fix (terminator quick-reject, zero output change) is ready.
• The changelog is left for the release, following prior PRs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.88%. Comparing base (bff75ed) to head (856320a).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #423      +/-   ##
==========================================
+ Coverage   95.85%   95.88%   +0.02%     
==========================================
  Files          64       64              
  Lines        3619     3643      +24     
==========================================
+ Hits         3469     3493      +24     
  Misses        150      150              

Flag	Coverage Δ
pytests	95.88% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.