👌 Fix quadratic inline tokenization on runs of special characters - #411
Closed
hdimer wants to merge 1 commit into
Closed
👌 Fix quadratic inline tokenization on runs of special characters#411hdimer wants to merge 1 commit into
hdimer wants to merge 1 commit into
Conversation
Long runs of characters that begin an inline rule but do not form a construct (bare `&`/`<`, incomplete entities, ...) were tokenised in O(n^2) time, from two causes: - the inline tokenizer's fallback appended to `state.pending` one character at a time; `str += ch` on an attribute cannot reuse the buffer in place, so each append is O(len). - the `entity` and `html_inline` rules matched their `^`-anchored regexes against `src[pos:]`, slicing the remaining source on every character. Accumulate `pending` through a list buffer materialised lazily, and anchor the entity/html regexes with `.match(src, pos)` instead of slicing. Rendering is now linear (e.g. `'&' * 500_000` drops from seconds to well under one); output is unchanged and the full suite passes.
chrisjsewell
pushed a commit
that referenced
this pull request
Sep 8, 2026
…cters
Long runs of characters that begin an inline rule but never complete a
construct -- bare `&`, `<`, `~`, `{`, incomplete entities -- were tokenised
in O(n^2) time, so a few hundred KB of such input stalls `render()` for
many seconds. Output was always correct; the cost is pure CPU.
Two independent quadratic factors, each hit once per character in a run:
1. `state.pending` accumulation. The inline tokenizer's fallback path did
`state.pending += state.src[state.pos]` one character at a time. Because
`pending` is an *attribute*, `str += ch` cannot use CPython's in-place
concatenation optimisation (the attribute holds a second reference), so
every append copies the whole accumulated string.
2. `src[pos:]` slicing in `entity` / `html_inline`. Both matched
`^`-anchored regexes against `state.src[pos:]`, copying the remainder of
the source on every `&` / `<`. On Python 3.10 this carried a second
quadratic factor inside the regex engine itself: before 3.11, `search()`
on a `^`-anchored pattern retries at every offset.
Neither is quadratic in the JavaScript markdown-it (rope-backed string
concatenation, and sticky/lastIndex regex matching), so this is specific to
the Python port.
Fix:
- `StateInline.pending` now accumulates through a lazily-materialised list
buffer via `append_pending()`, giving amortised O(1) appends. It is still
exposed as a plain `str` property, so existing readers and third-party
plugins doing `state.pending += x` keep working unchanged.
- `entity` and `html_inline` anchor with `.match(state.src, pos)` instead of
slicing; the leading `^` is dropped since `.match` already anchors at
`pos`. Neither pattern uses `\b` or lookbehind, so this is exactly
equivalent. `HTML_TAG_RE` has a single consumer; the separate
`HTML_OPEN_CLOSE_TAG_RE` is untouched.
All affected inputs now grow ~2.0x per doubling (linear) out to 640k
characters. `"&" * 400_000` drops from 6.8s to 1.3s on CPython 3.11, from
653s to 1.8s on CPython 3.10, and from 145s to 0.2s on PyPy. Runs of `[`
were also reported as superlinear, but once the `pending` quadratic is
removed they measure a flat 2.0x per doubling: that path is linear already
(bounded by the existing `skipToken` cache), just with a large constant.
Output is unchanged: a 47,936-case differential run over the repo fixtures,
the CommonMark spec, targeted constructs and fuzzed inputs across seven
presets compares rendered HTML, `renderInline`, and full token streams
byte-for-byte against the previous behaviour with zero differences. The
test suites of mdit-py-plugins, myst-parser, mdformat and rich are
unaffected.
Adds `test_long_special_char_runs_are_linear`, which asserts correct output
for the affected constructs and renders inputs that took ~16s before this
change, so a regression trips the global 10s test timeout.
Same class as the previously fixed #367 and #389. The root causes and this
fix were independently identified by Haim Dimer in #411, which this
supersedes.
Co-authored-by: Haim Dimer <haim@dimer.org>
chrisjsewell
added a commit
that referenced
this pull request
Sep 8, 2026
## Summary
Long runs of characters that begin an inline rule but never complete a
construct (bare `&`, `<`, `~`, `{`, incomplete entities, …) were
tokenised in **O(n²)** time, so a few hundred KB of such input stalls
`render()` for many seconds. Output was always correct; the cost is pure
CPU. This is the same class as #367 and #389.
Supersedes #411. The root causes and the core fix were independently
identified by @hdimer there, and the first commit here is code-identical
to that PR (credited as co-author, thank you!). The second commit closes
a gap found while auditing it that made the fix ineffective for a common
plugin configuration, and adds a few hardening changes.
## Root causes
Two independent quadratic factors, each hit once per character in a run:
1. **`state.pending` accumulation.** The inline tokenizer's fallback
path did `state.pending += state.src[state.pos]` one character at a
time. Because `pending` is an *attribute*, `str += ch` can't use
CPython's in-place concatenation optimisation (the attribute holds a
second reference), so every append copies the whole accumulated string.
2. **`src[pos:]` slicing in `entity` / `html_inline`.** Both matched
`^`-anchored regexes against `state.src[pos:]`, copying the remainder of
the source on every `&` / `<`. On Python 3.10 this carried a *second*
quadratic factor inside the regex engine: before 3.11, `search()` on a
`^`-anchored pattern retries at every offset.
Neither is quadratic in the JavaScript markdown-it (rope-backed concat,
sticky regex matching), so this is Python-port-specific.
## Fix
**Commit 1** (as in #411):
- `StateInline.pending` accumulates through a lazily-materialised list
buffer via a new `append_pending()` method, giving amortised O(1)
appends. It's still exposed as a plain `str` property, so existing
readers and plugins doing `state.pending += x` keep working.
- `entity` and `html_inline` anchor with `.match(state.src, pos)`
instead of slicing; the leading `^` is dropped since `.match` anchors at
`pos`. Neither pattern uses `\b` or lookbehind, so this is exactly
equivalent.
**Commit 2** (new):
- The buffered getter materialised with `self._pending += joined`,
itself an attribute concat. Any rule that reads `state.pending` before
its cheap character check therefore re-introduced the quadratic on every
character. The mdit-py-plugins `attrs` rule does exactly that, and
myst-parser's `attrs_inline` extension enables it: with it, `"~a~" *
400_000` still took **34 s** (5.7× per doubling) after commit 1. The
getter now moves the string into a local and drops the instance's
reference before concatenating, so CPython resizes it in place: **3.0 s,
2.0× per doubling**, verified on CPython 3.10–3.13.
- The setter no longer depends on `__init__` having run (a subclass
assigning `pending` before `super().__init__()` raised
`AttributeError`).
- `copy.copy(state)` no longer shares the mutable buffer with the
original.
## Results
`"&" * 400_000`, before → after:
| interpreter | before | after |
|---|---|---|
| CPython 3.10 | 653 s | 1.8 s |
| CPython 3.11 | 6.8 s | 1.3 s |
| CPython 3.13 | 6.6 s | 1.1 s |
| PyPy 7.3 | 145 s | 0.2 s |
All affected inputs now grow ~2.0× per doubling out to 640k characters.
Runs of `[` were also reported as superlinear, but once the `pending`
quadratic is removed they measure a flat 2.0× per doubling: that path
was already linear (bounded by the `skipToken` cache), just with a large
constant.
**Trade-off:** the property indirection costs ~2% on ordinary Markdown
(up to ~4.7% on inline-dense files) — measured over 55 inputs × 3
presets, 8 interleaved rounds. The regex half is a pure win at every
size. Memory is unchanged on realistic input. If wanted, most of the 2%
can be recovered by having `push()`/`pushPending()` read the private
fields directly; happy to do that as a follow-up once benchmarked
properly.
## Verification
- **Output unchanged:** 47,936-case differential (repo fixtures,
CommonMark spec, targeted constructs, 6k fuzzed inputs × 7 presets)
comparing HTML, `renderInline` and full token streams byte-for-byte: 0
differences.
- **Downstream:** mdit-py-plugins (511), myst-parser (1245), mdformat
(4282), rich and mdformat-gfm test suites give identical results against
this branch and `master`; a 61-document differential through a full
plugin stack shows 0 differing renders.
- Full suite passes on CPython 3.10, 3.11, 3.12, 3.13 and PyPy. Pinned
`ruff` and strict `mypy` clean.
- New tests: `test_long_special_char_runs_are_linear`,
`test_state_inline_pending_buffer_semantics`,
`test_pending_reader_rule_stays_linear`. Each linearity test exceeds the
global 10 s timeout on the pre-fix code.
## Notes for reviewers
- `HTML_TAG_RE`, `DIGITAL_RE` and `NAMED_RE` lose their `^` anchor
(they're now applied with `.match(src, pos)`). No external consumer was
found in any scanned package or via code search, but any third-party
`.search()` on them would now be unanchored. Probably worth a changelog
line. `HTML_OPEN_CLOSE_TAG_RE` is untouched.
- mdformat-gfm ships its own replacement `text` rule that still does
`state.pending +=`; it keeps its quadratic and gains nothing from this
fix.
- **Not covered here, follow-up PR:** with `html=True` (the `commonmark`
and `gfm-like` presets), runs of `<
Member
|
Thanks @hdimer, and apologies this sat for a while. You identified both root causes and the right fix here, and #423 (now merged) carries your change verbatim as its first commit, with you credited as co-author. While auditing it before merging we found one gap that #423 adds on top: the buffered Closing this in favour of #423. It will be in the next release. Much appreciated! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Long runs of characters that begin an inline rule but do not form a construct — bare
&/<, incomplete entities like&am&am…, etc. — were tokenised in O(n²) time. A single such character is fine; it is the count that blows up. For example,md.render("&" * 128_000)took over a second and grew quadratically.This is the same class as the previously fixed #367 and #389.
Root causes
Two independent quadratic factors, both hit once per character in these runs:
state.pendingaccumulation. The inline tokenizer's fallback path doesstate.pending += state.src[state.pos]one character at a time. Becausependingis an attribute,str += chcannot use CPython's in-place concatenation optimisation (the attribute keeps a second reference), so each append rebuilds the whole string — O(len) per character.src[pos:]slicing inentity/html_inline. These rules matched^-anchored regexes againststate.src[pos:], copying the remaining source on every&/<.In JavaScript
markdown-itneither is quadratic (rope-backed string concat, and the equivalent regexes are applied with a sticky/lastIndex match), so this is Python-port-specific.Fix
StateInline.pendingnow accumulates through a lazily-materialised list buffer (append_pending), giving amortised O(1) appends. It still presents as a plainstrto every reader (the getter joins the buffer on demand and caches).entityandhtml_inlineanchor their regexes with.match(state.src, pos)instead of slicing (the leading^is dropped, since.matchalready anchors atpos).HTML_TAG_REis only used byhtml_inline, so this is contained.Rendering is now linear in all these cases (measured slope ≈ 1.0, including the
html=Truehtml_inlinepath);"&" * 500_000drops from seconds to well under one. Output is unchanged — the full test suite passes.Tests
Added
test_long_special_char_runs_stay_linearintests/test_api/test_main.py: it asserts correct rendering for the affected constructs and renders a large input whose previous O(n²) behaviour would exceed the global 10s test timeout. Fullpytest tests/passes (982).ruff check/ruff format(0.15.12) andmypy(1.20.2, strict) are clean.Disclosure: this fix was developed with AI assistance; I reviewed it and stand behind it.