Discussion: Markdown correctness and extension points for downstream integrations

[RobinDev]

This is a discussion about which generally useful behavior belongs in Tempest and which should remain a downstream extension. The examples below were checked against upstream main / 1.2.2 with highlighting disabled.

1. Make configured rules work inside nested tokens

prependRules() and removeRules() affect the outer parser, but paragraph, heading, list, link, and table tokens call forToken() with hard-coded rule lists. A custom ^ rule, for example, fires for ^x at document start but not for a ^x, - ^x, or ## ^x. This prevents downstream syntax from being implemented as an ordinary Tempest rule throughout a document.

This overlaps #13 (configurable rules/tokens) and #18 (removeRules() inside nested contexts). The likely requirement is composable rules per context (block versus inline), rather than blindly copying block rules into every inline parser. It would be useful to know your preferred API before attempting a fix.

2. Correct generic link and image parsing

Input	Current result	Expected meaning
[x](/a "Title")	href="/a "Title""	href="/a" and title="Title"
[x](a b)	A link with a space in href	Literal text; the unbracketed destination is invalid
[x](<a b>)	href="&lt;a b&gt;"	A link to a b, with appropriate URL encoding
	Title folded into src	Separate image source and title

The general rule should parse destination, optional title, escapes, entities, and balanced parentheses before rendering. A renderer hook for parsed links/images would let integrations use those services without rewriting serialized <a> and <img> tags.

3. Preserve raw HTML and distinguish literal text from markup

<script>const x = "**a**";</script> currently becomes <script>const x = "<strong>a</strong>";</script>. The script body must remain raw. This is separate from #20, which asks whether raw HTML should be allowed, escaped, or stripped.

Other generic inline cases include a\*b retaining its backslash and a_b_c becoming emphasis. Escaping plain text/entities correctly in text tokens, while keeping raw HTML and code opaque, would nice.

4. Finish structural Markdown before adding CMS syntax

Examples still requiring downstream handling include #titre being parsed as a heading without the required separator; 2. First losing its list start number; ~~~ fences being parsed as strikethrough; and nested/loose lists and hard line breaks. Automatic heading IDs should also be optional or render-configurable.

Several narrower issues/PRs already cover parts of this: #27/#29, #28/#30, #31/#32, #33/#34, and #35/#36. I am listing them here for context. GFM task lists (- [x] Done, currently rendered as an empty-URL link) could be an optional extension.

5. Keep application behavior downstream

A small set of stable rule and rendering hooks in Tempest would make cool thing possible without pre-processing Markdown or post-processing HTML (Example).

@brendt, which of these would you want in the core parser, which as optional rules, and which should be solved through extension hooks? In particular, is the rule-configuration direction in #13/#18 the right prerequisite? I will wait for your feedback before opening any further PRs for the items in this discussion.

[brendt]

1. Make configured rules work inside nested tokens

Yes this is what #13 and #18 are about. I haven't really thought about an API yet. Feel free to suppose something in #13

[brendt]

2. Correct generic link and image parsing

Is this part of the official spec? If so we should support it

[brendt]

3. Preserve raw HTML and distinguish literal text from markup

Would be good to fix indeed

[brendt]

4. Finish structural Markdown before adding CMS syntax

Sure, any bug can be fixed

[brendt]

5. Keep application behavior downstream

I don't really understand. Care to elaborate?

[RobinDev]

2. Correct generic link and image parsing

Is this part of the official spec? If so we should support it

Yes — CommonMark 0.31.2, §6.3 Links and §6.4 Images. An inline link is [text](destination "title"): the title is a separate optional element, the destination may be wrapped in <...> (which is how a destination containing spaces is written), and a bare destination containing spaces makes the whole construct literal text.

Current main next to a reference implementation (league/commonmark, same input, highlighting off):

Input	Tempest main	CommonMark
[x](/a "Title")	<a href="/a &quot;Title&quot;">x</a>	<a href="/a" title="Title">x</a>
[x](a b)	<a href="a b">x</a>	[x](a b)
[x](</my uri>)	<a href="&lt;/my uri&gt;">x</a>	<a href="/my%20uri">x</a>
	<img src="/a &quot;Title&quot;" alt="alt">	<img src="/a" alt="alt" title="Title" />
[x](/a&amp;b)	<a href="/a&amp;amp;b">x</a>	<a href="/a&amp;b">x</a>

The last row is one I hadn't listed in the issue: entities in a destination are escaped twice.

Balanced parentheses and backslash escapes in destinations are already correct since #25 — [x](/a(b)c) and [x](/a\)b) both match CommonMark on main.

I'll open a PR that parses destination, optional title, angle-bracket destinations and entities before rendering, and carries title on LinkToken / ImageToken so it isn't folded into src/href.

[RobinDev]

5. Keep application behavior downstream

I don't really understand. Care to elaborate?

That point was badly written, sorry. It's two related asks, and the first one already exists in the codebase.

1. A rendering hook per token. ImageToken::parse() already does exactly what I'm describing:

if ($parser->imageFactory) {
    return $parser->imageFactory->create($this->src, $this->alt)->html;
}

That's the right shape. It's just wired to one concrete collaborator (ResponsiveImageFactory), for one token. Generalising it — an integration registers a renderer for a token class and receives the parsed token — is all point 5 asks for.

Why it matters downstream: in Pushword, images go through our own media pipeline (responsive sizes configured per site, an optional link wrapper, a placeholder comment when the file is missing). With no hook, we replace every image with a private-use-area marker before parsing, then substitute the rendered HTML back after: TempestImageRestorer. We do the same for link titles with a PWTITLE0TOKEN marker. It works, but it means the Markdown you parse is not the Markdown the user wrote, and any change to your output format silently breaks our string matching.

From our side it would look like:

$markdown->renderToken(ImageToken::class, fn (ImageToken $token) => $media->render($token->src, $token->alt, $token->title));
$markdown->renderToken(LinkToken::class, fn (LinkToken $token) => $links->render($token->content, $token->href));

2. The other half: app conventions in core should be removable. LinkToken::parse() does:

if (str_starts_with($href, '*')) {
    $href = substr($href, 1);
    $blank = ' target="_blank" rel="noopener noreferrer"';
}

* meaning "open in a new tab" isn't Markdown, it's a convention — same family as ::: divs, @@, {x:handle} and automatic heading IDs. I'm not arguing they should go away; they're useful and I'd use several of them. The ask is only that they be rules an integration can remove or replace. Today an app that wants [text](*something) to keep its asterisk has no way to say so, and the same is true of heading IDs, which we currently strip back out of the HTML.

So concretely: if #13 lands along the lines I proposed there, most of point 5 is solved — the conventions become removable rules — and what remains is the renderer hook.

[RobinDev]

3. Preserve raw HTML and distinguish literal text from markup

Would be good to fix indeed

The raw HTML half is #40: pre, script, style and textarea are raw text elements in CommonMark, and HtmlToken no longer re-parses their content as Markdown. <div>**a**</div> keeps parsing, since that's Tempest's own behaviour and a separate question from #20.

The other half — escaping in text tokens — I've left alone for now. a\*b keeps its backslash, a_b_c becomes emphasis, and " in a text node isn't escaped. That one needs a decision rather than a fix: TextToken::parse() currently returns its content verbatim, so escaping it correctly means deciding what happens to inline raw HTML that reaches a text token, and there's already a @todo about Markdown escaping in LinkToken. Happy to take it on once you've said which way you want it to go.

[RobinDev]

4. Finish structural Markdown before adding CMS syntax

Sure, any bug can be fixed

Four PRs, one per item, each with tests and a table of before/after:

• fix: keep the start number of an ordered list #41 — 2. First loses its list start number → <ol start="2">
• fix: parse tilde fenced code blocks #42 — ~~~ fences parsed as strikethrough → fenced code, and a longer fence may now contain a shorter one
• fix: require whitespace after an ATX heading marker #43 — #titre parsed as a heading without the required space, and ####### seven emitting an <h7>
• fix: render hard line breaks #44 — hard line breaks (␣␣\n and \\\n) rendered as soft breaks

Still open from that list, and not in any of these PRs:

• Loose lists. - a\n\n- b renders tight; CommonMark wraps each item in <p>.
• Automatic heading IDs. Right now every heading gets one and there's no way to turn it off — we strip them back out of the HTML downstream. This is Make tokens and rules more configurable #13 territory rather than a bug.
• GFM task lists. - [x] Done currently renders an empty-URL link. Best as an optional rule, I think.
• Indented code blocks. Four leading spaces stay a paragraph.

Tell me which of those you want and in what shape, and I'll pick them up.