feat!: flank emphasis by the CJK-friendly amendments

docs/archive/2026-10-18-cjk-friendly-emphasis/plan.md

@@ -0,0 +1,144 @@
+# CJK-friendly emphasis
+
+## Why
+
+CommonMark's flanking rules read `这是**重点。**然后` as text: a closing `**`
+after CJK punctuation and before a CJK character is not right-flanking, and
+Chinese and Japanese put no space there. Settle's notes are written in
+Chinese, so bold that ends in `。` or `」` silently fails to form in the
+editor and the index.
+
+- Scope: the flanking of every emphasis-like delimiter run (`*`, `_`, `~~`,
+  `==`) follows the markdown-cjk-friendly amendments; the syntax stays one
+  fixed syntax with no option.
+- Risk: a document that parsed one way can parse another way. The amendments
+  leave every CommonMark example unchanged (upstream checks this), and source
+  bytes never move on write, so the change shows only in the tree.
+
+## What changes
+
+### Inline syntax
+- `inline-syntax` › CJK-friendly emphasis flanking: added
+- `inline-syntax` › CommonMark inlines: modified — names the flanking
+  exception
+
+### Decisions
+- `0018` supersedes `0015`: the one normative syntax reads emphasis with the
+  CJK-friendly amendments.
+
+### Outside the specs
+- Settle (another repository), once this ships in a published release:
+  depend on the new version, and rebuild the vendored wasm binding from it.
+
+Specs:
+- `inline-syntax` (modified)
+
+## Design
+
+### Reference
+
+The amendments are
+<https://github.com/tats-u/markdown-cjk-friendly/blob/5b17173/specification.md>,
+with concrete ranges in its `ranges.md` (Unicode 18). The reference
+implementation is `micromark-extension-cjk-friendly` 2.0.2 with
+`micromark-extension-cjk-friendly-gfm-strikethrough`; its
+`classifyPrecedingCharacter` settles readings the text leaves loose.
+
+### Character classes
+
+A new crate-private module holds sorted inclusive ranges, copied from
+`ranges.md` at that commit, and three predicates: a CJK character, a
+non-emoji general-use variation selector (U+FE00–U+FE0E), and an ideographic
+variation selector (U+E0100–U+E01EF). Unicode punctuation stays the existing
+table.
+
+Each side of a run is classified once into whitespace (or a bounded side, or
+none), CJK, punctuation, CJK punctuation, ideographic variation selector, or
+other:
+
+- After the run: the next char alone.
+- Before the run: the previous char; when it is a non-emoji general-use
+  variation selector, the char before it decides, unless that is whitespace
+  or absent, where the selector reads as other (unspecified upstream). `‘`
+  `’` `“` `”` followed by U+FE01 read as CJK punctuation, as the reference
+  does.
+
+### Flanking
+
+With `punct` the Unicode punctuation set and `cjk` the CJK set:
+
+- left = !after.space && (!after.punct || before.space || before.punct ||
+  after.cjk || before.cjk || before.ivs)
+- right = !before.space && (!before.punct || after.space || after.punct ||
+  before.cjk || after.cjk)
+
+These are the amendments' definitions in the reduced form of the upstream
+implementers' tips. `delimiter_flanking_within` computes them for every
+caller: `*`, `_`, `~~`, and `==` runs. The `_` open rule's "preceded by
+punctuation" reads the previous base char, so a punctuation char followed by a
+variation selector counts; its close rule is CommonMark's. The GFM rule that
+lets a `*` run beside `~` open or close stays.
+
+### Serializer
+
+The serializer writes a span's delimiters around its content and relies on
+read-back tests; nothing in it reads flanking. Read-back can only improve for
+CJK text, and the seeded generator's `NOT_READING_BACK` list is checked as it
+stands.
+
+### Test scope
+
+- `tests/fixtures/syntax_decisions/cjk_friendly_emphasis.cases`: the inputs of
+  upstream `testcases/*.md` (`strong.md`, `korean.md`, `pseudo-emoji.md`,
+  `non-bmp.md`, `underscore-cjk-punct.md`, `gfm-strikethrough.md`,
+  `gfm-non-cjk.md`), with HTML verified against micromark with GFM and the
+  two cjk-friendly extensions, where this crate's renderer agrees elsewhere.
+- The conformance bench: no oracle case moves (CommonMark examples are
+  unchanged by the amendments); a GFM case that moves leaves the bench by the
+  conformance rule.
+- Unit tests of the classes at their range edges and of the variation
+  selector look-back.
+
+### Decisions
+- Always on, no option — because 0015 fixes one syntax with no configuration
+  and Settle's notes need it. Turned down: an opt-in parse option.
+- Every emphasis-like mark, `~~` and `==` included — because they share one
+  flanking function and the reference applies the amendments to
+  strikethrough too. Turned down: `*` and `_` only, which would make `~~重点。~~然后`
+  fail where `**重点。**然后` forms.
+- The ranges copied from the spec's `ranges.md` — because the CJK set excludes
+  default-emoji-presentation chars, which needs emoji data the crate's
+  generated tables do not carry; the spec publishes the set itself. Turned
+  down: generating from the UCD tables the crate already uses (15.1.0).
+- Follow the reference for the ambiguous quotes with U+FE01 — the spec text
+  defines them but its flanking rule does not name them.
+
+### Risks
+- [A non-CJK document pairs differently because a run becomes both-flanking]
+  → the amendments only widen flanking beside CJK chars and variation
+  selectors; the conformance bench and the goldens show any change, each read
+  in review.
+
+## Tasks
+
+### 1. Classes and flanking
+- [x] 1.1 Add the CJK and variation-selector ranges and predicates; verified by
+  unit tests at the range edges.
+  - The module doc names `ranges.md` at 5b17173 and its generator
+    (`node --run print-ranges`), so a Unicode update knows where to look.
+- [x] 1.2 Classify both sides of a run and compute flanking and the `_` open
+  rule by the amendments; verified by `cjk_friendly_emphasis.cases` matching
+  micromark's HTML and by unchanged conformance numbers and goldens.
+  - The flanking comment cites the spec's definitions and the reduction from
+    the upstream implementers' tips.
+  - `tests/syntax_decisions.rs`'s header admits cases taken from an
+    amendment's own test inputs, labelled with their upstream file.
+
+### 2. Docs
+- [x] 2.1 Merge the spec change and write decision 0018 superseding 0015;
+  verified by the requirement named in the new tests and `0015` moved to
+  `superseded/`.
+
+### 3. Integration
+- [x] 3.1 Run CI's commands (fmt, build, tests with `--profile ci --features
+  html`, docs, the wasm job); verified by each passing.

docs/decisions/0018-one-syntax-with-cjk-friendly-emphasis.md

@@ -0,0 +1,121 @@
+# 0018: One normative syntax, with CJK-friendly emphasis
+
+Status: Accepted
+Date: 2026-10-18
+Supersedes: 0015
+
+## Context
+
+0015 fixed two parse entry points and one syntax, with tags. Its emphasis is
+CommonMark's, whose flanking rules treat CJK punctuation like ASCII
+punctuation: `这是**重点。**然后` stays text, because the closing `**` follows
+`。` and precedes `然`, and Chinese and Japanese put no space there. Settle's
+notes are mostly Chinese, and the editor shows the bold the parse finds, so
+such bold silently fails to form. The markdown-cjk-friendly amendments fix
+flanking beside CJK characters; micromark and markdown-it implement them, and
+they leave every CommonMark example unchanged.
+
+## Decision
+
+There are two parse entry points, and both read one fixed syntax:
+`parse(input)` reads a document, and `parse_inline(text, definitions)` reads
+the inline content of one leaf block, applying no block rule and forming
+reference links only for the labels `definitions` holds. The syntax is:
+
+- CommonMark, with raw HTML and indented code, where delimiter runs flank
+  by the CommonMark CJK-friendly amendments
+  (<https://github.com/tats-u/markdown-cjk-friendly>, at commit `5b17173`,
+  Unicode 18 ranges) for `*`, `_`, `~~`, and `==` alike, reading `‘’“”`
+  with U+FE01 before a run as CJK punctuation as
+  `micromark-extension-cjk-friendly` does;
+- `<details>` HTML containers;
+- GFM tables, task list items, literal autolinks (the strict GFM extents,
+  plus `mailto:` and `xmpp:`; as in cmark-gfm, a `www.`, `http://`, or
+  `https://` literal does not form after a link or image `[` that no `]` has
+  closed yet, while an email address still links there), and `~~`
+  strikethrough;
+- footnote definitions, references, and inline footnotes `^[…]`, whose `^[`
+  opens no link text, so a URL inside one links;
+- GFM alerts, frontmatter, shortcodes, `==` highlight;
+- wiki links with the title after the pipe, whose content holds no unescaped
+  `[` or `]`, and which win over a defined reference label;
+- inline and block math;
+- directives, whose names are runs of ASCII letters joined by single `-`, and
+  whose text form is followed by whitespace or the end of the content, or by
+  ASCII punctuation when it has a non-empty label or non-blank attribute
+  braces;
+- tags: `#` followed by a Unicode letter and then any letters, numbers, `/`,
+  `_`, or `-`, where the character before the `#` is whitespace or the start
+  of the leaf's inline content. The delimiters of emphasis, strong,
+  strikethrough, and highlight are transparent to that test; the edges of a
+  link, image, inline footnote, or text directive, and every other inline
+  node, are not whitespace, and nothing inside code, math, autolinks, wiki
+  links, raw HTML, escapes, character references, shortcodes, or footnote
+  references is read.
+
+Link text holds no link: an autolink inside link text, image alt included,
+reads as its text, an angle-bracket one without its brackets, and a wiki link,
+also inside a text directive's label, keeps the brackets around it from forming
+a link. A tag is not a link, so link text holds tags.
+
+Subscript, superscript, insert, spoiler, underline, description lists,
+single-tilde strikethrough, relaxed literal autolinks, and MDX are not part of
+the syntax, and their AST nodes do not exist. The syntax is hard-coded: no
+public or private configuration value selects constructs.
+
+## Considered options
+
+- Keep CommonMark flanking: CJK writers must add a space or a zero-width
+  space after a closing delimiter, which shows in the text or hides in it.
+- Make the amendments an option of `parse`: the syntax is hard-coded and has
+  no configuration, and Settle, the crate's user, wants them on.
+- Apply the amendments to `*` and `_` only: `~~重点。~~然后` would stay text
+  where `**重点。**然后` forms, and the reference applies them to
+  strikethrough too.
+- Generate the CJK set from the Unicode tables the crate already uses
+  (15.1.0): the set excludes default emoji presentation characters, which
+  needs emoji data those tables do not carry; the amendments publish the set.
+- Keep `parse` as the only entry point and re-parse the whole document per
+  keystroke: 3–6 ms for 124 KB in wasm, and the editor must rebuild the
+  file's text before every parse.
+- Parse a paragraph alone with `parse`: block rules apply to it, and
+  reference links lose their definitions.
+- Keep the presets and add a normative one as the default: the configuration
+  space, the profile-by-profile tests, and the serializer's per-syntax read
+  back stay.
+- Keep a crate-private constant `Constructs`: dead branches stay alive for a
+  configuration that no longer exists.
+- Keep relaxed literal-autolink extents for `http(s)://` and `www.`: they
+  match only comrak's relaxed mode.
+- Block a bare text directive only before `@` or `.letter`: narrower, and it
+  keeps `(:note)` and `:h1` as directives.
+- Let a defined reference label beat a wiki link: a reading would depend on
+  definitions elsewhere in the document.
+- Nest an autolink's link inside link text, as cmark-gfm, commonmark.js, and
+  micromark do for `[this <http://and.com> that](url)`: link text would hold
+  a link, which the AST, validation, and the serializer otherwise rule out.
+- Leave tags to the caller: the editor and the index read them separately,
+  and the editor cannot get them from the tree it renders.
+- meowdown's tag rule, which also reads a tag that starts with a digit, such
+  as `#1a`: Settle's spec decided against it.
+- Recognize `#` in the inline scanner: whether `*` is transparent depends on
+  delimiter pairing, which the scanner has not done yet.
+
+## Consequences
+
+- Text beside CJK characters that read as literal delimiters may read as
+  emphasis, strikethrough, or highlight; a document's source is unchanged,
+  and only its tree differs.
+- A caller who relied on a removed construct reads it as text.
+- Text that held a tag holds a `Tag` node; a caller that read tags from text
+  reads the nodes.
+- Text that would form a tag where it is written is the builder's to escape
+  with `Escape('#')`, as for any shape that depends on its neighbours. A
+  `Tag` the builder places after a character that is not whitespace, or
+  before a character a name can hold, is written as it is and reads back
+  differently; validation checks only the name.
+- Tests assert one result per input; fixture profiles and preset loops go.
+- An oracle case that this syntax reads differently by decision leaves the
+  conformance bench; its input is checked exactly in
+  `tests/fixtures/syntax_decisions/`, with HTML verified against a reference
+  renderer or the decision.

docs/decisions/0015-parse-entry-points-with-tags.md → docs/decisions/superseded/0015-parse-entry-points-with-tags.md

@@ -1,6 +1,6 @@
 # 0015: Parse entry points and one normative syntax, with tags
 
-Status: Accepted
+Status: Superseded by 0018
 Date: 2026-10-14
 Supersedes: 0012
 

docs/overview.md

@@ -36,7 +36,8 @@ shipped: blank lines as nodes, item numbers, leaves written as their text,
 source reuse per node down to inlines, and inline syntax tokens for the
 editor's marks. Validation diagnostics name the node they are about, so an
 editor can fix a refused tree without guessing, and `Block::children()`
-walks block containers. Lists stay nested; the editor flattens them.
+walks block containers. Lists stay nested; the editor flattens them. Emphasis flanks by the CJK-friendly
+amendments, so bold that ends in CJK punctuation forms without a space.
 
 ## Non-goals
 

docs/specs/inline-syntax.md

@@ -13,7 +13,8 @@ HTML, and inline directives. Owned by the inline parser.
 The parser SHALL recognize CommonMark inline constructs as the CommonMark
 specification 0.31.2 defines them, including its precedence of code spans,
 links, and emphasis, with commonmark.js as the reference where the
-specification leaves a case open, except that link text holds no link: an
+specification leaves a case open, except that delimiter runs flank as
+CJK-friendly emphasis flanking states, and that link text holds no link: an
 autolink inside link text, image alt included, reads as its text, an
 angle-bracket one without its brackets, and a wikilink, alone or in a text
 directive's label, keeps the brackets around it from forming a link.
@@ -26,6 +27,21 @@ directive's label, keeps the brackets around it from forming a link.
 - **WHEN** `"[a [[b]] c](u)"` and `"[x :d[[[w]]] y](u)"` are parsed
 - **THEN** neither paragraph holds a `Link`: a wikilink, alone or in a text directive's label, keeps the brackets around it from forming a link, and the first paragraph holds `Text("[a ")`, a `WikiLink` with target `b`, and `Text(" c](u)")`
 
+### Requirement: CJK-friendly emphasis flanking
+The parser SHALL decide whether a `*`, `_`, `~~`, or `==` delimiter run is
+left-flanking or right-flanking, and whether a `_` or `__` run can open, by
+the CommonMark CJK-friendly amendments
+(<https://github.com/tats-u/markdown-cjk-friendly>, specification at commit
+`5b17173`, with its Unicode 18 ranges) in place of CommonMark's definitions,
+except that `‘`, `’`, `“`, or `”` followed by U+FE01 before a run SHALL read as
+CJK punctuation, as `micromark-extension-cjk-friendly` reads it. A run after a
+variation selector that follows whitespace, an ideographic variation
+selector, or the start of the content flanks in an unspecified way.
+
+#### Scenario: Strong ending in CJK punctuation
+- **WHEN** `"这是**重点。**然后"` is parsed
+- **THEN** the paragraph holds `Text("这是")`, a `Strong` containing `重点。`, and `Text("然后")`
+
 ### Requirement: Extension marks
 The parser SHALL read `==x==` as `Mark`, parsing its content as inline content,
 and `~~x~~` as `Delete`; a single `~`, a run of three or more `~`, and `++` and

src/lib.rs

@@ -12,6 +12,7 @@ mod leaves;
 mod memo;
 #[cfg(test)]
 mod test_support;
+mod unicode_cjk;
 mod unicode_letters;
 mod unicode_punctuation;