diff --git a/docs/archive/2026-10-18-cjk-friendly-emphasis/plan.md b/docs/archive/2026-10-18-cjk-friendly-emphasis/plan.md new file mode 100644 index 0000000..9ed11c1 --- /dev/null +++ b/docs/archive/2026-10-18-cjk-friendly-emphasis/plan.md @@ -0,0 +1,144 @@ +# CJK-friendly emphasis + +## Why + +CommonMark's flanking rules read `这是**重点。**然后` as text: a closing `**` +after CJK punctuation and before a CJK character is not right-flanking, and +Chinese and Japanese put no space there. Settle's notes are written in +Chinese, so bold that ends in `。` or `」` silently fails to form in the +editor and the index. + +- Scope: the flanking of every emphasis-like delimiter run (`*`, `_`, `~~`, + `==`) follows the markdown-cjk-friendly amendments; the syntax stays one + fixed syntax with no option. +- Risk: a document that parsed one way can parse another way. The amendments + leave every CommonMark example unchanged (upstream checks this), and source + bytes never move on write, so the change shows only in the tree. + +## What changes + +### Inline syntax +- `inline-syntax` › CJK-friendly emphasis flanking: added +- `inline-syntax` › CommonMark inlines: modified — names the flanking + exception + +### Decisions +- `0018` supersedes `0015`: the one normative syntax reads emphasis with the + CJK-friendly amendments. + +### Outside the specs +- Settle (another repository), once this ships in a published release: + depend on the new version, and rebuild the vendored wasm binding from it. + +Specs: +- `inline-syntax` (modified) + +## Design + +### Reference + +The amendments are +, +with concrete ranges in its `ranges.md` (Unicode 18). The reference +implementation is `micromark-extension-cjk-friendly` 2.0.2 with +`micromark-extension-cjk-friendly-gfm-strikethrough`; its +`classifyPrecedingCharacter` settles readings the text leaves loose. + +### Character classes + +A new crate-private module holds sorted inclusive ranges, copied from +`ranges.md` at that commit, and three predicates: a CJK character, a +non-emoji general-use variation selector (U+FE00–U+FE0E), and an ideographic +variation selector (U+E0100–U+E01EF). Unicode punctuation stays the existing +table. + +Each side of a run is classified once into whitespace (or a bounded side, or +none), CJK, punctuation, CJK punctuation, ideographic variation selector, or +other: + +- After the run: the next char alone. +- Before the run: the previous char; when it is a non-emoji general-use + variation selector, the char before it decides, unless that is whitespace + or absent, where the selector reads as other (unspecified upstream). `‘` + `’` `“` `”` followed by U+FE01 read as CJK punctuation, as the reference + does. + +### Flanking + +With `punct` the Unicode punctuation set and `cjk` the CJK set: + +- left = !after.space && (!after.punct || before.space || before.punct || + after.cjk || before.cjk || before.ivs) +- right = !before.space && (!before.punct || after.space || after.punct || + before.cjk || after.cjk) + +These are the amendments' definitions in the reduced form of the upstream +implementers' tips. `delimiter_flanking_within` computes them for every +caller: `*`, `_`, `~~`, and `==` runs. The `_` open rule's "preceded by +punctuation" reads the previous base char, so a punctuation char followed by a +variation selector counts; its close rule is CommonMark's. The GFM rule that +lets a `*` run beside `~` open or close stays. + +### Serializer + +The serializer writes a span's delimiters around its content and relies on +read-back tests; nothing in it reads flanking. Read-back can only improve for +CJK text, and the seeded generator's `NOT_READING_BACK` list is checked as it +stands. + +### Test scope + +- `tests/fixtures/syntax_decisions/cjk_friendly_emphasis.cases`: the inputs of + upstream `testcases/*.md` (`strong.md`, `korean.md`, `pseudo-emoji.md`, + `non-bmp.md`, `underscore-cjk-punct.md`, `gfm-strikethrough.md`, + `gfm-non-cjk.md`), with HTML verified against micromark with GFM and the + two cjk-friendly extensions, where this crate's renderer agrees elsewhere. +- The conformance bench: no oracle case moves (CommonMark examples are + unchanged by the amendments); a GFM case that moves leaves the bench by the + conformance rule. +- Unit tests of the classes at their range edges and of the variation + selector look-back. + +### Decisions +- Always on, no option — because 0015 fixes one syntax with no configuration + and Settle's notes need it. Turned down: an opt-in parse option. +- Every emphasis-like mark, `~~` and `==` included — because they share one + flanking function and the reference applies the amendments to + strikethrough too. Turned down: `*` and `_` only, which would make `~~重点。~~然后` + fail where `**重点。**然后` forms. +- The ranges copied from the spec's `ranges.md` — because the CJK set excludes + default-emoji-presentation chars, which needs emoji data the crate's + generated tables do not carry; the spec publishes the set itself. Turned + down: generating from the UCD tables the crate already uses (15.1.0). +- Follow the reference for the ambiguous quotes with U+FE01 — the spec text + defines them but its flanking rule does not name them. + +### Risks +- [A non-CJK document pairs differently because a run becomes both-flanking] + → the amendments only widen flanking beside CJK chars and variation + selectors; the conformance bench and the goldens show any change, each read + in review. + +## Tasks + +### 1. Classes and flanking +- [x] 1.1 Add the CJK and variation-selector ranges and predicates; verified by + unit tests at the range edges. + - The module doc names `ranges.md` at 5b17173 and its generator + (`node --run print-ranges`), so a Unicode update knows where to look. +- [x] 1.2 Classify both sides of a run and compute flanking and the `_` open + rule by the amendments; verified by `cjk_friendly_emphasis.cases` matching + micromark's HTML and by unchanged conformance numbers and goldens. + - The flanking comment cites the spec's definitions and the reduction from + the upstream implementers' tips. + - `tests/syntax_decisions.rs`'s header admits cases taken from an + amendment's own test inputs, labelled with their upstream file. + +### 2. Docs +- [x] 2.1 Merge the spec change and write decision 0018 superseding 0015; + verified by the requirement named in the new tests and `0015` moved to + `superseded/`. + +### 3. Integration +- [x] 3.1 Run CI's commands (fmt, build, tests with `--profile ci --features + html`, docs, the wasm job); verified by each passing. diff --git a/docs/decisions/0018-one-syntax-with-cjk-friendly-emphasis.md b/docs/decisions/0018-one-syntax-with-cjk-friendly-emphasis.md new file mode 100644 index 0000000..e3813fe --- /dev/null +++ b/docs/decisions/0018-one-syntax-with-cjk-friendly-emphasis.md @@ -0,0 +1,121 @@ +# 0018: One normative syntax, with CJK-friendly emphasis + +Status: Accepted +Date: 2026-10-18 +Supersedes: 0015 + +## Context + +0015 fixed two parse entry points and one syntax, with tags. Its emphasis is +CommonMark's, whose flanking rules treat CJK punctuation like ASCII +punctuation: `这是**重点。**然后` stays text, because the closing `**` follows +`。` and precedes `然`, and Chinese and Japanese put no space there. Settle's +notes are mostly Chinese, and the editor shows the bold the parse finds, so +such bold silently fails to form. The markdown-cjk-friendly amendments fix +flanking beside CJK characters; micromark and markdown-it implement them, and +they leave every CommonMark example unchanged. + +## Decision + +There are two parse entry points, and both read one fixed syntax: +`parse(input)` reads a document, and `parse_inline(text, definitions)` reads +the inline content of one leaf block, applying no block rule and forming +reference links only for the labels `definitions` holds. The syntax is: + +- CommonMark, with raw HTML and indented code, where delimiter runs flank + by the CommonMark CJK-friendly amendments + (, at commit `5b17173`, + Unicode 18 ranges) for `*`, `_`, `~~`, and `==` alike, reading `‘’“”` + with U+FE01 before a run as CJK punctuation as + `micromark-extension-cjk-friendly` does; +- `
` HTML containers; +- GFM tables, task list items, literal autolinks (the strict GFM extents, + plus `mailto:` and `xmpp:`; as in cmark-gfm, a `www.`, `http://`, or + `https://` literal does not form after a link or image `[` that no `]` has + closed yet, while an email address still links there), and `~~` + strikethrough; +- footnote definitions, references, and inline footnotes `^[…]`, whose `^[` + opens no link text, so a URL inside one links; +- GFM alerts, frontmatter, shortcodes, `==` highlight; +- wiki links with the title after the pipe, whose content holds no unescaped + `[` or `]`, and which win over a defined reference label; +- inline and block math; +- directives, whose names are runs of ASCII letters joined by single `-`, and + whose text form is followed by whitespace or the end of the content, or by + ASCII punctuation when it has a non-empty label or non-blank attribute + braces; +- tags: `#` followed by a Unicode letter and then any letters, numbers, `/`, + `_`, or `-`, where the character before the `#` is whitespace or the start + of the leaf's inline content. The delimiters of emphasis, strong, + strikethrough, and highlight are transparent to that test; the edges of a + link, image, inline footnote, or text directive, and every other inline + node, are not whitespace, and nothing inside code, math, autolinks, wiki + links, raw HTML, escapes, character references, shortcodes, or footnote + references is read. + +Link text holds no link: an autolink inside link text, image alt included, +reads as its text, an angle-bracket one without its brackets, and a wiki link, +also inside a text directive's label, keeps the brackets around it from forming +a link. A tag is not a link, so link text holds tags. + +Subscript, superscript, insert, spoiler, underline, description lists, +single-tilde strikethrough, relaxed literal autolinks, and MDX are not part of +the syntax, and their AST nodes do not exist. The syntax is hard-coded: no +public or private configuration value selects constructs. + +## Considered options + +- Keep CommonMark flanking: CJK writers must add a space or a zero-width + space after a closing delimiter, which shows in the text or hides in it. +- Make the amendments an option of `parse`: the syntax is hard-coded and has + no configuration, and Settle, the crate's user, wants them on. +- Apply the amendments to `*` and `_` only: `~~重点。~~然后` would stay text + where `**重点。**然后` forms, and the reference applies them to + strikethrough too. +- Generate the CJK set from the Unicode tables the crate already uses + (15.1.0): the set excludes default emoji presentation characters, which + needs emoji data those tables do not carry; the amendments publish the set. +- Keep `parse` as the only entry point and re-parse the whole document per + keystroke: 3–6 ms for 124 KB in wasm, and the editor must rebuild the + file's text before every parse. +- Parse a paragraph alone with `parse`: block rules apply to it, and + reference links lose their definitions. +- Keep the presets and add a normative one as the default: the configuration + space, the profile-by-profile tests, and the serializer's per-syntax read + back stay. +- Keep a crate-private constant `Constructs`: dead branches stay alive for a + configuration that no longer exists. +- Keep relaxed literal-autolink extents for `http(s)://` and `www.`: they + match only comrak's relaxed mode. +- Block a bare text directive only before `@` or `.letter`: narrower, and it + keeps `(:note)` and `:h1` as directives. +- Let a defined reference label beat a wiki link: a reading would depend on + definitions elsewhere in the document. +- Nest an autolink's link inside link text, as cmark-gfm, commonmark.js, and + micromark do for `[this that](url)`: link text would hold + a link, which the AST, validation, and the serializer otherwise rule out. +- Leave tags to the caller: the editor and the index read them separately, + and the editor cannot get them from the tree it renders. +- meowdown's tag rule, which also reads a tag that starts with a digit, such + as `#1a`: Settle's spec decided against it. +- Recognize `#` in the inline scanner: whether `*` is transparent depends on + delimiter pairing, which the scanner has not done yet. + +## Consequences + +- Text beside CJK characters that read as literal delimiters may read as + emphasis, strikethrough, or highlight; a document's source is unchanged, + and only its tree differs. +- A caller who relied on a removed construct reads it as text. +- Text that held a tag holds a `Tag` node; a caller that read tags from text + reads the nodes. +- Text that would form a tag where it is written is the builder's to escape + with `Escape('#')`, as for any shape that depends on its neighbours. A + `Tag` the builder places after a character that is not whitespace, or + before a character a name can hold, is written as it is and reads back + differently; validation checks only the name. +- Tests assert one result per input; fixture profiles and preset loops go. +- An oracle case that this syntax reads differently by decision leaves the + conformance bench; its input is checked exactly in + `tests/fixtures/syntax_decisions/`, with HTML verified against a reference + renderer or the decision. diff --git a/docs/decisions/0015-parse-entry-points-with-tags.md b/docs/decisions/superseded/0015-parse-entry-points-with-tags.md similarity index 99% rename from docs/decisions/0015-parse-entry-points-with-tags.md rename to docs/decisions/superseded/0015-parse-entry-points-with-tags.md index 576f4d5..c7707cb 100644 --- a/docs/decisions/0015-parse-entry-points-with-tags.md +++ b/docs/decisions/superseded/0015-parse-entry-points-with-tags.md @@ -1,6 +1,6 @@ # 0015: Parse entry points and one normative syntax, with tags -Status: Accepted +Status: Superseded by 0018 Date: 2026-10-14 Supersedes: 0012 diff --git a/docs/overview.md b/docs/overview.md index c78ab39..08f9b2b 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -36,7 +36,8 @@ shipped: blank lines as nodes, item numbers, leaves written as their text, source reuse per node down to inlines, and inline syntax tokens for the editor's marks. Validation diagnostics name the node they are about, so an editor can fix a refused tree without guessing, and `Block::children()` -walks block containers. Lists stay nested; the editor flattens them. +walks block containers. Lists stay nested; the editor flattens them. Emphasis flanks by the CJK-friendly +amendments, so bold that ends in CJK punctuation forms without a space. ## Non-goals diff --git a/docs/specs/inline-syntax.md b/docs/specs/inline-syntax.md index 6912265..ef0d2e1 100644 --- a/docs/specs/inline-syntax.md +++ b/docs/specs/inline-syntax.md @@ -13,7 +13,8 @@ HTML, and inline directives. Owned by the inline parser. The parser SHALL recognize CommonMark inline constructs as the CommonMark specification 0.31.2 defines them, including its precedence of code spans, links, and emphasis, with commonmark.js as the reference where the -specification leaves a case open, except that link text holds no link: an +specification leaves a case open, except that delimiter runs flank as +CJK-friendly emphasis flanking states, and that link text holds no link: an autolink inside link text, image alt included, reads as its text, an angle-bracket one without its brackets, and a wikilink, alone or in a text directive's label, keeps the brackets around it from forming a link. @@ -26,6 +27,21 @@ directive's label, keeps the brackets around it from forming a link. - **WHEN** `"[a [[b]] c](u)"` and `"[x :d[[[w]]] y](u)"` are parsed - **THEN** neither paragraph holds a `Link`: a wikilink, alone or in a text directive's label, keeps the brackets around it from forming a link, and the first paragraph holds `Text("[a ")`, a `WikiLink` with target `b`, and `Text(" c](u)")` +### Requirement: CJK-friendly emphasis flanking +The parser SHALL decide whether a `*`, `_`, `~~`, or `==` delimiter run is +left-flanking or right-flanking, and whether a `_` or `__` run can open, by +the CommonMark CJK-friendly amendments +(, specification at commit +`5b17173`, with its Unicode 18 ranges) in place of CommonMark's definitions, +except that `‘`, `’`, `“`, or `”` followed by U+FE01 before a run SHALL read as +CJK punctuation, as `micromark-extension-cjk-friendly` reads it. A run after a +variation selector that follows whitespace, an ideographic variation +selector, or the start of the content flanks in an unspecified way. + +#### Scenario: Strong ending in CJK punctuation +- **WHEN** `"这是**重点。**然后"` is parsed +- **THEN** the paragraph holds `Text("这是")`, a `Strong` containing `重点。`, and `Text("然后")` + ### Requirement: Extension marks The parser SHALL read `==x==` as `Mark`, parsing its content as inline content, and `~~x~~` as `Delete`; a single `~`, a run of three or more `~`, and `++` and diff --git a/src/lib.rs b/src/lib.rs index 3eb39c7..25a452f 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -12,6 +12,7 @@ mod leaves; mod memo; #[cfg(test)] mod test_support; +mod unicode_cjk; mod unicode_letters; mod unicode_punctuation; diff --git a/src/parse.rs b/src/parse.rs index e66e09f..436ba86 100644 --- a/src/parse.rs +++ b/src/parse.rs @@ -1568,7 +1568,8 @@ fn push_delimiter( }); } -/// The CommonMark roles of a `*`/`_`/`~~` run. +/// The CommonMark roles of a `*`/`_`/`~~` run, with flanking as the +/// CJK-friendly amendments define it (see `delimiter_flanking_within`). /// /// Flanking is computed on the whole run (CommonMark treats left/right-flanking /// as a property of the run, not of an individual delimiter), including the @@ -1592,10 +1593,8 @@ fn emphasis_roles( let flanking = delimiter_flanking_within(input, index, length, bounded_before, bounded_after); let (mut can_open, mut can_close) = if marker == b'_' { ( - flanking.left - && (!flanking.right || flanking.previous.is_some_and(is_flanking_punctuation)), - flanking.right - && (!flanking.left || flanking.next.is_some_and(is_flanking_punctuation)), + flanking.left && (!flanking.right || flanking.previous_punctuation), + flanking.right && (!flanking.left || flanking.next_punctuation), ) } else { (flanking.left, flanking.right) @@ -3917,6 +3916,68 @@ struct DelimiterFlanking { right: bool, previous: Option, next: Option, + /// The char before the run, read past a variation selector, is + /// punctuation (a "Unicode punctuation sequence"). + previous_punctuation: bool, + next_punctuation: bool, +} + +/// How one side of a delimiter run reads for flanking. +#[derive(Clone, Copy, Default)] +struct FlankingSide { + whitespace: bool, + punctuation: bool, + cjk: bool, + ideographic_variation_selector: bool, +} + +impl FlankingSide { + fn of(value: Option) -> Self { + let Some(value) = value else { + return Self { + whitespace: true, + ..Self::default() + }; + }; + Self { + whitespace: value.is_whitespace(), + punctuation: is_flanking_punctuation(value), + cjk: crate::unicode_cjk::is_cjk(value), + ideographic_variation_selector: crate::unicode_cjk::is_ideographic_variation_selector( + value, + ), + } + } + + /// The side before a run whose previous char is `previous` and, before + /// that, `before_previous`: a general-use variation selector reads as the + /// char it follows, unless that is whitespace or absent, where it reads as + /// a plain character (unspecified by the amendments), and `‘’“”` with + /// U+FE01 read as CJK punctuation, as micromark-extension-cjk-friendly + /// reads them. + fn before(previous: Option, before_previous: impl FnOnce() -> Option) -> Self { + match previous { + Some(selector) if crate::unicode_cjk::is_general_use_variation_selector(selector) => { + match before_previous() { + Some(base) if !base.is_whitespace() => { + if selector == '\u{FE01}' && matches!(base, '‘' | '’' | '“' | '”') { + return Self { + punctuation: true, + cjk: true, + ..Self::default() + }; + } + Self { + ideographic_variation_selector: false, + ..Self::of(Some(base)) + } + } + _ => Self::default(), + } + } + _ => Self::of(previous), + } + } } fn delimiter_flanking(input: &str, index: usize, marker_len: usize) -> DelimiterFlanking { @@ -3926,6 +3987,14 @@ fn delimiter_flanking(input: &str, index: usize, marker_len: usize) -> Delimiter /// `delimiter_flanking` for a run that may sit right inside a mark span: a /// bounded side flanks as if the input ended there, the way the span's content /// reads on its own. +/// +/// Left- and right-flanking follow the CommonMark CJK-friendly amendments +/// (, +/// section 6.2), in the reduced form of the amendments' implementers' tips: +/// a run is left-flanking when not followed by whitespace and either not +/// followed by punctuation, or preceded by whitespace or punctuation, or with +/// a CJK char on either side or an ideographic variation selector before it; +/// right-flanking is the mirror, without the selector. fn delimiter_flanking_within( input: &str, index: usize, @@ -3944,23 +4013,34 @@ fn delimiter_flanking_within( input[index + marker_len..].chars().next().map(source_char) }; - let previous_whitespace = previous.is_none_or(char::is_whitespace); - let next_whitespace = next.is_none_or(char::is_whitespace); - let previous_punctuation = previous.is_some_and(is_flanking_punctuation); - let next_punctuation = next.is_some_and(is_flanking_punctuation); - - let left = next.is_some() - && !next_whitespace - && !(next_punctuation && !previous_whitespace && !previous_punctuation); - let right = previous.is_some() - && !previous_whitespace - && !(previous_punctuation && !next_whitespace && !next_punctuation); + let before = FlankingSide::before(previous, || { + let mut chars = input[..index].chars(); + chars.next_back(); + chars.next_back().map(source_char) + }); + let after = FlankingSide::of(next); + + let left = !after.whitespace + && (!after.punctuation + || before.whitespace + || before.punctuation + || after.cjk + || before.cjk + || before.ideographic_variation_selector); + let right = !before.whitespace + && (!before.punctuation + || after.whitespace + || after.punctuation + || before.cjk + || after.cjk); DelimiterFlanking { left, right, previous, next, + previous_punctuation: before.punctuation, + next_punctuation: after.punctuation, } } diff --git a/src/unicode_cjk.rs b/src/unicode_cjk.rs new file mode 100644 index 0000000..3b5d456 --- /dev/null +++ b/src/unicode_cjk.rs @@ -0,0 +1,155 @@ +//! CJK characters and variation selectors for CJK-friendly emphasis flanking. +//! +//! The CommonMark CJK-friendly amendments +//! () define a CJK character +//! by East Asian Width, emoji presentation, and the Hangul script. The set is +//! copied from the amendments' `ranges.md` at commit `5b17173` (Unicode 18), +//! which upstream prints with `node --run print-ranges`; a new Unicode version +//! is taken from a newer `ranges.md` the same way. `no_std`, no allocation, no +//! dependency. + +/// Inclusive code-point ranges of CJK characters, sorted ascending and +/// non-overlapping. +static CJK_RANGES: [(char, char); 64] = [ + ('\u{1100}', '\u{11FF}'), + ('\u{20A9}', '\u{20A9}'), + ('\u{2329}', '\u{232A}'), + ('\u{2630}', '\u{2637}'), + ('\u{268A}', '\u{268F}'), + ('\u{2E80}', '\u{2E99}'), + ('\u{2E9B}', '\u{2EF3}'), + ('\u{2F00}', '\u{2FD5}'), + ('\u{2FF0}', '\u{303E}'), + ('\u{3041}', '\u{3096}'), + ('\u{3099}', '\u{30FF}'), + ('\u{3105}', '\u{312F}'), + ('\u{3131}', '\u{318E}'), + ('\u{3190}', '\u{31E5}'), + ('\u{31EF}', '\u{321E}'), + ('\u{3220}', '\u{3247}'), + ('\u{3250}', '\u{A48C}'), + ('\u{A490}', '\u{A4C6}'), + ('\u{A960}', '\u{A97C}'), + ('\u{AC00}', '\u{D7A3}'), + ('\u{D7B0}', '\u{D7C6}'), + ('\u{D7CB}', '\u{D7FB}'), + ('\u{F900}', '\u{FAFF}'), + ('\u{FE10}', '\u{FE19}'), + ('\u{FE30}', '\u{FE52}'), + ('\u{FE54}', '\u{FE66}'), + ('\u{FE68}', '\u{FE6B}'), + ('\u{FF01}', '\u{FFBE}'), + ('\u{FFC2}', '\u{FFC7}'), + ('\u{FFCA}', '\u{FFCF}'), + ('\u{FFD2}', '\u{FFD7}'), + ('\u{FFDA}', '\u{FFDC}'), + ('\u{FFE0}', '\u{FFE6}'), + ('\u{FFE8}', '\u{FFEE}'), + ('\u{16FE0}', '\u{16FE4}'), + ('\u{16FF0}', '\u{16FF6}'), + ('\u{17000}', '\u{18CDA}'), + ('\u{18CFF}', '\u{18D20}'), + ('\u{18D80}', '\u{18DF2}'), + ('\u{18E00}', '\u{19191}'), + ('\u{191A0}', '\u{191D2}'), + ('\u{1AFF0}', '\u{1AFF3}'), + ('\u{1AFF5}', '\u{1AFFB}'), + ('\u{1AFFD}', '\u{1AFFE}'), + ('\u{1B000}', '\u{1B128}'), + ('\u{1B132}', '\u{1B132}'), + ('\u{1B150}', '\u{1B152}'), + ('\u{1B155}', '\u{1B155}'), + ('\u{1B164}', '\u{1B168}'), + ('\u{1B170}', '\u{1B2FB}'), + ('\u{1D300}', '\u{1D356}'), + ('\u{1D360}', '\u{1D376}'), + ('\u{1F1AE}', '\u{1F1AE}'), + ('\u{1F200}', '\u{1F200}'), + ('\u{1F202}', '\u{1F202}'), + ('\u{1F210}', '\u{1F219}'), + ('\u{1F21B}', '\u{1F22E}'), + ('\u{1F230}', '\u{1F231}'), + ('\u{1F237}', '\u{1F237}'), + ('\u{1F23B}', '\u{1F23B}'), + ('\u{1F240}', '\u{1F248}'), + ('\u{1F260}', '\u{1F265}'), + ('\u{1F7DA}', '\u{1F7DA}'), + ('\u{20000}', '\u{3FFFD}'), +]; + +/// Whether `value` is a CJK character as the amendments define it. +pub(crate) fn is_cjk(value: char) -> bool { + value >= '\u{1100}' + && CJK_RANGES + .binary_search_by(|&(start, end)| { + if value < start { + core::cmp::Ordering::Greater + } else if value > end { + core::cmp::Ordering::Less + } else { + core::cmp::Ordering::Equal + } + }) + .is_ok() +} + +/// A non-emoji general-use variation selector: U+FE00–U+FE0E, the Variation +/// Selectors block without the emoji presentation selector U+FE0F. +pub(crate) fn is_general_use_variation_selector(value: char) -> bool { + ('\u{FE00}'..='\u{FE0E}').contains(&value) +} + +/// An ideographic variation selector: the Variation Selectors Supplement block. +pub(crate) fn is_ideographic_variation_selector(value: char) -> bool { + ('\u{E0100}'..='\u{E01EF}').contains(&value) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ranges_are_sorted_and_disjoint() { + assert!(CJK_RANGES.windows(2).all(|pair| pair[0].1 < pair[1].0)); + assert!(CJK_RANGES.iter().all(|&(start, end)| start <= end)); + } + + #[test] + fn cjk_edges() { + for cjk in [ + '\u{1100}', + '\u{11FF}', + '重', + '。', + '」', + '！', + '가', + '\u{3FFFD}', + '\u{1F202}', + ] { + assert!(is_cjk(cjk), "{cjk:?}"); + } + for other in [ + 'a', + '.', + '\u{10FF}', + '\u{1200}', + '\u{3040}', + '\u{1F600}', + '\u{3FFFE}', + '“', + ] { + assert!(!is_cjk(other), "{other:?}"); + } + } + + #[test] + fn variation_selectors() { + assert!(is_general_use_variation_selector('\u{FE00}')); + assert!(is_general_use_variation_selector('\u{FE0E}')); + assert!(!is_general_use_variation_selector('\u{FE0F}')); + assert!(is_ideographic_variation_selector('\u{E0100}')); + assert!(is_ideographic_variation_selector('\u{E01EF}')); + assert!(!is_ideographic_variation_selector('\u{E01F0}')); + } +} diff --git a/tests/fixtures/syntax_decisions/cjk_friendly_emphasis.cases b/tests/fixtures/syntax_decisions/cjk_friendly_emphasis.cases new file mode 100644 index 0000000..eca1736 --- /dev/null +++ b/tests/fixtures/syntax_decisions/cjk_friendly_emphasis.cases @@ -0,0 +1,866 @@ +# markdown-syntax AST->HTML conformance suite v1: syntax decisions +source: cjk_friendly_emphasis +decision: Delimiter runs flank by the CommonMark CJK-friendly amendments (decision 0018). +count: 123 + +--- case 1 options - label-bytes 106 input-bytes 67 expected-bytes 87 +from markdown-cjk-friendly 5b17173 testcases/strong.md:1; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +これは**私のやりたかったこと。**だからするの。 +--- expected +

これは私のやりたかったこと。だからするの。

+--- end +--- case 2 options - label-bytes 106 input-bytes 93 expected-bytes 148 +from markdown-cjk-friendly 5b17173 testcases/strong.md:3; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**[製品ほげ](./product-foo)**と**[製品ふが](./product-bar)**をお試しください +--- expected +

製品ほげと製品ふがをお試しください

+--- end +--- case 3 options - label-bytes 106 input-bytes 40 expected-bytes 71 +from markdown-cjk-friendly 5b17173 testcases/strong.md:5; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +単語と**[単語と](word-and)**単語 +--- expected +

単語と単語と単語

+--- end +--- case 4 options - label-bytes 106 input-bytes 64 expected-bytes 84 +from markdown-cjk-friendly 5b17173 testcases/strong.md:7; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**これは太字になりません。**ご注意ください。 +--- expected +

これは太字になりません。ご注意ください。

+--- end +--- case 5 options - label-bytes 106 input-bytes 82 expected-bytes 102 +from markdown-cjk-friendly 5b17173 testcases/strong.md:9; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +カッコに注意**（太字にならない）**文が続く場合に要警戒。 +--- expected +

カッコに注意（太字にならない）文が続く場合に要警戒。

+--- end +--- case 6 options - label-bytes 107 input-bytes 69 expected-bytes 100 +from markdown-cjk-friendly 5b17173 testcases/strong.md:11; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**[リンク](https://example.com)**も注意。（画像も同様） +--- expected +

リンクも注意。（画像も同様）

+--- end +--- case 7 options - label-bytes 107 input-bytes 38 expected-bytes 56 +from markdown-cjk-friendly 5b17173 testcases/strong.md:13; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +先頭の**\`コード\`も注意。** +--- expected +

先頭の`コード`も注意。

+--- end +--- case 8 options - label-bytes 107 input-bytes 38 expected-bytes 56 +from markdown-cjk-friendly 5b17173 testcases/strong.md:15; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**末尾の\`コード\`**も注意。 +--- expected +

末尾の`コード`も注意。

+--- end +--- case 9 options - label-bytes 107 input-bytes 42 expected-bytes 62 +from markdown-cjk-friendly 5b17173 testcases/strong.md:17; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +税込**¥10,000**で入手できます。 +--- expected +

税込¥10,000で入手できます。

+--- end +--- case 10 options - label-bytes 107 input-bytes 25 expected-bytes 45 +from markdown-cjk-friendly 5b17173 testcases/strong.md:19; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +正解は**④**です。 +--- expected +

正解は④です。

+--- end +--- case 11 options - label-bytes 107 input-bytes 46 expected-bytes 66 +from markdown-cjk-friendly 5b17173 testcases/strong.md:21; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**「こんにちわ」**といった +--- expected +

太郎は「こんにちわ」といった

+--- end +--- case 12 options - label-bytes 107 input-bytes 42 expected-bytes 72 +from markdown-cjk-friendly 5b17173 testcases/strong.md:23; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**"こんにちわ"**といった +--- expected +

太郎は"こんにちわ"といった

+--- end +--- case 13 options - label-bytes 107 input-bytes 40 expected-bytes 60 +from markdown-cjk-friendly 5b17173 testcases/strong.md:25; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**こんにちわ**といった +--- expected +

太郎はこんにちわといった

+--- end +--- case 14 options - label-bytes 107 input-bytes 36 expected-bytes 56 +from markdown-cjk-friendly 5b17173 testcases/strong.md:27; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**「Hello」**といった +--- expected +

太郎は「Hello」といった

+--- end +--- case 15 options - label-bytes 107 input-bytes 32 expected-bytes 62 +from markdown-cjk-friendly 5b17173 testcases/strong.md:29; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**"Hello"**といった +--- expected +

太郎は"Hello"といった

+--- end +--- case 16 options - label-bytes 107 input-bytes 30 expected-bytes 50 +from markdown-cjk-friendly 5b17173 testcases/strong.md:31; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**Hello**といった +--- expected +

太郎はHelloといった

+--- end +--- case 17 options - label-bytes 107 input-bytes 40 expected-bytes 60 +from markdown-cjk-friendly 5b17173 testcases/strong.md:33; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**「Oh my god」**といった +--- expected +

太郎は「Oh my god」といった

+--- end +--- case 18 options - label-bytes 107 input-bytes 36 expected-bytes 66 +from markdown-cjk-friendly 5b17173 testcases/strong.md:35; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**"Oh my god"**といった +--- expected +

太郎は"Oh my god"といった

+--- end +--- case 19 options - label-bytes 107 input-bytes 34 expected-bytes 54 +from markdown-cjk-friendly 5b17173 testcases/strong.md:37; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**Oh my god**といった +--- expected +

太郎はOh my godといった

+--- end +--- case 20 options - label-bytes 107 input-bytes 89 expected-bytes 135 +from markdown-cjk-friendly 5b17173 testcases/strong.md:39; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**C#**や**F#**は**「.NET」**というプラットフォーム上で動作します。 +--- expected +

C#やF#は「.NET」というプラットフォーム上で動作します。

+--- end +--- case 21 options - label-bytes 107 input-bytes 33 expected-bytes 53 +from markdown-cjk-friendly 5b17173 testcases/strong.md:41; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +IDが**001号**になります。 +--- expected +

IDが001号になります。

+--- end +--- case 22 options - label-bytes 107 input-bytes 39 expected-bytes 59 +from markdown-cjk-friendly 5b17173 testcases/strong.md:43; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +IDが**００１号**になります。 +--- expected +

IDが００１号になります。

+--- end +--- case 23 options - label-bytes 107 input-bytes 54 expected-bytes 74 +from markdown-cjk-friendly 5b17173 testcases/strong.md:45; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +Go**「初心者」**を対象とした記事です。 +--- expected +

Go「初心者」を対象とした記事です。

+--- end +--- case 24 options - label-bytes 107 input-bytes 48 expected-bytes 79 +from markdown-cjk-friendly 5b17173 testcases/strong.md:47; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**[リンク](https://example.com)**も注意。 +--- expected +

リンクも注意。

+--- end +--- case 25 options - label-bytes 107 input-bytes 31 expected-bytes 51 +from markdown-cjk-friendly 5b17173 testcases/strong.md:49; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**⻲田太郎**と申します +--- expected +

⻲田太郎と申します

+--- end +--- case 26 options - label-bytes 107 input-bytes 29 expected-bytes 49 +from markdown-cjk-friendly 5b17173 testcases/strong.md:51; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +・**㋐**:選択肢１つ目 +--- expected +

・㋐:選択肢１つ目

+--- end +--- case 27 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:53; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真，**她 +--- expected +

真，她

+--- end +--- case 28 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:55; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真。**她 +--- expected +

真。她

+--- end +--- case 29 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:57; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真、**她 +--- expected +

真、她

+--- end +--- case 30 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:59; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真；**她 +--- expected +

真；她

+--- end +--- case 31 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:61; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真：**她 +--- expected +

真：她

+--- end +--- case 32 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:63; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真？**她 +--- expected +

真？她

+--- end +--- case 33 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:65; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真！**她 +--- expected +

真！她

+--- end +--- case 34 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:67; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真“**她 +--- expected +

真“她

+--- end +--- case 35 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:69; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真”**她 +--- expected +

真”她

+--- end +--- case 36 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:71; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真‘**她 +--- expected +

真‘她

+--- end +--- case 37 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:73; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真’**她 +--- expected +

真’她

+--- end +--- case 38 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:75; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真（**她 +--- expected +

真（她

+--- end +--- case 39 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:77; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +真**（她** +--- expected +

真（她

+--- end +--- case 40 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:79; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真）**她 +--- expected +

真）她

+--- end +--- case 41 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:81; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真【**她 +--- expected +

真【她

+--- end +--- case 42 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:83; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +真**【她** +--- expected +

真【她

+--- end +--- case 43 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:85; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真】**她 +--- expected +

真】她

+--- end +--- case 44 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:87; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真《**她 +--- expected +

真《她

+--- end +--- case 45 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:89; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +真**《她** +--- expected +

真《她

+--- end +--- case 46 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:91; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真》**她 +--- expected +

真》她

+--- end +--- case 47 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:93; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真—**她 +--- expected +

真—她

+--- end +--- case 48 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:95; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真～**她 +--- expected +

真～她

+--- end +--- case 49 options - label-bytes 107 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:97; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真…**她 +--- expected +

真…她

+--- end +--- case 50 options - label-bytes 107 input-bytes 12 expected-bytes 32 +from markdown-cjk-friendly 5b17173 testcases/strong.md:99; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真·**她 +--- expected +

真·她

+--- end +--- case 51 options - label-bytes 108 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:101; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真〃**她 +--- expected +

真〃她

+--- end +--- case 52 options - label-bytes 108 input-bytes 11 expected-bytes 31 +from markdown-cjk-friendly 5b17173 testcases/strong.md:103; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真-**她 +--- expected +

真-她

+--- end +--- case 53 options - label-bytes 108 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:105; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真々**她 +--- expected +

真々她

+--- end +--- case 54 options - label-bytes 108 input-bytes 10 expected-bytes 30 +from markdown-cjk-friendly 5b17173 testcases/strong.md:107; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真**她 +--- expected +

真她

+--- end +--- case 55 options - label-bytes 108 input-bytes 14 expected-bytes 34 +from markdown-cjk-friendly 5b17173 testcases/strong.md:109; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真，** 她 +--- expected +

真， 她

+--- end +--- case 56 options - label-bytes 108 input-bytes 13 expected-bytes 33 +from markdown-cjk-friendly 5b17173 testcases/strong.md:111; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真**，她 +--- expected +

真，她

+--- end +--- case 57 options - label-bytes 108 input-bytes 29 expected-bytes 36 +from markdown-cjk-friendly 5b17173 testcases/strong.md:113; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**真，**​她 +--- expected +

真，​她

+--- end +--- case 58 options - label-bytes 108 input-bytes 37 expected-bytes 57 +from markdown-cjk-friendly 5b17173 testcases/strong.md:115; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +私は**⻲田太郎**と申します +--- expected +

私は⻲田太郎と申します

+--- end +--- case 59 options - label-bytes 108 input-bytes 37 expected-bytes 57 +from markdown-cjk-friendly 5b17173 testcases/strong.md:117; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +選択肢**㋐**: 1つ目の選択肢 +--- expected +

選択肢㋐: 1つ目の選択肢

+--- end +--- case 60 options - label-bytes 108 input-bytes 46 expected-bytes 66 +from markdown-cjk-friendly 5b17173 testcases/strong.md:119; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**さようなら︙**と太郎はいった。 +--- expected +

さようなら︙と太郎はいった。

+--- end +--- case 61 options - label-bytes 108 input-bytes 46 expected-bytes 66 +from markdown-cjk-friendly 5b17173 testcases/strong.md:121; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +.NET**（.NET Frameworkは不可）**では、 +--- expected +

.NET（.NET Frameworkは不可）では、

+--- end +--- case 62 options - label-bytes 108 input-bytes 53 expected-bytes 73 +from markdown-cjk-friendly 5b17173 testcases/strong.md:123; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +「禰󠄀」の偏は示ではなく**礻**です。 +--- expected +

「禰󠄀」の偏は示ではなく礻です。

+--- end +--- case 63 options - label-bytes 108 input-bytes 31 expected-bytes 51 +from markdown-cjk-friendly 5b17173 testcases/strong.md:125; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +Git**（注：不是GitHub）** +--- expected +

Git（注：不是GitHub）

+--- end +--- case 64 options - label-bytes 108 input-bytes 49 expected-bytes 69 +from markdown-cjk-friendly 5b17173 testcases/strong.md:127; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +太郎は**「こんにちわ」**といった。 +--- expected +

太郎は「こんにちわ」といった。

+--- end +--- case 65 options - label-bytes 108 input-bytes 39 expected-bytes 59 +from markdown-cjk-friendly 5b17173 testcases/strong.md:129; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𰻞𰻞**（ビャンビャン）**麺 +--- expected +

𰻞𰻞（ビャンビャン）麺

+--- end +--- case 66 options - label-bytes 108 input-bytes 35 expected-bytes 55 +from markdown-cjk-friendly 5b17173 testcases/strong.md:131; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𰻞𰻞**(ビャンビャン)**麺 +--- expected +

𰻞𰻞(ビャンビャン)麺

+--- end +--- case 67 options - label-bytes 108 input-bytes 78 expected-bytes 98 +from markdown-cjk-friendly 5b17173 testcases/strong.md:133; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +ハイパーテキストコーヒーポット制御プロトコル**(HTCPCP)** +--- expected +

ハイパーテキストコーヒーポット制御プロトコル(HTCPCP)

+--- end +--- case 68 options - label-bytes 108 input-bytes 12 expected-bytes 32 +from markdown-cjk-friendly 5b17173 testcases/strong.md:135; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +﨑**(崎)** +--- expected +

﨑(崎)

+--- end +--- case 69 options - label-bytes 108 input-bytes 52 expected-bytes 83 +from markdown-cjk-friendly 5b17173 testcases/strong.md:137; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +国際規格**[ECMA-262](https://tc39.es/ecma262/)** +--- expected +

国際規格ECMA-262

+--- end +--- case 70 options - label-bytes 108 input-bytes 21 expected-bytes 41 +from markdown-cjk-friendly 5b17173 testcases/strong.md:139; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +㐧**(第の俗字)** +--- expected +

㐧(第の俗字)

+--- end +--- case 71 options - label-bytes 108 input-bytes 34 expected-bytes 54 +from markdown-cjk-friendly 5b17173 testcases/strong.md:141; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𠮟**(こちらが正式表記)** +--- expected +

𠮟(こちらが正式表記)

+--- end +--- case 72 options - label-bytes 108 input-bytes 31 expected-bytes 51 +from markdown-cjk-friendly 5b17173 testcases/strong.md:143; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𪜈**(トモの合略仮名)** +--- expected +

𪜈(トモの合略仮名)

+--- end +--- case 73 options - label-bytes 108 input-bytes 22 expected-bytes 42 +from markdown-cjk-friendly 5b17173 testcases/strong.md:145; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𫠉**(馬の俗字)** +--- expected +

𫠉(馬の俗字)

+--- end +--- case 74 options - label-bytes 108 input-bytes 31 expected-bytes 51 +from markdown-cjk-friendly 5b17173 testcases/strong.md:147; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +谺𬤲**(こだま)**石神社 +--- expected +

谺𬤲(こだま)石神社

+--- end +--- case 75 options - label-bytes 108 input-bytes 28 expected-bytes 48 +from markdown-cjk-friendly 5b17173 testcases/strong.md:149; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +石𮧟**(いしただら)** +--- expected +

石𮧟(いしただら)

+--- end +--- case 76 options - label-bytes 108 input-bytes 54 expected-bytes 74 +from markdown-cjk-friendly 5b17173 testcases/strong.md:151; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**推荐几个框架：**React、Vue等前端框架。 +--- expected +

推荐几个框架：React、Vue等前端框架。

+--- end +--- case 77 options - label-bytes 108 input-bytes 43 expected-bytes 63 +from markdown-cjk-friendly 5b17173 testcases/strong.md:153; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +葛󠄀**(こちらが正式表記)**城市 +--- expected +

葛󠄀(こちらが正式表記)城市

+--- end +--- case 78 options - label-bytes 108 input-bytes 43 expected-bytes 63 +from markdown-cjk-friendly 5b17173 testcases/strong.md:155; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +禰󠄀**(こちらが正式表記)**豆子 +--- expected +

禰󠄀(こちらが正式表記)豆子

+--- end +--- case 79 options - label-bytes 108 input-bytes 17 expected-bytes 37 +from markdown-cjk-friendly 5b17173 testcases/strong.md:157; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𱟛**(U+317DB)** +--- expected +

𱟛(U+317DB)

+--- end +--- case 80 options - label-bytes 108 input-bytes 69 expected-bytes 89 +from markdown-cjk-friendly 5b17173 testcases/strong.md:159; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +阿寒湖アイヌシアターイコㇿ**(Akanko Ainu Theater Ikor)** +--- expected +

阿寒湖アイヌシアターイコㇿ(Akanko Ainu Theater Ikor)

+--- end +--- case 81 options - label-bytes 108 input-bytes 25 expected-bytes 45 +from markdown-cjk-friendly 5b17173 testcases/strong.md:161; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +あ𛀙**(か)**よろし +--- expected +

あ𛀙(か)よろし

+--- end +--- case 82 options - label-bytes 108 input-bytes 41 expected-bytes 61 +from markdown-cjk-friendly 5b17173 testcases/strong.md:163; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𮹝**(simplified form of 龘 in China)** +--- expected +

𮹝(simplified form of 龘 in China)

+--- end +--- case 83 options - label-bytes 108 input-bytes 47 expected-bytes 80 +from markdown-cjk-friendly 5b17173 testcases/strong.md:165; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +大塚︀**(U+585A U+FE00)** 大塚**(U+FA10)** +--- expected +

大塚︀(U+585A U+FE00) 大塚(U+FA10)

+--- end +--- case 84 options - label-bytes 108 input-bytes 24 expected-bytes 44 +from markdown-cjk-friendly 5b17173 testcases/strong.md:167; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +〽︎**(庵点)**は、 +--- expected +

〽︎(庵点)は、

+--- end +--- case 85 options - label-bytes 108 input-bytes 22 expected-bytes 42 +from markdown-cjk-friendly 5b17173 testcases/strong.md:169; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**“︁Git”︁**Hub +--- expected +

“︁Git”︁Hub

+--- end +--- case 86 options - label-bytes 106 input-bytes 64 expected-bytes 95 +from markdown-cjk-friendly 5b17173 testcases/korean.md:1; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**이 [링크](https://example.kr/)**만을 강조하고 싶다. +--- expected +

이 링크만을 강조하고 싶다.

+--- end +--- case 87 options - label-bytes 106 input-bytes 30 expected-bytes 50 +from markdown-cjk-friendly 5b17173 testcases/korean.md:3; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**스크립트(script)**라고 +--- expected +

스크립트(script)라고

+--- end +--- case 88 options - label-bytes 106 input-bytes 69 expected-bytes 100 +from markdown-cjk-friendly 5b17173 testcases/korean.md:5; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +패키지를 발행하려면 **`npm publish`**를 실행하십시오. +--- expected +

패키지를 발행하려면 npm publish를 실행하십시오.

+--- end +--- case 89 options - label-bytes 106 input-bytes 27 expected-bytes 47 +from markdown-cjk-friendly 5b17173 testcases/korean.md:7; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**안녕(hello)**하세요. +--- expected +

안녕(hello)하세요.

+--- end +--- case 90 options - label-bytes 106 input-bytes 10 expected-bytes 30 +from markdown-cjk-friendly 5b17173 testcases/korean.md:9; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +ᅡ**(a)** +--- expected +

ᅡ(a)

+--- end +--- case 91 options - label-bytes 107 input-bytes 10 expected-bytes 30 +from markdown-cjk-friendly 5b17173 testcases/korean.md:11; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +**(k)**ᄏ +--- expected +

(k)ᄏ

+--- end +--- case 92 options - label-bytes 112 input-bytes 9 expected-bytes 29 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:1; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**〰**a +--- expected +

a〰a

+--- end +--- case 93 options - label-bytes 112 input-bytes 9 expected-bytes 29 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:3; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**〽**a +--- expected +

a〽a

+--- end +--- case 94 options - label-bytes 112 input-bytes 10 expected-bytes 30 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:5; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**🈂**a +--- expected +

a🈂a

+--- end +--- case 95 options - label-bytes 112 input-bytes 10 expected-bytes 30 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:7; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**🈷**a +--- expected +

a🈷a

+--- end +--- case 96 options - label-bytes 112 input-bytes 9 expected-bytes 29 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:9; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**㊗**a +--- expected +

a㊗a

+--- end +--- case 97 options - label-bytes 113 input-bytes 9 expected-bytes 29 +from markdown-cjk-friendly 5b17173 testcases/pseudo-emoji.md:11; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**㊙**a +--- expected +

a㊙a

+--- end +--- case 98 options - label-bytes 107 input-bytes 9 expected-bytes 16 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:2; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**a«**a +--- expected +

a**a«**a

+--- end +--- case 99 options - label-bytes 107 input-bytes 9 expected-bytes 16 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:4; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**»a**a +--- expected +

a**»a**a

+--- end +--- case 100 options - label-bytes 107 input-bytes 10 expected-bytes 17 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:6; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**a∇**a +--- expected +

a**a∇**a

+--- end +--- case 101 options - label-bytes 107 input-bytes 10 expected-bytes 17 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:8; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**∇a**a +--- expected +

a**∇a**a

+--- end +--- case 102 options - label-bytes 108 input-bytes 11 expected-bytes 18 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:10; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**a𝜵**a +--- expected +

a**a𝜵**a

+--- end +--- case 103 options - label-bytes 108 input-bytes 11 expected-bytes 18 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:12; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**𝜵a**a +--- expected +

a**𝜵a**a

+--- end +--- case 104 options - label-bytes 108 input-bytes 11 expected-bytes 18 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:14; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**𐬻a**a +--- expected +

a**𐬻a**a

+--- end +--- case 105 options - label-bytes 108 input-bytes 11 expected-bytes 18 +from markdown-cjk-friendly 5b17173 testcases/non-bmp.md:16; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a**a𐬻**a +--- expected +

a**a𐬻**a

+--- end +--- case 106 options - label-bytes 120 input-bytes 25 expected-bytes 45 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:1; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +__注意__：注意事項 +--- expected +

注意：注意事項

+--- end +--- case 107 options - label-bytes 120 input-bytes 25 expected-bytes 45 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:3; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +注意：__注意事項__ +--- expected +

注意：注意事項

+--- end +--- case 108 options - label-bytes 120 input-bytes 29 expected-bytes 43 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:5; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +正體字。︁_Traditional._ +--- expected +

正體字。︁Traditional.

+--- end +--- case 109 options - label-bytes 120 input-bytes 40 expected-bytes 60 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:7; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +正體字。︁__Hong Kong and Taiwan.__ +--- expected +

正體字。︁Hong Kong and Taiwan.

+--- end +--- case 110 options - label-bytes 120 input-bytes 40 expected-bytes 54 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:9; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +简体字 / 新字体。︀_Simplified._ +--- expected +

简体字 / 新字体。︀Simplified.

+--- end +--- case 111 options - label-bytes 121 input-bytes 55 expected-bytes 75 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:11; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +简体字 / 新字体。︀__Mainland China or Japan.__ +--- expected +

简体字 / 新字体。︀Mainland China or Japan.

+--- end +--- case 112 options - label-bytes 121 input-bytes 22 expected-bytes 42 +from markdown-cjk-friendly 5b17173 testcases/underscore-cjk-punct.md:13; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +“︁Git”︁__Hub__ +--- expected +

“︁Git”︁Hub

+--- end +--- case 113 options - label-bytes 117 input-bytes 11 expected-bytes 25 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:1; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a~~a()~~あ +--- expected +

aa()あ

+--- end +--- case 114 options - label-bytes 117 input-bytes 11 expected-bytes 25 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:3; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +あ~~()a~~a +--- expected +

あ()aa

+--- end +--- case 115 options - label-bytes 117 input-bytes 12 expected-bytes 26 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:5; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +𩸽~~()a~~a +--- expected +

𩸽()aa

+--- end +--- case 116 options - label-bytes 117 input-bytes 12 expected-bytes 26 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:7; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a~~a()~~𩸽 +--- expected +

aa()𩸽

+--- end +--- case 117 options - label-bytes 117 input-bytes 15 expected-bytes 29 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:9; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +葛󠄀~~()a~~a +--- expected +

葛󠄀()aa

+--- end +--- case 118 options - label-bytes 118 input-bytes 14 expected-bytes 28 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:11; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +羽︀~~()a~~a +--- expected +

羽︀()aa

+--- end +--- case 119 options - label-bytes 118 input-bytes 12 expected-bytes 26 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:13; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +a~~「a~~」 +--- expected +

a「a」

+--- end +--- case 120 options - label-bytes 118 input-bytes 12 expected-bytes 26 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:15; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +「~~a」~~a +--- expected +

「a」a

+--- end +--- case 121 options - label-bytes 118 input-bytes 13 expected-bytes 34 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:17; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +~~a~~：~~a~~ +--- expected +

a：a

+--- end +--- case 122 options - label-bytes 118 input-bytes 27 expected-bytes 41 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:19; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +~~日本語。︀~~English. +--- expected +

日本語。︀English.

+--- end +--- case 123 options - label-bytes 118 input-bytes 18 expected-bytes 32 +from markdown-cjk-friendly 5b17173 testcases/gfm-strikethrough.md:21; verified: micromark-extension-cjk-friendly 2.0.2 +--- input +~~“︁a”︁~~a +--- expected +

“︁a”︁a

+--- end diff --git a/tests/one_syntax.rs b/tests/one_syntax.rs index ff1e51a..68b65ae 100644 --- a/tests/one_syntax.rs +++ b/tests/one_syntax.rs @@ -1,4 +1,4 @@ -//! Locks the one syntax `parse` reads (decisions/0015): the constructs it +//! Locks the one syntax `parse` reads (decisions/0018): the constructs it //! recognizes, the ones it reads as text, and the spellings it records. use markdown_syntax::prelude::*; diff --git a/tests/parse_inline_regressions.rs b/tests/parse_inline_regressions.rs index ce4be43..da15dca 100644 --- a/tests/parse_inline_regressions.rs +++ b/tests/parse_inline_regressions.rs @@ -546,16 +546,63 @@ mod review_unicode { } } - /// `a*。*b` — CJK full stop U+3002 is Unicode punctuation; same rule, no emphasis. + /// `a*。*b` — CJK full stop U+3002 is CJK punctuation, which does not + /// block flanking (inline-syntax: CJK-friendly emphasis flanking). #[test] - fn emphasis_does_not_open_before_cjk_punctuation() { + fn emphasis_opens_before_cjk_punctuation() { let nodes = paragraph_inlines("a*\u{3002}*b\n"); assert!( - !nodes.iter().any(|node| matches!(node, Inline::Emphasis(_))), - "no emphasis should form around CJK punctuation: {nodes:?}" + nodes.iter().any(|node| matches!(node, Inline::Emphasis(_))), + "emphasis forms around CJK punctuation: {nodes:?}" ); } + fn text(inline: &Inline) -> &str { + match inline { + Inline::Text(text) => &text.value, + other => panic!("expected text, got {other:?}"), + } + } + + // inline-syntax: CJK-friendly emphasis flanking + #[test] + fn strong_ends_in_cjk_punctuation_before_a_cjk_char() { + let nodes = paragraph_inlines("这是**重点。**然后\n"); + assert_eq!(nodes.len(), 3, "{nodes:?}"); + assert_eq!(text(&nodes[0]), "这是"); + match &nodes[1] { + Inline::Strong(strong) => { + assert_eq!(strong.children.len(), 1); + assert_eq!(text(&strong.children[0]), "重点。"); + } + other => panic!("expected strong, got {other:?}"), + } + assert_eq!(text(&nodes[2]), "然后"); + } + + // inline-syntax: CJK-friendly emphasis flanking + #[test] + fn every_emphasis_like_mark_flanks_beside_cjk() { + for (input, is_mark) in [ + ( + "这是~~重点。~~然后\n", + (|node: &Inline| matches!(node, Inline::Delete(_))) as fn(&Inline) -> bool, + ), + ("这是==重点。==然后\n", |node| { + matches!(node, Inline::Mark(_)) + }), + ("这是*重点。*然后\n", |node| { + matches!(node, Inline::Emphasis(_)) + }), + ] { + let nodes = paragraph_inlines(input); + assert!( + nodes.len() == 3 && is_mark(&nodes[1]), + "{input:?}: {nodes:?}" + ); + } + } + /// Control: a `*` between ASCII letters still opens/closes emphasis — the /// Unicode-punctuation change must not over-restrict intraword emphasis. #[test] diff --git a/tests/syntax_decisions.rs b/tests/syntax_decisions.rs index e29ee9d..9b37d57 100644 --- a/tests/syntax_decisions.rs +++ b/tests/syntax_decisions.rs @@ -4,8 +4,9 @@ //! //! Each case in `tests/fixtures/syntax_decisions/` names the upstream case it //! came from, numbered as in its file under `tests/fixtures/conformance/` at -//! commit 8ba83e2, before it left the bench, and what verified its expected -//! HTML: the reference renderers that +//! commit 8ba83e2, before it left the bench, or, for an amendment the syntax +//! adopts, the file and line of the amendment's own test input, and what +//! verified its expected HTML: the reference renderers that //! produce the same HTML (cmark-gfm, commonmark.js, micromark with GFM, math, //! and frontmatter extensions and single-tilde strikethrough off), or the //! decision that sets the reading where no reference shares it. The HTML must @@ -95,6 +96,7 @@ fn options(case: &Case) -> HtmlOptions { } // inline-syntax: CommonMark inlines +// inline-syntax: CJK-friendly emphasis flanking #[test] fn decision_cases_render_as_verified() { let mut paths: Vec<_> = fs::read_dir(ROOT) @@ -127,5 +129,5 @@ fn decision_cases_render_as_verified() { checked += 1; } } - assert_eq!(checked, 97, "syntax decision cases drifted"); + assert_eq!(checked, 220, "syntax decision cases drifted"); }