Spec 001 T030 follow-up: long-token breaks

specs/001-pathway-pdfs/contracts/cli.md

@@ -203,9 +203,11 @@ FR-011's check, alone. Also run by `book`.
     figure's link (a diagram's labels). Measured against the section, not in points, so a
     part Chromium printed scaled down still reads its names, its scale not a fault: the
     layout keeps realistic content inside the text column (long tokens may break, a table's
-    columns narrow to their longest words), but a table of very many columns can still be
-    wider; `no-stable-id`: its text holds no `Stable identifier` followed by its stable
-    identifier;
+    columns narrow to their longest words), but a table whose longest words, side by side,
+    are wider than the column can still be wider: a token of up to 40 characters does not
+    break, so three columns of 40-character words are wider, four of 30 and five of 21 (a
+    longer token breaks into pieces of 14 to 20, and takes six columns or more);
+    `no-stable-id`: its text holds no `Stable identifier` followed by its stable identifier;
   - `missing-figure`: a pathway's or reaction's section with a figure by the Figure rules,
     whose source is in `--files` (its diagram's two files, its EHLD; a reaction's is the
     graph), has no link to that figure's interactive form below its heading. The figure is

specs/001-pathway-pdfs/research.md

@@ -954,7 +954,10 @@ embedded.
     28 human events have a `<table>` in their summation, so a table wider than the text
     column, which Chromium printed by scaling its whole part down below the theme's sizes,
     would have reached the real book: the layout now keeps realistic content inside the
-    column (pages.css; a table of very many columns can still be wider), and the check
+    column (pages.css; a table whose longest words, side by side, are wider than the
+    column can still be wider: measured, three columns of unbroken 40-character words,
+    four of 30 or five of 21; a longer token, broken into pieces of 14 to 20, takes six
+    columns or more), and the check
     reads headings against the section's own body text, so a part printed scaled down is
     not a fault. No human summation holds a
     `<pre>` block, and no human event's name holds a soft hyphen, a zero-width or

specs/001-pathway-pdfs/tasks.md

@@ -536,6 +536,7 @@ passes.
       - **`overflow-wrap: anywhere` broke short columns' words (high).** In a table it let every column narrow to a character, so a table wider than the column broke its short columns' words (`Da`/`te`). `body` now has `overflow-wrap: break-word`, which keeps each column at its longest word, and the layout gives a token longer than 40 characters a `<wbr>` every 20 (`text()` in `src/layout/html.ts`, grapheme by grapheme, for every text it writes; `markupBreaks` for a summation's text runs, a character reference one unit; an attribute's value and the page's `<title>` never, through `plainText`), which adds no character to what is read. An edit's date is kept whole (`white-space: nowrap`), as a column narrowed to its longest word broke `2026-01-04` at its hyphens. Test: an edit's names, 30 of them, and a summation table of `Phosphofructokinase` beside a column of prose: exit 0, `Date` and `Action` whole and no fragment of them, every date whole, `Phosphofructokinase` whole four times; on #160's CSS it read `Da`, `te`, `Ac`. The overflow test is kept: its table is now four columns of long words, as wide as release 98's are (six columns of 28-letter words are wider than the column whatever breaks, and a part holding one is printed scaled down, which the check tolerates), and its 140-character token also sits in a table cell, where only its `<wbr>`s let the column narrow.
       - **Lows.** The docs no longer say nothing overflows: the layout keeps realistic content inside the column, a table of very many columns can still be wider, and the check's measure against the section tolerates a part printed scaled down (contracts/cli.md, research R8). `folded()` builds its pattern from FR-017's `NEVER_DRAWN` (`src/layout/characters.ts`), not a copy. `identifies()` reads a candidate's section only to the next place a section may start (an event's first link to its page, or an outline entry with an event's name). Recorded: the failed-removal test (`test/print/temporary.test.ts`) relies on `chmod` keeping a folder from being emptied, so it does not hold when run as root.
       - **Tests:** 439 before, 441 after (the labels and the short-columns tests). Each of these mutants made one fail: the page's commonest size with the figures' labels counted, as before (the labels test); `overflow-wrap: anywhere` again, and the dates allowed to break (the short-columns test, each); no `<wbr>` at all, and none in summations (the overflow test, each). Survived: the label's size alone, and the page's size without the figures' labels alone (each covers the other on the fixture; both kept, the second as the fallback for a section with no label); `identifies()` unbounded (no fixture section lacks its label, the only case it changes); no `overflow-wrap` and the tables' `max-width: 100%` (with the `<wbr>`s, nothing tested is wider than a line; both kept, as the review asked); the facts' grid at `minmax(0, 1fr)` and the contents' links at `min-width: 0`, taken out again.
+    - **Follow-up (#163):** #162's review's five lows. A token ends only at a space Chromium breaks a line at (space, tab, line feed, carriage return): a no-break space, U+202F or U+FEFF no longer ends one. A summation's markup is read whole (`markupBreaks`, which `markupText` no longer feeds run by run), its tags never counted or broken into, and a word counted whole through the inline tags (`a`, `b`, `em`, `i`, `span`, `strong`, `sub`, `sup`, `u`), any other tag ending it. A token over 40 breaks into equal pieces of at most 20 (41 is 14, 14 and 13, not 20, 20 and 1). The docs (contracts/cli.md, research R8) say what can still overflow, measured on a page with the book's styles at the text column's width: unbroken tokens side by side, three columns of 40 characters, four of 30, five of 21; a longer token, broken into pieces of 14 to 20, takes six columns or more. Tests: `test/layout/html.test.ts` (7) for `text()` and `markupBreaks`, 441 before, 448 after. Each of these mutants made one fail: the old `/^\s$/u` space (the spaces test, and the markup one through a no-break space); a carriage return and line feed not taken as a space (the spaces test); character references not split out (`&amp;` broken into, the references test); `text()` escaping before it breaks (the escaped test); tags taken as text (the attribute and inline-tags tests); every tag ending a word, and none (the inline-tags test, each); tags counted in a word's length (the same); a `<` before no tag name taken as a tag (the markup test); a break every 20 again (five tests).
     - **Known gaps:** the rules are measured on the fixture alone, whose names are short and whose figures small: T031 runs the check on release 98's book, and its time against R7's 9 minutes. A figure is known by its link: a drawing lost with its link kept is not seen. A heading that lost its link and every entry naming it is found nowhere, and is then three faults. A link landing on a page but not on its target is not a fault (R8: every link landing on a page). Between `bookCommand` returning and the process ending there are no handlers, so a signal then is the default action, exit 128 and its number, the book and its report already in place.
 
 - [ ] T031 [US1] The book on a private copy of release 98's graph.

src/layout/html.ts

@@ -12,34 +12,69 @@ export function plainText(value: string): string {
 }
 
 /**
- * A token longer than this, with no space in it, may break every BREAK_EVERY characters:
- * the pages break words only where they may (`overflow-wrap: break-word`, pages.css), so
- * that a table's short columns keep their words whole, and a token longer than the text
- * column (a run of letters, a long address) would otherwise overflow it, and Chromium then
- * prints its whole part scaled down.
+ * A token longer than this, with no space in it, is given places to break, in equal pieces
+ * of at most BREAK_EVERY characters: the pages break words only where they may
+ * (`overflow-wrap: break-word`, pages.css), so that a table's short columns keep their
+ * words whole, and a token longer than the text column (a run of letters, a long address)
+ * would otherwise overflow it, and Chromium then prints its whole part scaled down.
  */
 export const LONG_TOKEN = 40;
 export const BREAK_EVERY = 20;
 
 /**
- * Units of text (characters as a reader sees them, and a markup run's character references)
- * with `<wbr>` between every BREAK_EVERY of a token longer than LONG_TOKEN: a break
+ * The spaces a token ends at: those Chromium breaks a line at. A no-break space, U+202F or
+ * U+FEFF joins its neighbours, so a token runs through it; a space it could break at but
+ * that is not listed (an en space, say) only adds a `<wbr>` where a break already was.
+ * A carriage return and line feed are one grapheme.
+ */
+const BREAKABLE_SPACE = /^[ \t\n\r]+$/;
+
+/**
+ * A unit of text: a character as a reader sees it, a character reference, or a tag, which
+ * is no character, and which ends the token it is in unless it is inline (`joins`).
+ */
+interface Unit {
+  readonly text: string;
+  readonly tag?: 'joins' | 'ends';
+}
+
+/**
+ * Units with `<wbr>` between the pieces of a token longer than LONG_TOKEN, each piece of
+ * at most BREAK_EVERY units and none more than one shorter than another: a break
  * opportunity, which adds no character to the printed text.
  */
-function breakTokens(units: readonly string[], escape: (unit: string) => string): string {
+function breakTokens(units: readonly Unit[], escape: (unit: string) => string): string {
   let out = '';
-  let token: string[] = [];
+  let token: Unit[] = [];
   const flush = () => {
-    for (let i = 0; i < token.length; i++) {
-      if (token.length > LONG_TOKEN && i > 0 && i % BREAK_EVERY === 0) out += '<wbr>';
-      out += escape(token[i] ?? '');
+    const length = token.filter((unit) => unit.tag === undefined).length;
+    const pieces = length > LONG_TOKEN ? Math.ceil(length / BREAK_EVERY) : 1;
+    // the k-th piece ends at ceil(k * length / pieces) units, so the pieces differ by one at most
+    let next = 1;
+    let counted = 0;
+    for (const unit of token) {
+      if (unit.tag !== undefined) {
+        out += unit.text;
+        continue;
+      }
+      if (next < pieces && counted === Math.ceil((next * length) / pieces)) {
+        out += '<wbr>';
+        next++;
+      }
+      out += escape(unit.text);
+      counted++;
     }
     token = [];
   };
   for (const unit of units) {
-    if (/^\s$/u.test(unit)) {
+    if (unit.tag === 'joins') {
+      token.push(unit);
+    } else if (unit.tag === 'ends') {
+      flush();
+      out += unit.text;
+    } else if (BREAKABLE_SPACE.test(unit.text)) {
       flush();
-      out += escape(unit);
+      out += escape(unit.text);
     } else {
       token.push(unit);
     }
@@ -50,23 +85,46 @@ function breakTokens(units: readonly string[], escape: (unit: string) => string)
 
 /** Characters as a reader sees them: a break never parts a mark from its letter. */
 const GRAPHEMES = new Intl.Segmenter('en', { granularity: 'grapheme' });
-const graphemes = (value: string): string[] =>
-  Array.from(GRAPHEMES.segment(value), (part) => part.segment);
+const graphemes = (value: string): Unit[] =>
+  Array.from(GRAPHEMES.segment(value), (part) => ({ text: part.segment }));
 
 /** Text for an element's content, escaped, a long token given places to break. */
 export function text(value: string): string {
   return breakTokens(graphemes(value), plainText);
 }
 
 /**
- * A run of markup's text (raw: its character references as they are) given places to break
- * in a long token, as `text` gives plain text: for `markupText`'s runs of a summation.
+ * The inline tags of a summation (the sanitiser's, rules.ts): a word may run through them,
+ * as `Na<sub>2</sub>` or a word half in italics, and is counted whole. Any other tag, a
+ * paragraph's or a cell's, ends the word.
  */
-export function markupBreaks(run: string): string {
-  // each character reference one unit, every other text by its characters
-  const units = run
-    .split(/(&(?:#[0-9]+|#x[0-9a-f]+|[a-z][a-z0-9]*);)/giu)
-    .flatMap((part, i) => (i % 2 === 1 ? [part] : graphemes(part)));
+const INLINE_TAGS = new Set(['a', 'b', 'em', 'i', 'span', 'strong', 'sub', 'sup', 'u']);
+
+/** A tag (its attributes' quoted values may hold `>`), or a comment; a `<` before neither is text. */
+const TAG = /(<!--[\s\S]*?(?:-->|$)|<\/?([a-z][a-z0-9]*)(?:[^>"']|"[^"]*"|'[^']*')*>?)/giu;
+const REFERENCE = /(&(?:#[0-9]+|#x[0-9a-f]+|[a-z][a-z0-9]*);)/giu;
+
+/**
+ * A summation's sanitised markup, a long token in its text given places to break as `text`
+ * gives plain text: each character reference one unit, a tag never broken into and never
+ * counted, and a word counted whole through inline tags.
+ */
+export function markupBreaks(markup: string): string {
+  const units: Unit[] = [];
+  const parts = markup.split(TAG);
+  // split gives text, then each tag with its name (or undefined), then text again
+  for (let i = 0; i < parts.length; i += 3) {
+    const between = parts[i] ?? '';
+    between.split(REFERENCE).forEach((part, j) => {
+      if (j % 2 === 1) units.push({ text: part });
+      else units.push(...graphemes(part));
+    });
+    const tag = parts[i + 1];
+    if (tag === undefined) continue;
+    const name = parts[i + 2]?.toLowerCase();
+    const joins = tag.startsWith('<!--') || (name !== undefined && INLINE_TAGS.has(name));
+    units.push({ text: tag, tag: joins ? 'joins' : 'ends' });
+  }
   return breakTokens(units, (unit) => unit);
 }
 

src/layout/section.ts

@@ -6,7 +6,6 @@ import type { Reference } from '../graph/queries.ts';
 import type { Found, FoundRow, Place, Section } from '../model/types.ts';
 import { figureHtml, figureKey, type FigureSvgs, type Placement } from './figure.ts';
 import { anchorOf } from '../model/build.ts';
-import { markupText } from './characters.ts';
 import { Links, attribute, markupBreaks, text } from './html.ts';
 
 /** What laying out sections reports, for the run report and the service's log. */
@@ -95,7 +94,7 @@ export function sectionHtml(section: Section, input: SectionInput): string {
   if (section.summations.length > 0) {
     // sanitised by the model (rules.ts) to the tags the website's pages allow
     // a long token in its text given places to break, as every other text is (html.ts)
-    const broken = section.summations.map((summation) => markupText(summation, markupBreaks));
+    const broken = section.summations.map(markupBreaks);
     parts.push(`<div class="summation">${broken.join('\n')}</div>`);
   }
 

test/layout/html.test.ts

@@ -0,0 +1,87 @@
+// A long token's places to break (spec 001 T030): `<wbr>` in a token longer than 40
+// characters, in equal pieces of at most 20, never inside a character reference or a tag.
+import assert from 'node:assert/strict';
+import { describe, test } from 'node:test';
+
+import { BREAK_EVERY, LONG_TOKEN, markupBreaks, text } from '../../src/layout/html.ts';
+
+/** Each piece's length in characters, a character reference counted as one. */
+const pieces = (out: string): number[] =>
+  out.split('<wbr>').map((piece) => piece.replace(/&[^;]*;/g, '_').length);
+
+describe("text()'s breaks", () => {
+  test('a token of 40 characters is kept whole; longer, it breaks in equal pieces', () => {
+    assert.equal(text('a'.repeat(LONG_TOKEN)), 'a'.repeat(LONG_TOKEN));
+    assert.deepEqual(pieces(text('a'.repeat(41))), [14, 14, 13]);
+    for (let n = LONG_TOKEN + 1; n <= 200; n++) {
+      const lengths = pieces(text('a'.repeat(n)));
+      assert.equal(lengths.length, Math.ceil(n / BREAK_EVERY), `${String(n)}: pieces`);
+      assert.ok(Math.max(...lengths) <= BREAK_EVERY, `${String(n)}: at most 20`);
+      assert.ok(Math.max(...lengths) - Math.min(...lengths) <= 1, `${String(n)}: equal`);
+    }
+  });
+
+  test('a token ends at a space Chromium breaks at, and runs through one it does not', () => {
+    const half = 'a'.repeat(30);
+    for (const space of [' ', '\t', '\n', '\r', '\r\n'])
+      assert.equal(text(`${half}${space}${half}`), `${half}${space}${half}`, JSON.stringify(space));
+    for (const joiner of ['\u00a0', '\u202f', '\ufeff']) {
+      const out = text(`${half}${joiner}${half}`);
+      assert.deepEqual(pieces(out), [16, 15, 15, 15], JSON.stringify(joiner));
+      assert.equal(out.replaceAll('<wbr>', ''), `${half}${joiner}${half}`);
+    }
+  });
+
+  test('a character escaped is one unit, never broken into', () => {
+    const out = text('&'.repeat(41));
+    assert.equal(out, ['&amp;'.repeat(14), '&amp;'.repeat(14), '&amp;'.repeat(13)].join('<wbr>'));
+    assert.doesNotMatch(text(`${'<'.repeat(30)}>${'"'.repeat(30)}`), /&[a-z]*<wbr>/);
+  });
+});
+
+describe("markupBreaks()'s breaks", () => {
+  test('a character reference is one unit, never broken into', () => {
+    for (const reference of ['&amp;', '&#x3b1;', '&#945;']) {
+      const out = markupBreaks(reference.repeat(41));
+      assert.equal(
+        out,
+        [reference.repeat(14), reference.repeat(14), reference.repeat(13)].join('<wbr>')
+      );
+    }
+  });
+
+  test("a tag's attribute is never broken, nor counted in the word beside it", () => {
+    const href = `https://site.example/${'p'.repeat(60)}`;
+    const link = `<a href="${href}">link</a>`;
+    assert.equal(markupBreaks(`<p>${link}</p>`), `<p>${link}</p>`);
+    assert.equal(
+      markupBreaks(`<p>${'w'.repeat(30)}${link}</p>`),
+      `<p>${'w'.repeat(30)}${link}</p>`
+    );
+  });
+
+  test('a word runs through inline tags, and ends at any other', () => {
+    const x = 'x'.repeat(25);
+    const y = 'y'.repeat(25);
+    assert.equal(
+      markupBreaks(`<p>${x}<i>${y}</i></p>`),
+      `<p>${'x'.repeat(17)}<wbr>${'x'.repeat(8)}<i>${'y'.repeat(9)}<wbr>${'y'.repeat(16)}</i></p>`
+    );
+    assert.match(markupBreaks(`${x}<sub>2</sub>${y}`), /<wbr>/);
+    const cells = `<tr><td>${x}</td><td>${y}</td></tr>`;
+    assert.equal(markupBreaks(cells), cells);
+    assert.equal(markupBreaks(`${x}<br>${y}`), `${x}<br>${y}`);
+  });
+
+  test('its text breaks as text() does, a `<` before no tag name being text', () => {
+    const plain = `${'a'.repeat(45)} ${'b'.repeat(10)}`;
+    assert.equal(markupBreaks(plain), text(plain));
+    const lessThan = `${'a'.repeat(30)}<=${'b'.repeat(30)}`;
+    assert.equal(markupBreaks(lessThan).replaceAll('<wbr>', ''), lessThan);
+    assert.equal(markupBreaks(lessThan).split('<wbr>').length, 4);
+    assert.equal(
+      markupBreaks(`<p>${'a'.repeat(30)}\u00a0${'a'.repeat(30)}</p>`).split('<wbr>').length,
+      4
+    );
+  });
+});