Spec 001 T035: compare

specs/001-pathway-pdfs/contracts/cli.md

@@ -228,10 +228,44 @@ is a PDF, or a folder of today's PDFs, as today's book is one file per top-level
 pathway, and the new side likewise; a folder is compared file by file, by top-level
 pathway, and a file on one side only is reported as a difference.
 
-- **Output:** a JSON report of the differences by part and kind, with counts. Names
-  appear only as counts and hashes (FR-013).
-- **`--expected <file>`:** the recorded differences. With it, the exit is 1 only for a
-  difference not recorded.
+- **Output:** a JSON report of the differences by part and kind, with counts.
+  - Each difference has its `file`, its `part` (`file`, `cover`, `introduction`,
+    `analysis`, `sections`, `section`, `figure`, `contents` or `short-entries`), a `field`
+    where the part has several (a section's `summation`, the cover's `names`), its `kind`
+    (`value`, `count`, `text`, `order`, `missing` from the new side, `extra` on it, `kind`,
+    `target`, `missing-target` or `unreadable`), a `count`, and the sections it is in by
+    stable identifier.
+  - **What may be printed:** no text read from a PDF is printed unless it has one of these
+    shapes: a count, yes or no, a release number, one of a fixed list (the analysis's
+    types and resources, the parts, fields and kinds), a stable identifier
+    (`R-[A-Z]{3}-\d+`), an analysis token of the form the analysis service issues, or one
+    of Reactome's species. Anything else is printed as a hash and a length, and the
+    cover's names as a count and a hash (FR-013). The hashes are salted once a run, so
+    they say which values are equal within a report and cannot be checked against a
+    guessed name. The cover's title is never printed: a side's title is said only to be
+    its file's name or not (`same` or `differs`). A section's text is never printed.
+  - A list in another order is a difference of its own kind, `order`. The sections' order
+    is read by their longest common subsequence, so only the sections out of it are
+    reported; a section written twice is compared occurrence by occurrence.
+  - A contents entry whose target is not read on either side is `missing-target`.
+  - A file that cannot be read is named on standard error and reported as `unreadable`;
+    the other files are still compared.
+  - `unmatched`: the records of `--expected` that matched no difference, by their place in
+    its list.
+- **`--expected <file>`:** the recorded differences, as `{ "differences": [...] }`, each
+  with a `part`, a `kind` and a `reason`, and optionally a `field`, a `file` and `events`;
+  any other key is refused (exit 2). A record naming events covers only those events, and
+  never a difference that names none. Each difference it covers is reported with its
+  reason. With it, the exit is 1 only for a difference not recorded.
+- **`--reference-margin <pt>`:** the depth of the bands at the top and bottom of the
+  reference's pages read as margins (running headers, page numbers), 50 by default, as
+  pdf-export's own; T040 sets it for `event-pdf`'s files from their real margins.
+- **Exit:** 0 with no difference (or none not recorded), 1 with one, 2 on bad arguments or
+  a reference and a new side not both PDFs or both folders.
+- **What it does not know yet:** the reducer reads parts by their labels, pdf-export's
+  and other words for them; `event-pdf`'s own labels, margins, contents form and names'
+  separator are not known here, and are for T040 to add from its real files. A label not
+  known leaves its part unread, which shows as a difference.
 
 ## `pdf-export serve`
 

specs/001-pathway-pdfs/tasks.md

@@ -591,7 +591,7 @@ passes.
 
 **Goal:** FR-009 in the image (US4, SC-006), and FR-010, FR-011, FR-012, SC-002, SC-003, SC-007 and SC-010 (US5).
 
-- [ ] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare <today's folder> <new.pdf>` (contracts/cli.md).
+- [x] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare <today's folder> <new.pdf>` (contracts/cli.md).
   - Both sides are reduced to FR-012's parts:
     - the covers' distinct names, counted, across today's files;
     - the introduction's release and counts;
@@ -605,6 +605,40 @@ passes.
 
   Tests on fixture PDFs: each part's difference is caught, and no name is printed.
 
+  - **Done (#167, replacing #165 after its review):** `pdf-export compare <reference> <new> [--expected <file>] [--reference-margin <pt>]`, wired into `src/cli.ts`, as contracts/cli.md has it rather than this task's `<today's folder> <new.pdf>`: the book is a folder of PDFs, so each side is a PDF or a folder, a folder compared file by file by name, and a file on one side only is a difference.
+    - **Reading** (`src/compare/read.ts`): pdf.js's lines of text (pieces sharing a baseline), links and figures. The bands at a page's top and bottom (50 pt unless set; running headers, page numbers) and the text inside a figure (a vector diagram's labels) are left out. A figure is a cluster of images and painted paths, neither thin nor the whole page, within 30 pt of each other, a thin path (a connector) joining only the clusters it touches; a cluster at least 60 pt each way is a figure. A raster diagram and a vector one whose shapes are joined or near are found; a label outside its figure's shapes is read as text.
+    - **Reducing** (`src/compare/reduce.ts`), with no knowledge of which program wrote the file:
+      - the cover: its title (its largest text) as a hash and a length, a diagram below it, an excerpt note, and its distinct names, counted and hashed (the names are a run of characters joined by U+2060 or no-break spaces where U+2060 is there to read, research R2; otherwise parted by semicolons, as pdf-export writes them);
+      - the introduction's release and counts;
+      - the analysis's properties by label, from its heading to the first stable identifier or the next heading, each kept only where it has its shape and else withheld;
+      - each section, found at its `Stable identifier` label, its name the larger text above it: its parts by their labels, each taken only after the parts before it in FR-005's order, so a summation beginning with a label's word stays a summation; the summation is the unlabelled text after the facts (after the figure, or at the section's left edge);
+      - each part's items: lists split at their separators, literature at its numbers, the edit history at its dates; the literature and edit history, which name people, held as one hash an item;
+      - the figure's kind, `diagram`, `selection` or `reaction` from its link to the website, `reaction` for a section with a type, or else `figure` (present, kind not read), which matches any kind;
+      - the contents list's entries (each up to the line ending in its page number) and the section or front part each lands on;
+      - short entries at later places (FR-004), counted apart.
+        Text is compared folded: NFKC, with white space and FR-017's characters never drawn (U+2060 among them) left out.
+    - **Printing** (`src/compare/output.ts`): every value a difference holds passes through one printer, `Output`. Text read from a PDF is printed only where it has one of a few shapes: a count, yes or no, a release number, one of a fixed list (the analysis's types and resources, the parts, fields and kinds), a stable identifier `R-[A-Z]{3}-\d+`, an analysis token of the service's form (`canonicalToken`), or one of Reactome's species. Anything else is a hash and a length, and the cover's names a count and a hash. The hashes printed are salted once a run (never printed): equal values stay equal within a report, and a printed hash cannot be checked against a guessed name, which an unsalted hash of a short name could be. The reduced form's own hashes are unsalted, so two runs' reduced forms compare (T036); they never reach the report unsalted. Standard error names paths and an error's kind and code, never its message.
+    - **Comparing** (`src/compare/compare.ts`): each difference by part, field and kind (`value`, `count`, `text`, `order`, `missing`, `extra`, `kind`, `target`, `missing-target`, `unreadable`), with a count and its sections' stable identifiers. A list with the same items in another order is `order`. The sections' order is read by their longest common subsequence (Hunt and Szymanski), so only the sections out of it are `order`; a section's occurrences beyond the other side's are `missing` or `extra`, and each occurrence is compared with its own. A contents entry whose target is not read on either side is `missing-target`. A file that cannot be read is named on standard error and is `unreadable`, and the rest are compared. `--expected` takes `{ "differences": [...] }`, each with a part, kind and reason and optionally a field, file and events; any other key is exit 2; a record naming events covers only those, never a difference that names none; the records that matched nothing are listed in `unmatched`. The exit is 1 only for a difference not recorded.
+    - **Tests:** 37 in `test/compare/` (Chromium; no graph): `compare.test.ts` (20), `layouts.test.ts` (6), `units.test.ts` (11, no browser).
+      - A sample document (`samples.ts`, placeholder names only) is printed by pdf-export's layout and, as the reference, as HTML in `event-pdf`'s shape as the Background and R2 describe it, with wording written anew; real `event-pdf` files are never used. The two differ only in the analysis's resource. Each part's difference is caught, as in #165, and a contents target not read (`missing-target`), a file that cannot be read, `--reference-margin`, an unknown key in `--expected` and a record matching nothing.
+      - A second reference laid out unlike the sample (facts in a table with colons, the contents first in a table, vector figures with a connector and no link, references numbered in brackets, page numbers bottom right) reads the same as pdf-export's: no difference.
+      - The reviewer's covers (a title in a coloured band; a logo, a 16 pt title and a diagram close together, so a figure takes in the title) and analysis parts (followed by a section labelled `Identifier:` under its heading or not, with summation lines starting "Species…" and "Type…", and by text with neither a heading nor a stable identifier), names planted in each: none printed, nothing past the part read as a property, nothing not of its shape kept as a value.
+      - A fuzz-like test: 300 seeded random layouts per side, names planted after every label the reducer reads, at every size, joined with U+2060 or not; every string of each report is a key, a part, field or kind, a file name, a stable identifier, a hash, or a value of an allowed shape, and none holds a part of a name.
+      - The sections' order: `[A,B,A,C,D,E]` against `[A,B,C,D,E]` is `A` missing once and nothing out of order; one section moved from first to last is that one alone; each occurrence of a repeated section is compared with its own.
+      - The constants pinned: a landing 76 pt above a heading is on it and one 160 pt above is not (`LANDING_PT`); a 50 pt square is not a figure and a 70 pt one is (`FIGURE_MIN_PT`); shapes 20 pt apart are one figure and 45 pt apart two (`FIGURE_GAP_PT`).
+      - **Measured:** 4,000 synthetic sections (each a heading, facts, a figure, a summation and four literature references linked to their records) with a contents list of 4,000 entries reduce in 1.25 s; #165's reduction took 30.9 s on the same layout (2.7 s at 1,000: quadratic), and #165's review measured 115 s on its own. The test fails over 8 s. Links, figures and targets are read by page, labels compiled once, the regions found by binary search, and a file's bytes read once and handed to pdf.js uncopied.
+    - **Mutants, each caught:** from #165's review: values printed without their shape check, and a withheld value printed as its text (the shapes and fuzz-like tests); the analysis part not ended at a heading or stable identifier (the reviewer's layouts); analysis values kept without their shape; an unread contents target passing; links not read by page (the figures' kinds, and #165's whole quadratic reduction fails the timing test at 30.9 s); the sections' order by position, not the common subsequence; only a repeated section's first occurrence compared; an unreadable file stopping the run; unknown `--expected` keys accepted; an `events` record covering a difference with none; unmatched records not reported; the printed hashes unsalted; thin connectors not joining a figure's shapes (the second layout); `--reference-margin` ignored; `LANDING_PT` at 60 or 200, `FIGURE_MIN_PT` at 40 or 80, `FIGURE_GAP_PT` at 15 or 50; the titles not compared. Again from #165: names not read as U+2060 runs; names not made distinct; the names' hash not compared; literature not hashed; the release not read; an order read as text; contents targets not compared; figures not compared; a figure's text kept; the margins kept; labels taken in any order; short entries not set apart; a record's field ignored; recorded differences failing the run; the summation not split from the facts; a diagram anywhere on the cover. 38 in all, none survived.
+    - **From #165's review:** two highs, three mediums and lows, each reproduced first.
+      - **H1, the cover title printed (could be the names):** where a figure took in the title, the "title" was body text, the names included. The title is held as a hash and a length and never printed; a side's title is said to be its file's name (`same`) or not (`differs`).
+      - **H2, analysis values from section text:** the region ran to the next section found, and every line starting with a property's label was read. It now ends at the first stable identifier or the next heading, and a value is kept only of its shape. Both highs led to the rule above: one printer, every value through it, and the fuzz-like test.
+      - **M1, contents targets failed open:** a target not read on one side was no difference; it is `missing-target`.
+      - **M2, quadratic reduction:** 4,000 sections took 115 s; now links, figures and targets by page (1.25 s against 30.9 s on the test's layout, above), and no second copy of a file.
+      - **M3, order misreported:** a repeated section, or one moved, flagged every section after it; now the longest common subsequence, with each occurrence compared.
+      - **Lows:** the contract, the files' headers and this note no longer say more than is done; the margin band is a setting (`--reference-margin`, 50 by default), which T040 sets for `event-pdf`'s files; an unreadable PDF is named and is a difference, and the rest are compared; `--expected` refuses unknown keys, does not let an `events` record cover a difference that names none, and lists records matching nothing; the printed hashes are salted once a run (above).
+      - **Found by the second layout:** a vector figure whose shapes were joined only by a thin connector was two small clusters, neither a figure, and its labels were read into the facts; a thin path now joins the clusters it touches.
+    - **Found:** pdf-export's analysis part does not print the resource (`TOTAL`), which FR-012 lists among the properties: `analysis resource missing` until it does, for T040 to record or fix.
+    - **Known gaps, for T040:** `event-pdf`'s own labels, its cover's names' separator (whether U+2060 survives pdf.js's reading of its files), its margins and its contents' form are not known here, so the reference's shape is the Background's and R2's, and its wording this task's. A label not known leaves its part unread, which shows as a difference, never a pass; T040 adds `event-pdf`'s labels to `RULES` and `PROPERTIES` in `src/compare/reduce.ts` and sets `--reference-margin`. A table with stroked or shaded cells may be read as a figure (its borders thick enough, or its shading a filled shape), its text then left out: T040 checks `event-pdf`'s tables. A figure's label outside its shapes is read as text. A fact's value wrapped to the section's left edge would be read as the summation's start. A cover whose title a figure takes in compares its largest remaining text as its title (it says `differs`, nothing more). A summation and an analysis's found tables are compared as text and rows, not by their markup.
+
 - [ ] T036 [US5] The fixture's output as a baseline (the constitution's quality gate for a change to what is written).
   - `ci/fixture/expected.json` holds the fixture book and a fixture single-event PDF, reduced by T035's `compare`; `compare` gains `--reduced <file>` to take a reduced form as the reference, added to contracts/cli.md in this task (the fixture PDFs are not committed).
   - CI compares each PR's output with it, and fails on any difference that the PR does not also update.

src/cli.ts

@@ -2,7 +2,8 @@
 // pdf-export's command line (contracts/cli.md). Its commands come with spec 001's tasks:
 // serve (T022) runs the single-event service; preflight (T024) runs the book's preflight
 // alone; figures (T046) draws the book's figures into a folder; book (T029) writes the
-// book; check-book (T030) checks a written book for FR-011's faults; compare is to come.
+// book; check-book (T030) checks a written book for FR-011's faults; compare (T035)
+// compares with event-pdf's output by FR-012.
 import { constants as osConstants } from 'node:os';
 
 import { privateTemporary } from './print/temporary.ts';
@@ -93,6 +94,12 @@ if (import.meta.main) {
       out: (text) => process.stdout.write(text),
       err: (text) => process.stderr.write(text),
     });
+  } else if (args[0] === 'compare') {
+    const { compareCommand } = await import('./compare/compare.ts');
+    process.exitCode = await compareCommand(args.slice(1), {
+      out: (text) => process.stdout.write(text),
+      err: (text) => process.stderr.write(text),
+    });
   } else if (args[0] === 'figures') {
     const { figuresCommand } = await import('./figures/step.ts');
     process.exitCode = await figuresCommand(args.slice(1), process.env, {