diff --git a/specs/001-pathway-pdfs/contracts/cli.md b/specs/001-pathway-pdfs/contracts/cli.md index 0fc4f55..2a24936 100644 --- a/specs/001-pathway-pdfs/contracts/cli.md +++ b/specs/001-pathway-pdfs/contracts/cli.md @@ -228,10 +228,48 @@ is a PDF, or a folder of today's PDFs, as today's book is one file per top-level pathway, and the new side likewise; a folder is compared file by file, by top-level pathway, and a file on one side only is reported as a difference. -- **Output:** a JSON report of the differences by part and kind, with counts. Names - appear only as counts and hashes (FR-013). -- **`--expected `:** the recorded differences. With it, the exit is 1 only for a - difference not recorded. +- **Output:** a JSON report of the differences by part and kind, with counts. + - Each difference has its `file`, its `part` (`file`, `cover`, `introduction`, + `analysis`, `sections`, `section`, `figure`, `contents` or `short-entries`), a `field` + where the part has several (a section's `summation`, the cover's `names`), its `kind` + (`value`, `count`, `text`, `order`, `missing` from the new side, `extra` on it, `kind`, + `target`, `missing-target` or `unreadable`), a `count`, and the sections it is in by + stable identifier. + - **What may be printed:** no text read from a PDF is printed unless it has one of these + shapes: a count, yes or no, a release number, one of a fixed list (the analysis's + types and resources, the parts, fields and kinds), a stable identifier + (`R-[A-Z]{3}-\d+`), an analysis token of the form the analysis service issues (printed + in its one canonical form, so copies encoded otherwise are one token), or one + of Reactome's species. Anything else is printed as a hash and a length, and the + cover's names as a count and a hash (FR-013). The hashes are salted once a run, so + they say which values are equal within a report and cannot be checked against a + guessed name. The cover's title is never printed: a side's title is said only to be + its file's name or not (`same` or `differs`). A section's text is never printed. + - A list in another order is a difference of its own kind, `order`. The sections' order + is read by their longest common subsequence, so only the sections out of it are + reported; a section written twice is compared occurrence by occurrence. + - A contents entry whose target is not read on either side is `missing-target`. + - A file that cannot be read is named on standard error and reported as `unreadable`; + the other files are still compared. + - `unmatched`: the records of `--expected` that matched no difference, by their place in + its list. +- **`--expected `:** the recorded differences, as `{ "differences": [...] }`, each + with a `part`, a `kind` and a `reason`, and optionally a `field`, a `file` and `events`; + any other key is refused (exit 2). A record naming events covers only those events, and + never a difference that names none. Each difference it covers is reported with its + reason. With it, the exit is 1 only for a difference not recorded. +- **`--reference-margin `:** the depth of the bands at the top and bottom of the + reference's pages read as margins (running headers, page numbers), 50 by default, as + pdf-export's own; T040 sets it for `event-pdf`'s files from their real margins. +- **Standard error:** paths, and an error's kind and code, never its message. pdf.js's + warnings, which quote a file's content, are off, and anything a library writes to the + console during a comparison is withheld: counted, with the kinds of the errors it passed. +- **Exit:** 0 with no difference (or none not recorded), 1 with one, 2 on bad arguments, a + path that does not exist, or a reference and a new side not both PDFs or both folders. +- **What it does not know yet:** the reducer reads parts by their labels, pdf-export's + and other words for them; `event-pdf`'s own labels, margins, contents form and names' + separator are not known here, and are for T040 to add from its real files. A label not + known leaves its part unread, which shows as a difference. ## `pdf-export serve` diff --git a/specs/001-pathway-pdfs/tasks.md b/specs/001-pathway-pdfs/tasks.md index ad710a5..34c46c3 100644 --- a/specs/001-pathway-pdfs/tasks.md +++ b/specs/001-pathway-pdfs/tasks.md @@ -591,7 +591,7 @@ passes. **Goal:** FR-009 in the image (US4, SC-006), and FR-010, FR-011, FR-012, SC-002, SC-003, SC-007 and SC-010 (US5). -- [ ] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare ` (contracts/cli.md). +- [x] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare ` (contracts/cli.md). - Both sides are reduced to FR-012's parts: - the covers' distinct names, counted, across today's files; - the introduction's release and counts; @@ -605,6 +605,45 @@ passes. Tests on fixture PDFs: each part's difference is caught, and no name is printed. + - **Done (#168, replacing #165 and #167 after their reviews):** `pdf-export compare [--expected ] [--reference-margin ]`, wired into `src/cli.ts`, as contracts/cli.md has it rather than this task's ` `: the book is a folder of PDFs, so each side is a PDF or a folder, a folder compared file by file by name, and a file on one side only is a difference. + - **Reading** (`src/compare/read.ts`): pdf.js's lines of text (pieces sharing a baseline), links and figures. The bands at a page's top and bottom (50 pt unless set; running headers, page numbers) and the text inside a figure (a vector diagram's labels) are left out. A figure is a cluster of images and painted paths, neither thin nor the whole page, within 30 pt of each other; thin paths that touch are one line, which joins the clusters it touches only where it touches two or more, each at least 4 pt each way (a connector between nodes, never a table's borders or a rule); a cluster at least 60 pt each way is a figure. pdf.js reads with its warnings off. A raster diagram and a vector one whose shapes are joined or near are found; a label outside its figure's shapes is read as text. + - **Reducing** (`src/compare/reduce.ts`), with no knowledge of which program wrote the file: + - the cover: its title (its largest text) as a hash and a length, a diagram below it, an excerpt note, and its distinct names, counted and hashed (the names are a run of characters joined by U+2060 or no-break spaces where U+2060 is there to read, research R2; otherwise parted by semicolons, as pdf-export writes them); + - the introduction's release and counts; + - the analysis's properties by label, from its heading to the first stable identifier or the next heading, each kept only where it has its shape and else withheld; + - each section, found at its `Stable identifier` label, its name the larger text above it: its parts by their labels, each taken only after the parts before it in FR-005's order, so a summation beginning with a label's word stays a summation; the summation is the unlabelled text after the facts (after the figure, or at the section's left edge); + - each part's items: lists split at their separators, literature at its numbers, the edit history at its dates; the literature and edit history, which name people, held as one hash an item; + - the figure's kind, `diagram`, `selection` or `reaction` from its link to the website, `reaction` for a section with a type, or else `figure` (present, kind not read), which matches any kind; + - the contents list's entries (each up to the line ending in its page number) and the section or front part each lands on; + - short entries at later places (FR-004), counted apart. + Text is compared folded: NFKC, with white space and FR-017's characters never drawn (U+2060 among them) left out. + - **Printing** (`src/compare/output.ts`): every value a difference holds passes through one printer, `Output`. Text read from a PDF is printed only where it has one of a few shapes: a count, yes or no, a release number, one of a fixed list (the analysis's types and resources, the parts, fields and kinds), a stable identifier `R-[A-Z]{3}-\d+`, an analysis token of the service's form (`canonicalToken`), or one of Reactome's species. Anything else is a hash and a length, and the cover's names a count and a hash. The hashes printed are salted once a run (never printed): equal values stay equal within a report, and a printed hash cannot be checked against a guessed name, which an unsalted hash of a short name could be. The reduced form's own hashes are unsalted, so two runs' reduced forms compare (T036); they never reach the report unsalted. Standard error names paths and an error's kind and code, never its message. + - **Comparing** (`src/compare/compare.ts`): each difference by part, field and kind (`value`, `count`, `text`, `order`, `missing`, `extra`, `kind`, `target`, `missing-target`, `unreadable`), with a count and its sections' stable identifiers. A list with the same items in another order is `order`. The sections' order is read by their longest common subsequence (Hunt and Szymanski), so only the sections out of it are `order` (in about (n + matches) log n; a list of one identifier repeated throughout makes the matches, and so the time, quadratic, which a book never has: an event is written in full once a PDF); a section's occurrences beyond the other side's are `missing` or `extra`, and each occurrence is compared with its own. A contents entry whose target is not read on either side is `missing-target`. A file that cannot be read is named on standard error and is `unreadable`, and the rest are compared. `--expected` takes `{ "differences": [...] }`, each with a part, kind and reason and optionally a field, file and events; any other key is exit 2; a record naming events covers only those, never a difference that names none; the records that matched nothing are listed in `unmatched`. The exit is 1 only for a difference not recorded. + - **Tests:** 49 in `test/compare/` (Chromium; no graph): `compare.test.ts` (20), `layouts.test.ts` (8), `units.test.ts` (21, no browser). + - A sample document (`samples.ts`, placeholder names only) is printed by pdf-export's layout and, as the reference, as HTML in `event-pdf`'s shape as the Background and R2 describe it, with wording written anew; real `event-pdf` files are never used. The two differ only in the analysis's resource. Each part's difference is caught, as in #165, and a contents target not read (`missing-target`), a file that cannot be read, `--reference-margin`, an unknown key in `--expected` and a record matching nothing. + - A second reference laid out unlike the sample (facts in a table with colons, the contents first in a table, vector figures with a connector and no link, references numbered in brackets, page numbers bottom right) reads the same as pdf-export's: no difference. + - The reviewer's covers (a title in a coloured band; a logo, a 16 pt title and a diagram close together, so a figure takes in the title) and analysis parts (followed by a section labelled `Identifier:` under its heading or not, with summation lines starting "Species…" and "Type…", and by text with neither a heading nor a stable identifier), names planted in each: none printed, nothing past the part read as a property, nothing not of its shape kept as a value. + - A fuzz-like test: 300 seeded random layouts per side, names planted after every label the reducer reads, at every size, joined with U+2060 or not; every string of each report is a key, a part, field or kind, a file name, a stable identifier, a hash, or a value of an allowed shape, and none holds a part of a name. + - The sections' order: `[A,B,A,C,D,E]` against `[A,B,C,D,E]` is `A` missing once and nothing out of order; one section moved from first to last is that one alone; each occurrence of a repeated section is compared with its own. + - The constants pinned: a landing 76 pt above a heading is on it and one 160 pt above is not (`LANDING_PT`); a 50 pt square is not a figure and a 70 pt one is (`FIGURE_MIN_PT`); shapes 20 pt apart are one figure and 45 pt apart two (`FIGURE_GAP_PT`). + - **Measured:** 4,000 synthetic sections (each a heading, facts, a figure, a summation and four literature references linked to their records) with a contents list of 4,000 entries reduce in 1.25 s; #165's reduction took 30.9 s on the same layout (2.7 s at 1,000: quadratic), and #165's review measured 115 s on its own. The test fails over 8 s. Links, figures and targets are read by page, labels compiled once, the regions found by binary search, and a file's bytes read once and handed to pdf.js uncopied. + - **Mutants, each caught:** from #165's review: values printed without their shape check, and a withheld value printed as its text (the shapes and fuzz-like tests); the analysis part not ended at a heading or stable identifier (the reviewer's layouts); analysis values kept without their shape; an unread contents target passing; links not read by page (the figures' kinds, and #165's whole quadratic reduction fails the timing test at 30.9 s); the sections' order by position, not the common subsequence; only a repeated section's first occurrence compared; an unreadable file stopping the run; unknown `--expected` keys accepted; an `events` record covering a difference with none; unmatched records not reported; the printed hashes unsalted; thin connectors not joining a figure's shapes (the second layout); `--reference-margin` ignored; `LANDING_PT` at 60 or 200, `FIGURE_MIN_PT` at 40 or 80, `FIGURE_GAP_PT` at 15 or 50; the titles not compared. Again from #165: names not read as U+2060 runs; names not made distinct; the names' hash not compared; literature not hashed; the release not read; an order read as text; contents targets not compared; figures not compared; a figure's text kept; the margins kept; labels taken in any order; short entries not set apart; a record's field ignored; recorded differences failing the run; the summation not split from the facts; a diagram anywhere on the cover. 38 in all, none survived. + - **From #165's review:** two highs, three mediums and lows, each reproduced first. + - **H1, the cover title printed (could be the names):** where a figure took in the title, the "title" was body text, the names included. The title is held as a hash and a length and never printed; a side's title is said to be its file's name (`same`) or not (`differs`). + - **H2, analysis values from section text:** the region ran to the next section found, and every line starting with a property's label was read. It now ends at the first stable identifier or the next heading, and a value is kept only of its shape. Both highs led to the rule above: one printer, every value through it, and the fuzz-like test. + - **M1, contents targets failed open:** a target not read on one side was no difference; it is `missing-target`. + - **M2, quadratic reduction:** 4,000 sections took 115 s; now links, figures and targets by page (1.25 s against 30.9 s on the test's layout, above), and no second copy of a file. + - **M3, order misreported:** a repeated section, or one moved, flagged every section after it; now the longest common subsequence, with each occurrence compared. + - **Lows:** the contract, the files' headers and this note no longer say more than is done; the margin band is a setting (`--reference-margin`, 50 by default), which T040 sets for `event-pdf`'s files; an unreadable PDF is named and is a difference, and the rest are compared; `--expected` refuses unknown keys, does not let an `events` record cover a difference that names none, and lists records matching nothing; the printed hashes are salted once a run (above). + - **Found by the second layout:** a vector figure whose shapes were joined only by a thin connector was two small clusters, neither a figure, and its labels were read into the facts; a thin path now joins the clusters it touches. + - **From #167's review:** two mediums and lows, each reproduced first; standard output was sound. + - **M1, pdf.js printed PDF text on standard error:** its default verbosity wrote warnings (`Unknown command ""`) through `console.warn`, quoting the file's content. Every `getDocument` that reads a PDF which may hold names now has `verbosity: VerbosityLevel.ERRORS`: `src/compare/read.ts`, and also `src/check/book.ts` (check-book reads the real book) and `src/print/contents.ts` (it reads the printed PDFs for the contents' pages). `compare` also holds the console for the length of a comparison (`withConsoleHeld`): a library's messages are counted and said once, with the kinds of the errors passed, never their words. Tests: a PDF whose content holds `PlaceholderPersonJaneQExample` as an operator is read with no warning, and compared with nothing of it on standard error or the console; the held console counts a warning and an error and gives the error's kind alone. The verbosity in check-book and contents is a setting with no test of its own. + - **M2, a table with a shaded header read as a figure:** the hairline borders joined the header's fill into one cluster at least 60 pt each way, and an eight-row table lost its rows. A line of thin paths now joins only where it touches two or more clusters, each at least 4 pt each way. Tests: the reviewer's table (a `#dddddd` header, 0.5 pt borders, eight rows) printed and read with its eight rows and no figure; at the shapes' level, the table is no figure, a connector still joins two shapes, a rule between two figures joins nothing, and a rule to a mark too small to be a node does not widen its figure. + - **Lows:** assertions that kill the mutants that survived #167 (a rule joined within 30 pt; the part not ended at a heading, with a heading-only layout; `errorKind` giving the message, by standard error's exact form; the title always `same`, by its meaning side by side; `x of y of z` kept); the contract says a missing path is exit 2; the token is printed, and compared, in its canonical form; the common subsequence's quadratic case is recorded (above). + - **Mutants, each caught:** pdf.js's warnings on in `readLayout`; the console not held; a thin line joining whatever it touches (#167's rule: the reviewer's table, printed and as shapes); any cluster taken as a node; touching within 30 pt; `errorKind` giving the message; the title always `same`; `x of y of z` kept as a count; the analysis part not ended at a heading; the token printed as given. 10, none survived (#167's 38 were not run again: the host was loaded). + - **Found:** pdf-export's analysis part does not print the resource (`TOTAL`), which FR-012 lists among the properties: `analysis resource missing` until it does, for T040 to record or fix. + - **Known gaps, for T040:** `event-pdf`'s own labels, its cover's names' separator (whether U+2060 survives pdf.js's reading of its files), its margins and its contents' form are not known here, so the reference's shape is the Background's and R2's, and its wording this task's. A label not known leaves its part unread, which shows as a difference, never a pass; T040 adds `event-pdf`'s labels to `RULES` and `PROPERTIES` in `src/compare/reduce.ts` and sets `--reference-margin`. A table with stroked or shaded cells may be read as a figure (its borders thick enough, or its shading a filled shape), its text then left out: T040 checks `event-pdf`'s tables. A figure's label outside its shapes is read as text. A fact's value wrapped to the section's left edge would be read as the summation's start. A cover whose title a figure takes in compares its largest remaining text as its title (it says `differs`, nothing more). A summation and an analysis's found tables are compared as text and rows, not by their markup. + - [ ] T036 [US5] The fixture's output as a baseline (the constitution's quality gate for a change to what is written). - `ci/fixture/expected.json` holds the fixture book and a fixture single-event PDF, reduced by T035's `compare`; `compare` gains `--reduced ` to take a reduced form as the reference, added to contracts/cli.md in this task (the fixture PDFs are not committed). - CI compares each PR's output with it, and fails on any difference that the PR does not also update. diff --git a/src/check/book.ts b/src/check/book.ts index 6b08f79..6c316c4 100644 --- a/src/check/book.ts +++ b/src/check/book.ts @@ -24,7 +24,7 @@ import { readdirSync, readFileSync, statSync } from 'node:fs'; import { join } from 'node:path'; -import { getDocument } from 'pdfjs-dist/legacy/build/pdf.mjs'; +import { getDocument, VerbosityLevel } from 'pdfjs-dist/legacy/build/pdf.mjs'; import { printedDocument } from '../book/book.ts'; import { @@ -177,7 +177,11 @@ const isRecord = (value: unknown): value is Record => /** A PDF's text, links and outline, read with pdf.js. */ async function readPdf(path: string): Promise { - const task = getDocument({ data: new Uint8Array(readFileSync(path)) }); + // pdf.js's warnings off: they quote the file's content, and the book holds names + const task = getDocument({ + data: new Uint8Array(readFileSync(path)), + verbosity: VerbosityLevel.ERRORS, + }); try { const doc = await task.promise; const pieces: Piece[] = []; diff --git a/src/cli.ts b/src/cli.ts index 6df0dae..af54db9 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -2,7 +2,8 @@ // pdf-export's command line (contracts/cli.md). Its commands come with spec 001's tasks: // serve (T022) runs the single-event service; preflight (T024) runs the book's preflight // alone; figures (T046) draws the book's figures into a folder; book (T029) writes the -// book; check-book (T030) checks a written book for FR-011's faults; compare is to come. +// book; check-book (T030) checks a written book for FR-011's faults; compare (T035) +// compares with event-pdf's output by FR-012. import { constants as osConstants } from 'node:os'; import { privateTemporary } from './print/temporary.ts'; @@ -93,6 +94,12 @@ if (import.meta.main) { out: (text) => process.stdout.write(text), err: (text) => process.stderr.write(text), }); + } else if (args[0] === 'compare') { + const { compareCommand } = await import('./compare/compare.ts'); + process.exitCode = await compareCommand(args.slice(1), { + out: (text) => process.stdout.write(text), + err: (text) => process.stderr.write(text), + }); } else if (args[0] === 'figures') { const { figuresCommand } = await import('./figures/step.ts'); process.exitCode = await figuresCommand(args.slice(1), process.env, { diff --git a/src/compare/compare.ts b/src/compare/compare.ts new file mode 100644 index 0000000..a6a0c80 --- /dev/null +++ b/src/compare/compare.ts @@ -0,0 +1,581 @@ +// FR-012's comparison (spec 001 T035, contracts/cli.md `compare`, research R8): a reference +// (`event-pdf`'s output) and the new output, each a PDF or a folder of them, reduced to +// FR-012's parts (reduce.ts) and compared part by part. A folder is compared file by file, +// by name, as today's book is one file per top-level pathway, named after it; a file that +// cannot be read is a difference, and the rest are still compared. +// +// Each difference is reported by its part, its field and its kind, with a count and the +// stable identifiers of the sections it is in. Its values pass through one printer +// (output.ts), which prints text read from a PDF only where it has a strict shape that +// cannot hold a name, and anything else as a salted hash and a length (FR-013). A list in +// another order is a difference of its own kind, `order`, which `--expected` records like +// any other; the sections' order is read by their longest common subsequence, so a section +// moved is the one reported, and a section written twice is compared occurrence by +// occurrence. +import { readdirSync, readFileSync, statSync } from 'node:fs'; +import { basename, join } from 'node:path'; + +import { SettingsError } from '../preflight/command.ts'; +import { fold, hash, Output, type Difference } from './output.ts'; +import { MARGIN_BAND_PT, readLayout } from './read.ts'; +import { reduce, type ContentsItem, type Reduced, type ReducedSection } from './reduce.ts'; + +export type { Difference } from './output.ts'; + +/** A recorded difference: matches by part and kind, and by field, file and events if given. */ +export interface Recorded { + part: string; + kind: string; + field?: string; + file?: string; + events?: string[]; + reason: string; +} + +/** How a list differs: in its items, or only in their order. */ +function listKind(a: readonly string[], b: readonly string[]): 'text' | 'order' | null { + if (a.length === b.length && a.every((item, i) => item === b[i])) return null; + const sorted = (list: readonly string[]) => [...list].sort().join('\n'); + return sorted(a) === sorted(b) ? 'order' : 'text'; +} + +/** + * The longest common subsequence of two lists, as pairs of indexes (Hunt and Szymanski: for + * each item of `b`, the places it has in `a`, latest first, through a longest increasing + * sequence), in about (n + matches) log n. + */ +export function commonSubsequence(a: readonly string[], b: readonly string[]): [number, number][] { + const places = new Map(); + a.forEach((item, i) => { + const list = places.get(item); + if (list) list.push(i); + else places.set(item, [i]); + }); + const nodes: { i: number; j: number; prev: number }[] = []; + /** The node ending the best sequence of each length, by length - 1. */ + const tails: number[] = []; + b.forEach((item, j) => { + const at = places.get(item) ?? []; + for (let k = at.length - 1; k >= 0; k--) { + const i = at[k] ?? 0; + let lo = 0; + let hi = tails.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if ((nodes[tails[mid] ?? 0]?.i ?? 0) < i) lo = mid + 1; + else hi = mid; + } + nodes.push({ i, j, prev: lo > 0 ? (tails[lo - 1] ?? -1) : -1 }); + tails[lo] = nodes.length - 1; + } + }); + const pairs: [number, number][] = []; + for (let n = tails.at(-1) ?? -1; n >= 0;) { + const node = nodes[n]; + if (!node) break; + pairs.push([node.i, node.j]); + n = node.prev; + } + return pairs.reverse(); +} + +/** How many times each item is in a list. */ +function tally(list: readonly string[]): Map { + const counts = new Map(); + for (const item of list) counts.set(item, (counts.get(item) ?? 0) + 1); + return counts; +} + +/** + * The sections' differences in which and in what order: each stable identifier's occurrences + * beyond the other side's are missing or extra; of the rest, those outside the longest + * common subsequence are out of order. A section written twice is listed as often. + */ +function sectionOrder(ref: readonly string[], neu: readonly string[]) { + const [inRef, inNew] = [tally(ref), tally(neu)]; + const kept = tally(commonSubsequence(ref, neu).map(([i]) => ref[i] ?? '')); + const missing: string[] = []; + const extra: string[] = []; + const moved: string[] = []; + const repeat = (list: string[], id: string, times: number) => { + for (let k = 0; k < times; k++) list.push(id); + }; + for (const id of new Set(ref)) repeat(missing, id, (inRef.get(id) ?? 0) - (inNew.get(id) ?? 0)); + for (const id of new Set(neu)) { + const [r, n] = [inRef.get(id) ?? 0, inNew.get(id) ?? 0]; + repeat(extra, id, n - r); + repeat(moved, id, Math.min(r, n) - (kept.get(id) ?? 0)); + } + return { missing, extra, moved }; +} + +/** The pairs of sections compared: each identifier's k-th occurrence on each side. */ +function pairSections(ref: readonly ReducedSection[], neu: readonly ReducedSection[]) { + const occurrences = (list: readonly ReducedSection[]) => { + const by = new Map(); + for (const s of list) by.set(s.stId, [...(by.get(s.stId) ?? []), s]); + return by; + }; + const newBy = occurrences(neu); + const pairs: [ReducedSection, ReducedSection][] = []; + for (const [stId, list] of occurrences(ref)) { + const others = newBy.get(stId) ?? []; + list.forEach((s, k) => { + const other = others[k]; + if (other) pairs.push([s, other]); + }); + } + return pairs; +} + +/** A file's name without `.pdf`, as a title is compared with it. */ +const stem = (file: string) => fold(basename(file).replace(/\.pdf$/i, '')); + +/** + * Two reduced PDFs' differences, grouped by part, field and kind. `files` names each side's + * file, whose name's stem each cover's title is compared with. + */ +export function compareReduced( + file: string, + ref: Reduced, + neu: Reduced, + files: { reference: string; new: string } = { reference: file, new: file } +): Difference[] { + const out: Difference[] = []; + const add = (d: Omit & { count?: number }) => { + out.push({ file, count: d.events?.length ?? 1, ...d }); + }; + + // the cover: its title compared by its hash, and said only to be its file's name or not + if (ref.cover.title.hash !== neu.cover.title.hash) { + const named = (title: Reduced['cover']['title'], name: string) => + title.hash === hash(stem(name)) ? 'same' : 'differs'; + add({ + part: 'cover', + field: 'title', + kind: 'value', + reference: named(ref.cover.title, files.reference), + new: named(neu.cover.title, files.new), + }); + } + for (const field of ['diagram', 'excerpt'] as const) { + if (ref.cover[field] !== neu.cover[field]) { + add({ + part: 'cover', + field, + kind: 'value', + reference: ref.cover[field], + new: neu.cover[field], + }); + } + } + const names = { reference: ref.cover.names, new: neu.cover.names }; + if (ref.cover.names.count !== neu.cover.names.count) { + add({ part: 'cover', field: 'names', kind: 'count', ...names }); + } else if (ref.cover.names.hash !== neu.cover.names.hash) { + add({ part: 'cover', field: 'names', kind: 'value', ...names }); + } + + // the introduction + for (const field of ['release', 'pathways', 'reactions'] as const) { + const [a, b] = [ref.introduction[field], neu.introduction[field]]; + if (a !== b) add({ part: 'introduction', field, kind: 'value', reference: a, new: b }); + } + + // the analysis's properties + if ((ref.analysis === null) !== (neu.analysis === null)) { + add({ part: 'analysis', kind: ref.analysis ? 'missing' : 'extra' }); + } else if (ref.analysis && neu.analysis) { + const keys = [...new Set([...Object.keys(ref.analysis), ...Object.keys(neu.analysis)])].sort(); + for (const field of keys) { + const [a, b] = [ref.analysis[field], neu.analysis[field]]; + if (JSON.stringify(a) === JSON.stringify(b)) continue; + const kind = a === undefined ? 'extra' : b === undefined ? 'missing' : 'value'; + add({ part: 'analysis', field, kind, reference: a ?? null, new: b ?? null }); + } + } + + // the sections: which, and in what order + const order = sectionOrder( + ref.sections.map((s) => s.stId), + neu.sections.map((s) => s.stId) + ); + if (order.missing.length > 0) add({ part: 'sections', kind: 'missing', events: order.missing }); + if (order.extra.length > 0) add({ part: 'sections', kind: 'extra', events: order.extra }); + if (order.moved.length > 0) add({ part: 'sections', kind: 'order', events: order.moved }); + + // each section's parts and figure, grouped by field and kind across sections + const grouped = new Map(); + const note = ( + part: 'section' | 'figure', + field: string | undefined, + kind: Difference['kind'], + stId: string + ) => { + const key = `${part}\0${field ?? ''}\0${kind}`; + let d = grouped.get(key); + if (!d) { + d = { file, part, ...(field === undefined ? {} : { field }), kind, count: 0, events: [] }; + grouped.set(key, d); + } + d.events?.push(stId); + d.count++; + }; + for (const [a, b] of pairSections(ref.sections, neu.sections)) { + const byPart = (s: ReducedSection) => new Map(s.parts.map((p) => [p.part, p.items])); + const [pa, pb] = [byPart(a), byPart(b)]; + for (const part of new Set([...pa.keys(), ...pb.keys()])) { + const [ia, ib] = [pa.get(part), pb.get(part)]; + if (!ia) note('section', part, 'extra', a.stId); + else if (!ib) note('section', part, 'missing', a.stId); + else { + const kind = listKind(ia, ib); + if (kind) note('section', part, kind, a.stId); + } + } + // a figure of a kind not read is compared only as being there + if (a.figure !== b.figure) { + if (a.figure === 'none') note('figure', undefined, 'extra', a.stId); + else if (b.figure === 'none') note('figure', undefined, 'missing', a.stId); + else if (a.figure !== 'figure' && b.figure !== 'figure') { + note('figure', undefined, 'kind', a.stId); + } + } + } + out.push(...grouped.values()); + + if ((ref.contents === null) !== (neu.contents === null)) { + add({ part: 'contents', kind: ref.contents ? 'missing' : 'extra' }); + } else if (ref.contents && neu.contents) { + compareContents(ref.contents, neu.contents, add); + } + + if (ref.shortEntries !== neu.shortEntries) { + add({ + part: 'short-entries', + kind: 'count', + reference: ref.shortEntries, + new: neu.shortEntries, + }); + } + return out; +} + +/** The contents lists: their entries, their order, and each entry's target. */ +function compareContents( + ref: readonly ContentsItem[], + neu: readonly ContentsItem[], + add: (d: Omit & { count?: number }) => void +): void { + const remaining = neu.map((e) => e.title); + let gone = 0; + for (const entry of ref) { + const at = remaining.indexOf(entry.title); + if (at < 0) gone++; + else remaining.splice(at, 1); + } + if (gone > 0) add({ part: 'contents', kind: 'missing', count: gone }); + if (remaining.length > 0) add({ part: 'contents', kind: 'extra', count: remaining.length }); + if (gone === 0 && remaining.length === 0) { + const [ta, tb] = [ref.map((e) => e.title), neu.map((e) => e.title)]; + if (listKind(ta, tb) === 'order') add({ part: 'contents', kind: 'order' }); + } + // each entry's target, matched by title occurrence by occurrence: a target not read on + // either side is a difference of its own, never a pass + const targets = new Map(); + for (const entry of neu) + targets.set(entry.title, [...(targets.get(entry.title) ?? []), entry.target]); + let wrong = 0; + let unread = 0; + for (const entry of ref) { + const target = targets.get(entry.title)?.shift(); + if (target === undefined) continue; + if (entry.target === null || target === null) unread++; + else if (target !== entry.target) wrong++; + } + if (wrong > 0) add({ part: 'contents', kind: 'target', count: wrong }); + if (unread > 0) add({ part: 'contents', kind: 'missing-target', count: unread }); +} + +/** + * The differences with those `--expected` records marked, with their reasons, and the + * records that matched nothing, by their place in the file. A record naming events covers + * only those, and never a difference that names none. + */ +export function applyRecorded( + differences: readonly Difference[], + recorded: readonly Recorded[] +): { differences: Difference[]; unmatched: number[] } { + const used = new Set(); + const out: Difference[] = []; + for (const difference of differences) { + let rest: Difference | null = { ...difference }; + for (const [index, record] of recorded.entries()) { + if (!rest) break; + if (record.part !== rest.part || record.kind !== rest.kind) continue; + if (record.field !== undefined && record.field !== rest.field) continue; + if (record.file !== undefined && record.file !== rest.file) continue; + if (record.events === undefined) { + out.push({ ...rest, recorded: record.reason }); + used.add(index); + rest = null; + continue; + } + if (rest.events === undefined) continue; + const events: string[] = rest.events; + const covered = events.filter((e) => record.events?.includes(e)); + if (covered.length === 0) continue; + used.add(index); + out.push({ ...rest, events: covered, count: covered.length, recorded: record.reason }); + const left = events.filter((e) => !covered.includes(e)); + rest = left.length > 0 ? { ...rest, events: left, count: left.length } : null; + } + if (rest) out.push(rest); + } + return { differences: out, unmatched: recorded.flatMap((_, i) => (used.has(i) ? [] : [i])) }; +} + +const RECORD_KEYS = new Set(['part', 'kind', 'field', 'file', 'events', 'reason']); + +/** The recorded differences, from `--expected`'s file: `{ "differences": [...] }`. */ +export function readRecorded(path: string): Recorded[] { + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(path, 'utf8')); + } catch (error) { + throw new SettingsError( + `${path} cannot be read as JSON (${error instanceof Error ? error.name : 'error'})` + ); + } + const list = + parsed !== null && typeof parsed === 'object' && 'differences' in parsed + ? parsed.differences + : undefined; + if (!Array.isArray(list)) throw new SettingsError(`${path} holds no "differences" list`); + return list.map((item: unknown, i) => { + const r = item as Partial | null; + const text = (v: unknown) => typeof v === 'string' && v !== ''; + if (r === null || typeof r !== 'object' || Array.isArray(r)) { + throw new SettingsError(`${path}: difference ${String(i)} is not an object`); + } + const unknown = Object.keys(r).filter((key) => !RECORD_KEYS.has(key)); + if (unknown.length > 0) { + throw new SettingsError( + `${path}: difference ${String(i)} has keys not known: ${unknown.join(', ')}` + ); + } + const ok = + text(r.part) && + text(r.kind) && + text(r.reason) && + (r.field === undefined || text(r.field)) && + (r.file === undefined || text(r.file)) && + (r.events === undefined || (Array.isArray(r.events) && r.events.every(text))); + if (!ok) { + throw new SettingsError( + `${path}: difference ${String(i)} needs a part, a kind and a reason, and its field, file and events as text` + ); + } + return r as Recorded; + }); +} + +/** One PDF, reduced; its bytes read once and handed to pdf.js. */ +export async function reducePdf(path: string, margin = MARGIN_BAND_PT): Promise { + const bytes = readFileSync(path); + // a plain view over the Buffer's memory, not a copy: pdf.js refuses a Buffer + return reduce( + await readLayout(new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength), { margin }) + ); +} + +export interface CompareOptions { + /** The reference's margin band, in points (read.ts). */ + referenceMargin?: number; + /** Told of each file that could not be read, by path, never its content. */ + unreadable?: (path: string, error: unknown) => void; +} + +/** A reference and a new side, each a PDF or a folder of them, compared (contracts/cli.md). */ +export async function comparePaths( + reference: string, + neu: string, + { referenceMargin = MARGIN_BAND_PT, unreadable = () => undefined }: CompareOptions = {} +): Promise { + const folder = (path: string) => statSync(path).isDirectory(); + const [refFolder, newFolder] = [folder(reference), folder(neu)]; + if (refFolder !== newFolder) { + throw new SettingsError('The reference and the new side must both be PDFs or both be folders'); + } + /** Both sides reduced and compared; a side that cannot be read, a difference. */ + const pair = async (file: string, refPath: string, newPath: string): Promise => { + const read = async (path: string, margin: number) => { + try { + return await reducePdf(path, margin); + } catch (error) { + unreadable(path, error); + return null; + } + }; + const a = await read(refPath, referenceMargin); + const b = await read(newPath, MARGIN_BAND_PT); + const failed: Difference[] = []; + if (!a) failed.push({ file, part: 'file', field: 'reference', kind: 'unreadable', count: 1 }); + if (!b) failed.push({ file, part: 'file', field: 'new', kind: 'unreadable', count: 1 }); + if (!a || !b) return failed; + return compareReduced(file, a, b, { reference: refPath, new: newPath }); + }; + if (!refFolder) return pair(basename(neu), reference, neu); + const pdfs = (path: string) => + readdirSync(path) + .filter((name) => name.toLowerCase().endsWith('.pdf')) + .sort(); + const [refFiles, newFiles] = [pdfs(reference), pdfs(neu)]; + const out: Difference[] = []; + for (const file of refFiles) { + if (!newFiles.includes(file)) out.push({ file, part: 'file', kind: 'missing', count: 1 }); + } + for (const file of newFiles) { + if (!refFiles.includes(file)) out.push({ file, part: 'file', kind: 'extra', count: 1 }); + } + for (const file of refFiles.filter((f) => newFiles.includes(f))) { + out.push(...(await pair(file, join(reference, file), join(neu, file)))); + } + return out; +} + +export interface CompareSettings { + reference: string; + neu: string; + expected: string | null; + referenceMargin: number; +} + +export function compareSettings(args: readonly string[]): CompareSettings { + const paths: string[] = []; + const flags = new Map(); + for (let i = 0; i < args.length; i++) { + const arg = args[i] ?? ''; + if (!arg.startsWith('--')) { + paths.push(arg); + continue; + } + const [name = '', inline] = arg.split(/=(.*)/s, 2); + if (name !== '--expected' && name !== '--reference-margin') { + throw new SettingsError(`Unknown argument: ${name}`); + } + const value = inline ?? args[++i]; + if (value === undefined || value === '') throw new SettingsError(`${name} needs a value`); + if (flags.has(name)) throw new SettingsError(`${name} is given twice`); + flags.set(name, value); + } + const [reference, neu] = paths; + if (paths.length !== 2 || !reference || !neu) { + throw new SettingsError('Two paths are needed: pdf-export compare '); + } + const margin = flags.get('--reference-margin'); + const referenceMargin = margin === undefined ? MARGIN_BAND_PT : Number(margin); + if (!Number.isFinite(referenceMargin) || referenceMargin < 0 || referenceMargin > 300) { + throw new SettingsError('--reference-margin is a number of points, 0 to 300'); + } + return { reference, neu, expected: flags.get('--expected') ?? null, referenceMargin }; +} + +/** An error as it may be printed: its kind and code, never its message, which may quote a file. */ +export const errorKind = (error: unknown): string => { + if (!(error instanceof Error)) return 'error'; + const code = (error as { code?: unknown }).code; + return typeof code === 'string' && /^[A-Z_]+$/.test(code) ? `${error.name} ${code}` : error.name; +}; + +const CONSOLE = ['log', 'info', 'warn', 'error', 'debug'] as const; + +/** + * `run`, with the console's methods held for its length: a library writing to the console + * (pdf.js's warnings quote a file's content) is not printed, only counted, and said once on + * standard error with the kinds of the errors it passed, never its words. + */ +export async function withConsoleHeld( + run: () => Promise, + err: (text: string) => void +): Promise { + // the methods as they are, put back as they were + const saved = CONSOLE.map((name) => [name, Reflect.get(console, name) as unknown] as const); + let count = 0; + const kinds = new Set(); + const held = (...args: unknown[]) => { + count++; + for (const arg of args) if (arg instanceof Error) kinds.add(errorKind(arg)); + }; + for (const name of CONSOLE) console[name] = held; + try { + return await run(); + } finally { + for (const [name, method] of saved) Reflect.set(console, name, method); + if (count > 0) { + const of = kinds.size > 0 ? ` (${[...kinds].sort().join(', ')})` : ''; + err(`${String(count)} message(s) from a library withheld${of}\n`); + } + } +} + +/** + * `pdf-export compare [--expected ] [--reference-margin ]`: the + * report as JSON on standard output; exit 0 with no difference (or none not recorded, with + * `--expected`), 1 with one, 2 on bad arguments or any other error. A file that cannot be + * read is named on standard error and is a difference; the others are still compared. + */ +export async function compareCommand( + args: readonly string[], + { + out, + err, + output = new Output(), + }: { out: (text: string) => void; err: (text: string) => void; output?: Output } +): Promise { + let settings: CompareSettings; + let recorded: Recorded[] = []; + try { + settings = compareSettings(args); + if (settings.expected !== null) recorded = readRecorded(settings.expected); + } catch (error) { + err(`${error instanceof SettingsError ? error.message : errorKind(error)}\n`); + return 2; + } + let result: { differences: Difference[]; unmatched: number[] }; + try { + const { reference, neu, referenceMargin } = settings; + const found = await withConsoleHeld( + () => + comparePaths(reference, neu, { + referenceMargin, + unreadable: (path, error) => { + err(`Not read: ${path} (${errorKind(error)})\n`); + }, + }), + err + ); + result = applyRecorded(found, recorded); + } catch (error) { + err( + `The comparison stopped: ${error instanceof SettingsError ? error.message : errorKind(error)}\n` + ); + return 2; + } + const { differences, unmatched } = result; + const unrecorded = differences.filter((d) => d.recorded === undefined).length; + const report = { + reference: settings.reference, + new: settings.neu, + counts: { + differences: differences.length, + recorded: differences.length - unrecorded, + unrecorded, + }, + differences: differences.map((d) => output.difference(d)), + /** The records of `--expected` that matched no difference, by their place in its list. */ + unmatched, + }; + out(`${JSON.stringify(report, null, 2)}\n`); + return unrecorded > 0 ? 1 : 0; +} diff --git a/src/compare/output.ts b/src/compare/output.ts new file mode 100644 index 0000000..6325e42 --- /dev/null +++ b/src/compare/output.ts @@ -0,0 +1,293 @@ +// What `compare` may print (spec 001 T035, FR-012, FR-013): the one place a value read from a +// PDF reaches its report. Text read from a PDF is printed only where it has one of a few +// strict shapes, none of which can hold a person's name: +// +// - a count (digits); +// - yes or no; +// - a release number; +// - one of a fixed list: the analysis's types and resources, a part's or a kind's name; +// - a stable identifier, `R-` then three capitals, a hyphen and digits; +// - an analysis token of the form the analysis service issues (src/analysis/token.ts); +// - a species from the fixed list of Reactome's species below. +// +// Everything else is printed as a hash and a length (or a count and a hash, for a list of +// names). The hashes printed are salted once a run, so a report's hashes say which values +// are equal within it, and cannot be checked against a guessed name without the run's salt, +// which is never printed: a name is a short guess, and an unsalted hash of one would be read +// back by guessing. +import { createHash, randomBytes } from 'node:crypto'; + +import { canonicalToken } from '../analysis/token.ts'; +import { NEVER_DRAWN } from '../layout/characters.ts'; + +const UNPRINTED = new RegExp(`${NEVER_DRAWN.source}+`, 'gu'); + +/** Text as compared: compatibility forms folded, white space and joiners left out. */ +export const fold = (text: string): string => text.normalize('NFKC').replace(UNPRINTED, ''); + +/** A short hash of a text, as the reduced form holds what it does not keep as text. */ +export const hash = (text: string): string => + createHash('sha256').update(text).digest('hex').slice(0, 16); + +/** A value held only as its hash and its length. */ +export interface Withheld { + hash: string; + length: number; +} + +/** People's names, or a list that holds them: how many, and one hash of them all. */ +export interface Hashed { + count: number; + hash: string; +} + +export const withhold = (text: string): Withheld => ({ hash: hash(text), length: text.length }); + +/** Reactome's species: human and the species its pathways are inferred to. */ +export const SPECIES = [ + 'Homo sapiens', + 'Bos taurus', + 'Caenorhabditis elegans', + 'Canis familiaris', + 'Danio rerio', + 'Dictyostelium discoideum', + 'Drosophila melanogaster', + 'Gallus gallus', + 'Mus musculus', + 'Mycobacterium tuberculosis', + 'Plasmodium falciparum', + 'Rattus norvegicus', + 'Saccharomyces cerevisiae', + 'Schizosaccharomyces pombe', + 'Sus scrofa', + 'Xenopus tropicalis', +] as const; + +/** The analysis service's types and resources. */ +export const ANALYSIS_TYPES = ['OVERREPRESENTATION', 'EXPRESSION', 'SPECIES_COMPARISON', 'GSA']; +export const RESOURCES = [ + 'TOTAL', + 'UNIPROT', + 'ENSEMBL', + 'CHEBI', + 'IUPHAR', + 'MIRBASE', + 'NCBI_PROTEIN', + 'EMBL', + 'COMPOUND', + 'PUBCHEM_COMPOUND', +]; + +export type Shape = + 'count' | 'yesno' | 'release' | 'stId' | 'token' | 'species' | { oneOf: readonly string[] }; + +export const STABLE_ID = /^R-[A-Z]{3}-\d+$/; + +/** One of a list, compared folded and case aside, as the list writes it. */ +function oneOf(value: string, list: readonly string[]): string | null { + const read = fold(value).toLowerCase(); + return list.find((item) => fold(item).toLowerCase() === read) ?? null; +} + +/** + * A value as it may be printed in this shape, or null where it does not have it. Strings are + * read folded; booleans are yes or no. + */ +export function shaped(value: unknown, shape: Shape): string | number | null { + if (typeof value === 'boolean') return shape === 'yesno' ? (value ? 'yes' : 'no') : null; + if (typeof value === 'number') { + if (!Number.isSafeInteger(value) || value < 0) return null; + if (shape === 'count') return value; + if (shape === 'release') return value > 0 && value < 10_000 ? value : null; + return null; + } + if (typeof value !== 'string') return null; + const read = fold(value); + if (typeof shape === 'object') return oneOf(read, shape.oneOf); + switch (shape) { + case 'count': + return /^\d{1,12}$/.test(read) ? Number(read) : null; + case 'release': + return /^\d{1,4}$/.test(read) && Number(read) > 0 ? Number(read) : null; + case 'yesno': + return /^(?:yes|true)$/i.test(read) ? 'yes' : /^(?:no|false)$/i.test(read) ? 'no' : null; + case 'stId': + return STABLE_ID.test(read) ? read : null; + case 'species': + return oneOf(read, SPECIES); + case 'token': + try { + // in its one canonical form: copies encoded otherwise are one token + return canonicalToken(read); + } catch { + return null; + } + } +} + +/** A value read from a PDF as the reduced form holds it: as printed, or withheld. */ +export type Checked = { value: string | number } | Withheld; + +/** A value kept as it is where it has its shape, or else withheld. */ +export function checked(value: string, shape: Shape): Checked { + const kept = shaped(value, shape); + return kept === null ? withhold(fold(value)) : { value: kept }; +} + +/** The parts, fields and kinds a difference may name. */ +export const PARTS = [ + 'file', + 'cover', + 'introduction', + 'analysis', + 'sections', + 'section', + 'figure', + 'contents', + 'short-entries', +] as const; +export const KINDS = [ + 'value', + 'count', + 'text', + 'order', + 'missing', + 'extra', + 'kind', + 'target', + 'missing-target', + 'unreadable', +] as const; + +/** Each field whose values may be printed, and the shape they must have. */ +const VALUES: Readonly> = { + 'cover title': { oneOf: ['same', 'differs'] }, + 'cover diagram': 'yesno', + 'cover excerpt': 'yesno', + 'introduction release': 'release', + 'introduction pathways': 'count', + 'introduction reactions': 'count', + 'analysis token': 'token', + 'analysis type': { oneOf: ANALYSIS_TYPES }, + 'analysis species': 'species', + 'analysis projection': 'yesno', + 'analysis interactors': 'yesno', + 'analysis identifiersFound': 'count', + 'analysis identifiersTotal': 'count', + 'analysis pathways': 'count', + 'analysis resource': { oneOf: RESOURCES }, + 'short-entries ': 'count', +}; + +/** The fields a difference may name: the analysis's properties and a section's parts too. */ +const FIELDS = new Set([ + ...Object.keys(VALUES).map((key) => key.split(' ')[1] ?? ''), + 'names', + 'reference', + 'new', + 'name', + 'location', + 'stId', + 'type', + 'compartments', + 'diseases', + 'inferredFrom', + 'summation', + 'before', + 'after', + 'references', + 'edits', + 'entities', + 'interactors', +]); + +/** A difference as the comparison makes it, before it is printed. */ +export interface Difference { + /** The file compared, by its name in its folder (never read from a PDF). */ + file: string; + part: (typeof PARTS)[number]; + field?: string; + kind: (typeof KINDS)[number]; + count: number; + /** The sections it is in, by stable identifier. */ + events?: string[]; + reference?: unknown; + new?: unknown; + /** Why it is expected, from `--expected`'s file (never read from a PDF). */ + recorded?: string; +} + +const isRecord = (value: unknown): value is Record => + value !== null && typeof value === 'object'; + +/** + * The report's printer: every value a difference holds passes through here. A value of its + * field's shape is printed; a Hashed list of names as its count and salted hash; anything + * else, as a salted hash and a length. + */ +export class Output { + readonly #salt: string; + + /** A salt of its own each run, unless one is given (for tests). */ + constructor(salt: string = randomBytes(16).toString('hex')) { + this.#salt = salt; + } + + #salted(digest: string): string { + return hash(`${this.#salt}\0${digest}`); + } + + #withheld(value: unknown): { hash: string; length: number } { + const text = typeof value === 'string' ? fold(value) : JSON.stringify(value); + return { hash: this.#salted(hash(text)), length: text.length }; + } + + /** One side's value, as it may be printed. */ + value(key: string, value: unknown): unknown { + if (value === null || value === undefined) return null; + if (isRecord(value)) { + const keys = Object.keys(value).sort().join(','); + const digest = value['hash']; + const hex = typeof digest === 'string' && /^[0-9a-f]{16}$/.test(digest); + const count = value['count']; + const length = value['length']; + // already withheld or hashed by the reduction: its digest salted + if (hex && keys === 'count,hash' && typeof count === 'number') { + return { count, hash: this.#salted(digest) }; + } + if (hex && keys === 'hash,length' && typeof length === 'number') { + return { hash: this.#salted(digest), length }; + } + if (keys === 'value') return this.value(key, value['value']); + return this.#withheld(value); + } + const shape = VALUES[key]; + const kept = shape === undefined ? null : shaped(value, shape); + return kept ?? this.#withheld(value); + } + + /** A difference, every field checked. */ + difference(d: Difference): Record { + const key = `${d.part} ${d.field ?? ''}`; + const named = (value: string | undefined, list: Iterable) => + value === undefined + ? undefined + : [...list].includes(value as T) + ? value + : this.#withheld(value); + const out: Record = { + file: d.file, + part: named(d.part, PARTS), + kind: named(d.kind, KINDS), + count: Number.isSafeInteger(d.count) ? d.count : 0, + }; + if (d.field !== undefined) out['field'] = named(d.field, FIELDS); + if (d.events !== undefined) { + out['events'] = d.events.map((e) => (STABLE_ID.test(e) ? e : this.#withheld(e))); + } + if ('reference' in d) out['reference'] = this.value(key, d.reference); + if ('new' in d) out['new'] = this.value(key, d.new); + if (d.recorded !== undefined) out['recorded'] = d.recorded; + return out; + } +} diff --git a/src/compare/read.ts b/src/compare/read.ts new file mode 100644 index 0000000..1ece30f --- /dev/null +++ b/src/compare/read.ts @@ -0,0 +1,374 @@ +// What FR-012's comparison reads from a PDF (spec 001 T035, research R8): its lines of text, +// its links and its figures, with pdf.js, from either layout: pdf-export's or `event-pdf`'s. +// Nothing here knows which made the file; src/compare/reduce.ts reads the parts from these. +// +// - **Lines:** the pieces of text that share a baseline on a page, left to right, each with +// its left edge and its largest size. What is in a page's top or bottom margin band (the +// running headers and page numbers, which FR-012 does not compare; MARGIN_BAND_PT unless +// set) and what is inside a figure (a vector diagram's labels) is left out. +// - **Links:** each link's box, its address or where in the PDF it lands. +// - **Figures:** the boxes of what is drawn: images, and painted paths that are neither thin +// (a rule, a table's border) nor the whole page, clustered where they lie within +// FIGURE_GAP_PT of each other, a line of thin paths joining the clusters it touches only +// where it touches two or more (a diagram's connector, never a table's borders); a cluster +// at least FIGURE_MIN_PT each way is a figure. A raster diagram is one image; a vector one +// is many paths. A table with thick borders or shaded rows near one another could be read +// as one, its text then left out (for T040 to check on `event-pdf`). +import { getDocument, OPS, VerbosityLevel } from 'pdfjs-dist/legacy/build/pdf.mjs'; + +/** + * The bands at a page's top and bottom read as margins, by default: within pdf-export's 20 mm + * (56.7 pt, src/layout/figure.ts), where its running headers and page numbers are, and above + * the first line of its text area. A setting (`compare --reference-margin`): T040 sets it for + * `event-pdf`'s files from their real margins, not known here. + */ +export const MARGIN_BAND_PT = 50; + +/** Shapes nearer than this are one figure. */ +export const FIGURE_GAP_PT = 30; + +/** A figure is at least this wide and this tall: a logo's mark or a bullet is not. */ +export const FIGURE_MIN_PT = 60; + +/** A shape thinner than this either way is a rule or a border, not part of a figure. */ +const THIN_PT = 2; + +/** A place in a PDF: its page, counted from 0, and a height on it in points from its foot. */ +export interface Spot { + page: number; + y: number; +} + +/** A box on a page: left, bottom, right, top, in points. */ +export interface Box { + page: number; + x1: number; + y1: number; + x2: number; + y2: number; +} + +export interface Line extends Spot { + /** Its left edge. */ + x: number; + /** Its largest piece's size. */ + size: number; + /** Its pieces' text, joined as pdf.js gives them. */ + text: string; +} + +export interface Link { + box: Box; + uri: string | null; + /** Where an internal link lands, or null. */ + landing: Spot | null; +} + +export interface Layout { + pages: { width: number; height: number }[]; + /** In reading order: by page, then from the top down. */ + lines: Line[]; + links: Link[]; + figures: Box[]; +} + +type Matrix = [number, number, number, number, number, number]; + +const IDENTITY: Matrix = [1, 0, 0, 1, 0, 0]; + +/** `m` then `n`: a point is taken through `m`, then through `n` (PDF's row-vector order). */ +function multiply(m: Matrix, n: Matrix): Matrix { + return [ + m[0] * n[0] + m[1] * n[2], + m[0] * n[1] + m[1] * n[3], + m[2] * n[0] + m[3] * n[2], + m[2] * n[1] + m[3] * n[3], + m[4] * n[0] + m[5] * n[2] + n[4], + m[4] * n[1] + m[5] * n[3] + n[5], + ]; +} + +/** A rectangle [x1, y1, x2, y2] through a matrix, as the box around its corners. */ +function through(rect: readonly number[], m: Matrix, page: number): Box { + const [x1 = 0, y1 = 0, x2 = 0, y2 = 0] = rect; + const xs: number[] = []; + const ys: number[] = []; + for (const [x, y] of [ + [x1, y1], + [x1, y2], + [x2, y1], + [x2, y2], + ] as const) { + xs.push(m[0] * x + m[2] * y + m[4]); + ys.push(m[1] * x + m[3] * y + m[5]); + } + return { + page, + x1: Math.min(...xs), + y1: Math.min(...ys), + x2: Math.max(...xs), + y2: Math.max(...ys), + }; +} + +const isMatrix = (value: unknown): value is Matrix => + Array.isArray(value) && value.length === 6 && value.every((n) => typeof n === 'number'); + +const IMAGES = new Set([ + OPS.paintImageXObject, + OPS.paintInlineImageXObject, + OPS.paintImageMaskXObject, + OPS.paintImageXObjectRepeat, + OPS.paintInlineImageXObjectGroup, + OPS.paintImageMaskXObjectGroup, + OPS.paintSolidColorImageMask, +]); + +/** The shapes a page paints: images, and painted paths, each as its box. */ +function shapes(fnArray: readonly number[], argsArray: readonly unknown[], page: number): Box[] { + const boxes: Box[] = []; + let ctm: Matrix = IDENTITY; + const stack: Matrix[] = []; + for (let i = 0; i < fnArray.length; i++) { + const fn = fnArray[i]; + const args = argsArray[i]; + if (fn === OPS.save || fn === OPS.paintFormXObjectBegin) { + stack.push(ctm); + const matrix: unknown = + fn === OPS.paintFormXObjectBegin && Array.isArray(args) ? args[0] : null; + if (isMatrix(matrix)) ctm = multiply(matrix, ctm); + } else if (fn === OPS.restore || fn === OPS.paintFormXObjectEnd) { + ctm = stack.pop() ?? IDENTITY; + } else if (fn === OPS.transform && isMatrix(args)) { + ctm = multiply(args, ctm); + } else if (fn === OPS.constructPath && Array.isArray(args)) { + // [how it is painted, its drawing, its box]: a path only clipped to is not drawn + const [paint, , minMax] = args as [unknown, unknown, unknown]; + if (paint === OPS.endPath || minMax === null || typeof minMax !== 'object') continue; + const rect = Array.from(minMax as ArrayLike); + if (rect.length === 4 && rect.every(Number.isFinite)) boxes.push(through(rect, ctm, page)); + } else if (fn !== undefined && IMAGES.has(fn)) { + boxes.push(through([0, 0, 1, 1], ctm, page)); + } + } + return boxes; +} + +/** Two boxes within `gap` of each other. */ +const near = (a: Box, b: Box, gap: number): boolean => + a.x1 - gap <= b.x2 && b.x1 - gap <= a.x2 && a.y1 - gap <= b.y2 && b.y1 - gap <= a.y2; + +const union = (a: Box, b: Box): Box => ({ + page: a.page, + x1: Math.min(a.x1, b.x1), + y1: Math.min(a.y1, b.y1), + x2: Math.max(a.x2, b.x2), + y2: Math.max(a.y2, b.y2), +}); + +/** Boxes joined into clusters wherever any two are within `gap`, until none are. */ +function clustered(boxes: readonly Box[], gap: number): Box[] { + let clusters: Box[] = []; + for (const box of boxes) { + let merged = { ...box }; + // a box may join clusters that were apart until it came + clusters = clusters.filter((cluster) => { + if (!near(cluster, merged, gap)) return true; + merged = union(cluster, merged); + return false; + }); + clusters.push(merged); + } + // a merge can bring a cluster near one it was not near before + for (let again = true; again;) { + again = false; + for (let i = 0; i < clusters.length && !again; i++) { + for (let j = i + 1; j < clusters.length && !again; j++) { + const [a, b] = [clusters[i], clusters[j]]; + if (!a || !b || !near(a, b, gap)) continue; + clusters[i] = union(a, b); + clusters.splice(j, 1); + again = true; + } + } + } + return clusters; +} + +/** The shapes a thin path joins must be at least this big each way, as a diagram's nodes are. */ +const NODE_PT = 4; + +/** How near a thin path must come to a shape to touch it. */ +const TOUCH_PT = 1; + +/** + * The figures among a page's shapes. The shapes that are neither thin nor all of the page + * are clustered within FIGURE_GAP_PT. Thin paths that touch one another are one line (a + * connector drawn in segments); a line joins the clusters it touches only where it touches + * two or more, each at least NODE_PT each way, as a connector joins a diagram's nodes. A + * table's hairline borders, which touch at most its one shaded header, join nothing, nor + * does a rule beside a figure. A cluster at least FIGURE_MIN_PT each way is a figure. + */ +export function figureBoxes(boxes: readonly Box[], width: number, height: number): Box[] { + const kept = boxes.filter((b) => (b.x2 - b.x1) * (b.y2 - b.y1) < 0.9 * width * height); + const thin = (b: Box) => b.x2 - b.x1 < THIN_PT || b.y2 - b.y1 < THIN_PT; + let clusters = clustered( + kept.filter((b) => !thin(b)), + FIGURE_GAP_PT + ); + // the thin paths, as lines of paths that touch + const lines = kept.filter(thin); + const group = lines.map((_, i) => i); + const root = (i: number): number => { + let r = i; + while (group[r] !== r) r = group[r] ?? r; + group[i] = r; + return r; + }; + for (let a = 0; a < lines.length; a++) { + for (let b = a + 1; b < lines.length; b++) { + const [la, lb] = [lines[a], lines[b]]; + if (la && lb && near(la, lb, TOUCH_PT)) group[root(a)] = root(b); + } + } + const joined = new Map(); + lines.forEach((line, i) => { + joined.set(root(i), [...(joined.get(root(i)) ?? []), line]); + }); + for (const paths of joined.values()) { + const node = (c: Box) => c.x2 - c.x1 >= NODE_PT && c.y2 - c.y1 >= NODE_PT; + const touched = clusters.filter( + (c) => node(c) && paths.some((path) => near(c, path, TOUCH_PT)) + ); + if (touched.length < 2) continue; + const merged = [...touched, ...paths].reduce(union); + clusters = [...clusters.filter((c) => !touched.includes(c)), merged]; + } + clusters = clustered(clusters, FIGURE_GAP_PT); + return clusters.filter((c) => c.x2 - c.x1 >= FIGURE_MIN_PT && c.y2 - c.y1 >= FIGURE_MIN_PT); +} + +export const inside = (x: number, y: number, box: Box): boolean => + x >= box.x1 - 1 && x <= box.x2 + 1 && y >= box.y1 - 1 && y <= box.y2 + 1; + +type PdfJs = Awaited['promise']>; + +const isRecord = (value: unknown): value is Record => + value !== null && typeof value === 'object'; + +/** Where a destination lands, or null where it names no page of the PDF. */ +async function land(doc: PdfJs, dest: unknown): Promise { + try { + const explicit: unknown = typeof dest === 'string' ? await doc.getDestination(dest) : dest; + if (!Array.isArray(explicit)) return null; + const [ref, kind, left, top] = explicit as unknown[]; + let page: number; + if (typeof ref === 'number') { + if (!Number.isInteger(ref) || ref < 0 || ref >= doc.numPages) return null; + page = ref; + } else if (isRecord(ref) && 'num' in ref && 'gen' in ref) { + page = await doc.getPageIndex(ref as { num: number; gen: number }); + } else { + return null; + } + const name = isRecord(kind) && 'name' in kind ? kind['name'] : null; + const height = name === 'XYZ' ? top : name === 'FitH' || name === 'FitBH' ? left : null; + if (typeof height === 'number') return { page, y: height }; + const [, , , pageTop = 0] = (await doc.getPage(page + 1)).view; + return { page, y: pageTop }; + } catch { + return null; + } +} + +interface Piece { + x: number; + y: number; + size: number; + str: string; +} + +/** A page's pieces as lines: those sharing a baseline, within a point, left to right. */ +function linesOf(pieces: Piece[], page: number): Line[] { + const lines: Line[] = []; + const sorted = [...pieces].sort((a, b) => b.y - a.y || a.x - b.x); + let current: Piece[] = []; + const flush = () => { + const [first] = current; + if (!first) return; + const row = current.sort((a, b) => a.x - b.x); + lines.push({ + page, + y: first.y, + x: Math.min(...row.map((p) => p.x)), + size: Math.max(...row.map((p) => p.size)), + text: row.map((p) => p.str).join(''), + }); + current = []; + }; + for (const piece of sorted) { + const [first] = current; + if (first && Math.abs(first.y - piece.y) > 1) flush(); + current.push(piece); + } + flush(); + return lines; +} + +/** + * A PDF's lines, links and figures, read with pdf.js. The bytes are handed over: pdf.js may + * detach their buffer, so a caller keeping them passes a copy. A Node Buffer is refused: give + * a plain Uint8Array over its memory. + */ +export async function readLayout( + data: Uint8Array, + { margin = MARGIN_BAND_PT }: { margin?: number } = {} +): Promise { + // not copied: pdf.js takes the buffer it is given, which the caller gives up + // pdf.js's warnings off: they quote the file's content (`Unknown command "…"`) + const task = getDocument({ data, verbosity: VerbosityLevel.ERRORS }); + try { + const doc = await task.promise; + const layout: Layout = { pages: [], lines: [], links: [], figures: [] }; + for (let i = 0; i < doc.numPages; i++) { + const page = await doc.getPage(i + 1); + const [, , width = 0, height = 0] = page.view; + layout.pages.push({ width, height }); + const operators = await page.getOperatorList(); + const figures = figureBoxes(shapes(operators.fnArray, operators.argsArray, i), width, height); + layout.figures.push(...figures); + const pieces: Piece[] = []; + for (const item of (await page.getTextContent()).items) { + if (!('str' in item) || item.str === '') continue; + const [a = 0, b = 0, c = 0, d = 0, x = 0, y = 0] = item.transform as number[]; + if (y <= margin || y >= height - margin) continue; + if (figures.some((box) => inside(x, y, box))) continue; + pieces.push({ x, y, size: Math.max(Math.hypot(a, b), Math.hypot(c, d)), str: item.str }); + } + layout.lines.push(...linesOf(pieces, i).filter((line) => line.text.trim() !== '')); + for (const annot of (await page.getAnnotations()) as unknown[]) { + if (!isRecord(annot) || annot['subtype'] !== 'Link') continue; + const [x1 = 0, y1 = 0, x2 = 0, y2 = 0] = (annot['rect'] as number[] | undefined) ?? []; + const address = annot['unsafeUrl'] ?? annot['url']; + const uri = typeof address === 'string' ? address : null; + const dest = annot['dest']; + layout.links.push({ + box: { + page: i, + x1: Math.min(x1, x2), + y1: Math.min(y1, y2), + x2: Math.max(x1, x2), + y2: Math.max(y1, y2), + }, + uri, + landing: + uri === null && dest !== undefined && dest !== null ? await land(doc, dest) : null, + }); + } + page.cleanup(); + } + return layout; + } finally { + await task.destroy(); + } +} diff --git a/src/compare/reduce.ts b/src/compare/reduce.ts new file mode 100644 index 0000000..13d31dc --- /dev/null +++ b/src/compare/reduce.ts @@ -0,0 +1,603 @@ +// A PDF reduced to FR-012's parts (spec 001 T035, research R8), from either layout: the +// cover's title, diagram, excerpt note and distinct names; the introduction's release and +// counts; the analysis's properties; each section's parts and their text; the contents +// list's entries and their targets; and each section's figure's kind. +// +// Text is compared folded: compatibility forms folded (NFKC), and white space and the +// characters never drawn (FR-017's NEVER_DRAWN: joiners, U+2060 among them) left out, so a +// line broken elsewhere, or the cover's names written with U+2060 between their characters +// and no-break spaces between their words (`event-pdf`, research R2), read the same. +// +// What the reduced form holds as text (a section's parts, the contents' titles) is compared, +// never printed: src/compare/output.ts prints only values of strict shapes. People's names +// are not held as text at all: the cover's names are a count and a hash, its title (which +// a misread cover could take from the names) a hash and a length, a section's literature +// references and edit history one hash an item, and an analysis property not of its shape a +// hash and a length (FR-013). +// +// The parts are found by their labels: pdf-export's, and other words for them. `event-pdf`'s +// own labels are not known here, and are for T040 to add from its real files; a label not +// known leaves its part unread, which the comparison reports as a difference. +import { CONTRIBUTORS, INSTITUTIONS } from '../layout/text.ts'; +import { + ANALYSIS_TYPES, + RESOURCES, + checked, + fold, + hash, + withhold, + type Checked, + type Hashed, + type Shape, + type Withheld, +} from './output.ts'; +import type { Box, Layout, Line, Link, Spot } from './read.ts'; + +export { fold, hash } from './output.ts'; +export type { Hashed } from './output.ts'; + +export type FigureKind = 'none' | 'diagram' | 'selection' | 'reaction' | 'figure'; + +export interface ReducedPart { + part: PartName; + /** Its items, folded, in order; a part that names people holds a hash for each. */ + items: string[]; +} + +export interface ReducedSection { + stId: string; + /** Its parts in the order they are printed, its name first. */ + parts: ReducedPart[]; + /** + * Its figure: `diagram`, `selection` or `reaction` where its link to the website says + * which; `reaction` too for a reaction's section whose figure has no such link; else + * `figure`, a figure of a kind not read. + */ + figure: FigureKind; +} + +export interface ContentsItem { + title: string; + /** The section's stable identifier, or the front part's name, it lands on; null if none. */ + target: string | null; +} + +export interface Reduced { + cover: { title: Withheld; diagram: boolean; excerpt: boolean; names: Hashed }; + introduction: { release: number | null; pathways: number | null; reactions: number | null }; + /** The analysis's properties by name, each of its shape or withheld; null without one. */ + analysis: Record | null; + sections: ReducedSection[]; + /** Null where the PDF has no contents list. */ + contents: ContentsItem[] | null; + /** Short entries at an event's later places (FR-004), which today's files do not write. */ + shortEntries: number; +} + +/** Distinct names, folded, counted and hashed together; empty names left out. */ +export function hashNames(names: readonly string[]): Hashed { + const distinct = [...new Set(names.map(fold).filter((name) => name !== ''))].sort(); + return { count: distinct.length, hash: hash(distinct.join('\n')) }; +} + +/** + * The cover's names from their text. Where the names are written with U+2060 between their + * characters (research R2), each name is a run of characters joined by U+2060 or no-break + * spaces, and whatever else stands between runs parts them; otherwise names are parted by + * semicolons, as pdf-export writes them. + */ +export function coverNames(text: string): string[] { + if (text.includes('\u2060')) { + return text.match(/[^\s\u2060\u00a0](?:[\u2060\u00a0]+[^\s\u2060\u00a0])+/gu) ?? []; + } + return text.split(';'); +} + +export const PART_NAMES = [ + 'name', + 'location', + 'stId', + 'type', + 'compartments', + 'diseases', + 'inferredFrom', + 'summation', + 'before', + 'after', + 'references', + 'edits', + 'entities', + 'interactors', +] as const; + +export type PartName = (typeof PART_NAMES)[number]; + +/** A label at a line's start, whole and case aside, then a colon, white space or its end. */ +const labelPattern = (labels: readonly string[]): RegExp[] => + labels.map((label) => { + const words = label.split(' ').map((w) => w.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')); + return new RegExp(`^\\s*${words.join('\\s*')}\\s*(?::|\\s|$)`, 'i'); + }); + +interface PartRule { + part: PartName; + /** The labels it is known by, longest first where one begins another, compiled. */ + labels: readonly RegExp[]; + /** A fact, beside its label: a line of the section's own left edge after it ends it. */ + fact?: boolean; + /** Its items, within its text: parted by a pattern, or each on lines of their own. */ + split?: RegExp | 'numbered' | 'dated' | 'rows'; + /** Whether its items name people, and so are held as hashes. */ + names?: boolean; +} + +/** The labelled parts, in the order a section prints them (FR-005). */ +const RULES: readonly PartRule[] = [ + { part: 'location', labels: labelPattern(['location']), fact: true, split: /[›»>]/ }, + { part: 'stId', labels: labelPattern(['stable identifier', 'stable id']), fact: true }, + { part: 'type', labels: labelPattern(['reaction type', 'type']), fact: true }, + { + part: 'compartments', + labels: labelPattern(['compartments', 'compartment']), + fact: true, + split: /,/, + }, + { part: 'diseases', labels: labelPattern(['diseases', 'disease']), fact: true, split: /,/ }, + { part: 'inferredFrom', labels: labelPattern(['inferred from']), fact: true, split: /;/ }, + { part: 'summation', labels: [] }, + { + part: 'before', + labels: labelPattern(['preceded by', 'preceding events', 'preceding event']), + fact: true, + split: /;/, + }, + { + part: 'after', + labels: labelPattern(['followed by', 'following events', 'following event']), + fact: true, + split: /;/, + }, + { + part: 'references', + labels: labelPattern(['literature references', 'references']), + split: 'numbered', + names: true, + }, + { part: 'edits', labels: labelPattern(['edit history']), split: 'dated', names: true }, + { part: 'entities', labels: labelPattern(['entities found']), split: 'rows' }, + { part: 'interactors', labels: labelPattern(['interactors found']), split: 'rows' }, +]; + +const ruleOf = (part: PartName): PartRule | undefined => RULES.find((r) => r.part === part); +const orderOf = (part: PartName): number => PART_NAMES.indexOf(part); +const STABLE_ID_LABELS = ruleOf('stId')?.labels ?? []; +const LOCATION_LABELS = ruleOf('location')?.labels ?? []; + +/** A line's text as read: compatibility forms folded, U+2060 left out. */ +const plain = (text: string): string => text.normalize('NFKC').replace(/\u2060/g, ''); + +/** What follows a line's label, or null where it does not begin with one of them. */ +function labelled(text: string, labels: readonly RegExp[]): string | null { + const read = plain(text); + for (const label of labels) { + const match = label.exec(read); + if (match) return read.slice(match[0].length); + } + return null; +} + +const STABLE_ID = /R-[A-Z]{3}-\d+/; +const STABLE_ID_AT_END = /R-[A-Z]{3}-\d+$/; + +/** A heading is set at least this much larger than its body text (as check-book reads it). */ +const HEADING_RATIO = 11.25 / 10.5; + +/** + * How far below where a link lands its target's first line may be: Chromium's own + * destinations land up to about 76 pt above a heading's baseline (at the top of its page, or + * of the space above it), measured on pdf-export's single-event PDFs. + */ +export const LANDING_PT = 100; + +/** Whether `a` comes before `b` in reading order: an earlier page, or higher on the same one. */ +const before = (a: Spot, b: Spot): boolean => a.page < b.page || (a.page === b.page && a.y > b.y); + +/** The size most of the text is set at. */ +function bodySize(lines: readonly Line[]): number { + const counts = new Map(); + for (const line of lines) { + const size = Math.round(line.size * 100) / 100; + counts.set(size, (counts.get(size) ?? 0) + fold(line.text).length); + } + return [...counts].sort(([, m], [, n]) => n - m)[0]?.[0] ?? 0; +} + +const FRONT_HEADINGS = { + introduction: /^introduction$/i, + analysis: /^analysis/i, + contents: /^(?:table of )?contents$/i, +} as const; + +type FrontName = keyof typeof FRONT_HEADINGS; + +const linkOn = (line: Line, box: Box): boolean => + box.page === line.page && line.y >= box.y1 - 2 && line.y <= box.y2 + 2; + +/** Items grouped by their page, so each line looks only at its own page's. */ +function byPage(items: readonly T[], page: (item: T) => number): (p: number) => readonly T[] { + const pages = new Map(); + for (const item of items) { + const list = pages.get(page(item)); + if (list) list.push(item); + else pages.set(page(item), [item]); + } + return (p) => pages.get(p) ?? []; +} + +/** A layout reduced to FR-012's parts. */ +export function reduce(layout: Layout): Reduced { + const { lines, figures } = layout; + const linksOn = byPage(layout.links, (link) => link.box.page); + const figuresOn = byPage(figures, (box) => box.page); + const body = bodySize(lines.filter((line) => line.page > 0)); + const heading = (line: Line) => line.size >= body * HEADING_RATIO; + const internalOn = (line: Line): Link | undefined => + linksOn(line.page).find((link) => link.uri === null && linkOn(line, link.box)); + + // each section, at its stable identifier's label; its name the heading above it, above its + // location where that comes first + const anchors = lines.flatMap((line, i) => { + const rest = labelled(line.text, STABLE_ID_LABELS); + const stId = rest === null ? undefined : STABLE_ID.exec(fold(rest))?.[0]; + return stId === undefined ? [] : [{ i, stId }]; + }); + const starts = anchors.map(({ i }, n) => { + const floor = n > 0 ? (anchors[n - 1]?.i ?? 0) + 1 : 0; + let j = i - 1; + for (let k = i - 1; k >= Math.max(floor, i - 6); k--) { + if (labelled(lines[k]?.text ?? '', LOCATION_LABELS) !== null) { + j = k - 1; + break; + } + } + let start = j + 1; + while (start - 1 >= floor && lines[start - 1] && heading(lines[start - 1] as Line)) start--; + return start; + }); + const startSet = new Set(starts); + const firstSection = starts[0] ?? lines.length; + const lastAnchor = anchors.at(-1)?.i ?? -1; + + // the front parts' headings: the introduction's and the analysis's before the sections, the + // contents list's before them or after them all (today's files put it last) + const front = new Map(); + lines.forEach((line, i) => { + if (line.page === 0 || !heading(line) || startSet.has(i)) return; + const text = plain(line.text).trim(); + for (const name of Object.keys(FRONT_HEADINGS) as FrontName[]) { + if (front.has(name) || !FRONT_HEADINGS[name].test(text)) continue; + if (name !== 'contents' && i >= firstSection) continue; + if (name === 'contents' && i >= firstSection && i < lastAnchor) continue; + front.set(name, i); + } + }); + const boundaries = [...new Set([...starts, ...front.values(), lines.length])].sort( + (a, b) => a - b + ); + /** The first boundary after a line: the next section or front part, or the end. */ + const regionEnd = (from: number): number => { + let lo = 0; + let hi = boundaries.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if ((boundaries[mid] ?? Infinity) <= from) lo = mid + 1; + else hi = mid; + } + return boundaries[lo] ?? lines.length; + }; + const spotOf = (i: number): Spot | null => { + const line = lines[i]; + return line ? { page: line.page, y: line.y + line.size } : null; + }; + + // where an internal link lands: the first section or front part starting at or below it on + // its page, within LANDING_PT + const targetsOn = byPage( + [ + ...anchors.map(({ stId }, n) => ({ at: starts[n] ?? 0, name: stId })), + ...[...front].map(([name, at]) => ({ at, name })), + ].flatMap((t) => { + const line = lines[t.at]; + return line ? [{ ...t, line }] : []; + }), + (t) => t.line.page + ); + const resolve = (landing: Spot | null): string | null => { + if (!landing) return null; + let best: { name: string; y: number } | null = null; + for (const { line, name } of targetsOn(landing.page)) { + if (line.y > landing.y + 2 || line.y < landing.y - LANDING_PT) continue; + if (!best || line.y > best.y) best = { name, y: line.y }; + } + return best?.name ?? null; + }; + + let shortEntries = 0; + const sections = anchors.map(({ i, stId }, n): ReducedSection => { + const start = starts[n] ?? i; + const end = regionEnd(i); + const left = lines[i]?.x ?? 0; + const from = spotOf(start); + const to = spotOf(end); + const own: Box[] = []; + if (from) { + for (let page = from.page; page <= (to?.page ?? from.page); page++) { + for (const box of figuresOn(page)) { + const top = { page: box.page, y: box.y2 }; + if (!before(top, from) && (to === null || before(top, to))) own.push(box); + } + } + } + const parts: { part: PartName; texts: string[] }[] = []; + const headingLines: Line[] = []; + for (let k = start; k < i && lines[k] && heading(lines[k] as Line); k++) { + headingLines.push(lines[k] as Line); + } + parts.push({ part: 'name', texts: headingLines.map((line) => line.text) }); + let current: { part: PartName; texts: string[] } | null = null; + for (let k = start + headingLines.length; k < end; k++) { + const line = lines[k]; + if (!line) continue; + const order = current ? orderOf(current.part) : orderOf('name'); + let matched: { part: PartName; rest: string } | null = null; + for (const rule of RULES) { + if (orderOf(rule.part) <= order || rule.labels.length === 0) continue; + const rest = labelled(line.text, rule.labels); + if (rest !== null) { + matched = { part: rule.part, rest }; + break; + } + } + if (matched) { + current = { part: matched.part, texts: [matched.rest] }; + parts.push(current); + continue; + } + // a short entry: an event's name and stable identifier, linked to its section + if (STABLE_ID_AT_END.test(fold(line.text)) && internalOn(line)) { + shortEntries++; + continue; + } + const afterFigure = own.some( + (box) => box.page < line.page || (box.page === line.page && box.y1 > line.y) + ); + const fact = current ? ruleOf(current.part)?.fact === true : true; + if ( + fact && + orderOf(current?.part ?? 'name') < orderOf('summation') && + (afterFigure || line.x <= left + 1) + ) { + current = { part: 'summation', texts: [line.text] }; + parts.push(current); + continue; + } + current?.texts.push(line.text); + } + return { + stId, + parts: parts.map(({ part, texts }) => ({ part, items: items(part, texts) })), + figure: figureKind( + own, + parts.some((p) => p.part === 'type') + ), + }; + }); + + function figureKind(own: readonly Box[], reaction: boolean): FigureKind { + const [box] = own; + if (!box) return 'none'; + for (const link of linksOn(box.page)) { + if (link.uri === null) continue; + const overlaps = + link.box.x1 < box.x2 && + box.x1 < link.box.x2 && + link.box.y1 < box.y2 && + box.y1 < link.box.y2; + if (!overlaps) continue; + if (/\/PathwayBrowser\/#\/[^&]+&SEL=/.test(link.uri)) return 'selection'; + if (/\/PathwayBrowser\//.test(link.uri)) return 'diagram'; + if (/\/content\/detail\//.test(link.uri)) return 'reaction'; + } + return reaction ? 'reaction' : 'figure'; + } + + /** A front part's lines: after its heading, to the next section or front part. */ + const region = (name: FrontName): Line[] | null => { + const at = front.get(name); + return at === undefined ? null : lines.slice(at + 1, regionEnd(at)); + }; + const analysisLines = region('analysis'); + const contentsLines = region('contents'); + return { + cover: cover( + lines.filter((line) => line.page === 0), + figuresOn(0) + ), + introduction: introduction( + region('introduction') ?? lines.slice(0, firstSection).filter((line) => line.page > 0) + ), + analysis: analysisLines ? analysis(analysisLines, heading) : null, + sections, + contents: contentsLines + ? contents(contentsLines, (line) => resolve(internalOn(line)?.landing ?? null)) + : null, + shortEntries, + }; +} + +/** A part's items from its lines' texts, folded; hashed where they name people. */ +function items(part: PartName, texts: readonly string[]): string[] { + const rule = ruleOf(part); + let list: string[]; + const split = rule?.split; + if (split === 'numbered' || split === 'dated') { + const marker = split === 'numbered' ? /^\s*(?:\d+[.)]|\[\d+\])\s*/ : /^\s*\d{4}-\d{2}-\d{2}/; + const found: string[] = []; + for (const text of texts) { + const read = plain(text); + const last = found.pop(); + if (last !== undefined) found.push(marker.test(read) ? last : last + read); + if (marker.test(read)) found.push(split === 'numbered' ? read.replace(marker, '') : read); + // before the first item: the table's header, not an item + } + list = found; + } else if (split === 'rows') { + // each row a line; a first line with no digit is the table's header + const rows = texts.filter((text) => text.trim() !== ''); + list = rows.filter((text, i) => i > 0 || /\d/.test(text)); + } else if (split instanceof RegExp) { + list = texts.join(' ').split(split); + } else { + list = [texts.join(' ')]; + } + const folded = list.map(fold).filter((item) => item !== ''); + return rule?.names ? folded.map(hash) : folded; +} + +/** Lines before any of these on the cover end its names. */ +const AFTER_NAMES = new RegExp( + `\\d|institut|universit|licen[cs]|creative\\s*commons|excerpt|copyright|©|${INSTITUTIONS.map( + (i) => fold(i).slice(0, 12) + ).join('|')}`, + 'i' +); + +/** The cover's label before its names: pdf-export's, or a word that says what they are. */ +const NAMES_LABEL = /^\s*(?:authored|authors?|contributors?|edited|editors?|curated|curators?)\b/i; + +/** + * The cover. Its title is its largest text, held as a hash and a length: where a figure took + * in the real title, the largest text left could be the names. + */ +function cover(page: readonly Line[], figures: readonly Box[]): Reduced['cover'] { + const largest = Math.max(0, ...page.map((line) => line.size)); + const first = page.findIndex((line) => Math.abs(line.size - largest) < 0.5); + let last = first; + while (page[last + 1] && Math.abs((page[last + 1]?.size ?? 0) - largest) < 0.5) last++; + const titleLines = first < 0 ? [] : page.slice(first, last + 1); + const title = fold(titleLines.map((line) => line.text).join('')); + const below = titleLines.at(-1); + const diagram = figures.some((box) => box.page === 0 && below !== undefined && box.y2 < below.y); + const excerpt = page.some((line) => /excerpt/i.test(plain(line.text))); + + // the names: after their label, up to what follows them; or the lines written with U+2060 + const contributors = fold(CONTRIBUTORS).toLowerCase(); + const at = page.findIndex( + (line) => fold(line.text).toLowerCase() === contributors || NAMES_LABEL.test(plain(line.text)) + ); + let region: Line[] = []; + if (at >= 0) { + for (const line of page.slice(at + 1)) { + if (AFTER_NAMES.test(fold(line.text))) break; + region.push(line); + } + } else { + region = page.filter((line) => line.text.includes('\u2060')); + } + return { + title: withhold(title), + diagram, + excerpt, + names: hashNames(coverNames(region.map((line) => line.text).join('\n'))), + }; +} + +function introduction(region: readonly Line[]): Reduced['introduction'] { + const text = fold(region.map((line) => line.text).join(' ')); + const number = (pattern: RegExp): number | null => { + const found = pattern.exec(text)?.[1]; + return found === undefined ? null : Number(found.replace(/,/g, '')); + }; + return { + release: number(/release(?:version|number)?:?(\d+)/i), + pathways: number(/(\d[\d,]*)pathways/i), + reactions: number(/(\d[\d,]*)reactions/i), + }; +} + +/** The analysis's properties, the labels each is known by (longest first), and their shapes. */ +const PROPERTIES: readonly { name: string; labels: readonly RegExp[]; shape: Shape }[] = [ + { name: 'token', labels: labelPattern(['token']), shape: 'token' }, + { + name: 'type', + labels: labelPattern(['analysis type', 'type']), + shape: { oneOf: ANALYSIS_TYPES }, + }, + { name: 'species', labels: labelPattern(['species']), shape: 'species' }, + { + name: 'projection', + labels: labelPattern(['projected to human', 'projection']), + shape: 'yesno', + }, + { + name: 'interactors', + labels: labelPattern(['interactors included', 'include interactors', 'interactors']), + shape: 'yesno', + }, + { name: 'identifiers', labels: labelPattern(['identifiers found']), shape: 'count' }, + { name: 'pathways', labels: labelPattern(['pathways hit', 'pathways found']), shape: 'count' }, + { name: 'resource', labels: labelPattern(['resource']), shape: { oneOf: RESOURCES } }, +]; + +/** + * The analysis's properties, from its region up to the first stable identifier or the next + * heading (a section's, where its stable identifier was not read), each kept only where it + * has its shape (src/compare/output.ts), else withheld. + */ +function analysis( + region: readonly Line[], + heading: (line: Line) => boolean +): Record { + const properties: Record = {}; + for (const line of region) { + if (heading(line) || STABLE_ID.test(fold(line.text))) break; + for (const { name, labels, shape } of PROPERTIES) { + if (name in properties || (name === 'identifiers' && 'identifiersFound' in properties)) { + continue; + } + const rest = labelled(line.text, labels); + if (rest === null) continue; + const value = fold(rest); + if (name === 'identifiers') { + const [found = '', total, more] = value.split(/of/i); + properties['identifiersFound'] = checked(found, 'count'); + if (total !== undefined) properties['identifiersTotal'] = checked(total, 'count'); + // a value of another form than `x of y`, withheld whole + if (more !== undefined) properties['identifiersFound'] = checked(value, 'count'); + } else { + properties[name] = checked(value, shape); + } + break; + } + } + return properties; +} + +/** The contents list's entries: each up to a line ending in its page's number. */ +function contents(region: readonly Line[], target: (line: Line) => string | null): ContentsItem[] { + const entries: ContentsItem[] = []; + let pending: Line[] = []; + const flush = () => { + if (pending.length === 0) return; + const title = fold(pending.map((line) => line.text).join(' ')).replace(/[.·…]*\d+$/, ''); + const landed = pending.map(target).find((t) => t !== null) ?? null; + if (title !== '') entries.push({ title, target: landed }); + pending = []; + }; + for (const line of region) { + pending.push(line); + if (/\d\s*$/.test(plain(line.text))) flush(); + } + flush(); + return entries; +} diff --git a/src/print/contents.ts b/src/print/contents.ts index 7517242..4ec8362 100644 --- a/src/print/contents.ts +++ b/src/print/contents.ts @@ -25,7 +25,7 @@ import { type PDFContext, type PDFObject, } from '@cantoo/pdf-lib'; -import { getDocument } from 'pdfjs-dist/legacy/build/pdf.mjs'; +import { getDocument, VerbosityLevel } from 'pdfjs-dist/legacy/build/pdf.mjs'; import { layoutBookFront, type LayoutOptions } from '../layout/document.ts'; import { probeTargets, type Target } from './links.ts'; @@ -57,7 +57,8 @@ export interface Landed { /** Where each target landed in a PDF, from its named destinations. */ export async function landed(pdf: Uint8Array, targets: Iterable): Promise { // pdf.js may take the buffer over, so it is given a copy - const task = getDocument({ data: pdf.slice() }); + // pdf.js's warnings off: they quote the file's content, which holds names + const task = getDocument({ data: pdf.slice(), verbosity: VerbosityLevel.ERRORS }); try { const doc = await task.promise; const pages = new Map(); diff --git a/test/compare/compare.test.ts b/test/compare/compare.test.ts new file mode 100644 index 0000000..016de25 --- /dev/null +++ b/test/compare/compare.test.ts @@ -0,0 +1,473 @@ +// FR-012's comparison (spec 001 T035): each part's difference is caught, list order is a +// difference of its own that `--expected` records, and no name is printed. The PDFs are +// printed here from a sample document (samples.ts): pdf-export's own, and a reference in +// `event-pdf`'s shape. No assertion here prints a text read from a PDF: a failure says +// which part, never what a name is. +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { after, before, describe, test } from 'node:test'; +import { fileURLToPath } from 'node:url'; + +import { compareCommand, compareReduced, type Difference } from '../../src/compare/compare.ts'; +import { readLayout } from '../../src/compare/read.ts'; +import { coverNames, hashNames, reduce, type Reduced } from '../../src/compare/reduce.ts'; +import type { Document } from '../../src/model/types.ts'; +import { Printer } from '../../src/print/print.ts'; +import { + PEOPLE, + R1, + R2, + S, + joined, + printOurs, + printReference, + sampleAnalysis, + sampleDocument, + type ReferenceOptions, +} from './samples.ts'; + +/** A difference as the tests name it: part, field and kind, and its events. */ +const brief = (d: Difference) => + [d.part, d.field, d.kind, ...(d.events ?? [])].filter((w) => w !== undefined).join(' '); + +/** The resource, which `event-pdf` reports and pdf-export does not print. */ +const RESOURCE = 'analysis resource missing'; + +/** Every surname and initial of the placeholders, as a printed name could show them. */ +const NAME_PARTS = PEOPLE.flatMap((name) => name.split(/[\s,]+/)).filter((w) => w.length > 2); + +/** Whether a text holds any placeholder's name, or any part of one; never prints the text. */ +function assertNoName(text: string, where: string): void { + for (const part of NAME_PARTS) { + assert.ok(!text.includes(part), `a name was printed in ${where}`); + } +} + +/** The reference's first child section moved to the end. */ +function moveFirstChildLast(doc: Document): void { + const [root, child, ...rest] = doc.places; + if (root && child) doc.places = [root, ...rest, child]; +} + +describe('compare (FR-012)', { timeout: 300_000 }, () => { + let printer: Printer; + let folder: string; + let ours: Uint8Array; + let oursReduced: Reduced; + const analysis = sampleAnalysis(); + + // a copy each time: pdf.js may take the buffer it is given + const reduced = async (pdf: Uint8Array) => reduce(await readLayout(pdf.slice())); + /** The reference printed from a document changed by `change`, with these options. */ + const reference = async ( + change: (doc: Document) => void = () => undefined, + options: ReferenceOptions = {} + ) => { + const doc = sampleDocument(); + change(doc); + return printReference(printer, doc, { analysis, ...options }); + }; + /** The differences between a reference and pdf-export's sample, briefly. */ + const differences = async (ref: Uint8Array) => + compareReduced('sample.pdf', await reduced(ref), oursReduced).map(brief); + + before(async () => { + folder = mkdtempSync(join(tmpdir(), 'pdf-export-compare-')); + printer = await Printer.start(); + ours = await printOurs(printer, sampleDocument(), analysis); + oursReduced = await reduced(ours); + }); + after(async () => { + await printer.close(); + rmSync(folder, { recursive: true, force: true }); + }); + + test('the same document in both layouts differs in nothing but the resource', async () => { + assert.deepEqual(await differences(await reference()), [RESOURCE]); + }); + + test('each section is read with its parts, figure and items, names held as hashes', () => { + const byId = new Map(oursReduced.sections.map((s) => [s.stId, s])); + assert.deepEqual([...byId.keys()], ['R-HSA-9000001', R1.stId, S.stId, R2.stId]); + const parts = (stId: string) => byId.get(stId)?.parts.map((p) => p.part); + assert.deepEqual(parts(R1.stId), [ + 'name', + 'location', + 'stId', + 'type', + 'compartments', + 'inferredFrom', + 'summation', + 'after', + 'references', + 'edits', + 'entities', + ]); + // a summation beginning with a label's word is still a summation + assert.deepEqual(parts(S.stId), [ + 'name', + 'location', + 'stId', + 'compartments', + 'diseases', + 'summation', + 'references', + 'edits', + ]); + assert.deepEqual( + oursReduced.sections.map((s) => s.figure), + ['diagram', 'reaction', 'selection', 'reaction'] + ); + const edits = byId.get(R1.stId)?.parts.find((p) => p.part === 'edits')?.items ?? []; + assert.equal(edits.length, 2); + assert.ok( + edits.every((item) => /^[0-9a-f]{16}$/.test(item)), + 'edits are held as hashes' + ); + assertNoName(JSON.stringify(oursReduced), 'the reduced form'); + }); + + describe('the cover', () => { + test('its distinct names: another name, or one more, is caught as a count and hash', async () => { + const more = await differences( + await reference(undefined, { names: [...PEOPLE.slice(0, 3)] }) + ); + assert.deepEqual(more, ['cover names count', RESOURCE]); + const other = await differences( + await reference(undefined, { names: [PEOPLE[0], PEOPLE[3]] }) + ); + assert.deepEqual(other, ['cover names value', RESOURCE]); + // each side as a count and a hash, nothing else + const [names] = compareReduced( + 'x', + await reduced(await reference(undefined, { names: [PEOPLE[3]] })), + oursReduced + ); + assert.deepEqual(Object.keys(names?.reference ?? {}), ['count', 'hash']); + assert.deepEqual(Object.keys(names?.new ?? {}), ['count', 'hash']); + }); + + test('its title, diagram and excerpt note', async () => { + const changed = await differences( + await reference((doc) => { + doc.cover.title = 'Another title'; + doc.cover.figure = null; + doc.cover.excerpt = false; + }) + ); + assert.deepEqual(changed, [ + 'cover title value', + 'cover diagram value', + 'cover excerpt value', + RESOURCE, + ]); + }); + + test('names written with U+2060 and no-break spaces read as the same names', () => { + const plain = PEOPLE.slice(0, 3); + const written = plain.map(joined).join(', '); + assert.ok(written.includes('\u2060') && written.includes('\u00a0')); + assert.deepEqual(hashNames(coverNames(written)), hashNames(plain)); + // each once, however often it is written + assert.deepEqual(hashNames(coverNames(`${written}, ${joined(PEOPLE[0])}`)), hashNames(plain)); + assert.deepEqual(hashNames(coverNames(plain.join('; '))), hashNames(plain)); + assert.notDeepEqual(hashNames(coverNames(written)), hashNames(PEOPLE.slice(0, 2))); + }); + }); + + test("the introduction's release and counts", async () => { + const changed = await differences(await reference(undefined, { release: 97 })); + assert.deepEqual(changed, ['introduction release value', RESOURCE]); + const counts = await differences( + await reference((doc) => { + doc.release = { version: 98, pathways: 3, reactions: 1 }; + }) + ); + assert.deepEqual(counts, [ + 'introduction pathways value', + 'introduction reactions value', + RESOURCE, + ]); + }); + + test("the analysis's properties, not its explanatory text", async () => { + const p = analysis.properties; + const changed = await differences( + await reference(undefined, { + properties: [ + ['Token', p.token], + ['Analysis type', p.type], + ['Species', p.species], + ['Projection', 'false'], + ['Interactors', String(p.interactors)], + ['Identifiers found', '3 of 6'], + ['Pathways found', '5'], + ['Resource', 'TOTAL'], + ], + }) + ); + assert.deepEqual(changed, [ + 'analysis identifiersTotal value', + 'analysis pathways value', + 'analysis projection value', + RESOURCE, + ]); + }); + + describe('the sections', () => { + test("a part's text, a part missing, and a list's order, each by its section", async () => { + const changed = await differences( + await reference((doc) => { + for (const place of doc.places) { + if (!place.full) continue; + const { section } = place; + if (section.event.stId === R1.stId) section.summations = ['

Other text.

']; + if (section.event.stId === S.stId) section.diseases = []; + if (section.event.stId === R2.stId) section.event.compartments.reverse(); + } + }) + ); + assert.deepEqual(changed, [ + RESOURCE, + `section summation text ${R1.stId}`, + `section diseases extra ${S.stId}`, + `section compartments order ${R2.stId}`, + ]); + }); + + test('a section missing, and the sections in another order', async () => { + const missing = await differences( + await reference((doc) => { + doc.places = doc.places.filter((p) => !p.full || p.section.event.stId !== S.stId); + }) + ); + assert.ok(missing.includes(`sections extra ${S.stId}`), missing.join('\n')); + const moved = await differences(await reference(moveFirstChildLast)); + assert.ok(moved.includes(`sections order ${R1.stId}`), 'the order is missed'); + }); + + test('the references and the edit history, compared by their hashes', async () => { + const changed = await differences( + await reference((doc) => { + for (const place of doc.places) { + if (!place.full || place.section.event.stId !== R2.stId) continue; + place.section.edits = [...place.section.edits].reverse(); + const [first] = place.section.references; + if (first) place.section.references = [{ ...first, authors: [PEOPLE[0]] }]; + } + }) + ); + assert.deepEqual(changed, [ + RESOURCE, + `section references text ${R2.stId}`, + `section edits order ${R2.stId}`, + ]); + }); + }); + + test("the contents list's entries, their order and their targets", async () => { + const entries = (sampleDocument().contents ?? []).map((c) => ({ + title: c.title, + target: c.target, + })); + const fewer = await differences(await reference(undefined, { contents: entries.slice(1) })); + assert.deepEqual(fewer, [RESOURCE, 'contents extra']); + const [first, second, ...rest] = entries; + const swapped = await differences( + await reference(undefined, { contents: first && second ? [second, first, ...rest] : [] }) + ); + assert.deepEqual(swapped, [RESOURCE, 'contents order']); + const retargeted = await differences( + await reference(undefined, { + contents: entries.map((e) => + e.target === `s-${R1.stId}` ? { ...e, target: `s-${S.stId}` } : e + ), + }) + ); + assert.deepEqual(retargeted, [RESOURCE, 'contents target']); + // a target not read, on either side, is a difference of its own + const unread = await differences( + await reference(undefined, { + contents: entries.map((e) => + e.target === `s-${R1.stId}` ? { ...e, target: 'nowhere' } : e + ), + }) + ); + assert.deepEqual(unread, [RESOURCE, 'contents missing-target']); + }); + + test('the figures: one missing, and one of another kind', async () => { + const missing = await differences(await reference(undefined, { noFigure: [R1.stId] })); + assert.deepEqual(missing, [RESOURCE, `figure extra ${R1.stId}`]); + // pdf-export's own on both sides, where each figure's kind is read + const doc = sampleDocument(); + for (const place of doc.places) { + if (place.full && place.section.event.stId === S.stId) { + place.section.figure = { kind: 'diagram', stId: S.stId }; + } + } + const other = await reduced(await printOurs(printer, doc, analysis)); + assert.deepEqual(compareReduced('x', oursReduced, other).map(brief), [`figure kind ${S.stId}`]); + }); + + test('a short entry is counted, apart from the section before it', async () => { + const doc = sampleDocument(); + doc.places.push({ full: false, anchor: `s-${R1.stId}`, event: R1, path: [], depth: 1 }); + const other = await reduced(await printOurs(printer, doc, analysis)); + assert.deepEqual(compareReduced('x', oursReduced, other).map(brief), ['short-entries count']); + }); + + describe('the command', () => { + let refFolder: string; + let newFolder: string; + let expected: string; + const run = async (args: string[]) => { + let out = ''; + let err = ''; + const code = await compareCommand(args, { + out: (text) => (out += text), + err: (text) => (err += text), + }); + return { code, out, err }; + }; + + before(async () => { + refFolder = join(folder, 'reference'); + newFolder = join(folder, 'new'); + mkdirSync(refFolder); + mkdirSync(newFolder); + // names that differ, a section moved, and a file on each side alone + const ref = await reference(moveFirstChildLast, { names: [...PEOPLE] }); + writeFileSync(join(refFolder, 'Alpha signalling pathway.pdf'), ref); + writeFileSync(join(newFolder, 'Alpha signalling pathway.pdf'), ours); + writeFileSync(join(refFolder, 'Only today.pdf'), ref); + writeFileSync(join(newFolder, 'Only new.pdf'), ours); + expected = join(folder, 'expected.json'); + }); + + test('folders file by file, a file on one side only a difference; exit 1', async () => { + const { code, out, err } = await run([refFolder, newFolder]); + assert.equal(code, 1); + assert.equal(err, ''); + const report = JSON.parse(out) as { differences: Difference[] }; + assert.deepEqual( + report.differences.map((d) => `${d.file}: ${brief(d)}`), + [ + 'Only today.pdf: file missing', + 'Only new.pdf: file extra', + 'Alpha signalling pathway.pdf: cover names count', + 'Alpha signalling pathway.pdf: analysis resource missing', + `Alpha signalling pathway.pdf: sections order ${R1.stId}`, + ] + ); + }); + + test('--expected: recorded differences pass, with their reasons; exit 0 only if all are', async () => { + const records = [ + { part: 'file', kind: 'missing', reason: 'a sample' }, + { part: 'file', kind: 'extra', reason: 'a sample' }, + { part: 'cover', field: 'names', kind: 'count', reason: 'a sample' }, + { part: 'analysis', field: 'resource', kind: 'missing', reason: 'not printed yet' }, + { part: 'sections', kind: 'order', reason: "today's lists are in the database's order" }, + { part: 'contents', kind: 'order', reason: 'matches nothing' }, + ]; + writeFileSync(expected, JSON.stringify({ differences: records })); + const all = await run([refFolder, newFolder, '--expected', expected]); + assert.equal(all.code, 0); + const report = JSON.parse(all.out) as { + counts: { differences: number; recorded: number; unrecorded: number }; + differences: Difference[]; + unmatched: number[]; + }; + assert.deepEqual(report.counts, { differences: 5, recorded: 5, unrecorded: 0 }); + assert.deepEqual(report.unmatched, [5]); + assert.ok(report.differences.every((d) => typeof d.recorded === 'string')); + // the order's record left out: that difference alone fails the run + writeFileSync(expected, JSON.stringify({ differences: records.slice(0, 4) })); + const some = await run([refFolder, newFolder, `--expected=${expected}`]); + assert.equal(some.code, 1); + const unrecorded = (JSON.parse(some.out) as { differences: Difference[] }).differences.filter( + (d) => d.recorded === undefined + ); + assert.deepEqual(unrecorded.map(brief), [`sections order ${R1.stId}`]); + }); + + test('a file that cannot be read is named and is a difference; the others are compared', async () => { + const refs = join(folder, 'broken-reference'); + const news = join(folder, 'broken-new'); + mkdirSync(refs); + mkdirSync(news); + writeFileSync(join(refs, 'Broken.pdf'), 'not a PDF: Placeholder, A'); + writeFileSync(join(news, 'Broken.pdf'), ours); + writeFileSync(join(refs, 'Whole.pdf'), ours); + writeFileSync(join(news, 'Whole.pdf'), ours); + const { code, out, err } = await run([refs, news]); + assert.equal(code, 1); + // the error's kind alone, never its message + assert.match(err, /^Not read: .*Broken\.pdf \([A-Za-z]+(?: [A-Z_]+)?\)\n$/); + assertNoName(out + err, 'the report of an unreadable file'); + const report = JSON.parse(out) as { differences: Difference[] }; + assert.deepEqual( + report.differences.map((d) => `${d.file}: ${brief(d)}`), + ['Broken.pdf: file reference unreadable'] + ); + }); + + test('no name is printed, in the report or its errors', async () => { + const differ = await run([refFolder, newFolder]); + assertNoName(differ.out + differ.err, 'the report'); + const pdfs = await run([ + join(refFolder, 'Alpha signalling pathway.pdf'), + join(newFolder, 'Alpha signalling pathway.pdf'), + ]); + assert.equal(pdfs.code, 1); + assertNoName(pdfs.out + pdfs.err, 'the report of two PDFs'); + // a reference whose names are all others: still only a count and a hash + const report = JSON.parse(pdfs.out) as { differences: Difference[] }; + assert.ok(report.differences.some((d) => d.field === 'names')); + }); + + test('bad arguments: exit 2, saying what is needed', async () => { + assert.equal((await run([refFolder])).code, 2); + const mixed = await run([refFolder, join(newFolder, 'Only new.pdf')]); + assert.equal(mixed.code, 2); + assert.match(mixed.err, /both be PDFs or both be folders/); + assert.equal((await run([refFolder, newFolder, '--depth', '1'])).code, 2); + writeFileSync(expected, JSON.stringify({ differences: [{ part: 'cover' }] })); + const unrecorded = await run([refFolder, newFolder, '--expected', expected]); + assert.equal(unrecorded.code, 2); + assert.match(unrecorded.err, /needs a part, a kind and a reason/); + writeFileSync( + expected, + JSON.stringify({ + differences: [{ part: 'cover', kind: 'count', reason: 'x', feild: 'names' }], + }) + ); + const misspelt = await run([refFolder, newFolder, '--expected', expected]); + assert.equal(misspelt.code, 2); + assert.match(misspelt.err, /keys not known: feild/); + assert.equal((await run([refFolder, newFolder, '--reference-margin', 'wide'])).code, 2); + }); + + test("--reference-margin: the reference's margin band, in points", async () => { + const file = join(newFolder, 'Alpha signalling pathway.pdf'); + assert.equal((await run([file, file])).code, 0); + // a band 300 pt deep leaves the reference little of its text + const deep = await run([file, file, '--reference-margin', '300']); + assert.equal(deep.code, 1); + }); + + test('it is a command of the command line', () => { + const result = spawnSync( + process.execPath, + [fileURLToPath(new URL('../../src/cli.ts', import.meta.url)), 'compare'], + { encoding: 'utf8', env: { PATH: process.env['PATH'] ?? '' } } + ); + assert.equal(result.status, 2); + assert.match(result.stderr, /Two paths are needed/); + }); + }); +}); diff --git a/test/compare/layouts.test.ts b/test/compare/layouts.test.ts new file mode 100644 index 0000000..86a0981 --- /dev/null +++ b/test/compare/layouts.test.ts @@ -0,0 +1,244 @@ +// FR-012's comparison on layouts other than the sample's (spec 001 T035, from #165's review): +// covers whose title a figure takes in, an analysis part followed by a section the reducer +// cannot read, and a reference laid out unlike the sample's, so the reducer is not tried +// only on its own author's shape. Names are planted in each, and none may be printed. +import assert from 'node:assert/strict'; +import { after, before, describe, test } from 'node:test'; + +import { compareReduced } from '../../src/compare/compare.ts'; +import { Output, type Difference } from '../../src/compare/output.ts'; +import { readLayout } from '../../src/compare/read.ts'; +import { reduce, type Reduced } from '../../src/compare/reduce.ts'; +import { plainText } from '../../src/layout/html.ts'; +import { reference as referenceText } from '../../src/layout/section.ts'; +import { Printer } from '../../src/print/print.ts'; +import { PEOPLE, png, printOurs, sampleAnalysis, sampleDocument } from './samples.ts'; + +const OTHERS = ['Quixote, Zebulon', 'Marchbanks, Ottoline', 'Fairweather, Ignatius'] as const; +const NAME_PARTS = [...PEOPLE, ...OTHERS] + .flatMap((name) => name.split(/[\s,]+/)) + .filter((w) => w.length > 2); + +function assertNoName(text: string, where: string): void { + for (const part of NAME_PARTS) assert.ok(!text.includes(part), `a name was printed in ${where}`); +} + +const brief = (d: Difference) => + [d.part, d.field, d.kind, ...(d.events ?? [])].filter((w) => w !== undefined).join(' '); + +const t = plainText; + +const page = ( + body: string, + css = '' +) => `${body}`; + +describe('compare on other layouts', { timeout: 300_000 }, () => { + let printer: Printer; + const output = new Output(); + const reduced = async (html: string): Promise => + reduce(await readLayout(await printer.print(html, false))); + /** The two sides' differences as printed, and as the tests name them. */ + const compare = (file: string, a: Reduced, b: Reduced) => { + const found = compareReduced(file, a, b); + return { + printed: JSON.stringify(found.map((d) => output.difference(d))), + brief: found.map(brief), + }; + }; + + before(async () => { + printer = await Printer.start(); + }); + after(async () => { + await printer.close(); + }); + + describe('a cover whose title a figure takes in: the title is never printed, the names never', () => { + const cover = (top: string, names: readonly string[]) => + page(`${top} +

Authors

${names.map(t).join('; ')}

+

${t('Ontario Institute for Cancer Research')}

4 November 2026

+

Introduction

Release 98. It holds 2 pathways and 2 reactions.

`); + + test('a title in a coloured band', async () => { + const band = `

Alpha signalling pathway

`; + const a = await reduced(cover(band, PEOPLE.slice(0, 2))); + const b = await reduced(cover(band, OTHERS)); + const { printed, brief: found } = compare('Alpha signalling pathway.pdf', a, b); + assertNoName(printed, 'the report'); + assert.ok(found.includes('cover names count'), 'the names are compared'); + for (const d of JSON.parse(printed) as Difference[]) { + if (d.field === 'title') assert.ok(['same', 'differs'].includes(String(d.reference))); + } + }); + + test('a logo, a 16 pt title and a diagram close together', async () => { + const close = ` +

Alpha signalling pathway

+`; + const a = await reduced(cover(close, PEOPLE.slice(0, 3))); + const b = await reduced(cover(close, OTHERS.slice(0, 2))); + const { printed } = compare('Alpha signalling pathway.pdf', a, b); + assertNoName(printed, 'the report'); + }); + }); + + describe('an analysis part followed by text that is not its own', () => { + const analysis = (heading: boolean, label: string, people: readonly string[]) => + page(`

Alpha signalling pathway

+

Introduction

Release 98.

+

Analysis

+

Token: MjAyNjExMDQxMjAwMDBfMQ%3D%3D

Identifiers found: 3 of 5

+${heading ? '

Alpha signalling pathway

' : '

Alpha signalling pathway

'} +${label ? `

${label}

` : ''} +

Species of ${t(people[0] ?? '')} were studied, written for these tests.

+

Type ${t(people[1] ?? '')} cells hold the sample, written for these tests.

+

Projection: ${t(people[2] ?? '')}

Resource: ${t(people[0] ?? '')}

`); + + for (const [name, heading, label] of [ + ['a section labelled Identifier:, under its heading', true, 'Identifier: R-HSA-9000001'], + ['a section labelled Identifier:, with no heading', false, 'Identifier: R-HSA-9000001'], + ['a section under its heading, with no stable identifier', true, ''], + ['text with neither a heading nor a stable identifier', false, ''], + ] as const) { + test(name, async () => { + const a = await reduced(analysis(heading, label, PEOPLE.slice(0, 3))); + const b = await reduced(analysis(heading, label, OTHERS)); + const { printed, brief: found } = compare('x.pdf', a, b); + assertNoName(printed, 'the report'); + assert.equal( + a.analysis?.['identifiersTotal'] && 'value' in a.analysis['identifiersTotal'] + ? a.analysis['identifiersTotal'].value + : null, + 5 + ); + // what is read past the part, or not of its shape, is never a value + for (const key of ['species', 'type', 'projection', 'resource']) { + const value = a.analysis?.[key]; + assert.ok(value === undefined || !('value' in value), `${key} kept as a value`); + } + assert.ok(found.every((d) => !d.startsWith('analysis token'))); + // the part ends at the section's heading or its stable identifier: nothing after them + // is read as a property + if (heading || label) { + assert.deepEqual(Object.keys(a.analysis ?? {}).sort(), [ + 'identifiersFound', + 'identifiersTotal', + 'token', + ]); + } + }); + } + }); + + test('a table with a shaded header and hairline borders is read, not taken as a figure', async () => { + const rows = Array.from( + { length: 8 }, + (_, i) => `P0000${String(i)}P0000${String(i)}UniProt` + ).join(''); + const read = await reduced( + page( + `

Alpha signalling pathway

+

Beta binds gamma

+

Stable identifier: R-HSA-9000002

A summation, written for these tests.

+

Entities found

+${rows}
IdentifierMaps toResource
+
`, + 'td, th { border: 0.5pt solid #888888; padding: 2pt 4pt; text-align: left }' + ) + ); + assert.equal(read.sections.length, 1); + const section = read.sections[0]; + assert.equal(section?.figure, 'none'); + assert.equal(section.parts.find((p) => p.part === 'entities')?.items.length, 8); + }); + + test('a reference laid out unlike the sample: facts in a table, contents first, vector figures, numbered references in brackets', async () => { + const doc = sampleDocument(); + const props = sampleAnalysis(); + const ours = reduce(await readLayout(await printOurs(printer, doc, props))); + const sections = doc.places.flatMap((p) => (p.full ? [p.section] : [])); + const svg = (label: string) => + `${t(label)}`; + const row = (label: string, value: string) => `${t(label)}:${value}`; + const body = [ + `

${t(doc.cover.title)}

`, + `

${svg('cover')}

`, + `

Authored, edited, reviewed and revised by

${doc.cover.names.map(t).join('; ')}

`, + '

European Bioinformatics Institute

An excerpt of the book.

', + '

Introduction

Reactome release 98, holding 2 pathways and 2 reactions.

', + `

Analysis properties

${[ + row('Token', props.properties.token), + row('Type', props.properties.type), + row('Species', props.properties.species), + row('Projected to human', 'yes'), + row('Interactors included', 'no'), + row('Identifiers found', '3 of 5'), + row('Pathways hit', '4'), + ].join('')}
`, + `

Contents

${( + doc.contents ?? [] + ) + .map( + (c, i) => + `` + ) + .join('')}
${t(c.title)}${String(i + 3)}
`, + ...sections.map((s) => { + const e = s.event; + const facts = [ + s.path.length > 0 + ? row( + 'Location', + s.path.map((p) => `${t(p.name)}`).join(' » ') + ) + : '', + row('Stable identifier', t(e.stId)), + e.kind === 'reaction' && e.category ? row('Type', t(e.category)) : '', + row('Compartments', e.compartments.map((c) => t(c.name)).join(', ')), + s.diseases.length > 0 ? row('Diseases', s.diseases.map(t).join(', ')) : '', + s.inferredFrom.length > 0 + ? row( + 'Inferred from', + s.inferredFrom.map((i) => `${t(i.name)} (${t(i.speciesName)})`).join('; ') + ) + : '', + ].join(''); + const found = props.found.get(e.stId); + return `

${t(e.name)}

+${facts}
+${s.figure ? `

${svg(e.name)}

` : ''} +${s.summations.join('\n')} +${e.kind === 'reaction' && e.before.length > 0 ? `

Preceded by: ${e.before.map((b) => t(b.name)).join('; ')}

` : ''} +${e.kind === 'reaction' && e.after.length > 0 ? `

Followed by: ${e.after.map((b) => t(b.name)).join('; ')}

` : ''} +

References

${s.references.map((r, i) => `

[${String(i + 1)}] ${referenceText(r)}

`).join('')} +

Edit history

${s.edits.map((r) => `

${t(r.date)} ${t(r.kinds.map((k) => k.charAt(0).toUpperCase() + k.slice(1)).join(', '))} ${t(r.authors.join('; '))}

`).join('')} +${found ? `

Entities found

${found.entities.map((f) => `

${t(f.identifier)} ${t(f.mapsTo.join(', '))} ${t(f.resource)}

`).join('')}` : ''} +
`; + }), + ]; + const other = reduce( + await readLayout( + await printer.print( + page( + body.join('\n'), + 'th { text-align: left; font-weight: normal; padding-right: 10pt } @page { @bottom-right { content: counter(page) } }' + ), + false + ) + ) + ); + const { printed, brief: found } = compare('x.pdf', other, ours); + assertNoName(printed, 'the report'); + // the same content, read alike: the figures' kinds, which this layout's figures carry no + // link to say, match any kind; its analysis has no resource, as pdf-export's has none + assert.deepEqual(found, []); + }); +}); diff --git a/test/compare/samples.ts b/test/compare/samples.ts new file mode 100644 index 0000000..401eb50 --- /dev/null +++ b/test/compare/samples.ts @@ -0,0 +1,405 @@ +// Samples for FR-012's comparison's tests (spec 001 T035): one small document, printed +// both by pdf-export's own layout and as a reference in the shape of `event-pdf`'s files +// (the spec's Background and research R2: the contents list after the sections, its parts +// written `Label: value`, its figures as images, the cover's names with U+2060 between +// their characters and no-break spaces between their words). The reference's wording is +// written here; none is today's. Every name is a placeholder (FR-013): real `event-pdf` +// files are never used in tests. +import { crc32, deflateSync } from 'node:zlib'; + +import type { EventRef, Reference } from '../../src/graph/queries.ts'; +import type { FigureSvgs } from '../../src/layout/figure.ts'; +import { figureKey } from '../../src/layout/figure.ts'; +import { attribute, plainText } from '../../src/layout/html.ts'; +import { layoutSingle, pageStyles } from '../../src/layout/document.ts'; +import { reference } from '../../src/layout/section.ts'; +import { INSTITUTIONS } from '../../src/layout/text.ts'; +import type { + AnalysisProperties, + Document, + Figure, + Found, + Section, +} from '../../src/model/types.ts'; +import type { Printer } from '../../src/print/print.ts'; + +/** The placeholder people: authors of edits and of references. */ +export const PEOPLE = ['Placeholder, A', 'Example, B', 'Sample, C D', 'Dummy, E'] as const; + +const SITE = 'https://site.example'; + +const ref = (dbId: number, stId: string, name: string): EventRef => ({ dbId, stId, name }); + +const P = ref(1, 'R-HSA-9000001', 'Alpha signalling pathway'); +const R1 = ref(2, 'R-HSA-9000002', 'Beta binds gamma'); +const S = ref(3, 'R-HSA-9000003', 'Delta subpathway'); +const R2 = ref(4, 'R-HSA-9000004', 'Epsilon is phosphorylated'); + +function literature(dbId: number, title: string, authors: string[]): Reference { + return { + dbId, + kind: 'article', + title, + chapterTitle: null, + journal: 'Journal of Examples', + volume: 12, + pages: '1-9', + year: 2001, + pubMedId: 1000 + dbId, + isbn: null, + url: null, + publisher: null, + authors, + moreAuthors: false, + }; +} + +function section( + event: EventRef, + kind: 'pathway' | 'reaction', + depth: number, + path: EventRef[], + figure: Figure | null, + more: Partial
& { category?: string; before?: EventRef[]; after?: EventRef[] } = {} +): Section { + const { category = null, before = [], after = [], ...rest } = more; + return { + anchor: `s-${event.stId}`, + event: { + ...event, + kind, + schemaClass: kind === 'pathway' ? 'Pathway' : 'Reaction', + speciesName: 'Homo sapiens', + category, + hasDiagram: figure?.kind === 'diagram', + hasEHLD: false, + isInferred: false, + isInDisease: false, + species: { dbId: 48887, name: 'Homo sapiens' }, + compartments: [{ dbId: 70101, name: 'cytosol' }], + before, + after, + }, + path, + figure, + diseases: [], + inferredFrom: [], + summations: [`

${plainText(event.name)} is a sample event, written for these tests.

`], + references: [literature(event.dbId * 10, `A study of ${event.name}`, [PEOPLE[2], PEOPLE[3]])], + edits: [ + { date: '2020-01-01', kinds: ['authored'], authors: [PEOPLE[0]] }, + { date: '2021-02-03', kinds: ['reviewed'], authors: [PEOPLE[1]] }, + ], + depth, + ...rest, + }; +} + +/** The sample: a pathway at depth 1 with two reactions and a subpathway, with an analysis. */ +export function sampleDocument(): Document { + const diagram: Figure = { kind: 'diagram', stId: P.stId }; + const sections = [ + section(P, 'pathway', 0, [], diagram, { + references: [ + literature(11, 'A first study', [PEOPLE[2]]), + literature(12, 'A second study', [PEOPLE[3], PEOPLE[2]]), + ], + }), + section( + R1, + 'reaction', + 1, + [P], + { kind: 'reaction', stId: R1.stId }, + { + category: 'binding', + after: [R2], + inferredFrom: [ + { ...ref(20, 'R-MMU-9000020', 'Beta binds gamma'), speciesName: 'Mus musculus' }, + ], + } + ), + section( + S, + 'pathway', + 1, + [P], + { kind: 'ancestor', stId: P.stId, highlight: [R1.stId] }, + { + diseases: ['influenza'], + // a summation whose first word is a part's label, which only labels its part where it comes + summations: ['

Type 2 cells hold the sample subpathway, written for these tests.

'], + } + ), + section( + R2, + 'reaction', + 1, + [P], + { kind: 'reaction', stId: R2.stId }, + { + category: 'transition', + before: [R1], + } + ), + ]; + for (const s of sections) { + if (s.event.stId === R2.stId) { + s.event.compartments = [ + { dbId: 70101, name: 'cytosol' }, + { dbId: 7660, name: 'nucleoplasm' }, + ]; + } + } + return { + kind: 'single', + release: { version: 98, pathways: 2, reactions: 2 }, + cover: { + title: P.name, + release: null, + figure: diagram, + excerpt: true, + names: [PEOPLE[0], PEOPLE[1]], + }, + places: sections.map((s) => ({ full: true as const, section: s })), + contents: [ + { title: 'Introduction', target: 'introduction', depth: 0 }, + { title: 'Analysis properties', target: 'analysis', depth: 0 }, + ...sections.map((s) => ({ title: s.event.name, target: s.anchor, depth: s.depth })), + { title: 'Contents', target: 'contents', depth: 0 }, + ], + }; +} + +export function sampleAnalysis(): { properties: AnalysisProperties; found: Map } { + return { + properties: { + token: 'MjAyNjExMDQxMjAwMDBfMQ%3D%3D', + type: 'OVERREPRESENTATION', + species: 'Homo sapiens', + projection: true, + interactors: false, + identifiersFound: 3, + identifiersTotal: 5, + pathwaysHit: 4, + }, + found: new Map([ + [ + R1.stId, + { + entities: [{ identifier: 'P00001', mapsTo: ['P00001'], resource: 'UniProt' }], + interactors: [], + }, + ], + ]), + }; +} + +/** A figure: a few shapes and a label. */ +function svg(label: string): string { + return `${plainText(label)}`; +} + +/** Each section's figure, and the cover's, as pdf-export's layout reads them. */ +export function sampleFigures(doc: Document): FigureSvgs { + const figures = new Map(); + for (const place of doc.places) { + if (place.full && place.section.figure) { + figures.set(figureKey(place.section.figure), svg(place.section.event.name)); + } + } + if (doc.cover.figure) figures.set(figureKey(doc.cover.figure), svg(doc.cover.title)); + return figures; +} + +/** pdf-export's own single-event PDF of a document. */ +export async function printOurs( + printer: Printer, + doc: Document, + analysis: ReturnType | null +): Promise { + const { html } = layoutSingle(doc, { + site: SITE, + date: new Date('2026-11-04T12:00:00Z'), + figures: sampleFigures(doc), + styles: pageStyles(), + analysis, + }); + return printer.single(html); +} + +/** A PNG of one colour, as `event-pdf`'s raster figures stand in. */ +export function png(width: number, height: number): string { + const chunk = (type: string, data: Buffer) => { + const length = Buffer.alloc(4); + length.writeUInt32BE(data.length); + const body = Buffer.concat([Buffer.from(type, 'latin1'), data]); + const crc = Buffer.alloc(4); + crc.writeUInt32BE(crc32(body)); + return Buffer.concat([length, body, crc]); + }; + const header = Buffer.alloc(13); + header.writeUInt32BE(width, 0); + header.writeUInt32BE(height, 4); + header.set([8, 2, 0, 0, 0], 8); + const row = Buffer.concat([Buffer.from([0]), Buffer.alloc(width * 3, 0x9c)]); + const pixels = deflateSync(Buffer.concat(Array.from({ length: height }, () => row))); + const file = Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk('IHDR', header), + chunk('IDAT', pixels), + chunk('IEND', Buffer.alloc(0)), + ]); + return `data:image/png;base64,${file.toString('base64')}`; +} + +/** A name as the reference's cover writes it: U+2060 between its characters, no-break spaces between its words. */ +export const joined = (name: string): string => + Array.from(name.replace(/ /g, '\u00a0')).join('\u2060'); + +export interface ReferenceOptions { + analysis?: ReturnType | null; + /** The cover's names, if not the document's. */ + names?: readonly string[]; + /** The release line's number and the counts, if not the document's. */ + release?: number; + /** Sections drawn with no figure, by stable identifier. */ + noFigure?: readonly string[]; + /** The contents list's entries, if not the document's. */ + contents?: { title: string; target: string }[]; + /** The analysis's properties as written, if not those given. */ + properties?: [string, string][]; +} + +/** A reference PDF in `event-pdf`'s shape, of a document. */ +export async function printReference( + printer: Printer, + doc: Document, + options: ReferenceOptions = {} +): Promise { + const t = (s: string) => plainText(s); + const a = (href: string, s: string) => `${t(s)}`; + const names = options.names ?? doc.cover.names; + const parts: string[] = []; + // the cover: a raster logo, the title, a raster diagram, the names, then the rest + parts.push(`
+ +

${t(doc.cover.title)}

+${doc.cover.figure ? `` : ''} +

Authors and editors

+

${names.map(joined).join('; ')}

+

${INSTITUTIONS.map(t).join('
')}

+

Shared under a Creative Commons licence.

+

4 November 2026

+${doc.cover.excerpt ? '

An excerpt of the whole book.

' : ''} +
`); + const { release } = doc; + parts.push(`

Introduction

+

Sample introduction text, written for these tests.

+

Release ${String(options.release ?? release.version)}. It holds ${String(release.pathways)} pathways and ${String(release.reactions)} reactions.

`); + const analysis = options.analysis ?? null; + if (analysis) { + const p = analysis.properties; + const rows: [string, string][] = options.properties ?? [ + ['Token', p.token], + ['Analysis type', p.type], + ['Species', p.species], + ['Projection', String(p.projection)], + ['Interactors', String(p.interactors)], + ['Identifiers found', `${String(p.identifiersFound)} of ${String(p.identifiersTotal)}`], + ['Pathways found', String(p.pathwaysHit)], + ['Resource', 'TOTAL'], + ]; + parts.push(`

Analysis properties

+${rows.map(([k, v]) => `

${t(k)}: ${t(v)}

`).join('\n')} +

Explanatory text on the analysis, written for these tests.

`); + } + const sections: string[] = []; + for (const place of doc.places) { + if (!place.full) continue; + const s = place.section; + const e = s.event; + const lines: string[] = []; + lines.push( + `

${a(`${SITE}/content/detail/${e.stId}`, e.name)}

` + ); + if (s.path.length > 0) { + lines.push(`

Location: ${s.path.map((p) => a(`#s-${p.stId}`, p.name)).join(' > ')}

`); + } + lines.push(`

Stable Identifier: ${t(e.stId)}

`); + if (e.kind === 'reaction' && e.category) lines.push(`

Type: ${t(e.category)}

`); + if (e.compartments.length > 0) { + lines.push(`

Compartment: ${e.compartments.map((c) => t(c.name)).join(', ')}

`); + } + if (s.diseases.length > 0) lines.push(`

Disease: ${s.diseases.map(t).join(', ')}

`); + if (s.inferredFrom.length > 0) { + lines.push( + `

Inferred from: ${s.inferredFrom.map((i) => `${t(i.name)} (${t(i.speciesName)})`).join('; ')}

` + ); + } + if (s.figure && !options.noFigure?.includes(e.stId)) { + lines.push(`

`); + } + lines.push(...s.summations); + if (e.kind === 'reaction') { + if (e.before.length > 0) { + lines.push( + `

Preceding events: ${e.before.map((b) => a(`#s-${b.stId}`, b.name)).join('; ')}

` + ); + } + if (e.after.length > 0) { + lines.push( + `

Following events: ${e.after.map((b) => a(`#s-${b.stId}`, b.name)).join('; ')}

` + ); + } + } + if (s.references.length > 0) { + lines.push( + `

Literature References

    ${s.references.map((r) => `
  1. ${reference(r)}
    2. `).join('')}
` + ); + } + if (s.edits.length > 0) { + lines.push( + `

Edit history

${s.edits + .map( + (row) => + `` + ) + .join('')}
DateActionAuthor
${t(row.date)}${t(row.kinds.map((k) => k.charAt(0).toUpperCase() + k.slice(1)).join(', '))}${t(row.authors.join('; '))}
` + ); + } + const found = analysis?.found.get(e.stId); + if (found && found.entities.length > 0) { + lines.push( + `

Entities found

${found.entities + .map( + (r) => + `` + ) + .join('')}
InputIdentifiersResource
${t(r.identifier)}${t(r.mapsTo.join(', '))}${t(r.resource)}
` + ); + } + sections.push(`
${lines.join('\n')}
`); + } + parts.push(...sections); + const entries = + options.contents ?? (doc.contents ?? []).map((c) => ({ title: c.title, target: c.target })); + if (entries.length > 0) { + parts.push(`

Table of contents

+${entries.map((c, i) => `

${a(`#${c.target}`, c.title)} ........ ${String(i + 2)}

`).join('\n')}
`); + } + const html = ` +${parts.join('\n')} +`; + return printer.print(html, false); +} + +export { P, R1, R2, S }; diff --git a/test/compare/units.test.ts b/test/compare/units.test.ts new file mode 100644 index 0000000..741ff41 --- /dev/null +++ b/test/compare/units.test.ts @@ -0,0 +1,627 @@ +// FR-012's comparison's parts that need no browser (spec 001 T035, from #165's review): the +// one printer every reported value passes through, tried with names planted everywhere in +// random layouts; the sections' order by their longest common subsequence; the constants a +// layout is read by, pinned; and the reduction's time on a large book. Layouts here are +// built directly, as pdf.js's reading would give them. +import assert from 'node:assert/strict'; +import { describe, test } from 'node:test'; + +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { PDFDocument, PDFOperator, type PDFOperatorNames } from '@cantoo/pdf-lib'; + +import { + applyRecorded, + commonSubsequence, + compareCommand, + compareReduced, + errorKind, + withConsoleHeld, +} from '../../src/compare/compare.ts'; +import { + Output, + SPECIES, + ANALYSIS_TYPES, + RESOURCES, + type Difference, +} from '../../src/compare/output.ts'; +import { + figureBoxes, + readLayout, + type Box, + type Layout, + type Line, + type Link, +} from '../../src/compare/read.ts'; +import { hash, reduce, type Reduced, type ReducedSection } from '../../src/compare/reduce.ts'; +import { CONTRIBUTORS } from '../../src/layout/text.ts'; +import { PEOPLE } from './samples.ts'; + +/** Placeholder names, and more of them, to plant. */ +const NAMES = [...PEOPLE, 'Quixote, Zebulon', 'Marchbanks, Ottoline', 'Fairweather, Ignatius']; +const NAME_PARTS = NAMES.flatMap((name) => name.split(/[\s,]+/)).filter((w) => w.length > 2); + +/** Every string a report holds, at any depth. */ +function strings(value: unknown, out: string[] = []): string[] { + if (typeof value === 'string') out.push(value); + else if (Array.isArray(value)) for (const item of value) strings(item, out); + else if (value !== null && typeof value === 'object') { + for (const [key, item] of Object.entries(value)) { + out.push(key); + strings(item, out); + } + } + return out; +} + +/** The strings a printed difference may hold: its keys, names of its own, and the shapes. */ +const ALLOWED = new Set([ + ...['file', 'part', 'field', 'kind', 'count', 'events', 'reference', 'new', 'recorded'], + ...['hash', 'length', 'value', 'same', 'differs', 'yes', 'no'], + ...['cover', 'introduction', 'analysis', 'sections', 'section', 'figure', 'contents'], + ...['short-entries', 'title', 'diagram', 'excerpt', 'names', 'release', 'pathways'], + ...['reactions', 'token', 'type', 'species', 'projection', 'interactors'], + ...['identifiersFound', 'identifiersTotal', 'resource', 'missing', 'extra', 'text'], + ...['order', 'target', 'missing-target', 'unreadable', 'name', 'location', 'stId'], + ...['compartments', 'diseases', 'inferredFrom', 'summation', 'before', 'after'], + ...['references', 'edits', 'entities', 'value'], + ...SPECIES, + ...ANALYSIS_TYPES, + ...RESOURCES, +]); + +/** Fails, naming only where, if any string is neither allowed nor of an allowed shape. */ +function assertPrintable(printed: unknown, files: readonly string[], token: string): void { + for (const text of strings(printed)) { + const ok = + ALLOWED.has(text) || + files.includes(text) || + text === token || + /^R-[A-Z]{3}-\d+$/.test(text) || + /^[0-9a-f]{16}$/.test(text); + assert.ok(ok, 'a string of no allowed shape was printed'); + for (const part of NAME_PARTS) assert.ok(!text.includes(part), 'a name was printed'); + } +} + +/** A seeded random number source (mulberry32), so a failure can be run again. */ +function random(seed: number): () => number { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +const TOKEN = 'MjAyNjExMDQxMjAwMDBfMQ%3D%3D'; + +/** What a line may begin with: every label the reducer reads, and others. */ +const LEADS = [ + 'Stable identifier R-HSA-1234', + 'Stable Identifier: R-HSA-77', + 'Identifier: R-HSA-5', + 'Location', + 'Type', + 'Type:', + 'Species', + 'Species:', + 'Token', + 'Token:', + 'Analysis type', + 'Projection:', + 'Projected to human', + 'Interactors included', + 'Identifiers found', + 'Pathways hit', + 'Resource:', + 'Compartments', + 'Diseases', + 'Inferred from', + 'Preceded by', + 'Followed by', + 'Literature references', + 'Edit history', + 'Entities found', + 'Introduction', + 'Analysis properties', + 'Contents', + 'Table of contents', + 'Authors', + CONTRIBUTORS, + 'Release 98. It holds 3 pathways and 4 reactions.', + '1.', + '2020-01-01', + 'This document is an excerpt', + '', +]; + +/** A random layout: names planted after every kind of lead, at every size, everywhere. */ +function plantedLayout(next: () => number): Layout { + const pick = (list: readonly T[]): T => list[Math.floor(next() * list.length)] as T; + const pages = 1 + Math.floor(next() * 5); + const lines: Line[] = []; + const links: Link[] = []; + const figures: Box[] = []; + for (let page = 0; page < pages; page++) { + let y = 780; + while (y > 60) { + const size = pick([9, 10.5, 10.5, 10.5, 12, 14, 16, 18, 24, 28]); + const lead = pick(LEADS); + const tail = pick([ + pick(NAMES), + `${pick(NAMES)}; ${pick(NAMES)}`, + `R-HSA-${String(Math.floor(next() * 99))}`, + TOKEN, + '12', + 'yes', + `${pick(NAMES)} 4`, + ]); + const text = next() < 0.2 ? Array.from(tail).join('\u2060') : `${lead} ${tail}`; + lines.push({ page, y, x: 57 + Math.floor(next() * 3) * 12, size, text }); + if (next() < 0.15) { + links.push({ + box: { page, x1: 57, y1: y - 2, x2: 300, y2: y + size }, + uri: null, + landing: { page: Math.floor(next() * pages), y: 300 + next() * 500 }, + }); + } + if (next() < 0.05) { + figures.push({ page, x1: 100, y1: y - 120, x2: 400, y2: y - 10 }); + y -= 120; + } + y -= size + 4 + next() * 10; + } + } + return { + pages: Array.from({ length: pages }, () => ({ width: 595, height: 842 })), + lines, + links, + figures, + }; +} + +describe('what compare prints', () => { + test('only values of their shapes: anything else as a hash and a length', () => { + const output = new Output('fixed'); + assert.equal(output.value('analysis species', 'Homosapiens'), 'Homo sapiens'); + assert.equal(output.value('analysis type', 'overrepresentation'), 'OVERREPRESENTATION'); + assert.equal(output.value('analysis token', TOKEN), TOKEN); + assert.equal(output.value('introduction release', 98), 98); + assert.equal(output.value('analysis projection', true), 'yes'); + assert.equal(output.value('cover title', 'differs'), 'differs'); + for (const [key, value] of [ + ['analysis species', 'Placeholder, A'], + ['analysis token', 'Placeholder'], + ['analysis type', 'Type Example, B'], + ['cover title', 'Sample, C D'], + ['introduction release', 'Dummy'], + ['analysis pathways', '4 Dummy'], + ['section summation', 'Example'], + ['no such field', 'Example'], + ] as const) { + const printed = output.value(key, value); + assert.deepEqual(Object.keys(printed as object), ['hash', 'length'], key); + assertPrintable(printed, [], TOKEN); + } + // a hash the reduction made is salted: equal values stay equal within a run, and the + // printed hash is not the hash of the name + const names = { count: 2, hash: hash('Placeholder,A') }; + const a = new Output('one').value('cover names', names) as { hash: string }; + const b = new Output('two').value('cover names', names) as { hash: string }; + assert.notEqual(a.hash, b.hash); + assert.notEqual(a.hash, names.hash); + assert.deepEqual(new Output('one').value('cover names', names), a); + // a difference's own fields are checked too + const d = new Output('x').difference({ + file: 'f.pdf', + part: 'Dummy, E', + field: 'Placeholder', + kind: 'Sample', + count: 1, + events: ['R-HSA-1', 'Example, B'], + } as unknown as Difference); + assertPrintable(d, ['f.pdf'], TOKEN); + }); + + test('names planted everywhere in both sides, in random layouts: none is printed', () => { + const output = new Output('fuzz'); + for (let seed = 1; seed <= 300; seed++) { + const next = random(seed); + const ref = reduce(plantedLayout(next)); + const neu = reduce(plantedLayout(next)); + const printed = compareReduced('fuzz.pdf', ref, neu).map((d) => output.difference(d)); + assertPrintable(printed, ['fuzz.pdf'], TOKEN); + } + }); +}); + +/** A reduced PDF of these sections, each its stable identifier and a text. */ +function sections(...ids: (string | [string, string])[]): Reduced { + const list: ReducedSection[] = ids.map((id) => { + const [stId, text] = typeof id === 'string' ? [id, 'text'] : id; + return { stId, figure: 'none', parts: [{ part: 'summation', items: [text] }] }; + }); + return { + cover: { + title: { hash: hash('t'), length: 1 }, + diagram: false, + excerpt: false, + names: { count: 0, hash: hash('') }, + }, + introduction: { release: 98, pathways: 1, reactions: 1 }, + analysis: null, + sections: list, + contents: null, + shortEntries: 0, + }; +} + +const brief = (d: Difference) => + [d.part, d.field, d.kind, ...(d.events ?? [])].filter((w) => w !== undefined).join(' '); + +describe("the sections' order", () => { + const [A, B, C, D, E] = ['R-HSA-1', 'R-HSA-2', 'R-HSA-3', 'R-HSA-4', 'R-HSA-5'] as const; + + test('a section written twice is missing once, and nothing is out of order', () => { + const found = compareReduced('f', sections(A, B, A, C, D, E), sections(A, B, C, D, E)); + assert.deepEqual(found.map(brief), [`sections missing ${A}`]); + }); + + test('one section moved from first to last is the one out of order', () => { + const found = compareReduced('f', sections(A, B, C, D, E), sections(B, C, D, E, A)); + assert.deepEqual(found.map(brief), [`sections order ${A}`]); + }); + + test('each occurrence of a section written twice is compared with its own', () => { + const found = compareReduced( + 'f', + sections([A, 'x'], B, [A, 'y']), + sections([A, 'x'], B, [A, 'z']) + ); + assert.deepEqual(found.map(brief), [`section summation text ${A}`]); + }); + + test('the longest common subsequence, with repeats', () => { + const pairs = commonSubsequence(['a', 'b', 'a', 'c'], ['b', 'a', 'c', 'a']); + assert.equal(pairs.length, 3); + for (const [i, j] of pairs) assert.ok(i >= 0 && j >= 0); + // strictly increasing on both sides + pairs.reduce(([pi, pj], [i, j]) => { + assert.ok(i > pi && j > pj); + return [i, j]; + }); + }); +}); + +describe('--expected', () => { + test('a record naming events covers only those, never a difference that names none', () => { + const difference: Difference = { + file: 'f.pdf', + part: 'section', + field: 'summation', + kind: 'text', + count: 2, + events: ['R-HSA-1', 'R-HSA-2'], + }; + const split = applyRecorded( + [difference], + [{ part: 'section', kind: 'text', events: ['R-HSA-1'], reason: 'rewritten' }] + ); + assert.deepEqual( + split.differences.map((d) => [d.events, d.count, d.recorded]), + [ + [['R-HSA-1'], 1, 'rewritten'], + [['R-HSA-2'], 1, undefined], + ] + ); + const eventless: Difference = { file: 'f.pdf', part: 'contents', kind: 'order', count: 1 }; + const none = applyRecorded( + [eventless], + [{ part: 'contents', kind: 'order', events: ['R-HSA-1'], reason: 'x' }] + ); + assert.equal(none.differences[0]?.recorded, undefined); + assert.deepEqual(none.unmatched, [0]); + // a record for another field covers nothing, and is reported as matching nothing + const other = applyRecorded( + [difference], + [{ part: 'section', field: 'references', kind: 'text', reason: 'x' }] + ); + assert.equal(other.differences[0]?.recorded, undefined); + assert.deepEqual(other.unmatched, [0]); + }); +}); + +/** A book-like layout: a contents list of every section, then the sections. */ +function book(count: number): Layout { + const lines: Line[] = []; + const links: Link[] = []; + const figures: Box[] = []; + const perPage = 60; + const contentsPages = Math.ceil(count / perPage); + lines.push({ page: 1, y: 790, x: 57, size: 18, text: 'Contents' }); + for (let n = 0; n < count; n++) { + const page = 1 + Math.floor(n / perPage); + const y = 770 - (n % perPage) * 12; + lines.push({ page, y, x: 57, size: 10.5, text: `Event ${String(n)} ${String(n + 2)}` }); + const target = contentsPages + 1 + Math.floor(n / 4); + const top = 780 - (n % 4) * 180; + links.push({ + box: { page, x1: 57, y1: y - 2, x2: 400, y2: y + 10 }, + uri: null, + landing: { page: target, y: top + 20 }, + }); + } + for (let n = 0; n < count; n++) { + const page = contentsPages + 1 + Math.floor(n / 4); + const top = 780 - (n % 4) * 180; + lines.push( + { page, y: top, x: 57, size: 16, text: `Event ${String(n)}` }, + { page, y: top - 20, x: 57, size: 10.5, text: 'Location Parent' }, + { page, y: top - 34, x: 57, size: 10.5, text: `Stable identifier R-HSA-${String(n)}` }, + { page, y: top - 48, x: 57, size: 10.5, text: 'Compartments cytosol' }, + { page, y: top - 118, x: 57, size: 10.5, text: 'A summation, written for this test.' }, + { page, y: top - 126, x: 57, size: 10.5, text: 'Its second line.' }, + { page, y: top - 134, x: 57, size: 10.5, text: 'Literature references' } + ); + // literature, each linked to its record, as a real section's is + for (let r = 0; r < 4; r++) { + const y = top - 142 - r * 9; + lines.push({ page, y, x: 62, size: 10.5, text: `${String(r + 1)}. A reference.` }); + links.push({ + box: { page, x1: 62, y1: y - 2, x2: 300, y2: y + 8 }, + uri: `https://pubmed.example/${String(n * 4 + r)}`, + landing: null, + }); + } + links.push({ + box: { page, x1: 57, y1: top - 2, x2: 300, y2: top + 16 }, + uri: `https://site.example/content/detail/R-HSA-${String(n)}`, + landing: null, + }); + figures.push({ page, x1: 100, y1: top - 115, x2: 400, y2: top - 55 }); + } + lines.sort((a, b) => a.page - b.page || b.y - a.y); + const pages = contentsPages + 1 + Math.ceil(count / 4); + return { + pages: Array.from({ length: pages }, () => ({ width: 595, height: 842 })), + lines, + links, + figures, + }; +} + +describe('the reduction at scale', () => { + test('4,000 sections with a contents list of 4,000 entries reduce in seconds', () => { + const layout = book(4000); + const start = performance.now(); + const reduced = reduce(layout); + const seconds = (performance.now() - start) / 1000; + assert.equal(reduced.sections.length, 4000); + assert.equal(reduced.contents?.length, 4000); + assert.ok(reduced.contents.every((entry, n) => entry.target === `R-HSA-${String(n)}`)); + // #165's reduction took 31 s on this layout (and #165's review measured 115 s on its + // own); about 1.3 s now + assert.ok(seconds < 8, `${seconds.toFixed(1)} s`); + }); +}); + +describe('the constants a layout is read by', () => { + /** A contents entry landing `above` points over a section's heading. */ + const landing = (above: number): string | null => { + const layout: Layout = { + pages: [0, 1, 2].map(() => ({ width: 595, height: 842 })), + lines: [ + { page: 1, y: 790, x: 57, size: 18, text: 'Contents' }, + { page: 1, y: 770, x: 57, size: 10.5, text: 'Event 3' }, + { page: 2, y: 500, x: 57, size: 16, text: 'Event' }, + { page: 2, y: 480, x: 57, size: 10.5, text: 'Stable identifier R-HSA-1' }, + { page: 2, y: 460, x: 57, size: 10.5, text: 'A summation.' }, + ], + links: [ + { + box: { page: 1, x1: 57, y1: 768, x2: 300, y2: 780 }, + uri: null, + landing: { page: 2, y: 500 + above }, + }, + ], + figures: [], + }; + return reduce(layout).contents?.[0]?.target ?? null; + }; + + test('LANDING_PT: a landing 76 pt above a heading is on it, one 160 pt above is not', () => { + assert.equal(landing(76), 'R-HSA-1'); + assert.equal(landing(160), null); + }); + + test('FIGURE_MIN_PT: a 50 pt square is not a figure, a 70 pt one is', () => { + const square = (side: number): Box => ({ + page: 0, + x1: 100, + y1: 100, + x2: 100 + side, + y2: 100 + side, + }); + assert.equal(figureBoxes([square(50)], 595, 842).length, 0); + assert.equal(figureBoxes([square(70)], 595, 842).length, 1); + }); + + test('FIGURE_GAP_PT: shapes 20 pt apart are one figure, 45 pt apart two', () => { + const two = (gap: number): Box[] => [ + { page: 0, x1: 100, y1: 100, x2: 170, y2: 170 }, + { page: 0, x1: 170 + gap, y1: 100, x2: 240 + gap, y2: 170 }, + ]; + assert.equal(figureBoxes(two(20), 595, 842).length, 1); + assert.equal(figureBoxes(two(45), 595, 842).length, 2); + }); +}); + +describe('what a library writes to the console', () => { + const PLANTED = 'PlaceholderPersonJaneQExample'; + + /** A PDF whose page's content holds the planted word as an operator pdf.js does not know. */ + async function planted(): Promise { + const doc = await PDFDocument.create(); + doc.addPage().pushOperators(PDFOperator.of(PLANTED as unknown as PDFOperatorNames)); + return doc.save(); + } + + /** What reaches the console and standard error while `run` runs, and what it returns. */ + async function captured(run: () => Promise): Promise<{ text: string; value: T }> { + let text = ''; + const warn = Reflect.get(console, 'warn') as unknown; + const log = Reflect.get(console, 'log') as unknown; + const write = Reflect.get(process.stderr, 'write') as unknown; + const keep = (...args: unknown[]) => { + text += args.map(String).join(' '); + }; + Reflect.set(console, 'warn', keep); + Reflect.set(console, 'log', keep); + Reflect.set(process.stderr, 'write', (chunk: unknown) => { + text += String(chunk); + return true; + }); + try { + const value = await run(); + return { text, value }; + } finally { + Reflect.set(console, 'warn', warn); + Reflect.set(console, 'log', log); + Reflect.set(process.stderr, 'write', write); + } + } + + test('pdf.js reads a PDF quoting its content without a word: its warnings are off', async () => { + const bytes = await planted(); + let said = ''; + const warn = Reflect.get(console, 'warn') as unknown; + Reflect.set(console, 'warn', (...args: unknown[]) => { + said += args.map(String).join(' '); + }); + try { + await readLayout(bytes); + } finally { + Reflect.set(console, 'warn', warn); + } + assert.ok(!said.includes(PLANTED), 'pdf.js printed the planted word'); + }); + + test('compare prints none of it, on standard error or the console', async () => { + const folder = mkdtempSync(join(tmpdir(), 'pdf-export-console-')); + try { + const file = join(folder, 'planted.pdf'); + writeFileSync(file, await planted()); + let err = ''; + let out = ''; + const { text } = await captured(() => + compareCommand([file, file], { out: (t) => (out += t), err: (t) => (err += t) }) + ); + assert.ok(!(text + err + out).includes(PLANTED), 'the planted word was printed'); + } finally { + rmSync(folder, { recursive: true, force: true }); + } + }); + + test("a library's console messages are held while compare runs: counted, kinds only", async () => { + let err = ''; + const warn = Reflect.get(console, 'warn') as unknown; + await withConsoleHeld( + () => { + console.warn(`Unknown command "${PLANTED}"`); + console.error(new TypeError(`no ${PLANTED}`)); + return Promise.resolve(); + }, + (t) => (err += t) + ); + assert.equal(err, '2 message(s) from a library withheld (TypeError)\n'); + assert.equal(Reflect.get(console, 'warn'), warn); + assert.equal(errorKind(new RangeError(PLANTED)), 'RangeError'); + const coded = Object.assign(new Error(PLANTED), { code: 'ENOENT' }); + assert.equal(errorKind(coded), 'Error ENOENT'); + }); +}); + +describe('figures and tables', () => { + const square = (x: number, y: number, side = 70): Box => ({ + page: 0, + x1: x, + y1: y, + x2: x + side, + y2: y + side, + }); + + test('a connector joins the shapes at its two ends into one figure', () => { + const connector: Box = { page: 0, x1: 170, y1: 134.75, x2: 230, y2: 135.25 }; + assert.equal(figureBoxes([square(100, 100), square(230, 100)], 595, 842).length, 2); + assert.equal(figureBoxes([square(100, 100), connector, square(230, 100)], 595, 842).length, 1); + }); + + test('a rule between two figures, touching neither, joins nothing', () => { + const rule: Box = { page: 0, x1: 60, y1: 219.75, x2: 500, y2: 220.25 }; + assert.equal(figureBoxes([square(100, 130), rule, square(100, 240)], 595, 842).length, 2); + }); + + test('a rule from a figure to a mark too small to be a node does not widen the figure', () => { + const rule: Box = { page: 0, x1: 170, y1: 134.75, x2: 500, y2: 135.25 }; + const mark: Box = { page: 0, x1: 500, y1: 133.5, x2: 503, y2: 136.5 }; + const [figure] = figureBoxes([square(100, 100), rule, mark], 595, 842); + assert.equal(figure?.x2, 170); + }); + + test("a table's shaded header and hairline borders are not a figure", () => { + const shapes: Box[] = [{ page: 0, x1: 60, y1: 600, x2: 540, y2: 616 }]; + for (let row = 0; row <= 9; row++) { + const y = 616 - row * 16; + shapes.push({ page: 0, x1: 60, y1: y - 0.25, x2: 540, y2: y + 0.25 }); + } + for (const x of [60, 220, 380, 540]) { + shapes.push({ page: 0, x1: x - 0.25, y1: 472, x2: x + 0.25, y2: 616 }); + } + assert.equal(figureBoxes(shapes, 595, 842).length, 0); + }); +}); + +describe('the analysis and the cover, by their meaning', () => { + const reduced = (rows: string[]) => + reduce({ + pages: [0, 1].map(() => ({ width: 595, height: 842 })), + lines: [ + { page: 1, y: 790, x: 57, size: 18, text: 'Analysis' }, + ...rows.map((text, i) => ({ page: 1, y: 770 - i * 14, x: 57, size: 10.5, text })), + ], + links: [], + figures: [], + }).analysis ?? {}; + + test('identifiers found as `x of y` are counts; any other form is withheld whole', () => { + assert.deepEqual(reduced(['Identifiers found: 3 of 5']), { + identifiersFound: { value: 3 }, + identifiersTotal: { value: 5 }, + }); + const odd = reduced(['Identifiers found: 3 of 5 of 7']); + assert.ok(!('value' in (odd['identifiersFound'] ?? { value: 0 })), 'kept as a value'); + }); + + test('a token encoded otherwise is the same token, printed in its one form', () => { + const twice = TOKEN.replaceAll('%3D', '%253D'); + assert.deepEqual(reduced([`Token: ${twice}`]), reduced([`Token: ${TOKEN}`])); + assert.equal(new Output('x').value('analysis token', twice), TOKEN); + }); + + test("a cover's title is said to be its file's name, or not, side by side", () => { + const titled = (title: string): Reduced => ({ + ...sections(), + cover: { ...sections().cover, title: { hash: hash(title), length: title.length } }, + }); + const [d] = compareReduced('Alpha.pdf', titled('Alpha'), titled('Beta'), { + reference: 'today/Alpha.pdf', + new: 'new/Alpha.pdf', + }); + assert.deepEqual([d?.reference, d?.new], ['same', 'differs']); + }); +});