From 346bf586ebde411807863ec98dbcaa1b0ed9b97a Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Wed, 4 Nov 2026 04:19:41 +0000 Subject: [PATCH 1/2] Spec 001 T035: compare, FR-012's comparison with event-pdf's output pdf-export compare [--expected ], as contracts/cli.md has it: each side a PDF or a folder, a folder compared file by file by name. Both sides are read with pdf.js (lines, links, figures as clusters of images and painted paths) and reduced to FR-012's parts: the cover's title, diagram, excerpt note and distinct names (counted and hashed; U+2060 and no-break spaces normalised); the introduction's release and counts; the analysis's properties; each section's parts by their labels and their text; the contents' entries and targets; each figure's kind. The literature and edit history, which name people, are held as hashes. The JSON report gives each difference by part, field and kind with a count and stable identifiers, list order as a kind of its own; no section text and no name is printed. --expected takes the recorded differences, with their reasons. Tests print a sample document with pdf-export's layout and as a reference in event-pdf's shape (placeholder names, wording written anew): each part's difference is caught, and no name appears in any report. 21 mutants, each caught. Co-Authored-By: Claude Opus 5.5 --- specs/001-pathway-pdfs/contracts/cli.md | 17 +- specs/001-pathway-pdfs/tasks.md | 19 +- src/cli.ts | 9 +- src/compare/compare.ts | 397 +++++++++++++++++ src/compare/read.ts | 320 ++++++++++++++ src/compare/reduce.ts | 538 ++++++++++++++++++++++++ test/compare/compare.test.ts | 459 ++++++++++++++++++++ test/compare/samples.ts | 405 ++++++++++++++++++ 8 files changed, 2160 insertions(+), 4 deletions(-) create mode 100644 src/compare/compare.ts create mode 100644 src/compare/read.ts create mode 100644 src/compare/reduce.ts create mode 100644 test/compare/compare.test.ts create mode 100644 test/compare/samples.ts diff --git a/specs/001-pathway-pdfs/contracts/cli.md b/specs/001-pathway-pdfs/contracts/cli.md index 6bae651..7667327 100644 --- a/specs/001-pathway-pdfs/contracts/cli.md +++ b/specs/001-pathway-pdfs/contracts/cli.md @@ -224,8 +224,21 @@ pathway, and a file on one side only is reported as a difference. - **Output:** a JSON report of the differences by part and kind, with counts. Names appear only as counts and hashes (FR-013). -- **`--expected `:** the recorded differences. With it, the exit is 1 only for a - difference not recorded. + - Each difference has its `file`, its `part` (`file`, `cover`, `introduction`, + `analysis`, `sections`, `section`, `figure`, `contents` or `short-entries`), a `field` + where the part has several (a section's `summation`, the cover's `names`), its `kind` + (`value`, `count`, `text`, `order`, `missing` from the new side, `extra` on it, `kind` + or `target`), a `count`, and the sections it is in by stable identifier. + - Each side's value is given only where it names no one: the cover's title and flags, + the introduction's numbers, the analysis's properties, and the cover's names as a + count and a hash. A section's text is never printed. + - A list in another order is a difference of its own kind, `order`. +- **`--expected `:** the recorded differences, as `{ "differences": [...] }`, each + with a `part`, a `kind` and a `reason`, and optionally a `field`, a `file` and `events` + (a record naming events covers only those). Each difference it covers is reported with + its reason. With it, the exit is 1 only for a difference not recorded. +- **Exit:** 0 with no difference (or none not recorded), 1 with one, 2 on bad arguments, a + reference and a new side not both PDFs or both folders, or a file that cannot be read. ## `pdf-export serve` diff --git a/specs/001-pathway-pdfs/tasks.md b/specs/001-pathway-pdfs/tasks.md index 8909ecd..e6736e1 100644 --- a/specs/001-pathway-pdfs/tasks.md +++ b/specs/001-pathway-pdfs/tasks.md @@ -585,7 +585,7 @@ passes. **Goal:** FR-009 in the image (US4, SC-006), and FR-010, FR-011, FR-012, SC-002, SC-003, SC-007 and SC-010 (US5). -- [ ] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare ` (contracts/cli.md). +- [x] T035 [US5] FR-012's comparison, `src/compare/`: `pdf-export compare ` (contracts/cli.md). - Both sides are reduced to FR-012's parts: - the covers' distinct names, counted, across today's files; - the introduction's release and counts; @@ -599,6 +599,23 @@ passes. Tests on fixture PDFs: each part's difference is caught, and no name is printed. + - **Done (#PR):** `pdf-export compare [--expected ]`, wired into `src/cli.ts`, as contracts/cli.md now has it rather than this task's ` `: the book is a folder of PDFs, so each side is a PDF or a folder, a folder compared file by file by name, and a file on one side only is a difference. + - **Reading** (`src/compare/read.ts`): pdf.js's lines of text (pieces sharing a baseline), links and figures. A page's top and bottom 50 pt (running headers, page numbers) and the text inside a figure (a vector diagram's labels) are left out. A figure is a cluster of images and painted paths, neither thin nor the whole page, at least 60 pt each way, so a raster diagram and a vector one are both found. + - **Reducing** (`src/compare/reduce.ts`), with no knowledge of which program wrote the file: + - the cover: its title (its largest text), a diagram below it, an excerpt note, and its distinct names, counted and hashed (the names are a run of characters joined by U+2060 or no-break spaces where U+2060 is there to read, research R2; otherwise parted by semicolons, as pdf-export writes them); + - the introduction's release and counts, the analysis's properties by label (not its explanatory text); + - each section, found at its `Stable identifier` label, its name the larger text above it: its parts by their labels, each taken only after the parts before it in FR-005's order, so a summation beginning with a label's word stays a summation; the summation is the unlabelled text after the facts (after the figure, or at the section's left edge); + - each part's items: lists split at their separators, literature at its numbers, the edit history at its dates; the literature and edit history, which name people, held as one hash an item; + - the figure's kind, `diagram`, `selection` or `reaction` from its link to the website, `reaction` for a section with a type, or else `figure` (present, kind not read), which matches any kind; + - the contents list's entries (each up to the line ending in its page number) and the section or front part each lands on; + - short entries at later places (FR-004), counted apart. + Text is compared folded: NFKC, with white space and FR-017's characters never drawn (U+2060 among them) left out. + - **Comparing** (`src/compare/compare.ts`): a JSON report, each difference by part, field and kind (`value`, `count`, `text`, `order`, `missing`, `extra`, `kind`, `target`), with a count and its sections' stable identifiers; a list with the same items in another order is `order`, never `text`. Values are printed only where they name no one; a section's text never is, and the cover's names only as a count and a hash. `--expected` takes `{ "differences": [...] }`, each with a part, kind and reason and optionally a field, file and events (covering only those events); the exit is 1 only for a difference not recorded, 2 on bad arguments or an unreadable file. contracts/cli.md says so. + - **Tests:** `test/compare/compare.test.ts` (19, on Chromium; no graph). A sample document (`test/compare/samples.ts`, placeholder names only) is printed by pdf-export's layout and, as the reference, as HTML in `event-pdf`'s shape (contents last, `Label: value` parts, raster figures and logo, running headers, names with U+2060 and no-break spaces), with wording written anew. Real `event-pdf` files are never used. The same document in both differs only in the analysis's resource. Each part's difference is caught: the cover's names (count, and another name of the same count), title, diagram and excerpt; the release and counts; three analysis properties; a summation's text, a part missing, a list's order, a section missing, the sections' order, the literature's and edit history's hashes; the contents' entries, order and targets; a figure missing and of another kind; a short entry. `--expected` passes recorded differences and fails on the one left out; a record for some events or another field covers only those. No placeholder's name or part of one appears in any report, error or reduced form, and no assertion prints a text read from a PDF. + - **Mutants, each caught:** names not read as U+2060 runs; white space kept in folding; names not made distinct; the names' hash not compared; names printed in the clear in place of the hash (the no-name test); literature not hashed; the release not read; analysis values not compared; an order read as text; a section missing not reported; contents targets not compared; links landing nowhere; figures not compared; a figure's text kept; the margins kept; labels taken in any order; short entries not set apart; a record's field ignored; recorded differences failing the run; the summation not split from the facts; a diagram anywhere on the cover (above the title, a logo). + - **Found:** pdf-export's analysis part does not print the resource (`TOTAL`), which FR-012 lists among the properties: `analysis resource missing` until it does, for T040 to record or fix. + - **Known gaps, for T040:** `event-pdf`'s own labels, its cover's names' separator (whether U+2060 survives pdf.js's reading of its files), its margins and its contents' form are not known here, so the reference's shape is the Background's and R2's, and its wording this task's. A label not known leaves its part unread, which shows as a difference, never a pass; T040 adds `event-pdf`'s labels to `RULES` and `PROPERTIES` in `src/compare/reduce.ts`. A fact's value wrapped to the section's left edge would be read as the summation's start. A summation and an analysis's found tables are compared as text and rows, not by their markup. + - [ ] T036 [US5] The fixture's output as a baseline (the constitution's quality gate for a change to what is written). - `ci/fixture/expected.json` holds the fixture book and a fixture single-event PDF, reduced by T035's `compare`; `compare` gains `--reduced ` to take a reduced form as the reference, added to contracts/cli.md in this task (the fixture PDFs are not committed). - CI compares each PR's output with it, and fails on any difference that the PR does not also update. diff --git a/src/cli.ts b/src/cli.ts index 6df0dae..af54db9 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -2,7 +2,8 @@ // pdf-export's command line (contracts/cli.md). Its commands come with spec 001's tasks: // serve (T022) runs the single-event service; preflight (T024) runs the book's preflight // alone; figures (T046) draws the book's figures into a folder; book (T029) writes the -// book; check-book (T030) checks a written book for FR-011's faults; compare is to come. +// book; check-book (T030) checks a written book for FR-011's faults; compare (T035) +// compares with event-pdf's output by FR-012. import { constants as osConstants } from 'node:os'; import { privateTemporary } from './print/temporary.ts'; @@ -93,6 +94,12 @@ if (import.meta.main) { out: (text) => process.stdout.write(text), err: (text) => process.stderr.write(text), }); + } else if (args[0] === 'compare') { + const { compareCommand } = await import('./compare/compare.ts'); + process.exitCode = await compareCommand(args.slice(1), { + out: (text) => process.stdout.write(text), + err: (text) => process.stderr.write(text), + }); } else if (args[0] === 'figures') { const { figuresCommand } = await import('./figures/step.ts'); process.exitCode = await figuresCommand(args.slice(1), process.env, { diff --git a/src/compare/compare.ts b/src/compare/compare.ts new file mode 100644 index 0000000..bad7e56 --- /dev/null +++ b/src/compare/compare.ts @@ -0,0 +1,397 @@ +// FR-012's comparison (spec 001 T035, contracts/cli.md `compare`, research R8): a reference +// (`event-pdf`'s output) and the new output, each a PDF or a folder of them, reduced to +// FR-012's parts (reduce.ts) and compared part by part. A folder is compared file by file, +// by name, as today's book is one file per top-level pathway, named after it. +// +// Each difference is reported by its part, its field and its kind, with a count and the +// stable identifiers of the sections it is in. No text of a section is printed, and the +// cover's names only as a count and a hash (FR-013): the report holds a pathway's title, +// the introduction's numbers and the analysis's properties, none of which names a person. +// A list in another order is a difference of its own kind, `order`, which `--expected` +// records like any other (the spec's expected differences: today's lists are in the +// database's order, the new ones in the curators'). +import { readdirSync, readFileSync, statSync } from 'node:fs'; +import { basename, join } from 'node:path'; + +import { SettingsError } from '../preflight/command.ts'; +import { readLayout } from './read.ts'; +import { reduce, type Reduced, type ReducedPart } from './reduce.ts'; + +export interface Difference { + /** The file compared, by name. */ + file: string; + part: + | 'file' + | 'cover' + | 'introduction' + | 'analysis' + | 'sections' + | 'section' + | 'contents' + | 'figure' + | 'short-entries'; + /** Which of the part's fields: a cover's `names`, a section's `summation`; none for some. */ + field?: string; + /** `value`, `text`, `order`, `missing` (on the new side), `extra` or `kind`. */ + kind: string; + count: number; + /** The sections it is in, by stable identifier. */ + events?: string[]; + /** Each side's value, where it names no one: the cover's names as a count and a hash. */ + reference?: unknown; + new?: unknown; + /** Why it is expected, where `--expected` records it. */ + recorded?: string; +} + +/** A recorded difference: matches by part and kind, and by field, file and events if given. */ +export interface Recorded { + part: string; + kind: string; + field?: string; + file?: string; + events?: string[]; + reason: string; +} + +/** How a list differs: in its items, or only in their order. */ +function listKind(a: readonly string[], b: readonly string[]): 'text' | 'order' | null { + if (a.length === b.length && a.every((item, i) => item === b[i])) return null; + const sorted = (list: readonly string[]) => [...list].sort().join('\n'); + return sorted(a) === sorted(b) ? 'order' : 'text'; +} + +/** Two reduced PDFs' differences, grouped by part, field and kind. */ +export function compareReduced(file: string, ref: Reduced, neu: Reduced): Difference[] { + const out: Difference[] = []; + const add = (d: Omit & { count?: number }) => { + out.push({ file, count: d.events?.length ?? 1, ...d }); + }; + + // the cover + if (ref.cover.title !== neu.cover.title) { + add({ + part: 'cover', + field: 'title', + kind: 'value', + reference: ref.cover.title, + new: neu.cover.title, + }); + } + for (const field of ['diagram', 'excerpt'] as const) { + if (ref.cover[field] !== neu.cover[field]) { + add({ + part: 'cover', + field, + kind: 'value', + reference: ref.cover[field], + new: neu.cover[field], + }); + } + } + const names = { reference: ref.cover.names, new: neu.cover.names }; + if (ref.cover.names.count !== neu.cover.names.count) { + add({ part: 'cover', field: 'names', kind: 'count', ...names }); + } else if (ref.cover.names.hash !== neu.cover.names.hash) { + add({ part: 'cover', field: 'names', kind: 'value', ...names }); + } + + // the introduction + for (const field of ['release', 'pathways', 'reactions'] as const) { + const [a, b] = [ref.introduction[field], neu.introduction[field]]; + if (a !== b) add({ part: 'introduction', field, kind: 'value', reference: a, new: b }); + } + + // the analysis's properties + if ((ref.analysis === null) !== (neu.analysis === null)) { + add({ part: 'analysis', kind: ref.analysis ? 'missing' : 'extra' }); + } else if (ref.analysis && neu.analysis) { + const keys = [...new Set([...Object.keys(ref.analysis), ...Object.keys(neu.analysis)])].sort(); + for (const field of keys) { + const [a, b] = [ref.analysis[field], neu.analysis[field]]; + if (a === b) continue; + const kind = a === undefined ? 'extra' : b === undefined ? 'missing' : 'value'; + add({ part: 'analysis', field, kind, reference: a ?? null, new: b ?? null }); + } + } + + // the sections: which, and in what order + const refIds = ref.sections.map((s) => s.stId); + const newIds = neu.sections.map((s) => s.stId); + const missing = refIds.filter((id) => !newIds.includes(id)); + const extra = newIds.filter((id) => !refIds.includes(id)); + if (missing.length > 0) add({ part: 'sections', kind: 'missing', events: missing }); + if (extra.length > 0) add({ part: 'sections', kind: 'extra', events: extra }); + const common = refIds.filter((id) => newIds.includes(id)); + const commonNew = newIds.filter((id) => refIds.includes(id)); + const moved = commonNew.filter((id, i) => id !== common[i]); + if (moved.length > 0) add({ part: 'sections', kind: 'order', events: moved }); + + // each section's parts and figure, grouped by field and kind across sections + const grouped = new Map(); + const note = ( + field: string, + kind: string, + stId: string, + part: Difference['part'] = 'section' + ) => { + const key = `${part}\0${field}\0${kind}`; + let d = grouped.get(key); + if (!d) { + d = { file, part, ...(part === 'section' ? { field } : {}), kind, count: 0, events: [] }; + grouped.set(key, d); + } + d.events?.push(stId); + d.count++; + }; + for (const stId of common) { + const a = ref.sections.find((s) => s.stId === stId); + const b = neu.sections.find((s) => s.stId === stId); + if (!a || !b) continue; + const byPart = (parts: readonly ReducedPart[]) => new Map(parts.map((p) => [p.part, p.items])); + const [pa, pb] = [byPart(a.parts), byPart(b.parts)]; + for (const part of new Set([...pa.keys(), ...pb.keys()])) { + const [ia, ib] = [pa.get(part), pb.get(part)]; + if (!ia) note(part, 'extra', stId); + else if (!ib) note(part, 'missing', stId); + else { + const kind = listKind(ia, ib); + if (kind) note(part, kind, stId); + } + } + // a figure of a kind not read is compared only as being there + if (a.figure !== b.figure) { + if (a.figure === 'none') note('figure', 'extra', stId, 'figure'); + else if (b.figure === 'none') note('figure', 'missing', stId, 'figure'); + else if (a.figure !== 'figure' && b.figure !== 'figure') + note('figure', 'kind', stId, 'figure'); + } + } + out.push(...grouped.values()); + + // the contents list: its entries, their order and their targets + if ((ref.contents === null) !== (neu.contents === null)) { + add({ part: 'contents', kind: ref.contents ? 'missing' : 'extra' }); + } else if (ref.contents && neu.contents) { + const key = (e: { title: string; target: string | null }) => e.title; + const remaining = neu.contents.map(key); + let gone = 0; + for (const entry of ref.contents) { + const at = remaining.indexOf(key(entry)); + if (at < 0) gone++; + else remaining.splice(at, 1); + } + if (gone > 0) add({ part: 'contents', kind: 'missing', count: gone }); + if (remaining.length > 0) add({ part: 'contents', kind: 'extra', count: remaining.length }); + if (gone === 0 && remaining.length === 0) { + const [ta, tb] = [ref.contents.map(key), neu.contents.map(key)]; + if (listKind(ta, tb) === 'order') add({ part: 'contents', kind: 'order' }); + } + // an entry's target, where both sides read one: matched by title, in turn + const targets = new Map(); + for (const entry of neu.contents) { + targets.set(entry.title, [...(targets.get(entry.title) ?? []), entry.target]); + } + let wrong = 0; + for (const entry of ref.contents) { + const target = targets.get(entry.title)?.shift(); + if ( + entry.target !== null && + target !== undefined && + target !== null && + target !== entry.target + ) { + wrong++; + } + } + if (wrong > 0) add({ part: 'contents', kind: 'target', count: wrong }); + } + + if (ref.shortEntries !== neu.shortEntries) { + add({ + part: 'short-entries', + kind: 'count', + reference: ref.shortEntries, + new: neu.shortEntries, + }); + } + return out; +} + +/** + * The differences with those `--expected` records marked, with their reasons. A record + * naming events covers only those: a difference's other events stay unrecorded, apart. + */ +export function applyRecorded( + differences: readonly Difference[], + recorded: readonly Recorded[] +): Difference[] { + const out: Difference[] = []; + for (const difference of differences) { + let rest: Difference | null = { ...difference }; + for (const record of recorded) { + if (!rest) break; + if (record.part !== rest.part || record.kind !== rest.kind) continue; + if (record.field !== undefined && record.field !== rest.field) continue; + if (record.file !== undefined && record.file !== rest.file) continue; + if (record.events === undefined || rest.events === undefined) { + out.push({ ...rest, recorded: record.reason }); + rest = null; + continue; + } + const events: string[] = rest.events; + const covered = events.filter((e) => record.events?.includes(e)); + if (covered.length === 0) continue; + out.push({ ...rest, events: covered, count: covered.length, recorded: record.reason }); + const left = events.filter((e) => !covered.includes(e)); + rest = left.length > 0 ? { ...rest, events: left, count: left.length } : null; + } + if (rest) out.push(rest); + } + return out; +} + +/** The recorded differences, from `--expected`'s file: `{ "differences": [...] }`. */ +export function readRecorded(path: string): Recorded[] { + const parsed: unknown = JSON.parse(readFileSync(path, 'utf8')); + const list = + parsed !== null && typeof parsed === 'object' && 'differences' in parsed + ? parsed.differences + : undefined; + if (!Array.isArray(list)) throw new SettingsError(`${path} holds no "differences" list`); + return list.map((item: unknown, i) => { + const r = item as Partial | null; + const text = (v: unknown) => typeof v === 'string' && v !== ''; + const ok = + r !== null && + typeof r === 'object' && + text(r.part) && + text(r.kind) && + text(r.reason) && + (r.field === undefined || text(r.field)) && + (r.file === undefined || text(r.file)) && + (r.events === undefined || (Array.isArray(r.events) && r.events.every(text))); + if (!ok) { + throw new SettingsError( + `${path}: difference ${String(i)} needs a part, a kind and a reason, and its field, file and events as text` + ); + } + return r as Recorded; + }); +} + +/** One PDF, reduced. */ +export async function reducePdf(path: string): Promise { + return reduce(await readLayout(new Uint8Array(readFileSync(path)))); +} + +/** A reference and a new side, each a PDF or a folder of them, compared (contracts/cli.md). */ +export async function comparePaths(reference: string, neu: string): Promise { + const folder = (path: string) => statSync(path).isDirectory(); + const [refFolder, newFolder] = [folder(reference), folder(neu)]; + if (refFolder !== newFolder) { + throw new SettingsError('The reference and the new side must both be PDFs or both be folders'); + } + if (!refFolder) { + return compareReduced(basename(neu), await reducePdf(reference), await reducePdf(neu)); + } + const pdfs = (path: string) => + readdirSync(path) + .filter((name) => name.toLowerCase().endsWith('.pdf')) + .sort(); + const [refFiles, newFiles] = [pdfs(reference), pdfs(neu)]; + const out: Difference[] = []; + for (const file of refFiles) { + if (!newFiles.includes(file)) out.push({ file, part: 'file', kind: 'missing', count: 1 }); + } + for (const file of newFiles) { + if (!refFiles.includes(file)) out.push({ file, part: 'file', kind: 'extra', count: 1 }); + } + for (const file of refFiles.filter((f) => newFiles.includes(f))) { + out.push( + ...compareReduced( + file, + await reducePdf(join(reference, file)), + await reducePdf(join(neu, file)) + ) + ); + } + return out; +} + +export interface CompareSettings { + reference: string; + neu: string; + expected: string | null; +} + +export function compareSettings(args: readonly string[]): CompareSettings { + const paths: string[] = []; + let expected: string | null = null; + for (let i = 0; i < args.length; i++) { + const arg = args[i] ?? ''; + if (!arg.startsWith('--')) { + paths.push(arg); + continue; + } + const [name = '', inline] = arg.split(/=(.*)/s, 2); + if (name !== '--expected') throw new SettingsError(`Unknown argument: ${name}`); + const value = inline ?? args[++i]; + if (value === undefined || value === '') throw new SettingsError(`${name} needs a value`); + if (expected !== null) throw new SettingsError(`${name} is given twice`); + expected = value; + } + const [reference, neu] = paths; + if (paths.length !== 2 || !reference || !neu) { + throw new SettingsError('Two paths are needed: pdf-export compare '); + } + return { reference, neu, expected }; +} + +/** + * `pdf-export compare [--expected ]`: the report as JSON on standard + * output; exit 0 with no difference (or none not recorded, with `--expected`), 1 with one, + * 2 on bad arguments, a path that cannot be read or any other error. + */ +export async function compareCommand( + args: readonly string[], + { out, err }: { out: (text: string) => void; err: (text: string) => void } +): Promise { + let settings: CompareSettings; + let recorded: Recorded[] = []; + try { + settings = compareSettings(args); + if (settings.expected !== null) recorded = readRecorded(settings.expected); + } catch (error) { + if (error instanceof SettingsError) { + err(`${error.message}\n`); + return 2; + } + // the file's text is never printed: it may be anything + err( + `The recorded differences cannot be read: ${error instanceof Error ? error.name : 'error'}\n` + ); + return 2; + } + let differences: Difference[]; + try { + differences = applyRecorded(await comparePaths(settings.reference, settings.neu), recorded); + } catch (error) { + // the error's message (a path, or pdf.js's word on a file it cannot read), never a text + err(`The comparison stopped: ${error instanceof Error ? error.message : 'error'}\n`); + return 2; + } + const unrecorded = differences.filter((d) => d.recorded === undefined).length; + const report = { + reference: settings.reference, + new: settings.neu, + counts: { + differences: differences.length, + recorded: differences.length - unrecorded, + unrecorded, + }, + differences, + }; + out(`${JSON.stringify(report, null, 2)}\n`); + return unrecorded > 0 ? 1 : 0; +} diff --git a/src/compare/read.ts b/src/compare/read.ts new file mode 100644 index 0000000..481858c --- /dev/null +++ b/src/compare/read.ts @@ -0,0 +1,320 @@ +// What FR-012's comparison reads from a PDF (spec 001 T035, research R8): its lines of text, +// its links and its figures, with pdf.js, from either layout: pdf-export's or `event-pdf`'s. +// Nothing here knows which made the file; src/compare/reduce.ts reads the parts from these. +// +// - **Lines:** the pieces of text that share a baseline on a page, left to right, each with +// its left edge and its largest size. What is in a page's top or bottom MARGIN_BAND_PT (the +// running headers and page numbers, which FR-012 does not compare) and what is inside a +// figure (a vector diagram's labels) is left out. +// - **Links:** each link's box, its address or where in the PDF it lands. +// - **Figures:** the boxes of what is drawn: images, and painted paths that are neither thin +// (a rule, a table's border) nor the whole page, clustered where they lie within +// FIGURE_GAP_PT of each other; a cluster at least FIGURE_MIN_PT each way is a figure. A +// raster diagram is one image; a vector one is many paths. +import { getDocument, OPS } from 'pdfjs-dist/legacy/build/pdf.mjs'; + +/** + * The bands at a page's top and bottom read as margins: within pdf-export's 20 mm (56.7 pt, + * src/layout/figure.ts), where its running headers and page numbers are, and above the first + * line of its text area. `event-pdf`'s margins are to be confirmed on its real files (T040). + */ +export const MARGIN_BAND_PT = 50; + +/** Shapes nearer than this are one figure. */ +export const FIGURE_GAP_PT = 30; + +/** A figure is at least this wide and this tall: a logo's mark or a bullet is not. */ +export const FIGURE_MIN_PT = 60; + +/** A shape thinner than this either way is a rule or a border, not part of a figure. */ +const THIN_PT = 2; + +/** A place in a PDF: its page, counted from 0, and a height on it in points from its foot. */ +export interface Spot { + page: number; + y: number; +} + +/** A box on a page: left, bottom, right, top, in points. */ +export interface Box { + page: number; + x1: number; + y1: number; + x2: number; + y2: number; +} + +export interface Line extends Spot { + /** Its left edge. */ + x: number; + /** Its largest piece's size. */ + size: number; + /** Its pieces' text, joined as pdf.js gives them. */ + text: string; +} + +export interface Link { + box: Box; + uri: string | null; + /** Where an internal link lands, or null. */ + landing: Spot | null; +} + +export interface Layout { + pages: { width: number; height: number }[]; + /** In reading order: by page, then from the top down. */ + lines: Line[]; + links: Link[]; + figures: Box[]; +} + +type Matrix = [number, number, number, number, number, number]; + +const IDENTITY: Matrix = [1, 0, 0, 1, 0, 0]; + +/** `m` then `n`: a point is taken through `m`, then through `n` (PDF's row-vector order). */ +function multiply(m: Matrix, n: Matrix): Matrix { + return [ + m[0] * n[0] + m[1] * n[2], + m[0] * n[1] + m[1] * n[3], + m[2] * n[0] + m[3] * n[2], + m[2] * n[1] + m[3] * n[3], + m[4] * n[0] + m[5] * n[2] + n[4], + m[4] * n[1] + m[5] * n[3] + n[5], + ]; +} + +/** A rectangle [x1, y1, x2, y2] through a matrix, as the box around its corners. */ +function through(rect: readonly number[], m: Matrix, page: number): Box { + const [x1 = 0, y1 = 0, x2 = 0, y2 = 0] = rect; + const xs: number[] = []; + const ys: number[] = []; + for (const [x, y] of [ + [x1, y1], + [x1, y2], + [x2, y1], + [x2, y2], + ] as const) { + xs.push(m[0] * x + m[2] * y + m[4]); + ys.push(m[1] * x + m[3] * y + m[5]); + } + return { + page, + x1: Math.min(...xs), + y1: Math.min(...ys), + x2: Math.max(...xs), + y2: Math.max(...ys), + }; +} + +const isMatrix = (value: unknown): value is Matrix => + Array.isArray(value) && value.length === 6 && value.every((n) => typeof n === 'number'); + +const IMAGES = new Set([ + OPS.paintImageXObject, + OPS.paintInlineImageXObject, + OPS.paintImageMaskXObject, + OPS.paintImageXObjectRepeat, + OPS.paintInlineImageXObjectGroup, + OPS.paintImageMaskXObjectGroup, + OPS.paintSolidColorImageMask, +]); + +/** The shapes a page paints: images, and painted paths, each as its box. */ +function shapes(fnArray: readonly number[], argsArray: readonly unknown[], page: number): Box[] { + const boxes: Box[] = []; + let ctm: Matrix = IDENTITY; + const stack: Matrix[] = []; + for (let i = 0; i < fnArray.length; i++) { + const fn = fnArray[i]; + const args = argsArray[i]; + if (fn === OPS.save || fn === OPS.paintFormXObjectBegin) { + stack.push(ctm); + const matrix: unknown = + fn === OPS.paintFormXObjectBegin && Array.isArray(args) ? args[0] : null; + if (isMatrix(matrix)) ctm = multiply(matrix, ctm); + } else if (fn === OPS.restore || fn === OPS.paintFormXObjectEnd) { + ctm = stack.pop() ?? IDENTITY; + } else if (fn === OPS.transform && isMatrix(args)) { + ctm = multiply(args, ctm); + } else if (fn === OPS.constructPath && Array.isArray(args)) { + // [how it is painted, its drawing, its box]: a path only clipped to is not drawn + const [paint, , minMax] = args as [unknown, unknown, unknown]; + if (paint === OPS.endPath || minMax === null || typeof minMax !== 'object') continue; + const rect = Array.from(minMax as ArrayLike); + if (rect.length === 4 && rect.every(Number.isFinite)) boxes.push(through(rect, ctm, page)); + } else if (fn !== undefined && IMAGES.has(fn)) { + boxes.push(through([0, 0, 1, 1], ctm, page)); + } + } + return boxes; +} + +/** The figures among a page's shapes: clusters of the shapes that are neither thin nor all. */ +export function figureBoxes(boxes: readonly Box[], width: number, height: number): Box[] { + const clusters: Box[] = []; + const near = (a: Box, b: Box) => + a.x1 - FIGURE_GAP_PT <= b.x2 && + b.x1 - FIGURE_GAP_PT <= a.x2 && + a.y1 - FIGURE_GAP_PT <= b.y2 && + b.y1 - FIGURE_GAP_PT <= a.y2; + for (const box of boxes) { + const w = box.x2 - box.x1; + const h = box.y2 - box.y1; + if (w < THIN_PT || h < THIN_PT || w * h >= 0.9 * width * height) continue; + let merged = { ...box }; + // a box may join clusters that were apart until it came + for (let i = clusters.length - 1; i >= 0; i--) { + const cluster = clusters[i]; + if (!cluster || !near(cluster, merged)) continue; + merged = { + page: box.page, + x1: Math.min(cluster.x1, merged.x1), + y1: Math.min(cluster.y1, merged.y1), + x2: Math.max(cluster.x2, merged.x2), + y2: Math.max(cluster.y2, merged.y2), + }; + clusters.splice(i, 1); + } + clusters.push(merged); + } + // a merge can bring a cluster near one it was not near before + let again = true; + while (again) { + again = false; + for (let i = 0; i < clusters.length && !again; i++) { + for (let j = i + 1; j < clusters.length && !again; j++) { + const a = clusters[i]; + const b = clusters[j]; + if (!a || !b || !near(a, b)) continue; + clusters[i] = { + page: a.page, + x1: Math.min(a.x1, b.x1), + y1: Math.min(a.y1, b.y1), + x2: Math.max(a.x2, b.x2), + y2: Math.max(a.y2, b.y2), + }; + clusters.splice(j, 1); + again = true; + } + } + } + return clusters.filter((c) => c.x2 - c.x1 >= FIGURE_MIN_PT && c.y2 - c.y1 >= FIGURE_MIN_PT); +} + +export const inside = (x: number, y: number, box: Box): boolean => + x >= box.x1 - 1 && x <= box.x2 + 1 && y >= box.y1 - 1 && y <= box.y2 + 1; + +type PdfJs = Awaited['promise']>; + +const isRecord = (value: unknown): value is Record => + value !== null && typeof value === 'object'; + +/** Where a destination lands, or null where it names no page of the PDF. */ +async function land(doc: PdfJs, dest: unknown): Promise { + try { + const explicit: unknown = typeof dest === 'string' ? await doc.getDestination(dest) : dest; + if (!Array.isArray(explicit)) return null; + const [ref, kind, left, top] = explicit as unknown[]; + let page: number; + if (typeof ref === 'number') { + if (!Number.isInteger(ref) || ref < 0 || ref >= doc.numPages) return null; + page = ref; + } else if (isRecord(ref) && 'num' in ref && 'gen' in ref) { + page = await doc.getPageIndex(ref as { num: number; gen: number }); + } else { + return null; + } + const name = isRecord(kind) && 'name' in kind ? kind['name'] : null; + const height = name === 'XYZ' ? top : name === 'FitH' || name === 'FitBH' ? left : null; + if (typeof height === 'number') return { page, y: height }; + const [, , , pageTop = 0] = (await doc.getPage(page + 1)).view; + return { page, y: pageTop }; + } catch { + return null; + } +} + +interface Piece { + x: number; + y: number; + size: number; + str: string; +} + +/** A page's pieces as lines: those sharing a baseline, within a point, left to right. */ +function linesOf(pieces: Piece[], page: number): Line[] { + const lines: Line[] = []; + const sorted = [...pieces].sort((a, b) => b.y - a.y || a.x - b.x); + let current: Piece[] = []; + const flush = () => { + const [first] = current; + if (!first) return; + const row = current.sort((a, b) => a.x - b.x); + lines.push({ + page, + y: first.y, + x: Math.min(...row.map((p) => p.x)), + size: Math.max(...row.map((p) => p.size)), + text: row.map((p) => p.str).join(''), + }); + current = []; + }; + for (const piece of sorted) { + const [first] = current; + if (first && Math.abs(first.y - piece.y) > 1) flush(); + current.push(piece); + } + flush(); + return lines; +} + +/** A PDF's lines, links and figures, read with pdf.js. */ +export async function readLayout(data: Uint8Array): Promise { + // a copy: pdf.js takes the buffer it is given + const task = getDocument({ data: new Uint8Array(data) }); + try { + const doc = await task.promise; + const layout: Layout = { pages: [], lines: [], links: [], figures: [] }; + for (let i = 0; i < doc.numPages; i++) { + const page = await doc.getPage(i + 1); + const [, , width = 0, height = 0] = page.view; + layout.pages.push({ width, height }); + const operators = await page.getOperatorList(); + const figures = figureBoxes(shapes(operators.fnArray, operators.argsArray, i), width, height); + layout.figures.push(...figures); + const pieces: Piece[] = []; + for (const item of (await page.getTextContent()).items) { + if (!('str' in item) || item.str === '') continue; + const [a = 0, b = 0, c = 0, d = 0, x = 0, y = 0] = item.transform as number[]; + if (y <= MARGIN_BAND_PT || y >= height - MARGIN_BAND_PT) continue; + if (figures.some((box) => inside(x, y, box))) continue; + pieces.push({ x, y, size: Math.max(Math.hypot(a, b), Math.hypot(c, d)), str: item.str }); + } + layout.lines.push(...linesOf(pieces, i).filter((line) => line.text.trim() !== '')); + for (const annot of (await page.getAnnotations()) as unknown[]) { + if (!isRecord(annot) || annot['subtype'] !== 'Link') continue; + const [x1 = 0, y1 = 0, x2 = 0, y2 = 0] = (annot['rect'] as number[] | undefined) ?? []; + const address = annot['unsafeUrl'] ?? annot['url']; + const uri = typeof address === 'string' ? address : null; + const dest = annot['dest']; + layout.links.push({ + box: { + page: i, + x1: Math.min(x1, x2), + y1: Math.min(y1, y2), + x2: Math.max(x1, x2), + y2: Math.max(y1, y2), + }, + uri, + landing: + uri === null && dest !== undefined && dest !== null ? await land(doc, dest) : null, + }); + } + page.cleanup(); + } + return layout; + } finally { + await task.destroy(); + } +} diff --git a/src/compare/reduce.ts b/src/compare/reduce.ts new file mode 100644 index 0000000..581b92c --- /dev/null +++ b/src/compare/reduce.ts @@ -0,0 +1,538 @@ +// A PDF reduced to FR-012's parts (spec 001 T035, research R8), from either layout: the +// cover's title, diagram, excerpt note and distinct names; the introduction's release and +// counts; the analysis's properties; each section's parts and their text; the contents +// list's entries and their targets; and each section's figure's kind. +// +// Text is compared folded: compatibility forms folded (NFKC), and white space and the +// characters never drawn (FR-017's NEVER_DRAWN: joiners, U+2060 among them) left out, so a +// line broken elsewhere, or the cover's names written with U+2060 between their characters +// and no-break spaces between their words (`event-pdf`, research R2), read the same. +// +// People's names are never held as text: the cover's are a count and a hash, and a +// section's literature references and edit history, which name their authors, are held as +// one hash an item (FR-013). +// +// The parts are found by their labels. Each part has pdf-export's label and the others it +// is known by; `event-pdf`'s own are to be confirmed on its real files (T040), and a label +// not known leaves its part unread, which the comparison reports, never passes. +import { createHash } from 'node:crypto'; + +import { NEVER_DRAWN } from '../layout/characters.ts'; +import { CONTRIBUTORS, INSTITUTIONS } from '../layout/text.ts'; +import type { Box, Layout, Line, Spot } from './read.ts'; + +/** People's names, or a list that holds them: how many, and one hash of them all. */ +export interface Hashed { + count: number; + hash: string; +} + +export type FigureKind = 'none' | 'diagram' | 'selection' | 'reaction' | 'figure'; + +export interface ReducedPart { + part: PartName; + /** Its items, folded, in order; a part that names people holds a hash for each. */ + items: string[]; +} + +export interface ReducedSection { + stId: string; + /** Its parts in the order they are printed, its name first. */ + parts: ReducedPart[]; + /** + * Its figure: `diagram`, `selection` or `reaction` where its link to the website says + * which; `reaction` too for a reaction's section whose figure has no such link; else + * `figure`, a figure of a kind not read. + */ + figure: FigureKind; +} + +export interface ContentsItem { + title: string; + /** The section's stable identifier, or the front part's name, it lands on; null if none. */ + target: string | null; +} + +export interface Reduced { + cover: { title: string; diagram: boolean; excerpt: boolean; names: Hashed }; + introduction: { release: number | null; pathways: number | null; reactions: number | null }; + /** The analysis's properties by name, their values folded, yes or no for either; null without one. */ + analysis: Record | null; + sections: ReducedSection[]; + /** Null where the PDF has no contents list. */ + contents: ContentsItem[] | null; + /** Short entries at an event's later places (FR-004), which today's files do not write. */ + shortEntries: number; +} + +const UNPRINTED = new RegExp(`${NEVER_DRAWN.source}+`, 'gu'); + +/** Text as compared: compatibility forms folded, white space and joiners left out. */ +export const fold = (text: string): string => text.normalize('NFKC').replace(UNPRINTED, ''); + +/** A short hash of a text, as names and the parts naming people are held. */ +export const hash = (text: string): string => + createHash('sha256').update(text).digest('hex').slice(0, 16); + +/** Distinct names, folded, counted and hashed together; empty names left out. */ +export function hashNames(names: readonly string[]): Hashed { + const distinct = [...new Set(names.map(fold).filter((name) => name !== ''))].sort(); + return { count: distinct.length, hash: hash(distinct.join('\n')) }; +} + +/** + * The cover's names from their text. Where the names are written with U+2060 between their + * characters (research R2), each name is a run of characters joined by U+2060 or no-break + * spaces, and whatever else stands between runs parts them; otherwise names are parted by + * semicolons, as pdf-export writes them. + */ +export function coverNames(text: string): string[] { + if (text.includes('\u2060')) { + return text.match(/[^\s\u2060\u00a0](?:[\u2060\u00a0]+[^\s\u2060\u00a0])+/gu) ?? []; + } + return text.split(';'); +} + +export const PART_NAMES = [ + 'name', + 'location', + 'stId', + 'type', + 'compartments', + 'diseases', + 'inferredFrom', + 'summation', + 'before', + 'after', + 'references', + 'edits', + 'entities', + 'interactors', +] as const; + +export type PartName = (typeof PART_NAMES)[number]; + +interface PartRule { + part: PartName; + /** The labels it is known by, lower case, longest first where one begins another. */ + labels: readonly string[]; + /** A fact, beside its label: a line of the section's own left edge after it ends it. */ + fact?: boolean; + /** Its items, within its text: parted by a pattern, or each on lines of their own. */ + split?: RegExp | 'numbered' | 'dated' | 'rows'; + /** Whether its items name people, and so are held as hashes. */ + names?: boolean; +} + +/** The labelled parts, in the order a section prints them (FR-005). */ +const RULES: readonly PartRule[] = [ + { part: 'location', labels: ['location'], fact: true, split: /[›»>]/ }, + { part: 'stId', labels: ['stable identifier', 'stable id'], fact: true }, + { part: 'type', labels: ['reaction type', 'type'], fact: true }, + { part: 'compartments', labels: ['compartments', 'compartment'], fact: true, split: /,/ }, + { part: 'diseases', labels: ['diseases', 'disease'], fact: true, split: /,/ }, + { part: 'inferredFrom', labels: ['inferred from'], fact: true, split: /;/ }, + { part: 'summation', labels: [] }, + { + part: 'before', + labels: ['preceded by', 'preceding events', 'preceding event'], + fact: true, + split: /;/, + }, + { + part: 'after', + labels: ['followed by', 'following events', 'following event'], + fact: true, + split: /;/, + }, + { + part: 'references', + labels: ['literature references', 'references'], + split: 'numbered', + names: true, + }, + { part: 'edits', labels: ['edit history'], split: 'dated', names: true }, + { part: 'entities', labels: ['entities found'], split: 'rows' }, + { part: 'interactors', labels: ['interactors found'], split: 'rows' }, +]; + +const ruleOf = (part: PartName): PartRule | undefined => RULES.find((r) => r.part === part); +const orderOf = (part: PartName): number => PART_NAMES.indexOf(part); + +/** A line's text as read: compatibility forms folded, no-break spaces as spaces. */ +const plain = (text: string): string => text.normalize('NFKC').replace(/\u2060/g, ''); + +/** A label at a line's start, whole and case aside, then a colon, white space or its end. */ +function labelled(text: string, labels: readonly string[]): string | null { + const read = plain(text); + for (const label of labels) { + const words = label.split(' ').map((w) => w.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')); + const match = new RegExp(`^\\s*${words.join('\\s*')}\\s*(?::|\\s|$)`, 'i').exec(read); + if (match) return read.slice(match[0].length); + } + return null; +} + +const STABLE_ID = /R-[A-Z]{2,4}-\d+/; +const STABLE_ID_AT_END = /R-[A-Z]{2,4}-\d+$/; + +/** A heading is set at least this much larger than its body text (as check-book reads it). */ +const HEADING_RATIO = 11.25 / 10.5; + +/** How far below where a link lands its target's first line may be. */ +const LANDING_PT = 100; + +/** Whether `a` comes before `b` in reading order: an earlier page, or higher on the same one. */ +const before = (a: Spot, b: Spot): boolean => a.page < b.page || (a.page === b.page && a.y > b.y); + +/** The size most of the text is set at. */ +function bodySize(lines: readonly Line[]): number { + const counts = new Map(); + for (const line of lines) { + const size = Math.round(line.size * 100) / 100; + counts.set(size, (counts.get(size) ?? 0) + fold(line.text).length); + } + return [...counts].sort(([, m], [, n]) => n - m)[0]?.[0] ?? 0; +} + +const FRONT_HEADINGS = { + introduction: /^introduction$/i, + analysis: /^analysis/i, + contents: /^(?:table of )?contents$/i, +} as const; + +type FrontName = keyof typeof FRONT_HEADINGS; + +const linkOn = (line: Line, box: Box): boolean => + box.page === line.page && line.y >= box.y1 - 2 && line.y <= box.y2 + 2; + +/** A layout reduced to FR-012's parts. */ +export function reduce(layout: Layout): Reduced { + const { lines, links, figures } = layout; + const body = bodySize(lines.filter((line) => line.page > 0)); + const heading = (line: Line) => line.size >= body * HEADING_RATIO; + + // each section, at its stable identifier's label; its name the heading above it, above its + // location where that comes first + const anchors = lines.flatMap((line, i) => { + const rest = labelled(line.text, ruleOf('stId')?.labels ?? []); + const stId = rest === null ? undefined : STABLE_ID.exec(fold(rest))?.[0]; + return stId === undefined ? [] : [{ i, stId }]; + }); + const starts = anchors.map(({ i }, n) => { + const floor = n > 0 ? (anchors[n - 1]?.i ?? 0) + 1 : 0; + let j = i - 1; + for (let k = i - 1; k >= Math.max(floor, i - 6); k--) { + if (labelled(lines[k]?.text ?? '', ruleOf('location')?.labels ?? []) !== null) { + j = k - 1; + break; + } + } + let start = j + 1; + while (start - 1 >= floor && lines[start - 1] && heading(lines[start - 1] as Line)) start--; + return start; + }); + const firstSection = starts[0] ?? lines.length; + + // the front parts' headings: the introduction's and the analysis's before the sections, the + // contents list's before them or after them all (today's files put it last) + const front = new Map(); + lines.forEach((line, i) => { + if (line.page === 0 || !heading(line) || starts.includes(i)) return; + const text = plain(line.text).trim(); + for (const name of Object.keys(FRONT_HEADINGS) as FrontName[]) { + if (front.has(name) || !FRONT_HEADINGS[name].test(text)) continue; + if (name !== 'contents' && i >= firstSection) continue; + if (name === 'contents' && i >= firstSection && i < (anchors.at(-1)?.i ?? 0)) continue; + front.set(name, i); + } + }); + const boundaries = [...starts, ...front.values(), lines.length].sort((a, b) => a - b); + const regionEnd = (from: number) => boundaries.find((b) => b > from) ?? lines.length; + const spotOf = (i: number): Spot | null => { + const line = lines[i]; + return line ? { page: line.page, y: line.y + line.size } : null; + }; + + // where an internal link lands: the first section or front part starting at or below it + // on its page, within LANDING_PT (Chromium's own destinations land up to about 76 pt + // above a heading's baseline, at the top of its page or of the space above it) + const targets = [ + ...anchors.map(({ stId }, n) => ({ at: starts[n] ?? 0, name: stId })), + ...[...front].map(([name, at]) => ({ at, name })), + ]; + const resolve = (landing: Spot | null): string | null => { + if (!landing) return null; + let best: { name: string; y: number } | null = null; + for (const { at, name } of targets) { + const line = lines[at]; + if (!line || line.page !== landing.page) continue; + if (line.y > landing.y + 2 || line.y < landing.y - LANDING_PT) continue; + if (!best || line.y > best.y) best = { name, y: line.y }; + } + return best?.name ?? null; + }; + + let shortEntries = 0; + const sections = anchors.map(({ i, stId }, n): ReducedSection => { + const start = starts[n] ?? i; + const end = regionEnd(i); + const label = lines[i]; + const left = label?.x ?? 0; + const from = spotOf(start); + const to = spotOf(end); + const own = figures.filter((box) => { + const top = { page: box.page, y: box.y2 }; + return from !== null && !before(top, from) && (to === null || before(top, to)); + }); + const parts: { part: PartName; texts: string[] }[] = []; + const headingLines: Line[] = []; + for (let k = start; k < i && lines[k] && heading(lines[k] as Line); k++) { + headingLines.push(lines[k] as Line); + } + parts.push({ part: 'name', texts: headingLines.map((line) => line.text) }); + let current: { part: PartName; texts: string[] } | null = null; + for (let k = start + headingLines.length; k < end; k++) { + const line = lines[k]; + if (!line) continue; + const order = current ? orderOf(current.part) : orderOf('name'); + let matched: { part: PartName; rest: string } | null = null; + for (const rule of RULES) { + if (orderOf(rule.part) <= order || rule.labels.length === 0) continue; + const rest = labelled(line.text, rule.labels); + if (rest !== null) { + matched = { part: rule.part, rest }; + break; + } + } + if (matched) { + current = { part: matched.part, texts: [matched.rest] }; + parts.push(current); + continue; + } + // a short entry: an event's name and stable identifier, linked to its section + const internal = links.some((link) => link.uri === null && linkOn(line, link.box)); + if (internal && STABLE_ID_AT_END.test(fold(line.text))) { + shortEntries++; + continue; + } + const afterFigure = own.some( + (box) => box.page < line.page || (box.page === line.page && box.y1 > line.y) + ); + const fact = current ? ruleOf(current.part)?.fact === true : true; + if ( + fact && + orderOf(current?.part ?? 'name') < orderOf('summation') && + (afterFigure || line.x <= left + 1) + ) { + current = { part: 'summation', texts: [line.text] }; + parts.push(current); + continue; + } + current?.texts.push(line.text); + } + return { + stId, + parts: parts.map(({ part, texts }) => ({ part, items: items(part, texts) })), + figure: figureKind( + own, + parts.some((p) => p.part === 'type') + ), + }; + }); + + function figureKind(own: readonly Box[], reaction: boolean): FigureKind { + const [box] = own; + if (!box) return 'none'; + for (const link of links) { + if (link.box.page !== box.page || link.uri === null) continue; + const overlaps = + link.box.x1 < box.x2 && + box.x1 < link.box.x2 && + link.box.y1 < box.y2 && + box.y1 < link.box.y2; + if (!overlaps) continue; + if (/\/PathwayBrowser\/#\/[^&]+&SEL=/.test(link.uri)) return 'selection'; + if (/\/PathwayBrowser\//.test(link.uri)) return 'diagram'; + if (/\/content\/detail\//.test(link.uri)) return 'reaction'; + } + return reaction ? 'reaction' : 'figure'; + } + + return { + cover: cover( + lines.filter((line) => line.page === 0), + figures + ), + introduction: introduction( + front.has('introduction') + ? lines.slice( + (front.get('introduction') ?? 0) + 1, + regionEnd(front.get('introduction') ?? 0) + ) + : lines.slice(0, firstSection).filter((line) => line.page > 0) + ), + analysis: front.has('analysis') + ? analysis( + lines.slice((front.get('analysis') ?? 0) + 1, regionEnd(front.get('analysis') ?? 0)) + ) + : null, + sections, + contents: front.has('contents') + ? contents( + lines.slice((front.get('contents') ?? 0) + 1, regionEnd(front.get('contents') ?? 0)), + (line) => + resolve( + links.find((link) => link.uri === null && link.landing && linkOn(line, link.box)) + ?.landing ?? null + ) + ) + : null, + shortEntries, + }; +} + +/** A part's items from its lines' texts, folded; hashed where they name people. */ +function items(part: PartName, texts: readonly string[]): string[] { + const rule = ruleOf(part); + let list: string[]; + const split = rule?.split; + if (split === 'numbered' || split === 'dated') { + const marker = split === 'numbered' ? /^\s*(?:\d+[.)]|\[\d+\])\s*/ : /^\s*\d{4}-\d{2}-\d{2}/; + const found: string[] = []; + for (const text of texts) { + const read = plain(text); + const last = found.pop(); + if (last !== undefined) found.push(marker.test(read) ? last : last + read); + if (marker.test(read)) found.push(split === 'numbered' ? read.replace(marker, '') : read); + // before the first item: the table's header, not an item + } + list = found; + } else if (split === 'rows') { + // each row a line; a first line with no digit is the table's header + const rows = texts.filter((text) => text.trim() !== ''); + list = rows.filter((text, i) => i > 0 || /\d/.test(text)); + } else if (split instanceof RegExp) { + list = texts.join(' ').split(split); + } else { + list = [texts.join(' ')]; + } + const folded = list.map(fold).filter((item) => item !== ''); + return rule?.names ? folded.map(hash) : folded; +} + +/** Lines before any of these on the cover end its names. */ +const AFTER_NAMES = new RegExp( + `\\d|institut|universit|licen[cs]|creative\\s*commons|excerpt|copyright|©|${INSTITUTIONS.map( + (i) => fold(i).slice(0, 12) + ).join('|')}`, + 'i' +); + +/** The cover's label before its names: pdf-export's, or a word that says what they are. */ +const NAMES_LABEL = /^\s*(?:authored|authors?|contributors?|edited|editors?|curated|curators?)\b/i; + +function cover(page: readonly Line[], figures: readonly Box[]): Reduced['cover'] { + const largest = Math.max(0, ...page.map((line) => line.size)); + const first = page.findIndex((line) => Math.abs(line.size - largest) < 0.5); + let last = first; + while (page[last + 1] && Math.abs((page[last + 1]?.size ?? 0) - largest) < 0.5) last++; + const titleLines = first < 0 ? [] : page.slice(first, last + 1); + const title = titleLines + .map((line) => plain(line.text).replace(/\s+/g, ' ').trim()) + .join(' ') + .trim(); + const below = titleLines.at(-1); + const diagram = figures.some((box) => box.page === 0 && below !== undefined && box.y2 < below.y); + const excerpt = page.some((line) => /excerpt/i.test(plain(line.text))); + + // the names: after their label, up to what follows them; or the lines written with U+2060 + const contributors = fold(CONTRIBUTORS).toLowerCase(); + const at = page.findIndex( + (line) => fold(line.text).toLowerCase() === contributors || NAMES_LABEL.test(plain(line.text)) + ); + let region: Line[] = []; + if (at >= 0) { + for (const line of page.slice(at + 1)) { + if (AFTER_NAMES.test(fold(line.text))) break; + region.push(line); + } + } else { + region = page.filter((line) => line.text.includes('\u2060')); + } + return { + title, + diagram, + excerpt, + names: hashNames(coverNames(region.map((line) => line.text).join('\n'))), + }; +} + +function introduction(region: readonly Line[]): Reduced['introduction'] { + const text = fold(region.map((line) => line.text).join(' ')); + const number = (pattern: RegExp): number | null => { + const found = pattern.exec(text)?.[1]; + return found === undefined ? null : Number(found.replace(/,/g, '')); + }; + return { + release: number(/release(?:version|number)?:?(\d+)/i), + pathways: number(/(\d[\d,]*)pathways/i), + reactions: number(/(\d[\d,]*)reactions/i), + }; +} + +/** The analysis's properties and the labels each is known by, longest first. */ +const PROPERTIES: readonly (readonly [string, readonly string[]])[] = [ + ['token', ['token']], + ['type', ['analysis type', 'type']], + ['species', ['species']], + ['projection', ['projected to human', 'projection']], + ['interactors', ['interactors included', 'include interactors', 'interactors']], + ['identifiers', ['identifiers found']], + ['pathways', ['pathways hit', 'pathways found']], + ['resource', ['resource']], +]; + +function analysis(region: readonly Line[]): Record { + const properties: Record = {}; + for (const line of region) { + for (const [name, labels] of PROPERTIES) { + if (name in properties || (name === 'identifiers' && 'identifiersFound' in properties)) { + continue; + } + const rest = labelled(line.text, labels); + if (rest === null) continue; + const value = fold(rest); + const yes = /^(?:yes|true)$/i.test(value); + const no = /^(?:no|false)$/i.test(value); + if (name === 'identifiers') { + const [found, total] = value.split(/of/i); + properties['identifiersFound'] = found ?? ''; + if (total !== undefined) properties['identifiersTotal'] = total; + } else { + properties[name] = yes ? 'yes' : no ? 'no' : value; + } + break; + } + } + return properties; +} + +/** The contents list's entries: each up to a line ending in its page's number. */ +function contents(region: readonly Line[], target: (line: Line) => string | null): ContentsItem[] { + const entries: ContentsItem[] = []; + let pending: Line[] = []; + const flush = () => { + if (pending.length === 0) return; + const title = fold(pending.map((line) => line.text).join(' ')).replace(/[.·…]*\d+$/, ''); + const landed = pending.map(target).find((t) => t !== null) ?? null; + if (title !== '') entries.push({ title, target: landed }); + pending = []; + }; + for (const line of region) { + pending.push(line); + if (/\d\s*$/.test(plain(line.text))) flush(); + } + flush(); + return entries; +} diff --git a/test/compare/compare.test.ts b/test/compare/compare.test.ts new file mode 100644 index 0000000..0efa116 --- /dev/null +++ b/test/compare/compare.test.ts @@ -0,0 +1,459 @@ +// FR-012's comparison (spec 001 T035): each part's difference is caught, list order is a +// difference of its own that `--expected` records, and no name is printed. The PDFs are +// printed here from a sample document (samples.ts): pdf-export's own, and a reference in +// `event-pdf`'s shape. No assertion here prints a text read from a PDF: a failure says +// which part, never what a name is. +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { after, before, describe, test } from 'node:test'; +import { fileURLToPath } from 'node:url'; + +import { + applyRecorded, + compareCommand, + compareReduced, + type Difference, +} from '../../src/compare/compare.ts'; +import { readLayout } from '../../src/compare/read.ts'; +import { coverNames, hashNames, reduce, type Reduced } from '../../src/compare/reduce.ts'; +import type { Document } from '../../src/model/types.ts'; +import { Printer } from '../../src/print/print.ts'; +import { + PEOPLE, + R1, + R2, + S, + joined, + printOurs, + printReference, + sampleAnalysis, + sampleDocument, + type ReferenceOptions, +} from './samples.ts'; + +/** A difference as the tests name it: part, field and kind, and its events. */ +const brief = (d: Difference) => + [d.part, d.field, d.kind, ...(d.events ?? [])].filter((w) => w !== undefined).join(' '); + +/** The resource, which `event-pdf` reports and pdf-export does not print. */ +const RESOURCE = 'analysis resource missing'; + +/** Every surname and initial of the placeholders, as a printed name could show them. */ +const NAME_PARTS = PEOPLE.flatMap((name) => name.split(/[\s,]+/)).filter((w) => w.length > 2); + +/** Whether a text holds any placeholder's name, or any part of one; never prints the text. */ +function assertNoName(text: string, where: string): void { + for (const part of NAME_PARTS) { + assert.ok(!text.includes(part), `a name was printed in ${where}`); + } +} + +describe('compare (FR-012)', { timeout: 300_000 }, () => { + let printer: Printer; + let folder: string; + let ours: Uint8Array; + let oursReduced: Reduced; + const analysis = sampleAnalysis(); + + const reduced = async (pdf: Uint8Array) => reduce(await readLayout(pdf)); + /** The reference printed from a document changed by `change`, with these options. */ + const reference = async ( + change: (doc: Document) => void = () => undefined, + options: ReferenceOptions = {} + ) => { + const doc = sampleDocument(); + change(doc); + return printReference(printer, doc, { analysis, ...options }); + }; + /** The differences between a reference and pdf-export's sample, briefly. */ + const differences = async (ref: Uint8Array) => + compareReduced('sample.pdf', await reduced(ref), oursReduced).map(brief); + + before(async () => { + folder = mkdtempSync(join(tmpdir(), 'pdf-export-compare-')); + printer = await Printer.start(); + ours = await printOurs(printer, sampleDocument(), analysis); + oursReduced = await reduced(ours); + }); + after(async () => { + await printer.close(); + rmSync(folder, { recursive: true, force: true }); + }); + + test('the same document in both layouts differs in nothing but the resource', async () => { + assert.deepEqual(await differences(await reference()), [RESOURCE]); + }); + + test('each section is read with its parts, figure and items, names held as hashes', () => { + const byId = new Map(oursReduced.sections.map((s) => [s.stId, s])); + assert.deepEqual([...byId.keys()], ['R-HSA-9000001', R1.stId, S.stId, R2.stId]); + const parts = (stId: string) => byId.get(stId)?.parts.map((p) => p.part); + assert.deepEqual(parts(R1.stId), [ + 'name', + 'location', + 'stId', + 'type', + 'compartments', + 'inferredFrom', + 'summation', + 'after', + 'references', + 'edits', + 'entities', + ]); + // a summation beginning with a label's word is still a summation + assert.deepEqual(parts(S.stId), [ + 'name', + 'location', + 'stId', + 'compartments', + 'diseases', + 'summation', + 'references', + 'edits', + ]); + assert.deepEqual( + oursReduced.sections.map((s) => s.figure), + ['diagram', 'reaction', 'selection', 'reaction'] + ); + const edits = byId.get(R1.stId)?.parts.find((p) => p.part === 'edits')?.items ?? []; + assert.equal(edits.length, 2); + assert.ok( + edits.every((item) => /^[0-9a-f]{16}$/.test(item)), + 'edits are held as hashes' + ); + assertNoName(JSON.stringify(oursReduced), 'the reduced form'); + }); + + describe('the cover', () => { + test('its distinct names: another name, or one more, is caught as a count and hash', async () => { + const more = await differences( + await reference(undefined, { names: [...PEOPLE.slice(0, 3)] }) + ); + assert.deepEqual(more, ['cover names count', RESOURCE]); + const other = await differences( + await reference(undefined, { names: [PEOPLE[0], PEOPLE[3]] }) + ); + assert.deepEqual(other, ['cover names value', RESOURCE]); + // each side as a count and a hash, nothing else + const [names] = compareReduced( + 'x', + await reduced(await reference(undefined, { names: [PEOPLE[3]] })), + oursReduced + ); + assert.deepEqual(Object.keys(names?.reference ?? {}), ['count', 'hash']); + assert.deepEqual(Object.keys(names?.new ?? {}), ['count', 'hash']); + }); + + test('its title, diagram and excerpt note', async () => { + const changed = await differences( + await reference((doc) => { + doc.cover.title = 'Another title'; + doc.cover.figure = null; + doc.cover.excerpt = false; + }) + ); + assert.deepEqual(changed, [ + 'cover title value', + 'cover diagram value', + 'cover excerpt value', + RESOURCE, + ]); + }); + + test('names written with U+2060 and no-break spaces read as the same names', () => { + const plain = PEOPLE.slice(0, 3); + const written = plain.map(joined).join(', '); + assert.ok(written.includes('\u2060') && written.includes('\u00a0')); + assert.deepEqual(hashNames(coverNames(written)), hashNames(plain)); + // each once, however often it is written + assert.deepEqual(hashNames(coverNames(`${written}, ${joined(PEOPLE[0])}`)), hashNames(plain)); + assert.deepEqual(hashNames(coverNames(plain.join('; '))), hashNames(plain)); + assert.notDeepEqual(hashNames(coverNames(written)), hashNames(PEOPLE.slice(0, 2))); + }); + }); + + test("the introduction's release and counts", async () => { + const changed = await differences(await reference(undefined, { release: 97 })); + assert.deepEqual(changed, ['introduction release value', RESOURCE]); + const counts = await differences( + await reference((doc) => { + doc.release = { version: 98, pathways: 3, reactions: 1 }; + }) + ); + assert.deepEqual(counts, [ + 'introduction pathways value', + 'introduction reactions value', + RESOURCE, + ]); + }); + + test("the analysis's properties, not its explanatory text", async () => { + const p = analysis.properties; + const changed = await differences( + await reference(undefined, { + properties: [ + ['Token', p.token], + ['Analysis type', p.type], + ['Species', p.species], + ['Projection', 'false'], + ['Interactors', String(p.interactors)], + ['Identifiers found', '3 of 6'], + ['Pathways found', '5'], + ['Resource', 'TOTAL'], + ], + }) + ); + assert.deepEqual(changed, [ + 'analysis identifiersTotal value', + 'analysis pathways value', + 'analysis projection value', + RESOURCE, + ]); + }); + + describe('the sections', () => { + test("a part's text, a part missing, and a list's order, each by its section", async () => { + const changed = await differences( + await reference((doc) => { + for (const place of doc.places) { + if (!place.full) continue; + const { section } = place; + if (section.event.stId === R1.stId) section.summations = ['

Other text.

']; + if (section.event.stId === S.stId) section.diseases = []; + if (section.event.stId === R2.stId) section.event.compartments.reverse(); + } + }) + ); + assert.deepEqual(changed, [ + RESOURCE, + `section summation text ${R1.stId}`, + `section diseases extra ${S.stId}`, + `section compartments order ${R2.stId}`, + ]); + }); + + test('a section missing, and the sections in another order', async () => { + const missing = await differences( + await reference((doc) => { + doc.places = doc.places.filter((p) => !p.full || p.section.event.stId !== S.stId); + }) + ); + assert.ok(missing.includes(`sections extra ${S.stId}`), missing.join('\n')); + const moved = await differences( + await reference((doc) => { + const [first, ...rest] = doc.places; + if (first) doc.places = [first, ...rest.reverse()]; + }) + ); + assert.ok(moved.includes(`sections order ${R1.stId} ${R2.stId}`), 'the order is missed'); + }); + + test('the references and the edit history, compared by their hashes', async () => { + const changed = await differences( + await reference((doc) => { + for (const place of doc.places) { + if (!place.full || place.section.event.stId !== R2.stId) continue; + place.section.edits = [...place.section.edits].reverse(); + const [first] = place.section.references; + if (first) place.section.references = [{ ...first, authors: [PEOPLE[0]] }]; + } + }) + ); + assert.deepEqual(changed, [ + RESOURCE, + `section references text ${R2.stId}`, + `section edits order ${R2.stId}`, + ]); + }); + }); + + test("the contents list's entries, their order and their targets", async () => { + const entries = (sampleDocument().contents ?? []).map((c) => ({ + title: c.title, + target: c.target, + })); + const fewer = await differences(await reference(undefined, { contents: entries.slice(1) })); + assert.deepEqual(fewer, [RESOURCE, 'contents extra']); + const [first, second, ...rest] = entries; + const swapped = await differences( + await reference(undefined, { contents: first && second ? [second, first, ...rest] : [] }) + ); + assert.deepEqual(swapped, [RESOURCE, 'contents order']); + const retargeted = await differences( + await reference(undefined, { + contents: entries.map((e) => + e.target === `s-${R1.stId}` ? { ...e, target: `s-${S.stId}` } : e + ), + }) + ); + assert.deepEqual(retargeted, [RESOURCE, 'contents target']); + }); + + test('the figures: one missing, and one of another kind', async () => { + const missing = await differences(await reference(undefined, { noFigure: [R1.stId] })); + assert.deepEqual(missing, [RESOURCE, `figure extra ${R1.stId}`]); + // pdf-export's own on both sides, where each figure's kind is read + const doc = sampleDocument(); + for (const place of doc.places) { + if (place.full && place.section.event.stId === S.stId) { + place.section.figure = { kind: 'diagram', stId: S.stId }; + } + } + const other = await reduced(await printOurs(printer, doc, analysis)); + assert.deepEqual(compareReduced('x', oursReduced, other).map(brief), [`figure kind ${S.stId}`]); + }); + + test('a short entry is counted, apart from the section before it', async () => { + const doc = sampleDocument(); + doc.places.push({ full: false, anchor: `s-${R1.stId}`, event: R1, path: [], depth: 1 }); + const other = await reduced(await printOurs(printer, doc, analysis)); + assert.deepEqual(compareReduced('x', oursReduced, other).map(brief), ['short-entries count']); + }); + + describe('the command', () => { + let refFolder: string; + let newFolder: string; + let expected: string; + const run = async (args: string[]) => { + let out = ''; + let err = ''; + const code = await compareCommand(args, { + out: (text) => (out += text), + err: (text) => (err += text), + }); + return { code, out, err }; + }; + + before(async () => { + refFolder = join(folder, 'reference'); + newFolder = join(folder, 'new'); + mkdirSync(refFolder); + mkdirSync(newFolder); + // names that differ, sections in another order, and a file on each side alone + const ref = await reference( + (doc) => { + const [first, ...rest] = doc.places; + if (first) doc.places = [first, ...rest.reverse()]; + }, + { names: [...PEOPLE] } + ); + writeFileSync(join(refFolder, 'Alpha signalling pathway.pdf'), ref); + writeFileSync(join(newFolder, 'Alpha signalling pathway.pdf'), ours); + writeFileSync(join(refFolder, 'Only today.pdf'), ref); + writeFileSync(join(newFolder, 'Only new.pdf'), ours); + expected = join(folder, 'expected.json'); + }); + + test('folders file by file, a file on one side only a difference; exit 1', async () => { + const { code, out, err } = await run([refFolder, newFolder]); + assert.equal(code, 1); + assert.equal(err, ''); + const report = JSON.parse(out) as { differences: Difference[] }; + assert.deepEqual( + report.differences.map((d) => `${d.file}: ${brief(d)}`), + [ + 'Only today.pdf: file missing', + 'Only new.pdf: file extra', + 'Alpha signalling pathway.pdf: cover names count', + 'Alpha signalling pathway.pdf: analysis resource missing', + `Alpha signalling pathway.pdf: sections order ${R1.stId} ${R2.stId}`, + ] + ); + }); + + test('--expected: recorded differences pass, with their reasons; exit 0 only if all are', async () => { + const records = [ + { part: 'file', kind: 'missing', reason: 'a sample' }, + { part: 'file', kind: 'extra', reason: 'a sample' }, + { part: 'cover', field: 'names', kind: 'count', reason: 'a sample' }, + { part: 'analysis', field: 'resource', kind: 'missing', reason: 'not printed yet' }, + { part: 'sections', kind: 'order', reason: "today's lists are in the database's order" }, + ]; + writeFileSync(expected, JSON.stringify({ differences: records })); + const all = await run([refFolder, newFolder, '--expected', expected]); + assert.equal(all.code, 0); + const report = JSON.parse(all.out) as { + counts: { differences: number; recorded: number; unrecorded: number }; + differences: Difference[]; + }; + assert.deepEqual(report.counts, { differences: 5, recorded: 5, unrecorded: 0 }); + assert.ok(report.differences.every((d) => typeof d.recorded === 'string')); + // the order's record left out: that difference alone fails the run + writeFileSync(expected, JSON.stringify({ differences: records.slice(0, 4) })); + const some = await run([refFolder, newFolder, `--expected=${expected}`]); + assert.equal(some.code, 1); + const unrecorded = (JSON.parse(some.out) as { differences: Difference[] }).differences.filter( + (d) => d.recorded === undefined + ); + assert.deepEqual(unrecorded.map(brief), [`sections order ${R1.stId} ${R2.stId}`]); + }); + + test('a record naming events covers only those', () => { + const difference: Difference = { + file: 'f.pdf', + part: 'section', + field: 'summation', + kind: 'text', + count: 2, + events: [R1.stId, R2.stId], + }; + const split = applyRecorded( + [difference], + [{ part: 'section', kind: 'text', events: [R1.stId], reason: 'rewritten' }] + ); + assert.deepEqual( + split.map((d) => [d.events, d.count, d.recorded]), + [ + [[R1.stId], 1, 'rewritten'], + [[R2.stId], 1, undefined], + ] + ); + // a record for another field covers nothing + const other = applyRecorded( + [difference], + [{ part: 'section', field: 'references', kind: 'text', reason: 'x' }] + ); + assert.equal(other[0]?.recorded, undefined); + }); + + test('no name is printed, in the report or its errors', async () => { + const differ = await run([refFolder, newFolder]); + assertNoName(differ.out + differ.err, 'the report'); + const pdfs = await run([ + join(refFolder, 'Alpha signalling pathway.pdf'), + join(newFolder, 'Alpha signalling pathway.pdf'), + ]); + assert.equal(pdfs.code, 1); + assertNoName(pdfs.out + pdfs.err, 'the report of two PDFs'); + // a reference whose names are all others: still only a count and a hash + const report = JSON.parse(pdfs.out) as { differences: Difference[] }; + assert.ok(report.differences.some((d) => d.field === 'names')); + }); + + test('bad arguments: exit 2, saying what is needed', async () => { + assert.equal((await run([refFolder])).code, 2); + const mixed = await run([refFolder, join(newFolder, 'Only new.pdf')]); + assert.equal(mixed.code, 2); + assert.match(mixed.err, /both be PDFs or both be folders/); + assert.equal((await run([refFolder, newFolder, '--depth', '1'])).code, 2); + writeFileSync(expected, JSON.stringify({ differences: [{ part: 'cover' }] })); + const unrecorded = await run([refFolder, newFolder, '--expected', expected]); + assert.equal(unrecorded.code, 2); + assert.match(unrecorded.err, /needs a part, a kind and a reason/); + }); + + test('it is a command of the command line', () => { + const result = spawnSync( + process.execPath, + [fileURLToPath(new URL('../../src/cli.ts', import.meta.url)), 'compare'], + { encoding: 'utf8', env: { PATH: process.env['PATH'] ?? '' } } + ); + assert.equal(result.status, 2); + assert.match(result.stderr, /Two paths are needed/); + }); + }); +}); diff --git a/test/compare/samples.ts b/test/compare/samples.ts new file mode 100644 index 0000000..401eb50 --- /dev/null +++ b/test/compare/samples.ts @@ -0,0 +1,405 @@ +// Samples for FR-012's comparison's tests (spec 001 T035): one small document, printed +// both by pdf-export's own layout and as a reference in the shape of `event-pdf`'s files +// (the spec's Background and research R2: the contents list after the sections, its parts +// written `Label: value`, its figures as images, the cover's names with U+2060 between +// their characters and no-break spaces between their words). The reference's wording is +// written here; none is today's. Every name is a placeholder (FR-013): real `event-pdf` +// files are never used in tests. +import { crc32, deflateSync } from 'node:zlib'; + +import type { EventRef, Reference } from '../../src/graph/queries.ts'; +import type { FigureSvgs } from '../../src/layout/figure.ts'; +import { figureKey } from '../../src/layout/figure.ts'; +import { attribute, plainText } from '../../src/layout/html.ts'; +import { layoutSingle, pageStyles } from '../../src/layout/document.ts'; +import { reference } from '../../src/layout/section.ts'; +import { INSTITUTIONS } from '../../src/layout/text.ts'; +import type { + AnalysisProperties, + Document, + Figure, + Found, + Section, +} from '../../src/model/types.ts'; +import type { Printer } from '../../src/print/print.ts'; + +/** The placeholder people: authors of edits and of references. */ +export const PEOPLE = ['Placeholder, A', 'Example, B', 'Sample, C D', 'Dummy, E'] as const; + +const SITE = 'https://site.example'; + +const ref = (dbId: number, stId: string, name: string): EventRef => ({ dbId, stId, name }); + +const P = ref(1, 'R-HSA-9000001', 'Alpha signalling pathway'); +const R1 = ref(2, 'R-HSA-9000002', 'Beta binds gamma'); +const S = ref(3, 'R-HSA-9000003', 'Delta subpathway'); +const R2 = ref(4, 'R-HSA-9000004', 'Epsilon is phosphorylated'); + +function literature(dbId: number, title: string, authors: string[]): Reference { + return { + dbId, + kind: 'article', + title, + chapterTitle: null, + journal: 'Journal of Examples', + volume: 12, + pages: '1-9', + year: 2001, + pubMedId: 1000 + dbId, + isbn: null, + url: null, + publisher: null, + authors, + moreAuthors: false, + }; +} + +function section( + event: EventRef, + kind: 'pathway' | 'reaction', + depth: number, + path: EventRef[], + figure: Figure | null, + more: Partial
& { category?: string; before?: EventRef[]; after?: EventRef[] } = {} +): Section { + const { category = null, before = [], after = [], ...rest } = more; + return { + anchor: `s-${event.stId}`, + event: { + ...event, + kind, + schemaClass: kind === 'pathway' ? 'Pathway' : 'Reaction', + speciesName: 'Homo sapiens', + category, + hasDiagram: figure?.kind === 'diagram', + hasEHLD: false, + isInferred: false, + isInDisease: false, + species: { dbId: 48887, name: 'Homo sapiens' }, + compartments: [{ dbId: 70101, name: 'cytosol' }], + before, + after, + }, + path, + figure, + diseases: [], + inferredFrom: [], + summations: [`

${plainText(event.name)} is a sample event, written for these tests.

`], + references: [literature(event.dbId * 10, `A study of ${event.name}`, [PEOPLE[2], PEOPLE[3]])], + edits: [ + { date: '2020-01-01', kinds: ['authored'], authors: [PEOPLE[0]] }, + { date: '2021-02-03', kinds: ['reviewed'], authors: [PEOPLE[1]] }, + ], + depth, + ...rest, + }; +} + +/** The sample: a pathway at depth 1 with two reactions and a subpathway, with an analysis. */ +export function sampleDocument(): Document { + const diagram: Figure = { kind: 'diagram', stId: P.stId }; + const sections = [ + section(P, 'pathway', 0, [], diagram, { + references: [ + literature(11, 'A first study', [PEOPLE[2]]), + literature(12, 'A second study', [PEOPLE[3], PEOPLE[2]]), + ], + }), + section( + R1, + 'reaction', + 1, + [P], + { kind: 'reaction', stId: R1.stId }, + { + category: 'binding', + after: [R2], + inferredFrom: [ + { ...ref(20, 'R-MMU-9000020', 'Beta binds gamma'), speciesName: 'Mus musculus' }, + ], + } + ), + section( + S, + 'pathway', + 1, + [P], + { kind: 'ancestor', stId: P.stId, highlight: [R1.stId] }, + { + diseases: ['influenza'], + // a summation whose first word is a part's label, which only labels its part where it comes + summations: ['

Type 2 cells hold the sample subpathway, written for these tests.

'], + } + ), + section( + R2, + 'reaction', + 1, + [P], + { kind: 'reaction', stId: R2.stId }, + { + category: 'transition', + before: [R1], + } + ), + ]; + for (const s of sections) { + if (s.event.stId === R2.stId) { + s.event.compartments = [ + { dbId: 70101, name: 'cytosol' }, + { dbId: 7660, name: 'nucleoplasm' }, + ]; + } + } + return { + kind: 'single', + release: { version: 98, pathways: 2, reactions: 2 }, + cover: { + title: P.name, + release: null, + figure: diagram, + excerpt: true, + names: [PEOPLE[0], PEOPLE[1]], + }, + places: sections.map((s) => ({ full: true as const, section: s })), + contents: [ + { title: 'Introduction', target: 'introduction', depth: 0 }, + { title: 'Analysis properties', target: 'analysis', depth: 0 }, + ...sections.map((s) => ({ title: s.event.name, target: s.anchor, depth: s.depth })), + { title: 'Contents', target: 'contents', depth: 0 }, + ], + }; +} + +export function sampleAnalysis(): { properties: AnalysisProperties; found: Map } { + return { + properties: { + token: 'MjAyNjExMDQxMjAwMDBfMQ%3D%3D', + type: 'OVERREPRESENTATION', + species: 'Homo sapiens', + projection: true, + interactors: false, + identifiersFound: 3, + identifiersTotal: 5, + pathwaysHit: 4, + }, + found: new Map([ + [ + R1.stId, + { + entities: [{ identifier: 'P00001', mapsTo: ['P00001'], resource: 'UniProt' }], + interactors: [], + }, + ], + ]), + }; +} + +/** A figure: a few shapes and a label. */ +function svg(label: string): string { + return `${plainText(label)}`; +} + +/** Each section's figure, and the cover's, as pdf-export's layout reads them. */ +export function sampleFigures(doc: Document): FigureSvgs { + const figures = new Map(); + for (const place of doc.places) { + if (place.full && place.section.figure) { + figures.set(figureKey(place.section.figure), svg(place.section.event.name)); + } + } + if (doc.cover.figure) figures.set(figureKey(doc.cover.figure), svg(doc.cover.title)); + return figures; +} + +/** pdf-export's own single-event PDF of a document. */ +export async function printOurs( + printer: Printer, + doc: Document, + analysis: ReturnType | null +): Promise { + const { html } = layoutSingle(doc, { + site: SITE, + date: new Date('2026-11-04T12:00:00Z'), + figures: sampleFigures(doc), + styles: pageStyles(), + analysis, + }); + return printer.single(html); +} + +/** A PNG of one colour, as `event-pdf`'s raster figures stand in. */ +export function png(width: number, height: number): string { + const chunk = (type: string, data: Buffer) => { + const length = Buffer.alloc(4); + length.writeUInt32BE(data.length); + const body = Buffer.concat([Buffer.from(type, 'latin1'), data]); + const crc = Buffer.alloc(4); + crc.writeUInt32BE(crc32(body)); + return Buffer.concat([length, body, crc]); + }; + const header = Buffer.alloc(13); + header.writeUInt32BE(width, 0); + header.writeUInt32BE(height, 4); + header.set([8, 2, 0, 0, 0], 8); + const row = Buffer.concat([Buffer.from([0]), Buffer.alloc(width * 3, 0x9c)]); + const pixels = deflateSync(Buffer.concat(Array.from({ length: height }, () => row))); + const file = Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk('IHDR', header), + chunk('IDAT', pixels), + chunk('IEND', Buffer.alloc(0)), + ]); + return `data:image/png;base64,${file.toString('base64')}`; +} + +/** A name as the reference's cover writes it: U+2060 between its characters, no-break spaces between its words. */ +export const joined = (name: string): string => + Array.from(name.replace(/ /g, '\u00a0')).join('\u2060'); + +export interface ReferenceOptions { + analysis?: ReturnType | null; + /** The cover's names, if not the document's. */ + names?: readonly string[]; + /** The release line's number and the counts, if not the document's. */ + release?: number; + /** Sections drawn with no figure, by stable identifier. */ + noFigure?: readonly string[]; + /** The contents list's entries, if not the document's. */ + contents?: { title: string; target: string }[]; + /** The analysis's properties as written, if not those given. */ + properties?: [string, string][]; +} + +/** A reference PDF in `event-pdf`'s shape, of a document. */ +export async function printReference( + printer: Printer, + doc: Document, + options: ReferenceOptions = {} +): Promise { + const t = (s: string) => plainText(s); + const a = (href: string, s: string) => `${t(s)}`; + const names = options.names ?? doc.cover.names; + const parts: string[] = []; + // the cover: a raster logo, the title, a raster diagram, the names, then the rest + parts.push(`
+ +

${t(doc.cover.title)}

+${doc.cover.figure ? `` : ''} +

Authors and editors

+

${names.map(joined).join('; ')}

+

${INSTITUTIONS.map(t).join('
')}

+

Shared under a Creative Commons licence.

+

4 November 2026

+${doc.cover.excerpt ? '

An excerpt of the whole book.

' : ''} +
`); + const { release } = doc; + parts.push(`

Introduction

+

Sample introduction text, written for these tests.

+

Release ${String(options.release ?? release.version)}. It holds ${String(release.pathways)} pathways and ${String(release.reactions)} reactions.

`); + const analysis = options.analysis ?? null; + if (analysis) { + const p = analysis.properties; + const rows: [string, string][] = options.properties ?? [ + ['Token', p.token], + ['Analysis type', p.type], + ['Species', p.species], + ['Projection', String(p.projection)], + ['Interactors', String(p.interactors)], + ['Identifiers found', `${String(p.identifiersFound)} of ${String(p.identifiersTotal)}`], + ['Pathways found', String(p.pathwaysHit)], + ['Resource', 'TOTAL'], + ]; + parts.push(`

Analysis properties

+${rows.map(([k, v]) => `

${t(k)}: ${t(v)}

`).join('\n')} +

Explanatory text on the analysis, written for these tests.

`); + } + const sections: string[] = []; + for (const place of doc.places) { + if (!place.full) continue; + const s = place.section; + const e = s.event; + const lines: string[] = []; + lines.push( + `

${a(`${SITE}/content/detail/${e.stId}`, e.name)}

` + ); + if (s.path.length > 0) { + lines.push(`

Location: ${s.path.map((p) => a(`#s-${p.stId}`, p.name)).join(' > ')}

`); + } + lines.push(`

Stable Identifier: ${t(e.stId)}

`); + if (e.kind === 'reaction' && e.category) lines.push(`

Type: ${t(e.category)}

`); + if (e.compartments.length > 0) { + lines.push(`

Compartment: ${e.compartments.map((c) => t(c.name)).join(', ')}

`); + } + if (s.diseases.length > 0) lines.push(`

Disease: ${s.diseases.map(t).join(', ')}

`); + if (s.inferredFrom.length > 0) { + lines.push( + `

Inferred from: ${s.inferredFrom.map((i) => `${t(i.name)} (${t(i.speciesName)})`).join('; ')}

` + ); + } + if (s.figure && !options.noFigure?.includes(e.stId)) { + lines.push(`

`); + } + lines.push(...s.summations); + if (e.kind === 'reaction') { + if (e.before.length > 0) { + lines.push( + `

Preceding events: ${e.before.map((b) => a(`#s-${b.stId}`, b.name)).join('; ')}

` + ); + } + if (e.after.length > 0) { + lines.push( + `

Following events: ${e.after.map((b) => a(`#s-${b.stId}`, b.name)).join('; ')}

` + ); + } + } + if (s.references.length > 0) { + lines.push( + `

Literature References

    ${s.references.map((r) => `
  1. ${reference(r)}
    2. `).join('')}
` + ); + } + if (s.edits.length > 0) { + lines.push( + `

Edit history

${s.edits + .map( + (row) => + `` + ) + .join('')}
DateActionAuthor
${t(row.date)}${t(row.kinds.map((k) => k.charAt(0).toUpperCase() + k.slice(1)).join(', '))}${t(row.authors.join('; '))}
` + ); + } + const found = analysis?.found.get(e.stId); + if (found && found.entities.length > 0) { + lines.push( + `

Entities found

${found.entities + .map( + (r) => + `` + ) + .join('')}
InputIdentifiersResource
${t(r.identifier)}${t(r.mapsTo.join(', '))}${t(r.resource)}
` + ); + } + sections.push(`
${lines.join('\n')}
`); + } + parts.push(...sections); + const entries = + options.contents ?? (doc.contents ?? []).map((c) => ({ title: c.title, target: c.target })); + if (entries.length > 0) { + parts.push(`

Table of contents

+${entries.map((c, i) => `

${a(`#${c.target}`, c.title)} ........ ${String(i + 2)}

`).join('\n')}
`); + } + const html = ` +${parts.join('\n')} +`; + return printer.print(html, false); +} + +export { P, R1, R2, S }; From 1ac54a36410e71d266058c55d74095a38927e2d8 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Wed, 4 Nov 2026 04:20:24 +0000 Subject: [PATCH 2/2] Spec 001 T035: the PR's number Co-Authored-By: Claude Opus 5.5 --- specs/001-pathway-pdfs/tasks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/001-pathway-pdfs/tasks.md b/specs/001-pathway-pdfs/tasks.md index e6736e1..ce26b3a 100644 --- a/specs/001-pathway-pdfs/tasks.md +++ b/specs/001-pathway-pdfs/tasks.md @@ -599,7 +599,7 @@ passes. Tests on fixture PDFs: each part's difference is caught, and no name is printed. - - **Done (#PR):** `pdf-export compare [--expected ]`, wired into `src/cli.ts`, as contracts/cli.md now has it rather than this task's ` `: the book is a folder of PDFs, so each side is a PDF or a folder, a folder compared file by file by name, and a file on one side only is a difference. + - **Done (#165):** `pdf-export compare [--expected ]`, wired into `src/cli.ts`, as contracts/cli.md now has it rather than this task's ` `: the book is a folder of PDFs, so each side is a PDF or a folder, a folder compared file by file by name, and a file on one side only is a difference. - **Reading** (`src/compare/read.ts`): pdf.js's lines of text (pieces sharing a baseline), links and figures. A page's top and bottom 50 pt (running headers, page numbers) and the text inside a figure (a vector diagram's labels) are left out. A figure is a cluster of images and painted paths, neither thin nor the whole page, at least 60 pt each way, so a raster diagram and a vector one are both found. - **Reducing** (`src/compare/reduce.ts`), with no knowledge of which program wrote the file: - the cover: its title (its largest text), a diagram below it, an excerpt note, and its distinct names, counted and hashed (the names are a run of characters joined by U+2060 or no-break spaces where U+2060 is there to read, research R2; otherwise parted by semicolons, as pdf-export writes them);