Add syndication feed support

CHANGELOG.md

@@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Terminology server: a syndication feed (`modules.tx.syndication.enabled`, off by default) - the content the library loads, published at `{endpoint}/syndication` as a [Terminology Syndication Feed](https://build.fhir.org/ig/FHIR/fhir-syndication-ig/en/) (Atom, NCTS ASF profile) for Ontoserver, other FHIRsmith servers and people. As the library loads, each source is recorded in a store in `terminology-cache/syndication`, from the time syndication is turned on: npm, url and core packages (linked at npm.fhir.org, `FHIR_Package`), the versioned resources in file and folder sources (copied into the store), and the binaries (SNOMED caches, LOINC databases etc, linked at `base.url`, in a FHIRsmith category scheme, with `<rights>`) that are published there - not internal sources, OCL, `#current` packages or the full CPT database. Every link has a SHA-256. The store only grows: no retracts, and a later update to the same content is a new entry (so `contentItemVersion` is not unique, deliberately). The feed is paged as RFC 5005 archived feeds of 30 entries; a browser gets it as a page, and the tx pages have a Syndication nav item and feed autodiscovery link
 - Registry: the software page shows one row per server rather than one per endpoint - the endpoints of a server (its FHIR versions) that run the same software and version are collapsed under their common base URL, with the FHIR versions listed and any unreachable endpoint named. Endpoints running different software keep their own rows
 - Registry: the API can be secured (`modules.registry.security`), in any combination of three ways: a single `apiToken` every client sends; `users` the administrator manages on `/tx-reg/admin/users`, each with their own tokens (256 random bits, stored as a SHA-256 hash, shown once, labelled, optionally expiring, revocable; usage counted per token) and optionally a password to manage their own tokens on `/tx-reg/account`; and `oidc`, JWTs from an OpenID Connect provider verified against its published keys for issuer, audience, expiry and a required scope or role. Only the JSON API is closed - the HTML pages, the OpenAPI description and the login page stay open - and a request without a valid token gets 401 with `WWW-Authenticate: Bearer`. Failed authentications are limited per address (`security.failedAuthPerWindow`, default 60 in 15 minutes, then 429; successful requests aren't counted). Tokens are sent as `Authorization: Bearer {token}` or `Api-Key: {token}`, the headers the HL7 Java tooling already sends for a server in fhir-settings.json. Without `security` the registry is open, as before. New dependency: `jose`
 - Registry: documents (`/tx-reg/docs`, when `security.users` is configured) - markdown or JSON documents the registry stores and serves, each with a name (its address), a type, whether it is public, and when and by whom it was last edited. Public documents are readable by anyone, the others by any logged in user (a browser session, never an API token); administrators create, edit, rename and delete them in a plain text editor. A browser gets the document rendered (markdown as written, raw HTML included - the authors are the administrators; JSON pretty-printed), anything else the raw file with its media type. Every change is logged with the full content, and the history page shows each past version

config-template.json

@@ -253,6 +253,14 @@
       // several instances can route cached requests back to the instance that holds the
       // cache. Leave unset for a single server. See nginx.md.
       // "instanceCode": "tx1",
+      // The syndication feed (Terminology Syndication Feed IG) at <endpoint>/syndication. When
+      // it is on, the packages, resource files and published binaries the library loads are
+      // recorded in <data>/terminology-cache/syndication as they load - only from the time it
+      // is turned on - and the feed is built from that record. The full CPT database is never
+      // syndicated. Every endpoint shares the one store.
+      "syndication": {
+        "enabled": false
+      },
       // $closure: closure tables are kept on the server, across sessions and restarts, so
       // the operation is off unless it is turned on here. When it is on, only one server
       // instance may use the database - route every $closure request to it (see nginx.md).

server.js

@@ -282,7 +282,7 @@
     try {
       serverLog.info('Initializing module: tx...');
       modules.tx = new TXModule(stats.forModule('tx'));
-      await modules.tx.initialize(config.modules.tx, app);
+      await modules.tx.initialize(config.modules.tx, app, { hostName: config.hostName });
     } catch (error) {
       serverLog.error('Failed to initialize TX module:', error);
       throw error;
@@ -856,7 +856,7 @@
     }
     // Run usage tracker in background after startup
     if (modules.tx && modules.tx.library) {
       setImmediate(async () => {
         try {
           serverLog.info('Starting ConceptUsageTracker scan...');
           let count = await modules.tx.usageTracker.scanValueSets(modules.tx.library);

tests/tx/fixtures/syndication-folder/codesystem.json

@@ -0,0 +1,14 @@
+{
+  "resourceType": "CodeSystem",
+  "id": "syndication-test",
+  "url": "http://example.org/fhir/CodeSystem/syndication-test",
+  "version": "1.0.0",
+  "name": "SyndicationTest",
+  "title": "Syndication Test",
+  "description": "A code system for the syndication feed tests",
+  "status": "active",
+  "content": "complete",
+  "concept": [
+    { "code": "a", "display": "A" }
+  ]
+}

tests/tx/fixtures/syndication-folder/valueset.json

@@ -0,0 +1,10 @@
+{
+  "resourceType": "ValueSet",
+  "id": "syndication-test-unversioned",
+  "url": "http://example.org/fhir/ValueSet/syndication-test-unversioned",
+  "name": "SyndicationTestUnversioned",
+  "status": "active",
+  "compose": {
+    "include": [ { "system": "http://example.org/fhir/CodeSystem/syndication-test" } ]
+  }
+}

tests/tx/fixtures/test-library-syndication.yaml

@@ -0,0 +1,9 @@
+# Test library for the syndication feed tests: a folder source (one versioned resource,
+# which is syndicated, and one unversioned one, which isn't) alongside the automatic R5
+# core loading (a version specific package)
+
+base:
+  url: https://storage.googleapis.com/tx-fhir-org
+
+sources:
+  - folder:tests/tx/fixtures/syndication-folder=synd

tests/tx/syndication-feed.test.js

@@ -0,0 +1,199 @@
+/**
+ * The syndication feed (tx/syndication/syndication-feed.js): the Atom documents built from
+ * the store - NCTS ASF content, and RFC 5005 archived-feed paging.
+ */
+
+const { XMLParser } = require('fast-xml-parser');
+const { renderAtom, renderHtml, selectPage, feedId, esc, PAGE_SIZE } = require('../../tx/syndication/syndication-feed');
+const { NCTS_SCHEME, FHIRSMITH_SCHEME } = require('../../tx/syndication/syndication-store');
+
+const BASE = 'https://tx.example.org/tx/r4';
+
+function entry(n, extra = {}) {
+  return {
+    id: `urn:uuid:00000000-0000-4000-8000-${String(n).padStart(12, '0')}`,
+    inducted: new Date(Date.parse('2026-10-23T00:00:00Z') + n * 1000).toISOString(),
+    key: `package:p${n}#1.0.0`,
+    kind: 'package',
+    category: { scheme: NCTS_SCHEME, term: 'FHIR_Package', label: 'FHIR Package' },
+    identifier: `http://example.org/p${n}`,
+    version: `http://example.org/p${n}|1.0.0`,
+    title: `Package ${n}`,
+    link: { href: `https://npm.fhir.org/web/p${n}-1.0.0.tgz`, type: 'application/gzip', length: 100 + n, sha256: 'ab'.repeat(32) },
+    ...extra
+  };
+}
+
+function fakeStore(entries) {
+  return {
+    id: '11111111-2222-4333-8444-555555555555',
+    created: '2026-10-22T00:00:00.000Z',
+    entriesFor: (v) => entries.filter(e => !e.fhirVersion || e.fhirVersion === v)
+  };
+}
+
+function parse(xml) {
+  const parser = new XMLParser({ ignoreAttributes: false, attributeNamePrefix: '@', parseTagValue: false, parseAttributeValue: false, isArray: (name) => ['entry', 'link'].includes(name) });
+  return parser.parse(xml).feed;
+}
+
+function links(feed) {
+  const map = {};
+  for (const l of feed.link) {
+    map[l['@rel']] = l['@href'];
+  }
+  return map;
+}
+
+function render(entries, page = null, extra = {}) {
+  return renderAtom({ store: fakeStore(entries), endpointPath: '/tx/r4', fhirVersion: '4.0.1', base: BASE, page, title: 'Test Server', serverVersion: '1.2.3', ...extra });
+}
+
+describe('paging', () => {
+  test('pages are full archives, then the rest in the current document', () => {
+    const all = Array.from({ length: 65 }, (_, i) => entry(i));
+    expect(PAGE_SIZE).toBe(30);
+    expect(selectPage(all, null)).toEqual({ archives: 2, entries: all.slice(60) });
+    expect(selectPage(all, 1).entries).toEqual(all.slice(0, 30));
+    expect(selectPage(all, 2).entries).toEqual(all.slice(30, 60));
+    expect(selectPage(all, 3)).toBeNull();
+    expect(selectPage(all, 0)).toBeNull();
+    expect(selectPage(all.slice(0, 30), null)).toEqual({ archives: 1, entries: [] });
+  });
+
+  test('the current document links to the newest archive', () => {
+    const feed = parse(render(Array.from({ length: 65 }, (_, i) => entry(i))));
+    expect(links(feed)).toEqual({
+      self: `${BASE}/syndication`,
+      current: `${BASE}/syndication`,
+      'prev-archive': `${BASE}/syndication/archive/2`
+    });
+    expect(feed['fh:archive']).toBeUndefined();
+    expect(feed.entry).toHaveLength(5);
+  });
+
+  test('archives are marked, and linked back to the one before', () => {
+    const all = Array.from({ length: 95 }, (_, i) => entry(i));
+    const first = parse(render(all, 1));
+    expect(first['fh:archive']).toBeDefined();
+    // no next-archive anywhere: it would make the newest archive change
+    expect(links(first)).toEqual({
+      self: `${BASE}/syndication/archive/1`,
+      current: `${BASE}/syndication`
+    });
+    expect(links(parse(render(all, 2)))['prev-archive']).toBe(`${BASE}/syndication/archive/1`);
+    const newest = links(parse(render(all, 3)));
+    expect(newest['prev-archive']).toBe(`${BASE}/syndication/archive/2`);
+    expect(newest['next-archive']).toBeUndefined();
+    expect(render(all, 4)).toBeNull();
+  });
+
+  test('an archive does not change as entries are added', () => {
+    const all = Array.from({ length: 31 }, (_, i) => entry(i));
+    const before = render(all, 1);
+    all.push(...Array.from({ length: 40 }, (_, i) => entry(100 + i)));
+    expect(render(all, 1)).toBe(before);
+  });
+});
+
+describe('the feed', () => {
+  test('carries the feed level elements the IG asks for', () => {
+    const feed = parse(render([entry(1), entry(2)]));
+    expect(feed['@xmlns']).toBe('http://www.w3.org/2005/Atom');
+    expect(feed['@xmlns:ncts']).toBe('http://ns.electronichealth.net.au/ncts/syndication/asf/extensions/1.0.0');
+    expect(feed.id).toBe(feedId('11111111-2222-4333-8444-555555555555', '/tx/r4'));
+    expect(feed.title).toBe('Test Server Syndication Feed (FHIR 4.0)');
+    expect(feed.updated).toBe(entry(2).inducted);
+    expect(feed['ncts:atomSyndicationFormatProfile']).toBe('http://ns.electronichealth.net.au/ncts/syndication/asf/profile/1.0.0');
+    expect(feed.generator['#text']).toBe('FHIRsmith');
+  });
+
+  test('the feed id is stable and differs by endpoint', () => {
+    const a = feedId('s', '/tx/r4');
+    expect(a).toMatch(/^urn:uuid:[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/);
+    expect(feedId('s', '/tx/r4')).toBe(a);
+    expect(feedId('s', '/tx/r5')).not.toBe(a);
+  });
+
+  test('an empty feed dates from when the store started', () => {
+    const feed = parse(render([]));
+    expect(feed.updated).toBe('2026-10-22T00:00:00.000Z');
+    expect(feed.entry).toBeUndefined();
+  });
+
+  test('entries are newest first, with the NCTS elements', () => {
+    const feed = parse(render([entry(1), entry(2, { summary: 'second <one>' })]));
+    expect(feed.entry.map(e => e.title)).toEqual(['Package 2', 'Package 1']);
+    const e = feed.entry[0];
+    expect(e.id).toBe(entry(2).id);
+    expect(e.updated).toBe(entry(2).inducted);
+    expect(e.published).toBe(entry(2).inducted);
+    expect(e.summary).toBe('second <one>');
+    expect(e.category['@term']).toBe('FHIR_Package');
+    expect(e.category['@scheme']).toBe(NCTS_SCHEME);
+    expect(e.link[0]['@rel']).toBe('alternate');
+    expect(e.link[0]['@href']).toBe('https://npm.fhir.org/web/p2-1.0.0.tgz');
+    expect(e.link[0]['@length']).toBe('102');
+    expect(e.link[0]['@ncts:sha256Hash']).toBe('ab'.repeat(32));
+    expect(e['ncts:contentItemIdentifier']).toBe('http://example.org/p2');
+    expect(e['ncts:contentItemVersion']).toBe('http://example.org/p2|1.0.0');
+    expect(e['ncts:fhirVersion']).toBe('4.0');
+  });
+
+  test('version specific entries only appear in their own endpoint\'s feed', () => {
+    const core4 = entry(1, { fhirVersion: '4.0', title: 'R4 core' });
+    const core5 = entry(2, { fhirVersion: '5.0', title: 'R5 core' });
+    const r4 = parse(render([core4, core5, entry(3)]));
+    expect(r4.entry.map(e => e.title)).toEqual(['Package 3', 'R4 core']);
+    const r5 = parse(render([core4, core5, entry(3)], null, { fhirVersion: '5.0', endpointPath: '/tx/r5' }));
+    expect(r5.entry.map(e => e.title)).toEqual(['Package 3', 'R5 core']);
+    expect(r5.entry.map(e => e['ncts:fhirVersion'])).toEqual(['5.0', '5.0']);
+  });
+
+  test('resource files are linked under the endpoint; binaries carry rights and no fhirVersion', () => {
+    const res = entry(1, {
+      kind: 'resource', category: { scheme: NCTS_SCHEME, term: 'FHIR_CodeSystem' },
+      link: { file: 'f.json', type: 'application/fhir+json', length: 5, sha256: 'cd'.repeat(32) }
+    });
+    const bin = entry(2, {
+      kind: 'binary', category: { scheme: FHIRSMITH_SCHEME, term: 'LOINC_DB', label: 'LOINC_DB' }, rights: 'Licensed',
+      link: { href: 'https://storage.example.org/Loinc.db', type: 'application/octet-stream', length: 5, sha256: 'ef'.repeat(32) }
+    });
+    const feed = parse(render([res, bin]));
+    expect(feed.entry[0].rights).toBe('Licensed');
+    expect(feed.entry[0]['ncts:fhirVersion']).toBeUndefined();
+    expect(feed.entry[1].link[0]['@href']).toBe(`${BASE}/syndication/files/f.json`);
+    expect(feed.entry[1]['ncts:fhirVersion']).toBe('4.0');
+  });
+});
+
+describe('escaping', () => {
+  test('characters XML does not allow are dropped, so metadata cannot break a document', () => {
+    expect(esc('a\u0000b\u0008c\u000Bd\u001Fe\uFFFEf')).toBe('abcdef');
+    expect(esc('tab\tnl\ncr\r')).toBe('tab\tnl\ncr\r');
+    expect(esc('pair \uD83D\uDE00 lone \uD83D x \uDE00')).toBe('pair \uD83D\uDE00 lone  x ');
+    expect(esc('<a href="x">&\'</a>')).toBe('&lt;a href=&quot;x&quot;&gt;&amp;&apos;&lt;/a&gt;');
+    const xml = render([entry(1, { summary: 'bad \u0001 summary', title: 'bad \u0002 title' })]);
+    const feed = parse(xml);
+    expect(feed.entry[0].summary).toBe('bad  summary');
+    expect([...xml].some(ch => ch.charCodeAt(0) < 9)).toBe(false);
+  });
+});
+
+describe('the HTML view', () => {
+  test('lists the entries and escapes them', () => {
+    const html = renderHtml({ store: fakeStore([entry(1, { title: '<script>x</script>' })]), endpointPath: '/tx/r4', fhirVersion: '4.0.1', base: BASE, page: null });
+    expect(html).toContain('&lt;script&gt;x&lt;/script&gt;');
+    expect(html).not.toContain('<script>');
+    expect(html).toContain('https://npm.fhir.org/web/p1-1.0.0.tgz');
+  });
+
+  test('links between the documents', () => {
+    const all = Array.from({ length: 65 }, (_, i) => entry(i));
+    const current = renderHtml({ store: fakeStore(all), endpointPath: '/tx/r4', fhirVersion: '4.0', base: BASE, page: null });
+    expect(current).toContain(`${BASE}/syndication/archive/2">Older`);
+    const first = renderHtml({ store: fakeStore(all), endpointPath: '/tx/r4', fhirVersion: '4.0', base: BASE, page: 1 });
+    expect(first).toContain(`${BASE}/syndication/archive/2">Newer`);
+    expect(renderHtml({ store: fakeStore(all), endpointPath: '/tx/r4', fhirVersion: '4.0', base: BASE, page: 9 })).toBeNull();
+  });
+});