diff --git a/CHANGELOG.md b/CHANGELOG.md index 78d7efe3..4afc9cdd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,41 @@ All notable changes to Health Intersections FHIRsmith will be documented in this The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.18.2] - 2026-10-dd + +### Security + +- + +### Added + +- SNOMED CT ECL: member filters (`^ refset {{ M ... }}`) - `effectiveTime` (with a time value or set), `active`, `moduleId` (any sub expression) and the reference set's own fields (`targetComponentId = << X`, `mapTarget = "..."`, `mapGroup >= #2`, ...), comma-separated and repeatable. A member filter anywhere other than directly after a memberOf constraint is a syntax error (`INVALID_ECL`). +- SNOMED CT ECL: concept filters (`{{ C ... }}`) - `active`, `definitionStatus` (primitive/defined), `definitionStatusId`, `moduleId` (a concept, a bracketed concept list or any sub expression) and `effectiveTime` +- SNOMED CT ECL: `AND` and `OR` between refinements - between attributes (`: A = x AND B = y`, `: A = x OR B = y`, with brackets for nesting), within an attribute group (`{ A = x OR B = y }`) and between groups (`{ ... } OR { ... }`). Mixing AND and OR without brackets is rejected, as for compound expressions. Previously only `,` (conjunction) was accepted +- SNOMED CT ECL: description filters (`{{ D ... }}`) - `term` (match and `wild:` search terms, single or a set), `language`, `typeId` and `type` (syn/fsn/def), `dialectId` and `dialect` (en-us, en-gb and the other aliases in the ECL specification) with acceptability sets, `moduleId`, `effectiveTime`, `active` and `id`. The filters in one block apply to the same description; only active descriptions are considered unless `active` is given +- SNOMED CT ECL: history supplements (`{{ + HISTORY }}`, `HISTORY-MIN` / `-MOD` / `-MAX`, `HISTORY (*)`, `HISTORY (refset expression)`) - the inactive concepts that the selected historical association reference sets associate with the result are added to it +- SNOMED CT ECL: concrete values in refinements (`= #500`, `>= #1.5`, `!= #-1`, `= "text"`, `= true`). The importer now reads `sct2_RelationshipConcreteValues_*` into a new cache section (cache version 19, after 18's alternate identifiers); editions imported before this report a concrete-value refinement as unsupported. The test edition cache is `sct_test_20250909e.cache` +- SNOMED CT ECL: `!=` in refinements (`: A != X` - at least one A whose value is not in X; `[0..0] A != X` - every A is in X), reversed attributes (`: R A = X` - the concepts that are the value of A on a concept in X), and `,` as a conjunction between expressions as well as between refinements +- SNOMED CT ECL: comments (`/* ... */`), top and bottom of set (`!!> (...)`, `!!< (...)`), and attribute names that are expressions rather than a single concept (`: << 762705008 = ...`, `: (A OR B) = ...`, `: ^ refset = ...`, `: * = ...`, and the same after `.` in dotted expressions) +- SNOMED CT ECL: reference set field selection on member-of (`^ [targetComponentId] refset`) is parsed and reported as `UNSUPPORTED_ECL` (the selected fields, and many referenced components, are descriptions) rather than as a syntax error +- SNOMED CT ECL: alternate identifiers (`LOINC#14394-1`, `"LOINC#14394-1"`): the scheme alias names an identifier scheme (LOINC = 30051010000102 |LOINC code ID|, as imported from the LOINC Extension) and the code resolves to the concept(s) carrying it, which then take hierarchy operators like any set. An unknown scheme or a code nobody carries is `INVALID_ECL`, as an unknown concept is + +### Changed + +- Move the hard-coded link about adding to the terminology server ecosystem to the registry control file + +### Fixed + +- SNOMED CT ECL: a hierarchy operator on a bracketed expression was silently dropped - `< (A OR B)` returned A and B themselves rather than their descendants. It now applies to every member of the set, as do operators on `*` and on `^ refset`, which were previously rejected as unsupported +- SNOMED CT ECL: `^` on a reference set whose members are descriptions (a language reference set) returned an empty expansion; it is now reported as `UNSUPPORTED_ECL` when the reference set is named, and skipped when it merely falls inside a computed operand such as `^ *` +- SNOMED CT ECL: an attribute name with no comparison (`: 363698007` with nothing after it) was accepted by the parser and failed later as unsupported; it is a syntax error +- SNOMED CT ECL: an evaluation error is now `UNSUPPORTED_ECL` unless it is about a concept that does not exist (`INVALID_ECL`); it was classified by matching the error text, which filed `= true` as a syntax error. A concept id that is not 6-18 digits is rejected by the parser + +### Tx Conformance Statement + +(paste) + + ## [0.18.1] - 2026-10-25 @@ -79,12 +114,6 @@ Note: We are heading towards a 1.0.0 release. - `$lookup`: a warning when the `displayLanguage` asked for has no display for the code, and the display is in the default language (`LOOKUP_DISPLAY_LANGUAGE_FALLBACK`) - ICD-11: the `classificationCode` extension (provisional url `http://id.who.int/icd/schema/extension/classificationCode`) on a Coding carries the short code for an entity id, saving a lookup. When present it is checked in `$validate-code` (CodeSystem and ValueSet): a wrong value, or the extension on a code with no short code, on a short code, or on a system other than MMS, is an error - ICD-11 `$lookup`: postcoordination is reported as one `postcoordination` property per axis (subproperties `axis` and `value` Codings, a `/` member's value with its own nested postcoordination), replacing the WHO `postcoordinationScale` shape. With the new parameter `post-coordination-metadata=true` every declared axis is listed with `required`, `allowMultipleValues` and its value set; otherwise only the valued axes. `source` (the Foundation entity) is a Coding, and asking for `source.details` adds what a `$lookup` on it would return, designations included. A single-valued axis with several values gets a message in `$validate-code` -- SNOMED CT ECL: member filters (`^ refset {{ M ... }}`) - `effectiveTime` (with a time value or set), `active`, `moduleId` (any sub expression) and the reference set's own fields (`targetComponentId = << X`, `mapTarget = "..."`, `mapGroup >= #2`, ...), comma-separated and repeatable. A member filter anywhere other than directly after a memberOf constraint is a syntax error (`INVALID_ECL`). Description filters (`{{ D }}`), concept filters (`{{ C }}`) and history supplements (`{{ + }}`) are now parsed and rejected as `UNSUPPORTED_ECL` rather than failing as a lexer error -- SNOMED CT ECL: concept filters (`{{ C ... }}`) - `active`, `definitionStatus` (primitive/defined), `definitionStatusId`, `moduleId` (a concept, a bracketed concept list or any sub expression) and `effectiveTime` -- SNOMED CT ECL: `AND` and `OR` between refinements - between attributes (`: A = x AND B = y`, `: A = x OR B = y`, with brackets for nesting), within an attribute group (`{ A = x OR B = y }`) and between groups (`{ ... } OR { ... }`). Mixing AND and OR without brackets is rejected, as for compound expressions. Previously only `,` (conjunction) was accepted -- SNOMED CT ECL: description filters (`{{ D ... }}`) - `term` (match and `wild:` search terms, single or a set), `language`, `typeId` and `type` (syn/fsn/def), `dialectId` and `dialect` (en-us, en-gb and the other aliases in the ECL specification) with acceptability sets, `moduleId`, `effectiveTime`, `active` and `id`. The filters in one block apply to the same description; only active descriptions are considered unless `active` is given -- SNOMED CT ECL: comments (`/* ... */`), top and bottom of set (`!!> (...)`, `!!< (...)`), and attribute names that are expressions rather than a single concept (`: << 762705008 = ...`, `: (A OR B) = ...`, `: ^ refset = ...`, `: * = ...`, and the same after `.` in dotted expressions) -- SNOMED CT ECL: concrete values in refinements (`= #5`, `!= "x"`, `= match:"x"`, `= true`, ...) and alternate identifiers (`"ICD10#A01.1"`, `SCTID#123`) and reference set field selection on member-of (`^ [targetComponentId] refset`) are now parsed, so they are reported as `UNSUPPORTED_ECL` with a message saying what is missing (concrete value relationships / alternate identifier reference sets are not loaded) rather than as a syntax error - SNOMED CT: `tx/sct/documentation.md` - what the server supports for SNOMED CT: editions, ECL, expressions and the MRCM, implicit concept maps and translation - `utilities/make-sct-test-extension.js`: builds the extension concepts (module TEST1000003) in the SNOMED CT test edition used by the tx tests - one of each shape (fully defined ungrouped, fully defined role-grouped, equivalent to an International concept, primitive), plus inactive ones with and without historical associations - README: each module's status - in production; in production but still under development; under development / prototyping; or on the back-burner @@ -106,9 +135,6 @@ Note: We are heading towards a 1.0.0 release. - ValueSet `$validate-code`: an include with more than one filter only checked the first. The other filters' checks return a message when the code fails them, and a message was taken as a pass - LOINC: the `exists` filter ignored its value. On a relationship property (`METHOD_TYP`, `COMPONENT`, `document-kind`, ...) the value was looked up as a code or display, so `exists true` and `exists false` both returned every code with the relationship (because a code displayed as "True" and one as "False" exist), and on a property (`CLASSTYPE`, `ORDER_OBS`, ...) `exists false` meant `exists true`. `true` now returns the codes that have the property, `false` every other code in the code system - Parts, AnswerLists and Answers included, so add `CLASSTYPE exists true` to restrict to LOINC terms (e.g. LOINC terms with no method: `METHOD_TYP exists false` + `CLASSTYPE exists true`). Any other value is not supported - Packages: a package download or page request was counted twice in the statistics (once by the validation step, once by the handler), and tarball downloads were not told from requests for a version's JSON document; downloads are now counted as `download`, version documents as `version`, each once -- SNOMED CT ECL: a hierarchy operator on a bracketed expression was silently dropped - `< (A OR B)` returned A and B themselves rather than their descendants. It now applies to every member of the set, as do operators on `*` and on `^ refset`, which were previously rejected as unsupported -- SNOMED CT ECL: `^` on a reference set whose members are descriptions (a language reference set) returned an empty expansion; it is now reported as `UNSUPPORTED_ECL` -- SNOMED CT ECL: an evaluation error is now `UNSUPPORTED_ECL` unless it is about a concept that does not exist (`INVALID_ECL`); it was classified by matching the error text, which filed `= true` as a syntax error. A concept id that is not 6-18 digits is rejected by the parser - SNOMED CT expressions: the MRCM content type taken as "postcoordinated" was 723594008, which is *all precoordinated content* (postcoordinated is 723595009). So the body-structure part-of attributes (|Regional part of| etc., precoordinated only) were allowed in expressions, |Has ingredient| (postcoordinated only) was not checked, and for the attributes with a precoordinated-specific range that range was used in place of the general one - SHL: every access to a live link (`/shl/access/{uuid}`) hung without a response - the access-logging callback was a plain `function`, so `this.db` was undefined in it and the request was never answered (since 0.3.0). A `GET` on a link is now the U flag (direct file) access the SMART Health Links spec defines: a link with one file returns it as a JWE (`application/jose`), and `recipient` is read from the query and logged (it was always `anonymous`); a link with more than one file, or a VHL, still gets the manifest / VHL bundle. The manifest now has `status` (`finalized`), file `location`s are always https (they followed `req.secure`, so http behind a proxy without trust proxy), and `embedded` is the JWE itself (it was the base64 of it, as stored) diff --git a/registry/api.js b/registry/api.js index 4ba83fa7..ccc87973 100644 --- a/registry/api.js +++ b/registry/api.js @@ -774,7 +774,7 @@ class RegistryAPI { } html += '\n'; - html += `${row.security}/td>\n`; + html += `${escape(row.security)}\n`; html += '\n'; } diff --git a/registry/crawler.js b/registry/crawler.js index e130bf85..9f480fda 100644 --- a/registry/crawler.js +++ b/registry/crawler.js @@ -160,6 +160,8 @@ class RegistryCrawler { } newData.doco = masterJson.documentation || ''; + newData.registration = masterJson.registration || ''; + newData.ecosystemDoco = masterJson['ecosystem-documentation'] || ''; // Process each registry const registries = masterJson.registries || []; diff --git a/registry/documentation/internals.md b/registry/documentation/internals.md index c3676ee2..b6ee64a7 100644 --- a/registry/documentation/internals.md +++ b/registry/documentation/internals.md @@ -28,7 +28,10 @@ How the module works, for anyone changing it. The user-facing behaviour is in `RegistryCrawler.crawl()` runs every `crawlInterval` minutes (first 5 seconds after start), and never concurrently with itself. A scan builds a fresh `ServerRegistries` from scratch: -1. fetch the master registration file (`formatVersion` must be `"1"`); +1. fetch the master registration file (`formatVersion` must be `"1"`); its optional + `registration` and `ecosystem-documentation` URLs are the "Register your own server" and + "Documentation" links on the home page (each is omitted if the file doesn't give it), since + they depend on the ecosystem; 2. for each registry it lists, fetch the registry file and build a `ServerRegistry`; 3. for each server, a `ServerInformation` with its claims (`authoritative`, `authoritative-valuesets`, `languages`, `usage`, `exclusions`), and for each of its diff --git a/registry/documentation/readme.md b/registry/documentation/readme.md index 367ae838..23aa6fef 100644 --- a/registry/documentation/readme.md +++ b/registry/documentation/readme.md @@ -27,8 +27,9 @@ it, [api.md](api.md) says so. **The registration files.** The ecosystem is declared by a *master registration file* that lists *registries*, each of which lists *servers*, each of which lists its endpoints (one per -FHIR version) and its claims. The format is documented in the ig-registry repository's -[tx-registry-doco.md](https://github.com/FHIR/ig-registry/blob/master/tx-registry-doco.md). +FHIR version) and its claims. The format is documented on the IG's ecosystem page; HL7's own +ecosystem (its locations, and how to register a server in it) is documented in the ig-registry +repository's [tx-registry-doco.md](https://github.com/FHIR/ig-registry/blob/master/tx-registry-doco.md). HL7's master file is https://fhir.github.io/ig-registry/tx-servers.json, and its registries are the per-affiliate `*-tx-servers.json` files in the same repository (HL7 International, HL7 Australia, HL7 Europe, and so on). A server's registration carries: diff --git a/registry/model.js b/registry/model.js index ed6cdcf7..f83232ef 100644 --- a/registry/model.js +++ b/registry/model.js @@ -362,6 +362,8 @@ class ServerRegistries { constructor() { this.address = ''; this.doco = ''; + this.registration = ''; // URL: how to register your own server in this ecosystem + this.ecosystemDoco = ''; // URL: documentation for this ecosystem this.lastRun = null; // Date object this.outcome = ''; this.registries = []; // Array of ServerRegistry @@ -386,6 +388,8 @@ class ServerRegistries { this.lastRun = source.lastRun; this.outcome = source.outcome; this.doco = source.doco; + this.registration = source.registration; + this.ecosystemDoco = source.ecosystemDoco; source.registries.forEach(sourceRegistry => { const existing = this.getRegistry(sourceRegistry.code); @@ -402,6 +406,8 @@ class ServerRegistries { version: '1', address: this.address, doco: this.doco, + registration: this.registration, + 'ecosystem-documentation': this.ecosystemDoco, 'last-run': this.lastRun?.toISOString() || '', outcome: this.outcome, registries: this.registries.map(r => r.toJSON()) @@ -416,6 +422,8 @@ class ServerRegistries { const instance = new ServerRegistries(); instance.address = json.address || ''; instance.doco = json.doco || ''; + instance.registration = json.registration || ''; + instance.ecosystemDoco = json['ecosystem-documentation'] || ''; instance.lastRun = json['last-run'] ? new Date(json['last-run']) : null; instance.outcome = json.outcome || ''; instance.registries = (json.registries || []).map(r => ServerRegistry.fromJSON(r)); diff --git a/registry/registry.js b/registry/registry.js index c1e847fa..932bd829 100644 --- a/registry/registry.js +++ b/registry/registry.js @@ -807,8 +807,15 @@ class RegistryModule { const lastRunDate = new Date(stats.lastRun); lastUpdatedText = `Last Updated: ${lastRunDate.toLocaleString()}`; } - html += `

${lastUpdatedText}. Register your own server`+ - ` - see Documentation

`; + // what goes here depends on the ecosystem, so it comes from the master registration file + html += `

${lastUpdatedText}.`; + if (data.registration) { + html += ` Register your own server`; + } + if (data.ecosystemDoco) { + html += `${data.registration ? ' -' : ''} see Documentation`; + } + html += '

'; html += ''; html += ''; diff --git a/tests/cs/cs-snomed-ecl.test.js b/tests/cs/cs-snomed-ecl.test.js index 0d16a7b5..1861b113 100644 --- a/tests/cs/cs-snomed-ecl.test.js +++ b/tests/cs/cs-snomed-ecl.test.js @@ -91,13 +91,9 @@ describe('ECL Validator Test Suite', () => { expect(e.msgId).toBe('INVALID_ECL'); }); - test.each([ - ['< 64572001 {{ + HISTORY-MIN }}', 'history supplements'], - ])('%s is reported as unsupported', (ecl, what) => { - const e = errorFor(ecl); - expect(e).not.toBeNull(); - expect(e.msgId).toBe('UNSUPPORTED_ECL'); - expect(e.message).toContain(what); + test('a malformed history supplement is a syntax error', () => { + expect(errorFor('< 64572001 {{ + FOO }}').msgId).toBe('INVALID_ECL'); + expect(errorFor('< 64572001 {{ + HISTORY-MIN (*) }}').msgId).toBe('INVALID_ECL'); }); }); @@ -202,11 +198,11 @@ describe('ECL Validator Test Suite', () => { '("ICD10#A01.1")', '< SCTID#64572001', '< ICD10#A01.1 |Typhoid|', - ])('alternate identifier %s parses and is reported as unsupported', (ecl) => { + ])('alternate identifier %s with an unknown scheme is invalid', (ecl) => { const e = errorFor(ecl); expect(e).not.toBeNull(); - expect(e.msgId).toBe('UNSUPPORTED_ECL'); - expect(e.message).toContain('alternate identifiers'); + expect(e.msgId).toBe('INVALID_ECL'); + expect(e.message).toContain('scheme'); }); test.each([ @@ -341,6 +337,125 @@ describe('ECL Validator Test Suite', () => { }); }); + describe('ECL != and reversed (R) attributes', () => { + const codes = (ecl) => (snomedServices.filterECL(ecl, true, opContext).descendants || []) + .map(i => snomedServices.concepts.getConceptId(i).toString()).sort(); + const count = (ecl) => codes(ecl).length; + + test('!= matches concepts with an attribute value outside the set', () => { + // the medicinal products: precise active ingredient is halibut liver oil or barium sulfate + expect(codes('< 763158003 : 762949000 != 126081001')).toEqual(['1204474000']); + expect(codes('* : 762949000 != 126081001')).toEqual(['1204474000']); + // products with no precise active ingredient do not match + expect(count('< 763158003 : 127489000 != 126081001')).toBe(0); + // [0..0] != : every value is in the set (or there is no attribute) + expect(count('< 763158003 : [0..0] 762949000 != < 105590001')).toBe(count('< 763158003')); + expect(codes('< 763158003 : [0..0] 762949000 != 396014007')).not.toContain('329238006'); + expect(codes('< 763158003 : [0..0] 762949000 != 396014007')).toContain('1204474000'); + }); + + test('R yields the values of an attribute on the given concepts', () => { + expect(codes('< 105590001 : R 127489000 = 776168003')).toEqual(['126081001']); + expect(codes('< 105590001 : R * = 329238006')).toEqual(['126081001']); + expect(codes('< 105590001 : R 762949000 = *')).toEqual(['126081001', '396014007']); + expect(codes('< 105590001 : R 762949000 = < 763158003')).toEqual(['126081001', '396014007']); + expect(count('< 105590001 : [0..*] R 127489000 = 776168003')).toBe(count('< 105590001')); + expect(count('<< 105590001 : R 363698007 = *')).toBe(0); + }); + + test('a comma is a conjunction at the expression level', () => { + expect(codes('138875005 , 138875005')).toEqual(['138875005']); + expect(count('138875005 , 10200004')).toBe(0); + expect(codes('< 64572001 , < 64572001 , 64572001')).toEqual([]); + expect(() => snomedServices.filterECL('< 64572001 , < 10200004 OR < 39607008', true, opContext)).toThrow('cannot be mixed'); + }); + + test('an attribute without a comparison is a syntax error', () => { + for (const ecl of ['< 64572001 : 363698007', '(< 64572001 : 363698007 = 10200004 OR 404684003)']) { + let e = null; + try { snomedServices.filterECL(ecl, true, opContext); } catch (x) { e = x; } + expect(e).not.toBeNull(); + expect(e.msgId).toBe('INVALID_ECL'); + expect(e.message).toContain('comparison operator'); + } + }); + }); + + describe('ECL history supplements and concrete values (test edition cache)', () => { + // These need the full test edition (association reference sets with targets, + // and a cache carrying the concrete value relationships - version 19), so they + // run against the server's test cache when it is present and skip otherwise. + const cachePath = join(__dirname, '../../data/terminology-cache/sct_test_20250909e.cache'); + let svc = null; + beforeAll(async () => { + if (!require('fs').existsSync(cachePath)) return; + const factory = new SnomedServicesFactory(opContext.i18n, cachePath); + await factory.load(); + svc = factory.snomedServices; + }); + const codes = (ecl) => (svc.filterECL(ecl, true, opContext).descendants || []) + .map(i => svc.concepts.getConceptId(i).toString()).sort(); + const have = () => svc !== null && svc.concreteValues !== null; + + test('history supplements add the inactive concepts associated with the result', () => { + if (!have()) return; + expect(codes('<< 74400008 {{ + HISTORY }}')).toEqual(['307530000', '74400008']); + expect(codes('<< 74400008 {{ + HISTORY-MAX }}')).toEqual(['307530000', '74400008']); + expect(codes('<< 74400008 {{ + HISTORY-MOD }}')).toEqual(['307530000', '74400008']); + expect(codes('<< 74400008 {{ + HISTORY-MIN }}')).toEqual(['74400008']); // no SAME AS refset in the edition + expect(codes('<< 4896000 {{ + HISTORY (900000000000526001) }}')).toEqual(codes('<< 4896000').concat(['1000381000003107']).sort()); + expect(codes('<< 4896000 {{ + HISTORY (*) }}')).toEqual(codes('<< 4896000 {{ + HISTORY }}')); + expect(codes('(74400008 OR 4896000) {{ + HISTORY }}')).toEqual(['1000381000003107', '307530000', '4896000', '74400008']); + expect(codes('< 64572001 {{ + HISTORY }}').length).toBe(codes('< 64572001').length + 2); + }); + + test('concrete values: numeric comparisons', () => { + if (!have()) return; + expect(codes('< 763158003 : 1142139005 = #1')).toEqual(['1204474000', '329238006', '776168003', '779430005']); + expect(codes('< 763158003 : 1142139005 = #1.0')).toEqual(codes('< 763158003 : 1142139005 = #1')); + expect(codes('< 763158003 : 1142135004 = #4000')).toEqual(['329238006']); + expect(codes('< 763158003 : 1142139005 != #1')).toEqual([]); + expect(codes('< 763158003 : 1142138002 < #1000')).toEqual(['1204474000']); + expect(codes('< 763158003 : 1142135004 <= #4000')).toEqual(['329238006']); + expect(codes('< 763158003 : 1142135004 > #1000')).toEqual(['329238006']); + expect(codes('< 763158003 : 1142138002 >= #600')).toEqual(['1204474000']); + expect(codes('< 763158003 : 1142139005 = #-1')).toEqual([]); + expect(codes('< 763158003 : [0..0] 1142139005 = #1')).toEqual(['406459008', '769034009']); + expect(codes('< 763158003 : { 1142135004 = #4000 }')).toEqual(['329238006']); + expect(codes('< 763158003 : * = #4000')).toEqual(['329238006']); + expect(codes('< 763158003 : 1142139005 = #1 AND 1142135004 = #4000')).toEqual(['329238006']); + }); + + test('concrete values: a value of another kind never matches', () => { + if (!have()) return; + expect(codes('< 763158003 : 1142139005 = "x"')).toEqual([]); + expect(codes('< 763158003 : 1142139005 != "x"')).toEqual([]); + expect(codes('< 763158003 : 1142139005 = true')).toEqual([]); + }); + + test('alternate identifiers resolve through the identifier scheme alias', () => { + if (!have()) return; + expect(codes('LOINC#14394-1')).toEqual(['63241010000109']); + expect(codes('"LOINC#14394-1"')).toEqual(['63241010000109']); + expect(codes('loinc#14394-1 |Urea nitrogen|')).toEqual(['63241010000109']); + expect(codes('>> LOINC#14394-1')).toEqual(codes('>> 63241010000109')); + expect(codes('(LOINC#14394-1 OR LOINC#2067-7)')).toEqual(['57371010000105', '63241010000109']); + let e = null; + try { svc.filterECL('LOINC#99999-9', true, opContext); } catch (x) { e = x; } + expect(e.msgId).toBe('INVALID_ECL'); + expect(e.message).toContain('No concept has the alternate identifier'); + }); + + test('an edition without concrete values reports them as unsupported', () => { + // the committed jest cache predates the concrete value section + let e = null; + try { snomedServices.filterECL('< 763158003 : 1142139005 = #1', true, opContext); } catch (x) { e = x; } + expect(e).not.toBeNull(); + expect(e.msgId).toBe('UNSUPPORTED_ECL'); + expect(e.message).toContain('concrete value'); + }); + }); + describe('ECL refinement conjunction and disjunction', () => { const count = (ecl) => (snomedServices.filterECL(ecl, true, opContext).descendants || []).length; diff --git a/tests/registry/registry-home-links.test.js b/tests/registry/registry-home-links.test.js new file mode 100644 index 00000000..9eb04c42 --- /dev/null +++ b/tests/registry/registry-home-links.test.js @@ -0,0 +1,47 @@ +// The "Register your own server" and "Documentation" links on the registry home page +// depend on the ecosystem, so they come from the master registration file + +const { ServerRegistries } = require('../../registry/model'); +const RegistryModule = require('../../registry/registry'); + +function homePage(master) { + const data = new ServerRegistries(); + data.registration = master.registration || ''; + data.ecosystemDoco = master['ecosystem-documentation'] || ''; + const module = new RegistryModule({ countRequest() {} }); + module.api = { getStatistics: () => ({}), getData: () => data }; + module.crawler = { getData: () => data }; + module.releases = { latest: () => null, source: 'test', describe: () => ({}) }; + return module.buildHtmlContent(); +} + +describe('registry home page: ecosystem links', () => { + test('both links come from the master file', async () => { + const html = await homePage({ + registration: 'https://example.org/register.html', + 'ecosystem-documentation': 'https://example.org/ecosystem.html' + }); + expect(html).toContain('Register your own server - see Documentation'); + expect(html).not.toContain('github.com/FHIR/ig-registry'); + expect(html).not.toContain('fhir-tx-ecosystem-ig'); + }); + + test('a link the master file does not give is left out', async () => { + let html = await homePage({ 'ecosystem-documentation': 'https://example.org/ecosystem.html' }); + expect(html).not.toContain('Register your own server'); + expect(html).toContain('see Documentation'); + + html = await homePage({ registration: 'https://example.org/register.html' }); + expect(html).toContain('Register your own server

'); + expect(html).not.toContain('Documentation'); + }); + + test('the links survive a save and load', () => { + const data = new ServerRegistries(); + data.registration = 'https://example.org/register.html'; + data.ecosystemDoco = 'https://example.org/ecosystem.html'; + const copy = ServerRegistries.fromJSON(JSON.parse(JSON.stringify(data.toJSON()))); + expect(copy.registration).toBe(data.registration); + expect(copy.ecosystemDoco).toBe(data.ecosystemDoco); + }); +}); diff --git a/tests/registry/registry-hostile-content.test.js b/tests/registry/registry-hostile-content.test.js index fecc8863..d94278d2 100644 --- a/tests/registry/registry-hostile-content.test.js +++ b/tests/registry/registry-hostile-content.test.js @@ -43,7 +43,7 @@ describe('registry: hostile registry and server content', () => { const u = new URL(req.url, 'http://x'); let body; if (u.pathname === '/master.json') { - body = { formatVersion: '1', documentation: BAD, registries: [{ code: 'evil', name: 'Evil' + BAD, url: `${base}/registry.json`, authority: BAD }] }; + body = { formatVersion: '1', documentation: BAD, registration: EVIL_URL, 'ecosystem-documentation': 'https://evil.org/' + EVIL_ATTR, registries: [{ code: 'evil', name: 'Evil' + BAD, url: `${base}/registry.json`, authority: BAD }] }; } else if (u.pathname === '/registry.json') { body = { formatVersion: '1', diff --git a/tests/registry/test-data.json b/tests/registry/test-data.json index 4cb83863..60a4eb83 100644 --- a/tests/registry/test-data.json +++ b/tests/registry/test-data.json @@ -1,7 +1,9 @@ { "version": "1", "address": "https://fhir.github.io/ig-registry/tx-servers.json", - "doco": "See https://github.com/FHIR/ig-registry/blob/master/tx-registry-doco.md", + "doco": "See http://hl7.org/fhir/uv/tx-ecosystem/ecosystem.html", + "registration": "https://github.com/FHIR/ig-registry/blob/master/tx-registry-doco.md", + "ecosystem-documentation": "http://hl7.org/fhir/uv/tx-ecosystem/ecosystem.html", "last-run": "2025-08-08T22:50:55.096Z", "outcome": "Processed OK - 12.7 MB", "registries": [ diff --git a/tx/cs/cs-snomed.js b/tx/cs/cs-snomed.js index 167eac95..c1bddbf2 100644 --- a/tx/cs/cs-snomed.js +++ b/tx/cs/cs-snomed.js @@ -37,7 +37,7 @@ const SNOMED_FILTER_ANALYSIS = Symbol('snomedFilterAnalysis'); * returned as. An identifier in an unknown scheme is kept but not reported. */ const IDENTIFIER_SCHEMES = { - '30051010000102': { system: 'http://loinc.org', property: 'loinc' } // |LOINC code identifier| (LOINC Extension) + '30051010000102': { system: 'http://loinc.org', property: 'loinc', alias: 'LOINC' } // |LOINC code identifier| (LOINC Extension) }; /** The $expand / ECL-free filter on alternate identifiers: mapped: exists true|false */ @@ -206,6 +206,39 @@ class SnomedServices { this._mrcmRangeCache = new Map(); this.#loadIdentifiers(sharedData.identifiers || []); + this.#loadConcreteValues(sharedData.concreteValues); + } + + /** + * Index the concrete value relationships (cache version 19: rows of + * [sourceId, typeId, group, value], value as written in RF2 - "#500", "#5.5", + * a quoted string, or true/false) by source concept index, for the ECL + * evaluator. `concreteValues` stays null when the cache predates them, which + * is how a concrete-value refinement knows to report itself as unsupported + * rather than answering with nothing. + */ + #loadConcreteValues(rows) { + this.concreteValues = null; + if (!rows) return; + this.concreteValues = new Map(); + for (const [sourceId, typeId, group, raw] of rows) { + const src = this.concepts.findConcept(sourceId); + const type = this.concepts.findConcept(typeId); + if (!src.found || !type.found) continue; + let value; + if (/^#-?\d+(\.\d+)?$/.test(raw)) { + value = { kind: 'number', value: Number(raw.substring(1)), decimal: raw.includes('.') }; + } else if (/^".*"$/.test(raw)) { + value = { kind: 'string', value: raw.substring(1, raw.length - 1) }; + } else if (/^(true|false)$/i.test(raw)) { + value = { kind: 'boolean', value: raw.toLowerCase() === 'true' }; + } else { + value = { kind: 'string', value: raw }; + } + const list = this.concreteValues.get(src.index) || []; + list.push({ relType: type.index, group: group || 0, concrete: value }); + this.concreteValues.set(src.index, list); + } } /** @@ -912,14 +945,16 @@ class SnomedServices { * hierarchy operators on *, ^ and bracketed sets (applied per member) * AND / OR / MINUS compound expressions * !!> / !!< (top / bottom of set) - * refinements (= against an expression, attribute AND/OR/",", groups, - * attribute and group cardinality, attribute names that are expressions), + * refinements (= and != against an expression, reversed (R) attributes, + * attribute AND/OR/",", groups, attribute and group cardinality, + * attribute names that are expressions), * dotted expressions, {{ C ... }} concept filters, {{ D ... }} description - * filters, comments + * filters, {{ + HISTORY... }} history supplements, concrete (numeric / + * string / boolean) values in refinements (cache version 19+), comments + * + * alternate identifiers (LOINC#code, from the identifier scheme aliases) * - * Everything else (reverse attributes, != and concrete (numeric/string/ - * boolean) values in refinements, alternate identifiers, reference set field - * selection on ^, {{ + ... }} history supplements) throws an informative + * Everything else (reference set field selection on ^) throws an informative * error which is reported as UNSUPPORTED_ECL. */ @@ -1068,9 +1103,6 @@ class SnomedServices { // ({{ M ... }}) are handled in _evalMemberOf; concept filters ({{ C ... }}) // and description filters ({{ D ... }}) are applied to the result below. if (node.filters && node.filters.length > 0) { - if (node.filters.some(fc => fc.kind === '+')) { - throw new Error('ECL history supplements ({{ + ... }}) are not supported'); - } const inner = { ...node, filters: undefined }; let ctx = this._evalSubExpression(inner, opContext); const conceptFilters = node.filters.filter(fc => fc.kind === 'C'); @@ -1080,12 +1112,26 @@ class SnomedServices { for (const fc of node.filters.filter(fc => fc.kind === 'D')) { ctx = this._applyDescriptionFilter(ctx, fc.filters, opContext); } + for (const fc of node.filters.filter(fc => fc.kind === '+')) { + ctx = this._applyHistorySupplement(ctx, fc, opContext); + } return ctx; } - // Alternate identifiers (ECL 2.1 "scheme#code") + // Alternate identifiers (ECL 2.1 "scheme#code"): the concept(s) that carry + // the code in that identifier scheme, then the operator applies to them as + // to any set. The scheme is an alias from the ECL specification (LOINC) or + // the identifier scheme concept's own id. if (focus.type === ECLNodeType.ALT_IDENTIFIER) { - throw new Error(`ECL alternate identifiers (${focus.scheme}#${focus.code}) are not supported - no alternate identifier reference sets are loaded`); + const set = this._eclAltIdentifier(focus); + if (!operator) { + const result = new SnomedFilterContext(); + result.descendants = set; + return result; + } + const base = new SnomedFilterContext(); + base.descendants = set; + return this._eclApplyOperatorToSet(base, operator, opContext); } // Top / bottom of set: the members of the focus set with no ancestor @@ -1160,6 +1206,32 @@ class SnomedServices { return result; }; + /** + * Resolve an ECL alternate identifier (scheme#code) to concept indexes. The + * scheme is a known alias (IDENTIFIER_SCHEMES[].alias, case-insensitive) or + * an identifier scheme concept id. An unknown scheme, or a code nobody has, + * is an invalid expression - the same standing as an unknown concept. + */ + _eclAltIdentifier = function (node) { + let schemeId = null; + if (/^\d{6,18}$/.test(node.scheme)) { + schemeId = node.scheme; + } else { + for (const [id, def] of Object.entries(IDENTIFIER_SCHEMES)) { + if (def.alias && def.alias.toLowerCase() === node.scheme.toLowerCase()) schemeId = id; + } + } + const invalid = (msg) => { const e = new Error(msg); e.eclInvalid = true; return e; }; + if (!schemeId) { + throw invalid(`The alternate identifier scheme "${node.scheme}" is not known`); + } + const concepts = this.getConceptsForIdentifier(schemeId, node.code); + if (concepts.length === 0) { + throw invalid(`No concept has the alternate identifier ${node.scheme}#${node.code}`); + } + return concepts.map(id => this.concepts.findConcept(id)).filter(r => r.found).map(r => r.index); + }; + /** * !!> / !!< : keep the concepts in `set` that have no ancestor (top) or no * descendant (bottom) that is also in `set`. Walks the stated parents of each @@ -1317,6 +1389,9 @@ class SnomedServices { } const refsetConcepts = this._eclResolveSet(this._evalECLNode(memberOfNode.refSet, opContext), opContext); const memberFilters = this._eclPrepareMemberFilters(memberOfNode.memberFilters, opContext); + const rs = memberOfNode.refSet; + const explicit = rs.type === ECLNodeType.CONCEPT_REFERENCE || + (rs.type === ECLNodeType.SUB_EXPRESSION_CONSTRAINT && !rs.operator && rs.focus.type === ECLNodeType.CONCEPT_REFERENCE); const members = new Set(); for (const refsetIdx of refsetConcepts) { @@ -1334,9 +1409,12 @@ class SnomedServices { } const memberList = this.refSetMembers.getMembers(membersRef) || []; // A reference set whose members are descriptions (a language reference - // set, say) has no concept members to return. Say so rather than - // answering with an empty set, which would read as "no members". - if (memberList.length > 0 && !memberList.some(m => m.kind === 0)) { + // set, say) has no concept members to return. When it was named + // explicitly, say so rather than answering with an empty set, which would + // read as "no members". When it merely falls inside a computed operand + // (^ *, ^ (< 900000000000455006)) it contributes nothing and the rest of + // the operand still counts. + if (explicit && memberList.length > 0 && !memberList.some(m => m.kind === 0)) { throw new Error(`The members of reference set ${this.concepts.getConceptId(refsetIdx)} are descriptions, not concepts, so ^ cannot be evaluated for it`); } for (const m of memberList) { @@ -1356,6 +1434,82 @@ class SnomedServices { return result; }; + /** + * Apply a history supplement ({{ + HISTORY... }}): add to the result the + * inactive concepts that the selected historical association reference sets + * associate with a concept in the result (the association rows whose + * targetComponentId is in the result). + * + * Profiles (ECL 2.x): MIN = SAME AS; MOD = SAME AS, WAS A, REPLACED BY, + * PARTIALLY EQUIVALENT TO; MAX (and a bare HISTORY) = every reference set + * under 900000000000522004 |Historical association|; HISTORY (expr) = the + * reference sets the expression yields. A reference set the edition does not + * carry simply contributes nothing. + */ + _applyHistorySupplement = function (ctx, hs, opContext) { + const HISTORICAL_ASSOCIATION = '900000000000522004'; + const PROFILES = { + MIN: ['900000000000527005'], + MOD: ['900000000000527005', '900000000000528000', '900000000000526001', '1186924009'] + }; + let refsetIdxs; + if (hs.profile === 'EXPLICIT') { + refsetIdxs = this._eclResolveSet(this._evalECLNode(hs.refsets, opContext), opContext); + } else if (hs.profile === 'MAX') { + const root = this.concepts.findConcept(HISTORICAL_ASSOCIATION); + refsetIdxs = root.found ? this._eclToIndexArray(this.filterIsA(HISTORICAL_ASSOCIATION, false)) : []; + } else { + refsetIdxs = PROFILES[hs.profile].map(id => this.concepts.findConcept(id)).filter(r => r.found).map(r => r.index); + } + + const base = this._eclResolveSet(ctx, opContext); + const inResult = new Set(base); + const added = new Set(); + for (const refsetIdx of refsetIdxs) { + if (opContext) opContext.deadCheck('ecl:history'); + const byTarget = this._historyIndex(refsetIdx); + if (!byTarget) continue; + for (const [target, sources] of byTarget) { + if (inResult.has(target)) { + for (const src of sources) { + if (!inResult.has(src)) added.add(src); + } + } + } + } + const result = new SnomedFilterContext(); + result.descendants = base.concat([...added]); + return result; + }; + + /** + * target concept index -> [referenced (inactive) concept indexes] for one + * association reference set, built once per edition. null when the concept + * is not a reference set with concept-valued members. + */ + _historyIndex = function (refsetIdx) { + this._historyIndexes = this._historyIndexes || new Map(); + if (this._historyIndexes.has(refsetIdx)) { + return this._historyIndexes.get(refsetIdx); + } + let index = null; + const refSet = this._findRefSetRecord(refsetIdx); + if (refSet && refSet.membersByRef && refSet.membersByRef !== 0xFFFFFFFF) { + for (const m of this.refSetMembers.getMembers(refSet.membersByRef) || []) { + if (m.kind !== 0 || !m.values) continue; + const values = this.refs.getReferences(m.values); + // targetComponentId is the first additional field, stored as (value, type) with type 1 = concept + if (!values || values.length < 2 || values[1] !== 1) continue; + index = index || new Map(); + const list = index.get(values[0]) || []; + list.push(m.ref); + index.set(values[0], list); + } + } + this._historyIndexes.set(refsetIdx, index); + return index; + }; + /** * Apply {{ C ... }} concept filters to a result: each concept is tested on * its own active flag, module, effectiveTime and definition status. Several @@ -1855,17 +2009,11 @@ class SnomedServices { * @returns {boolean} */ _attributeMatches = function (rels, attr, groupFilter, opContext) { - if (attr.reverse) { - throw new Error('ECL reverse attributes (R) are not yet supported'); - } if (!attr.comparison) { throw new Error('ECL attribute without a comparison is not supported'); } - if (attr.comparison.type !== ECLNodeType.EXPRESSION_COMPARISON) { - throw new Error('ECL concrete values (numeric, string and boolean attribute values) are not supported - concrete value relationships are not loaded'); - } - if (attr.comparison.operator !== ECLTokenType.EQUALS) { - throw new Error('ECL != in refinements is not yet supported'); + if (attr.comparison.type !== ECLNodeType.EXPRESSION_COMPARISON && !this.concreteValues) { + throw new Error('ECL concrete values (numeric, string and boolean attribute values) cannot be evaluated - this edition was imported without its concrete value relationships'); } const count = this._countAttributeMatches(rels, attr, groupFilter, opContext); @@ -1895,15 +2043,30 @@ class SnomedServices { // fresh per request, so memoising on the node is safe within a request. let resolved = attr._eclResolved; if (!resolved) { - resolved = { - types: this._eclAttributeTypes(attr.name, opContext), - valueSet: new Set(this._eclResolveSet(this._evalECLNode(attr.comparison.value, opContext), opContext)) - }; + resolved = { types: this._eclAttributeTypes(attr.name, opContext) }; + if (attr.comparison.type === ECLNodeType.EXPRESSION_COMPARISON) { + resolved.valueSet = new Set(this._eclResolveSet(this._evalECLNode(attr.comparison.value, opContext), opContext)); + } else { + resolved.concrete = this._eclConcreteMatcher(attr.comparison); + } attr._eclResolved = resolved; } const types = resolved.types; const valueSet = resolved.valueSet; + if (resolved.concrete) { + // A concrete-value comparison: only the concrete rows count, each + // occurrence being a (group, value) pair + const matched = new Set(); + for (const rel of rels) { + if (!rel.concrete) continue; + if (!types.any && !types.types.has(rel.relType)) continue; + if (groupFilter !== null && rel.group !== groupFilter) continue; + if (resolved.concrete(rel.concrete)) matched.add(rel.group + ':' + JSON.stringify(rel.concrete.value)); + } + return matched.size; + } + // ECL cardinality counts OCCURRENCES of a matching attribute, per role group // — not distinct values. ECL runs against the necessary normal form, where a // role group has no redundant attributes, so the same (type, value) in two @@ -1912,13 +2075,26 @@ class SnomedServices { // form, not to de-duplicating across groups). We therefore count distinct // (group, target) pairs: within a group filter this is the matching values // in that group; ungrouped it is the total occurrences across all groups. + // + // "!=" counts the occurrences whose value is NOT in the set (so "A != X" + // matches a concept with at least one A whose value is something other than + // X, and "[0..0] A != X" one whose every A is X). + // + // A reversed attribute ("R A = X") is matched against the concept's INBOUND + // relationships: the concept is the value of an A attribute on some concept + // in X. _conceptRelRecords supplies those tagged `inbound`; the records built + // for a post-coordinated expression have none, so R matches nothing there. + const notEquals = attr.comparison.operator === ECLTokenType.NOT_EQUALS; + const reverse = !!attr.reverse; const matched = new Set(); for (const rel of rels) { if (opContext) opContext.deadCheck('ecl:countAttributeMatches'); - if (!rel.active) continue; + if (!rel.active || rel.concrete) continue; + if (!!rel.inbound !== reverse) continue; if (!types.any && !types.types.has(rel.relType)) continue; if (groupFilter !== null && rel.group !== groupFilter) continue; - if (valueSet.has(rel.target)) matched.add(rel.group + ':' + rel.target); + const value = reverse ? rel.source : rel.target; + if (valueSet.has(value) !== notEquals) matched.add(rel.group + ':' + value); } return matched.size; }; @@ -1983,12 +2159,58 @@ class SnomedServices { // target, group} — the shape the refinement matchers consume. Sharing this // shape lets the same matchers run against a post-coordinated expression's // relationships (see _expressionRelRecords). + /** + * A predicate over a stored concrete value for one comparison node: + * numeric: = != < <= > >= against a number (integers and decimals compare as + * numbers, so #5 = #5.0) + * string: = / != against typed search terms (match / wild semantics, as for + * description terms), any of a set + * boolean: = / != true/false + * A value of another kind never matches (and never "not-matches" either: a + * string attribute is not "!= #5"). + */ + _eclConcreteMatcher = function (comparison) { + switch (comparison.type) { + case ECLNodeType.NUMERIC_COMPARISON: { + const n = Number(comparison.value); + const op = { [ECLTokenType.EQUALS]: '=', [ECLTokenType.NOT_EQUALS]: '!=', [ECLTokenType.LT]: '<', [ECLTokenType.LTE]: '<=', [ECLTokenType.GT]: '>', [ECLTokenType.GTE]: '>=' }[comparison.operator] || comparison.operator; + return (v) => v.kind === 'number' && this._eclCompareValues(v.value, op, [n], (a, b) => a - b); + } + case ECLNodeType.STRING_COMPARISON: { + const matchers = comparison.values.map(t => this._eclTermMatcher(typeof t === 'string' ? { kind: 'match', value: t } : t)); + const neg = comparison.operator === ECLTokenType.NOT_EQUALS; + return (v) => v.kind === 'string' && (matchers.some(m => m(v.value)) !== neg); + } + case ECLNodeType.BOOLEAN_COMPARISON: { + const neg = comparison.operator === ECLTokenType.NOT_EQUALS; + return (v) => v.kind === 'boolean' && ((v.value === comparison.value) !== neg); + } + default: + throw new Error(`Unsupported comparison ${comparison.type}`); + } + }; + _conceptRelRecords = function (conceptIdx) { - const relIdxs = this.getConceptRelationships(conceptIdx); const out = []; - for (const relIdx of relIdxs) { + for (const relIdx of this.getConceptRelationships(conceptIdx)) { out.push(this.relationships.getRelationship(relIdx)); } + // concrete value rows, for numeric / string / boolean comparisons + if (this.concreteValues) { + for (const cv of this.concreteValues.get(conceptIdx) || []) { + out.push({ active: true, relType: cv.relType, group: cv.group, target: -1, concrete: cv.concrete }); + } + } + // inbound relationships, for reversed (R) attributes + const concept = this.concepts.getConcept(conceptIdx); + if (concept.inbounds) { + for (const relIdx of this.refs.getReferences(concept.inbounds) || []) { + const rel = this.relationships.getRelationship(relIdx); + if (rel.active && rel.relType !== this.isAIndex) { + out.push({ ...rel, inbound: true }); + } + } + } return out; }; diff --git a/tx/fixtures/test-cases.yml b/tx/fixtures/test-cases.yml index 2daaec9c..2659ef0e 100644 --- a/tx/fixtures/test-cases.yml +++ b/tx/fixtures/test-cases.yml @@ -13,7 +13,7 @@ sources: - ucum:tx/data/ucum-essence.xml - loinc!:Loinc_2.83.db - snomed!:sct_intl_20250201.cache - - snomed:sct_test_20250909d.cache + - snomed:sct_test_20250909e.cache - omop:omop_v20250227.db - icd11:icd11-2026-01-b.db - npm:hl7.terminology=tho diff --git a/tx/importers/import-sct.module.js b/tx/importers/import-sct.module.js index 71de86eb..4aa2eda5 100644 --- a/tx/importers/import-sct.module.js +++ b/tx/importers/import-sct.module.js @@ -1139,6 +1139,9 @@ class SnomedImporter { // Step 10a: Read alternate identifiers (e.g. the LOINC codes of the LOINC Extension) await this.readIdentifiers(); + // Step 10b: Read concrete value relationships (sct2_RelationshipConcreteValues) + await this.readConcreteValues(); + // Step 11: Link concepts this.linkConcepts(); @@ -1169,6 +1172,7 @@ class SnomedImporter { descriptions: [], relationships: [], identifiers: [], + concreteValues: [], refsetDirectories: [] }; @@ -1224,6 +1228,8 @@ class SnomedImporter { files.relationships.push(filePath); } else if (files.identifiers && firstLine.startsWith('alternateIdentifier\teffectiveTime\tactive\tmoduleId\tidentifierSchemeId\treferencedComponentId')) { files.identifiers.push(filePath); + } else if (files.concreteValues && firstLine.startsWith('id\teffectiveTime\tactive\tmoduleId\tsourceId\tvalue\trelationshipGroup\ttypeId\tcharacteristicTypeId\tmodifierId')) { + files.concreteValues.push(filePath); } } catch (error) { console.log(`DEBUG: Error reading file ${filePath}: ${error.message}`); @@ -1782,6 +1788,58 @@ class SnomedImporter { } } + /** + * Read the concrete value relationship files (sct2_RelationshipConcreteValues_*): + * the numeric and string attribute values of concepts (strength, count of base of + * active ingredient, ...). Kept as rows [sourceId, typeId, group, value] - the value + * as written in RF2 ("#500", "#5.5", "\"text\"", "true") - for the ECL evaluator; + * they are not relationships to concepts, so they do not join the relationship + * table. Only active, inferred rows; where a row appears more than once the latest + * effectiveTime wins. + */ + async readConcreteValues() { + const rows = new Map(); // id -> {effectiveTime, active, row} + for (const file of this.files.concreteValues || []) { + const rl = readline.createInterface({ + input: fs.createReadStream(file), + crlfDelay: Infinity + }); + let lineCount = 0; + for await (const line of rl) { + lineCount++; + if (lineCount === 1 || !line) continue; + const parts = line.split('\t'); + if (parts.length < 10) continue; + const [id, effectiveTime, active, , sourceId, value, group, typeId, characteristicTypeId] = parts.map(p => p.trim()); + if (characteristicTypeId !== '900000000000011006') continue; // inferred only + const prior = rows.get(id); + if (!prior || prior.effectiveTime <= effectiveTime) { + rows.set(id, { effectiveTime, active: active === '1', row: [sourceId, typeId, parseInt(group, 10) || 0, value] }); + } + } + } + this.concreteValues = []; + let missing = 0; + for (const r of rows.values()) { + if (!r.active) continue; + let known = false; + try { + known = this.conceptMap.has(BigInt(r.row[0])) && this.conceptMap.has(BigInt(r.row[1])); + } catch { + known = false; + } + if (known) { + this.concreteValues.push(r.row); + } else { + missing++; + } + } + this.concreteValues.sort((a, b) => (a.join('|') < b.join('|') ? -1 : 1)); + if (this.concreteValues.length > 0 || missing > 0) { + console.log(`Concrete values: ${this.concreteValues.length}${missing ? ` (${missing} for concepts not in this edition, ignored)` : ''}`); + } + } + async readRelationships() { // Estimate total lines let totalLines = 0; @@ -2823,11 +2881,14 @@ class SnomedImporter { progressBar?.update(50); const identifiers = this.identifiers || []; + const concreteValues = this.concreteValues || []; await writer.writeCache({ - // 18 = 17 plus the alternate identifiers. Only used when there are some, so that - // caches without them stay readable by servers that only know 17 - version: identifiers.length > 0 ? '18' : '17', + // 17 = the base format; 18 = 17 plus the alternate identifiers; 19 = 18 plus + // the concrete value relationships. Each is only used when it has something + // to carry, so that caches without them stay readable by older servers. + version: concreteValues.length > 0 ? '19' : identifiers.length > 0 ? '18' : '17', identifiers: identifiers, + concreteValues: concreteValues, versionUri: this.isTesting ? this.config.uri.replace("/sct/", "/xsct/") : this.config.uri, versionDate: this.extractDateFromUri(this.config.uri), @@ -2914,10 +2975,14 @@ class SnomedCacheWriter { } buffers.push(this.writeInteger(data.defaultLanguage)); - if (data.version === '18') { + if (data.version === '18' || data.version === '19') { // alternate identifiers, as JSON: [[scheme, module, code, conceptId], ...] buffers.push(this.writeBytes(Buffer.from(JSON.stringify(data.identifiers || []), 'utf8'))); } + if (data.version === '19') { + // concrete value relationships, as JSON: [[sourceId, typeId, group, value], ...] + buffers.push(this.writeBytes(Buffer.from(JSON.stringify(data.concreteValues || []), 'utf8'))); + } // Write to file const finalBuffer = Buffer.concat(buffers); diff --git a/tx/sct/documentation.md b/tx/sct/documentation.md index f858b26f..fdc9e0d9 100644 --- a/tx/sct/documentation.md +++ b/tx/sct/documentation.md @@ -35,14 +35,20 @@ that the server does not use. `filterECL` is the one that matters. reference set's own fields (concept, integer and string typed from the RF2 file name). Since inactive member rows are dropped at import, `active = false` never matches. -- `AND` / `OR` / `MINUS` (and `,`), with the ECL 6.4 rule that conjunction and +- Refinements: `=` and `!=` against an expression value (`!=` counts the + occurrences whose value is outside the set, so `A != X` needs at least one + A that is not X, and `[0..0] A != X` means every A is X); reversed + attributes (`R A = X`: the concept is the value of an A on something in X, + evaluated over the concept's inbound relationships); attribute names that + are a concept, `*`, or any sub expression (`<< 762705008 = ...`); attribute + sets joined by `AND`, `OR` or `,`, with brackets; attribute groups, + including `AND`/`OR` inside and between groups; attribute and group + cardinality, including `[0..0]`. Cardinality counts occurrences per role + group, per the SNOMED Computable Languages Group ruling (see + `_countAttributeMatches`). +- `AND` / `OR` / `MINUS`, and `,` as a conjunction at the expression level as + well as in refinements, with the ECL 6.4 rule that conjunction and disjunction cannot be mixed without brackets. -- Refinements: `=` against an expression value; attribute names that are a - concept, `*`, or any sub expression (`<< 762705008 = ...`); attribute sets - joined by `AND`, `OR` or `,`, with brackets; attribute groups, including - `AND`/`OR` inside and between groups; attribute and group cardinality, - including `[0..0]`. Cardinality counts occurrences per role group, per the - SNOMED Computable Languages Group ruling (see `_countAttributeMatches`). - Dotted expressions (`< X . 363698007`, `. << 363698007`, `. (A OR B)`). - `{{ C ... }}` concept filters on `active`, `definitionStatus`, `definitionStatusId`, `moduleId` and `effectiveTime`. @@ -56,26 +62,37 @@ that the server does not use. `filterECL` is the one that matters. must be satisfied by a single description; separate blocks may be satisfied by different descriptions. Only active descriptions are considered unless the block has an `active` filter. +- Concrete values in refinements - numeric (`= #5`, `>= #1.5`, `!= #-1`), + string (`= "x"`, `= match:"x"`, a set) and boolean (`= true`) - against the + concrete value relationships, which the importer reads from + `sct2_RelationshipConcreteValues_*` into cache version 19. Numbers compare + as numbers (`#5` matches `#5.0`); strings use the same match / wild + semantics as description terms; a value of another kind never matches. + An edition imported before version 19 has no concrete values and reports + such a refinement as unsupported rather than answering with nothing. +- History supplements: `{{ + HISTORY }}` (= MAX), `HISTORY-MIN` (SAME AS), + `HISTORY-MOD` (SAME AS, WAS A, REPLACED BY, PARTIALLY EQUIVALENT TO), + `HISTORY-MAX` (every reference set under 900000000000522004 |Historical + association|), `HISTORY (*)` and `HISTORY (expression)`: the inactive + concepts that the selected association reference sets associate with a + concept in the result are added to it. A reference set the edition does not + carry contributes nothing. Whether a bare `HISTORY` means MAX is taken from + Snow Owl's reading of the specification - to be confirmed. +- Alternate identifiers (`LOINC#14394-1`, `"LOINC#14394-1"`): the scheme + alias names an identifier scheme (`IDENTIFIER_SCHEMES` in `cs-snomed.js`: + LOINC = 30051010000102, from the LOINC Extension's `sct2_Identifier` file, + cache version 18+), the code resolves to the concept(s) carrying it, and + hierarchy operators apply to them as to any set. An unknown scheme or a + code nobody carries is `INVALID_ECL`. - Comments (`/* ... */`), terms (`|...|`, not checked against descriptions). ### Parsed but reported as unsupported -- Reverse attributes (`R 363698007 = ...`). -- `!=` against an expression value in a refinement. -- Concrete values in refinements - numeric (`= #5`, `>= #1.5`), string - (`= "x"`, `= match:"x"`), boolean (`= true`). The importer does not read - `sct2_RelationshipConcreteValues_*`, so no concrete values are in the cache - to compare against. Supporting them means extending the importer and the - relationship record, and rebuilding every cache. -- Alternate identifiers (`"ICD10#A01.1"`, `SCTID#123`). No alternate - identifier reference sets are loaded. - Reference set field selection on member-of (`^ [targetComponentId] refset`, `^ [referencedComponentId, targetComponentId] refset`). Deliberate: the selected fields, and the referenced components of many reference sets, are descriptions or other non-concept components, which an expansion has no way to represent. -- History supplements (`{{ + HISTORY-MIN / MOD / MAX }}`). The association - reference sets they draw on are loaded, so this is implementable. ### Not parsed @@ -99,7 +116,13 @@ that the server does not use. `filterECL` is the one that matters. - `^` on a concept that is not a reference set yields an empty result rather than an error (Snowstorm's behaviour; agreed in the tx-ecosystem suite). `^` on a reference set whose members are descriptions (a language reference - set) is reported as unsupported - there are no concepts to return. + set) is reported as unsupported when that reference set is named explicitly + - there are no concepts to return. Inside a computed operand (`^ *`, + `^ (< 900000000000455006)`) such a reference set contributes nothing and + the rest of the operand still counts. +- An attribute name without a comparison (`< X : 363698007`, or `(< X : A = v + OR B)` where the OR binds inside the refinement) is a syntax error, as the + grammar requires. - The ECL result for a value set filter is cached per (value set, code system version), so the first expansion of a large constraint is slow and later ones are not. @@ -181,8 +204,9 @@ role group pairing. - Stated relationships and the OWL axiom reference sets. Everything is driven by the distributed inferred relationships; there is no classification. -- Concrete value relationships (see ECL above). -- Alternate identifier reference sets. +- Concrete value relationships are read into cache version 19 (October 2026); + caches built before that have none, and a concrete-value ECL refinement on + them is reported as unsupported. - Inactive reference set member rows (only active rows are stored). - Module ids and definition status on concepts are recorded by the current importer; caches built before mid-2026 have neither (every concept looks diff --git a/tx/sct/ecl.js b/tx/sct/ecl.js index 0b3ac2fe..f42bf0d4 100644 --- a/tx/sct/ecl.js +++ b/tx/sct/ecl.js @@ -538,8 +538,11 @@ class ECLParser { // this compound sequence and reject a mix (e.g. "<< A OR << B AND << C"). let compoundOp = null; - while (this.match(ECLTokenType.AND, ECLTokenType.OR, ECLTokenType.MINUS)) { - const operator = this.current; + // "," is a conjunction in the ECL grammar (conjunction = "AND" / ","), at + // this level as well as in refinements; a comma that reaches here was not + // claimed by a refinement. + while (this.match(ECLTokenType.AND, ECLTokenType.OR, ECLTokenType.MINUS, ECLTokenType.COMMA)) { + const operator = this.match(ECLTokenType.COMMA) ? { ...this.current, type: ECLTokenType.AND } : this.current; if (compoundOp !== null && operator.type !== compoundOp) { throw new Error( 'Ambiguous ECL expression: conjunction, disjunction and exclusion ' + @@ -726,18 +729,30 @@ class ECLParser { return { type: ECLNodeType.FILTER_CONSTRAINT, kind, filters }; } - if (kind !== 'M') { - // Skip to the closing }} keeping the raw text for diagnostics - const raw = []; - while (!this.match(ECLTokenType.RFILTER)) { - if (this.match(ECLTokenType.EOF)) { - this.error('Unterminated filter constraint - missing }}'); + if (kind === '+') { + // historySupplement = "{{" "+" "HISTORY" [ "-MIN" / "-MOD" / "-MAX" / "(" (expressionConstraint / "*") ")" ] "}}" + if (!this.match(ECLTokenType.SCTID) || !/^HISTORY(-MIN|-MOD|-MAX)?$/i.test(this.current.value)) { + this.error('Expected HISTORY, HISTORY-MIN, HISTORY-MOD or HISTORY-MAX after {{ +'); + } + const word = this.current.value.toUpperCase(); + this.advance(); + const node = { type: ECLNodeType.FILTER_CONSTRAINT, kind, profile: word === 'HISTORY' ? 'MAX' : word.substring(8) }; + if (this.match(ECLTokenType.LPAREN)) { + if (node.profile !== 'MAX' || word !== 'HISTORY') { + this.error('A history profile (HISTORY-MIN/MOD/MAX) cannot also name reference sets'); } - raw.push(this.current.value); this.advance(); + if (this.match(ECLTokenType.WILDCARD)) { + this.advance(); + node.profile = 'MAX'; + } else { + node.profile = 'EXPLICIT'; + node.refsets = this.parseExpressionConstraint(); + } + this.expect(ECLTokenType.RPAREN); } - this.advance(); // }} - return { type: ECLNodeType.FILTER_CONSTRAINT, kind, raw: raw.join(' ') }; + this.expect(ECLTokenType.RFILTER); + return node; } const filters = [this.parseMemberFilter()]; @@ -1430,6 +1445,14 @@ class ECLParser { }; } + if (!comparison) { + // eclAttribute = [cardinality] [reverseFlag] eclAttributeName comparison - a + // bare name is a syntax error. This is what "(A : B = C OR D)" trips on: + // the OR binds inside the refinement, so D is an attribute name without a + // comparison. + this.error('Expected a comparison operator (=, !=, <, <=, >, >=) after the attribute name'); + } + return { type: ECLNodeType.ATTRIBUTE, cardinality, diff --git a/tx/sct/structures.js b/tx/sct/structures.js index 679bf811..f0ae641c 100644 --- a/tx/sct/structures.js +++ b/tx/sct/structures.js @@ -1358,7 +1358,7 @@ class SnomedFileReader { } // Determine settings based on cache version string (matching Pascal logic) - if (result.cacheVersion === '17' || result.cacheVersion === '18') { // SNOMED_CACHE_VERSION_CURRENT (18 = 17 + alternate identifiers) + if (result.cacheVersion === '17' || result.cacheVersion === '18' || result.cacheVersion === '19') { // 18 = 17 + alternate identifiers; 19 = 18 + concrete values result.hasLangs = true; } else if (result.cacheVersion === '16') { // Assuming this is UTF8 version for now result.isUTF16 = false; @@ -1384,9 +1384,16 @@ class SnomedFileReader { // Alternate identifiers (version 18): [[scheme, module, code, conceptId], ...] result.identifiers = []; - if (result.cacheVersion === '18') { + if (result.cacheVersion === '18' || result.cacheVersion === '19') { result.identifiers = JSON.parse(this.readBytes().toString('utf8')); } + // Concrete value relationships (version 19): [[sourceId, typeId, group, value], ...]. + // null (not []) when the cache predates them, so the server can tell "none" from + // "not loaded". + result.concreteValues = null; + if (result.cacheVersion === '19') { + result.concreteValues = JSON.parse(this.readBytes().toString('utf8')); + } return result; } diff --git a/tx/tx.fhir.org.yml b/tx/tx.fhir.org.yml index b254ffbd..e8b46833 100644 --- a/tx/tx.fhir.org.yml +++ b/tx/tx.fhir.org.yml @@ -32,7 +32,7 @@ sources: - snomed:sct_us_20230301.cache - snomed:sct_us_20240301.cache - snomed:sct_us_20250901.cache - - snomed:sct_test_20250909d.cache + - snomed:sct_test_20250909e.cache - cpt:CodeSystem-cpt.db|cpt-2023-fragment-0.1.db - omop:omop_v20250227.db - npm:hl7.terminology=tho