diff --git a/conda-recipe/meta.yaml b/conda-recipe/meta.yaml index 385fc12..ea05eff 100644 --- a/conda-recipe/meta.yaml +++ b/conda-recipe/meta.yaml @@ -17,6 +17,7 @@ build: - vcf-rdfizer=vcf_rdfizer:main - vcf-rdfizer-rules=vcf_rdfizer_rules:main - vcf-rdfizer-link=vcf_rdfizer_link:main + - vcf-rdfizer-policy=vcf_rdfizer_policy:main requirements: host: @@ -36,16 +37,20 @@ test: - vcf-rdfizer --help - vcf-rdfizer-rules --help - vcf-rdfizer-link list + - vcf-rdfizer-policy --version # The vendored vocabulary is package data, not code, so no import touches # it: a dropped package-data glob would pass every check above and only # surface when SHACL validation runs. One file per glob in pyproject.toml. - python -c "import importlib.resources as r; d = r.files('vcf_rdfizer_data'); assert d.joinpath('VOCABULARY_PROVENANCE.json').is_file() and d.joinpath('shacl').joinpath('vcf-4.5.shacl.ttl').is_file() and d.joinpath('ontology').joinpath('vcf-core-vocabulary.bundle.ttl').is_file()" + # The policy plug-in's profile and default purpose vocabulary are package data too. + - python -c "import importlib.resources as r; d = r.files('vcf_rdfizer_data.policy'); assert d.joinpath('vcf-core-profile.ttl').is_file() and d.joinpath('duo-subset.ttl').is_file()" imports: - vcf_rdfizer # The vocabulary terms, VCF-version model and lexical parsers. Importing it # here catches a packaging miss that would otherwise only surface at # conversion time, since vcf_rdfizer imports it at module load. - vcf_rdfizer_vocab + - vcf_rdfizer_policies about: home: https://github.com/ecrum19/VCF-RDFizer diff --git a/docs/README.md b/docs/README.md index 8d6e44d..b05531e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -54,6 +54,12 @@ deciding whether the tool fits your problem, read graph: an ODRL profile with graph selectors, three enforcement tiers, and verification — plus a candid account of why access control is not anonymization when the genotypes are themselves identifiers. +- **[Policy attachment v0.1.0](policy-demonstrator.md)** — *implemented; + walkthrough in `examples/policy/`.* The first slice of the privacy design: + ODRL policies attached to any resource or declared graph selection, a + partition rule that says what a withheld resource takes with it, per-request + release views, and checks — including a VCF-text oracle — that each view + withholds exactly what it should. Selectors and partitions are Turtle, not code. ### What the graph looks like diff --git a/docs/limitations.md b/docs/limitations.md index d265cb1..9869791 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -267,7 +267,12 @@ record what an artifact was permitted to contain. Three consequences today: amount of label removal makes a released genotype graph non-identifying. The plan is [`privacy-policy-design.md`](privacy-policy-design.md), which is -explicit that what it offers is *governed release*, not anonymization. +explicit that what it offers is *governed release*, not anonymization. Its first +slice exists as a demonstrator, [`vcf-rdfizer-policy`](policy-demonstrator.md) +v0.1.0: ODRL policies on files and on declared graph selections (region and +variant ship; others are Turtle declarations), verified release views, in +memory and after conversion. It changes none of the points above for a +graph produced by a normal conversion. **No clinical claims.** The tool transcribes a VCF. It does not interpret, annotate, prioritize, or assess pathogenicity, and its output should not be diff --git a/docs/policy-demonstrator.md b/docs/policy-demonstrator.md new file mode 100644 index 0000000..78e8fe2 --- /dev/null +++ b/docs/policy-demonstrator.md @@ -0,0 +1,446 @@ +# Policy attachment — v0.1.0 + +*Part of the [VCF-RDFizer documentation](README.md). Status: **implemented +(v0.1.0)**; walkthrough in [`examples/policy/`](../examples/policy/README.md); +tests in [vcf-rdfizer-testing `plugin-tests/policy/`](https://github.com/ecrum19/vcf-rdfizer-testing/tree/main/plugin-tests). +This is the first slice of [`privacy-policy-design.md`](privacy-policy-design.md): +it uses that document's vocabulary and rules, implements a subset of them, and +states plainly what it leaves out.* + +`vcf-rdfizer-policy` attaches ODRL policies to an RDF graph and produces one +release view per request. It is built from three generic steps, and each is +configured in Turtle, not code: + +1. **Select.** A rule's target is either one resource IRI or a *selection*. A + selection is computed by a selector type, declared as a SPARQL `SELECT`. +2. **Partition.** A *profile* says what a selected resource owns, so that + withholding a record also withholds its call, its alleles and its genotypes. +3. **Decide.** ODRL semantics: a binding permission must own a resource, no + binding prohibition may, and deny wins. + +The bundled **VCF Core profile** makes this work on graphs written by +VCF-RDFizer, with region and variant selectors. Anyone can add a selector, such +as FILTER state, QUAL, an INFO key or genes from a linkset, by declaring it. +The same engine partitions any other RDF graph with a different profile. + +This is **governed release, not anonymization**, and every string the tool +emits says so. See [`privacy-policy-design.md` §1](privacy-policy-design.md#1-the-uncomfortable-premise). + +--- + +## 1. Why + +Consent for genomic data is recorded per dataset. A repository record carries +one data-use code, and VCF itself has no place to state a policy at all. Real +policies are finer than that: + +- *this participant* consented to health research but not to clinical use; +- *this region* holds actionable incidental findings, readable only for + clinical care; +- *this variant* is restricted to disease-specific research; +- *this participant* withdrew, and that overrides everything else. + +An RDF graph can hold every one of those statements next to the data it +governs, in standard vocabularies: ODRL for the rules, GA4GH DUO for the +purposes, PROV for what was released. It can also answer questions that mix +the two (§6). A VCF-based workflow would need an external spreadsheet, and its +enforcement would live in someone's scripts. + +--- + +## 2. How it works + +```text +policy.ttl ──► rules ─────────────────────────────┐ + │ target │ kind, assignee, purpose constraints + ▼ ▼ + ┌───────────┐ resources ┌───────────┐ owned sets ┌──────────┐ +graph ────►│ select │──────────────►│ partition │───────────────►│ decide │──► view, decisions, + └───────────┘ └───────────┘ └──────────┘ manifest + selector types ownership rule ODRL, deny wins + (SPARQL, profile) (profile) (engine.py) +``` + +| Layer | What it is | Where | +| --- | --- | --- | +| Engine | Select, partition, decide; knows only ODRL, selector declarations, ownership rules and a purpose hierarchy. It never mentions VCF | `vcf_rdfizer_policies/engine.py` | +| Profile | Selector types, the ownership rule, and the reporting-unit query, in Turtle | `vcf_rdfizer_data/policy/vcf-core-profile.ttl` (bundled), or `--profile` | +| Purposes | An RDFS or SKOS hierarchy | `vcf_rdfizer_data/policy/duo-subset.ttl` (bundled), or `--purposes` | +| VCF oracle | An independent check of a view against the source VCF text (§7) | `vcf_rdfizer_policies/vcf_oracle.py` | + +### 2.1 Selector types + +A selector type is a SPARQL `SELECT` that projects `?resource`, plus the +parameters a policy supplies. Each `vcfp:parameter` is a property the +selector node must carry, and its value is bound to the query variable named +after the property's local name: `vcfp:start` binds `?start`. An optional +`vcfp:violations` query lists reasons the selector cannot apply to a graph. +Any row it returns stops evaluation; the VCF Core selectors use it to require +that every file declares the policy's assembly. + +This is the shipped region selector, in full: + +```turtle +vcfp:RegionSelector a vcfp:SelectorType ; + vcfp:parameter vcfp:assembly , vcfp:chrom , vcfp:start , vcfp:end ; + vcfp:query """ + PREFIX vcfc: + SELECT ?resource WHERE { + ?resource a vcfc:VCFRecord ; vcfc:chrom ?c ; vcfc:pos ?pos . + FILTER(STR(?c) = STR(?chrom) && ?pos >= ?start && ?pos <= ?end) }""" ; + vcfp:violations """ … files whose vcfc:referenceGenome is not ?assembly … """ . +``` + +A policy uses it like this: + +```turtle +ex:brca1 a odrl:Asset , vcfp:GraphSelection ; + vcfp:selector [ a vcfp:RegionSelector ; vcfp:assembly "GRCh38" ; + vcfp:chrom "chr17" ; vcfp:start 43044295 ; vcfp:end 43125483 ] . +``` + +Selector types are read from the profile files, and also from the policy file +itself, so a policy can bring its own (§8). + +### 2.2 Partitioning + +A withheld resource takes with it everything it **owns**: + +- the resources its `vcfp:ownershipPath` (a SPARQL property path) reaches; +- when `vcfp:iriSubtree` is true, every IRI beneath any of those, after a `#` + or `/`. + +The VCF Core profile sets the path to `vcfc:hasCall/vcfc:hasSampleCall?` and +turns the subtree rule on, which matches the converter's IRIs +([conversion §6](conversion.md#6-iri-templates)): + +- a file owns everything under `file://NAME#`: its header, sample set and + records; +- a record `…#record/9` owns its alleles (`…#record/9/allele/…`), its call + `…#call/9` with that call's INFO values and condensed matrix, and, through + the call, its expanded per-sample calls `…#sample/9/P001`. + +A triple is released when its subject is released, and its object too if the +object is a node of the graph. So no view ever points at something it doesn't +contain. + +### 2.3 Reporting units + +`vcfp:unitQuery` names what the per-unit report counts. It must project +`?resource` and `?group`, and any other variables become report columns. The +VCF Core profile reports records grouped by file, with chrom, pos, ref and alt. + +--- + +## 3. Policies + +Namespace `vcfp:` = `https://w3id.org/vcf-rdfizer/policy#`. + +| Construct | Supported | Notes | +| --- | --- | --- | +| `odrl:Set` / `Policy` / `Offer` / `Agreement` | yes | | +| `odrl:permission`, `odrl:prohibition` | yes | | +| `odrl:action odrl:read` | yes | The only action evaluated | +| `odrl:target` = a resource IRI | yes | The resource and everything it owns, e.g. `` | +| `odrl:target` = a `vcfp:GraphSelection` | yes | One `vcfp:selector`, whose type the profile or policy declares | +| `odrl:assignee` | yes | An IRI, or `odrl:All` | +| `odrl:constraint` on `odrl:purpose`, `isAnyOf` / `isNoneOf` | yes | Terms of the purpose vocabulary (§3.1) | +| `odrl:duty` | recorded | Copied into the manifest as obligations. **Not enforced** | +| `odrl:conflict odrl:prohibit` | required | Deny wins; any other value is refused | +| `vcfp:transform vcfp:drop` | yes | The only effect | +| anything else on a rule (`odrl:refinement`, `odrl:remedy`, …) | **refused** | It could change what the rule means | +| `generalize`, `pseudonymize`, `threshold`, `maskVectorPositions` | **refused** | Later versions (§10) | + +**What the engine cannot evaluate, it refuses.** A missing parameter, an +undeclared selector type, a purpose outside the vocabulary, a selector whose +`violations` query returns rows: each stops the run with an error, never a +warning. A tool that silently skipped a rule would demonstrate the opposite of +its purpose. + +### 3.1 Purposes + +A requester's purpose satisfies a term when it **is that term or a narrower +one**, following `rdfs:subClassOf` or `skos:broader`. So any RDFS or SKOS +vocabulary works, passed with `--purposes`. Names resolve as full IRIs or +through the vocabulary's own prefixes. + +The default is four GA4GH DUO terms (`duo-subset.ttl`), copied from DUO release +2021-02-23 together with their subclass links: + +| Code | IRI | In the example | +| --- | --- | --- | +| GRU, general research use | `DUO:0000042` | P001, P002 consent; the general-research requester | +| HMB, health/medical/biomedical research | `DUO:0000006` | P003 consent | +| DS, disease-specific research | `DUO:0000007` | P005 consent; the Alzheimer's study | +| CC, clinical care use | `DUO:0000043` | P001, P002 consent; the clinical requester | + +DS ⊑ HMB ⊑ GRU, so a disease-specific study falls within a general-research +consent, but not the reverse. DUO files CC as a data use *modifier*, not a +permission. Here it is a purpose like the others, and matches only itself. + +--- + +## 4. Decision rules (normative) + +A **request** is `(assignee a, purpose p)`. A rule *binds* a request when its +assignee is `odrl:All` or `a`, and each constraint holds: + +- `isAnyOf S` holds when `p ⊑ s` for some `s` in `S`; +- `isNoneOf S` holds when `p ⊑ s` for none. + +For each binding rule, the engine selects its target's resources (§2.1) and +takes everything they own (§2.2). + +**A resource is released if and only if:** + +1. some binding **permission** owns it — the default-deny rule: no + permission, no release; **and** +2. no binding **prohibition** owns it — deny wins. + +A permission on a file and a prohibition on a region inside it combine as +"everything in the file except the region". No "most-specific target wins" +rule is needed, which is fortunate, since ODRL does not define one. + +--- + +## 5. The example: one cohort, three requesters + +`examples/policy/` holds five synthetic participants, one small single-sample +VCF each: 23–34 records, 139 in all. The positions are GRCh38 and fall in real +loci: *BRCA1* (chr17:43,044,295–43,125,483), *APOE* (including the real +rs429358 T>C and rs7412 C>T), and background sites on chr1 and chr20. + +Every other allele, and every genotype, is synthetic, drawn from a fixed seed. +The design's first premise is that genotypes identify people, so a privacy +example should not be built on real ones. + +The generator also plants boundary cases: +- records one base inside and one base outside each end of the *BRCA1* window; +- a decoy T>G at rs429358's position, to test that the variant rule matches + alleles and not just position. + +Single-sample files make each participant's consent a policy on their own +file, and a withdrawal the withholding of one file. They also mean the +condensed profile's per-sample vectors have one entry each, so nothing ever +has to be masked *inside* a literal. + +`policy.ttl` holds: +- five consents, one per file (P004's includes the withdrawal, a prohibition); +- a cohort policy with two prohibitions, *BRCA1* unless the purpose is CC and + rs429358 unless the purpose is DS. + +| Requester | Purpose | P001 | P002 | P003 | P004 | P005 | *BRCA1* | rs429358 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| General-research consortium | GRU | 21 | 16 | — | — | — | withheld | withheld | +| Alzheimer's consortium | DS | 22 | 16 | 18 | — | 20 | withheld | **released** | +| Clinical genetics lab | CC | 33 | 23 | — | — | — | **released** | withheld | + +The numbers are records released from each file. Every cell follows from §4: +- P003's HMB consent admits DS but not GRU; +- P004's withdrawal is a prohibition, so it wins everywhere; +- each cohort rule is lifted for exactly one purpose. + +Released records total 37, 76 and 56 of 139, with 7,869, 4,649 and 6,539 +triples withheld in the expanded profile. The condensed profile releases the +same records. + +--- + +## 6. What it writes + +### 6.1 `attach`: policies in the graph + +`attach` merges the policies into the data. It links each directly targeted +resource, and each resource a selection selects, to its policy: + +```turtle + odrl:hasPolicy ex:consent-P003 . +ex:brca1 vcfp:selects , … . + odrl:hasPolicy ex:cohort . +``` + +Policy and data are then one graph, and one query asks which records a +general-research collaborator may not receive, and under which rule. It needs +no knowledge of the selectors: + +```sparql +PREFIX odrl: +PREFIX vcfc: +SELECT ?record ?chrom ?pos ?rule WHERE { + ?record a vcfc:VCFRecord ; vcfc:chrom ?chrom ; vcfc:pos ?pos ; odrl:hasPolicy ?policy . + ?policy odrl:prohibition ?rule . } +``` + +### 6.2 `evaluate`: one request's release + +| File | Contents | +| --- | --- | +| `view.nt` | The released triples, sorted | +| `decisions.csv` | One row per reporting unit: the unit query's columns, released or not, and the rule that decided it | +| `summary.json` | Counts per group and per deciding reason | +| `manifest.ttl` | The `vcfp:ReleaseView`, below | + +```turtle +<#release> a vcfp:ReleaseView ; + vcfp:derivedFrom , … , ; + vcfp:policy ex:cohort , ex:consent-P001 , … ; + vcfp:policyDigest "sha256:…" ; + vcfp:request [ odrl:assignee ; + odrl:purpose obo:DUO_0000007 ] ; + vcfp:recordsReleased 76 ; vcfp:recordsWithheld 63 ; + vcfp:groupsWithheld 1 ; vcfp:triplesWithheld 4649 ; + vcfp:obligation [ odrl:action odrl:attribute ] ; + vcfp:disclosureModel "governed release; not anonymization" ; + prov:wasGeneratedBy ; + prov:generatedAtTime "…"^^xsd:dateTime . +``` + +The policy is recorded by digest as well as by IRI, since an IRI's content can +change after the fact. Views keep the source IRIs, as there is no +pseudonymization yet (§10), and `vcfp:disclosureModel` says so. + +--- + +## 7. Checking a view + +`vcf-rdfizer-policy check --view DIR --rdf SOURCE --policy P [--vcf …]` fails +on each of the following. + +**Structural checks, for any graph:** +1. The manifest's policy digest doesn't match the policy. +2. Anything a binding prohibition owns appears in the view, as subject or + object. +3. A subject in the view is owned by no binding permission (default-deny). +4. A triple points at a node of the source that the view does not contain. + +**The VCF oracle (`--vcf`):** it reads the source VCFs as text and builds a +minimal VCF Core graph of the fixed columns except INFO: CHROM, POS, ID, REF, +ALT, QUAL and FILTER. It then evaluates the same policy on that graph. The +view's records must equal the records released there **exactly**: an extra +record is a leak, and a missing one is over-withholding. + +The structural checks reuse the engine's selectors, so they confirm that a +view honours the policy, but they cannot catch a wrong selector. The oracle +can, because its input never went through the converter. A selector that +reads INFO, FORMAT or the header is outside what the oracle models, so don't +pass `--vcf` for such a policy. + +**The checks earned their place early.** In development, a withdrawn file's +header and sample set stayed in every view even though every record decision +was right, so nothing looked wrong from the counts. The check named the leaked +IRI. That failure is why the partition is defined by ownership (§2.2) rather +than by a list of record subtrees. + +**The tests** are in +[vcf-rdfizer-testing `plugin-tests/policy/`](https://github.com/ecrum19/vcf-rdfizer-testing/tree/main/plugin-tests), +kept apart from this code and run by hand. They cover: +- the §5 grid cell by cell, in both profiles; +- the refusals; +- the generality cases: a selector declared in a policy, a non-VCF graph with + a property-path partition, a SKOS vocabulary; +- mutation tests, where each planted fault must fail `check`: a reinstated + prohibited record, a restored triple of the withdrawn file, a deleted + record, a dangling reference, a swapped view, and a changed policy. + +--- + +## 8. Extending it without code + +**A new selector.** Declare a `vcfp:SelectorType`, either in the policy file +or in a file passed with `--profile`. +[`examples/policy/custom-selector.ttl`](../examples/policy/custom-selector.ttl) +adds "records whose QUAL is below a threshold" as one short declaration, and +withholds 38 of the example's 139 records at a threshold of 60. Selectors for +FILTER state, an INFO value, a sample (for multi-sample files), or genes from a +linkset all follow the same pattern. + +**Another kind of graph.** Pass a profile with a different ownership rule and +unit query. For a graph of documents that own their sections through +`ex:hasSection`, with opaque IRIs: + +```turtle +ex:Docs a vcfp:Profile ; + vcfp:ownershipPath "*" ; + vcfp:iriSubtree false ; + vcfp:unitQuery "SELECT ?resource ?group WHERE { ?group ?resource }" . +``` + +**Other purposes.** Pass `--purposes` a vocabulary: full DUO, or a local SKOS +scheme. + +--- + +## 9. The command + +```bash +vcf-rdfizer-policy explain --policy policy.ttl +vcf-rdfizer-policy attach --rdf P00*.nt.gz --policy policy.ttl -o annotated.nt +vcf-rdfizer-policy evaluate --rdf P00*.nt.gz --policy policy.ttl \ + --assignee https://example.org/party/alz-consortium --purpose DUO:0000007 -o views/alz +vcf-rdfizer-policy check --view views/alz --rdf P00*.nt.gz --policy policy.ttl --vcf P00*.vcf +``` + +Every subcommand takes: +- `--profile`, repeatable; a file or `vcf-core` (the default); +- `--purposes`, the purpose vocabulary. + +The command runs on the host and needs `rdflib`, not Docker. It evaluates in +memory, and refuses graphs over 5M triples. + +| Exit code | Meaning | +| --- | --- | +| 0 | Success; for `check`, every check passed | +| 1 | `check` found a failure | +| 2 | The policy, profile, graph or request cannot be evaluated | + +```text +vcf_rdfizer_policy.py the command +vcf_rdfizer_policies/ + engine.py select, partition, decide -- generic + policy.py ODRL -> rules; refuses what it cannot evaluate + profile.py selector types and the ownership rule, from Turtle + vocabulary.py purpose hierarchies (RDFS / SKOS) + release.py evaluate, attach, manifest + check.py the structural checks -- generic + vcf_oracle.py the VCF-text oracle + graphs.py loading, IRI hierarchy +vcf_rdfizer_data/policy/ + vcf-core-profile.ttl the VCF Core profile: region and variant selectors, ownership, units + duo-subset.ttl the default purpose vocabulary + vcfp-0.1.ttl the profile terms +examples/policy/ the cohort, policy.ttl, custom-selector.ttl, run_demo.sh +``` + +--- + +## 10. Beyond v0.1.0 + +Mapped onto the full design's build order +([`privacy-policy-design.md` §13](privacy-policy-design.md#13-build-order)). +Selectors are now declarations, so the full design's sample, field, header and +class selectors are Turtle, not engine work. What remains needs code: + +| Version | Adds | Design § | +| --- | --- | --- | +| **v0.2** | Multi-sample files: a sample selector (a declaration) for expanded graphs, and `maskVectorPositions` for condensed ones, which rewrites a literal and so needs code; the `generalize` effect (genotype → carrier status) | §4.2, §8 | +| **v0.3** | Enforcement during conversion (TSV and emitter tiers), and a streaming evaluator, lifting the size limit | §5 | +| **v0.4** | Pseudonymization: IRI re-minting with per-release keys | §7 | +| **v0.5** | Full DUO with release pinning and MONDO qualifiers; `policy diff`; enforced duties with an audit sink | §4.3, §12 | +| later | `threshold`; query-time rewriting for an operated endpoint | §5, §9 | + +Two rules carry forward unchanged: + +- **What cannot be enforced stops the run.** A version may widen what can be + enforced, but never turn that error into a warning. +- **Every version ships with its checks and mutation tests.** "A crude + redaction that is verified is worth more than a sophisticated one that is + not." + +--- + +## See also + +- [Privacy policy design](privacy-policy-design.md) — the full design this is a slice of +- [Data linking](datalinking.md) — the sibling plug-in; the same manifest and provenance pattern +- [Conversion §6](conversion.md#6-iri-templates) — the IRIs the VCF Core ownership rule follows +- [Validation methodology](validation-methodology.md) — why a policy claim needs an oracle diff --git a/docs/privacy-policy-design.md b/docs/privacy-policy-design.md index 228663a..ce0a55d 100644 --- a/docs/privacy-policy-design.md +++ b/docs/privacy-policy-design.md @@ -6,6 +6,12 @@ contract before code depends on it — and, in this case, to be explicit about what a policy layer can and cannot achieve, because the failure mode of a privacy feature is a false sense of safety.* +*The first slice is implemented and specified separately: +[`policy-demonstrator.md`](policy-demonstrator.md), v0.1.0 — ODRL attached to +files, regions and variants over single-sample fixtures, with per-request views +verified against an oracle. It implements a subset of this document and cites +the sections it defers.* + VCF-RDFizer currently makes one decision about disclosure: it converts everything. Every sample, every genotype, every header line, every free-text `Description` goes into the graph, and every artifact is all-or-nothing. The diff --git a/docs/roadmap.md b/docs/roadmap.md index 649efe3..1658386 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -141,7 +141,10 @@ The proposal is an ODRL profile whose assets are **graph selectors** (by class, predicate, sample, genomic region, declared field or pattern), compiled to a release plan and enforced at the cheapest available point in the existing pipeline — TSV pre-filtering, emitter-time filtering, or a post-hoc pass. -Full design in [`privacy-policy-design.md`](privacy-policy-design.md). +Full design in [`privacy-policy-design.md`](privacy-policy-design.md). The first +slice, a v0.1.0 demonstrator over single-sample fixtures, is specified in +[`policy-demonstrator.md`](policy-demonstrator.md); its §10 maps later versions +onto the full design's build order. Two findings from that design are worth surfacing here because they affect work outside it: diff --git a/examples/policy/P001.vcf b/examples/policy/P001.vcf new file mode 100644 index 0000000..06fc7eb --- /dev/null +++ b/examples/policy/P001.vcf @@ -0,0 +1,46 @@ +##fileformat=VCFv4.3 +##fileDate=20260925 +##source=vcf-rdfizer-policy-demonstrator-fixture +##reference=GRCh38 +##contig= +##contig= +##contig= +##contig= +##INFO= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT P001 +chr1 11913838 . A T 87 PASS DP=38 GT:DP 1/1:38 +chr1 33711661 . G C 61 PASS DP=37 GT:DP 1/1:37 +chr1 45899293 . C T 74 PASS DP=52 GT:DP 0/1:52 +chr1 138834739 . T G 70 PASS DP=46 GT:DP 0/1:46 +chr1 139602254 . G T 84 PASS DP=49 GT:DP 1/1:49 +chr1 162068500 . A G 57 PASS DP=33 GT:DP 1/1:33 +chr1 227621759 . C G 88 PASS DP=51 GT:DP 0/1:51 +chr17 43044294 . A C 62 PASS DP=45 GT:DP 0/1:45 +chr17 43044295 . C G 91 PASS DP=46 GT:DP 0/1:46 +chr17 43055276 . G A 86 PASS DP=51 GT:DP 0/1:51 +chr17 43059666 . G A 55 PASS DP=24 GT:DP 0/1:24 +chr17 43061329 . G A 75 PASS DP=40 GT:DP 1/1:40 +chr17 43075071 . C G 70 PASS DP=50 GT:DP 1/1:50 +chr17 43087473 . A C 52 PASS DP=45 GT:DP 0/1:45 +chr17 43095759 . C T 52 PASS DP=51 GT:DP 1/1:51 +chr17 43100876 . G T 93 PASS DP=36 GT:DP 0/1:36 +chr17 43108081 . T C 60 PASS DP=22 GT:DP 0/1:22 +chr17 43111166 . C A 71 PASS DP=29 GT:DP 0/1:29 +chr17 43118559 . T C 56 PASS DP=50 GT:DP 0/1:50 +chr17 43125483 . C T 76 PASS DP=40 GT:DP 1/1:40 +chr17 43125484 . G C 89 PASS DP=59 GT:DP 0/1:59 +chr19 44906345 . T G 62 PASS DP=52 GT:DP 0/1:52 +chr19 44907272 . T G 85 PASS DP=29 GT:DP 0/1:29 +chr19 44907527 . G C 78 PASS DP=23 GT:DP 1/1:23 +chr19 44908684 rs429358 T C 56 PASS DP=58 GT:DP 0/1:58 +chr20 12657015 . G A 94 PASS DP=39 GT:DP 0/1:39 +chr20 13974064 . C G 73 PASS DP=22 GT:DP 0/1:22 +chr20 22188117 . A C 99 PASS DP=47 GT:DP 0/1:47 +chr20 30588233 . T G 55 PASS DP=22 GT:DP 0/1:22 +chr20 36112198 . C A 50 PASS DP=38 GT:DP 1/1:38 +chr20 38576612 . C G 94 PASS DP=26 GT:DP 0/1:26 +chr20 46050645 . A G 71 PASS DP=36 GT:DP 0/1:36 +chr20 47351745 . C A 81 PASS DP=24 GT:DP 1/1:24 +chr20 50908083 . G C 89 PASS DP=49 GT:DP 0/1:49 diff --git a/examples/policy/P002.vcf b/examples/policy/P002.vcf new file mode 100644 index 0000000..f0e257c --- /dev/null +++ b/examples/policy/P002.vcf @@ -0,0 +1,35 @@ +##fileformat=VCFv4.3 +##fileDate=20260925 +##source=vcf-rdfizer-policy-demonstrator-fixture +##reference=GRCh38 +##contig= +##contig= +##contig= +##contig= +##INFO= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT P002 +chr1 11913838 . A T 90 PASS DP=47 GT:DP 0/1:47 +chr1 45899293 . C T 68 PASS DP=54 GT:DP 0/1:54 +chr1 139602254 . G T 56 PASS DP=34 GT:DP 1/1:34 +chr1 162068500 . A G 61 PASS DP=35 GT:DP 1/1:35 +chr1 231965065 . C T 94 PASS DP=57 GT:DP 0/1:57 +chr1 232595915 . T C 57 PASS DP=27 GT:DP 0/1:27 +chr17 43061329 . G A 98 PASS DP=32 GT:DP 0/1:32 +chr17 43075071 . C G 80 PASS DP=52 GT:DP 0/1:52 +chr17 43087473 . A C 53 PASS DP=58 GT:DP 1/1:58 +chr17 43095759 . C T 96 PASS DP=37 GT:DP 1/1:37 +chr17 43111166 . C A 92 PASS DP=32 GT:DP 0/1:32 +chr17 43118500 . C T 67 PASS DP=29 GT:DP 0/1:29 +chr17 43124696 . T A 52 PASS DP=56 GT:DP 1/1:56 +chr19 44907675 . A C 77 PASS DP=22 GT:DP 0/1:22 +chr19 44908684 . T G 50 PASS DP=28 GT:DP 1/1:28 +chr19 44908822 rs7412 C T 55 PASS DP=38 GT:DP 0/1:38 +chr20 12657015 . G A 82 PASS DP=31 GT:DP 0/1:31 +chr20 13974064 . C G 65 PASS DP=56 GT:DP 0/1:56 +chr20 36112198 . C A 55 PASS DP=20 GT:DP 0/1:20 +chr20 38576612 . C G 63 PASS DP=39 GT:DP 0/1:39 +chr20 46050645 . A G 86 PASS DP=34 GT:DP 1/1:34 +chr20 47351745 . C A 51 PASS DP=20 GT:DP 0/1:20 +chr20 50908083 . G C 72 PASS DP=60 GT:DP 0/1:60 diff --git a/examples/policy/P003.vcf b/examples/policy/P003.vcf new file mode 100644 index 0000000..cb1f6ba --- /dev/null +++ b/examples/policy/P003.vcf @@ -0,0 +1,39 @@ +##fileformat=VCFv4.3 +##fileDate=20260925 +##source=vcf-rdfizer-policy-demonstrator-fixture +##reference=GRCh38 +##contig= +##contig= +##contig= +##contig= +##INFO= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT P003 +chr1 33711661 . G C 53 PASS DP=20 GT:DP 0/1:20 +chr1 45899293 . C T 70 PASS DP=20 GT:DP 0/1:20 +chr1 138834739 . T G 92 PASS DP=35 GT:DP 1/1:35 +chr1 152491455 . A G 56 PASS DP=42 GT:DP 0/1:42 +chr1 231965065 . C T 69 PASS DP=22 GT:DP 0/1:22 +chr1 232595915 . T C 62 PASS DP=47 GT:DP 0/1:47 +chr17 43055276 . G A 59 PASS DP=50 GT:DP 1/1:50 +chr17 43059666 . G A 91 PASS DP=43 GT:DP 1/1:43 +chr17 43061329 . G A 85 PASS DP=35 GT:DP 1/1:35 +chr17 43075071 . C G 52 PASS DP=50 GT:DP 0/1:50 +chr17 43087473 . A C 65 PASS DP=38 GT:DP 1/1:38 +chr17 43095759 . C T 69 PASS DP=24 GT:DP 0/1:24 +chr17 43100876 . G T 69 PASS DP=33 GT:DP 0/1:33 +chr17 43108081 . T C 79 PASS DP=46 GT:DP 0/1:46 +chr17 43124696 . T A 54 PASS DP=22 GT:DP 1/1:22 +chr19 44906345 . T G 75 PASS DP=57 GT:DP 0/1:57 +chr19 44907272 . T G 95 PASS DP=35 GT:DP 0/1:35 +chr19 44907527 . G C 51 PASS DP=33 GT:DP 1/1:33 +chr19 44907675 . A C 52 PASS DP=27 GT:DP 0/1:27 +chr19 44908684 rs429358 T C 60 PASS DP=33 GT:DP 0/1:33 +chr19 44908822 rs7412 C T 58 PASS DP=39 GT:DP 0/1:39 +chr20 13974064 . C G 68 PASS DP=27 GT:DP 0/1:27 +chr20 20513631 . T A 51 PASS DP=44 GT:DP 0/1:44 +chr20 22188117 . A C 68 PASS DP=44 GT:DP 0/1:44 +chr20 30588233 . T G 58 PASS DP=47 GT:DP 0/1:47 +chr20 36112198 . C A 60 PASS DP=40 GT:DP 1/1:40 +chr20 38576612 . C G 85 PASS DP=23 GT:DP 1/1:23 diff --git a/examples/policy/P004.vcf b/examples/policy/P004.vcf new file mode 100644 index 0000000..cc609b5 --- /dev/null +++ b/examples/policy/P004.vcf @@ -0,0 +1,38 @@ +##fileformat=VCFv4.3 +##fileDate=20260925 +##source=vcf-rdfizer-policy-demonstrator-fixture +##reference=GRCh38 +##contig= +##contig= +##contig= +##contig= +##INFO= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT P004 +chr1 11913838 . A T 76 PASS DP=43 GT:DP 0/1:43 +chr1 33711661 . G C 64 PASS DP=29 GT:DP 0/1:29 +chr1 45899293 . C T 74 PASS DP=48 GT:DP 0/1:48 +chr1 138834739 . T G 61 PASS DP=45 GT:DP 0/1:45 +chr1 139602254 . G T 89 PASS DP=31 GT:DP 1/1:31 +chr1 231965065 . C T 98 PASS DP=25 GT:DP 0/1:25 +chr1 232595915 . T C 76 PASS DP=28 GT:DP 1/1:28 +chr17 43055276 . G A 81 PASS DP=31 GT:DP 0/1:31 +chr17 43059666 . G A 82 PASS DP=49 GT:DP 0/1:49 +chr17 43075071 . C G 52 PASS DP=50 GT:DP 0/1:50 +chr17 43087473 . A C 65 PASS DP=37 GT:DP 0/1:37 +chr17 43095759 . C T 65 PASS DP=24 GT:DP 0/1:24 +chr17 43108081 . T C 99 PASS DP=31 GT:DP 0/1:31 +chr17 43118500 . C T 83 PASS DP=27 GT:DP 0/1:27 +chr17 43118559 . T C 73 PASS DP=23 GT:DP 1/1:23 +chr17 43124696 . T A 51 PASS DP=52 GT:DP 0/1:52 +chr19 44907272 . T G 96 PASS DP=50 GT:DP 0/1:50 +chr19 44907675 . A C 68 PASS DP=30 GT:DP 1/1:30 +chr19 44908684 rs429358 T C 80 PASS DP=58 GT:DP 0/1:58 +chr20 12657015 . G A 79 PASS DP=28 GT:DP 1/1:28 +chr20 13974064 . C G 98 PASS DP=27 GT:DP 0/1:27 +chr20 20513631 . T A 85 PASS DP=27 GT:DP 0/1:27 +chr20 30588233 . T G 85 PASS DP=26 GT:DP 0/1:26 +chr20 36112198 . C A 54 PASS DP=35 GT:DP 0/1:35 +chr20 47351745 . C A 55 PASS DP=31 GT:DP 0/1:31 +chr20 50908083 . G C 95 PASS DP=58 GT:DP 0/1:58 diff --git a/examples/policy/P005.vcf b/examples/policy/P005.vcf new file mode 100644 index 0000000..079e72e --- /dev/null +++ b/examples/policy/P005.vcf @@ -0,0 +1,41 @@ +##fileformat=VCFv4.3 +##fileDate=20260925 +##source=vcf-rdfizer-policy-demonstrator-fixture +##reference=GRCh38 +##contig= +##contig= +##contig= +##contig= +##INFO= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT P005 +chr1 11913838 . A T 71 PASS DP=25 GT:DP 0/1:25 +chr1 33711661 . G C 72 PASS DP=56 GT:DP 0/1:56 +chr1 45899293 . C T 87 PASS DP=47 GT:DP 0/1:47 +chr1 138834739 . T G 51 PASS DP=47 GT:DP 1/1:47 +chr1 139602254 . G T 87 PASS DP=26 GT:DP 1/1:26 +chr1 152491455 . A G 66 PASS DP=40 GT:DP 0/1:40 +chr1 231965065 . C T 50 PASS DP=25 GT:DP 0/1:25 +chr1 232595915 . T C 58 PASS DP=50 GT:DP 1/1:50 +chr17 43055276 . G A 75 PASS DP=54 GT:DP 0/1:54 +chr17 43059666 . G A 98 PASS DP=55 GT:DP 0/1:55 +chr17 43087473 . A C 94 PASS DP=23 GT:DP 1/1:23 +chr17 43095759 . C T 91 PASS DP=27 GT:DP 1/1:27 +chr17 43100876 . G T 51 PASS DP=35 GT:DP 0/1:35 +chr17 43111166 . C A 68 PASS DP=31 GT:DP 0/1:31 +chr17 43118500 . C T 56 PASS DP=47 GT:DP 0/1:47 +chr17 43118559 . T C 68 PASS DP=22 GT:DP 0/1:22 +chr17 43124696 . T A 54 PASS DP=58 GT:DP 1/1:58 +chr19 44907675 . A C 82 PASS DP=57 GT:DP 0/1:57 +chr19 44908684 rs429358 T C 67 PASS DP=43 GT:DP 0/1:43 +chr19 44908822 rs7412 C T 84 PASS DP=34 GT:DP 1/1:34 +chr20 12657015 . G A 95 PASS DP=60 GT:DP 0/1:60 +chr20 13974064 . C G 56 PASS DP=43 GT:DP 0/1:43 +chr20 20513631 . T A 81 PASS DP=21 GT:DP 0/1:21 +chr20 22188117 . A C 80 PASS DP=44 GT:DP 0/1:44 +chr20 36112198 . C A 74 PASS DP=28 GT:DP 0/1:28 +chr20 38576612 . C G 92 PASS DP=55 GT:DP 0/1:55 +chr20 46050645 . A G 59 PASS DP=53 GT:DP 1/1:53 +chr20 47351745 . C A 82 PASS DP=28 GT:DP 0/1:28 +chr20 50908083 . G C 99 PASS DP=20 GT:DP 1/1:20 diff --git a/examples/policy/README.md b/examples/policy/README.md new file mode 100644 index 0000000..1ef7b65 --- /dev/null +++ b/examples/policy/README.md @@ -0,0 +1,106 @@ +# Policy attachment example (v0.1.0) + +One synthetic cohort, one set of ODRL policies, three requesters, and three +different release views, each checked against the source VCFs. How the engine +works (select → partition → decide, each configured in Turtle) is in +[`docs/policy-demonstrator.md`](../../docs/policy-demonstrator.md). + +This is **governed release, not anonymization.** Views keep the original IRIs, +and a released genotype still identifies the person it came from. + +## Run it + +```bash +examples/policy/run_demo.sh /tmp/policy-demo # needs rdflib; no Docker +PROFILE=condensed examples/policy/run_demo.sh /tmp/policy-demo-condensed +``` + +It explains the policy, attaches it to the graph, and then for each requester +evaluates a view and checks it. Finally it prints the decision grid: + +```text +requester P001.vcf P002.vcf P003.vcf P004.vcf P005.vcf triples withheld +alz 22 16 18 withheld 20 4649 +clinical 33 23 withheld withheld withheld 6539 +gru 21 16 withheld withheld withheld 7869 +``` + +Each number is the records released from that file. Where P001 and P002 differ +between requesters, that's the cohort rules at work. *BRCA1* records are +released only for clinical care, and the APOE ε4 variant (rs429358) only for +disease-specific research. P004 withdrew. + +## What is here + +| File | What it is | +| --- | --- | +| `make_fixture.py` | Deterministic generator for the five VCFs and `fixture.json` | +| `P001.vcf` … `P005.vcf` | One synthetic participant each, GRCh38, 23–34 records | +| `fixture.json` | Seed, loci, each participant's consent, and the three requesters | +| `policy.ttl` | Five consents (one per file) and the cohort policy (*BRCA1*, ε4) | +| `custom-selector.ttl` | A selector type declared in the policy itself (below) | +| `converted/` | The VCFs converted once per sample profile, so nothing needs Docker; see `PROVENANCE.json` | +| `run_demo.sh` | The walkthrough above | + +The positions are real GRCh38 coordinates in real loci, and rs429358 and +rs7412 are the real variants. Every other allele, and every genotype, is +synthetic. To regenerate the fixture, run +`python3 examples/policy/make_fixture.py`. + +## The pieces, one at a time + +```bash +vcf-rdfizer-policy explain --policy policy.ttl +vcf-rdfizer-policy attach --rdf converted/expanded/P00*.nt.gz --policy policy.ttl -o annotated.nt +vcf-rdfizer-policy evaluate --rdf converted/expanded/P00*.nt.gz --policy policy.ttl \ + --assignee https://example.org/party/alz-consortium --purpose DUO:0000007 -o views/alz +vcf-rdfizer-policy check --view views/alz --rdf converted/expanded/P00*.nt.gz \ + --policy policy.ttl --vcf P00*.vcf +``` + +`evaluate` writes the following into its output directory: +- `view.nt`, the released triples; +- `decisions.csv`, one row per record with the rule that decided it; +- `summary.json`, the counts; +- `manifest.ttl`, the policy's digest, the request, what was withheld and the + obligations accepted. + +`check` runs the structural checks against the source graph. With `--vcf`, it +also re-derives the expected records straight from the VCF text and fails on +any difference. + +## Write your own selector + +Selector types are SPARQL, declared in Turtle, and there is nothing to code. +`custom-selector.ttl` declares one inside the policy that uses it: + +```turtle +ex:QualityBelow a vcfp:SelectorType ; + vcfp:parameter ex:threshold ; # bound to ?threshold + vcfp:query """ + PREFIX vcfc: + SELECT ?resource WHERE { + ?resource a vcfc:VCFRecord ; vcfc:hasCall ?call . + ?call vcfc:qual ?qual . + FILTER(?qual < ?threshold) }""" . + +ex:low-quality a odrl:Asset , vcfp:GraphSelection ; + vcfp:selector [ a ex:QualityBelow ; ex:threshold 60 ] . +``` + +With every file released for general research except that selection, 38 of +the 139 records are withheld: + +```bash +vcf-rdfizer-policy evaluate --rdf converted/expanded/P00*.nt.gz --policy custom-selector.ttl \ + --assignee https://example.org/party/anyone --purpose DUO:0000042 -o views/quality +``` + +The query must project `?resource`. Each `vcfp:parameter` names a property of +the selector node, bound to the variable of the same local name. A selector +type can also carry a `vcfp:violations` query, and any row it returns stops +evaluation; the shipped region and variant selectors use one to check the +assembly. + +To share a selector across policies, put it in a file and pass +`--profile vcf-core --profile my-selectors.ttl`. diff --git a/examples/policy/converted/PROVENANCE.json b/examples/policy/converted/PROVENANCE.json new file mode 100644 index 0000000..521badd --- /dev/null +++ b/examples/policy/converted/PROVENANCE.json @@ -0,0 +1,10 @@ +{ + "description": "The five fixture VCFs converted once per sample profile, committed so the demonstrator and its tests run without Docker.", + "host": "vcf-bench-1", + "converted": "2026-09-25", + "image": "ecrum19/vcf-rdfizer@sha256:1904e96dde12ab2e2e70d8ee1267765c293ab100a8bd1b14d5b009b2bf8e34aa", + "image_tag": "ecrum19/vcf-rdfizer:3.1.0", + "wrapper_commit": "3da1a5a4a85204c74ed986a953f9098286d1bc80", + "command": "vcf_rdfizer.py --mode full --input --sample-representation {expanded|condensed} --representations none --rdf-storage-mode plain --rdf-compression gzip --image ecrum19/vcf-rdfizer --image-version 3.1.0 --no-build", + "outputs": {"expanded": "expanded/P00n.nt.gz", "condensed": "condensed/P00n.nt.gz"} +} diff --git a/examples/policy/converted/condensed/P001.nt.gz b/examples/policy/converted/condensed/P001.nt.gz new file mode 100644 index 0000000..49e8890 Binary files /dev/null and b/examples/policy/converted/condensed/P001.nt.gz differ diff --git a/examples/policy/converted/condensed/P002.nt.gz b/examples/policy/converted/condensed/P002.nt.gz new file mode 100644 index 0000000..7ec8efb Binary files /dev/null and b/examples/policy/converted/condensed/P002.nt.gz differ diff --git a/examples/policy/converted/condensed/P003.nt.gz b/examples/policy/converted/condensed/P003.nt.gz new file mode 100644 index 0000000..215a812 Binary files /dev/null and b/examples/policy/converted/condensed/P003.nt.gz differ diff --git a/examples/policy/converted/condensed/P004.nt.gz b/examples/policy/converted/condensed/P004.nt.gz new file mode 100644 index 0000000..776cec5 Binary files /dev/null and b/examples/policy/converted/condensed/P004.nt.gz differ diff --git a/examples/policy/converted/condensed/P005.nt.gz b/examples/policy/converted/condensed/P005.nt.gz new file mode 100644 index 0000000..c4004db Binary files /dev/null and b/examples/policy/converted/condensed/P005.nt.gz differ diff --git a/examples/policy/converted/expanded/P001.nt.gz b/examples/policy/converted/expanded/P001.nt.gz new file mode 100644 index 0000000..a42f6da Binary files /dev/null and b/examples/policy/converted/expanded/P001.nt.gz differ diff --git a/examples/policy/converted/expanded/P002.nt.gz b/examples/policy/converted/expanded/P002.nt.gz new file mode 100644 index 0000000..c4587f4 Binary files /dev/null and b/examples/policy/converted/expanded/P002.nt.gz differ diff --git a/examples/policy/converted/expanded/P003.nt.gz b/examples/policy/converted/expanded/P003.nt.gz new file mode 100644 index 0000000..b796887 Binary files /dev/null and b/examples/policy/converted/expanded/P003.nt.gz differ diff --git a/examples/policy/converted/expanded/P004.nt.gz b/examples/policy/converted/expanded/P004.nt.gz new file mode 100644 index 0000000..182f471 Binary files /dev/null and b/examples/policy/converted/expanded/P004.nt.gz differ diff --git a/examples/policy/converted/expanded/P005.nt.gz b/examples/policy/converted/expanded/P005.nt.gz new file mode 100644 index 0000000..fb33bfb Binary files /dev/null and b/examples/policy/converted/expanded/P005.nt.gz differ diff --git a/examples/policy/custom-selector.ttl b/examples/policy/custom-selector.ttl new file mode 100644 index 0000000..3282a83 --- /dev/null +++ b/examples/policy/custom-selector.ttl @@ -0,0 +1,53 @@ +# Extending the policy engine without code: a selector type declared in the +# policy file itself. +# +# ex:QualityBelow selects records whose call has QUAL under a threshold. The +# declaration is a SPARQL SELECT projecting ?resource; its one parameter, +# ex:threshold, is bound to ?threshold from the selector node below. The engine +# loads it alongside the bundled VCF Core profile and treats it exactly like +# the shipped region and variant selectors. +# +# vcf-rdfizer-policy evaluate --rdf converted/expanded/P00*.nt.gz \ +# --policy custom-selector.ttl --assignee https://example.org/party/anyone \ +# --purpose DUO:0000042 -o views/quality + +@prefix odrl: . +@prefix vcfp: . +@prefix obo: . +@prefix ex: . + +# --- The new selector type -------------------------------------------------- +ex:QualityBelow a vcfp:SelectorType ; + vcfp:parameter ex:threshold ; + vcfp:query """ + PREFIX vcfc: + SELECT ?resource WHERE { + ?resource a vcfc:VCFRecord ; vcfc:hasCall ?call . + ?call vcfc:qual ?qual . + FILTER(?qual < ?threshold) }""" . + +# --- A policy that uses it ------------------------------------------------------ +ex:low-quality a odrl:Asset , vcfp:GraphSelection ; + vcfp:selector [ a ex:QualityBelow ; ex:threshold 60 ] . + +ex:release a odrl:Set ; + odrl:uid ex:release ; + odrl:conflict odrl:prohibit ; + # Every file may be read for general research ... + odrl:permission [ odrl:target ; odrl:action odrl:read ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + # ... except records below the quality threshold, which no one receives. + odrl:prohibition [ odrl:target ex:low-quality ; odrl:action odrl:read ] . diff --git a/examples/policy/fixture.json b/examples/policy/fixture.json new file mode 100644 index 0000000..cb70b27 --- /dev/null +++ b/examples/policy/fixture.json @@ -0,0 +1,95 @@ +{ + "description": "Synthetic single-sample cohort for the v0.1.0 policy demonstrator. Genotypes are synthetic; see make_fixture.py.", + "seed": 20260925, + "assembly": "GRCh38", + "participants": { + "P001": { + "consent": [ + "DUO_0000042", + "DUO_0000043" + ] + }, + "P002": { + "consent": [ + "DUO_0000042", + "DUO_0000043" + ] + }, + "P003": { + "consent": [ + "DUO_0000006" + ] + }, + "P004": { + "consent": [ + "DUO_0000042" + ], + "withdrawn": true + }, + "P005": { + "consent": [ + "DUO_0000007" + ] + } + }, + "files": { + "P001.vcf": { + "participant": "P001", + "records": 34 + }, + "P002.vcf": { + "participant": "P002", + "records": 23 + }, + "P003.vcf": { + "participant": "P003", + "records": 27 + }, + "P004.vcf": { + "participant": "P004", + "records": 26 + }, + "P005.vcf": { + "participant": "P005", + "records": 29 + } + }, + "loci": { + "brca1": { + "chrom": "chr17", + "start": 43044295, + "end": 43125483 + }, + "apoe_e4": { + "chrom": "chr19", + "pos": 44908684, + "ref": "T", + "alt": "C", + "id": "rs429358" + }, + "apoe_e2": { + "chrom": "chr19", + "pos": 44908822, + "ref": "C", + "alt": "T", + "id": "rs7412" + } + }, + "requesters": { + "gru": { + "assignee": "https://example.org/party/research-consortium", + "purpose": "DUO:0000042", + "label": "General-research consortium" + }, + "alz": { + "assignee": "https://example.org/party/alz-consortium", + "purpose": "DUO:0000007", + "label": "Alzheimer's consortium" + }, + "clinical": { + "assignee": "https://example.org/party/clinical-genetics", + "purpose": "DUO:0000043", + "label": "Clinical genetics lab" + } + } +} diff --git a/examples/policy/make_fixture.py b/examples/policy/make_fixture.py new file mode 100644 index 0000000..8e51a8d --- /dev/null +++ b/examples/policy/make_fixture.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +"""Generate the policy demonstrator's synthetic cohort. + + python3 examples/policy/make_fixture.py [OUT_DIR] # writes P001.vcf ... P005.vcf, fixture.json + +Five participants, one single-sample VCF each. Positions are GRCh38 and fall in +real loci -- BRCA1, APOE, and background sites on chr1 and chr20 -- so the region +and variant selectors are exercised against real coordinates. Two variants are +real and named: rs429358 (the APOE e4-defining SNP) and rs7412. Every other +allele, and every genotype, is synthetic and drawn from a fixed seed: the +genotypes identify no one, which is the point of using a fixture for a privacy +demonstration. REF alleles other than the two named variants are not checked +against the reference genome. + +The site catalogue also plants the cases the tests need: + * records one base outside each end of the BRCA1 window, so the region + bounds are tested as inclusive; + * a decoy at rs429358's position with a different ALT, so the variant + selector is tested on alleles and not on position alone. + +The output is deterministic: rerunning it rewrites byte-identical files. +""" + +from __future__ import annotations + +import json +import random +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SEED = 20260925 +ASSEMBLY = "GRCh38" +FILE_DATE = "20260925" + +CONTIGS = {"chr1": 248956422, "chr17": 83257441, "chr19": 58617616, "chr20": 64444167} + +BRCA1 = {"chrom": "chr17", "start": 43044295, "end": 43125483} +APOE_E4 = {"chrom": "chr19", "pos": 44908684, "ref": "T", "alt": "C", "id": "rs429358"} +APOE_E2 = {"chrom": "chr19", "pos": 44908822, "ref": "C", "alt": "T", "id": "rs7412"} + +# Consent per participant, as DUO terms. P004 withdrew. +PARTICIPANTS = { + "P001": {"consent": ["DUO_0000042", "DUO_0000043"]}, + "P002": {"consent": ["DUO_0000042", "DUO_0000043"]}, + "P003": {"consent": ["DUO_0000006"]}, + "P004": {"consent": ["DUO_0000042"], "withdrawn": True}, + "P005": {"consent": ["DUO_0000007"]}, +} + +REQUESTERS = { + "gru": {"assignee": "https://example.org/party/research-consortium", "purpose": "DUO:0000042", + "label": "General-research consortium"}, + "alz": {"assignee": "https://example.org/party/alz-consortium", "purpose": "DUO:0000007", + "label": "Alzheimer's consortium"}, + "clinical": {"assignee": "https://example.org/party/clinical-genetics", "purpose": "DUO:0000043", + "label": "Clinical genetics lab"}, +} + +BASES = "ACGT" + + +def site_catalogue(rng: random.Random) -> list[dict]: + """Every site any participant may carry, before sampling.""" + sites = [] + + def snv(chrom, pos, tag, ident="."): + ref = rng.choice(BASES) + alt = rng.choice([b for b in BASES if b != ref]) + sites.append({"chrom": chrom, "pos": pos, "ref": ref, "alt": alt, "id": ident, "tag": tag}) + + # BRCA1: twelve sites inside the window, and one just outside each end. + for pos in sorted(rng.sample(range(BRCA1["start"] + 1, BRCA1["end"]), 12)): + snv("chr17", pos, "brca1") + snv("chr17", BRCA1["start"] - 1, "brca1-edge-outside") + snv("chr17", BRCA1["end"] + 1, "brca1-edge-outside") + snv("chr17", BRCA1["start"], "brca1-edge-inside") + snv("chr17", BRCA1["end"], "brca1-edge-inside") + + # APOE: the two named variants, a decoy at the e4 position, and neighbours. + sites.append({**APOE_E4, "tag": "apoe-e4"}) + sites.append({**APOE_E2, "tag": "apoe-e2"}) + sites.append({"chrom": "chr19", "pos": APOE_E4["pos"], "ref": "T", "alt": "G", "id": ".", + "tag": "apoe-e4-decoy"}) + for pos in sorted(rng.sample(range(44905791, 44909393), 4)): + if pos not in (APOE_E4["pos"], APOE_E2["pos"]): + snv("chr19", pos, "apoe-region") + + # Background, governed by consent alone. + for chrom, lo, hi in (("chr1", 1_000_000, 240_000_000), ("chr20", 1_000_000, 60_000_000)): + for pos in sorted(rng.sample(range(lo, hi), 10)): + snv(chrom, pos, "background") + return sites + + +def carried(rng: random.Random, participant: str, sites: list[dict]) -> list[dict]: + """The sites one participant carries, with the scenario's cases forced in.""" + forced = { + "apoe-e4": participant in ("P001", "P003", "P004", "P005"), + "apoe-e4-decoy": participant == "P002", + "brca1-edge-outside": participant == "P001", + "brca1-edge-inside": participant == "P001", + } + chosen = [] + for site in sites: + if site["tag"] in forced: + keep = forced[site["tag"]] + elif site["tag"] == "apoe-e2": + keep = participant in ("P002", "P003", "P005") + else: + keep = rng.random() < 0.7 + if keep: + chosen.append(site) + return chosen + + +def write_vcf(path: Path, participant: str, records: list[dict], rng: random.Random) -> None: + order = {c: i for i, c in enumerate(CONTIGS)} + records = sorted(records, key=lambda s: (order[s["chrom"]], s["pos"], s["alt"])) + lines = [ + "##fileformat=VCFv4.3", + f"##fileDate={FILE_DATE}", + "##source=vcf-rdfizer-policy-demonstrator-fixture", + f"##reference={ASSEMBLY}", + ] + lines += [f"##contig=" for c, n in CONTIGS.items()] + lines += [ + '##INFO=', + '##FORMAT=', + '##FORMAT=', + "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t" + participant, + ] + for site in records: + depth = rng.randint(20, 60) + genotype = "1/1" if rng.random() < 0.3 else "0/1" + qual = rng.randint(50, 99) + lines.append("\t".join([site["chrom"], str(site["pos"]), site["id"], site["ref"], site["alt"], + str(qual), "PASS", f"DP={depth}", "GT:DP", f"{genotype}:{depth}"])) + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def main(out_dir: Path = HERE) -> None: + out_dir = Path(out_dir) + rng = random.Random(SEED) + sites = site_catalogue(rng) + files = {} + for participant in PARTICIPANTS: + records = carried(rng, participant, sites) + write_vcf(out_dir / f"{participant}.vcf", participant, records, rng) + files[f"{participant}.vcf"] = {"participant": participant, "records": len(records)} + + fixture = { + "description": "Synthetic single-sample cohort for the v0.1.0 policy demonstrator. " + "Genotypes are synthetic; see make_fixture.py.", + "seed": SEED, + "assembly": ASSEMBLY, + "participants": PARTICIPANTS, + "files": files, + "loci": {"brca1": BRCA1, "apoe_e4": APOE_E4, "apoe_e2": APOE_E2}, + "requesters": REQUESTERS, + } + (out_dir / "fixture.json").write_text(json.dumps(fixture, indent=2) + "\n", encoding="utf-8") + for name, info in files.items(): + print(f"wrote {name}: {info['records']} records") + + +if __name__ == "__main__": + main(Path(sys.argv[1]) if len(sys.argv) > 1 else HERE) diff --git a/examples/policy/policy.ttl b/examples/policy/policy.ttl new file mode 100644 index 0000000..4c73a37 --- /dev/null +++ b/examples/policy/policy.ttl @@ -0,0 +1,73 @@ +# Policies for the demonstrator cohort (docs/policy-demonstrator.md §2, §4). +# +# One consent policy per participant, attached to that participant's file, and +# one cohort policy with two rules that cut across files. Every rule reads the +# graph (odrl:read); deny wins (odrl:conflict odrl:prohibit); anything no +# permission covers is withheld. + +@prefix odrl: . +@prefix vcfp: . +@prefix obo: . +@prefix ex: . + +# --- Cohort-wide rules ------------------------------------------------------ + +ex:brca1 a odrl:Asset , vcfp:GraphSelection ; + vcfp:selector [ a vcfp:RegionSelector ; vcfp:assembly "GRCh38" ; + vcfp:chrom "chr17" ; vcfp:start 43044295 ; vcfp:end 43125483 ] . + +ex:apoe-e4 a odrl:Asset , vcfp:GraphSelection ; + vcfp:selector [ a vcfp:VariantSelector ; vcfp:assembly "GRCh38" ; + vcfp:chrom "chr19" ; vcfp:pos 44908684 ; vcfp:ref "T" ; vcfp:alt "C" ] . + +ex:cohort a odrl:Set ; + odrl:uid ex:cohort ; + odrl:profile ; + odrl:conflict odrl:prohibit ; + # Actionable incidental findings: readable only for clinical care. + odrl:prohibition [ odrl:target ex:brca1 ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isNoneOf ; + odrl:rightOperand obo:DUO_0000043 ] ] ; + # APOE e4: readable only for disease-specific research. + odrl:prohibition [ odrl:target ex:apoe-e4 ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isNoneOf ; + odrl:rightOperand obo:DUO_0000007 ] ] . + +# --- Participant consents --------------------------------------------------- + +ex:consent-P001 a odrl:Set ; odrl:uid ex:consent-P001 ; + odrl:profile ; odrl:conflict odrl:prohibit ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 , obo:DUO_0000043 ] ; + odrl:duty [ odrl:action odrl:attribute ] ] . + +ex:consent-P002 a odrl:Set ; odrl:uid ex:consent-P002 ; + odrl:profile ; odrl:conflict odrl:prohibit ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 , obo:DUO_0000043 ] ; + odrl:duty [ odrl:action odrl:attribute ] ] . + +ex:consent-P003 a odrl:Set ; odrl:uid ex:consent-P003 ; + odrl:profile ; odrl:conflict odrl:prohibit ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000006 ] ; + odrl:duty [ odrl:action odrl:attribute ] ] . + +# P004 consented to general research and later withdrew. The withdrawal is a +# prohibition, so it overrides the permission under deny-wins. +ex:consent-P004 a odrl:Set ; odrl:uid ex:consent-P004 ; + odrl:profile ; odrl:conflict odrl:prohibit ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000042 ] ] ; + odrl:prohibition [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ] . + +ex:consent-P005 a odrl:Set ; odrl:uid ex:consent-P005 ; + odrl:profile ; odrl:conflict odrl:prohibit ; + odrl:permission [ odrl:target ; odrl:action odrl:read ; odrl:assignee odrl:All ; + odrl:constraint [ odrl:leftOperand odrl:purpose ; odrl:operator odrl:isAnyOf ; + odrl:rightOperand obo:DUO_0000007 ] ; + odrl:duty [ odrl:action odrl:attribute ] ] . diff --git a/examples/policy/run_demo.sh b/examples/policy/run_demo.sh new file mode 100755 index 0000000..68eb549 --- /dev/null +++ b/examples/policy/run_demo.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# The v0.1.0 policy demonstrator end to end, without Docker: +# attach the policies, evaluate one release view per requester, check each view +# against the source VCFs, and print the decision grid. +# +# examples/policy/run_demo.sh [OUT_DIR] # default: ./policy-demo-out (must not exist) +# +# PROFILE=condensed runs it on the condensed graphs instead of the expanded ones. +set -euo pipefail + +HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +OUT="${1:-policy-demo-out}" +PROFILE="${PROFILE:-expanded}" +POLICY="$HERE/policy.ttl" +RDF=("$HERE"/converted/"$PROFILE"/P00*.nt.gz) +VCF=("$HERE"/P00*.vcf) +POLICY_CMD=(python3 "$HERE/../../vcf_rdfizer_policy.py") + +[[ -e "$OUT" ]] && { echo "error: $OUT exists; choose a new directory" >&2; exit 2; } +mkdir -p "$OUT" + +echo "== the policy" +"${POLICY_CMD[@]}" explain --policy "$POLICY" + +echo; echo "== attach: policies in the graph" +"${POLICY_CMD[@]}" attach --rdf "${RDF[@]}" --policy "$POLICY" -o "$OUT/cohort-annotated.nt" + +# One view per requester in fixture.json: key, assignee IRI, DUO purpose. +while read -r key assignee purpose; do + echo; echo "== $key ($purpose)" + "${POLICY_CMD[@]}" evaluate --rdf "${RDF[@]}" --policy "$POLICY" \ + --assignee "$assignee" --purpose "$purpose" -o "$OUT/views/$key" + "${POLICY_CMD[@]}" check --view "$OUT/views/$key" --policy "$POLICY" --rdf "${RDF[@]}" --vcf "${VCF[@]}" +done < <(python3 -c 'import json,sys +for k, r in json.load(open(sys.argv[1]))["requesters"].items(): print(k, r["assignee"], r["purpose"])' "$HERE/fixture.json") + +echo; echo "== decision grid (records released per file)" +python3 - "$OUT/views" <<'PY' +import json, sys +from pathlib import Path +views = sorted(Path(sys.argv[1]).iterdir()) +summaries = {v.name: json.loads((v / "summary.json").read_text()) for v in views} +files = sorted(next(iter(summaries.values()))["groups"]) +print("requester".ljust(10) + "".join(f.split("//")[1].ljust(10) for f in files) + "triples withheld") +for name, s in summaries.items(): + cells = "".join((str(s["groups"][f]["records_released"]) if s["groups"][f]["released"] else "withheld").ljust(10) + for f in files) + print(name.ljust(10) + cells + str(s["triples_withheld"])) +PY diff --git a/pyproject.toml b/pyproject.toml index 0db051c..fffcc40 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -37,6 +37,7 @@ Issues = "https://github.com/ecrum19/VCF-RDFizer/issues" vcf-rdfizer = "vcf_rdfizer:main" vcf-rdfizer-rules = "vcf_rdfizer_rules:main" vcf-rdfizer-link = "vcf_rdfizer_link:main" +vcf-rdfizer-policy = "vcf_rdfizer_policy:main" [project.optional-dependencies] dev = [ @@ -45,13 +46,14 @@ dev = [ ] [tool.setuptools] -py-modules = ["vcf_rdfizer", "vcf_rdfizer_gzip", "vcf_rdfizer_rules", "vcf_rdfizer_link", "vcf_rdfizer_vocab", "vcf_rdfizer_cottas"] -packages = ["vcf_rdfizer_data", "vcf_rdfizer_data.rules", "vcf_rdfizer_data.linkers", "vcf_rdfizer_linking"] +py-modules = ["vcf_rdfizer", "vcf_rdfizer_gzip", "vcf_rdfizer_rules", "vcf_rdfizer_link", "vcf_rdfizer_policy", "vcf_rdfizer_vocab", "vcf_rdfizer_cottas"] +packages = ["vcf_rdfizer_data", "vcf_rdfizer_data.rules", "vcf_rdfizer_data.linkers", "vcf_rdfizer_data.policy", "vcf_rdfizer_linking", "vcf_rdfizer_policies"] include-package-data = true [tool.setuptools.package-data] "vcf_rdfizer_data.rules" = ["default_rules.ttl"] "vcf_rdfizer_data.linkers" = ["*/linker.ttl", "*/resolver.py", "*/genes.gff3", "*/README.md"] +"vcf_rdfizer_data.policy" = ["*.ttl"] # Vendored from the published vocabulary so SHACL validation works from an # installed package with no separate checkout. Digests are pinned in # vcf_rdfizer_data/VOCABULARY_PROVENANCE.json. diff --git a/vcf_rdfizer_data/policy/__init__.py b/vcf_rdfizer_data/policy/__init__.py new file mode 100644 index 0000000..faba12d --- /dev/null +++ b/vcf_rdfizer_data/policy/__init__.py @@ -0,0 +1 @@ +"""Bundled data for the policy demonstrator: the vcfp profile terms and a DUO subset.""" diff --git a/vcf_rdfizer_data/policy/duo-subset.ttl b/vcf_rdfizer_data/policy/duo-subset.ttl new file mode 100644 index 0000000..24051c9 --- /dev/null +++ b/vcf_rdfizer_data/policy/duo-subset.ttl @@ -0,0 +1,28 @@ +# The four GA4GH Data Use Ontology terms the v0.1.0 policy demonstrator uses, +# with their subclass links exactly as DUO states them. +# +# Source: http://purl.obolibrary.org/obo/duo/releases/2021-02-23/duo.owl +# (EBISPOT/DUO master, read 2026-09-25). Only rdfs:subClassOf and labels are +# copied. A purpose satisfies a consented term when it is that term or a +# subclass of it, so disease-specific research falls within a +# general-research consent and not the reverse. +# +# Clinical care use is, in DUO, a data use *modifier* rather than a permission. +# v0.1.0 treats it as a purpose like the others; it matches only itself. + +@prefix obo: . +# So DUO:0000007 resolves as a prefixed name, with no special-case code. +@prefix DUO: . +@prefix rdfs: . +@prefix owl: . + + a owl:Ontology . + +obo:DUO_0000042 rdfs:label "general research use" ; + rdfs:subClassOf obo:DUO_0000001 . +obo:DUO_0000006 rdfs:label "health or medical or biomedical research" ; + rdfs:subClassOf obo:DUO_0000042 . +obo:DUO_0000007 rdfs:label "disease specific research" ; + rdfs:subClassOf obo:DUO_0000006 . +obo:DUO_0000043 rdfs:label "clinical care use" ; + rdfs:subClassOf obo:DUO_0000017 . diff --git a/vcf_rdfizer_data/policy/vcf-core-profile.ttl b/vcf_rdfizer_data/policy/vcf-core-profile.ttl new file mode 100644 index 0000000..096bac9 --- /dev/null +++ b/vcf_rdfizer_data/policy/vcf-core-profile.ttl @@ -0,0 +1,63 @@ +# The VCF Core profile for vcf-rdfizer-policy: everything the engine needs to know +# about graphs written by VCF-RDFizer, declared here rather than coded. +# +# * vcfp:SelectorType -- a kind of graph selection. Its vcfp:query is a SPARQL +# SELECT that must project ?resource. Each vcfp:parameter is a property the +# policy's selector node must carry; its value is bound to the query variable +# named after the property's local name (vcfp:start -> ?start). An optional +# vcfp:violations query lists reasons the selector cannot be applied to this +# graph; any row stops evaluation. +# * vcfp:Profile -- how the graph is partitioned. A withheld resource takes with +# it everything it owns: the resources reached by vcfp:ownershipPath (a SPARQL +# property path), and, when vcfp:iriSubtree is true, every IRI beneath any of +# them (…#record/9 owns …#record/9/allele/0). vcfp:unitQuery names what the +# per-record report counts: ?resource and ?group are required, other +# variables become report columns. +# +# New selectors need no code: declare another vcfp:SelectorType here, in a +# file passed with --profile, or in the policy file itself. + +@prefix vcfp: . +@prefix rdfs: . + +vcfp:VCFCore a vcfp:Profile ; + rdfs:label "VCF Core graphs as VCF-RDFizer writes them" ; + # A record owns its call, and through the call its per-sample calls + # (expanded profile); the condensed matrix already sits under the call's IRI. + vcfp:ownershipPath "/?" ; + vcfp:iriSubtree true ; + vcfp:unitQuery """ + PREFIX vcfc: + SELECT ?resource ?group ?chrom ?pos ?ref (GROUP_CONCAT(?a; separator=",") AS ?alt) + WHERE { ?group vcfc:hasRecord ?resource . + ?resource vcfc:chrom ?chrom ; vcfc:pos ?pos ; vcfc:ref ?ref . + OPTIONAL { ?resource vcfc:alt ?a } } + GROUP BY ?resource ?group ?chrom ?pos ?ref""" . + +vcfp:RegionSelector a vcfp:SelectorType ; + rdfs:comment "Records whose POS lies in [start, end] on chrom, 1-based and inclusive." ; + vcfp:parameter vcfp:assembly , vcfp:chrom , vcfp:start , vcfp:end ; + vcfp:query """ + PREFIX vcfc: + SELECT ?resource WHERE { + ?resource a vcfc:VCFRecord ; vcfc:chrom ?c ; vcfc:pos ?pos . + FILTER(STR(?c) = STR(?chrom) && ?pos >= ?start && ?pos <= ?end) }""" ; + vcfp:violations """ + PREFIX vcfc: + SELECT ?file ?declared WHERE { + ?file a vcfc:VCFFile . OPTIONAL { ?file vcfc:referenceGenome ?declared } + FILTER(!BOUND(?declared) || STR(?declared) != STR(?assembly)) }""" . + +vcfp:VariantSelector a vcfp:SelectorType ; + rdfs:comment "Records with exactly this chrom, pos, ref and one ALT equal to alt." ; + vcfp:parameter vcfp:assembly , vcfp:chrom , vcfp:pos , vcfp:ref , vcfp:alt ; + vcfp:query """ + PREFIX vcfc: + SELECT ?resource WHERE { + ?resource a vcfc:VCFRecord ; vcfc:chrom ?c ; vcfc:pos ?p ; vcfc:ref ?r ; vcfc:alt ?a . + FILTER(STR(?c) = STR(?chrom) && ?p = ?pos && STR(?r) = STR(?ref) && STR(?a) = STR(?alt)) }""" ; + vcfp:violations """ + PREFIX vcfc: + SELECT ?file ?declared WHERE { + ?file a vcfc:VCFFile . OPTIONAL { ?file vcfc:referenceGenome ?declared } + FILTER(!BOUND(?declared) || STR(?declared) != STR(?assembly)) }""" . diff --git a/vcf_rdfizer_data/policy/vcfp-0.1.ttl b/vcf_rdfizer_data/policy/vcfp-0.1.ttl new file mode 100644 index 0000000..4969f8f --- /dev/null +++ b/vcf_rdfizer_data/policy/vcfp-0.1.ttl @@ -0,0 +1,41 @@ +# Terms of the VCF-RDFizer policy profile that v0.1.0 defines. The engine reads +# ODRL policies; these terms describe the three things it is configured with: +# selections (what a rule targets), profiles (how a graph is partitioned), and +# release views (what it writes). docs/policy-demonstrator.md explains each. + +@prefix vcfp: . +@prefix odrl: . +@prefix rdf: . +@prefix rdfs: . + + a odrl:Profile ; + rdfs:label "VCF-RDFizer policy profile, v0.1.0" . + +# --- Selections -------------------------------------------------------------- +vcfp:GraphSelection a rdfs:Class ; rdfs:subClassOf odrl:Asset ; + rdfs:comment "An asset that is a selection of a graph, described by exactly one vcfp:selector." . +vcfp:selector a rdf:Property ; rdfs:domain vcfp:GraphSelection ; + rdfs:comment "The selector node: typed with a vcfp:SelectorType, carrying its parameters." . +vcfp:selects a rdf:Property ; rdfs:domain vcfp:GraphSelection ; + rdfs:comment "Written by `attach`: a resource the selection selects." . +vcfp:SelectorType a rdfs:Class ; + rdfs:comment "A kind of selection, declared by a SPARQL SELECT that projects ?resource." . +vcfp:query a rdf:Property ; rdfs:domain vcfp:SelectorType . +vcfp:parameter a rdf:Property ; rdfs:domain vcfp:SelectorType ; + rdfs:comment "A required property of the selector node, bound to the query variable named after its local name." . +vcfp:violations a rdf:Property ; rdfs:domain vcfp:SelectorType ; + rdfs:comment "A SPARQL SELECT whose rows are reasons the selector cannot be applied; any row stops evaluation." . + +# --- Partitioning -------------------------------------------------------------- +vcfp:Profile a rdfs:Class ; + rdfs:comment "How a graph is partitioned, and what the per-unit report counts." . +vcfp:ownershipPath a rdf:Property ; rdfs:domain vcfp:Profile ; + rdfs:comment "A SPARQL property path: a withheld resource also withholds what it reaches." . +vcfp:iriSubtree a rdf:Property ; rdfs:domain vcfp:Profile ; + rdfs:comment "When true, a withheld resource also withholds every IRI beneath it (after '#' or '/')." . +vcfp:unitQuery a rdf:Property ; rdfs:domain vcfp:Profile ; + rdfs:comment "A SPARQL SELECT of the reported units: ?resource and ?group required." . + +# --- Release views --------------------------------------------------------------- +vcfp:ReleaseView a rdfs:Class ; + rdfs:comment "A graph produced under a policy for one request. Governed release, not anonymization." . diff --git a/vcf_rdfizer_policies/__init__.py b/vcf_rdfizer_policies/__init__.py new file mode 100644 index 0000000..855dc0f --- /dev/null +++ b/vcf_rdfizer_policies/__init__.py @@ -0,0 +1,33 @@ +"""Policy demonstrator v0.1.0: attach ODRL policies to RDF graphs and release governed views. + +Three generic steps, each configured in Turtle rather than code: + + select a rule's target is an IRI, or a selection computed by a declared + SPARQL selector type (profile.py) + partition the profile's ownership rule extends each selection to what it owns + decide ODRL: a binding permission must own a resource, and no binding + prohibition may; deny wins (engine.py) + +The bundled VCF Core profile makes it work on VCF-RDFizer graphs; vcf_oracle.py +adds an independent check against the source VCFs. This is governed release, not +anonymization. The specification is docs/policy-demonstrator.md. +""" + +VERSION = "0.1.0" + +ODRL = "http://www.w3.org/ns/odrl/2/" +VCFP = "https://w3id.org/vcf-rdfizer/policy#" +VCFC = "https://w3id.org/vcf-core/vocab#" +OBO = "http://purl.obolibrary.org/obo/" + +#: Stated in every release manifest, so no artifact can be mistaken for more. +DISCLOSURE_MODEL = "governed release; not anonymization" + + +class PolicyError(ValueError): + """A policy, graph or request that v0.1.0 cannot evaluate. + + Always fatal. A rule that is parsed and then not applied would leave the + operator believing a release is governed when it is not, so the tool stops + instead of guessing or skipping. + """ diff --git a/vcf_rdfizer_policies/check.py b/vcf_rdfizer_policies/check.py new file mode 100644 index 0000000..8aaa750 --- /dev/null +++ b/vcf_rdfizer_policies/check.py @@ -0,0 +1,66 @@ +"""Verify a release view: structural checks for any graph, plus an optional oracle. + +Given the view, the policy, and the source graph the view was made from: + +1. the view was produced under this policy (its manifest's digest matches); +2. nothing a binding prohibition owns appears in the view, as subject or object; +3. every subject in the view is owned by a binding permission (default-deny); +4. no triple points at a node of the source that the view does not contain. + +Checks 2-4 reuse the engine's selectors and partition, so they confirm the view +honours the policy; they cannot catch a mistake in a selector itself. That is +what an oracle is for: `vcf_oracle.compare` re-derives the expected records +from the VCF text, an input the graph and its conversion never touched. +Each failure is returned as one human-readable line. +""" + +from pathlib import Path + +from . import ODRL, VCFP +from .engine import Partition, Request, applies, select +from .policy import policy_digest + +#: Enough to diagnose; a broken view can otherwise fail thousands of times. +LIMIT = 20 + + +def read_view(view_dir: Path): + """(view graph, manifest graph, request) from a directory written by evaluate.""" + import rdflib + + view_dir = Path(view_dir) + manifest = rdflib.Graph().parse(str(view_dir / "manifest.ttl"), format="turtle") + request_node = next(manifest.objects(None, rdflib.URIRef(VCFP + "request"))) + request = Request(str(manifest.value(request_node, rdflib.URIRef(ODRL + "assignee"))), + str(manifest.value(request_node, rdflib.URIRef(ODRL + "purpose")))) + return rdflib.Graph().parse(str(view_dir / "view.nt"), format="nt"), manifest, request + + +def check_view(view, manifest, request, *, policy_path, rules, profile, vocabulary, source) -> list: + """Every way `view` departs from the policy, given the `source` it was made from.""" + import rdflib + + failures = [] + recorded = str(next(manifest.objects(None, rdflib.URIRef(VCFP + "policyDigest")), None)) + if recorded != policy_digest(policy_path): + failures.append(f"view was produced under a different policy ({recorded})") + + partition = Partition(source, profile) + binding = [(rule, partition.owned(select(source, rule.target))) + for rule in rules if applies(rule, request, vocabulary)] + terms = {t for triple in view for t in (triple[0], triple[2]) + if isinstance(t, (rdflib.URIRef, rdflib.BNode))} + for rule, owned in binding: + if rule.kind == "prohibition": + present = sorted(str(t) for t in terms if partition.contains(owned, t)) + failures += [f"prohibited content present: {rule.label} owns <{t}>" for t in present[:LIMIT]] + + granted = [owned for rule, owned in binding if rule.kind == "permission"] + ungoverned = sorted(str(s) for s in set(view.subjects()) + if not any(partition.contains(owned, s) for owned in granted)) + failures += [f"no permission covers <{s}>" for s in ungoverned[:LIMIT]] + + nodes, present = set(source.subjects()), set(view.subjects()) + dangling = sorted((str(s), str(o)) for s, _, o in view if o in nodes and o not in present) + failures += [f"dangling reference: <{s}> -> <{o}>" for s, o in dangling[:LIMIT]] + return failures diff --git a/vcf_rdfizer_policies/engine.py b/vcf_rdfizer_policies/engine.py new file mode 100644 index 0000000..1a73721 --- /dev/null +++ b/vcf_rdfizer_policies/engine.py @@ -0,0 +1,136 @@ +"""The engine: select, partition, decide (docs/policy-demonstrator.md §4). + +Nothing here knows about VCF. A rule's target selects resources (a declared +SPARQL selector, or one IRI); the profile's ownership rule extends each +selection to everything it owns; and a resource is released when some binding +permission owns it and no binding prohibition does. +""" + +from dataclasses import dataclass + +from . import PolicyError +from .graphs import ancestors +from .policy import Direct + + +@dataclass(frozen=True) +class Request: + assignee: str + purpose: str # a term IRI of the purpose vocabulary + + +def applies(rule, request, vocabulary) -> bool: + """Does the rule bind this request: assignee matches, and every constraint holds?""" + if rule.assignee is not None and rule.assignee != request.assignee: + return False + for constraint in rule.constraints: + inside = any(vocabulary.within(request.purpose, term) for term in constraint.purposes) + if inside != (constraint.operator == "isAnyOf"): + return False + return True + + +def select(graph, target) -> set: + """The resources a target selects in `graph`.""" + import rdflib + + if isinstance(target, Direct): + return {rdflib.URIRef(target.iri)} + rows = graph.query(target.selector.query, initBindings=dict(target.bindings)) + return {row.resource for row in rows} + + +def check_preconditions(graph, rules) -> None: + """Run each selection's declared violations query; any row stops evaluation.""" + for rule in rules: + selector = getattr(rule.target, "selector", None) + if selector is None or selector.violations is None: + continue + rows = list(graph.query(selector.violations, initBindings=dict(rule.target.bindings))) + if rows: + shown = "; ".join(" ".join(str(v) for v in row if v is not None) for row in rows[:3]) + raise PolicyError(f"{rule.label} cannot be applied to this graph: {shown}") + + +class Partition: + """The profile's ownership rule, applied to one graph.""" + + def __init__(self, graph, profile): + self.graph, self.profile = graph, profile + + def owned(self, roots) -> frozenset: + """The roots and everything their ownership path reaches (IRI subtrees are implicit).""" + found = set(roots) + if self.profile.ownership_path: + query = (f"SELECT DISTINCT ?owned WHERE {{ ?root {self.profile.ownership_path} ?owned }}") + for root in roots: + found.update(row.owned for row in self.graph.query(query, initBindings={"root": root})) + return frozenset(found) + + def contains(self, owned, term) -> bool: + """Is `term` in `owned`, or (with iriSubtree) beneath an IRI that is?""" + if term in owned: + return True + if not self.profile.iri_subtree or not hasattr(term, "startswith"): + return False + from rdflib import URIRef + + return any(URIRef(a) in owned for a in ancestors(str(term))) + + +@dataclass +class Evaluation: + """Per-rule ownership, and the release decision for any term.""" + partition: Partition + binding: list # (rule, owned) for every rule that binds the request + + def decide(self, term): + """(released, reason) for one resource; deny wins, and default-deny.""" + for rule, owned in self.binding: + if rule.kind == "prohibition" and self.partition.contains(owned, term): + return False, f"withheld: {rule.label}" + for rule, owned in self.binding: + if rule.kind == "permission" and self.partition.contains(owned, term): + return True, f"released: {rule.label}" + return False, "withheld: no permission covers it for this purpose" + + +def evaluation(graph, rules, request, profile, vocabulary) -> Evaluation: + check_preconditions(graph, rules) + partition = Partition(graph, profile) + binding = [(rule, partition.owned(select(graph, rule.target))) + for rule in rules if applies(rule, request, vocabulary)] + return Evaluation(partition, binding) + + +def view(graph, evaluation) -> tuple: + """(released triples, number withheld). A triple is released when its subject is, + and when its object, if it is a node of the graph, is released too -- so a view + never points at something it does not contain.""" + import rdflib + + nodes = set(graph.subjects()) + cache = {} + + def released(term): + if term not in cache: + cache[term] = evaluation.decide(term)[0] + return cache[term] + + kept, withheld = [], 0 + for s, p, o in graph: + if released(s) and (not isinstance(o, (rdflib.URIRef, rdflib.BNode)) or o not in nodes or released(o)): + kept.append((s, p, o)) + else: + withheld += 1 + return kept, withheld + + +def units(graph, profile) -> list: + """The profile's reporting units: dicts with 'resource', 'group' and any other columns.""" + if profile.unit_query is None: + return [] + rows = graph.query(profile.unit_query) + names = [str(v) for v in rows.vars] + found = [{n: row[n] for n in names} for row in rows] + return sorted(found, key=lambda u: (str(u["group"]), str(u["resource"]))) diff --git a/vcf_rdfizer_policies/graphs.py b/vcf_rdfizer_policies/graphs.py new file mode 100644 index 0000000..fdc79b5 --- /dev/null +++ b/vcf_rdfizer_policies/graphs.py @@ -0,0 +1,46 @@ +"""Load RDF inputs, and walk an IRI's hierarchy.""" + +import gzip +from pathlib import Path + +from . import PolicyError + +#: The engine evaluates in memory. Above this, the demonstrator is the wrong tool. +MAX_TRIPLES = 5_000_000 + + +def _open(path: Path, mode: str): + return gzip.open(path, mode) if path.name.endswith(".gz") else open(path, mode) + + +def load(paths) -> "rdflib.Graph": + """Parse .nt / .nt.gz files into one graph, after a size check.""" + import rdflib + + paths = [Path(p) for p in paths] + total = 0 + for path in paths: + with _open(path, "rb") as handle: + total += sum(1 for _ in handle) + if total > MAX_TRIPLES: + raise PolicyError(f"{total:,} triples exceeds the in-memory limit of {MAX_TRIPLES:,}; " + "the demonstrator is for fixtures, not cohorts") + graph = rdflib.Graph() + for path in paths: + with _open(path, "rb") as handle: + graph.parse(handle, format="nt") + return graph + + +def ancestors(iri: str): + """The IRI itself, then each prefix of it that ends just before a '#' or '/'. + + file://P1.vcf#record/9/allele/0 -> itself, file://P1.vcf#record/9/allele, + file://P1.vcf#record/9, file://P1.vcf#record, file://P1.vcf. The scheme's own + slashes are skipped, so nothing shorter than the authority is yielded. + """ + yield iri + start = iri.find("://") + 3 if "://" in iri else 0 + for index in range(len(iri) - 1, start, -1): + if iri[index] in "#/": + yield iri[:index] diff --git a/vcf_rdfizer_policies/policy.py b/vcf_rdfizer_policies/policy.py new file mode 100644 index 0000000..35cd2b4 --- /dev/null +++ b/vcf_rdfizer_policies/policy.py @@ -0,0 +1,157 @@ +"""Read an ODRL policy file into rules, refusing anything the engine cannot evaluate. + +Supported (docs/policy-demonstrator.md §3): odrl:permission and odrl:prohibition +with action odrl:read; a target that is either a resource IRI or a +vcfp:GraphSelection whose selector type the profile declares; an assignee; +purpose constraints with odrl:isAnyOf / odrl:isNoneOf; duties, which are +recorded but not enforced; and odrl:conflict odrl:prohibit. Anything else -- +an unknown selector type, a missing parameter, an effect other than drop, an +unrecognised property on a rule -- raises PolicyError. Silently ignoring a rule +would be worse than refusing the policy. +""" + +from dataclasses import dataclass +import hashlib +from pathlib import Path + +from . import ODRL, VCFP, PolicyError +from .profile import variable + +RDF_TYPE = "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" +POLICY_CLASSES = {ODRL + name for name in ("Policy", "Set", "Offer", "Agreement")} +#: Properties a rule may carry. Anything else (odrl:refinement, odrl:remedy, ...) +#: could change what the rule means, so it is refused rather than dropped. +RULE_PROPERTIES = {RDF_TYPE, ODRL + "target", ODRL + "action", ODRL + "assignee", + ODRL + "assigner", ODRL + "constraint", ODRL + "duty"} + + +@dataclass(frozen=True) +class Direct: + """A target that is one resource, e.g. .""" + iri: str + + +@dataclass(frozen=True) +class Selection: + """A target computed by a declared selector type with these parameter bindings.""" + asset: str + selector: object # profile.SelectorType + bindings: tuple # ((variable name, rdflib term), ...) + + +@dataclass(frozen=True) +class Constraint: + operator: str # "isAnyOf" or "isNoneOf" + purposes: frozenset # term IRIs + + +@dataclass(frozen=True) +class Rule: + kind: str # "permission" or "prohibition" + policy: str # the IRI of the policy the rule belongs to + target: object # Direct | Selection + assignee: str = None # None means odrl:All + constraints: tuple = () + duties: tuple = () # ODRL action IRIs; recorded, not enforced + + @property + def label(self) -> str: + """Short, stable name for reports: kind and target.""" + return f"{self.kind} on <{getattr(self.target, 'asset', None) or self.target.iri}>" + + +def policy_digest(path: Path) -> str: + """sha256 of the policy file's bytes, as recorded in every manifest.""" + return "sha256:" + hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def read_graph(path: Path): + import rdflib + + return rdflib.Graph().parse(str(path), format="turtle") + + +def load_rules(graph, profile, vocabulary) -> list: + """Every rule of every policy in `graph`, validated against the profile and vocabulary.""" + import rdflib + + policies = {s for s, o in graph.subject_objects(rdflib.RDF.type) if str(o) in POLICY_CLASSES} + if not policies: + raise PolicyError("no odrl:Policy, Set, Offer or Agreement found") + rules = [] + for policy in sorted(policies, key=str): + if str(graph.value(policy, rdflib.URIRef(ODRL + "conflict"))) != ODRL + "prohibit": + raise PolicyError(f"<{policy}>: odrl:conflict odrl:prohibit (deny wins) is required") + if graph.value(policy, rdflib.URIRef(ODRL + "obligation")) is not None: + raise PolicyError(f"<{policy}>: odrl:obligation is not supported") + for kind in ("permission", "prohibition"): + for node in graph.objects(policy, rdflib.URIRef(ODRL + kind)): + rules.append(_rule(graph, str(policy), kind, node, profile, vocabulary)) + return rules + + +def _rule(graph, policy, kind, node, profile, vocabulary): + import rdflib + + where = f"{kind} in <{policy}>" + unknown = {str(p) for p in graph.predicates(node)} - RULE_PROPERTIES + if unknown: + raise PolicyError(f"{where} uses unsupported properties: {sorted(unknown)}") + if str(graph.value(node, rdflib.URIRef(ODRL + "action"))) != ODRL + "read": + raise PolicyError(f"{where}: the only supported action is odrl:read") + assignee = graph.value(node, rdflib.URIRef(ODRL + "assignee")) + constraints = tuple(_constraint(graph, c, vocabulary) + for c in graph.objects(node, rdflib.URIRef(ODRL + "constraint"))) + return Rule(kind, policy, _target(graph, graph.value(node, rdflib.URIRef(ODRL + "target")), profile, where), + None if assignee is None or str(assignee) == ODRL + "All" else str(assignee), + constraints, + tuple(sorted(_duty(graph, d) for d in graph.objects(node, rdflib.URIRef(ODRL + "duty"))))) + + +def _target(graph, node, profile, where): + import rdflib + + if node is None: + raise PolicyError(f"{where} has no odrl:target") + selectors = list(graph.objects(node, rdflib.URIRef(VCFP + "selector"))) + if not selectors: + if not isinstance(node, rdflib.URIRef): + raise PolicyError(f"{where}: a target must be an IRI or a vcfp:GraphSelection") + return Direct(str(node)) + if len(selectors) != 1: + raise PolicyError(f"<{node}>: a GraphSelection needs exactly one vcfp:selector") + kind = str(graph.value(selectors[0], rdflib.RDF.type) or "") + if kind not in profile.selectors: + raise PolicyError(f"<{node}>: selector type <{kind or '(none)'}> is not declared by the profile") + selector = profile.selectors[kind] + bindings = [] + for prop in selector.parameters: + value = graph.value(selectors[0], rdflib.URIRef(prop)) + if value is None: + raise PolicyError(f"<{node}>: a {kind.rsplit('#', 1)[-1]} needs <{prop}>") + bindings.append((variable(prop), value)) + return Selection(str(node), selector, tuple(bindings)) + + +def _constraint(graph, node, vocabulary): + import rdflib + + left = str(graph.value(node, rdflib.URIRef(ODRL + "leftOperand"))) + operator = str(graph.value(node, rdflib.URIRef(ODRL + "operator"))) + if left != ODRL + "purpose": + raise PolicyError(f"only odrl:purpose constraints are supported, not <{left}>") + if operator not in (ODRL + "isAnyOf", ODRL + "isNoneOf"): + raise PolicyError(f"only odrl:isAnyOf and odrl:isNoneOf are supported, not <{operator}>") + values = list(graph.objects(node, rdflib.URIRef(ODRL + "rightOperand"))) + if not values: + raise PolicyError("a purpose constraint has no odrl:rightOperand") + return Constraint(operator.rsplit("/", 1)[1], frozenset(vocabulary.resolve(str(v)) for v in values)) + + +def _duty(graph, node): + import rdflib + + transform = graph.value(node, rdflib.URIRef(VCFP + "transform")) + if transform is not None and str(transform) != VCFP + "drop": + raise PolicyError(f"effect <{transform}> is not supported; only vcfp:drop") + return str(graph.value(node, rdflib.URIRef(ODRL + "action"))) diff --git a/vcf_rdfizer_policies/profile.py b/vcf_rdfizer_policies/profile.py new file mode 100644 index 0000000..d980ed3 --- /dev/null +++ b/vcf_rdfizer_policies/profile.py @@ -0,0 +1,102 @@ +"""Profiles: the selector types and the partitioning rule, read from Turtle. + +A profile is what makes the engine specific to a kind of graph without any +code. It declares the selector types a policy may use -- each a SPARQL SELECT +that projects ?resource -- and how a withheld resource takes others with it. +The bundled VCF Core profile (vcf_rdfizer_data/policy/vcf-core-profile.ttl) +covers graphs written by VCF-RDFizer; its comments document every term. +""" + +from dataclasses import dataclass, field +from importlib.resources import files +from pathlib import Path + +from . import VCFP, PolicyError + +#: The name `--profile vcf-core` refers to, and the default when none is given. +BUNDLED = {"vcf-core": "vcf-core-profile.ttl"} + + +@dataclass(frozen=True) +class SelectorType: + iri: str + query: str + parameters: tuple # property IRIs; each binds ? + violations: str = None # optional SELECT; any row means "cannot apply here" + + +@dataclass(frozen=True) +class Profile: + iri: str + ownership_path: str = None + iri_subtree: bool = False + unit_query: str = None + selectors: dict = field(default_factory=dict) # type IRI -> SelectorType + + +def variable(prop: str) -> str: + """The query variable a parameter property binds: vcfp:start -> 'start'.""" + return prop.rstrip("/#").replace("#", "/").rsplit("/", 1)[1] + + +def load_profile(sources=(), extra_graph=None) -> Profile: + """Merge profile files (paths or bundled names) and any declarations in `extra_graph`. + + `extra_graph` is the policy's own graph, so a policy can declare the selector + types it uses. Exactly one vcfp:Profile must result. + """ + import rdflib + + graph = rdflib.Graph() + for source in sources or ("vcf-core",): + path = files("vcf_rdfizer_data.policy") / BUNDLED[source] if source in BUNDLED else Path(source) + graph.parse(str(path), format="turtle") + if extra_graph is not None: + graph += extra_graph + + def one(node, name, cast=str): + value = graph.value(node, rdflib.URIRef(VCFP + name)) + return None if value is None else cast(value) + + profiles = list(graph.subjects(rdflib.RDF.type, rdflib.URIRef(VCFP + "Profile"))) + if len(profiles) != 1: + raise PolicyError(f"expected exactly one vcfp:Profile, found {len(profiles)}") + node = profiles[0] + selectors = {} + for kind in graph.subjects(rdflib.RDF.type, rdflib.URIRef(VCFP + "SelectorType")): + selectors[str(kind)] = _selector(graph, kind, one) + return Profile( + iri=str(node), + ownership_path=one(node, "ownershipPath"), + iri_subtree=bool(one(node, "iriSubtree", lambda v: v.toPython())), + unit_query=_checked(one(node, "unitQuery"), f"<{node}> vcfp:unitQuery", ("resource", "group")), + selectors=selectors, + ) + + +def _selector(graph, kind, one) -> SelectorType: + import rdflib + + query = one(kind, "query") + if query is None: + raise PolicyError(f"selector type <{kind}> has no vcfp:query") + parameters = tuple(sorted(str(p) for p in graph.objects(kind, rdflib.URIRef(VCFP + "parameter")))) + return SelectorType(str(kind), _checked(query, f"<{kind}> vcfp:query", ("resource",)), + parameters, _checked(one(kind, "violations"), f"<{kind}> vcfp:violations", ())) + + +def _checked(query, where, required): + """Parse a declared query now, so a typo fails at load time, not mid-evaluation.""" + if query is None: + return None + from rdflib.plugins.sparql import prepareQuery + + try: + prepared = prepareQuery(query) + except Exception as error: # rdflib raises several parser exception types + raise PolicyError(f"{where} does not parse: {error}") from None + projected = {str(v) for v in prepared.algebra.get("PV", [])} + missing = [v for v in required if v not in projected] + if missing: + raise PolicyError(f"{where} must project {', '.join('?' + v for v in missing)}") + return query diff --git a/vcf_rdfizer_policies/release.py b/vcf_rdfizer_policies/release.py new file mode 100644 index 0000000..a7f2245 --- /dev/null +++ b/vcf_rdfizer_policies/release.py @@ -0,0 +1,151 @@ +"""Evaluate a policy for one request, write the release, and attach policies to a graph. + +`evaluate` produces the view (engine.view), a decision for every reporting unit +the profile names, and a decision for each unit group. `write_release` writes +them with a manifest. `attach` writes the policies into the data instead, so +they can be queried alongside it. +""" + +from dataclasses import dataclass, field +from datetime import datetime, timezone +import csv +import json +from pathlib import Path + +from . import DISCLOSURE_MODEL, ODRL, VCFP, VERSION +from .engine import check_preconditions, evaluation, select, units, view +from .policy import Direct + + +@dataclass +class Release: + request: object + view: list = field(default_factory=list) # released (s, p, o) + triples_withheld: int = 0 + units: list = field(default_factory=list) # (unit dict, released, reason) + groups: dict = field(default_factory=dict) # group IRI -> (released, reason) + duties: tuple = () # of the permissions that released something + + +def evaluate(graph, rules, request, profile, vocabulary) -> Release: + decided = evaluation(graph, rules, request, profile, vocabulary) + release = Release(request) + release.view, release.triples_withheld = view(graph, decided) + for unit in units(graph, profile): + release.units.append((unit, *decided.decide(unit["resource"]))) + for group in sorted({u["group"] for u, _, _ in release.units}, key=str): + release.groups[str(group)] = decided.decide(group) + released_subjects = {s for s, _, _ in release.view} + release.duties = tuple(sorted({ + duty for rule, owned in decided.binding if rule.kind == "permission" + and any(decided.partition.contains(owned, s) for s in released_subjects) + for duty in rule.duties})) + return release + + +def summary(release) -> dict: + """Counts per group and per deciding reason, for the manifest and the paper figure.""" + counts = {g: {"released": ok, "reason": why, "records_released": 0, "records_withheld": 0} + for g, (ok, why) in release.groups.items()} + reasons = {} + for unit, ok, why in release.units: + counts[str(unit["group"])]["records_released" if ok else "records_withheld"] += 1 + reasons[why] = reasons.get(why, 0) + 1 + return { + "request": {"assignee": release.request.assignee, "purpose": release.request.purpose}, + "groups": counts, + "records_released": sum(ok for _, ok, _ in release.units), + "records_withheld": sum(not ok for _, ok, _ in release.units), + "triples_released": len(release.view), + "triples_withheld": release.triples_withheld, + "reasons": reasons, + } + + +def write_release(release, out_dir: Path, *, policies, digest: str) -> None: + """Write view.nt, decisions.csv, summary.json and manifest.ttl into a new directory.""" + import rdflib + + out_dir = Path(out_dir) + if out_dir.exists() and any(out_dir.iterdir()): + raise FileExistsError(f"{out_dir} is not empty; a release is never overwritten") + out_dir.mkdir(parents=True, exist_ok=True) + + graph = rdflib.Graph() + for triple in release.view: + graph.add(triple) + lines = sorted(line for line in graph.serialize(format="nt").splitlines() if line.strip()) + (out_dir / "view.nt").write_text("\n".join(lines) + "\n", encoding="utf-8") + + columns = list(release.units[0][0]) if release.units else ["resource", "group"] + with (out_dir / "decisions.csv").open("w", newline="", encoding="utf-8") as handle: + writer = csv.writer(handle) + writer.writerow(columns + ["released", "reason"]) + for unit, ok, why in release.units: + writer.writerow([("" if unit[c] is None else str(unit[c])) for c in columns] + [ok, why]) + + counts = summary(release) + (out_dir / "summary.json").write_text(json.dumps(counts, indent=2) + "\n", encoding="utf-8") + (out_dir / "manifest.ttl").write_text( + manifest(counts, policies=policies, digest=digest, sources=release.groups, duties=release.duties), + encoding="utf-8") + + +def manifest(counts, *, policies, digest, sources, duties) -> str: + """The vcfp:ReleaseView description (docs/policy-demonstrator.md §6), as Turtle.""" + import rdflib + from rdflib.namespace import PROV, XSD + + vcfp, odrl = rdflib.Namespace(VCFP), rdflib.Namespace(ODRL) + g = rdflib.Graph() + for prefix, namespace in (("vcfp", vcfp), ("odrl", odrl), ("prov", PROV)): + g.bind(prefix, namespace) + node, request = rdflib.URIRef("#release"), rdflib.BNode() + g.add((node, rdflib.RDF.type, vcfp.ReleaseView)) + for source in sorted(sources): + g.add((node, vcfp.derivedFrom, rdflib.URIRef(source))) + for policy in sorted(policies): + g.add((node, vcfp.policy, rdflib.URIRef(policy))) + g.add((node, vcfp.policyDigest, rdflib.Literal(digest))) + g.add((node, vcfp.request, request)) + g.add((request, odrl.assignee, rdflib.URIRef(counts["request"]["assignee"]))) + g.add((request, odrl.purpose, rdflib.URIRef(counts["request"]["purpose"]))) + for key, prop in (("records_released", "recordsReleased"), ("records_withheld", "recordsWithheld"), + ("triples_withheld", "triplesWithheld")): + g.add((node, vcfp[prop], rdflib.Literal(counts[key]))) + g.add((node, vcfp.groupsWithheld, rdflib.Literal(sum(not c["released"] for c in counts["groups"].values())))) + for duty in duties: + obligation = rdflib.BNode() + g.add((node, vcfp.obligation, obligation)) + g.add((obligation, odrl.action, rdflib.URIRef(duty))) + g.add((node, vcfp.disclosureModel, rdflib.Literal(DISCLOSURE_MODEL))) + g.add((node, PROV.wasGeneratedBy, rdflib.URIRef(f"urn:vcf-rdfizer-policy:{VERSION}"))) + g.add((node, PROV.generatedAtTime, + rdflib.Literal(datetime.now(timezone.utc).replace(microsecond=0).isoformat(), datatype=XSD.dateTime))) + return g.serialize(format="turtle") + + +def attach(graph, policy_graph, rules) -> dict: + """Merge the policies into `graph` and link each governed resource to its policy. + + A direct target gets odrl:hasPolicy. A selection records what it selects + (vcfp:selects), and each selected resource gets odrl:hasPolicy too, so a + SPARQL query needs no knowledge of the selectors. Returns counts per asset. + """ + import rdflib + + has_policy, selects = rdflib.URIRef(ODRL + "hasPolicy"), rdflib.URIRef(VCFP + "selects") + check_preconditions(graph, rules) + selections = {rule.target: select(graph, rule.target) for rule in rules} # before merging + graph += policy_graph + counts = {} + for rule in rules: + policy = rdflib.URIRef(rule.policy) + chosen = selections[rule.target] + asset = None if isinstance(rule.target, Direct) else rdflib.URIRef(rule.target.asset) + for resource in chosen: + graph.add((resource, has_policy, policy)) + if asset is not None: + graph.add((asset, selects, resource)) + counts[getattr(rule.target, "asset", None) or rule.target.iri] = len(chosen) + return counts diff --git a/vcf_rdfizer_policies/vcf_oracle.py b/vcf_rdfizer_policies/vcf_oracle.py new file mode 100644 index 0000000..2456d8f --- /dev/null +++ b/vcf_rdfizer_policies/vcf_oracle.py @@ -0,0 +1,73 @@ +"""The VCF-side oracle: which records a correct view of VCF-RDFizer graphs releases. + +It reads the source VCFs as text -- no converted graph, no conversion -- and +builds a small VCF Core graph of the fixed columns except INFO: each file with +its reference genome; each record with CHROM, POS, ID, REF and ALT; and its +call with QUAL and FILTER, in the shapes and under the IRIs VCF-RDFizer uses +(file://NAME, #record/N, #call/N). A selector that reads INFO, FORMAT or the +header is outside what the oracle models; its views are covered by the +structural checks in check.py only, so do not pass --vcf for such a policy. +The same policy is then evaluated on that graph. The view's records must equal +the records released there exactly: an extra one is a leak, a missing one is +over-withholding. Its independence comes from its input, not from a second +copy of the rules. +""" + +from decimal import Decimal +import gzip +from pathlib import Path +import re + +from . import VCFC +from .engine import evaluation, units + + +def graph_from_vcfs(paths): + """A minimal VCF Core graph built directly from VCF text.""" + import rdflib + + vcfc = rdflib.Namespace(VCFC) + graph = rdflib.Graph() + for path in map(Path, paths): + name = re.sub(r"\.gz$", "", path.name) + file_iri = rdflib.URIRef(f"file://{name}") + graph.add((file_iri, rdflib.RDF.type, vcfc.VCFFile)) + row = 0 + opener = gzip.open if path.name.endswith(".gz") else open + with opener(path, "rt", encoding="utf-8") as handle: + for line in handle: + if line.startswith("##reference="): + graph.add((file_iri, vcfc.referenceGenome, rdflib.Literal(line.split("=", 1)[1].strip()))) + if line.startswith("#"): + continue + row += 1 + chrom, pos, ident, ref, alt, qual, filters = line.rstrip("\n").split("\t")[:7] + record = rdflib.URIRef(f"{file_iri}#record/{row}") + call = rdflib.URIRef(f"{file_iri}#call/{row}") + graph.add((file_iri, vcfc.hasRecord, record)) + graph.add((record, rdflib.RDF.type, vcfc.VCFRecord)) + graph.add((record, vcfc.hasCall, call)) + graph.add((record, vcfc.chrom, rdflib.Literal(chrom))) + graph.add((record, vcfc.pos, rdflib.Literal(int(pos)))) + graph.add((record, vcfc.ref, rdflib.Literal(ref))) + for allele in alt.split(","): + if allele != ".": + graph.add((record, vcfc.alt, rdflib.Literal(allele))) + for value in ident.split(";"): + if value != ".": + graph.add((record, vcfc.recordId, rdflib.Literal(value))) + if qual != ".": + graph.add((call, vcfc.qual, rdflib.Literal(Decimal(qual)))) + if filters != ".": + graph.add((call, vcfc.filter, rdflib.Literal(filters))) + return graph + + +def compare(view, vcf_paths, *, rules, request, profile, vocabulary) -> list: + """Failures where the view's records differ from the oracle's released records.""" + oracle = graph_from_vcfs(vcf_paths) + decided = evaluation(oracle, rules, request, profile, vocabulary) + expected = {str(u["resource"]) for u in units(oracle, profile) if decided.decide(u["resource"])[0]} + actual = {str(u["resource"]) for u in units(view, profile)} + return ([f"leak: <{r}> is released but the policy withholds it" for r in sorted(actual - expected)] + + [f"over-withheld: <{r}> should have been released" for r in sorted(expected - actual)]) diff --git a/vcf_rdfizer_policies/vocabulary.py b/vcf_rdfizer_policies/vocabulary.py new file mode 100644 index 0000000..1346388 --- /dev/null +++ b/vcf_rdfizer_policies/vocabulary.py @@ -0,0 +1,59 @@ +"""Purpose vocabularies: which purposes fall within which. + +A purpose satisfies a term when it is that term or narrower, following +rdfs:subClassOf or skos:broader, so any RDFS or SKOS vocabulary works. The +bundled four-term DUO subset (vcf_rdfizer_data/policy/duo-subset.ttl) is only +the default; full DUO or a local vocabulary is passed with --purposes. +""" + +from importlib.resources import files +from pathlib import Path + +from . import PolicyError + +BROADER = ("http://www.w3.org/2000/01/rdf-schema#subClassOf", + "http://www.w3.org/2004/02/skos/core#broader") + + +class Vocabulary: + def __init__(self, graph): + import rdflib + + self.parents = {} + for predicate in BROADER: + for narrow, broad in graph.subject_objects(rdflib.URIRef(predicate)): + self.parents.setdefault(str(narrow), set()).add(str(broad)) + self.terms = set(self.parents) | {b for broads in self.parents.values() for b in broads} + self._namespaces = graph.namespace_manager + + @classmethod + def load(cls, path=None) -> "Vocabulary": + """The vocabulary at `path`, or the bundled DUO subset.""" + import rdflib + + source = Path(path) if path else files("vcf_rdfizer_data.policy") / "duo-subset.ttl" + return cls(rdflib.Graph().parse(str(source), format="turtle")) + + def resolve(self, value: str) -> str: + """A full IRI, or a prefixed name the vocabulary binds (DUO:0000007), as a known term.""" + iri = value.strip() + if ":" in iri and "://" not in iri: + try: + iri = str(self._namespaces.expand_curie(iri)) + except ValueError: + raise PolicyError(f"{value!r}: prefix is not bound by the purpose vocabulary") from None + if iri not in self.terms: + raise PolicyError(f"{value} is not a term of the purpose vocabulary") + return iri + + def within(self, purpose: str, term: str) -> bool: + """True when `purpose` is `term` or a descendant of it.""" + seen, frontier = set(), [purpose] + while frontier: + node = frontier.pop() + if node == term: + return True + if node not in seen: + seen.add(node) + frontier.extend(self.parents.get(node, ())) + return False diff --git a/vcf_rdfizer_policy.py b/vcf_rdfizer_policy.py new file mode 100644 index 0000000..876ea12 --- /dev/null +++ b/vcf_rdfizer_policy.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +"""vcf-rdfizer-policy: attach ODRL policies to RDF graphs, evaluate them, and check the result. + + vcf-rdfizer-policy explain --policy policy.ttl + vcf-rdfizer-policy attach --rdf P*.nt.gz --policy policy.ttl -o annotated.nt + vcf-rdfizer-policy evaluate --rdf P*.nt.gz --policy policy.ttl --assignee IRI --purpose DUO:0000007 -o views/alz + vcf-rdfizer-policy check --view views/alz --rdf P*.nt.gz --policy policy.ttl [--vcf P*.vcf] + +Selectors and the ownership rule come from a profile (--profile, default the +bundled VCF Core profile; selector types may also be declared in the policy +file), and purposes from a vocabulary (--purposes, default a DUO subset). +v0.1.0 demonstrator: governed release, not anonymization. Runs on the host; no +Docker. Exit codes: 0 success, 1 a check failed, 2 the policy, graph or request +cannot be evaluated. See docs/policy-demonstrator.md. +""" + +import argparse +from pathlib import Path +import sys + +from vcf_rdfizer_policies import VERSION, PolicyError + + +def _require_rdflib(): + try: + import rdflib # noqa: F401 + except ModuleNotFoundError: + raise PolicyError("vcf-rdfizer-policy requires 'rdflib': python -m pip install rdflib") from None + + +def _setup(args): + """(policy graph, profile, vocabulary, rules) from the common arguments.""" + from vcf_rdfizer_policies.policy import load_rules, read_graph + from vcf_rdfizer_policies.profile import load_profile + from vcf_rdfizer_policies.vocabulary import Vocabulary + + graph = read_graph(args.policy) + profile = load_profile(args.profile, extra_graph=graph) + vocabulary = Vocabulary.load(args.purposes) + return graph, profile, vocabulary, load_rules(graph, profile, vocabulary) + + +def cmd_explain(args): + _, profile, _, rules = _setup(args) + for rule in rules: + who = "anyone" if rule.assignee is None else f"<{rule.assignee}>" + terms = [f"purpose {c.operator} {', '.join(sorted(p.rsplit('/', 1)[1] for p in c.purposes))}" + for c in rule.constraints] + what = f" (a {rule.target.selector.iri.rsplit('#', 1)[-1]})" if hasattr(rule.target, "selector") else "" + print(f"{rule.label}{what}: applies to {who}" + (" when " + " and ".join(terms) if terms else "") + + (f"; duties: {', '.join(d.rsplit('/', 1)[1] for d in rule.duties)}" if rule.duties else "")) + print(f"Deny wins; anything no permission covers is withheld. Profile: <{profile.iri}>.") + return 0 + + +def cmd_attach(args): + from vcf_rdfizer_policies.graphs import load + from vcf_rdfizer_policies.release import attach + + policy_graph, _, _, rules = _setup(args) + out = Path(args.out) + if out.exists(): + raise FileExistsError(f"{out} exists; attach never overwrites") + graph = load(args.rdf) + for asset, n in sorted(attach(graph, policy_graph, rules).items()): + print(f"{asset}: {n} resource(s) linked") + out.write_text(graph.serialize(format="nt"), encoding="utf-8") + print(f"wrote {out}") + return 0 + + +def cmd_evaluate(args): + from vcf_rdfizer_policies.engine import Request + from vcf_rdfizer_policies.graphs import load + from vcf_rdfizer_policies.policy import policy_digest + from vcf_rdfizer_policies.release import evaluate, summary, write_release + + _, profile, vocabulary, rules = _setup(args) + request = Request(args.assignee, vocabulary.resolve(args.purpose)) + release = evaluate(load(args.rdf), rules, request, profile, vocabulary) + write_release(release, args.out, policies={r.policy for r in rules}, digest=policy_digest(args.policy)) + counts = summary(release) + print(f"released {counts['records_released']} record(s), withheld {counts['records_withheld']}; " + f"{counts['triples_withheld']} triple(s) withheld -> {args.out}") + return 0 + + +def cmd_check(args): + from vcf_rdfizer_policies.check import check_view, read_view + from vcf_rdfizer_policies.graphs import load + from vcf_rdfizer_policies.vcf_oracle import compare + + _, profile, vocabulary, rules = _setup(args) + view, manifest, request = read_view(args.view) + failures = check_view(view, manifest, request, policy_path=args.policy, rules=rules, + profile=profile, vocabulary=vocabulary, source=load(args.rdf)) + if args.vcf: + failures += compare(view, args.vcf, rules=rules, request=request, profile=profile, vocabulary=vocabulary) + for failure in failures: + print(f"FAIL {failure}") + print("PASS" if not failures else f"{len(failures)} failure(s)") + return 0 if not failures else 1 + + +def build_parser(): + parser = argparse.ArgumentParser(prog="vcf-rdfizer-policy", description=__doc__.split("\n\n")[0]) + parser.add_argument("--version", action="version", version=f"%(prog)s {VERSION}") + common = argparse.ArgumentParser(add_help=False) + common.add_argument("--policy", required=True, type=Path, help="ODRL policy file (Turtle)") + common.add_argument("--profile", action="append", default=[], + help="profile file, or 'vcf-core' (default); repeatable") + common.add_argument("--purposes", type=Path, help="RDFS/SKOS purpose vocabulary (default: a DUO subset)") + sub = parser.add_subparsers(dest="command", required=True) + + explain = sub.add_parser("explain", parents=[common], help="list the policy's rules in plain language") + explain.set_defaults(run=cmd_explain) + + attach = sub.add_parser("attach", parents=[common], help="write the policies into the graph") + attach.add_argument("--rdf", nargs="+", required=True, help=".nt / .nt.gz inputs") + attach.add_argument("-o", "--out", required=True, help="annotated .nt to create") + attach.set_defaults(run=cmd_attach) + + evaluate = sub.add_parser("evaluate", parents=[common], help="write one request's release view") + evaluate.add_argument("--rdf", nargs="+", required=True, help=".nt / .nt.gz inputs") + evaluate.add_argument("--assignee", required=True, help="the requesting party's IRI") + evaluate.add_argument("--purpose", required=True, help="a vocabulary term, e.g. DUO:0000007") + evaluate.add_argument("-o", "--out", required=True, type=Path, help="new or empty directory") + evaluate.set_defaults(run=cmd_evaluate) + + check = sub.add_parser("check", parents=[common], help="verify a view against its source") + check.add_argument("--view", required=True, type=Path, help="a directory written by evaluate") + check.add_argument("--rdf", nargs="+", required=True, help="the source the view was made from") + check.add_argument("--vcf", nargs="+", help="source VCFs, for the independent record oracle") + check.set_defaults(run=cmd_check) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + try: + _require_rdflib() + return args.run(args) + except (PolicyError, FileExistsError, FileNotFoundError) as error: + print(f"error: {error}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main())