diff --git a/docs/generators/jsonld-context.rst b/docs/generators/jsonld-context.rst index da5e4df794..b90bf131e5 100644 --- a/docs/generators/jsonld-context.rst +++ b/docs/generators/jsonld-context.rst @@ -132,6 +132,8 @@ Docs Command Line ^^^^^^^^^^^^ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. currentmodule:: linkml.generators.jsonldcontextgen .. click:: linkml.generators.jsonldcontextgen:cli diff --git a/docs/generators/owl.rst b/docs/generators/owl.rst index 4b6f076fe8..7ebcb2fb1d 100644 --- a/docs/generators/owl.rst +++ b/docs/generators/owl.rst @@ -67,6 +67,13 @@ Mapping .. note:: The current default settings for ``metaclasses`` and ``type-objects`` may change in the future +Prefix normalization +^^^^^^^^^^^^^^^^^^^^ + +``--normalize-prefixes`` writes the conventional names of well-known namespaces, the same in +every generator; see :ref:`prefix-normalization`. + + Enums and PermissibleValues ^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/generators/shacl.rst b/docs/generators/shacl.rst index 3e88f0090f..42767f04ce 100644 --- a/docs/generators/shacl.rst +++ b/docs/generators/shacl.rst @@ -87,6 +87,8 @@ Example Output: Command Line ^^^^^^^^^^^^ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. currentmodule:: linkml.generators.shaclgen .. click:: linkml.generators.shaclgen:cli diff --git a/docs/generators/shex.rst b/docs/generators/shex.rst index 6bebcf67b0..8db18d939c 100644 --- a/docs/generators/shex.rst +++ b/docs/generators/shex.rst @@ -64,6 +64,8 @@ Docs Command Line ^^^^^^^^^^^^ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. currentmodule:: linkml.generators.shexgen .. click:: linkml.generators.shexgen:cli diff --git a/docs/generators/sparql.rst b/docs/generators/sparql.rst index bc2fcd2ef3..f1c4ac5d3b 100644 --- a/docs/generators/sparql.rst +++ b/docs/generators/sparql.rst @@ -60,6 +60,8 @@ Docs Command Line ^^^^^^^^^^^^ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. currentmodule:: linkml.generators.sparqlgen .. click:: linkml.generators.sparqlgen:cli diff --git a/docs/generators/sssom.rst b/docs/generators/sssom.rst index 73f0df407c..5d0f9dc8d2 100644 --- a/docs/generators/sssom.rst +++ b/docs/generators/sssom.rst @@ -16,6 +16,8 @@ Docs Command Line ^^^^^^^^^^^^ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. click:: linkml.generators.sssomgen:cli :prog: gen-sssom :nested: full diff --git a/docs/generators/yarrrml.rst b/docs/generators/yarrrml.rst index 80b8608f86..57a8c09d22 100644 --- a/docs/generators/yarrrml.rst +++ b/docs/generators/yarrrml.rst @@ -133,6 +133,8 @@ Overview Command Line ------------ +``--normalize-prefixes`` is described under :ref:`prefix-normalization`. + .. code:: bash linkml generate yarrrml path/to/schema.yaml > mappings.yml diff --git a/docs/schemas/uris-and-mappings.md b/docs/schemas/uris-and-mappings.md index 090f51e4b9..f32cba418e 100644 --- a/docs/schemas/uris-and-mappings.md +++ b/docs/schemas/uris-and-mappings.md @@ -50,6 +50,49 @@ LinkML also provides a way to import prefix maps to avoid repetitively declaring best practice to explicitly declare prefix maps, and to use the linter to check for consistency with standard prefix registries like bioregistry and prefix.cc. +(prefix-normalization)= +### Conventional prefix names in generated artifacts + +A schema may declare its own name for a well-known namespace, such as `dce` for +`http://purl.org/dc/elements/1.1/`. By default every generator writes the names the schema +declares. With `--normalize-prefixes` the generators write the conventional name, here `dc`, and +all of them write the same name for each namespace. Only names change; every IRI stays the same. + +| generator | output for `dce: http://purl.org/dc/elements/1.1/` | +|---|---| +| gen-owl, gen-shacl, gen-shex, gen-sparql, gen-yarrrml, gen-sssom | `dc` only | +| gen-jsonld-context | `dc` in what it writes; `dce` stays defined | + +The two groups differ because their prefixes do different jobs: + +- In a self-contained document such as Turtle, ShExC or SPARQL, a prefix only abbreviates IRIs + within that document: prefixes "are not a formal part of the RDF data model" + ([RDF 1.1 Concepts §1.4](https://www.w3.org/TR/rdf11-concepts/#vocabularies)). The document + uses the conventional name alone. It renames only the prefixes it declares. +- A JSON-LD context is shared with the data written against it. A compact IRI whose prefix the + context does not define is read as an IRI of its own + ([JSON-LD 1.1 §4.1.5](https://www.w3.org/TR/json-ld11/#compact-iris)), so data written as + `dce:title` would stop expanding. The context keeps every name the schema declares and adds the + conventional one, so data may use either. JSON-LD compaction picks the shortest name + ([JSON-LD 1.1 API, IRI Compaction](https://www.w3.org/TR/json-ld11-api/#iri-compaction)), so + compacted data may still use the declared one. With `--no-prefixes` nothing is renamed, since the + prefixes then come from another context, such as the one gen-prefix-map writes. + +The conventional names come from two curated sources, the `linked_data` context of +[prefixmaps](https://github.com/linkml/prefixmaps) and the names rdflib binds by default; +crowd-sourced registries such as prefix.cc are not used. One plan per schema decides the names. A +name that the schema or one of its imports declares keeps denoting its namespace, since LinkML +expands the CURIEs of data with all of them: `linkml:types` declares `schema` for +`http://schema.org/`, so a schema's `sdo` for `https://schema.org/` keeps its name unless the +schema declares `schema` itself. A prefix also keeps its declared name, with a warning, when two +namespaces want the same conventional name, or when the name is that of a class, slot or +permissible value, which a JSON-LD context uses as a term. + +gen-prefix-map doesn't offer the option, since a prefix map gives each namespace one name, the +schema's. gen-python, gen-pydantic and gen-namespaces don't either, since their prefix names are part +of the generated code's interface, and gen-rdf and gen-jsonld write CURIEs as literal data. To +change the names that users of a schema write, rename the prefixes in the schema itself. + ## class uri and slot uri Slot and class URIs in LinkML provide the *meaning* for a class or slot, and give a robust and unique place to consistently find information about a class or slot in a LinkML model on the web. diff --git a/packages/linkml/src/linkml/generators/common/prefixes.py b/packages/linkml/src/linkml/generators/common/prefixes.py new file mode 100644 index 0000000000..d619630902 --- /dev/null +++ b/packages/linkml/src/linkml/generators/common/prefixes.py @@ -0,0 +1,308 @@ +"""Conventional prefix names for ``--normalize-prefixes``. + +A schema may declare its own name for a well-known namespace, such as ``sdo`` for +``https://schema.org/``. With ``--normalize-prefixes`` every generator that prints +prefix names writes the conventional name, ``schema``, and all of them write the +same name for each namespace: :func:`preferred_prefixes` computes one plan per +schema, as a :class:`curies.Converter` whose records hold the preferred name and +the other declared names as synonyms. + +How a generator applies the plan depends on what its prefixes are: + +- In a self-contained document (Turtle, RDF/XML, ShExC, SPARQL, YARRRML, SSSOM) a + prefix only abbreviates IRIs in that document; prefixes "are not a formal part + of the RDF data model" (RDF 1.1 Concepts §1.4). Such a document uses the + preferred names only. +- A JSON-LD context is shared with the data written against it, and a compact + IRI whose prefix the context no longer defines is read as an IRI of its own + (JSON-LD 1.1 §4.1.5). The context keeps every declared name and adds the + preferred one. + +>>> from linkml_runtime.utils.schemaview import SchemaView +>>> sv = SchemaView(''' +... id: http://example.org/s +... name: s +... prefixes: {ex: http://example.org/, sdo: https://schema.org/} +... default_prefix: ex +... classes: +... Person: {class_uri: sdo:Person} +... ''') +>>> converter = preferred_prefixes(sv) +>>> converter.standardize_curie("sdo:Person") +'schema:Person' +>>> converter.expand("sdo:Person") == converter.expand("schema:Person") == "https://schema.org/Person" +True +""" + +import logging +from collections.abc import Callable, Collection, Mapping +from functools import lru_cache +from typing import TYPE_CHECKING, Any + +import click +from curies import Converter, Record + +from linkml_runtime.utils.formatutils import camelcase, underscore +from linkml_runtime.utils.schemaview import SchemaView + +if TYPE_CHECKING: + from rdflib import Graph + +logger = logging.getLogger(__name__) + + +def well_known_prefix_map() -> dict[str, str]: + """Return a mapping from namespace URI to its conventional prefix name. + + The names come from two curated sources, which agree wherever both name a + namespace: + + - the ``linked_data`` context of `prefixmaps `_, + curated by the LinkML maintainers (``sh`` for SHACL, ``schema`` for + ``http://schema.org/``); + - the names rdflib binds by default (``Graph(bind_namespaces="rdflib")``), + which the RDF generators serialize with, and which therefore take precedence + (``schema`` for ``https://schema.org/``, ``dc`` for DCMI elements). + + prefixmaps' broader ``merged`` context is not used: it gives a namespace the + name of the first registry that lists it, such as ``alice`` for + ``http://example.org/`` from prefix.cc, which is no convention. + """ + return dict(_cached_well_known_prefix_map()) + + +@lru_cache(maxsize=1) +def _cached_well_known_prefix_map() -> dict[str, str]: + """Internal cached builder for well_known_prefix_map().""" + from prefixmaps import load_context + from rdflib import Graph + + ns_to_prefix = {rec.namespace: rec.prefix for rec in load_context("linked_data").prefix_expansions} + for prefix, namespace in Graph(bind_namespaces="rdflib").namespaces(): + if str(prefix): + ns_to_prefix[str(namespace)] = str(prefix) + return ns_to_prefix + + +def prefix_renames(prefixes: Mapping[str, str], taken: Collection[str] = frozenset()) -> dict[str, str]: + """Return the renames ``{prefix: conventional name}`` for the declared *prefixes*. + + *prefixes* maps each declared prefix to its namespace. A prefix is renamed to + the conventional name of its namespace (:func:`well_known_prefix_map`) when that + name belongs to the namespace alone: a declared name belongs to the namespaces + that declare it, because shared outputs keep every declared name, and any other + name to the namespaces that want it. A name in *taken*, the other names of the + outputs such as JSON-LD terms, belongs to none. A prefix that is not renamed + keeps its name, with a warning. + + >>> prefix_renames({"sdo": "https://schema.org/", "ex": "http://example.org/"}) + {'sdo': 'schema'} + >>> prefix_renames({"sdo": "https://schema.org/", "schema": "http://schema.org/"}) + {} + >>> prefix_renames({"sdo": "https://schema.org/"}, taken={"schema"}) + {} + """ + known = well_known_prefix_map() + wanted = {prefix: known[str(ns)] for prefix, ns in prefixes.items() if known.get(str(ns), prefix) != prefix} + declared: dict[str, set[str]] = {} + for prefix, ns in prefixes.items(): + declared.setdefault(prefix, set()).add(str(ns)) + wanting: dict[str, set[str]] = {} + for prefix, name in wanted.items(): + wanting.setdefault(name, set()).add(str(prefixes[prefix])) + renames = {} + for prefix, name in sorted(wanted.items()): + owners = declared.get(name) or wanting[name] + if name in taken or owners != {str(prefixes[prefix])}: + logger.warning( + "Prefix %r is not renamed to %r: that name also denotes another namespace or term", prefix, name + ) + else: + renames[prefix] = name + return renames + + +def preferred_prefixes(sv: SchemaView) -> Converter: + """Return the prefix names that ``--normalize-prefixes`` writes for the schema of *sv*. + + Each record's prefix is the name every generator writes for its namespace, and + its synonyms are the namespace's other declared names, which shared outputs keep. + The plan is computed from the schema, not from one output, so all generators + agree: a name that the schema or an import declares stays with its namespace, + and a conventional name that is a term of the schema's JSON-LD context, such as + the name of a slot, is not given to a prefix in any output. + """ + declared = _declared_prefixes(sv) + renames = prefix_renames(declared, taken=_term_names(sv)) + names: dict[str, list[str]] = {} + for prefix, ns in declared.items(): + names.setdefault(ns, []).append(prefix) + records = [] + for ns, ns_names in names.items(): + preferred = next((renames[name] for name in ns_names if name in renames), ns_names[0]) + records.append(Record(prefix=preferred, uri_prefix=ns, prefix_synonyms=[n for n in ns_names if n != preferred])) + return Converter(records) + + +def _declared_prefixes(sv: SchemaView) -> dict[str, str]: + """Return every prefix that the schema of *sv* and its imports declare, with its namespace. + + LinkML expands the CURIEs of data with all of them (``SchemaView.namespaces()`` + once the imports are loaded), so each keeps denoting its namespace in every + output: ``linkml:types`` declares ``schema`` for ``http://schema.org/``, so no + other namespace gets that name. The schema's own declaration wins over an + import's, as in SchemaLoader. + """ + declared = {str(p.prefix_prefix): str(p.prefix_reference) for p in sv.schema.prefixes.values()} + for schema_name in sv.imports_closure(): + for p in sv.schema_map[schema_name].prefixes.values(): + declared.setdefault(str(p.prefix_prefix), str(p.prefix_reference)) + return declared + + +def _term_names(sv: SchemaView) -> set[str]: + """Return the names a JSON-LD context of *sv*'s schema uses as terms or bare values. + + These are the class and slot names as the context generator writes them and + the texts of permissible values, which a ``@type: @vocab`` slot reads as terms + first (JSON-LD 1.1 API, IRI Expansion, step 5). + """ + names = {camelcase(name) for name in sv.all_classes()} + names |= {underscore(slot.alias or slot.name) for slot in sv.all_slots().values()} + names |= {text for enum in sv.all_enums().values() for text in enum.permissible_values} + return names + + +def document_prefixes( + declared: Mapping[str, str], plan: Converter, undeclared: Collection[str] = frozenset() +) -> Converter: + """Return the prefixes of a self-contained document that declares *declared* (name to namespace) under *plan*. + + Each record's prefix is the name the document uses for a namespace it declares, + the preferred one of *plan* where the plan has one, and its synonyms are the + declared names it replaces. A declaration of a name that *plan* gives to another + namespace is dropped. Only declared prefixes are renamed, and not to a name in + *undeclared*, the prefixes the document writes without declaring them, so such a + CURIE keeps what it denotes. + + >>> plan = Converter([Record(prefix="schema", uri_prefix="https://schema.org/", prefix_synonyms=["sdo"])]) + >>> doc = document_prefixes({"sdo": "https://schema.org/", "schema": "http://schema.org/"}, plan) + >>> [(r.prefix, r.uri_prefix, r.prefix_synonyms) for r in doc.records] + [('schema', 'https://schema.org/', ['sdo'])] + """ + names: dict[str, list[str]] = {} + for name, namespace in declared.items(): + planned = plan.prefix_map.get(name) + if planned is None or planned == str(namespace): + names.setdefault(str(namespace), []).append(name) + records = [] + for namespace, ns_names in names.items(): + preferred = plan.reverse_prefix_map.get(namespace, ns_names[0]) + if preferred in undeclared and preferred not in ns_names: + preferred = ns_names[0] + records.append( + Record(prefix=preferred, uri_prefix=namespace, prefix_synonyms=[n for n in ns_names if n != preferred]) + ) + return Converter(records) + + +def added_preferred_prefixes( + plan: Converter, defined: Mapping[str, str], terms: Collection[str] = frozenset() +) -> Converter: + """Return the preferred names that an output keeping its *defined* prefixes adds. + + A JSON-LD context is shared with the data written against it, so it keeps every + prefix it defines (name to namespace) and adds the preferred name of the + namespace from *plan* (:func:`preferred_prefixes`); data using either name + expands alike. Each record's synonyms are the defined names it stands for. + A preferred name in *terms*, or defined for another namespace, is not added. + + >>> plan = Converter([Record(prefix="schema", uri_prefix="https://schema.org/", prefix_synonyms=["sdo"])]) + >>> [(r.prefix, r.prefix_synonyms) for r in added_preferred_prefixes(plan, {"sdo": "https://schema.org/"}).records] + [('schema', ['sdo'])] + """ + synonyms: dict[str, list[str]] = {} + for prefix in sorted(defined): + name = plan.standardize_prefix(prefix) + if name in (None, prefix) or name in terms or defined.get(name, defined[prefix]) != defined[prefix]: + continue + synonyms.setdefault(name, []).append(prefix) + return Converter( + [Record(prefix=name, uri_prefix=defined[names[0]], prefix_synonyms=names) for name, names in synonyms.items()] + ) + + +def standardize_prefix(text: str, converter: Converter) -> str: + """Return *text* with the prefix of a compact IRI replaced by its preferred name. + + Only the part before the first colon changes, so templates such as + ``ex:Person/$(id)`` keep their rest. A suffix starting with ``//`` makes *text* + an absolute IRI (JSON-LD 1.1 §4.1.5), which is left alone. + + >>> converter = Converter([Record(prefix="schema", uri_prefix="https://schema.org/", prefix_synonyms=["sdo"])]) + >>> standardize_prefix("sdo:Person", converter), standardize_prefix("https://schema.org/x", converter) + ('schema:Person', 'https://schema.org/x') + """ + prefix, colon, rest = text.partition(":") + if colon and not rest.startswith("//"): + preferred = converter.standardize_prefix(prefix) + if preferred is not None: + return preferred + colon + rest + return text + + +def curie_prefixes(node: Any) -> set[str]: + """Return the prefixes of the compact IRIs among the string values of the JSON structure *node*. + + >>> sorted(curie_prefixes({"s": "ex:a/$(id)", "o": ["sdo:Person", "https://schema.org/x", "text"]})) + ['ex', 'sdo'] + """ + if isinstance(node, dict): + return set().union(*(curie_prefixes(value) for value in node.values())) + if isinstance(node, list): + return set().union(*(curie_prefixes(value) for value in node)) + if isinstance(node, str): + prefix, colon, rest = node.partition(":") + if colon and not rest.startswith("//"): + return {prefix} + return set() + + +def standardize_values(node: Any, converter: Converter) -> Any: + """Return the JSON structure *node* with :func:`standardize_prefix` applied to its string values. + + Keys are left alone: in a JSON-LD context they are the terms data refers to. + Dicts and lists keep their type, such as a list that a YAML dumper writes in + flow style. + """ + if isinstance(node, dict): + return type(node)((key, standardize_values(value, converter)) for key, value in node.items()) + if isinstance(node, list): + return type(node)(standardize_values(value, converter) for value in node) + if isinstance(node, str): + return standardize_prefix(node, converter) + return node + + +def bind_preferred_prefixes(graph: "Graph", converter: Converter) -> None: + """Bind the preferred name of each namespace of *converter* in *graph*. + + A name that rdflib or the metamodel bound for another namespace is taken over; + prefixes only abbreviate IRIs in the serialization (Turtle 1.1 §2.4), so no IRI + changes. + """ + from rdflib import Namespace + + for record in converter.records: + graph.bind(record.prefix, Namespace(record.uri_prefix), override=True, replace=True) + + +def normalize_prefixes_option(f: Callable) -> Callable: + """Add ``--normalize-prefixes`` to the command line of a generator that honours it.""" + return click.option( + "--normalize-prefixes/--no-normalize-prefixes", + default=False, + show_default=True, + help="Write the conventional names of well-known namespaces (e.g. schema for https://schema.org/ " + "where the schema declares sdo) in every generator alike.", + )(f) diff --git a/packages/linkml/src/linkml/generators/jsonldcontextgen.py b/packages/linkml/src/linkml/generators/jsonldcontextgen.py index 5462b29f8b..13008834a2 100644 --- a/packages/linkml/src/linkml/generators/jsonldcontextgen.py +++ b/packages/linkml/src/linkml/generators/jsonldcontextgen.py @@ -14,6 +14,13 @@ from rdflib import SKOS, XSD, Namespace from linkml._version import __version__ +from linkml.generators.common.prefixes import ( + added_preferred_prefixes, + normalize_prefixes_option, + preferred_prefixes, + standardize_prefix, + standardize_values, +) from linkml.utils.deprecation import deprecated_fields from linkml.utils.generator import Generator, shared_arguments from linkml_runtime.linkml_model.meta import ClassDefinition, EnumDefinition, SlotDefinition @@ -58,6 +65,9 @@ class ContextGenerator(Generator): prefixes: bool | None = True flatprefixes: bool | None = False fix_multivalue_containers: bool | None = False + normalize_prefixes: bool = False + """Also define the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce``, and write compact IRIs with them.""" exclude_imports: bool = False """If True, elements from imported schemas won't be included in the generated context""" exclude_external_imports: bool = False @@ -191,9 +201,19 @@ def end_schema( self.context_body["@base"] = os.path.relpath(base, os.path.dirname(self.schema.source_file)) else: self.context_body["@base"] = base + preferred = None + if self.normalize_prefixes and prefixes: + preferred = added_preferred_prefixes( + preferred_prefixes(self.prefix_schemaview()), + {prefix: str(self.namespaces[prefix]) for prefix in self.emit_prefixes}, + terms=self.context_body.keys() | self.slot_class_maps.keys(), + ) if prefixes: - for prefix in sorted(self.emit_prefixes): - url = str(self.namespaces[prefix]) + urls = {prefix: str(self.namespaces[prefix]) for prefix in self.emit_prefixes} + if preferred is not None: + urls |= {record.prefix: record.uri_prefix for record in preferred.records} + for prefix in sorted(urls): + url = urls[prefix] # Derived from line # ~5223 in pyld/lib/jsonld.py if bool(re.match(r".*[:/\?#\[\]@]$", url)) or flatprefixes: context_content[prefix] = url @@ -207,6 +227,8 @@ def end_schema( context_content[k] = v for k, v in self.slot_class_maps.items(): context_content[k] = v + if preferred is not None: + context_content = standardize_values(json.loads(as_json(context_content)), preferred) context["@context"] = context_content if output and not self.embed_context_in_frame: with open(output, "w", encoding="UTF-8") as outf: @@ -234,6 +256,8 @@ def end_schema( if root_name: root_cls = self.schema.classes[root_name] frame["@type"] = root_cls.class_uri or root_cls.name + if preferred is not None: + frame["@type"] = standardize_prefix(frame["@type"], preferred) for prop, rule in self.frame_body.items(): frame[prop] = rule @@ -616,6 +640,7 @@ def serialize( help="Use class_uri/slot_uri CURIEs as class and non-identifier slot context keys; identifier slots retain their " + "slot-name key and map to @id.", ) +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, emit_frame, embed_context_in_frame, output, **args): """Generate jsonld @context definition from LinkML model""" diff --git a/packages/linkml/src/linkml/generators/owlgen.py b/packages/linkml/src/linkml/generators/owlgen.py index 3f16c4965c..7ea4e43b89 100644 --- a/packages/linkml/src/linkml/generators/owlgen.py +++ b/packages/linkml/src/linkml/generators/owlgen.py @@ -19,6 +19,7 @@ from linkml import METAMODEL_NAMESPACE_NAME from linkml._version import __version__ +from linkml.generators.common.prefixes import bind_preferred_prefixes, normalize_prefixes_option, preferred_prefixes from linkml.generators.common.subproperty import is_xsd_anyuri_range from linkml.utils.deprecation import deprecation_warning from linkml.utils.generator import Generator, shared_arguments @@ -122,6 +123,10 @@ class OwlSchemaGenerator(Generator): """Suffix to add to the schema name to create the ontology URI, e.g. .owl.ttl""" # ObjectVars + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" + diff_stable: bool = False """Reduce blank-node label churn. See :ref:`rdf-in-version-control`.""" @@ -319,6 +324,8 @@ def as_graph(self) -> Graph: self.graph.bind(prefix, self.metamodel.namespaces[prefix]) for pfx in schema.prefixes.values(): self.graph.namespace_manager.bind(pfx.prefix_prefix, URIRef(pfx.prefix_reference)) + if self.normalize_prefixes: + bind_preferred_prefixes(graph, preferred_prefixes(self.schemaview)) graph.add((base, RDF.type, OWL.Ontology)) # Add main schema elements @@ -1856,6 +1863,7 @@ def slot_owl_type(self, slot: SlotDefinition) -> URIRef: "https://linkml.io/linkml/howtos/collaborative-development.html#rdf-in-version-control" ), ) +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile: str, metadata_profile: str, **kwargs: Any) -> None: """Generate an OWL representation of a LinkML model diff --git a/packages/linkml/src/linkml/generators/shaclgen.py b/packages/linkml/src/linkml/generators/shaclgen.py index eb6b48e9d6..42f7b180b7 100644 --- a/packages/linkml/src/linkml/generators/shaclgen.py +++ b/packages/linkml/src/linkml/generators/shaclgen.py @@ -11,6 +11,7 @@ from rdflib.namespace import RDF, RDFS, SH, XSD from linkml._version import __version__ +from linkml.generators.common.prefixes import bind_preferred_prefixes, normalize_prefixes_option, preferred_prefixes from linkml.generators.common.subproperty import get_subproperty_values, is_uri_range from linkml.generators.shacl.shacl_data_type import ShaclDataType from linkml.generators.shacl.shacl_ifabsent_processor import ShaclIfAbsentProcessor @@ -142,6 +143,10 @@ class ShaclGenerator(Generator): ignores any per-slot ``in_language``. """ + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" + diff_stable: bool = False """Reduce blank-node label churn. See :ref:`rdf-in-version-control`.""" @@ -210,6 +215,8 @@ def as_graph(self) -> Graph: for pfx in self.schema.prefixes.values(): g.bind(str(pfx.prefix_prefix), pfx.prefix_reference) + if self.normalize_prefixes: + bind_preferred_prefixes(g, preferred_prefixes(self.schemaview)) for c in sv.all_classes(imports=not self.exclude_imports).values(): @@ -972,6 +979,7 @@ def add_simple_data_type(func: Callable, r: ElementName) -> None: "https://linkml.io/linkml/howtos/collaborative-development.html#rdf-in-version-control" ), ) +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, **args): """Generate SHACL turtle from a LinkML model""" diff --git a/packages/linkml/src/linkml/generators/shexgen.py b/packages/linkml/src/linkml/generators/shexgen.py index e74e45225e..b73116da3b 100644 --- a/packages/linkml/src/linkml/generators/shexgen.py +++ b/packages/linkml/src/linkml/generators/shexgen.py @@ -13,6 +13,7 @@ from linkml import METAMODEL_NAMESPACE, METAMODEL_NAMESPACE_NAME from linkml._version import __version__ +from linkml.generators.common.prefixes import bind_preferred_prefixes, normalize_prefixes_option, preferred_prefixes from linkml.generators.common.subproperty import get_subproperty_values from linkml.utils.generator import Generator, shared_arguments from linkml_runtime.linkml_model.meta import ( @@ -42,6 +43,9 @@ class ShExGenerator(Generator): # ObjectVars diff_stable: bool = False """Reduce blank-node label churn in RDF output. See :ref:`rdf-in-version-control`.""" + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" shex: Schema = field(default_factory=lambda: Schema()) # ShEx Schema being generated shapes: list = field(default_factory=lambda: []) @@ -180,10 +184,14 @@ def end_schema(self, output: str | None = None, **_) -> str: g = Graph() g.parse(data=shex, format="json-ld", version="1.1") g.bind("owl", OWL) + if self.normalize_prefixes: + bind_preferred_prefixes(g, preferred_prefixes(self.prefix_schemaview())) shex = canonicalize_rdf_graph(g, output_format="turtle", diff_stable=self.diff_stable) elif self.format == "shex": g = Graph() self.namespaces.load_graph(g) + if self.normalize_prefixes: + bind_preferred_prefixes(g, preferred_prefixes(self.prefix_schemaview())) shex = str(ShExC(self.shex, base=sfx(self.namespaces._base), namespaces=g)) if output: @@ -270,6 +278,7 @@ def _get_subproperty_values(self, slot: SlotDefinition) -> list: "https://linkml.io/linkml/howtos/collaborative-development.html#rdf-in-version-control" ), ) +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, **args): """Generate a ShEx Schema for a LinkML model""" diff --git a/packages/linkml/src/linkml/generators/sparqlgen.py b/packages/linkml/src/linkml/generators/sparqlgen.py index 3497164246..5fcf7f9300 100644 --- a/packages/linkml/src/linkml/generators/sparqlgen.py +++ b/packages/linkml/src/linkml/generators/sparqlgen.py @@ -9,16 +9,22 @@ from jinja2 import Template from linkml._version import __version__ +from linkml.generators.common.prefixes import ( + document_prefixes, + normalize_prefixes_option, + preferred_prefixes, + standardize_prefix, +) from linkml.utils.generator import Generator, shared_arguments -from linkml_runtime.linkml_model.meta import Prefix +from linkml_runtime.linkml_model.meta import Element, Prefix from linkml_runtime.utils.formatutils import underscore from linkml_runtime.utils.schemaview import SchemaView logger = logging.getLogger(__name__) template = """ -{% for pfxn, pfx in schema.prefixes.items() -%} -PREFIX {{pfxn}}: <{{pfx.prefix_reference}}> +{% for pfxn, pfx in prefixes.items() -%} +PREFIX {{pfxn}}: <{{pfx}}> {% endfor %} {% for cn, c in schema.classes.items() if not c.mixin and not c.abstract %} @@ -30,11 +36,11 @@ # @CHECK permitted_{{cn}} SELECT ?g ?s ?p WHERE { GRAPH ?g { - ?s rdf:type {{ schema_view.get_uri(cn) }} ; + ?s rdf:type {{ uri(cn) }} ; ?p ?o . FILTER ( ?p NOT IN ( {% for sn in schema_view.class_slots(cn) -%} - {{ schema_view.get_uri(schema_view.get_slot(sn, attributes=True)) }}, + {{ uri(schema_view.get_slot(sn, attributes=True)) }}, {% endfor -%} rdf:type )) } @@ -52,11 +58,11 @@ ?subject ?predicate WHERE { GRAPH ?graph { - ?subject rdf:type {{ schema_view.get_uri(cn) }} . - FILTER NOT EXISTS { ?subject {{ schema_view.get_uri(slot) }} ?o } + ?subject rdf:type {{ uri(cn) }} . + FILTER NOT EXISTS { ?subject {{ uri(slot) }} ?o } } VALUES ?check { linkml:required } - VALUES ?predicate { {{schema_view.get_uri(slot)}} } + VALUES ?predicate { {{uri(slot)}} } {{ extra }} } {{ limit }} {% endif %} @@ -71,19 +77,19 @@ ?object WHERE { GRAPH ?graph { - ?subject rdf:type {{ schema_view.get_uri(cn) }} ; + ?subject rdf:type {{ uri(cn) }} ; ?predicate ?object . FILTER NOT EXISTS { ?object rdf:type ?otype . FILTER ( ?otype IN ( {% for a in schema_view.class_descendants(slot.range) -%} - {{ schema_view.get_uri(a) }} + {{ uri(a) }} {{ ", " if not loop.last else "" }} {% endfor -%} )) } } VALUES ?check { linkml:range } - VALUES ?predicate { {{ schema_view.get_uri(slot) }} } + VALUES ?predicate { {{ uri(slot) }} } {{ extra }} } {{ limit }} {% endif %} @@ -140,9 +146,14 @@ class SparqlGenerator(Generator): named_graphs: list[str] | None = None limit: int | None = None sparql: str | None = None + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" def __post_init__(self): super().__post_init__() + # The plan reads the schema as given, before materialization adds the imports' prefixes. + self._prefix_plan = preferred_prefixes(self.schemaview) if self.normalize_prefixes else None # Materialization rewrites the schema (attributes -> slots, extra prefixes), so work on a # copy: mutating the shared SchemaView corrupts the input for any generator that runs after # us under gen-project (#3770). Copying the whole view preserves import-resolution context. @@ -161,7 +172,20 @@ def generate_sparql(self, named_graphs=None, limit: int = None): limit = f"LIMIT {limit}" else: limit = "" - sparql = template_obj.render(schema_view=self.schemaview, schema=self.schema, limit=limit, extra=extra) + sv = self.schemaview + prefixes = {str(name): str(p.prefix_reference) for name, p in self.schema.prefixes.items()} + plan = None + if self._prefix_plan is not None: + plan = document_prefixes(prefixes, self._prefix_plan) + prefixes = {record.prefix: record.uri_prefix for record in plan.records} + + def uri(element: Element | str) -> str: + curie = sv.get_uri(element) + return curie if plan is None else standardize_prefix(curie, plan) + + sparql = template_obj.render( + schema_view=sv, schema=self.schema, prefixes=prefixes, uri=uri, limit=limit, extra=extra + ) self.sparql = sparql queries = self.split_sparql(sparql) return queries @@ -196,6 +220,7 @@ def split_sparql(sparql: str) -> dict[str, str]: @shared_arguments(SparqlGenerator) @click.command(name="sparql") @click.option("--dir", "-d", help="Directory in which queries will be deposited") +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, dir, **kwargs): """Generate SPARQL queries for validation diff --git a/packages/linkml/src/linkml/generators/sssomgen.py b/packages/linkml/src/linkml/generators/sssomgen.py index ade9c92c4f..f63b6b3045 100644 --- a/packages/linkml/src/linkml/generators/sssomgen.py +++ b/packages/linkml/src/linkml/generators/sssomgen.py @@ -5,6 +5,13 @@ import click from linkml._version import __version__ +from linkml.generators.common.prefixes import ( + curie_prefixes, + document_prefixes, + normalize_prefixes_option, + preferred_prefixes, + standardize_prefix, +) from linkml.utils.generator import Generator, shared_arguments from linkml_runtime.linkml_model.meta import ( LINKML, @@ -58,6 +65,14 @@ class SSSOMGenerator(Generator): "exact_mappings": "skos:exactMatch", } + # The columns whose values are entity references, i.e. CURIEs (SSSOM: ``EntityReference`` slots) + curie_columns = frozenset({"subject_id", "predicate_id", "object_id", "match_type"}) + + # ObjectVars + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" + def __post_init__(self): super().__post_init__() self.sourcefile = self.schema @@ -163,6 +178,16 @@ def end_schema(self, context: str = None, **_) -> None: metadata["creator_id"] = "linkml_user" metadata["mapping_date"] = date.today().strftime("%Y-%m-%d") metadata["curie_map"] = {k: v.prefix_reference for k, v in schema.prefixes.items()} + rows = self.table_as_list + if self.normalize_prefixes: + columns = [i for i, name in enumerate(self.msdf_columns) if name in self.curie_columns] + curies = {row[i] for row in rows for i in columns} + undeclared = curie_prefixes(list(curies)) - metadata["curie_map"].keys() + plan = document_prefixes(metadata["curie_map"], preferred_prefixes(self.prefix_schemaview()), undeclared) + metadata["curie_map"] = {record.prefix: record.uri_prefix for record in plan.records} + rows = [ + [standardize_prefix(cell, plan) if i in columns else cell for i, cell in enumerate(row)] for row in rows + ] with open(self.output_file, "w", encoding="UTF-8") as sssom_tsv: for k, v in metadata.items(): if k != "curie_map": @@ -176,12 +201,13 @@ def end_schema(self, context: str = None, **_) -> None: # Write column names first sssom_tsv.writelines("\t".join(self.msdf_columns) + "\n") # Write the msdf next - sssom_tsv.writelines("\t".join(i) + "\n" for i in self.table_as_list) + sssom_tsv.writelines("\t".join(i) + "\n" for i in rows) @shared_arguments(SSSOMGenerator) @click.command(name="sssom") @click.option("-o", "--output", help="Output file name") +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, **kwargs): """Generate SSSOM TSV to represent a LinkML model""" diff --git a/packages/linkml/src/linkml/generators/yarrrmlgen.py b/packages/linkml/src/linkml/generators/yarrrmlgen.py index 6847dafbd1..19aaee049e 100644 --- a/packages/linkml/src/linkml/generators/yarrrmlgen.py +++ b/packages/linkml/src/linkml/generators/yarrrmlgen.py @@ -8,6 +8,13 @@ import yaml from linkml._version import __version__ +from linkml.generators.common.prefixes import ( + curie_prefixes, + document_prefixes, + normalize_prefixes_option, + preferred_prefixes, + standardize_values, +) from linkml.utils.generator import Generator, shared_arguments from linkml_runtime.linkml_model.meta import ClassDefinition @@ -45,6 +52,9 @@ class YarrrmlGenerator(Generator): """YARRRML source shorthand, e.g. data.json~jsonpath or data.csv~csv.""" iterator_template: str | None = None """JSONPath iterator template; supports {Class}.""" + normalize_prefixes: bool = False + """Write the conventional prefix names of :mod:`linkml.generators.common.prefixes`, + e.g. ``dc`` where the schema declares ``dce`` for ``http://purl.org/dc/elements/1.1/``.""" def __post_init__(self): super().__post_init__() @@ -180,6 +190,10 @@ def as_dict(self) -> dict[str, Any]: mappings[str(cls.name)] = mapping_dict prefixes = self._prefixes_with_defaults() + if self.normalize_prefixes: + plan = document_prefixes(prefixes, preferred_prefixes(sv), curie_prefixes(mappings) - prefixes.keys()) + prefixes = {record.prefix: record.uri_prefix for record in plan.records} + mappings = standardize_values(mappings, plan) return {"prefixes": prefixes, "mappings": mappings} def _is_json_source(self) -> bool: @@ -366,6 +380,7 @@ def _po_list_for_class(self, c: ClassDefinition, inline_owners: dict) -> list[di "--iterator-template", help='JSONPath iterator template; supports {Class}, default: "$[*]"', ) +@normalize_prefixes_option @click.version_option(__version__, "-V", "--version") def cli(yamlfile, source, iterator_template, **args): """Generate YARRRML mappings from a LinkML schema.""" diff --git a/packages/linkml/src/linkml/utils/generator.py b/packages/linkml/src/linkml/utils/generator.py index 8c916ccb03..09f6be49aa 100644 --- a/packages/linkml/src/linkml/utils/generator.py +++ b/packages/linkml/src/linkml/utils/generator.py @@ -59,6 +59,7 @@ ) from linkml_runtime.utils.formatutils import camelcase, underscore from linkml_runtime.utils.namespaces import Namespaces +from linkml_runtime.utils.schemaview import load_schema_wrap logger = logging.getLogger(__name__) @@ -279,6 +280,7 @@ def __post_init__(self) -> None: schema = self.schema if isinstance(schema, Path): schema = str(schema) + self._schema_source = schema if self.uses_schemaloader: self._initialize_using_schemaloader(schema) @@ -799,6 +801,28 @@ def slot_range_path(self, slot_or_name: str | SlotDefinition) -> list[str]: # Class return self.class_identifier_path(slot.range, bool(slot.inlined)) + def prefix_schemaview(self) -> SchemaView: + """Return a SchemaView of the schema as given, with its imports unmerged. + + Prefix normalization reads the schema's own prefixes and those of its imports + from it (:func:`~linkml.generators.common.prefixes.preferred_prefixes`), the + same way for generators built on SchemaView and on SchemaLoader, whose + resolved schema has its imports merged. + """ + if self.schemaview is not None: + return self.schemaview + source = self._schema_source + if isinstance(source, Generator): + return source.prefix_schemaview() + if not isinstance(source, str | SchemaDefinition): + source = self.schema + elif isinstance(source, str) and "\n" in source and self.base_dir: + # SchemaView resolves local imports against the schema's file, which YAML text + # has none of; resolve them against base_dir, as SchemaLoader does. + source = load_schema_wrap(source) + source.source_file = os.path.join(self.base_dir, f"{source.name}.yaml") + return SchemaView(source, importmap=self.importmap, base_dir=self.base_dir) + def aliased_slot_name(self, slot: SlotDefinitionName | SlotDefinition) -> SlotDefinitionName: """Return the overloaded slot name -- the alias if one exists otherwise the actual name diff --git a/tests/linkml/test_data/test_sqlite.py b/tests/linkml/test_data/test_sqlite.py index 38dce4ffb8..8b28979d91 100644 --- a/tests/linkml/test_data/test_sqlite.py +++ b/tests/linkml/test_data/test_sqlite.py @@ -71,9 +71,17 @@ def test_mixin(tmp_outputs): assert diff == "" +@pytest.fixture +def restore_csv_field_size_limit(): + """Restore the process-wide CSV field size limit that a test changes.""" + limit = csv.field_size_limit() + yield + csv.field_size_limit(limit) + + @pytest.mark.parametrize("diff", [1, -1]) @pytest.mark.parametrize("size", [20, 200000]) -def test_csv_limit(tmp_outputs, diff, size): +def test_csv_limit(tmp_outputs, diff, size, restore_csv_field_size_limit): """ Tests https://github.com/linkml/linkml/issues/815 """ diff --git a/tests/linkml/test_generators/test_normalize_prefixes.py b/tests/linkml/test_generators/test_normalize_prefixes.py new file mode 100644 index 0000000000..77217a0bda --- /dev/null +++ b/tests/linkml/test_generators/test_normalize_prefixes.py @@ -0,0 +1,1001 @@ +"""Tests for ``--normalize-prefixes`` in every generator that writes prefix names. + +The option writes the conventional name of a well-known namespace, such as ``schema`` for +``https://schema.org/`` where a schema declares ``sdo``, and every generator writes the same +name (:mod:`linkml.generators.common.prefixes`). Only names change: + +- a self-contained document (OWL, SHACL, ShEx, SPARQL, YARRRML, SSSOM) means the same with and + without the option, since prefixes are "not a formal part of the RDF data model" + (RDF 1.1 Concepts §1.4; Turtle 1.1 §2.4); +- a JSON-LD context or a prefix map keeps every declared name and adds the conventional one, so + data written against it expands the same (JSON-LD 1.1 §4.1.5). +""" + +import json +import logging +import re +import textwrap +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pytest +import yaml +from click.testing import CliRunner +from curies import Converter, Record +from jsonasobj import as_json +from prefixmaps import load_context +from pyparsing import ParseResults +from pyshexc.parser_impl.generate_shexj import parse as parse_shexc +from rdflib import Graph +from rdflib.compare import isomorphic +from rdflib.plugins.shared.jsonld.context import Context +from rdflib.plugins.sparql.parser import parseQuery +from rdflib.plugins.sparql.parserutils import CompValue + +from linkml.cli.main import linkml as linkml_cli +from linkml.generators.common.prefixes import ( + added_preferred_prefixes, + document_prefixes, + preferred_prefixes, + prefix_renames, + standardize_prefix, + well_known_prefix_map, +) +from linkml.generators.jsonldcontextgen import ContextGenerator +from linkml.generators.owlgen import OwlSchemaGenerator +from linkml.generators.prefixmapgen import PrefixGenerator +from linkml.generators.shaclgen import ShaclGenerator +from linkml.generators.shexgen import ShExGenerator +from linkml.generators.sparqlgen import SparqlGenerator +from linkml.generators.sssomgen import SSSOMGenerator +from linkml.generators.yarrrmlgen import YarrrmlGenerator +from linkml_runtime.utils.schemaview import SchemaView + +SCHEMA_ORG = "https://schema.org/" +SCHEMA_ORG_HTTP = "http://schema.org/" +DC_ELEMENTS = "http://purl.org/dc/elements/1.1/" +DC_TERMS = "http://purl.org/dc/terms/" +FOAF = "http://xmlns.com/foaf/0.1/" +TIME = "http://www.w3.org/2006/time#" +BIBO = "http://purl.org/ontology/bibo/" +PROV = "http://www.w3.org/ns/prov#" +SKOS = "http://www.w3.org/2004/02/skos/core#" + + +def _schema(name: str, prefixes: str, body: str, default_prefix: str = "ex", imports: str = "linkml:types") -> str: + """A schema with the ``ex`` and ``linkml`` prefixes, the given extra prefix lines, imports and elements.""" + return ( + f"id: https://example.org/{name}\nname: {name}\ndefault_prefix: {default_prefix}\nprefixes:\n" + " ex: https://example.org/\n linkml: https://w3id.org/linkml/\n" + + textwrap.indent(textwrap.dedent(prefixes), " ") + + "imports:\n" + + "".join(f" - {i}\n" for i in imports.split()) + + textwrap.dedent(body) + ) + + +# Each case: the schema files (the main one is schema.yaml) and the name every generator writes for each namespace. +CASES: dict[str, tuple[dict[str, str], dict[str, str]]] = { + "prov_alias": ( + { + "schema.yaml": _schema( + "prov", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Agent: + class_uri: w3prov:Agent + exact_mappings: [w3prov:Person] + attributes: + id: {identifier: true, range: uriorcurie} + label: {slot_uri: w3prov:value} + delegate: {range: Agent, multivalued: true, slot_uri: w3prov:actedOnBehalfOf} + """, + ) + }, + {PROV: "prov"}, + ), + # linkml:types declares ``schema`` for http://schema.org/, so https://schema.org/ keeps ``sdo`` + "schema_org_alias": ( + { + "schema.yaml": _schema( + "sdo", + "sdo: https://schema.org/\n", + """\ + classes: + Person: + class_uri: sdo:Person + attributes: + id: {identifier: true, range: uriorcurie} + knows: {range: Person, multivalued: true, slot_uri: sdo:knows} + """, + ) + }, + {SCHEMA_ORG: "sdo", SCHEMA_ORG_HTTP: "schema"}, + ), + # an import declares ``prov`` for another namespace, although nothing uses it + "import_holds_a_name": ( + { + "module.yaml": _schema("module", "prov: http://example.org/not-prov/\n", ""), + "schema.yaml": _schema( + "holder", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Agent: + class_uri: w3prov:Agent + """, + imports="linkml:types module", + ), + }, + {PROV: "w3prov"}, + ), + "dce_alias": ( + { + "schema.yaml": _schema( + "dce", + "dce: http://purl.org/dc/elements/1.1/\n", + """\ + classes: + Record: + attributes: + title: {slot_uri: dce:title} + """, + ) + }, + {DC_ELEMENTS: "dc"}, + ), + # both namespaces want ``schema``, which linkml:types declares for http://schema.org/ + "http_and_https_schema_org": ( + { + "schema.yaml": _schema( + "both", + "sdo: https://schema.org/\nsdohttp: http://schema.org/\n", + """\ + classes: + Person: + class_uri: sdo:Person + attributes: + full_name: {slot_uri: sdo:name} + old_name: {slot_uri: sdohttp:alternateName} + """, + ) + }, + {SCHEMA_ORG: "sdo", SCHEMA_ORG_HTTP: "schema"}, + ), + # linkml:types declares ``schema`` for http://schema.org/, which this schema uses + "imported_name_in_use": ( + { + "schema.yaml": _schema( + "imported", + "sdo: https://schema.org/\n", + """\ + classes: + Person: + class_uri: sdo:Person + attributes: + full_name: {slot_uri: sdo:name} + legacy: {slot_uri: "schema:legacyName"} + """, + ) + }, + {SCHEMA_ORG: "sdo", SCHEMA_ORG_HTTP: "schema"}, + ), + # ``dc`` stays DCMI terms' name in shared outputs, so DCMI elements cannot take it + "dc_terms_and_dce": ( + { + "schema.yaml": _schema( + "dcboth", + "dc: http://purl.org/dc/terms/\ndce: http://purl.org/dc/elements/1.1/\n", + """\ + classes: + Doc: + attributes: + title: {slot_uri: dc:title} + creator: {slot_uri: dce:creator} + """, + ) + }, + {DC_TERMS: "dcterms", DC_ELEMENTS: "dce"}, + ), + "case_only_alias": ( + { + "schema.yaml": _schema( + "case", + "FOAF: http://xmlns.com/foaf/0.1/\n", + """\ + classes: + Agent: + class_uri: FOAF:Agent + attributes: + nick: {slot_uri: FOAF:nick} + """, + ) + }, + {FOAF: "foaf"}, + ), + # ``time`` is a slot, so a JSON-LD context has a term of that name + "prefix_name_is_a_slot": ( + { + "schema.yaml": _schema( + "termclash", + "owltime: http://www.w3.org/2006/time#\n", + """\ + classes: + Interval: + class_uri: owltime:Interval + attributes: + begin: {slot_uri: owltime:hasBeginning} + time: {range: string} + """, + ) + }, + {TIME: "owltime"}, + ), + # ``dc`` is a slot with a class-scoped range + "scoped_slot_named_like_a_prefix": ( + { + "schema.yaml": _schema( + "scoped", + "dce: http://purl.org/dc/elements/1.1/\n", + """\ + slots: + dc: {range: string} + classes: + Doc: + slots: [dc] + slot_usage: + dc: {range: integer} + attributes: + title: {slot_uri: dce:title} + """, + ) + }, + {DC_ELEMENTS: "dce"}, + ), + # ``dc`` is a permissible value, which a ``@type: @vocab`` slot reads as a term first + "enum_value_named_like_a_prefix": ( + { + "schema.yaml": _schema( + "vocab", + "dce: http://purl.org/dc/elements/1.1/\n", + """\ + classes: + Doc: + attributes: + title: {slot_uri: dce:title} + vocabulary: {range: VocabularyEnum} + enums: + VocabularyEnum: + permissible_values: + dc: {meaning: dce:title} + other: {} + """, + ) + }, + {DC_ELEMENTS: "dce"}, + ), + "alias_as_default_prefix": ( + { + "schema.yaml": _schema( + "dflt", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Agent: + attributes: + id: {identifier: true, range: uriorcurie} + label: {range: string} + delegate: {range: Agent, slot_uri: w3prov:actedOnBehalfOf} + """, + default_prefix="w3prov", + ) + }, + {PROV: "prov"}, + ), + "alias_in_type_and_enum_meaning": ( + { + "schema.yaml": _schema( + "typeuri", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + types: + ProvTime: + typeof: string + uri: w3prov:InstantaneousEvent + classes: + Activity: + class_uri: w3prov:Activity + attributes: + started: {range: ProvTime} + role: {range: RoleEnum, slot_uri: w3prov:hadRole} + enums: + RoleEnum: + permissible_values: + Author: {meaning: w3prov:Role} + """, + ) + }, + {PROV: "prov"}, + ), + "prefixes_declared_in_an_import": ( + { + "module.yaml": _schema( + "module", + "mybibo: http://purl.org/ontology/bibo/\nw3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Book: + class_uri: mybibo:Book + attributes: + source: {slot_uri: w3prov:hadPrimarySource} + """, + ), + "schema.yaml": _schema( + "main", + "", + """\ + classes: + Thesis: + is_a: Book + class_uri: mybibo:Thesis + """, + imports="linkml:types module", + ), + }, + {BIBO: "bibo", PROV: "prov"}, + ), + # rdflib binds ``schema`` to https://schema.org/ by default; the schema's own declaration wins + "conventional_http_schema_org": ( + { + "schema.yaml": _schema( + "httpschema", + "schema: http://schema.org/\n", + """\ + classes: + Person: + class_uri: schema:Person + attributes: + full_name: {slot_uri: schema:name} + """, + ) + }, + {SCHEMA_ORG_HTTP: "schema"}, + ), +} + + +def _write(tmp_path: Path, case: str) -> str: + """Write the files of *case* to *tmp_path* and return the path of its main schema.""" + for name, text in CASES[case][0].items(): + (tmp_path / name).write_text(text, encoding="utf-8") + return str(tmp_path / "schema.yaml") + + +# ── Self-contained documents ────────────────────────────────────────── + + +def _rdf_prefixes(text: str, rdf_format: str) -> dict[str, str]: + return {str(p): str(n) for p, n in Graph(bind_namespaces="none").parse(data=text, format=rdf_format).namespaces()} + + +def _prefix_lines(text: str) -> dict[str, str]: + """The ``PREFIX name: `` declarations of ShExC and SPARQL.""" + return dict(re.findall(r"^PREFIX\s+([\w.-]*):\s*<([^>]*)>", text, flags=re.MULTILINE)) + + +def _expand_curies(node: Any, prefixes: dict[str, str]) -> Any: + """Expand every string of *node* whose prefix *prefixes* declares.""" + if isinstance(node, dict): + return {key: _expand_curies(value, prefixes) for key, value in node.items()} + if isinstance(node, list): + return [_expand_curies(value, prefixes) for value in node] + if isinstance(node, str): + prefix, colon, rest = node.partition(":") + if colon and prefix in prefixes and not rest.startswith("//"): + return prefixes[prefix] + rest + return node + + +def _sparql_meaning(text: str) -> dict[str, Any]: + """Each query of a SPARQL file as its parse tree, with prefixed names expanded by its own PREFIX lines. + + rdflib's query algebra binds one prefix per namespace, so it cannot read a file that declares + two names for one namespace, which SPARQL allows; the parse tree has the names as written. + """ + + def expand(node: Any, prefixes: dict[str, str]) -> Any: + if isinstance(node, CompValue): + if node.name == "pname": + return prefixes[node["prefix"] if "prefix" in node else ""] + node.get("localname", "") + return node.name, tuple((key, expand(value, prefixes)) for key, value in node.items()) + if isinstance(node, list | ParseResults): + return tuple(expand(value, prefixes) for value in node) + return str(node) + + meaning = {} + for name, query in SparqlGenerator.split_sparql(text).items(): + prologue, body = parseQuery(query) + prefixes = {(d["prefix"] if "prefix" in d else ""): str(d["iri"]) for d in prologue if d.name == "PrefixDecl"} + meaning[name] = expand(body, prefixes) + return meaning + + +def _sssom(text: str) -> tuple[dict[str, str], list[list[str]]]: + """The ``curie_map`` and the rows of an SSSOM TSV.""" + curie_map, rows, in_map = {}, [], False + for line in text.splitlines(): + if line.startswith("#curie_map:"): + in_map = True + elif in_map and line.startswith("# "): + name, _, iri = line[2:].partition(": ") + curie_map[name] = iri + elif not line.startswith("#"): + in_map = False + rows.append(line.split("\t")) + return curie_map, rows + + +@dataclass(frozen=True) +class Document: + """A self-contained output: how to generate it, read its prefixes, and compare its meaning.""" + + produce: Callable[[str, bool, Path], str] + prefixes: Callable[[str], dict[str, str]] + meaning: Callable[[str], Any] + same: Callable[[Any, Any], bool] = lambda a, b: a == b + + +def _rdf_document(generator: type, rdf_format: str, parse_format: str, **flags: Any) -> Document: + return Document( + produce=lambda path, normalize, _: generator( + path, normalize_prefixes=normalize, format=rdf_format, **flags + ).serialize(), + prefixes=lambda text: _rdf_prefixes(text, parse_format), + meaning=lambda text: Graph().parse(data=text, format=parse_format), + same=isomorphic, + ) + + +def _sssom_produce(path: str, normalize: bool, tmp_path: Path) -> str: + output = tmp_path / f"out-{normalize}.sssom.tsv" + SSSOMGenerator(path, output=str(output), normalize_prefixes=normalize).serialize() + return output.read_text(encoding="utf-8") + + +DOCUMENTS = { + "owl-ttl": _rdf_document(OwlSchemaGenerator, "ttl", "turtle"), + "owl-xml": _rdf_document(OwlSchemaGenerator, "xml", "xml"), + "shacl-ttl": _rdf_document(ShaclGenerator, "ttl", "turtle"), + "shex-rdf": _rdf_document(ShExGenerator, "rdf", "turtle"), + "shexc": Document( + produce=lambda path, normalize, _: ShExGenerator(path, normalize_prefixes=normalize).serialize(), + prefixes=_prefix_lines, + meaning=lambda text: json.loads(as_json(parse_shexc(text))), + ), + "sparql": Document( + produce=lambda path, normalize, _: SparqlGenerator(path, normalize_prefixes=normalize).serialize(), + prefixes=_prefix_lines, + meaning=_sparql_meaning, + ), + "yarrrml": Document( + produce=lambda path, normalize, _: YarrrmlGenerator(path, normalize_prefixes=normalize).serialize(), + prefixes=lambda text: yaml.safe_load(text)["prefixes"], + meaning=lambda text: _expand_curies(yaml.safe_load(text)["mappings"], yaml.safe_load(text)["prefixes"]), + ), + "sssom": Document( + produce=_sssom_produce, + prefixes=lambda text: _sssom(text)[0], + meaning=lambda text: _expand_curies(_sssom(text)[1], _sssom(text)[0]), + ), +} + +# Formats and options that change which prefixes an RDF output binds or prints. +RDF_VARIANTS = { + "owl-n3": _rdf_document(OwlSchemaGenerator, "n3", "n3"), + "owl-json-ld": _rdf_document(OwlSchemaGenerator, "json-ld", "json-ld"), + "owl-nt": _rdf_document(OwlSchemaGenerator, "nt", "nt"), + "owl-diff-stable": _rdf_document(OwlSchemaGenerator, "ttl", "turtle", diff_stable=True), + "owl-mapped-uris": _rdf_document(OwlSchemaGenerator, "ttl", "turtle", use_native_uris=False), + "owl-no-metaclasses": _rdf_document(OwlSchemaGenerator, "ttl", "turtle", metaclasses=False, type_objects=False), + "owl-metadata-profile-rdfs": _rdf_document(OwlSchemaGenerator, "ttl", "turtle", metadata_profile="rdfs"), + "shacl-diff-stable": _rdf_document(ShaclGenerator, "ttl", "turtle", diff_stable=True), + "shacl-annotations": _rdf_document(ShaclGenerator, "ttl", "turtle", include_annotations=True), +} + + +# gen-shex's RDF output reads the ShEx JSON-LD context, which the network marker serves from local files. +DOCUMENT_NAMES = [pytest.param(d, marks=pytest.mark.network) if d == "shex-rdf" else d for d in sorted(DOCUMENTS)] + + +@pytest.mark.parametrize("case", sorted(CASES)) +@pytest.mark.parametrize("document", DOCUMENT_NAMES) +def test_document_means_the_same(tmp_path, document, case): + """With the option a self-contained document denotes the same IRIs as without it.""" + doc = DOCUMENTS[document] + path = _write(tmp_path, case) + off, on = (doc.produce(path, normalize, tmp_path) for normalize in (False, True)) + assert doc.same(doc.meaning(off), doc.meaning(on)) + + +@pytest.mark.parametrize("case", sorted(CASES)) +@pytest.mark.parametrize("document", DOCUMENT_NAMES) +def test_document_writes_the_planned_names(tmp_path, document, case): + """Every generator writes the same name for a namespace, and no other name for it.""" + doc = DOCUMENTS[document] + path = _write(tmp_path, case) + planned = CASES[case][1] + off = doc.prefixes(doc.produce(path, False, tmp_path)) + on = doc.prefixes(doc.produce(path, True, tmp_path)) + for namespace, name in planned.items(): + names = {prefix for prefix, iri in on.items() if iri == namespace} + assert names <= {name}, f"{namespace} is written as {names}" + if namespace in off.values(): + assert on.get(name) == namespace + + +@pytest.mark.parametrize("variant", sorted(RDF_VARIANTS)) +@pytest.mark.parametrize("case", ["prov_alias", "conventional_http_schema_org", "prefixes_declared_in_an_import"]) +def test_rdf_variants(tmp_path, variant, case): + """Other formats and options of gen-owl and gen-shacl keep the graph and use the planned names.""" + doc = RDF_VARIANTS[variant] + path = _write(tmp_path, case) + off, on = (doc.produce(path, normalize, tmp_path) for normalize in (False, True)) + assert doc.same(doc.meaning(off), doc.meaning(on)) + for namespace, name in CASES[case][1].items(): + assert {p for p, iri in doc.prefixes(on).items() if iri == namespace} <= {name} + + +@pytest.mark.parametrize("generator", [OwlSchemaGenerator, ShaclGenerator]) +def test_no_rdflib_generated_names(tmp_path, generator): + """A declared conventional name wins over the name rdflib binds by default, so no ``schema1`` appears.""" + path = _write(tmp_path, "conventional_http_schema_org") + prefixes = _rdf_prefixes(generator(path, normalize_prefixes=True).serialize(), "turtle") + assert prefixes["schema"] == SCHEMA_ORG_HTTP + assert not [p for p in prefixes if re.fullmatch(r"schema\d+", p)] + + +# ── JSON-LD contexts and prefix maps ────────────────────────────────── + + +def _context(path: str, normalize: bool, **flags: Any) -> dict: + return json.loads(ContextGenerator(path, normalize_prefixes=normalize, **flags).serialize())["@context"] + + +def _context_prefixes(context: dict) -> dict[str, str]: + """The prefix definitions of a context: IRIs as strings or ``@prefix`` term definitions.""" + prefixes = {} + for key, value in context.items(): + if key.startswith("@") or ":" in key: + continue + if isinstance(value, str): + prefixes[key] = value + elif isinstance(value, dict) and value.get("@prefix") is True: + prefixes[key] = value["@id"] + return prefixes + + +def _expanded_terms(context: dict, parent: Context | None = None) -> set: + """The IRIs each term of a JSON-LD context expands to, without the term names. + + Uses rdflib's JSON-LD context processing; scoped contexts are expanded within their parent. + """ + ctx = Context(context) if parent is None else parent.subcontext(context) + expanded = {("@vocab", ctx.vocab)} + for name in context: + term = ctx.terms.get(name) + if term is not None: + scoped = frozenset(_expanded_terms(term.context, ctx)) if isinstance(term.context, dict) else None + expanded.add((term.id, term.type, str(term.container), term.reverse, scoped)) + return expanded + + +def _terms(context: dict) -> dict: + """The term definitions of a context, without its prefix definitions.""" + return { + key: value + for key, value in context.items() + if not key.startswith("@") and isinstance(value, dict) and value.get("@prefix") is not True + } + + +def _data_graph(context: Any, terms: dict, prefixes: list[str], vocabulary: list[str]) -> Graph: + """Parse, with *context*, a document that uses every term of *terms* (term definitions). + + It has one node per prefix of *prefixes*, identified by a compact IRI with that prefix. + IRI-valued terms get a compact IRI and a relative IRI, vocabulary-valued ones every name of + *vocabulary* (the permissible values, the only names valid data uses there), and the others a + string. Every term is also a type of every node. + """ + + def value(definition: dict, prefix: str) -> Any: + coercion = definition.get("@type") + if coercion == "@id": + return [f"{prefix}:object", "relative"] + if coercion == "@vocab": + return vocabulary + return "text" + + nodes = [ + {"@id": f"{prefix}:node", "@type": sorted(terms), **{t: value(d, prefix) for t, d in terms.items()}} + for prefix in prefixes + ] + document = {"@context": context, "@graph": nodes} + return Graph().parse(data=json.dumps(document), format="json-ld", base="http://base.test/") + + +def _schema_namespaces(path: str) -> dict[str, str]: + """The prefixes of a schema and its imports, with which LinkML expands the CURIEs of data.""" + sv = SchemaView(path) + sv.imports_closure() + return {str(prefix): str(namespace) for prefix, namespace in sv.namespaces().items()} + + +def _vocabulary(path: str) -> list[str]: + sv = SchemaView(path) + return sorted(text for enum in sv.all_enums().values() for text in enum.permissible_values) + + +CONTEXT_FLAGS = { + "default": {}, + "use-curies": {"use_curies": True}, + "flatprefixes": {"flatprefixes": True}, + "fix-multivalue-containers": {"fix_multivalue_containers": True}, + "no-mergeimports": {"mergeimports": False}, +} + + +@pytest.mark.parametrize("case", sorted(CASES)) +@pytest.mark.parametrize("flags", sorted(CONTEXT_FLAGS)) +def test_context_terms_expand_the_same(tmp_path, flags, case): + """Every term of the context expands to the IRIs it expands to without the option.""" + path = _write(tmp_path, case) + off, on = (_context(path, normalize, **CONTEXT_FLAGS[flags]) for normalize in (False, True)) + assert _expanded_terms(on) == _expanded_terms(off) + assert _terms(on).keys() == _terms(off).keys() + + +@pytest.mark.parametrize("case", sorted(CASES)) +@pytest.mark.parametrize("flags", sorted(CONTEXT_FLAGS)) +def test_context_keeps_declared_names_and_adds_planned_ones(tmp_path, flags, case): + path = _write(tmp_path, case) + off, on = (_context_prefixes(_context(path, normalize, **CONTEXT_FLAGS[flags])) for normalize in (False, True)) + assert off.items() <= on.items() + for namespace, name in CASES[case][1].items(): + if namespace in off.values(): + assert on.get(name) == namespace + + +@pytest.mark.parametrize("case", sorted(CASES)) +@pytest.mark.parametrize("use_curies", [False, True]) +def test_data_expands_the_same(tmp_path, case, use_curies): + """Data written with a prefix of the schema or its imports means the same under both contexts. Where + the default context leaves such a prefix out and the normalized one defines it, the data gets the + namespace the schema declares, with which LinkML expands it. Data written with the planned names + means the same as with the declared ones.""" + path = _write(tmp_path, case) + off, on = (_context(path, normalize, use_curies=use_curies) for normalize in (False, True)) + off_prefixes, on_prefixes = _context_prefixes(off), _context_prefixes(on) + schema_prefixes = _schema_namespaces(path) + terms, vocabulary = _terms(off), _vocabulary(path) + plan = preferred_prefixes(SchemaView(path)) + for prefix in sorted(off_prefixes.keys() | schema_prefixes.keys()): + expected_context = off + if prefix not in off_prefixes and prefix in on_prefixes: + assert on_prefixes[prefix] == schema_prefixes[prefix] + expected_context = [{prefix: schema_prefixes[prefix]}, off] + expected = _data_graph(expected_context, terms, [prefix], vocabulary) + assert isomorphic(_data_graph(on, terms, [prefix], vocabulary), expected), prefix + planned = plan.standardize_prefix(prefix) or prefix + assert isomorphic(_data_graph(on, terms, [planned], vocabulary), expected), planned + + +@pytest.mark.parametrize("case", ["prov_alias", "prefix_name_is_a_slot", "prefixes_declared_in_an_import"]) +@pytest.mark.parametrize("prefixes_normalized", [False, True]) +@pytest.mark.parametrize("model_normalized", [False, True]) +def test_prefix_and_model_contexts_combine(tmp_path, case, prefixes_normalized, model_normalized): + """A prefix-only context (``--no-model``) and a model-only context (``--no-prefixes``) combine to the + same meaning, whichever of them was normalized.""" + path = _write(tmp_path, case) + full = _context(path, False) + combined = [ + _context(path, prefixes_normalized, model=False), + _context(path, model_normalized, prefixes=False), + ] + declared = sorted(_context_prefixes(full)) + vocabulary = _vocabulary(path) + expected = _data_graph(full, _terms(full), declared, vocabulary) + assert isomorphic(_data_graph(combined, _terms(full), declared, vocabulary), expected) + + +@pytest.mark.parametrize("case", ["prov_alias", "prefixes_declared_in_an_import"]) +def test_prefix_map_combines_with_a_model_only_context(tmp_path, case): + """The prefixes of ``--no-prefixes`` come from gen-prefix-map, whose names the context then keeps.""" + path = _write(tmp_path, case) + full = _context(path, False) + prefix_map = json.loads(PrefixGenerator(path).serialize()) + combined = [prefix_map, _context(path, True, prefixes=False)] + declared = sorted(_context_prefixes(full)) + expected = _data_graph(full, _terms(full), declared, _vocabulary(path)) + assert isomorphic(_data_graph(combined, _terms(full), declared, _vocabulary(path)), expected) + + +def test_no_prefixes_context_is_unchanged(tmp_path): + """Without prefix definitions a context writes compact IRIs only with the names other contexts define.""" + path = _write(tmp_path, "prov_alias") + assert _context(path, True, prefixes=False) == _context(path, False, prefixes=False) + + +@pytest.mark.parametrize("base_normalized", [False, True]) +@pytest.mark.parametrize("extension_normalized", [False, True]) +@pytest.mark.parametrize("exclude", ["exclude_imports", "exclude_external_imports"]) +def test_contexts_of_an_import_and_its_importer_combine(tmp_path, base_normalized, extension_normalized, exclude): + """The context of an importing schema, without the imported elements, combines with the imported + schema's context, whichever was normalized; the importer's ``time`` slot shares the name ``time``.""" + (tmp_path / "base.yaml").write_text( + _schema( + "base", + "owltime: http://www.w3.org/2006/time#\n", + """\ + classes: + Interval: + class_uri: owltime:Interval + attributes: + begin: {slot_uri: owltime:hasBeginning} + """, + ), + encoding="utf-8", + ) + (tmp_path / "schema.yaml").write_text( + _schema( + "extension", + "owltime: http://www.w3.org/2006/time#\n", + """\ + classes: + Meeting: + is_a: Interval + attributes: + time: {range: string} + """, + imports="linkml:types base", + ), + encoding="utf-8", + ) + base, extension = str(tmp_path / "base.yaml"), str(tmp_path / "schema.yaml") + expected_contexts = [_context(base, False), _context(extension, False, **{exclude: True})] + contexts = [_context(base, base_normalized), _context(extension, extension_normalized, **{exclude: True})] + terms = _terms(expected_contexts[0]) | _terms(expected_contexts[1]) + declared = sorted(_context_prefixes(expected_contexts[0]) | _context_prefixes(expected_contexts[1])) + expected = _data_graph(expected_contexts, terms, declared, []) + assert isomorphic(_data_graph(contexts, terms, declared, []), expected) + + +@pytest.mark.parametrize("embed", [False, True]) +def test_frame_type_expands_the_same(tmp_path, embed): + """The frame's ``@type`` is written with the planned name and denotes the same class.""" + (tmp_path / "schema.yaml").write_text( + _schema( + "frame", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Activity: + tree_root: true + class_uri: w3prov:Activity + attributes: + association: {range: Association, inlined: true, slot_uri: w3prov:qualifiedAssociation} + Association: + class_uri: w3prov:Association + attributes: + role: {slot_uri: w3prov:hadRole} + """, + ), + encoding="utf-8", + ) + path = str(tmp_path / "schema.yaml") + frames = [] + for normalize in (False, True): + output = tmp_path / f"out-{normalize}.context.jsonld" + gen = ContextGenerator(path, normalize_prefixes=normalize, emit_frame=True, embed_context_in_frame=embed) + gen.serialize(output=str(output)) + frame = json.loads(output.with_suffix(".frame.jsonld").read_text(encoding="utf-8")) + context = frame["@context"] if embed else json.loads(output.read_text(encoding="utf-8"))["@context"] + frames.append((frame, Context(context))) + (off, off_ctx), (on, on_ctx) = frames + assert on["@type"] == "prov:Activity" + assert on_ctx.expand(on["@type"]) == off_ctx.expand(off["@type"]) == PROV + "Activity" + assert on.keys() == off.keys() + + +@pytest.mark.parametrize("generator", [ContextGenerator, ShExGenerator]) +def test_yaml_text_input_with_local_imports(tmp_path, generator): + """A schema given as YAML text resolves its local imports against ``base_dir``, as without the option.""" + path = Path(_write(tmp_path, "prefixes_declared_in_an_import")) + from_text = generator(path.read_text(encoding="utf-8"), base_dir=str(tmp_path), normalize_prefixes=True) + assert from_text.serialize() == generator(str(path), normalize_prefixes=True).serialize() + + +def test_sssom_keeps_a_name_it_writes_undeclared(tmp_path): + """gen-sssom writes ``skos:exactMatch`` without declaring ``skos``, so a declared ``skosx`` for SKOS keeps + its name there, while documents that declare what they write use ``skos``.""" + (tmp_path / "schema.yaml").write_text( + _schema( + "skos", + "skosx: http://www.w3.org/2004/02/skos/core#\n", + """\ + classes: + Topic: + exact_mappings: [skosx:Concept] + """, + ), + encoding="utf-8", + ) + path = str(tmp_path / "schema.yaml") + sssom = DOCUMENTS["sssom"] + off, on = (sssom.produce(path, normalize, tmp_path) for normalize in (False, True)) + assert sssom.same(sssom.meaning(off), sssom.meaning(on)) + assert sssom.prefixes(on)["skosx"] == SKOS + assert _rdf_prefixes(OwlSchemaGenerator(path, normalize_prefixes=True).serialize(), "turtle")["skos"] == SKOS + + +def test_sssom_rewrites_only_curie_columns(tmp_path): + (tmp_path / "schema.yaml").write_text( + _schema( + "label", + "w3prov: http://www.w3.org/ns/prov#\n", + """\ + classes: + Agent: + title: "w3prov:Agent as used in the old feed" + class_uri: w3prov:Agent + exact_mappings: [w3prov:Person] + """, + ), + encoding="utf-8", + ) + _, rows = _sssom(_sssom_produce(str(tmp_path / "schema.yaml"), True, tmp_path)) + assert ["prov:Agent", "w3prov:Agent as used in the old feed", "skos:exactMatch", "", "prov:Person"] == rows[1][:5] + + +# ── The plan ────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("case", sorted(CASES)) +def test_plan(tmp_path, case): + """One plan per schema gives each namespace its name.""" + plan = preferred_prefixes(SchemaView(_write(tmp_path, case))) + assert {ns: plan.reverse_prefix_map[ns] for ns in CASES[case][1]} == CASES[case][1] + + +def test_plan_warns_when_a_prefix_keeps_its_name(tmp_path, caplog): + with caplog.at_level(logging.WARNING): + preferred_prefixes(SchemaView(_write(tmp_path, "http_and_https_schema_org"))) + assert "Prefix 'sdo' is not renamed to 'schema'" in caplog.text + assert "Prefix 'sdohttp' is not renamed" not in caplog.text + + +@pytest.mark.parametrize( + "prefixes,taken,expected", + [ + ({"sdo": SCHEMA_ORG}, set(), {"sdo": "schema"}), + ({"schema": SCHEMA_ORG}, set(), {}), + ({"ex": "http://example.org/"}, set(), {}), + ({"sdo": SCHEMA_ORG, "schema": SCHEMA_ORG}, set(), {"sdo": "schema"}), + ({"sdo": SCHEMA_ORG, "sdohttp": SCHEMA_ORG_HTTP}, set(), {}), + ({"sdo": SCHEMA_ORG, "schema": "http://example.org/s/"}, set(), {}), + ({"sdo": SCHEMA_ORG}, {"schema"}, {}), + ({"dc": DC_TERMS, "dce": DC_ELEMENTS}, set(), {"dc": "dcterms"}), + ({"FOAF": FOAF}, set(), {"FOAF": "foaf"}), + ], + ids=[ + "alias", + "conventional", + "unknown", + "alias_and_conventional", + "two_namespaces", + "name_declared_elsewhere", + "name_taken", + "declared_name_stays", + "case_only", + ], +) +def test_prefix_renames(prefixes, taken, expected): + assert prefix_renames(prefixes, taken) == expected + + +def test_document_prefixes_rename_only_declared_names(): + """A document renames the prefixes it declares; a CURIE with an undeclared prefix keeps its meaning.""" + plan = Converter( + [ + Record(prefix="schema", uri_prefix=SCHEMA_ORG, prefix_synonyms=["sdo"]), + Record(prefix="bibo", uri_prefix=BIBO, prefix_synonyms=["mybibo"]), + ] + ) + document = document_prefixes({"sdo": SCHEMA_ORG, "ex": "http://example.org/"}, plan) + assert {r.prefix: r.uri_prefix for r in document.records} == {"schema": SCHEMA_ORG, "ex": "http://example.org/"} + assert document.standardize_prefix("mybibo") is None + + +def test_standardize_prefix_leaves_absolute_iris_alone(): + """A suffix starting with ``//`` makes a value an absolute IRI, not a compact one (JSON-LD 1.1 §4.1.5).""" + converter = Converter([Record(prefix="web", uri_prefix="http://example.org/", prefix_synonyms=["http"])]) + assert standardize_prefix("http://schema.org/Person", converter) == "http://schema.org/Person" + assert standardize_prefix("http:Person", converter) == "web:Person" + + +def test_added_preferred_prefixes_skip_terms_and_other_namespaces(): + plan = Converter([Record(prefix="schema", uri_prefix=SCHEMA_ORG, prefix_synonyms=["sdo"])]) + assert not added_preferred_prefixes(plan, {"sdo": SCHEMA_ORG}, terms={"schema"}).records + assert not added_preferred_prefixes(plan, {"sdo": SCHEMA_ORG, "schema": SCHEMA_ORG_HTTP}).records + + +# ── Where the names come from ───────────────────────────────────────── + + +@pytest.mark.parametrize( + "namespace,expected", + [ + ("http://www.opengis.net/ont/geosparql#", "geo"), + (SCHEMA_ORG, "schema"), + ("https://www.w3.org/2003/01/geo/wgs84_pos#", "wgs"), + (SCHEMA_ORG_HTTP, "schema"), + ("http://www.w3.org/ns/shacl#", "sh"), + (DC_ELEMENTS, "dc"), + ], +) +def test_well_known_prefix_map_names(namespace, expected): + assert well_known_prefix_map()[namespace] == expected + + +def test_well_known_prefix_map_uses_curated_sources_only(): + """The names are those of prefixmaps' curated ``linked_data`` context and of rdflib, which wins; + no name of a crowd-sourced registry, such as prefix.cc's ``alice`` for http://example.org/.""" + curated = {rec.namespace: rec.prefix for rec in load_context("linked_data").prefix_expansions} + rdflib_map = {str(ns): str(pfx) for pfx, ns in Graph(bind_namespaces="rdflib").namespaces() if str(pfx)} + wk = well_known_prefix_map() + assert wk == {**curated, **rdflib_map} + assert "http://example.org/" not in wk + + +def test_well_known_prefix_map_returns_a_copy(): + well_known_prefix_map()["http://never-in-any-real-prefix-map.test/"] = "test" + assert "http://never-in-any-real-prefix-map.test/" not in well_known_prefix_map() + + +# ── Command line ────────────────────────────────────────────────────── + +OFFERING = ["owl", "shacl", "shex", "sparql", "yarrrml", "sssom", "jsonld-context"] + + +@pytest.mark.parametrize("command", OFFERING) +def test_cli_applies_the_option(command, tmp_path): + path = _write(tmp_path, "prov_alias") + out = tmp_path / "out" + extra = {"sssom": ["-o", str(out)], "sparql": ["-d", str(out)]}.get(command, []) + runner = CliRunner() + outputs = {} + for flag in ("--normalize-prefixes", "--no-normalize-prefixes", None): + args = ["generate", command, *extra, *([flag] if flag else []), path] + result = runner.invoke(linkml_cli, args) + assert result.exit_code == 0, result.output + if command == "sssom": + outputs[flag] = out.read_text(encoding="utf-8") + elif command == "sparql": + outputs[flag] = "".join(f.read_text(encoding="utf-8") for f in sorted(out.glob("*.rq"))) + else: + outputs[flag] = result.output + assert outputs["--no-normalize-prefixes"] == outputs[None] + assert re.search(r"\bprov:\w", outputs["--normalize-prefixes"]) + assert not re.search(r"\bprov:\w", outputs[None]) + + +@pytest.mark.parametrize("command", ["prefix-map", "rdf", "jsonld", "python", "pydantic", "json-schema"]) +def test_cli_rejects_the_option_where_names_are_data_or_api(command, tmp_path): + """A prefix map gives each namespace one name, the schema's; gen-rdf and gen-jsonld write CURIEs as + literal data; gen-python and gen-pydantic make prefix names part of the generated code; gen-json-schema + writes none.""" + path = _write(tmp_path, "prov_alias") + result = CliRunner().invoke(linkml_cli, ["generate", command, "--normalize-prefixes", path]) + assert result.exit_code == 2 and "No such option" in result.output