Repository navigation
[HWORKS-3744] Ontology factory: auto-generated ontology and text-to-SQL over data sources - #788
Open
jimdowling wants to merge 4 commits into
Open
jimdowling wants to merge 4 commits into
jimdowling wants to merge 4 commits into
Conversation
…QL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 The built-in ontology factory builds, per system, an ontology of the tables in a project's SQL data sources and a text-to-SQL app over it: a custom Python app in the new ontology-pipeline environment, five jobs, two online feature groups with embedding indexes, and two vLLM deployments the project's ontology systems share, or OpenAI-compatible endpoints instead. hopsworks-ee adds the factory and identity signing for custom apps, hopsworks-api the hops-ontology skill, hops ontology and the MCP tools, docker-images and hopsworks-helm the environment, and hopsworks-front the factory's menu entry. The Clients page gets a section on the ontology factory after the dlt ingestion factory: what a system builds, what it needs (sources with SHARED credentials, a time-sliced GPU or external endpoints, and hopsable_proxy_enabled for the CLI and MCP), the form and the build phases, the sales example with its expected answers, the roles in the app, its pages, hops ontology and the MCP tools, the jobs and their arguments, and delete. The From Template menu and the list of built-in factories include it, and the CLI page, the environments list and the Analytics landing link to it. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…QL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 A system the ontology factory builds compiles its semantic layer: its ontology.yaml switches compilation on, and the build's compile run and the scheduled ones no longer skip. docs: the ontology section says a new system starts with semantic compilation on and with its PII and distinct-value answers as Settings, and that an edit in the app is re-embedded at once. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…io into HWORKS-3744
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The documentation incorrectly promises permanent source removal and broad embedding retries, omitting empty-source reseeding behavior and the jobs’ limited scope of terms and column attributes.
2 open issues
What changed in this PR
This PR documents the ontology factory and connects its build-and-use guide to the existing Hopsworks client, Analytics, and Python environment documentation.
Changes:
- Adds ontology setup, model requirements, app usage, CLI/MCP access, jobs, and cleanup instructions.
- Documents the sales example and adds ontology to the built-in factory list.
- Links the guide from the CLI, Analytics, and environment pages.
| File | Description |
|---|---|
docs/user_guides/projects/python/python_env_overview.md |
Adds the ontology-pipeline environment. |
docs/user_guides/client_installation/index.md |
Adds the ontology guide and updates factory listings. |
docs/user_guides/client_installation/cli.md |
Introduces hops ontology and links to the guide. |
docs/user_guides/analytics/index.md |
Adds an ontology and text-to-SQL entry. |
Review effort: Balanced
| **Add Source** takes a data source's name, optional schemas and an optional table filter, connects to the source once, and starts the ingest job for it. | ||
| It refuses, with HTTP 422, a source of an unsupported type, one with PROVIDED credentials, and one that reads a database another source already reads, and, with HTTP 502, one it cannot reach. | ||
| **Remove** starts the reset job, which deletes what was ingested from that source: its catalog, its part of the ontology, and their vectors. | ||
| The `sources` of `ontology.yaml` seed a new system only: once the app has a source, **Sources** is the record, so a source removed there stays removed. |
| **Semantic Compilation** has **Enable Semantic Compilation**: the compile job compiles only while it is on, and switching it on starts a compile run. | ||
| A new system starts with it on, from `semantic_compilation: true` in its `ontology.yaml`; the `pii_detection` and `distinct_value_probing` answers set **PII Settings** and **Semantic Input** the same way, and a later change in **Settings** stays. | ||
| **Reset** there deletes the ontology of every database and compiles it again. | ||
| An edit in the app re-embeds the edited object, so it shows in the catalog and in the vector search at once; an edit whose embedding fails is embedded again by the next compile or embed run. |
…QL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 An ontology system's app and jobs now run in ontology-<commit>, a clone of python-feature-pipeline with Auto-Ontology installed, instead of an ontology-pipeline base environment. The ontology guide names the environment, its build phase and that deleting a system keeps it, and the environments page no longer lists ontology-pipeline. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs for the
ontologyfactory, which ports NVIDIA Auto-Ontology into Hopsworks.ontology-pipeline) and the Analytics landing.Tests
Cluster verification
Not run. There is no reachable cluster with a GPU:
jim-ontologyhas no GPU workers, and the GPU cluster offered for this ticket (jim-qwen38) answereddial tcp 10.117.180.10:6443: connect: no route to hoston 2026-11-09. Not covered yet:ontology-pipelineimage, and deploying ee, helm, front and the image on a cluster.loadtest/tests/workflows/ontology/(markerse2e_platformande2e_ontology).find_neighbors, and signed identity through the app proxy.hops factory run ontology --preset wwi-exampleend to end.The branch merges the current upstream default branch.
Related PRs
JIRA: https://hopsworks.atlassian.net/browse/HWORKS-3744
🤖 Generated with Claude Code