From 9e1de81a8eed4bd03b38217720ca31fba5cf6c29 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 9 Nov 2026 08:12:37 +0100 Subject: [PATCH 1/3] [HWORKS-3744] Ontology factory: auto-generated ontology and text-to-SQL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 The built-in ontology factory builds, per system, an ontology of the tables in a project's SQL data sources and a text-to-SQL app over it: a custom Python app in the new ontology-pipeline environment, five jobs, two online feature groups with embedding indexes, and two vLLM deployments the project's ontology systems share, or OpenAI-compatible endpoints instead. hopsworks-ee adds the factory and identity signing for custom apps, hopsworks-api the hops-ontology skill, hops ontology and the MCP tools, docker-images and hopsworks-helm the environment, and hopsworks-front the factory's menu entry. The Clients page gets a section on the ontology factory after the dlt ingestion factory: what a system builds, what it needs (sources with SHARED credentials, a time-sliced GPU or external endpoints, and hopsable_proxy_enabled for the CLI and MCP), the form and the build phases, the sales example with its expected answers, the roles in the app, its pages, hops ontology and the MCP tools, the jobs and their arguments, and delete. The From Template menu and the list of built-in factories include it, and the CLI page, the environments list and the Analytics landing link to it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/analytics/index.md | 2 + docs/user_guides/client_installation/cli.md | 1 + docs/user_guides/client_installation/index.md | 206 +++++++++++++++++- .../projects/python/python_env_overview.md | 1 + 4 files changed, 208 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/analytics/index.md b/docs/user_guides/analytics/index.md index 862ad401ba..f10052697b 100644 --- a/docs/user_guides/analytics/index.md +++ b/docs/user_guides/analytics/index.md @@ -38,6 +38,8 @@ Start with a query, then turn it into a dashboard. Expose a data source as a catalog so external tables join feature data. - [External BI tools](../../concepts/mlops/bi_tools.md) How a BI tool connects to the offline store through Trino. +- [Ontology and text-to-SQL][build-an-ontology-and-text-to-sql-app] + Ask questions about a SQL data source's tables in plain language and get the answer with its SQL. diff --git a/docs/user_guides/client_installation/cli.md b/docs/user_guides/client_installation/cli.md index c1434dbb04..7c3466422e 100644 --- a/docs/user_guides/client_installation/cli.md +++ b/docs/user_guides/client_installation/cli.md @@ -47,6 +47,7 @@ Inside the [project terminal][terminal] no login is needed, `hops` is pointed at Every asset type has a subcommand: `project`, `fg`, `fv`, `td`, `model`, `deployment`, `job`, `datasource`, `sql`. `hops --help` lists the verbs. +`hops ontology` asks an [ontology system][build-an-ontology-and-text-to-sql-app] questions about the data in its sources. ```bash hops project use fraud_detection diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index a594849b85..efb67be8c5 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -159,7 +159,7 @@ It writes your answers into the draft, which the form shows as it changes, and n What you type in the form goes to Claude Code with your next message. Say in the chat that you want to create the system, and Claude Code checks the required answers and the page creates it, as **Create** does; **Back** and **Create** are at the top right of the page. The form's chat tab in the Terminal closes once the system is created. -**From Template** lists the factories without the description: **ML System** holds **Batch ML system**, **Real-time ML system**, **Batch and streaming ML system** and **Agentic system**, **Analytics** the analytics layer factories, and each has a **Blueprints** submenu of its examples, each opening its factory's form filled in. +**From Template** lists the factories without the description: **ML System** holds **Batch ML system**, **Real-time ML system**, **Batch and streaming ML system**, **Agentic system** and **Ontology and text-to-SQL**, **Analytics** the analytics layer factories, and each has a **Blueprints** submenu of its examples, each opening its factory's form filled in. The cogwheel opens **Manage factories**. **Existing Systems**, shown once a factory has built a system, lists the project's systems for every member: each with its factory, type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a Git repository; the box above it filters the list. **Login to GitHub** runs `github-login` in a Terminal tab; the page shows whether the terminal's GitHub CLI is logged in, which the build needs to create the repository. @@ -365,11 +365,213 @@ ingestion: The chat stays open while it builds: it shows the rows each table loaded, with the dropped and pseudonymized columns as they are, and asks whether they are right before it schedules the job, builds the dashboard and verifies a run against the source's row counts. +### Build an ontology and text-to-SQL app + +The `ontology` factory builds an ontology of the tables in the project's SQL data sources, and an app that answers questions about their data by writing and running SQL. +Its jobs read the sources' metadata (schemas, tables, columns, keys and views) into a catalog, and an LLM compiles the catalog into business terms: concepts such as Customer or Invoice Line, with descriptions and synonyms, mapped to tables, columns, joins and SQL expressions. +Members review and edit the terms in the app, and its chat answers a question with the SQL it ran and the rows it returned. + +**From Template > ML System > Ontology and text-to-SQL** opens the factory's form, and **From Template > ML System > Blueprints > Example: sales database** opens it filled in for the sales example. +From a terminal: + +```bash +hops factory run ontology # answer the form's questions in the terminal +hops factory run ontology --preset wwi-example # the sales example, named sales-ontology +``` + +Both start Claude Code on `/hops-factory-ontology ` in the system's directory, which builds the system phase by phase. +A project can hold several ontology systems, and each one builds: + +| Part | Name | +| --- | --- | +| App: the HTTP API, the web UI and the chat | ``, a custom app in the `ontology-pipeline` environment | +| App tables: the catalog, the ontology, conversations and settings | `
`, in the project's online database | +| Jobs, in the `ontology-pipeline` environment | `-ingest`, `-compile`, `-embed`, `-reset`, `-import` | +| Online feature groups, each with an embedding index | `_data_objects` (tables and columns), `_semantic` (the ontology) | +| Models in the Model Registry, imported from Hugging Face | `Nemotron_3_5_Embed_1B_BF16`, `NVIDIA_Nemotron_3_5_Lightning_30B_A3B_FP8` | +| vLLM deployments | `nemotronembed`, `nemotronllm` | + +`` is the name with hyphens replaced by underscores, and `
` is the app's table prefix. +The models and the deployments are shared by the project's ontology systems: a second system reuses those the first one made, and a system with an external model makes neither that model nor its deployment. +Each embedding index holds vectors of the dimension chosen in the form, compared by cosine similarity. +The system's directory holds `ontology.yaml`, the system's configuration and the only file written from the answers, and the launchers of the app (`app.py`) and of the jobs (`jobs/`), which run the Auto-Ontology runtime that the `ontology-pipeline` environment ships. + +#### Before you build an ontology + +- **Data sources**: each source is a data source of the project with SHARED credentials, of type MySQL, PostgreSQL or ClickHouse ([SQL][how-to-set-up-an-sql-data-source]), [Snowflake][how-to-set-up-a-snowflake-data-source] or Databricks ([Unity Catalog][how-to-set-up-a-unity-catalog-data-source]). + The build refuses a source with PROVIDED credentials, because those answer with one member's credentials while the app and the jobs serve every member. + Two sources cannot read the same database. +- **A GPU, for local models**: the LLM and the embedding model run as two vLLM deployments that share one GPU, each requesting one `nvidia.com/gpu`. + This needs the NVIDIA device plugin on the GPU node configured for time-slicing with `replicas: 2` and `renameByDefault: false`; neither the Helm chart nor Hopsworks configures it. + The LLM takes 80% of the GPU's memory and the embedding model 12%, and the LLM's checkpoint, which the build downloads from Hugging Face, is about 32 GB. + A Hugging Face token is needed only when Hugging Face asks for a login or a license acceptance. +- **External models, instead of a GPU**: an OpenAI-compatible endpoint for the LLM, the embedding model, or both, set in the form's **Models** section. + Their URLs, model names and keys are saved as your [account environment variables][account-level-environment-variables] `LLM_URL`, `LLM_MODEL`, `LLM_API_KEY`, `EMBEDDING_URL`, `EMBEDDING_MODEL` and `EMBEDDING_API_KEY`, never in `system.yaml`. + Hopsworks sets a member's account variables in every job and app that member starts, and a scheduled run starts as its job's creator. + The embeddings endpoint must return vectors of the dimension chosen in the form. +- **`hopsable_proxy_enabled`, for the CLI and MCP**: `hops ontology` and the MCP tools reach the app with a proxy token, which Hopsworks mints only when the cluster variable `hopsable_proxy_enabled` is `true` (`hopsworks.variables.hopsable_proxy_enabled` in the Helm values, `false` by default); otherwise it refuses the token with error code `120012`. + The app's pages in Hopsworks do not need it. +- **A Data owner**: a Data owner builds the system: the sales example creates tables, which a Data scientist cannot, and deleting a system's assets needs a Data owner. + +#### The ontology form + +- **Basics**: **Name**, which is also the system's directory, the app's name, the prefix of its jobs, feature groups and tables, and, as `hops-`, its new GitHub repository; **Create new GitHub repo**; and **Who asks questions, and about what?**, such as "sales managers ask about orders, customers and stock levels", from which the build draws its test questions. +- **Data sources**: one entry per source, with **Data source name**, **Schemas**, an optional comma-separated allowlist (empty reads every schema the credentials can read), and **Table name pattern**, an optional regular expression a table's name must match. +- **Models**, collapsed: **LLM**, Nemotron 3.5 Lightning 30B (FP8) served in the project with vLLM (the default) or an OpenAI-compatible endpoint; **Embedding model**, Nemotron 3.5 Embed 1B served in the project with vLLM (the default) or an OpenAI-compatible endpoint; **Embedding dimension**, 2048 (the default) or 1024, which halves the vector index's memory; the endpoints, model names and keys of external models; and the **Hugging Face token**. +- **Schedule**, collapsed: when ingest and compile run, **Every day** (the default, at 02:00 and 03:00 UTC), **Every week** (Mondays at the same times) or **Only when started by hand or from the app**; and **Detect personal data**, off by default, which tags the columns that hold personal data and leaves their sample values out of the catalog and the embeddings. + +The build runs these phases: + +| Phase | What the build does | +| --- | --- | +| Check the sources and write ontology.yaml | Checks that every source exists, has a supported type and SHARED credentials, and that an external model's account variables are set, then writes `ontology.yaml` | +| Import the models and deploy them with vLLM | Imports each local model from Hugging Face at a pinned revision and deploys it, or reuses what another ontology system made; skipped when both models are external | +| Ontology app | Creates and starts the app, which creates its tables | +| Jobs, and ingest the sources' metadata | Creates the five jobs, schedules ingest and compile, and runs ingest | +| Compile the ontology | Runs compile and embed, then checks that every table has a term, except a table whose columns are all foreign keys, and that every ontology row has its vector in `_semantic` | +| Verify | Asks three questions drawn from **Who asks questions, and about what?**, or the sales example's five, records each answer, its SQL and its time in `system.yaml`, and runs `hops factory system status` | + +Compiling sends many LLM calls: on one GPU it takes tens of minutes for a few dozen tables. + +`--preset wwi-example`, or **Blueprints > Example: sales database**, builds the system `sales-ontology` over generated data. +Its first phase runs a script that creates six tables, `sales_state_provinces`, `sales_cities`, `sales_customers`, `sales_products`, `sales_orders` and `sales_order_lines`, in the project's online database, fills them from a fixed seed, and creates the MySQL data source `sales_mysql` over them with SHARED credentials, the system's only source. +The tables carry the `sales_` prefix because a member can create tables only in the project's online database, not in a database of their own. +Its last phase asks five questions and compares each answer with the one the data gives: + +| Question | Answer | +| --- | --- | +| How many orders were placed in March 2025? | 14 | +| What was the total revenue of delivered orders? | 107467.00 | +| Which state has the most customers? | Georgia, with 7 | +| Which products have fewer than 20 units in stock? | Build your own robot kit, Office desk toy, Ride on toy sedan car | +| Which salesperson brought in the most revenue from delivered orders? | Archer Lamble, with 33722.00 | + +#### Roles in the ontology app + +The Hopsworks app proxy signs the caller's name into every request, and the app reads the caller's role from the project's members, cached for 60 seconds. + +| Project role | In the app | +| --- | --- | +| Data owner | Everything: chat, edits to the catalog and the ontology, saved analyses, **Analytics** and **Settings** | +| Data scientist | Chat, edits to the catalog and the ontology, and saved analyses | +| Observer, Feature store restricted | The catalog, the terms, tags and saved analyses, read-only and without sample values; no chat | +| Not a member | Refused | + +Observers and Feature store restricted members get no chat because the chat runs SQL on the data sources, and no sample values because they cannot preview data sources in Hopsworks. +Only Data owners and Data scientists can mint the proxy token, so only they can use `hops ontology` and the MCP tools; an API key needs the `JOB` scope. +The app, and the jobs it starts, run as the member who last started or redeployed the app. + +#### Use an ontology + +The app is the system's **App** view on its Factory page, served at `/hopsworks-api/pythonapp///`; `hops app url ` prints its address, and **Start app** starts the system's deployments and then the app. +Its pages: + +- **Chat**: ask a question in plain language; the answer shows the SQL that ran, the rows and, when the result has one, a chart. + Conversations are saved per member, and a follow-up question resolves against its conversation's earlier questions. +- **Discovery**: search the catalog and the ontology. + Members without chat open the app here. +- **Terms**: the business terms with their descriptions, synonyms, columns and SQL expressions, to review and edit. +- **Analysis**: saved SQL analyses, which the chat uses as examples. +- **Exploration**: the graph of the terms and the joins between them. +- **Data**: the catalog of databases, schemas, tables and columns. +- **Analytics**, for Data owners: the questions asked in the last 30 days, and who asked them. +- **Settings**, for Data owners: **Sources**, **Tags**, **Rules**, **PII Settings**, **Semantic Input**, **Semantic Compilation**, **Agent Settings** and **Import / Export**. + +**Sources** lists the data sources the ontology reads, with their schemas and table filter. +**Add Source** takes a data source's name, optional schemas and an optional table filter, connects to the source once, and starts the ingest job for it. +It refuses, with HTTP 422, a source of an unsupported type, one with PROVIDED credentials, and one that reads a database another source already reads, and, with HTTP 502, one it cannot reach. +**Remove** starts the reset job, which deletes what was ingested from that source: its catalog, its part of the ontology, and their vectors. +The `sources` of `ontology.yaml` seed a new system only: once the app has a source, **Sources** is the record, so a source removed there stays removed. + +**Semantic Compilation** has **Enable Semantic Compilation**: the compile job compiles only while it is on, and switching it on starts a compile run. +**Reset** there deletes the ontology of every database and compiles it again. +An edit in the app shows in the catalog at once, and in the vector search after the next compile or embed run. +**Import / Export** exports the catalog and the ontology as YAML, in the native format or Apache Ossie's, and imports such a file. + +The app answers at most four questions at a time; past that it answers HTTP 429 with `Retry-After`. +The chat runs only single `SELECT` or `WITH` statements, with the source's credentials, and on MySQL, PostgreSQL and ClickHouse in a read-only session. + +From a terminal: + +```bash +hops ontology status sales-ontology # can it answer, and what stops it; exits 1 when it cannot +hops ontology ask sales-ontology "Which state has the most customers?" # the answer, its SQL and up to 100 rows +hops ontology ask sales-ontology "How many orders did they place?" --conversation-id # a follow-up in a saved conversation +``` + +`hops ontology ask` prints each step the agent takes as it takes it, then the answer, its SQL and up to 100 rows, and says how many rows the query returned when it returned more. +With `--json` it prints `answer`, `sql`, `rows`, `row_count`, `truncated`, `reasoning` and `chart` instead. +Without `--conversation-id` a question is answered on its own: no history, nothing saved and no chart. +It waits 900 seconds for the answer, which `--timeout` changes. + +`hops ontology status` prints the app's state, whether it is serving, its URL, whether the ontology is compiled, whether a source is configured to run SQL on, whether a database is ingested, and the ingested databases, then a `blocker:` line for each thing that stops the system from answering: + +- **The app is ``; start it with `hops app start `**: read `hops app logs ` when it does not start. +- **No database is ingested yet**: run `hops job run -ingest --wait`; its log names the source that failed. +- **The ontology is not compiled yet**: switch on **Enable Semantic Compilation** in the app's **Settings**, which starts a compile run, or run the compile job with `--force`. +- **No data source is configured**: add one with **Add Source** on the app's **Sources** page. + +When an answer comes back empty, `hops ontology status` separates a system that is not ready from a question the ontology does not cover. + +The Hopsworks MCP server, `hopsworks-mcp`, which ships with the `hopsworks` library, has the same reads as tools; each takes the system's name as `system`: + +| Tool | Returns | +| --- | --- | +| `check_readiness` | Whether the system can answer, and what stops it | +| `search_terms` | The terms whose name contains a string | +| `get_term` | One term, with its description, synonyms and related terms | +| `get_term_columns` | The columns a term maps to | +| `get_term_sql_attributes` | The SQL expressions under a term, such as a margin or a ratio | +| `get_sql_attribute` | One SQL expression, with its term and its purpose | +| `describe_table` | One table, with its columns, the terms that reference it and the SQL expressions over it | +| `check_answerable` | How well the ontology covers the entities of a question, without running the agent | +| `ask_question` | The answer, its SQL and up to 100 rows; it takes tens of seconds and reports each step as progress | + +#### Ontology jobs + +The five jobs run in the `ontology-pipeline` environment with 2 cores, 4096 MB and no GPU. +Their default arguments name the system's configuration, `--config /ontology.yaml`, where `` is the system's directory relative to the project, such as `Users//sales-ontology`. +One job works on the ontology at a time: a job that starts while another runs fails with `another ontology pipeline job is running`. + +- `-ingest [--source ]` reads the sources' metadata into the catalog and embeds the tables and columns into `_data_objects`, then starts the compile job when compilation is switched on. + `--source` ingests one source. + It runs on its schedule, and **Add Source** starts it for the new source; it exits 1 when a source fails. +- `-compile [--force]` compiles the catalog into terms, joins and SQL expressions, and embeds them into `_semantic`. + It compiles only while **Enable Semantic Compilation** is on, and `--force` compiles regardless. + It runs on its schedule, after ingest, and when compilation is switched on, and exits 1 when the compilation fails. +- `-embed [--database ]` embeds every ontology row whose text changed since it was last embedded, of one database with `--database`. + It fails when a row is left without its vector. +- `-reset --semantic [--database ]` deletes the ontology and its vectors, of one database with `--database`, and compiles it again; **Reset** starts it. + `-reset --all --database ` deletes a database's catalog, ontology and vectors; **Remove** on **Sources** starts it. +- `-import --file [--merge]` imports an export, native or Apache Ossie YAML, from a file in the project, for a file too large to upload on **Import / Export**, and embeds it. + The file's definitions replace the existing ones, and `--merge` keeps the ones the file does not mention. + +An execution runs with the arguments it is given instead of the job's default arguments, so arguments given by hand repeat `--config`: + +```bash +hops job run sales-ontology-ingest --wait # the default arguments +hops job run sales-ontology-ingest --wait --args "--config Users//sales-ontology/ontology.yaml --source sales_mysql" +``` + +A scheduled run starts as its job's creator, which `system.yaml` records for each job. + +#### Delete an ontology + +**Delete** on the system's page, or `hops factory system delete --assets`, deletes the app with its tables, the jobs, the two feature groups and the system's directory; it needs a Data owner. +The models and the deployments the project's ontology systems share are kept while another system names them. +A deployment's description names the system that created it, so it is deleted with that system when no other system names it, and otherwise stays until you delete it with `hops deployment delete `. +A model's description names the Hugging Face commit it came from and no system, so the models are kept; `hops model delete --version ` deletes one. +The data sources the system reads, the sales example's tables, and the vLLM configurations in `Resources/ontology-vllm` are kept. + +```bash +hops factory system delete sales-ontology --assets # the app and its tables, the jobs, the feature groups and the directory +``` + ### Create your own factory A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. The project's own factories are listed under **From Template** with the built-in ones of their kind, each opening the form it generates. -The nine built-in factories, `ml-batch`, `ml-realtime`, `ml-streaming`, `ml-agent`, `analytics-bronze`, `analytics-silver`, `analytics-gold`, `analytics-pipeline` and `analytics-ingestion`, are read-only; clone one to change it. +The ten built-in factories, `ml-batch`, `ml-realtime`, `ml-streaming`, `ml-agent`, `ontology`, `analytics-bronze`, `analytics-silver`, `analytics-gold`, `analytics-pipeline` and `analytics-ingestion`, are read-only; clone one to change it. A form has no conditions: every question of a section is shown, and a section with defaults can be collapsed to a summary of its answers with **Edit**. **Factory > Manage factories** lists every factory with its version and how many systems it built. diff --git a/docs/user_guides/projects/python/python_env_overview.md b/docs/user_guides/projects/python/python_env_overview.md index b0af414b0c..358f75b945 100644 --- a/docs/user_guides/projects/python/python_env_overview.md +++ b/docs/user_guides/projects/python/python_env_overview.md @@ -41,6 +41,7 @@ The `FEATURE ENGINEERING` environments can be used in [Jupyter notebooks](../jup - `spark-feature-pipeline` for writing feature pipelines using PySpark - `dbt-pipeline` extends `python-feature-pipeline` with dbt and the Trino and DuckDB adapters, for transformations written as dbt models +- `ontology-pipeline` extends `python-feature-pipeline` with the Auto-Ontology runtime and its web UI, which the app and the jobs of an [ontology system][build-an-ontology-and-text-to-sql-app] run in ### Model training From 661e1300ca4d8d9db98ab90243140d77cfbdcf8d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 9 Nov 2026 11:59:31 +0100 Subject: [PATCH 2/3] [HWORKS-3744] Ontology factory: auto-generated ontology and text-to-SQL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 A system the ontology factory builds compiles its semantic layer: its ontology.yaml switches compilation on, and the build's compile run and the scheduled ones no longer skip. docs: the ontology section says a new system starts with semantic compilation on and with its PII and distinct-value answers as Settings, and that an edit in the app is re-embedded at once. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/client_installation/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index efb67be8c5..fda597ea8d 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -484,8 +484,9 @@ It refuses, with HTTP 422, a source of an unsupported type, one with PROVIDED cr The `sources` of `ontology.yaml` seed a new system only: once the app has a source, **Sources** is the record, so a source removed there stays removed. **Semantic Compilation** has **Enable Semantic Compilation**: the compile job compiles only while it is on, and switching it on starts a compile run. +A new system starts with it on, from `semantic_compilation: true` in its `ontology.yaml`; the `pii_detection` and `distinct_value_probing` answers set **PII Settings** and **Semantic Input** the same way, and a later change in **Settings** stays. **Reset** there deletes the ontology of every database and compiles it again. -An edit in the app shows in the catalog at once, and in the vector search after the next compile or embed run. +An edit in the app re-embeds the edited object, so it shows in the catalog and in the vector search at once; an edit whose embedding fails is embedded again by the next compile or embed run. **Import / Export** exports the catalog and the ontology as YAML, in the native format or Apache Ossie's, and imports such a file. The app answers at most four questions at a time; past that it answers HTTP 429 with `Retry-After`. From 9ce8001813b6f2613ab273a3711700a739bb4555 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 11 Nov 2026 08:25:10 +0100 Subject: [PATCH 3/3] [HWORKS-3744] Ontology factory: auto-generated ontology and text-to-SQL over data sources https://hopsworks.atlassian.net/browse/HWORKS-3744 An ontology system's app and jobs now run in ontology-, a clone of python-feature-pipeline with Auto-Ontology installed, instead of an ontology-pipeline base environment. The ontology guide names the environment, its build phase and that deleting a system keeps it, and the environments page no longer lists ontology-pipeline. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/client_installation/index.md | 13 ++++++++----- .../projects/python/python_env_overview.md | 1 - 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index fda597ea8d..b037603d54 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -384,17 +384,19 @@ A project can hold several ontology systems, and each one builds: | Part | Name | | --- | --- | -| App: the HTTP API, the web UI and the chat | ``, a custom app in the `ontology-pipeline` environment | +| Python environment: a clone of `python-feature-pipeline` with Auto-Ontology installed | `ontology-`, where `` is the first 8 characters of the Auto-Ontology commit it installs | +| App: the HTTP API, the web UI and the chat | ``, a custom app in that environment | | App tables: the catalog, the ontology, conversations and settings | `
`, in the project's online database | -| Jobs, in the `ontology-pipeline` environment | `-ingest`, `-compile`, `-embed`, `-reset`, `-import` | +| Jobs, in that environment | `-ingest`, `-compile`, `-embed`, `-reset`, `-import` | | Online feature groups, each with an embedding index | `_data_objects` (tables and columns), `_semantic` (the ontology) | | Models in the Model Registry, imported from Hugging Face | `Nemotron_3_5_Embed_1B_BF16`, `NVIDIA_Nemotron_3_5_Lightning_30B_A3B_FP8` | | vLLM deployments | `nemotronembed`, `nemotronllm` | `` is the name with hyphens replaced by underscores, and `
` is the app's table prefix. -The models and the deployments are shared by the project's ontology systems: a second system reuses those the first one made, and a system with an external model makes neither that model nor its deployment. +The environment, the models and the deployments are shared by the project's ontology systems: a second system reuses those the first one made, and a system with an external model makes neither that model nor its deployment. +Deleting a system keeps the environment. Each embedding index holds vectors of the dimension chosen in the form, compared by cosine similarity. -The system's directory holds `ontology.yaml`, the system's configuration and the only file written from the answers, and the launchers of the app (`app.py`) and of the jobs (`jobs/`), which run the Auto-Ontology runtime that the `ontology-pipeline` environment ships. +The system's directory holds `ontology.yaml`, the system's configuration and the only file written from the answers, and the launchers of the app (`app.py`) and of the jobs (`jobs/`), which run the Auto-Ontology runtime installed in the system's environment. #### Before you build an ontology @@ -425,6 +427,7 @@ The build runs these phases: | Phase | What the build does | | --- | --- | | Check the sources and write ontology.yaml | Checks that every source exists, has a supported type and SHARED credentials, and that an external model's account variables are set, then writes `ontology.yaml` | +| Python environment with Auto-Ontology | Clones `python-feature-pipeline` and installs Auto-Ontology into the clone, or reuses the environment another ontology system made | | Import the models and deploy them with vLLM | Imports each local model from Hugging Face at a pinned revision and deploys it, or reuses what another ontology system made; skipped when both models are external | | Ontology app | Creates and starts the app, which creates its tables | | Jobs, and ingest the sources' metadata | Creates the five jobs, schedules ingest and compile, and runs ingest | @@ -530,7 +533,7 @@ The Hopsworks MCP server, `hopsworks-mcp`, which ships with the `hopsworks` libr #### Ontology jobs -The five jobs run in the `ontology-pipeline` environment with 2 cores, 4096 MB and no GPU. +The five jobs run in the system's `ontology-` environment with 2 cores, 4096 MB and no GPU. Their default arguments name the system's configuration, `--config /ontology.yaml`, where `` is the system's directory relative to the project, such as `Users//sales-ontology`. One job works on the ontology at a time: a job that starts while another runs fails with `another ontology pipeline job is running`. diff --git a/docs/user_guides/projects/python/python_env_overview.md b/docs/user_guides/projects/python/python_env_overview.md index 358f75b945..b0af414b0c 100644 --- a/docs/user_guides/projects/python/python_env_overview.md +++ b/docs/user_guides/projects/python/python_env_overview.md @@ -41,7 +41,6 @@ The `FEATURE ENGINEERING` environments can be used in [Jupyter notebooks](../jup - `spark-feature-pipeline` for writing feature pipelines using PySpark - `dbt-pipeline` extends `python-feature-pipeline` with dbt and the Trino and DuckDB adapters, for transformations written as dbt models -- `ontology-pipeline` extends `python-feature-pipeline` with the Auto-Ontology runtime and its web UI, which the app and the jobs of an [ontology system][build-an-ontology-and-text-to-sql-app] run in ### Model training