feat(enrichment): the metadata cache and the engine that fills it

CLAUDE.md

@@ -44,7 +44,7 @@ bun run build                          # webapp first, then cargo — the order
 
 Tests need no service container: `test_app()` builds a `Config` over a `TempDir` and the suite drives the same router `main` serves. FFmpeg and ffprobe must be on `PATH`.
 
-The integration suite is seventeen targets under `tests/`, one per subject — `auth`, `scanner`, `identity`, `catalog`, `subsonic_contract`, `subsonic_browse`, `subsonic_fields`, `subsonic_methods`, `native_api`, `sync`, `media`, `service`, `cli`, `scrobbling`, `uploads`, `canvas`, `migrations` — nearly all of them sharing `tests/support/mod.rs`. `migrations` is the exception and deliberately so: every other target starts from `test_app()`, which builds its database from nothing, and that is exactly the path an upgrade does not take. It seeds a database at each **released** schema, boots the server on it, and reads the rows back through the services — and it is where anything else that needs a database *behind* this binary belongs, such as proving that a key change reads without migrating and copies the pair aside before it does. `autotests = false`, so each is a target only because `Cargo.toml` declares it: a new file under `tests/` is compiled by nothing until a `[[test]]` entry names it, and the count above is worth checking against that list rather than trusted. `support/mod.rs` is the exception that proves it — shared module code, declared nowhere, a target never. File a new test by the surface it exercises, and check that by what it calls rather than by what its name says.
+The integration suite is eighteen targets under `tests/`, one per subject — `auth`, `scanner`, `identity`, `catalog`, `subsonic_contract`, `subsonic_browse`, `subsonic_fields`, `subsonic_methods`, `native_api`, `sync`, `media`, `service`, `cli`, `scrobbling`, `enrichment`, `uploads`, `canvas`, `migrations` — nearly all of them sharing `tests/support/mod.rs`. `migrations` is the exception and deliberately so: every other target starts from `test_app()`, which builds its database from nothing, and that is exactly the path an upgrade does not take. It seeds a database at each **released** schema, boots the server on it, and reads the rows back through the services — and it is where anything else that needs a database *behind* this binary belongs, such as proving that a key change reads without migrating and copies the pair aside before it does. `autotests = false`, so each is a target only because `Cargo.toml` declares it: a new file under `tests/` is compiled by nothing until a `[[test]]` entry names it, and the count above is worth checking against that list rather than trusted. `support/mod.rs` is the exception that proves it — shared module code, declared nowhere, a target never. File a new test by the surface it exercises, and check that by what it calls rather than by what its name says.
 
 `cargo build` works without a client build — `build.rs` embeds a placeholder page when `webapp/dist` is absent. A real `vite build` overwrites it.
 

Cargo.toml

@@ -154,3 +154,7 @@ path = "tests/canvas.rs"
 [[test]]
 name = "scrobbling"
 path = "tests/scrobbling.rs"
+
+[[test]]
+name = "enrichment"
+path = "tests/enrichment.rs"

docs/operations.md

@@ -169,6 +169,47 @@ Every tunable is a field on `Config` in
 [`src/config.rs`](../src/config.rs), with its environment variable documented on
 it. That file is the authority; this page is not a copy of it.
 
+## Switching metadata enrichment on
+
+**Off by default, and an upgrade leaves it off.** A server that has not been
+told otherwise makes no request to any metadata provider.
+[RFC-011](rfcs/RFC-011-metadata-enrichment.md) carries the reasoning.
+
+```bash
+WAVEFLOW_METADATA_PROVIDERS=lastfm
+WAVEFLOW_METADATA_LASTFM_KEY=…   # your own Last.fm API application's key
+```
+
+| Variable | Default | What it does |
+| --- | --- | --- |
+| `WAVEFLOW_METADATA_PROVIDERS` | empty | The switch: an ordered, comma-separated list. Empty is off. Today the only provider is `lastfm` |
+| `WAVEFLOW_METADATA_LASTFM_KEY` | — | Required when `lastfm` is listed; the server refuses to start without it. No key ships in the binary |
+| `WAVEFLOW_METADATA_IMAGES` | on | `off` keeps the lookups and refuses artist image downloads |
+| `WAVEFLOW_METADATA_TTL_DAYS` | 30 | How long an answer with content is kept before it is asked again |
+| `WAVEFLOW_METADATA_MISS_TTL_DAYS` | 7 | How long "the provider knows nothing about this artist" is kept |
+| `WAVEFLOW_METADATA_RATE_PER_SEC` | 4 | Requests per second, per provider |
+| `WAVEFLOW_METADATA_REQUEST_TIMEOUT_SECS` | 10 | How long one request may take before it counts as unanswered |
+
+**Scrobbling's Last.fm key does not switch this on.** An application you
+registered for submitting listens is a different consent from lookups about your
+whole catalogue. You may place the same key in both variables; each one switches
+on only its own feature.
+
+**What the server starts contacting.** `ws.audioscrobbler.com`, over https, and
+only when somebody opens an artist. Nothing is fetched at scan time. Each
+request carries the artist's name and your key, and no redirect is followed.
+Requests are spaced to the rate above, and simultaneous visits to one artist
+share one request.
+
+**What it stores.** In `waveflow.db`, for each provider and each artist name
+asked about: the biography, the ranked list of similar artists by name, whether
+the provider knew the artist, and when to ask again. Nothing in it belongs to an
+account, and nothing records who asked. A failed request stores nothing, so the
+next visit asks again.
+
+**What it logs.** The provider and a verdict, such as `answered HTTP 503` —
+never the request URL, which carries your key.
+
 ## Probes
 
 | Path | Answers |

docs/rfcs/RFC-011-metadata-enrichment.md

@@ -1,10 +1,18 @@
 # RFC-011 — Server-side metadata enrichment
 
 - **Status**: Proposed
-- **Implemented by**: nothing yet. The *Status* field above never flips in this
-  project; this line is the one that says what is running, and today it says
-  the answer is still the empty container.
+- **Implemented by**: in progress, in the order of
+  [the implementation plan](../metadata-enrichment-plan.md):
+  InstaZDLL/WaveFlow#855 (the core's clients accept a client, an address and a
+  key), #270 (the pin moved onto them), and the pull request closing #265 (the
+  cache and the engine that fills it). No surface answers from the cache yet,
+  so every container is still the empty one. The *Status* field above never
+  flips in this project; this line is the one that says what is running.
 - **Date**: 2026-10-14
+- **Revised**: 2026-10-15, while implementing the cache (#265). Decisions 4, 6
+  and 10 rested on attachment points the code does not hold; each correction
+  below is marked and says what the first version asserted, rather than
+  erasing it.
 - **Authors**: the WaveFlow project
 - **Depends on**: [RFC-002](RFC-002-waveflow-server-v2.md),
   [RFC-010](RFC-010-external-scrobbling.md)
@@ -170,6 +178,14 @@ The cache is therefore keyed on the **canonical name**, as the desktop's own
 - `metadata_album` — the same shape for album notes, keyed on the provider and
   the canonical `(artist, title)` pair that `album_match.rs` already computes.
 
+**Revised on 2026-10-15.** The migration that shipped
+(`20261015000000_metadata_artist.sql`) differs from this list in three places.
+The biography is two columns, `bio_summary` and `bio_full`, because that is how
+the providers return it. `image_url` is not there: portraits add their columns
+with the fetch that fills them. And `metadata_album` was not created — the
+open question on album notes below decides whether it exists, and
+`album_match.rs` was not even in the pinned core when this was written.
+
 ### One row per provider, merged on read
 
 **The provider belongs in the key, not in a column**, and the first draft of
@@ -204,6 +220,16 @@ matching a provider's ranked candidate list back to the name asked for is
 `select_by_name`. Neither is re-implemented here; both have a desktop issue
 behind them (#342) recording what naive equality dropped.
 
+**Revised on 2026-10-15: the key is not `normalize_name`.** The paragraph above
+asserted it was. Every `artist` row already stores a `canonical_name`, computed
+by `waveflow_core::scanner::canonical_name`, and the two functions disagree:
+`normalize_name` also folds diacritics and turns `&` into `and`, so "Björk" is
+`björk` in the column and `bjork` through it. The cache is keyed on the
+column's form, which lets the read path join
+`metadata_artist.name_canonical = artist.canonical_name` with nothing
+recomputed. `normalize_name` and `select_by_name` keep the job they were written
+for — matching a provider's candidates — where a provider returns candidates.
+
 **Nothing in these tables is account data**, which is what makes an
 instance-wide cache correct rather than merely convenient. Anything that ever
 becomes account-specific — a per-person override, a hidden artist — belongs in
@@ -267,6 +293,13 @@ for someone to reload the page. The mechanism already exists and decision 8
 explains why it may now carry an artist: RFC-007's library event stream, whose
 decision 6 declined an artist event for a reason this RFC removes.
 
+**Revised on 2026-10-15: only one client is told.** The paragraph above implied
+every client is. The desktop is, because it mirrors the library through that
+feed. The web client is not: the only event stream it reads is a scan's
+progress. It re-asks instead, a bounded number of times while the answer says
+the lookup is pending, and then stops — which is the web client's step of the
+plan, not a property this mechanism gives it.
+
 ## Decision 7 — similar artists resolve to artists the caller can see, and to nothing else
 
 `artist.getSimilar` returns names. Subsonic's `similarArtist` entries carry an
@@ -415,6 +448,16 @@ them empty.
 - `getArtistInfo` / `getArtistInfo2` — `biography`, `similarArtist` entries
   (decision 7), and `smallImageUrl` / `mediumImageUrl` / `largeImageUrl`
   pointing at this server's own artwork route (decision 8).
+
+  **Revised on 2026-10-15: the three image URLs stay absent.** The bullet above
+  asserted a route they could point at. Neither artwork route serves a URL a
+  client loads as-is: `media::artwork` wants the native bearer, and
+  `getCoverArt` wants Subsonic credentials in the query string, which would put
+  a member's token into a URL that clients cache and log. A client finds the
+  portrait through `coverArt`, which every artist element already carries. The
+  compatibility campaign measures whether a client in the matrix reads
+  `largeImageUrl` at all; a sealed artwork ticket can be added then, while a URL
+  scheme cannot be withdrawn once clients have cached it.
 - `getSimilarSongs` / `getSimilarSongs2` — tracks by the neighbours the caller
   can see.
 - `getAlbumInfo` / `getAlbumInfo2` — `notes` beside the `musicBrainzId` they

migrations-v2/20261015000000_metadata_artist.sql

@@ -0,0 +1,53 @@
+-- What the metadata providers said about an artist — RFC-011 decisions 4 and 5.
+--
+-- **Instance-wide, and nothing in it is account data.** A biography is the same
+-- text whoever asks, and `artist` rows are per library, so keying on
+-- `artist.id` would fetch the same answer once per library. The key is the
+-- name instead, in the form `artist.canonical_name` already stores
+-- (`waveflow_core::scanner::canonical_name`), so the read path joins on that
+-- column with nothing recomputed. RFC-011 first named
+-- `name_match::normalize_name` here; the two disagree — "Björk" is `björk` in
+-- the column and `bjork` through it — and its erratum says so.
+--
+-- **The provider is part of the key.** Each provider keeps its own row, its
+-- own outcome and its own clock, and the fields are merged at read time in the
+-- operator's order. One row per name would have to pick a winner when written,
+-- and would let one provider's "never heard of them" silence another that has
+-- a biography.
+--
+-- `provider` carries no CHECK: the vocabulary is the server's `Provider` enum,
+-- and widening a CHECK in SQLite means rebuilding the table. `outcome` does,
+-- because its two words are decision 5's and do not grow — the third outcome,
+-- unreached, is the absence of a row.
+--
+-- Album notes (`metadata_album`) wait for the RFC's open question on them.
+-- Portraits add their columns with the fetch that fills them.
+CREATE TABLE metadata_artist (
+    provider TEXT NOT NULL,
+    name_canonical TEXT NOT NULL,
+    outcome TEXT NOT NULL CHECK (outcome IN ('hit', 'miss')),
+    bio_summary TEXT,
+    bio_full TEXT,
+    fetched_at INTEGER NOT NULL,
+    expires_at INTEGER NOT NULL,
+    PRIMARY KEY (provider, name_canonical)
+) STRICT, WITHOUT ROWID;
+
+-- The ranked neighbours, as the provider gave them. Kept by name and resolved
+-- to local artists only when read, through the caller's own libraries
+-- (decision 7): an id here would name a row in somebody's library.
+--
+-- Rewritten whole with its parent row, in the same transaction, so a list is
+-- never half one answer and half the next. The cascade only matters to a row
+-- deleted outright; a refresh updates the parent in place.
+CREATE TABLE metadata_artist_similar (
+    provider TEXT NOT NULL,
+    name_canonical TEXT NOT NULL,
+    rank INTEGER NOT NULL,
+    similar_name TEXT NOT NULL,
+    similar_canonical TEXT NOT NULL,
+    score REAL NOT NULL,
+    PRIMARY KEY (provider, name_canonical, rank),
+    FOREIGN KEY (provider, name_canonical)
+        REFERENCES metadata_artist(provider, name_canonical) ON DELETE CASCADE
+) STRICT, WITHOUT ROWID;