release: v1.15.0 — the export gets a phase breakdown, and four rounds of C-343 land (#501)

docs/ADRs/022_tag_based_deployment_gate.md

@@ -51,7 +51,7 @@ Normally, `HEAD` points to a branch (e.g., `main`), and the branch points to a c
 This is **normal and expected on the server**. It means:
 - The server is looking at a frozen snapshot, not a moving branch.
 - No `git pull` or `git push` can accidentally change what's checked out.
-- The only way to change the running version is to update `~/.views-deploy-tag`.
+- Nothing changes the running version except an operator deploying — and deploying is the whole procedure below, not just the tag file.
 
 On developer machines, you work on branches as usual. Detached HEAD is only relevant on the deployment server.
 
@@ -112,7 +112,7 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co
 ### Positive
 
 - **Explicit version control.** The operator decides exactly what runs. No surprises from pushed commits.
-- **One-line rollback.** `echo 'v1.0.0' > ~/.views-deploy-tag` reverts to the previous release.
+- **Cheap rollback.** The previous release is always available — git never deletes tags — so reverting is the same short procedure as deploying, run with the earlier tag. (It is **not** one line; see the amendment below.)
 - **Audit trail.** The deploy tag file, git log, and provenance ledger together show what version produced what data.
 - **Safe for multiple maintainers.** Two people can push to `development` without affecting the server.
 - **No infrastructure dependencies.** No Docker registry, no CI/CD pipeline, no deployment service. Just git tags and a text file.
@@ -127,34 +127,32 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co
 
 ## Deployment Procedure
 
-**Deploy a new version:**
+**On your laptop:** the release ritual in `docs/guides/publishing_to_pypi.md` — bump, promote to `main`, cut a **GitHub Release** (a local `git tag && git push` does not trigger the publish workflow, and both branches have required pull requests since 2026-07-31).
 
-1. On your laptop: `git tag v1.2.0 && git push --tags`
-2. On the server: `echo 'v1.2.0' > ~/.views-deploy-tag`
-3. Wait for cron (21st of month) or run manually
+**On the server:** every block of `docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, `git fetch --tags && git checkout <tag>`, `uv sync`, then verification against three sources: the tag file, the checked-out tag, and the **installed distribution version**. A rollback is the same procedure with the earlier tag.
 
-**Roll back:**
+The commands are deliberately not reproduced here. This ADR records *why* the gate exists; the quickref is its single operational home, and **if they disagree, the quickref wins.**
 
-1. On the server: `echo 'v1.0.0' > ~/.views-deploy-tag`
-2. Run the pipeline manually or wait for cron
+### Amendment 2026-10-13 — writing the tag file is not deploying (C-343)
 
-**Verify:**
+This section previously gave deploying as three steps — local tag, tag file, wait for cron — and rollback as one. Both were wrong, and the error was not theoretical: on 2026-08-08 the production host ran v1.10.0 for five days while the tag file read v1.11.0, with a green status page, a pinging heartbeat, and no error at any point. v1.11.0 existed solely to raise the `views-frames` floor, and the floor on the box was still the frozen 1.0.0.
 
-```bash
-cat ~/.views-deploy-tag                    # configured version
-git describe --tags --exact-match          # checked-out version
-tail -5 logs/refresh.log                   # last run output
-```
+The gate described in **Decision** is unaffected and still does what it claims: the pipeline refuses to start without a valid tag, and checks that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. It does not, for two reasons established by experiment rather than by reading:
+
+1. Bash buffers `scripts/refresh_pipeline.sh` before the gate's `git checkout` runs, so a change to that script takes effect on the **next** run — one month, on a monthly cron.
+2. The pipeline never invokes `uv sync`, so a dependency change never lands at all, however many times it runs.
+
+The **Verify** block was also incomplete in the way that mattered: it listed the tag file and `git describe` but not the installed distribution, which is the only one of the three that reflects `uv sync`. In August it was the installed version that was wrong.
 
-Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, section "Deployment and releases."
+C-343 is **open**: the three guides and this ADR now agree, but nothing mechanically prevents the procedure from being skipped. The proposed instrument — a pre-flight check comparing the deploy tag against the HEAD the script *started* from, failing loud per ADR-011 rather than self-correcting one run late — is tracked in #424 and not built.
 
 ---
 
 ## References
 
 - ADR-011 (Fail Loud, No Stale Data Serving) — justifies fail-loud on misconfiguration
 - ADR-018 (Operational Resilience Policy) — operator-mediated bounded staleness
-- `scripts/refresh_pipeline.sh` lines 76-98 — implementation
+- `scripts/refresh_pipeline.sh`, the "Deployment gate" block — implementation
 - `reports/archive/technical_risk_register_resolved.md` C-98 — risk that motivated this ADR
 - `docs/guides/hetzner_deployment_guide.md` — operational procedures
 - `docs/guides/data_serving_guide.md` section 9 — conceptual explanation

docs/ADRs/041_content_addressed_skip.md

@@ -107,3 +107,18 @@ This ADR covers assembly and export only. It does not cover:
 - **Output integrity is verified on every skip:** a corrupt grid.npy triggers a rebuild, not silent serving.
 - **Source set changes are detected:** adding or removing a source (e.g., adding SHDI) prevents false skips.
 - **Operator override exists:** `--force` provides an escape hatch when data hasn't changed but code has.
+
+---
+
+### Amendment 2026-10-18 — the export cost in Context is 5× low, and the skip is worth more than claimed
+
+**Context** states that assembly is ~46 min and export ~57 min, and **Consequences** derives "no-change deploys drop from ~103 min to ~30 sec" from that pair. The 2026-10-16 production run measured the export at **18,427s (5h07m)** for a 3.1 GB store — roughly **5× the figure above**, and 88% of a 21,007s (5h50m) run.
+
+Nothing in this ADR's decision is affected. The skip mechanism works as specified, and the digest gates behave as described. Two statements derived from the stale timings are wrong:
+
+1. **The export is not comparable in cost to assembly. It dominates the pipeline.** Any future reasoning that treats the two as peers — a decision about which to optimise, which to parallelise, or which to make skippable — starts from a false premise.
+2. **This ADR understates its own benefit by about 3.4×.** A no-change deploy avoids roughly **~350 min**, not ~103. The skip is the single largest runtime saving in the pipeline.
+
+**The cause of the 5× is not established and is deliberately not guessed at here.** The store grew from 1.8 GB (2026-03) to 3.1 GB, which accounts for some of it but not a factor of five. Candidates worth measuring: disk I/O on the CPX32, the `12 × 360 × 720` chunking against a larger store, and compression settings. Tracked as a separate investigation in **#501**, which asks for the three export phases to be timed separately before anything is changed.
+
+The per-step operational figures live in `docs/guides/server_quickref.md`, which owns them. The numbers in **Context** above are left as written — they were measured in June 2026 and this amendment dates the correction rather than rewriting the record.

docs/guides/data_serving_guide.md

@@ -174,10 +174,14 @@ Example: "Run the data pipeline on the 21st of every month at midnight"
 That line means: minute 0, hour 0, day 21, every month, every
 day-of-week. Run the script.
 
-Your pipeline takes about 15-30 minutes (harvesting is the slow
-part — it calls the UCDP and ACLED APIs). After it finishes, the new zarr
-and parquet files are in the same directory the web server serves.
-Consumers automatically get the new data on their next request.
+Your pipeline takes about **six hours**. Harvesting is *not* the slow part,
+despite being the part that calls external APIs: harvest through compile is
+~43 minutes, and the zarr export alone is ~5 hours. Per-step timings live in
+[`server_quickref.md`](server_quickref.md#run-the-pipeline).
+
+After it finishes, the new zarr and parquet files are in the same directory the
+web server serves. Consumers automatically get the new data on their next
+request.
 
 **No restart needed.** The web server serves whatever files are
 in the directory. When the pipeline replaces them, the next
@@ -250,13 +254,22 @@ notice for days.
 
 With the tag system: the server keeps running the old working version
 until you *explicitly* tell it to switch. You test the new version,
-tag it, and only then update the server. If the new version breaks,
-you write the old tag name back and you're running known-good code
-again.
-
-**The full deployment procedure** (create a tag, push it, update the
-server file, rollback if needed) is documented in
-`hetzner_deployment_guide.md` under "Deployment and releases".
+release it, and only then deploy it. If the new version breaks, you
+deploy the previous tag and you're running known-good code again.
+
+**One thing this explanation makes tempting, and it is wrong.** Because
+the pipeline checks the tag out for you, it looks as though changing the
+tag file *is* deploying. It is not. The script is already running when it
+does that checkout, so a change to the script itself only takes effect
+the run after — a month, on a monthly cron — and the script never
+installs dependencies, so a dependency change never lands at all. That
+misreading left the server on v1.10.0 for five days with every signal
+reading v1.11.0 (C-343, August 2026).
+
+**The deployment procedure** — tag file, checkout, `uv sync`, then verify
+three sources agree — lives in `server_quickref.md` under "Deploy a new
+version", and only there. A rollback is the same procedure with the
+earlier tag.
 
 The architectural rationale (why tags instead of branch tracking,
 Docker, or CI/CD) is in **ADR-022**.

docs/guides/hetzner_deployment_guide.md

@@ -27,7 +27,7 @@ If the server is already set up and you're just deploying a tag that **already e
 # 1. SSH into the server (replace <your-user> with your username)
 ssh <your-user>@204.168.219.108
 
-# 2. Start tmux (pipeline runs 3-4 hours; SSH will drop without a multiplexer)
+# 2. Start tmux (the run takes ~6 hours; SSH will drop without a multiplexer)
 tmux new -s deploy
 
 # 3. Deploy and run
@@ -181,13 +181,23 @@ source ~/.profile
 
 ### 2.1 Run the full pipeline
 
+> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time.
+
 ```bash
+tmux new -s pipeline
 cd ~/views-datafactory
 bash scripts/refresh_pipeline.sh
 ```
 
-This takes 25-45 minutes (harvesting calls external APIs and downloads
-large raster files). Watch the output — each step prints PASS or FAIL.
+This takes about **six hours**. Watch the output — each step prints PASS or
+FAIL. Per-step timings are in
+[`server_quickref.md`](server_quickref.md#run-the-pipeline), which owns them.
+
+> **This figure said "25-45 minutes" until 2026-10-18, and that was the
+> dangerous direction to be wrong in.** A run at the two-hour mark looks hung
+> against a 45-minute estimate, and the reasonable response to a hung run is to
+> kill it. The cost is not harvesting — harvest through compile is ~43 minutes.
+> **The zarr export alone is ~5 hours and prints nothing while it works.**
 
 **Note:** The pipeline includes ACLED (8 features), GHS-POP
 population data (1 feature), GHS-BUILT-S built-up surface data
@@ -434,7 +444,10 @@ crontab -l
 
 ### 4.5 Test manually
 
+> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time.
+
 ```bash
+tmux new -s pipeline
 cd ~/views-datafactory
 bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log
 ```
@@ -457,8 +470,8 @@ See ADR-018 for the rationale.
 2. Create a check (or verify the existing one):
    - **Name:** `views-datafactory pipeline refresh`
    - **Period:** 30 days (matches the `0 0 21 * *` cron)
-   - **Grace period:** 48 hours (pipeline runs 2-3 hours; 48h covers
-     delays, restarts, and weekend non-response)
+   - **Grace period:** 48 hours (the run takes ~6 hours; 48h still leaves
+     room for delays, restarts, and weekend non-response)
    - **Tags:** `views`, `datafactory`, `hetzner`
 3. Copy the ping URL: `https://hc-ping.com/<uuid>`
 
@@ -852,8 +865,11 @@ fixed order:
    diverge a little further after **every** release, and
    `git log main..development` starts counting work that already shipped.
 
-5. **Deploy** = update `~/.views-deploy-tag` on the server to the
-   new tag. The pipeline checks out whatever tag that file contains.
+5. **Deploy** = three steps on the server, not one: the tag file, then
+   `git fetch --tags && git checkout <tag>`, then `uv sync`. Writing the
+   tag file alone is **not** deploying — that mistake left the host on
+   v1.10.0 for five days while every signal read v1.11.0 (C-343).
+   `docs/guides/server_quickref.md` holds the commands.
 
 **Why this order:** If you tag before bumping, the tag points at code
 with the old version string — wrong. If you tag on `development`
@@ -871,48 +887,54 @@ first, promote second, tag third, back-merge fourth.
 > corrected, because a second copy of the procedure is what let the two guides drift apart
 > in the first place (#402). One procedure, one home.
 
-**On the server** (SSH in):
-
-```bash
-# Update the deploy tag file
-echo 'vX.Y.Z' > ~/.views-deploy-tag
-```
+**On the server** (SSH in): run **every block** in
+`docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, then
+`git fetch --tags && git checkout <tag>`, then `uv sync`, then the three
+verification commands. The commands are there and not here on purpose.
+
+> **This section used to say "update `~/.views-deploy-tag`. That's it."** It was
+> wrong, and it is the sentence C-343 is about: on 2026-08-08 the tag file read
+> `v1.11.0`, `git describe` read `v1.10.0`, and the installed `views-frames` was
+> still the frozen `1.0.0` — three sources of truth disagreeing, with a green
+> status page and a pinging heartbeat and no error anywhere. The in-pipeline
+> `git checkout "$DEPLOY_TAG"` does not rescue it: bash has already buffered
+> `scripts/refresh_pipeline.sh`, so a change to that script lands one run later —
+> a month, on a monthly cron — and `uv sync` is never invoked by the pipeline at
+> all, so a dependency change never lands however many times it runs. C-343 was
+> corrected in `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md`
+> in #431 and **this third copy was missed**, which is the same drift #402 is
+> about. The commands live in the quickref and nowhere else; if this file and the
+> quickref ever disagree, **the quickref wins.**
 
-That's it. The next cron run (21st of the month) will automatically
-use the new tag. If you want to apply it right now instead of waiting:
+### How to roll back
 
-```bash
-# Run the pipeline manually (same command cron uses)
-cd ~/views-datafactory
-bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log
-```
+If a new version breaks the pipeline, re-run the same three deploy steps with the
+previous tag — a rollback is a deploy, and skipping `uv sync` on the way back
+leaves the dependencies of the version you are abandoning. Tags are never
+deleted by git, so every previous version stays available.
 
-### How to roll back
+Then run the pipeline manually to apply it immediately rather than waiting for
+the next cron run.
 
-If a new version breaks the pipeline:
+> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time.
 
 ```bash
-# On the server — point back to the previous version
-echo 'vPREVIOUS' > ~/.views-deploy-tag
+tmux new -s pipeline
+sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log'
 ```
 
-The next pipeline run will check out that tag and run its code
-instead. The old version is always available — git never deletes tags.
-
-You can also run the pipeline manually to roll back immediately
-instead of waiting for the next cron run.
-
 ### How to check what version is deployed
 
-```bash
-# What version is the server configured to run?
-cat ~/.views-deploy-tag
+**Three sources, and they must agree** — the tag file, the checked-out tag, and
+the installed distribution. Checking only the tag file is what C-343 was: it is a
+statement of intent, not evidence. The commands are step 4 of
+`docs/guides/server_quickref.md` §"Deploy a new version"; they are not repeated
+here, because a second copy of them is how this section came to be wrong.
 
-# What version is actually checked out right now?
-git describe --tags --exact-match 2>/dev/null || git log --oneline -1
+For when the last pipeline run happened, `tail` the refresh log:
 
-# When did the last pipeline run happen?
-tail -5 logs/refresh.log
+```bash
+sudo -u views-deploy bash -c 'tail -5 ~/views-datafactory/logs/refresh.log'
 ```
 
 ### Log rotation

docs/guides/hetzner_deployment_log.md

@@ -295,6 +295,12 @@ or mismatched path.
 
 **Total: approximately 2.5 hours for a clean run.**
 
+> **These are March 2026 measurements and are no longer current.** The
+> 2026-10-16 run took **5h50m**, with the zarr export alone at ~5h07m against a
+> 3.1 GB store (this table has it at ~14 min against 1.8 GB). This is a dated
+> log and the figures above are left as measured; current per-step timings live
+> in [`server_quickref.md`](server_quickref.md#run-the-pipeline).
+
 Subsequent runs are much faster because most data is cached (only new
 UCDP versions are fetched).