diff --git a/docs/ADRs/022_tag_based_deployment_gate.md b/docs/ADRs/022_tag_based_deployment_gate.md index 37a8c1f..19e030e 100644 --- a/docs/ADRs/022_tag_based_deployment_gate.md +++ b/docs/ADRs/022_tag_based_deployment_gate.md @@ -51,7 +51,7 @@ Normally, `HEAD` points to a branch (e.g., `main`), and the branch points to a c This is **normal and expected on the server**. It means: - The server is looking at a frozen snapshot, not a moving branch. - No `git pull` or `git push` can accidentally change what's checked out. -- The only way to change the running version is to update `~/.views-deploy-tag`. +- Nothing changes the running version except an operator deploying — and deploying is the whole procedure below, not just the tag file. On developer machines, you work on branches as usual. Detached HEAD is only relevant on the deployment server. @@ -112,7 +112,7 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ### Positive - **Explicit version control.** The operator decides exactly what runs. No surprises from pushed commits. -- **One-line rollback.** `echo 'v1.0.0' > ~/.views-deploy-tag` reverts to the previous release. +- **Cheap rollback.** The previous release is always available — git never deletes tags — so reverting is the same short procedure as deploying, run with the earlier tag. (It is **not** one line; see the amendment below.) - **Audit trail.** The deploy tag file, git log, and provenance ledger together show what version produced what data. - **Safe for multiple maintainers.** Two people can push to `development` without affecting the server. - **No infrastructure dependencies.** No Docker registry, no CI/CD pipeline, no deployment service. Just git tags and a text file. @@ -127,26 +127,24 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ## Deployment Procedure -**Deploy a new version:** +**On your laptop:** the release ritual in `docs/guides/publishing_to_pypi.md` — bump, promote to `main`, cut a **GitHub Release** (a local `git tag && git push` does not trigger the publish workflow, and both branches have required pull requests since 2026-07-31). -1. On your laptop: `git tag v1.2.0 && git push --tags` -2. On the server: `echo 'v1.2.0' > ~/.views-deploy-tag` -3. Wait for cron (21st of month) or run manually +**On the server:** every block of `docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, `git fetch --tags && git checkout `, `uv sync`, then verification against three sources: the tag file, the checked-out tag, and the **installed distribution version**. A rollback is the same procedure with the earlier tag. -**Roll back:** +The commands are deliberately not reproduced here. This ADR records *why* the gate exists; the quickref is its single operational home, and **if they disagree, the quickref wins.** -1. On the server: `echo 'v1.0.0' > ~/.views-deploy-tag` -2. Run the pipeline manually or wait for cron +### Amendment 2026-10-13 — writing the tag file is not deploying (C-343) -**Verify:** +This section previously gave deploying as three steps — local tag, tag file, wait for cron — and rollback as one. Both were wrong, and the error was not theoretical: on 2026-08-08 the production host ran v1.10.0 for five days while the tag file read v1.11.0, with a green status page, a pinging heartbeat, and no error at any point. v1.11.0 existed solely to raise the `views-frames` floor, and the floor on the box was still the frozen 1.0.0. -```bash -cat ~/.views-deploy-tag # configured version -git describe --tags --exact-match # checked-out version -tail -5 logs/refresh.log # last run output -``` +The gate described in **Decision** is unaffected and still does what it claims: the pipeline refuses to start without a valid tag, and checks that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. It does not, for two reasons established by experiment rather than by reading: + +1. Bash buffers `scripts/refresh_pipeline.sh` before the gate's `git checkout` runs, so a change to that script takes effect on the **next** run — one month, on a monthly cron. +2. The pipeline never invokes `uv sync`, so a dependency change never lands at all, however many times it runs. + +The **Verify** block was also incomplete in the way that mattered: it listed the tag file and `git describe` but not the installed distribution, which is the only one of the three that reflects `uv sync`. In August it was the installed version that was wrong. -Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, section "Deployment and releases." +C-343 is **open**: the three guides and this ADR now agree, but nothing mechanically prevents the procedure from being skipped. The proposed instrument — a pre-flight check comparing the deploy tag against the HEAD the script *started* from, failing loud per ADR-011 rather than self-correcting one run late — is tracked in #424 and not built. --- @@ -154,7 +152,7 @@ Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, se - ADR-011 (Fail Loud, No Stale Data Serving) — justifies fail-loud on misconfiguration - ADR-018 (Operational Resilience Policy) — operator-mediated bounded staleness -- `scripts/refresh_pipeline.sh` lines 76-98 — implementation +- `scripts/refresh_pipeline.sh`, the "Deployment gate" block — implementation - `reports/archive/technical_risk_register_resolved.md` C-98 — risk that motivated this ADR - `docs/guides/hetzner_deployment_guide.md` — operational procedures - `docs/guides/data_serving_guide.md` section 9 — conceptual explanation diff --git a/docs/ADRs/041_content_addressed_skip.md b/docs/ADRs/041_content_addressed_skip.md index 34fafb9..58d0c15 100644 --- a/docs/ADRs/041_content_addressed_skip.md +++ b/docs/ADRs/041_content_addressed_skip.md @@ -107,3 +107,18 @@ This ADR covers assembly and export only. It does not cover: - **Output integrity is verified on every skip:** a corrupt grid.npy triggers a rebuild, not silent serving. - **Source set changes are detected:** adding or removing a source (e.g., adding SHDI) prevents false skips. - **Operator override exists:** `--force` provides an escape hatch when data hasn't changed but code has. + +--- + +### Amendment 2026-10-18 — the export cost in Context is 5× low, and the skip is worth more than claimed + +**Context** states that assembly is ~46 min and export ~57 min, and **Consequences** derives "no-change deploys drop from ~103 min to ~30 sec" from that pair. The 2026-10-16 production run measured the export at **18,427s (5h07m)** for a 3.1 GB store — roughly **5× the figure above**, and 88% of a 21,007s (5h50m) run. + +Nothing in this ADR's decision is affected. The skip mechanism works as specified, and the digest gates behave as described. Two statements derived from the stale timings are wrong: + +1. **The export is not comparable in cost to assembly. It dominates the pipeline.** Any future reasoning that treats the two as peers — a decision about which to optimise, which to parallelise, or which to make skippable — starts from a false premise. +2. **This ADR understates its own benefit by about 3.4×.** A no-change deploy avoids roughly **~350 min**, not ~103. The skip is the single largest runtime saving in the pipeline. + +**The cause of the 5× is not established and is deliberately not guessed at here.** The store grew from 1.8 GB (2026-03) to 3.1 GB, which accounts for some of it but not a factor of five. Candidates worth measuring: disk I/O on the CPX32, the `12 × 360 × 720` chunking against a larger store, and compression settings. Tracked as a separate investigation in **#501**, which asks for the three export phases to be timed separately before anything is changed. + +The per-step operational figures live in `docs/guides/server_quickref.md`, which owns them. The numbers in **Context** above are left as written — they were measured in June 2026 and this amendment dates the correction rather than rewriting the record. diff --git a/docs/guides/data_serving_guide.md b/docs/guides/data_serving_guide.md index 71cf3a3..b3cd2c1 100644 --- a/docs/guides/data_serving_guide.md +++ b/docs/guides/data_serving_guide.md @@ -174,10 +174,14 @@ Example: "Run the data pipeline on the 21st of every month at midnight" That line means: minute 0, hour 0, day 21, every month, every day-of-week. Run the script. -Your pipeline takes about 15-30 minutes (harvesting is the slow -part — it calls the UCDP and ACLED APIs). After it finishes, the new zarr -and parquet files are in the same directory the web server serves. -Consumers automatically get the new data on their next request. +Your pipeline takes about **six hours**. Harvesting is *not* the slow part, +despite being the part that calls external APIs: harvest through compile is +~43 minutes, and the zarr export alone is ~5 hours. Per-step timings live in +[`server_quickref.md`](server_quickref.md#run-the-pipeline). + +After it finishes, the new zarr and parquet files are in the same directory the +web server serves. Consumers automatically get the new data on their next +request. **No restart needed.** The web server serves whatever files are in the directory. When the pipeline replaces them, the next @@ -250,13 +254,22 @@ notice for days. With the tag system: the server keeps running the old working version until you *explicitly* tell it to switch. You test the new version, -tag it, and only then update the server. If the new version breaks, -you write the old tag name back and you're running known-good code -again. - -**The full deployment procedure** (create a tag, push it, update the -server file, rollback if needed) is documented in -`hetzner_deployment_guide.md` under "Deployment and releases". +release it, and only then deploy it. If the new version breaks, you +deploy the previous tag and you're running known-good code again. + +**One thing this explanation makes tempting, and it is wrong.** Because +the pipeline checks the tag out for you, it looks as though changing the +tag file *is* deploying. It is not. The script is already running when it +does that checkout, so a change to the script itself only takes effect +the run after — a month, on a monthly cron — and the script never +installs dependencies, so a dependency change never lands at all. That +misreading left the server on v1.10.0 for five days with every signal +reading v1.11.0 (C-343, August 2026). + +**The deployment procedure** — tag file, checkout, `uv sync`, then verify +three sources agree — lives in `server_quickref.md` under "Deploy a new +version", and only there. A rollback is the same procedure with the +earlier tag. The architectural rationale (why tags instead of branch tracking, Docker, or CI/CD) is in **ADR-022**. diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 6cd8a91..f852129 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -27,7 +27,7 @@ If the server is already set up and you're just deploying a tag that **already e # 1. SSH into the server (replace with your username) ssh @204.168.219.108 -# 2. Start tmux (pipeline runs 3-4 hours; SSH will drop without a multiplexer) +# 2. Start tmux (the run takes ~6 hours; SSH will drop without a multiplexer) tmux new -s deploy # 3. Deploy and run @@ -181,13 +181,23 @@ source ~/.profile ### 2.1 Run the full pipeline +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh ``` -This takes 25-45 minutes (harvesting calls external APIs and downloads -large raster files). Watch the output — each step prints PASS or FAIL. +This takes about **six hours**. Watch the output — each step prints PASS or +FAIL. Per-step timings are in +[`server_quickref.md`](server_quickref.md#run-the-pipeline), which owns them. + +> **This figure said "25-45 minutes" until 2026-10-18, and that was the +> dangerous direction to be wrong in.** A run at the two-hour mark looks hung +> against a 45-minute estimate, and the reasonable response to a hung run is to +> kill it. The cost is not harvesting — harvest through compile is ~43 minutes. +> **The zarr export alone is ~5 hours and prints nothing while it works.** **Note:** The pipeline includes ACLED (8 features), GHS-POP population data (1 feature), GHS-BUILT-S built-up surface data @@ -434,7 +444,10 @@ crontab -l ### 4.5 Test manually +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log ``` @@ -457,8 +470,8 @@ See ADR-018 for the rationale. 2. Create a check (or verify the existing one): - **Name:** `views-datafactory pipeline refresh` - **Period:** 30 days (matches the `0 0 21 * *` cron) - - **Grace period:** 48 hours (pipeline runs 2-3 hours; 48h covers - delays, restarts, and weekend non-response) + - **Grace period:** 48 hours (the run takes ~6 hours; 48h still leaves + room for delays, restarts, and weekend non-response) - **Tags:** `views`, `datafactory`, `hetzner` 3. Copy the ping URL: `https://hc-ping.com/` @@ -852,8 +865,11 @@ fixed order: diverge a little further after **every** release, and `git log main..development` starts counting work that already shipped. -5. **Deploy** = update `~/.views-deploy-tag` on the server to the - new tag. The pipeline checks out whatever tag that file contains. +5. **Deploy** = three steps on the server, not one: the tag file, then + `git fetch --tags && git checkout `, then `uv sync`. Writing the + tag file alone is **not** deploying — that mistake left the host on + v1.10.0 for five days while every signal read v1.11.0 (C-343). + `docs/guides/server_quickref.md` holds the commands. **Why this order:** If you tag before bumping, the tag points at code with the old version string — wrong. If you tag on `development` @@ -871,48 +887,54 @@ first, promote second, tag third, back-merge fourth. > corrected, because a second copy of the procedure is what let the two guides drift apart > in the first place (#402). One procedure, one home. -**On the server** (SSH in): - -```bash -# Update the deploy tag file -echo 'vX.Y.Z' > ~/.views-deploy-tag -``` +**On the server** (SSH in): run **every block** in +`docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, then +`git fetch --tags && git checkout `, then `uv sync`, then the three +verification commands. The commands are there and not here on purpose. + +> **This section used to say "update `~/.views-deploy-tag`. That's it."** It was +> wrong, and it is the sentence C-343 is about: on 2026-08-08 the tag file read +> `v1.11.0`, `git describe` read `v1.10.0`, and the installed `views-frames` was +> still the frozen `1.0.0` — three sources of truth disagreeing, with a green +> status page and a pinging heartbeat and no error anywhere. The in-pipeline +> `git checkout "$DEPLOY_TAG"` does not rescue it: bash has already buffered +> `scripts/refresh_pipeline.sh`, so a change to that script lands one run later — +> a month, on a monthly cron — and `uv sync` is never invoked by the pipeline at +> all, so a dependency change never lands however many times it runs. C-343 was +> corrected in `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` +> in #431 and **this third copy was missed**, which is the same drift #402 is +> about. The commands live in the quickref and nowhere else; if this file and the +> quickref ever disagree, **the quickref wins.** -That's it. The next cron run (21st of the month) will automatically -use the new tag. If you want to apply it right now instead of waiting: +### How to roll back -```bash -# Run the pipeline manually (same command cron uses) -cd ~/views-datafactory -bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log -``` +If a new version breaks the pipeline, re-run the same three deploy steps with the +previous tag — a rollback is a deploy, and skipping `uv sync` on the way back +leaves the dependencies of the version you are abandoning. Tags are never +deleted by git, so every previous version stays available. -### How to roll back +Then run the pipeline manually to apply it immediately rather than waiting for +the next cron run. -If a new version breaks the pipeline: +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. ```bash -# On the server — point back to the previous version -echo 'vPREVIOUS' > ~/.views-deploy-tag +tmux new -s pipeline +sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' ``` -The next pipeline run will check out that tag and run its code -instead. The old version is always available — git never deletes tags. - -You can also run the pipeline manually to roll back immediately -instead of waiting for the next cron run. - ### How to check what version is deployed -```bash -# What version is the server configured to run? -cat ~/.views-deploy-tag +**Three sources, and they must agree** — the tag file, the checked-out tag, and +the installed distribution. Checking only the tag file is what C-343 was: it is a +statement of intent, not evidence. The commands are step 4 of +`docs/guides/server_quickref.md` §"Deploy a new version"; they are not repeated +here, because a second copy of them is how this section came to be wrong. -# What version is actually checked out right now? -git describe --tags --exact-match 2>/dev/null || git log --oneline -1 +For when the last pipeline run happened, `tail` the refresh log: -# When did the last pipeline run happen? -tail -5 logs/refresh.log +```bash +sudo -u views-deploy bash -c 'tail -5 ~/views-datafactory/logs/refresh.log' ``` ### Log rotation diff --git a/docs/guides/hetzner_deployment_log.md b/docs/guides/hetzner_deployment_log.md index 128be9f..b42ff3e 100644 --- a/docs/guides/hetzner_deployment_log.md +++ b/docs/guides/hetzner_deployment_log.md @@ -295,6 +295,12 @@ or mismatched path. **Total: approximately 2.5 hours for a clean run.** +> **These are March 2026 measurements and are no longer current.** The +> 2026-10-16 run took **5h50m**, with the zarr export alone at ~5h07m against a +> 3.1 GB store (this table has it at ~14 min against 1.8 GB). This is a dated +> log and the figures above are left as measured; current per-step timings live +> in [`server_quickref.md`](server_quickref.md#run-the-pipeline). + Subsequent runs are much faster because most data is cached (only new UCDP versions are fetched). diff --git a/docs/guides/server_operations.md b/docs/guides/server_operations.md index 324b824..898efba 100644 --- a/docs/guides/server_operations.md +++ b/docs/guides/server_operations.md @@ -10,9 +10,9 @@ For initial setup, see `hetzner_deployment_guide.md`. | What | Command | |------|---------| | SSH in | `ssh @204.168.219.108` | -| Check current version | `cat /home/views-deploy/.views-deploy-tag` | -| Set version to deploy | `echo "v1.2.3" \| sudo tee /home/views-deploy/.views-deploy-tag` | -| Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` | +| Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) | +| Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) | +| Run pipeline | **start `tmux` first** — see `server_quickref.md` §"Run the pipeline" | | Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` | | Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` | | List raw data | `sudo -u views-deploy ls -la /home/views-deploy/views-datafactory/data/raw/ucdp_annual/` | @@ -113,24 +113,24 @@ GitHub tag ### Deploy a new version -On your laptop: -```bash -# 1. Tag the release (from main branch) -git tag v1.2.4 -git push origin v1.2.4 -``` - -On the server: -```bash -# 2. SSH in -ssh @204.168.219.108 - -# 3. Set the new version -echo "v1.2.4" | sudo tee /home/views-deploy/.views-deploy-tag - -# 4. Run the pipeline -sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' -``` +**On your laptop:** the release ritual in `publishing_to_pypi.md` — bump, +promote to `main`, then cut a **GitHub Release**, which is what triggers the +publish workflow. A local `git tag && git push` does not, and both branches have +required pull requests since 2026-07-31, so the sequence that used to sit here +cannot be run any more. + +**On the server:** every block in `server_quickref.md` §"Deploy a new version" — +the tag file, `git fetch --tags && git checkout `, `uv sync`, then the three +verification commands. + +> **This section used to stop at the tag file**, and that is C-343: on 2026-08-08 +> the host ran v1.10.0 for five days while the tag file, and therefore every +> signal an operator checked, read v1.11.0. The pipeline's own `git checkout` +> does not rescue it — bash has already buffered `scripts/refresh_pipeline.sh`, +> so a change to that script lands one run later (a month, on a monthly cron), +> and `uv sync` is never invoked by the pipeline at all. The commands live in the +> quickref and nowhere else; **if this file and the quickref disagree, the +> quickref wins.** ### Run individual pipeline steps manually @@ -261,7 +261,10 @@ For JSON output (monitoring integration): add `--json` flag. If you suspect stale data, delete the assembled output and re-run: +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline sudo -u views-deploy bash -c 'cd ~/views-datafactory && rm -rf data/assembled data/compiled data/viewpoint data/consolidated' sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' ``` diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 401aa6e..f14ce24 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -12,23 +12,76 @@ All commands below assume you are on the server. ## Deploy a new version +**All four blocks. Step 1 alone is not a deploy (C-343)** — the pipeline's own +`git checkout` lands a change to `refresh_pipeline.sh` one run later, and never +runs `uv sync` at all. + ```bash -# Set version +# 1. Set version sudo -u views-deploy bash -c 'source ~/.profile && echo v1.2.6 > ~/.views-deploy-tag' -# Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) +# 2. Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && git fetch --tags && git checkout -- uv.lock && git checkout v1.2.6' -# Sync dependencies +# 3. Sync dependencies sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv sync' + +# 4. Verify — all three must name the same version. +# Every path goes inside bash -c: a bare ~ is expanded by YOUR shell +# before sudo runs, so it reads your home, not views-deploy's. +sudo -u views-deploy bash -c 'cat ~/.views-deploy-tag' +sudo -u views-deploy bash -c 'cd ~/views-datafactory && git describe --tags --exact-match 2>/dev/null || git log --oneline -1' +sudo -u views-deploy bash -c 'ls ~/views-datafactory/.venv/lib/python*/site-packages | grep views_datafactory' ``` +A **rollback is a deploy**: same four blocks, previous tag. + ## Run the pipeline +**Start tmux first. Not optional.** The run takes about six hours and SSH will +drop inside that window; without a multiplexer the drop sends SIGHUP and kills +the run. This is not hypothetical — see the note below. + +**Expect ~6 hours, and do not kill a run for "taking too long".** Measured +2026-10-16: **5h50m** (21,007s) with the assembly cached. The zarr export alone +runs for roughly five hours and prints nothing at all while it works — a silent +terminal is the normal state, not a hang. + +| Step | Time | Note | +|------|------|------| +| Harvest → consolidate → viewpoint → compile | ~43 min | all sources, most cached | +| Assemble grid | ~45 min | skipped when inputs are unchanged (ADR-041) | +| **Export zarr** | **~5h07m** | 18,427s for a 3.1 GB store — **88% of the run** | +| Export parquet | ~12 s | 29.5M rows | +| Health check + consumer contract | <1 min | | + +The assembly figure is ADR-041's; every other figure is from the 2026-10-16 run, +which had the assembly cached. A cold run that must rebuild the 37 GB grid is +therefore longer than 5h50m, not shorter. + ```bash -# Normal run (output streams to terminal + logs to file) +# 1. ALWAYS start tmux first +tmux new -s pipeline + +# 2. Then run, INSIDE tmux (output streams to terminal + logs to file) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' +# Detach: Ctrl-b then d Reattach: tmux attach -t pipeline +# Confirm you are inside tmux: a status bar sits at the BOTTOM of the +# terminal, e.g. "[pipeline] 0:sudo*". The shell prompt does NOT change, +# so the prompt alone never tells you. +``` + +> **2026-10-16: a run was killed this way.** `client_loop: send disconnect: +> Broken pipe` ended a foreground run after the 37 GB assembly had completed +> but before the zarr export. Nothing was corrupted — assembly renames from +> `grid.npy.tmp`, so `grid.npy` was whole, and `.exports_required` recorded the +> owed export — but `grid.zarr` was left three weeks stale while `grid.npy` was +> current, and the work had to be re-run. The lock needs no cleanup (`flock` is +> released by the kernel when the process dies), but the `/start` heartbeat ping +> is left dangling and healthchecks.io will flag it at the ~24h grace timeout. + +```bash # Force re-fetch: delete data first, then run normally # (there is no --force flag; the pipeline skips existing data by default) ``` @@ -77,17 +130,22 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv ## Check status ```bash -# Current deployed version -sudo -u views-deploy cat ~/.views-deploy-tag +# Every path goes inside bash -c — see the note in step 4 above. + +# Version the server is CONFIGURED to run (intent, not evidence — +# use all three checks in §"Deploy a new version" step 4 to confirm) +sudo -u views-deploy bash -c 'cat ~/.views-deploy-tag' # Last run duration -sudo -u views-deploy cat ~/views-datafactory/logs/pipeline_duration.json +sudo -u views-deploy bash -c 'cat ~/views-datafactory/logs/pipeline_duration.json' -# Last failure -sudo -u views-deploy cat ~/views-datafactory/logs/pipeline_failure.json 2>/dev/null || echo "No recent failure" +# Last failure. The fallback is INSIDE bash -c on purpose: outside it, an +# unreadable path makes `cat` fail and the || branch print "No recent +# failure" — a permission error reported as good news. +sudo -u views-deploy bash -c 'cat ~/views-datafactory/logs/pipeline_failure.json 2>/dev/null || echo "No recent failure"' # Raw data size -sudo -u views-deploy du -sh ~/views-datafactory/data/raw/*/ +sudo -u views-deploy bash -c 'du -sh ~/views-datafactory/data/raw/*/' # Zarr metadata (from laptop) curl -s http://204.168.219.108/grid.zarr/.zattrs | python3 -m json.tool diff --git a/pyproject.toml b/pyproject.toml index 020ae4a..cec4e03 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "views-datafactory" -version = "1.14.0" +version = "1.15.0" description = "Data factory for the VIEWS conflict forecasting platform — provenance, harvesting, consolidation, viewpoint construction, compilation, and assembly." authors = [ { name = "Simon Polichinel von der Maase", email = "simmaa@prio.org" }, diff --git a/reports/product_development_plan12.md b/reports/product_development_plan12.md index 31bd9cc..b33cf83 100644 --- a/reports/product_development_plan12.md +++ b/reports/product_development_plan12.md @@ -1,4 +1,4 @@ -# Product Development Plan v12 — current through v1.14, 11 registry sources, 79 Features, on PyPI +# Product Development Plan v12 — current through v1.15, 11 registry sources, 79 Features, on PyPI **Date:** 2026-06-29 (v1.10 addendum 2026-07-31) **Supersedes:** product_development_plan11.md (2026-05-08) @@ -7,6 +7,53 @@ --- +## v1.15 Addendum (2026-10-19) + +**v1.15.0 (the export's 5h07m gets a breakdown, and four rounds of C-343 land).** This release +contains **no library code at all** — `git diff --name-only v1.14.0 development -- src/` is empty, +so the wheel is byte-identical to v1.14.0 apart from its version metadata. That is not an oversight +and it is worth stating plainly, because it explains what the release is *for*: the server deploys +by **git tag** (ADR-022), and the two things this release carries — a script and a set of guides — +reach production no other way. A consumer installing from PyPI gets nothing new. The server gets +the only instrument that can answer #501. + +**What the server gets.** `scripts/export_zarr.py` now times four phases — `scan` (the `source_sums` +loop, one strided traversal of `grid.npy` per feature), `write` (`to_zarr` plus metadata +consolidation), `verify` (the ADR-011/C-144 round-trip integrity read) and `swap` (the `.tmp` +rename) — plus an explicit `other`, so the printed columns account for the whole of the `Time:` +line rather than a selected part of it. The breakdown prints to `logs/refresh.log` **and** persists +as `phase_seconds` in `data/assembled/export_ledger.jsonl`, which is the half that matters: logs +rotate, the ledger does not, and one measurement in a rotating log is not a trend. + +**Why it exists.** The 2026-10-16 production run took **21,007s (5h50m)**, of which the zarr export +alone was **18,427s (5h07m) — 88% of the pipeline**. ADR-041's Context had the export at ~57 +minutes. Nothing in the repo could attribute the other four hours to a phase, so nothing could be +optimised without guessing. The candidate explanation is pre-registered on #501 with a falsification +criterion committed **before** any production figure exists: `grid.npy` is C-contiguous with +features on the last axis, so `grid[:, :, :, i]` strides 316 bytes at F=79 and a 4 KB page yields +~13 useful float32s out of 1024 — **if `scan` comes in under ~25% of the export, that explanation is +refuted and abandoned.** The synthetic baseline, where 0.1 GB fits in RAM, puts `scan` at 10.1% — +the smallest named phase. The entire claim is that crossing the RAM boundary inverts that. + +**What else lands.** Four rounds of C-343 ("writing the deploy tag is not deploying"), each a +correction to the previous one's blind spot: #497 found four more copies of the one-step claim; +#498 found that `~` after `sudo` expands as the caller, so six verification commands read the wrong +home; #499 gave the procedure one home and added the tmux rule; #500 found that the authority +established in #499 carried a duration figure **nobody had ever measured** — seven places claimed a +pipeline duration, spanning 24×, and two of the wrong ones had been written two days earlier as the +fix. The through-line, now recorded: *a consolidation concentrates error as effectively as it +concentrates authority, so it must verify what it promotes, not merely absorb what it demotes.* + +**What this release does *not* fix.** The phase breakdown is *recorded* but nothing *compares* it +across runs — `check_health.py` has freshness SLOs and no duration SLO, so a second 5× regression +would still be invisible until someone read a log by hand. Flagged rather than built: one datapoint +does not justify a trend watcher, and the instrument that would have caught the first regression is +what this release ships. No change was made to chunking, compression, the slicing pattern or the +integrity check — each is a plausible remedy for a cause that is **not yet established**, and the +integrity check is an ADR-011/C-144 guarantee that must not be weakened to buy time. + +--- + ## v1.14 Addendum (2026-10-12) **v1.14.0 (ACLED pagination walks cursors — a provider-forced mechanism change).** ACLED deprecates diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index ab75ac6..c68683e 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3241,6 +3241,74 @@ Bash buffers the script and does not re-read it; git writes in place (inode unch **Remediated 2026-08-08** by running the three documented steps; `uv sync` moved views-frames 1.0.0 → 1.10.2 and views-datafactory 1.10.0 → 1.11.0, and `FeatureFrame` was imported on the host afterwards to confirm the ten-minor-version jump is clean. The entry stays **open** because nothing prevents recurrence. + +**Update 2026-10-13 — the fix reached two of six copies, and the four it missed are the ones an operator reads.** Found by the operator asking, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not, and the reason the question had to be asked is that #431 corrected `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` and stopped there. Four further copies of the one-step claim were still live: + +| Document | What it said | +|---|---| +| `docs/guides/hetzner_deployment_guide.md` §"How to deploy a new version" | the tag file, then *"That's it."* — **and this is the file `refresh_pipeline.sh` itself points to "for full detail"** | +| `docs/guides/server_operations.md` §"Deploy a new version" + the summary table | local `git tag && git push`, then the tag file, then run the pipeline | +| `docs/ADRs/022_tag_based_deployment_gate.md` §Consequences + §Deployment Procedure | *"One-line rollback"* as a claimed **benefit**; three-step deploy with no checkout and no `uv sync`; a Verify block listing the tag file and `git describe` but **not the installed distribution** — the one of the three that was actually wrong in August | +| `docs/guides/data_serving_guide.md` §9 | *"If the new version breaks, you write the old tag name back and you're running known-good code again."* | + +All four corrected, each pointing at `server_quickref.md` as the single operational home with "if they disagree, the quickref wins", and `server_quickref.md` grew the verification block it was missing (step 4: tag file, checked-out tag, **installed distribution version**). ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite — the Decision it records is sound; what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. + +**Update 2026-10-16 — the corrected procedure was run, and the verification step did not work.** v1.14.0 deployed cleanly: `git checkout` moved HEAD from `10c4d24` (v1.13.0) to `2b22a64` (v1.14.0), and `uv sync` reported `- views-datafactory==1.13.0` / `+ views-datafactory==1.14.0` — direct confirmation, on the box, that step 3 is load-bearing and that the tag file alone would have left 1.13.0 installed. + +Then step 4's first command failed: + +``` +$ sudo -u views-deploy cat ~/.views-deploy-tag +cat: /home/simmaa_prio/.views-deploy-tag: Permission denied +``` + +**`~` is expanded by the calling shell, before `sudo` runs.** Every such command reads the *operator's* home, not `views-deploy`'s. Six were live in the guides, two of them added by the 2026-10-13 correction above. All now wrapped in `bash -c '...'`, where the inner shell expands `~` as the target user. + +One of the six failed green rather than loud, which is why this is recorded here rather than left as a typo: + +``` +sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null || echo "No recent failure" +``` + +The `||` sat **outside** `bash -c`, so a permission error on the wrong home directory printed **"No recent failure"** — an unreadable path reported as good news, in the command an operator runs to ask whether the last pipeline run broke. Belongs to the **mechanisms that fail green** cluster alongside the rest of this entry. The fallback is now inside the quoted command. + +**Addendum 2026-10-16 — consolidating onto one authority moved content *out* of reach, not just into it.** The 2026-10-13 correction made `server_quickref.md` the single operational home and told every other document "if they disagree, the quickref wins." `hetzner_deployment_guide.md` carried a rule the quickref did not: **start `tmux` before a pipeline run**, stated at its line 30 with the reason ("pipeline runs 3-4 hours; SSH will drop without a multiplexer") and backed by a post-mortem section titled *"tmux saved us from the SSH drop"* (`2026-05-25_deployment_v1221.md`). Promoting the quickref silently demoted that rule. + +It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly completed but before the zarr export, leaving `grid.npy` current and `grid.zarr` three weeks stale. **Every designed defence held** — `assemble_grid.py` renames from `grid.npy.tmp` so the grid was whole, `.exports_required` recorded the owed export, `flock` released with the process, and `check_health.py` reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run and a window in which served data lagged the grid. + +Every interactive `refresh_pipeline.sh` invocation in the guides now starts with `tmux new -s pipeline`, and the quickref carries the rule, the dated evidence and the detail that catches people out: **tmux does not change the shell prompt**, so only the status bar tells you whether you are inside it. Cron invocations are deliberately left alone — cron has no controlling terminal and no SIGHUP to receive. + +**No new register entry.** The failure is loud, the defences worked, and the trigger would be perpetual ("whenever someone runs the pipeline"), which the trigger-quality gate rejects. What is registrable is the pattern above, and it belongs to this entry: a consolidation is only safe if the authority absorbs everything the demoted copies knew. Checking that was not part of the 2026-10-13 work and should have been. + +**Addendum 2026-10-18 — the new authority carried a number nobody had measured.** The 2026-10-16 addendum above records that consolidation can move content *out* of reach. This one records the complement, which is worse: **consolidation concentrates error as effectively as it concentrates authority.** When `server_quickref.md` was promoted on 2026-10-13 it was given "the run takes 3-4 hours." That figure was a guess. No measurement existed for it in the repo, and the document that stated it had just been declared the winner over every document that disagreed. + +Seven places claimed a pipeline duration, and they spanned a factor of 24: + +| Location | Claimed | Kind of document | +|---|---|---| +| `data_serving_guide.md:177` | 15-30 min | live guide | +| `hetzner_deployment_guide.md:192` | 25-45 min | live guide | +| `hetzner_deployment_log.md:296` | ~2.5 h | dated log (March 2026) | +| `hetzner_deployment_guide.md:466` | 2-3 h | live guide (justifies the 48h healthcheck grace) | +| `server_quickref.md:41` | 3-4 h | **the authority, set 2026-10-13** | +| `hetzner_deployment_guide.md:30` | 3-4 h | live guide | +| `ADR-041:12` | ~6 h | ADR | +| **2026-10-16 run, measured** | **5h50m** | — | + +The closest figure was in an **ADR about something else**. ADR-041 got it right incidentally, while arguing for content-addressed skipping. The document whose job is to be correct about it was wrong by 40%, and it was wrong because the correction that created its authority did not come with a measurement. + +**The direction of the error is the operational hazard, not the magnitude.** An over-estimate is harmless; an under-estimate licenses intervention on a healthy run. "25-45 minutes" tells an operator two hours in that the run is hung, and the reasonable response to a hung run is to kill it — reproducing the 2026-10-16 outage from a document instead of a dropped connection. Every stale figure also mis-attributed the cost to harvesting; harvest through compile is ~43 min, and the zarr export alone is ~5h07m, 88% of the run. + +Corrected by kind rather than uniformly, which is the part worth keeping: the four live guides now carry the measured figure and point at the quickref; **ADR-041 and the March deployment log were not rewritten** but carry dated amendments, because both are records of what was true when written and falsifying a record to make it current is a different defect from the one being fixed. + +**No new register entry, again.** The trigger would be perpetual. What is registrable belongs here: a consolidation must verify what it promotes, not only absorb what it demotes. Two rounds of this entry have now found that the 2026-10-13 work was sound in mechanism and unverified in content. + +**One finding escapes this entry and is not a documentation problem.** ADR-041 measured the zarr export at ~57 min in June 2026; it measured **18,427s (5h07m)** on 2026-10-16, a ~5x increase against a store that grew 1.8 GB → 3.1 GB. The cause is unestablished and deliberately not guessed at. Filed as **#501**, because it is a performance question requiring instrumentation rather than a figure requiring correction, and because ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not the ~103 min it claims. + +**The shape worth noticing.** C-343 is now on its fourth round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. + +**What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. + **The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix. Cross-ref: C-337 (the fix this failed to deliver), C-342 (the other lockfile-versus-reality entry from the same day), C-341 (gates that run nowhere), C-98/ADR-022 (the deployment gate this undermines). Part of the **mechanisms that fail green** cluster. GitHub: #421. diff --git a/scripts/export_zarr.py b/scripts/export_zarr.py index cf0aa69..96d143b 100644 --- a/scripts/export_zarr.py +++ b/scripts/export_zarr.py @@ -208,6 +208,12 @@ def main() -> int: t0 = time.monotonic() + # Where the export's wall time actually goes (#501). The total + # below has never been broken down, so the 5h07m measured on + # 2026-10-16 could not be attributed to a phase. One dict so the + # printed table and the ledger entry cannot drift apart. + phase_seconds: dict[str, float] = {} + # Load grid and metadata grid = np.load(grid_path, mmap_mode="r") pgids = np.load(pgids_path) @@ -235,7 +241,15 @@ def main() -> int: n_w, ) - # Build xarray Dataset — one variable per feature + # Build xarray Dataset — one variable per feature. + # + # Timed separately because this loop reads the whole grid once per + # feature. grid is [T, H, W, F] in C order, so grid[:, :, :, i] is a + # last-axis slice: it touches every page of the file while using + # 1/F of each page. With the grid larger than RAM the page cache + # cannot absorb that, and the sums here are computed only to be + # compared against the zarr read-back further down. + _t_scan = time.monotonic() data_vars = {} source_sums: dict[str, float] = {} for i, name in enumerate(feature_names): @@ -246,6 +260,7 @@ def main() -> int: feature_slice, ) print(f" {i:2d}: {name}") + phase_seconds["scan"] = time.monotonic() - _t_scan coords = { "time": time_steps, @@ -370,15 +385,22 @@ def main() -> int: shutil.rmtree(tmp_output) try: + # Second traversal of the grid: xarray pulls each feature_slice + # again to compress and write it. + _t_write = time.monotonic() ds.to_zarr( tmp_output, mode="w", encoding=encoding, ) zarr.consolidate_metadata(str(tmp_output)) + phase_seconds["write"] = time.monotonic() - _t_write print() print("Round-trip integrity check...") + # Reads back from the zarr store, not the memmap — fully + # decompresses every feature to re-sum it. + _t_verify = time.monotonic() store = zarr.open(str(tmp_output), mode="r") n_checked = 0 for name in feature_names: @@ -394,8 +416,12 @@ def main() -> int: return 1 n_checked += 1 del store + phase_seconds["verify"] = time.monotonic() - _t_verify print(f" {n_checked} features verified (sums match)") + # The C-144 atomic swap. Measured so it can be ruled out + # rather than assumed cheap. + _t_swap = time.monotonic() old_output = output.parent / (output.name + ".old") if old_output.exists(): shutil.rmtree(old_output) @@ -404,6 +430,7 @@ def main() -> int: os.rename(str(tmp_output), str(output)) if old_output.exists(): shutil.rmtree(old_output) + phase_seconds["swap"] = time.monotonic() - _t_swap except Exception: if tmp_output.exists(): shutil.rmtree(tmp_output) @@ -431,12 +458,27 @@ def main() -> int: f"{n_h} lat x {n_w} lon") print(f"Time: {elapsed:.1f}s") + # Phase breakdown (#501). "other" is whatever the named phases do + # not cover — grid load, coordinate construction, digests, the + # sentinel — and is reported rather than omitted so the columns + # always account for the whole of Time above. + print("Phases:") + _denom = max(elapsed, 1e-9) + for _label, _secs in phase_seconds.items(): + print(f" {_label:<7}{_secs:>10.1f}s {100 * _secs / _denom:>5.1f}%") + _other = elapsed - sum(phase_seconds.values()) + print(f" {'other':<7}{_other:>10.1f}s {100 * _other / _denom:>5.1f}%") + append_ledger_entry(_ledger_path, { "stage": "export", "outcome": "success", "source_digest": grid_digest, "ledger_version": LEDGER_VERSION, "digest_algorithm": DIGEST_SCHEME, + "elapsed_seconds": round(elapsed, 1), + "phase_seconds": { + k: round(v, 1) for k, v in phase_seconds.items() + }, }) _ledger_written = True diff --git a/tests/test_synthetic_pipeline.py b/tests/test_synthetic_pipeline.py index 97e5fc5..d530925 100644 --- a/tests/test_synthetic_pipeline.py +++ b/tests/test_synthetic_pipeline.py @@ -226,6 +226,60 @@ def test_zarr_source_digest(self, pipeline: dict) -> None: def test_sentinel_cleared(self, pipeline: dict) -> None: assert not (pipeline["assembled"] / ".exports_required").exists() + # ── #501: the export's phase breakdown ───────────────── + + @staticmethod + def _export_entry(pipeline: dict) -> dict: + """The successful export entry from the run's own ledger.""" + ledger = pipeline["assembled"] / "export_ledger.jsonl" + assert ledger.exists(), "export wrote no ledger" + entries = [ + json.loads(line) + for line in ledger.read_text().splitlines() + if line.strip() + ] + successes = [ + e for e in entries + if e.get("stage") == "export" and e.get("outcome") == "success" + ] + assert successes, f"no successful export entry in {entries}" + return successes[-1] + + def test_phase_timings_recorded(self, pipeline: dict) -> None: + """Every phase #501 asks about reaches the ledger, by name. + + The export took 5h07m on 2026-10-16 with no breakdown, so the + cost could not be attributed. If a phase label is dropped here, + that blindness returns silently — hence an exact-set assertion + rather than a subset one. + """ + entry = self._export_entry(pipeline) + assert "phase_seconds" in entry, entry + assert set(entry["phase_seconds"]) == { + "scan", "write", "verify", "swap" + }, entry["phase_seconds"] + + def test_phase_timings_are_nonnegative(self, pipeline: dict) -> None: + phases = self._export_entry(pipeline)["phase_seconds"] + assert all(v >= 0 for v in phases.values()), phases + + def test_phases_do_not_exceed_the_total(self, pipeline: dict) -> None: + """The phases must fit inside the reported total. + + They need not sum to it — 'other' covers load, coordinates and + digests. But a sum ABOVE the total means a timer spans work it + does not own, which would misattribute exactly the cost #501 + exists to locate. + """ + entry = self._export_entry(pipeline) + elapsed = entry["elapsed_seconds"] + assert elapsed > 0, entry + total = sum(entry["phase_seconds"].values()) + assert total <= elapsed + 0.5, ( + f"phases sum to {total}s but total is {elapsed}s: " + f"{entry['phase_seconds']}" + ) + # ── Consumer data loading ────────────────────────────────── diff --git a/uv.lock b/uv.lock index e0035d8..dab6e2a 100644 --- a/uv.lock +++ b/uv.lock @@ -1698,7 +1698,7 @@ wheels = [ [[package]] name = "views-datafactory" -version = "1.14.0" +version = "1.15.0" source = { editable = "." } dependencies = [ { name = "aiohttp" },