From 32f91db4120251c0743d54a5a23711327e219c42 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Tue, 13 Oct 2026 00:42:18 +0200 Subject: [PATCH 1/6] =?UTF-8?q?docs:=20writing=20the=20deploy=20tag=20is?= =?UTF-8?q?=20not=20deploying=20=E2=80=94=20four=20more=20copies=20said=20?= =?UTF-8?q?it=20did=20(C-343)=20(#497)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit C-343 was registered in #431 and corrected in scripts/refresh_pipeline.sh and docs/guides/server_quickref.md. Four further copies of the one-step claim were still live, and they are the ones an operator reads: - hetzner_deployment_guide.md said the tag file, then "That's it." This is the file refresh_pipeline.sh itself points to "for full detail". - server_operations.md gave a local `git tag && git push` (which cannot trigger the publish workflow, and cannot be run at all since branch protection) plus the tag file, with no checkout and no `uv sync`. - ADR-022 listed "One-line rollback" as a benefit of the gate, gave a three-step deploy with neither checkout nor `uv sync`, and offered a Verify block naming the tag file and `git describe` but not the installed distribution — the one of the three that was wrong in August. - data_serving_guide.md told newcomers that writing the old tag name back puts you on known-good code. server_quickref.md is now the single operational home and grew the verification it was missing: step 4 checks all three sources, including the installed version, which is the only one that reflects `uv sync`. The four others point there and say the quickref wins on disagreement. ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite: the Decision it records is sound, and what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. C-343 is updated rather than duplicated — same risk, new evidence that the fix reached two of six copies. It stays open; the pre-flight instrument that would catch a seventh copy is still #424. Found because the operator asked, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- docs/ADRs/022_tag_based_deployment_gate.md | 32 +++++----- docs/guides/data_serving_guide.md | 23 ++++--- docs/guides/hetzner_deployment_guide.md | 72 ++++++++++++---------- docs/guides/server_operations.md | 40 ++++++------ docs/guides/server_quickref.md | 20 ++++-- reports/technical_risk_register.md | 14 +++++ 6 files changed, 120 insertions(+), 81 deletions(-) diff --git a/docs/ADRs/022_tag_based_deployment_gate.md b/docs/ADRs/022_tag_based_deployment_gate.md index 37a8c1f4..19e030e0 100644 --- a/docs/ADRs/022_tag_based_deployment_gate.md +++ b/docs/ADRs/022_tag_based_deployment_gate.md @@ -51,7 +51,7 @@ Normally, `HEAD` points to a branch (e.g., `main`), and the branch points to a c This is **normal and expected on the server**. It means: - The server is looking at a frozen snapshot, not a moving branch. - No `git pull` or `git push` can accidentally change what's checked out. -- The only way to change the running version is to update `~/.views-deploy-tag`. +- Nothing changes the running version except an operator deploying — and deploying is the whole procedure below, not just the tag file. On developer machines, you work on branches as usual. Detached HEAD is only relevant on the deployment server. @@ -112,7 +112,7 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ### Positive - **Explicit version control.** The operator decides exactly what runs. No surprises from pushed commits. -- **One-line rollback.** `echo 'v1.0.0' > ~/.views-deploy-tag` reverts to the previous release. +- **Cheap rollback.** The previous release is always available — git never deletes tags — so reverting is the same short procedure as deploying, run with the earlier tag. (It is **not** one line; see the amendment below.) - **Audit trail.** The deploy tag file, git log, and provenance ledger together show what version produced what data. - **Safe for multiple maintainers.** Two people can push to `development` without affecting the server. - **No infrastructure dependencies.** No Docker registry, no CI/CD pipeline, no deployment service. Just git tags and a text file. @@ -127,26 +127,24 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ## Deployment Procedure -**Deploy a new version:** +**On your laptop:** the release ritual in `docs/guides/publishing_to_pypi.md` — bump, promote to `main`, cut a **GitHub Release** (a local `git tag && git push` does not trigger the publish workflow, and both branches have required pull requests since 2026-07-31). -1. On your laptop: `git tag v1.2.0 && git push --tags` -2. On the server: `echo 'v1.2.0' > ~/.views-deploy-tag` -3. Wait for cron (21st of month) or run manually +**On the server:** every block of `docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, `git fetch --tags && git checkout `, `uv sync`, then verification against three sources: the tag file, the checked-out tag, and the **installed distribution version**. A rollback is the same procedure with the earlier tag. -**Roll back:** +The commands are deliberately not reproduced here. This ADR records *why* the gate exists; the quickref is its single operational home, and **if they disagree, the quickref wins.** -1. On the server: `echo 'v1.0.0' > ~/.views-deploy-tag` -2. Run the pipeline manually or wait for cron +### Amendment 2026-10-13 — writing the tag file is not deploying (C-343) -**Verify:** +This section previously gave deploying as three steps — local tag, tag file, wait for cron — and rollback as one. Both were wrong, and the error was not theoretical: on 2026-08-08 the production host ran v1.10.0 for five days while the tag file read v1.11.0, with a green status page, a pinging heartbeat, and no error at any point. v1.11.0 existed solely to raise the `views-frames` floor, and the floor on the box was still the frozen 1.0.0. -```bash -cat ~/.views-deploy-tag # configured version -git describe --tags --exact-match # checked-out version -tail -5 logs/refresh.log # last run output -``` +The gate described in **Decision** is unaffected and still does what it claims: the pipeline refuses to start without a valid tag, and checks that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. It does not, for two reasons established by experiment rather than by reading: + +1. Bash buffers `scripts/refresh_pipeline.sh` before the gate's `git checkout` runs, so a change to that script takes effect on the **next** run — one month, on a monthly cron. +2. The pipeline never invokes `uv sync`, so a dependency change never lands at all, however many times it runs. + +The **Verify** block was also incomplete in the way that mattered: it listed the tag file and `git describe` but not the installed distribution, which is the only one of the three that reflects `uv sync`. In August it was the installed version that was wrong. -Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, section "Deployment and releases." +C-343 is **open**: the three guides and this ADR now agree, but nothing mechanically prevents the procedure from being skipped. The proposed instrument — a pre-flight check comparing the deploy tag against the HEAD the script *started* from, failing loud per ADR-011 rather than self-correcting one run late — is tracked in #424 and not built. --- @@ -154,7 +152,7 @@ Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, se - ADR-011 (Fail Loud, No Stale Data Serving) — justifies fail-loud on misconfiguration - ADR-018 (Operational Resilience Policy) — operator-mediated bounded staleness -- `scripts/refresh_pipeline.sh` lines 76-98 — implementation +- `scripts/refresh_pipeline.sh`, the "Deployment gate" block — implementation - `reports/archive/technical_risk_register_resolved.md` C-98 — risk that motivated this ADR - `docs/guides/hetzner_deployment_guide.md` — operational procedures - `docs/guides/data_serving_guide.md` section 9 — conceptual explanation diff --git a/docs/guides/data_serving_guide.md b/docs/guides/data_serving_guide.md index 71cf3a3b..8c38a76d 100644 --- a/docs/guides/data_serving_guide.md +++ b/docs/guides/data_serving_guide.md @@ -250,13 +250,22 @@ notice for days. With the tag system: the server keeps running the old working version until you *explicitly* tell it to switch. You test the new version, -tag it, and only then update the server. If the new version breaks, -you write the old tag name back and you're running known-good code -again. - -**The full deployment procedure** (create a tag, push it, update the -server file, rollback if needed) is documented in -`hetzner_deployment_guide.md` under "Deployment and releases". +release it, and only then deploy it. If the new version breaks, you +deploy the previous tag and you're running known-good code again. + +**One thing this explanation makes tempting, and it is wrong.** Because +the pipeline checks the tag out for you, it looks as though changing the +tag file *is* deploying. It is not. The script is already running when it +does that checkout, so a change to the script itself only takes effect +the run after — a month, on a monthly cron — and the script never +installs dependencies, so a dependency change never lands at all. That +misreading left the server on v1.10.0 for five days with every signal +reading v1.11.0 (C-343, August 2026). + +**The deployment procedure** — tag file, checkout, `uv sync`, then verify +three sources agree — lives in `server_quickref.md` under "Deploy a new +version", and only there. A rollback is the same procedure with the +earlier tag. The architectural rationale (why tags instead of branch tracking, Docker, or CI/CD) is in **ADR-022**. diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 6cd8a91a..4fc2bc12 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -852,8 +852,11 @@ fixed order: diverge a little further after **every** release, and `git log main..development` starts counting work that already shipped. -5. **Deploy** = update `~/.views-deploy-tag` on the server to the - new tag. The pipeline checks out whatever tag that file contains. +5. **Deploy** = three steps on the server, not one: the tag file, then + `git fetch --tags && git checkout `, then `uv sync`. Writing the + tag file alone is **not** deploying — that mistake left the host on + v1.10.0 for five days while every signal read v1.11.0 (C-343). + `docs/guides/server_quickref.md` holds the commands. **Why this order:** If you tag before bumping, the tag points at code with the old version string — wrong. If you tag on `development` @@ -871,48 +874,51 @@ first, promote second, tag third, back-merge fourth. > corrected, because a second copy of the procedure is what let the two guides drift apart > in the first place (#402). One procedure, one home. -**On the server** (SSH in): - -```bash -# Update the deploy tag file -echo 'vX.Y.Z' > ~/.views-deploy-tag -``` - -That's it. The next cron run (21st of the month) will automatically -use the new tag. If you want to apply it right now instead of waiting: - -```bash -# Run the pipeline manually (same command cron uses) -cd ~/views-datafactory -bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log -``` +**On the server** (SSH in): run **every block** in +`docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, then +`git fetch --tags && git checkout `, then `uv sync`, then the three +verification commands. The commands are there and not here on purpose. + +> **This section used to say "update `~/.views-deploy-tag`. That's it."** It was +> wrong, and it is the sentence C-343 is about: on 2026-08-08 the tag file read +> `v1.11.0`, `git describe` read `v1.10.0`, and the installed `views-frames` was +> still the frozen `1.0.0` — three sources of truth disagreeing, with a green +> status page and a pinging heartbeat and no error anywhere. The in-pipeline +> `git checkout "$DEPLOY_TAG"` does not rescue it: bash has already buffered +> `scripts/refresh_pipeline.sh`, so a change to that script lands one run later — +> a month, on a monthly cron — and `uv sync` is never invoked by the pipeline at +> all, so a dependency change never lands however many times it runs. C-343 was +> corrected in `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` +> in #431 and **this third copy was missed**, which is the same drift #402 is +> about. The commands live in the quickref and nowhere else; if this file and the +> quickref ever disagree, **the quickref wins.** ### How to roll back -If a new version breaks the pipeline: +If a new version breaks the pipeline, re-run the same three deploy steps with the +previous tag — a rollback is a deploy, and skipping `uv sync` on the way back +leaves the dependencies of the version you are abandoning. Tags are never +deleted by git, so every previous version stays available. + +Then run the pipeline manually to apply it immediately rather than waiting for +the next cron run: ```bash -# On the server — point back to the previous version -echo 'vPREVIOUS' > ~/.views-deploy-tag +sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' ``` -The next pipeline run will check out that tag and run its code -instead. The old version is always available — git never deletes tags. - -You can also run the pipeline manually to roll back immediately -instead of waiting for the next cron run. - ### How to check what version is deployed -```bash -# What version is the server configured to run? -cat ~/.views-deploy-tag +**Three sources, and they must agree** — the tag file, the checked-out tag, and +the installed distribution. Checking only the tag file is what C-343 was: it is a +statement of intent, not evidence. The commands are step 4 of +`docs/guides/server_quickref.md` §"Deploy a new version"; they are not repeated +here, because a second copy of them is how this section came to be wrong. -# What version is actually checked out right now? -git describe --tags --exact-match 2>/dev/null || git log --oneline -1 +For when the last pipeline run happened, `tail` the refresh log: -# When did the last pipeline run happen? -tail -5 logs/refresh.log +```bash +sudo -u views-deploy tail -5 ~/views-datafactory/logs/refresh.log ``` ### Log rotation diff --git a/docs/guides/server_operations.md b/docs/guides/server_operations.md index 324b8247..33083303 100644 --- a/docs/guides/server_operations.md +++ b/docs/guides/server_operations.md @@ -10,8 +10,8 @@ For initial setup, see `hetzner_deployment_guide.md`. | What | Command | |------|---------| | SSH in | `ssh @204.168.219.108` | -| Check current version | `cat /home/views-deploy/.views-deploy-tag` | -| Set version to deploy | `echo "v1.2.3" \| sudo tee /home/views-deploy/.views-deploy-tag` | +| Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) | +| Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) | | Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` | | Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` | | Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` | @@ -113,24 +113,24 @@ GitHub tag ### Deploy a new version -On your laptop: -```bash -# 1. Tag the release (from main branch) -git tag v1.2.4 -git push origin v1.2.4 -``` - -On the server: -```bash -# 2. SSH in -ssh @204.168.219.108 - -# 3. Set the new version -echo "v1.2.4" | sudo tee /home/views-deploy/.views-deploy-tag - -# 4. Run the pipeline -sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' -``` +**On your laptop:** the release ritual in `publishing_to_pypi.md` — bump, +promote to `main`, then cut a **GitHub Release**, which is what triggers the +publish workflow. A local `git tag && git push` does not, and both branches have +required pull requests since 2026-07-31, so the sequence that used to sit here +cannot be run any more. + +**On the server:** every block in `server_quickref.md` §"Deploy a new version" — +the tag file, `git fetch --tags && git checkout `, `uv sync`, then the three +verification commands. + +> **This section used to stop at the tag file**, and that is C-343: on 2026-08-08 +> the host ran v1.10.0 for five days while the tag file, and therefore every +> signal an operator checked, read v1.11.0. The pipeline's own `git checkout` +> does not rescue it — bash has already buffered `scripts/refresh_pipeline.sh`, +> so a change to that script lands one run later (a month, on a monthly cron), +> and `uv sync` is never invoked by the pipeline at all. The commands live in the +> quickref and nowhere else; **if this file and the quickref disagree, the +> quickref wins.** ### Run individual pipeline steps manually diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 401aa6e5..0b5627cc 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -12,17 +12,28 @@ All commands below assume you are on the server. ## Deploy a new version +**All four blocks. Step 1 alone is not a deploy (C-343)** — the pipeline's own +`git checkout` lands a change to `refresh_pipeline.sh` one run later, and never +runs `uv sync` at all. + ```bash -# Set version +# 1. Set version sudo -u views-deploy bash -c 'source ~/.profile && echo v1.2.6 > ~/.views-deploy-tag' -# Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) +# 2. Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && git fetch --tags && git checkout -- uv.lock && git checkout v1.2.6' -# Sync dependencies +# 3. Sync dependencies sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv sync' + +# 4. Verify — all three must name the same version +sudo -u views-deploy cat ~/.views-deploy-tag +sudo -u views-deploy bash -c 'cd ~/views-datafactory && git describe --tags --exact-match 2>/dev/null || git log --oneline -1' +sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv run python -c "import importlib.metadata as m; print(m.version(\"views-datafactory\"))"' ``` +A **rollback is a deploy**: same four blocks, previous tag. + ## Run the pipeline ```bash @@ -77,7 +88,8 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv ## Check status ```bash -# Current deployed version +# Version the server is CONFIGURED to run (intent, not evidence — +# use all three checks in §"Deploy a new version" step 4 to confirm) sudo -u views-deploy cat ~/.views-deploy-tag # Last run duration diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index ab75ac67..e248cb07 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3241,6 +3241,20 @@ Bash buffers the script and does not re-read it; git writes in place (inode unch **Remediated 2026-08-08** by running the three documented steps; `uv sync` moved views-frames 1.0.0 → 1.10.2 and views-datafactory 1.10.0 → 1.11.0, and `FeatureFrame` was imported on the host afterwards to confirm the ten-minor-version jump is clean. The entry stays **open** because nothing prevents recurrence. + +**Update 2026-10-13 — the fix reached two of six copies, and the four it missed are the ones an operator reads.** Found by the operator asking, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not, and the reason the question had to be asked is that #431 corrected `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` and stopped there. Four further copies of the one-step claim were still live: + +| Document | What it said | +|---|---| +| `docs/guides/hetzner_deployment_guide.md` §"How to deploy a new version" | the tag file, then *"That's it."* — **and this is the file `refresh_pipeline.sh` itself points to "for full detail"** | +| `docs/guides/server_operations.md` §"Deploy a new version" + the summary table | local `git tag && git push`, then the tag file, then run the pipeline | +| `docs/ADRs/022_tag_based_deployment_gate.md` §Consequences + §Deployment Procedure | *"One-line rollback"* as a claimed **benefit**; three-step deploy with no checkout and no `uv sync`; a Verify block listing the tag file and `git describe` but **not the installed distribution** — the one of the three that was actually wrong in August | +| `docs/guides/data_serving_guide.md` §9 | *"If the new version breaks, you write the old tag name back and you're running known-good code again."* | + +All four corrected, each pointing at `server_quickref.md` as the single operational home with "if they disagree, the quickref wins", and `server_quickref.md` grew the verification block it was missing (step 4: tag file, checked-out tag, **installed distribution version**). ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite — the Decision it records is sound; what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. + +**What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. + **The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix. Cross-ref: C-337 (the fix this failed to deliver), C-342 (the other lockfile-versus-reality entry from the same day), C-341 (gates that run nowhere), C-98/ADR-022 (the deployment gate this undermines). Part of the **mechanisms that fail green** cluster. GitHub: #421. From 4a935a3a93fcffe423509a09d140fcbcc502c94f Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Fri, 16 Oct 2026 16:42:22 +0200 Subject: [PATCH 2/6] =?UTF-8?q?docs:=20`~`=20after=20sudo=20expands=20as?= =?UTF-8?q?=20the=20caller=20=E2=80=94=20six=20verification=20commands=20r?= =?UTF-8?q?ead=20the=20wrong=20home=20(C-343)=20(#498)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by running the corrected deploy procedure on the production host. v1.14.0 deployed cleanly, then step 4's first check failed: $ sudo -u views-deploy cat ~/.views-deploy-tag cat: /home/simmaa_prio/.views-deploy-tag: Permission denied The calling shell expands `~` before sudo runs, so the command reads the operator's home rather than views-deploy's. Six such commands were live in the guides; two were added by the C-343 correction in #497. All are now wrapped in `bash -c '...'`, where the inner shell expands `~` as the target user. One of the six failed green rather than loud: sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null \ || echo "No recent failure" The `||` sat outside `bash -c`, so a permission error on the wrong home printed "No recent failure" — in the command an operator runs to ask whether the last pipeline run broke. The fallback is now inside the quoted command. Step 4's third check also drops the escaped-quote python one-liner for `ls .venv/.../site-packages | grep views_datafactory`, which needs no quoting and prints the dist-info directory — the same evidence form C-343 used in August. C-343 updated with the deploy outcome: `uv sync` reported 1.13.0 -> 1.14.0 on the box, which is direct confirmation that step 3 is load-bearing and the tag file alone would have left 1.13.0 installed. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- docs/guides/hetzner_deployment_guide.md | 2 +- docs/guides/server_quickref.md | 22 ++++++++++++++-------- reports/technical_risk_register.md | 21 +++++++++++++++++++++ 3 files changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 4fc2bc12..431eef1e 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -918,7 +918,7 @@ here, because a second copy of them is how this section came to be wrong. For when the last pipeline run happened, `tail` the refresh log: ```bash -sudo -u views-deploy tail -5 ~/views-datafactory/logs/refresh.log +sudo -u views-deploy bash -c 'tail -5 ~/views-datafactory/logs/refresh.log' ``` ### Log rotation diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 0b5627cc..7364ec9e 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -26,10 +26,12 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && git # 3. Sync dependencies sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv sync' -# 4. Verify — all three must name the same version -sudo -u views-deploy cat ~/.views-deploy-tag +# 4. Verify — all three must name the same version. +# Every path goes inside bash -c: a bare ~ is expanded by YOUR shell +# before sudo runs, so it reads your home, not views-deploy's. +sudo -u views-deploy bash -c 'cat ~/.views-deploy-tag' sudo -u views-deploy bash -c 'cd ~/views-datafactory && git describe --tags --exact-match 2>/dev/null || git log --oneline -1' -sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv run python -c "import importlib.metadata as m; print(m.version(\"views-datafactory\"))"' +sudo -u views-deploy bash -c 'ls ~/views-datafactory/.venv/lib/python*/site-packages | grep views_datafactory' ``` A **rollback is a deploy**: same four blocks, previous tag. @@ -88,18 +90,22 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv ## Check status ```bash +# Every path goes inside bash -c — see the note in step 4 above. + # Version the server is CONFIGURED to run (intent, not evidence — # use all three checks in §"Deploy a new version" step 4 to confirm) -sudo -u views-deploy cat ~/.views-deploy-tag +sudo -u views-deploy bash -c 'cat ~/.views-deploy-tag' # Last run duration -sudo -u views-deploy cat ~/views-datafactory/logs/pipeline_duration.json +sudo -u views-deploy bash -c 'cat ~/views-datafactory/logs/pipeline_duration.json' -# Last failure -sudo -u views-deploy cat ~/views-datafactory/logs/pipeline_failure.json 2>/dev/null || echo "No recent failure" +# Last failure. The fallback is INSIDE bash -c on purpose: outside it, an +# unreadable path makes `cat` fail and the || branch print "No recent +# failure" — a permission error reported as good news. +sudo -u views-deploy bash -c 'cat ~/views-datafactory/logs/pipeline_failure.json 2>/dev/null || echo "No recent failure"' # Raw data size -sudo -u views-deploy du -sh ~/views-datafactory/data/raw/*/ +sudo -u views-deploy bash -c 'du -sh ~/views-datafactory/data/raw/*/' # Zarr metadata (from laptop) curl -s http://204.168.219.108/grid.zarr/.zattrs | python3 -m json.tool diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index e248cb07..489cbe27 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3253,6 +3253,27 @@ Bash buffers the script and does not re-read it; git writes in place (inode unch All four corrected, each pointing at `server_quickref.md` as the single operational home with "if they disagree, the quickref wins", and `server_quickref.md` grew the verification block it was missing (step 4: tag file, checked-out tag, **installed distribution version**). ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite — the Decision it records is sound; what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. +**Update 2026-10-16 — the corrected procedure was run, and the verification step did not work.** v1.14.0 deployed cleanly: `git checkout` moved HEAD from `10c4d24` (v1.13.0) to `2b22a64` (v1.14.0), and `uv sync` reported `- views-datafactory==1.13.0` / `+ views-datafactory==1.14.0` — direct confirmation, on the box, that step 3 is load-bearing and that the tag file alone would have left 1.13.0 installed. + +Then step 4's first command failed: + +``` +$ sudo -u views-deploy cat ~/.views-deploy-tag +cat: /home/simmaa_prio/.views-deploy-tag: Permission denied +``` + +**`~` is expanded by the calling shell, before `sudo` runs.** Every such command reads the *operator's* home, not `views-deploy`'s. Six were live in the guides, two of them added by the 2026-10-13 correction above. All now wrapped in `bash -c '...'`, where the inner shell expands `~` as the target user. + +One of the six failed green rather than loud, which is why this is recorded here rather than left as a typo: + +``` +sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null || echo "No recent failure" +``` + +The `||` sat **outside** `bash -c`, so a permission error on the wrong home directory printed **"No recent failure"** — an unreadable path reported as good news, in the command an operator runs to ask whether the last pipeline run broke. Belongs to the **mechanisms that fail green** cluster alongside the rest of this entry. The fallback is now inside the quoted command. + +**The shape worth noticing.** C-343 is now on its third round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. + **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. **The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix. From 024c5c5e7c48583b60665fd03b05fd581bbae320 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Fri, 16 Oct 2026 20:01:10 +0200 Subject: [PATCH 3/6] =?UTF-8?q?docs:=20start=20tmux=20before=20a=20pipelin?= =?UTF-8?q?e=20run=20=E2=80=94=20the=20rule=20the=20quickref=20didn't=20ca?= =?UTF-8?q?rry=20(C-343)=20(#499)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-10-13 consolidation made server_quickref.md the single operational home and told every other guide "if they disagree, the quickref wins." hetzner_deployment_guide.md carried a rule the quickref did not — start tmux before a pipeline run, stated at its line 30 with the reason, and backed by a post-mortem section called "tmux saved us from the SSH drop". Promoting the quickref silently demoted that rule. It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly had completed but before the zarr export, leaving grid.npy current and grid.zarr three weeks stale. Every designed defence held: assemble_grid.py renames from grid.npy.tmp so the grid was whole, .exports_required recorded the owed export, flock released with the process, and check_health.py reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run. Every interactive refresh_pipeline.sh invocation in the guides now opens with `tmux new -s pipeline`: - server_quickref.md §"Run the pipeline" — the rule, the dated evidence, detach/reattach, and the detail that catches people out: tmux does not change the shell prompt, so only the bottom status bar tells you whether you are inside it - hetzner_deployment_guide.md §2.1, §4.5, and the rollback section added in #497 - server_operations.md summary table and the full-rebuild procedure Cron invocations are deliberately unchanged — cron has no controlling terminal and no SIGHUP to receive. The ~/.profile wrong-vs-right example in server_operations.md is also unchanged; adding tmux would obscure the point it exists to make. C-343 gets the pattern rather than a new entry: a consolidation is only safe if the new authority absorbs everything the demoted copies knew, and checking that was not part of the 2026-10-13 work. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- docs/guides/hetzner_deployment_guide.md | 11 ++++++++++- docs/guides/server_operations.md | 5 ++++- docs/guides/server_quickref.md | 25 ++++++++++++++++++++++++- reports/technical_risk_register.md | 8 ++++++++ 4 files changed, 46 insertions(+), 3 deletions(-) diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 431eef1e..6f69508e 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -181,7 +181,10 @@ source ~/.profile ### 2.1 Run the full pipeline +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh ``` @@ -434,7 +437,10 @@ crontab -l ### 4.5 Test manually +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log ``` @@ -901,9 +907,12 @@ leaves the dependencies of the version you are abandoning. Tags are never deleted by git, so every previous version stays available. Then run the pipeline manually to apply it immediately rather than waiting for -the next cron run: +the next cron run. + +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. ```bash +tmux new -s pipeline sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' ``` diff --git a/docs/guides/server_operations.md b/docs/guides/server_operations.md index 33083303..898efba0 100644 --- a/docs/guides/server_operations.md +++ b/docs/guides/server_operations.md @@ -12,7 +12,7 @@ For initial setup, see `hetzner_deployment_guide.md`. | SSH in | `ssh @204.168.219.108` | | Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) | | Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) | -| Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` | +| Run pipeline | **start `tmux` first** — see `server_quickref.md` §"Run the pipeline" | | Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` | | Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` | | List raw data | `sudo -u views-deploy ls -la /home/views-deploy/views-datafactory/data/raw/ucdp_annual/` | @@ -261,7 +261,10 @@ For JSON output (monitoring integration): add `--json` flag. If you suspect stale data, delete the assembled output and re-run: +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline sudo -u views-deploy bash -c 'cd ~/views-datafactory && rm -rf data/assembled data/compiled data/viewpoint data/consolidated' sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' ``` diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 7364ec9e..4aac18be 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -38,10 +38,33 @@ A **rollback is a deploy**: same four blocks, previous tag. ## Run the pipeline +**Start tmux first. Not optional.** The run takes 3-4 hours and SSH will drop +inside that window; without a multiplexer the drop sends SIGHUP and kills the +run. This is not hypothetical — see the note below. + ```bash -# Normal run (output streams to terminal + logs to file) +# 1. ALWAYS start tmux first +tmux new -s pipeline + +# 2. Then run, INSIDE tmux (output streams to terminal + logs to file) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' +# Detach: Ctrl-b then d Reattach: tmux attach -t pipeline +# Confirm you are inside tmux: a status bar sits at the BOTTOM of the +# terminal, e.g. "[pipeline] 0:sudo*". The shell prompt does NOT change, +# so the prompt alone never tells you. +``` + +> **2026-10-16: a run was killed this way.** `client_loop: send disconnect: +> Broken pipe` ended a foreground run after the 37 GB assembly had completed +> but before the zarr export. Nothing was corrupted — assembly renames from +> `grid.npy.tmp`, so `grid.npy` was whole, and `.exports_required` recorded the +> owed export — but `grid.zarr` was left three weeks stale while `grid.npy` was +> current, and the work had to be re-run. The lock needs no cleanup (`flock` is +> released by the kernel when the process dies), but the `/start` heartbeat ping +> is left dangling and healthchecks.io will flag it at the ~24h grace timeout. + +```bash # Force re-fetch: delete data first, then run normally # (there is no --force flag; the pipeline skips existing data by default) ``` diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 489cbe27..eba91c75 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3272,6 +3272,14 @@ sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null || echo "No rec The `||` sat **outside** `bash -c`, so a permission error on the wrong home directory printed **"No recent failure"** — an unreadable path reported as good news, in the command an operator runs to ask whether the last pipeline run broke. Belongs to the **mechanisms that fail green** cluster alongside the rest of this entry. The fallback is now inside the quoted command. +**Addendum 2026-10-16 — consolidating onto one authority moved content *out* of reach, not just into it.** The 2026-10-13 correction made `server_quickref.md` the single operational home and told every other document "if they disagree, the quickref wins." `hetzner_deployment_guide.md` carried a rule the quickref did not: **start `tmux` before a pipeline run**, stated at its line 30 with the reason ("pipeline runs 3-4 hours; SSH will drop without a multiplexer") and backed by a post-mortem section titled *"tmux saved us from the SSH drop"* (`2026-05-25_deployment_v1221.md`). Promoting the quickref silently demoted that rule. + +It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly completed but before the zarr export, leaving `grid.npy` current and `grid.zarr` three weeks stale. **Every designed defence held** — `assemble_grid.py` renames from `grid.npy.tmp` so the grid was whole, `.exports_required` recorded the owed export, `flock` released with the process, and `check_health.py` reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run and a window in which served data lagged the grid. + +Every interactive `refresh_pipeline.sh` invocation in the guides now starts with `tmux new -s pipeline`, and the quickref carries the rule, the dated evidence and the detail that catches people out: **tmux does not change the shell prompt**, so only the status bar tells you whether you are inside it. Cron invocations are deliberately left alone — cron has no controlling terminal and no SIGHUP to receive. + +**No new register entry.** The failure is loud, the defences worked, and the trigger would be perpetual ("whenever someone runs the pipeline"), which the trigger-quality gate rejects. What is registrable is the pattern above, and it belongs to this entry: a consolidation is only safe if the authority absorbs everything the demoted copies knew. Checking that was not part of the 2026-10-13 work and should have been. + **The shape worth noticing.** C-343 is now on its third round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. From 8dfcdf98a16ebbd2acffa4d8112c3fc58960cae7 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 18 Oct 2026 18:46:50 +0200 Subject: [PATCH 4/6] =?UTF-8?q?docs:=20seven=20places=20claimed=20a=20pipe?= =?UTF-8?q?line=20duration,=20spanning=2024x=20=E2=80=94=20the=20authority?= =?UTF-8?q?'s=20figure=20was=20never=20measured=20(C-343)=20(#500)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-10-16 run took 5h50m. The guides claimed 15-30 min, 25-45 min, ~2.5h, 2-3h, 3-4h and ~6h, and the closest figure was in an ADR about something else. `server_quickref.md` was made the single operational authority on 2026-10-13 and given "3-4 hours" — a guess, with no measurement anywhere in the repo, promoted over every document that disagreed. The complement of the tmux finding: consolidation concentrates error as effectively as it concentrates authority. The direction of the error is the hazard. An under-estimate licenses intervention on a healthy run: "25-45 minutes" tells an operator two hours in that the run is hung, and killing a hung run is the reasonable response — reproducing the 2026-10-16 outage from a document instead of a dropped SSH connection. Every stale figure also blamed harvesting; harvest through compile is ~43 min and the zarr export alone is ~5h07m, 88% of the run. Corrected by document kind, not uniformly: - four live guides carry the measured figure and point at the quickref, which owns the per-step breakdown - ADR-041 and the March 2026 deployment log are NOT rewritten — both are records of what was true when written, and falsifying a record to make it current is a different defect from the one being fixed. Both carry dated amendments instead. Also corrected: ADR-041's healthcheck-grace rationale derived 48h from "2-3 hours"; the conclusion survives at ~6h, the reasoning did not. One finding escapes the docs fix and is filed separately: the zarr export was ~57 min in June 2026 and 18,427s (5h07m) on 2026-10-16, a ~5x increase against a store that grew 1.8 -> 3.1 GB. Cause unestablished and deliberately not guessed at. ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not ~103. Verification: ruff clean, validate_docs.sh PASSED, 48 register-reading and citation tests plus 38 deployment-doc guards green. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- docs/ADRs/041_content_addressed_skip.md | 15 ++++++++++++++ docs/guides/data_serving_guide.md | 12 +++++++---- docs/guides/hetzner_deployment_guide.md | 17 +++++++++++----- docs/guides/hetzner_deployment_log.md | 6 ++++++ docs/guides/server_quickref.md | 23 ++++++++++++++++++--- reports/technical_risk_register.md | 27 ++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 13 deletions(-) diff --git a/docs/ADRs/041_content_addressed_skip.md b/docs/ADRs/041_content_addressed_skip.md index 34fafb9e..58d0c15f 100644 --- a/docs/ADRs/041_content_addressed_skip.md +++ b/docs/ADRs/041_content_addressed_skip.md @@ -107,3 +107,18 @@ This ADR covers assembly and export only. It does not cover: - **Output integrity is verified on every skip:** a corrupt grid.npy triggers a rebuild, not silent serving. - **Source set changes are detected:** adding or removing a source (e.g., adding SHDI) prevents false skips. - **Operator override exists:** `--force` provides an escape hatch when data hasn't changed but code has. + +--- + +### Amendment 2026-10-18 — the export cost in Context is 5× low, and the skip is worth more than claimed + +**Context** states that assembly is ~46 min and export ~57 min, and **Consequences** derives "no-change deploys drop from ~103 min to ~30 sec" from that pair. The 2026-10-16 production run measured the export at **18,427s (5h07m)** for a 3.1 GB store — roughly **5× the figure above**, and 88% of a 21,007s (5h50m) run. + +Nothing in this ADR's decision is affected. The skip mechanism works as specified, and the digest gates behave as described. Two statements derived from the stale timings are wrong: + +1. **The export is not comparable in cost to assembly. It dominates the pipeline.** Any future reasoning that treats the two as peers — a decision about which to optimise, which to parallelise, or which to make skippable — starts from a false premise. +2. **This ADR understates its own benefit by about 3.4×.** A no-change deploy avoids roughly **~350 min**, not ~103. The skip is the single largest runtime saving in the pipeline. + +**The cause of the 5× is not established and is deliberately not guessed at here.** The store grew from 1.8 GB (2026-03) to 3.1 GB, which accounts for some of it but not a factor of five. Candidates worth measuring: disk I/O on the CPX32, the `12 × 360 × 720` chunking against a larger store, and compression settings. Tracked as a separate investigation in **#501**, which asks for the three export phases to be timed separately before anything is changed. + +The per-step operational figures live in `docs/guides/server_quickref.md`, which owns them. The numbers in **Context** above are left as written — they were measured in June 2026 and this amendment dates the correction rather than rewriting the record. diff --git a/docs/guides/data_serving_guide.md b/docs/guides/data_serving_guide.md index 8c38a76d..b3cd2c18 100644 --- a/docs/guides/data_serving_guide.md +++ b/docs/guides/data_serving_guide.md @@ -174,10 +174,14 @@ Example: "Run the data pipeline on the 21st of every month at midnight" That line means: minute 0, hour 0, day 21, every month, every day-of-week. Run the script. -Your pipeline takes about 15-30 minutes (harvesting is the slow -part — it calls the UCDP and ACLED APIs). After it finishes, the new zarr -and parquet files are in the same directory the web server serves. -Consumers automatically get the new data on their next request. +Your pipeline takes about **six hours**. Harvesting is *not* the slow part, +despite being the part that calls external APIs: harvest through compile is +~43 minutes, and the zarr export alone is ~5 hours. Per-step timings live in +[`server_quickref.md`](server_quickref.md#run-the-pipeline). + +After it finishes, the new zarr and parquet files are in the same directory the +web server serves. Consumers automatically get the new data on their next +request. **No restart needed.** The web server serves whatever files are in the directory. When the pipeline replaces them, the next diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 6f69508e..f852129c 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -27,7 +27,7 @@ If the server is already set up and you're just deploying a tag that **already e # 1. SSH into the server (replace with your username) ssh @204.168.219.108 -# 2. Start tmux (pipeline runs 3-4 hours; SSH will drop without a multiplexer) +# 2. Start tmux (the run takes ~6 hours; SSH will drop without a multiplexer) tmux new -s deploy # 3. Deploy and run @@ -189,8 +189,15 @@ cd ~/views-datafactory bash scripts/refresh_pipeline.sh ``` -This takes 25-45 minutes (harvesting calls external APIs and downloads -large raster files). Watch the output — each step prints PASS or FAIL. +This takes about **six hours**. Watch the output — each step prints PASS or +FAIL. Per-step timings are in +[`server_quickref.md`](server_quickref.md#run-the-pipeline), which owns them. + +> **This figure said "25-45 minutes" until 2026-10-18, and that was the +> dangerous direction to be wrong in.** A run at the two-hour mark looks hung +> against a 45-minute estimate, and the reasonable response to a hung run is to +> kill it. The cost is not harvesting — harvest through compile is ~43 minutes. +> **The zarr export alone is ~5 hours and prints nothing while it works.** **Note:** The pipeline includes ACLED (8 features), GHS-POP population data (1 feature), GHS-BUILT-S built-up surface data @@ -463,8 +470,8 @@ See ADR-018 for the rationale. 2. Create a check (or verify the existing one): - **Name:** `views-datafactory pipeline refresh` - **Period:** 30 days (matches the `0 0 21 * *` cron) - - **Grace period:** 48 hours (pipeline runs 2-3 hours; 48h covers - delays, restarts, and weekend non-response) + - **Grace period:** 48 hours (the run takes ~6 hours; 48h still leaves + room for delays, restarts, and weekend non-response) - **Tags:** `views`, `datafactory`, `hetzner` 3. Copy the ping URL: `https://hc-ping.com/` diff --git a/docs/guides/hetzner_deployment_log.md b/docs/guides/hetzner_deployment_log.md index 128be9f6..b42ff3e8 100644 --- a/docs/guides/hetzner_deployment_log.md +++ b/docs/guides/hetzner_deployment_log.md @@ -295,6 +295,12 @@ or mismatched path. **Total: approximately 2.5 hours for a clean run.** +> **These are March 2026 measurements and are no longer current.** The +> 2026-10-16 run took **5h50m**, with the zarr export alone at ~5h07m against a +> 3.1 GB store (this table has it at ~14 min against 1.8 GB). This is a dated +> log and the figures above are left as measured; current per-step timings live +> in [`server_quickref.md`](server_quickref.md#run-the-pipeline). + Subsequent runs are much faster because most data is cached (only new UCDP versions are fetched). diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 4aac18be..f14ce242 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -38,9 +38,26 @@ A **rollback is a deploy**: same four blocks, previous tag. ## Run the pipeline -**Start tmux first. Not optional.** The run takes 3-4 hours and SSH will drop -inside that window; without a multiplexer the drop sends SIGHUP and kills the -run. This is not hypothetical — see the note below. +**Start tmux first. Not optional.** The run takes about six hours and SSH will +drop inside that window; without a multiplexer the drop sends SIGHUP and kills +the run. This is not hypothetical — see the note below. + +**Expect ~6 hours, and do not kill a run for "taking too long".** Measured +2026-10-16: **5h50m** (21,007s) with the assembly cached. The zarr export alone +runs for roughly five hours and prints nothing at all while it works — a silent +terminal is the normal state, not a hang. + +| Step | Time | Note | +|------|------|------| +| Harvest → consolidate → viewpoint → compile | ~43 min | all sources, most cached | +| Assemble grid | ~45 min | skipped when inputs are unchanged (ADR-041) | +| **Export zarr** | **~5h07m** | 18,427s for a 3.1 GB store — **88% of the run** | +| Export parquet | ~12 s | 29.5M rows | +| Health check + consumer contract | <1 min | | + +The assembly figure is ADR-041's; every other figure is from the 2026-10-16 run, +which had the assembly cached. A cold run that must rebuild the 37 GB grid is +therefore longer than 5h50m, not shorter. ```bash # 1. ALWAYS start tmux first diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index eba91c75..c68683e8 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3280,7 +3280,32 @@ Every interactive `refresh_pipeline.sh` invocation in the guides now starts with **No new register entry.** The failure is loud, the defences worked, and the trigger would be perpetual ("whenever someone runs the pipeline"), which the trigger-quality gate rejects. What is registrable is the pattern above, and it belongs to this entry: a consolidation is only safe if the authority absorbs everything the demoted copies knew. Checking that was not part of the 2026-10-13 work and should have been. -**The shape worth noticing.** C-343 is now on its third round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. +**Addendum 2026-10-18 — the new authority carried a number nobody had measured.** The 2026-10-16 addendum above records that consolidation can move content *out* of reach. This one records the complement, which is worse: **consolidation concentrates error as effectively as it concentrates authority.** When `server_quickref.md` was promoted on 2026-10-13 it was given "the run takes 3-4 hours." That figure was a guess. No measurement existed for it in the repo, and the document that stated it had just been declared the winner over every document that disagreed. + +Seven places claimed a pipeline duration, and they spanned a factor of 24: + +| Location | Claimed | Kind of document | +|---|---|---| +| `data_serving_guide.md:177` | 15-30 min | live guide | +| `hetzner_deployment_guide.md:192` | 25-45 min | live guide | +| `hetzner_deployment_log.md:296` | ~2.5 h | dated log (March 2026) | +| `hetzner_deployment_guide.md:466` | 2-3 h | live guide (justifies the 48h healthcheck grace) | +| `server_quickref.md:41` | 3-4 h | **the authority, set 2026-10-13** | +| `hetzner_deployment_guide.md:30` | 3-4 h | live guide | +| `ADR-041:12` | ~6 h | ADR | +| **2026-10-16 run, measured** | **5h50m** | — | + +The closest figure was in an **ADR about something else**. ADR-041 got it right incidentally, while arguing for content-addressed skipping. The document whose job is to be correct about it was wrong by 40%, and it was wrong because the correction that created its authority did not come with a measurement. + +**The direction of the error is the operational hazard, not the magnitude.** An over-estimate is harmless; an under-estimate licenses intervention on a healthy run. "25-45 minutes" tells an operator two hours in that the run is hung, and the reasonable response to a hung run is to kill it — reproducing the 2026-10-16 outage from a document instead of a dropped connection. Every stale figure also mis-attributed the cost to harvesting; harvest through compile is ~43 min, and the zarr export alone is ~5h07m, 88% of the run. + +Corrected by kind rather than uniformly, which is the part worth keeping: the four live guides now carry the measured figure and point at the quickref; **ADR-041 and the March deployment log were not rewritten** but carry dated amendments, because both are records of what was true when written and falsifying a record to make it current is a different defect from the one being fixed. + +**No new register entry, again.** The trigger would be perpetual. What is registrable belongs here: a consolidation must verify what it promotes, not only absorb what it demotes. Two rounds of this entry have now found that the 2026-10-13 work was sound in mechanism and unverified in content. + +**One finding escapes this entry and is not a documentation problem.** ADR-041 measured the zarr export at ~57 min in June 2026; it measured **18,427s (5h07m)** on 2026-10-16, a ~5x increase against a store that grew 1.8 GB → 3.1 GB. The cause is unestablished and deliberately not guessed at. Filed as **#501**, because it is a performance question requiring instrumentation rather than a figure requiring correction, and because ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not the ~103 min it claims. + +**The shape worth noticing.** C-343 is now on its fourth round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. From 7ce9b6b25fc37156bf302404f0cd3e39bba7e6ea Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 18 Oct 2026 19:50:40 +0200 Subject: [PATCH 5/6] =?UTF-8?q?perf:=20time=20the=20zarr=20export's=20four?= =?UTF-8?q?=20phases=20=E2=80=94=205h07m=20with=20no=20breakdown=20could?= =?UTF-8?q?=20not=20be=20attributed=20(#501)=20(#502)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-10-16 run measured the export at 18,427s, 88% of the pipeline and ~5x ADR-041's June figure of ~57 min. Nothing recorded WHERE that time went, so the regression could be stated but not located. Four phases are now timed, printed in the export's summary and written to export_ledger.jsonl as `phase_seconds`: scan the source_sums loop — one traversal of grid.npy per feature write ds.to_zarr + consolidate_metadata — the second traversal verify the round-trip integrity check — full decompression swap the C-144 .tmp -> rename dance Plus `other`, printed explicitly so the columns account for the whole of the existing `Time:` line rather than silently omitting a remainder. The ledger is the point: a print lives in logs/refresh.log, which rotates. `phase_seconds` in the ledger makes the breakdown queryable across runs, which is what turns a single measurement into a trend. Verified empirically, not inferred: - grid.npy is C_CONTIGUOUS with features as the contiguous last axis, so grid[:, :, :, i] strides F x 4 bytes between adjacent (t, h, w) — 316 bytes at F=79. A 4 KB page carries ~13 useful float32s out of 1024. - `source_sums` has exactly one consumer: the verify comparison. The scan phase's full-grid read exists solely to feed that check. NOT optimised. No change to chunking, compression, the slicing pattern, or the integrity check. Each is a plausible remedy for a cause that is not yet established, and the integrity check is an ADR-011/C-144 guarantee that must not be weakened to save time. #501 carries a pre-registered falsification criterion so the first production measurement is a test rather than a confirmation. Three guards in TestZarrExport, both substantive ones drilled by sabotage rather than assumed: delete phase_seconds["swap"] -> test_phase_timings_recorded FAILED inflate the scan timer past total -> test_phases_do_not_exceed_the_total FAILED The phase labels are asserted as an exact set; a subset assertion would let a dropped phase restore the original blindness silently. Verification: full suite exit 0 (0 FAILED, 0 ERROR), ruff clean, mypy src/ clean (89 files), validate_docs.sh PASSED, export-touching test files re-run green against the final tree. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- scripts/export_zarr.py | 44 +++++++++++++++++++++++++- tests/test_synthetic_pipeline.py | 54 ++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 1 deletion(-) diff --git a/scripts/export_zarr.py b/scripts/export_zarr.py index cf0aa693..96d143bd 100644 --- a/scripts/export_zarr.py +++ b/scripts/export_zarr.py @@ -208,6 +208,12 @@ def main() -> int: t0 = time.monotonic() + # Where the export's wall time actually goes (#501). The total + # below has never been broken down, so the 5h07m measured on + # 2026-10-16 could not be attributed to a phase. One dict so the + # printed table and the ledger entry cannot drift apart. + phase_seconds: dict[str, float] = {} + # Load grid and metadata grid = np.load(grid_path, mmap_mode="r") pgids = np.load(pgids_path) @@ -235,7 +241,15 @@ def main() -> int: n_w, ) - # Build xarray Dataset — one variable per feature + # Build xarray Dataset — one variable per feature. + # + # Timed separately because this loop reads the whole grid once per + # feature. grid is [T, H, W, F] in C order, so grid[:, :, :, i] is a + # last-axis slice: it touches every page of the file while using + # 1/F of each page. With the grid larger than RAM the page cache + # cannot absorb that, and the sums here are computed only to be + # compared against the zarr read-back further down. + _t_scan = time.monotonic() data_vars = {} source_sums: dict[str, float] = {} for i, name in enumerate(feature_names): @@ -246,6 +260,7 @@ def main() -> int: feature_slice, ) print(f" {i:2d}: {name}") + phase_seconds["scan"] = time.monotonic() - _t_scan coords = { "time": time_steps, @@ -370,15 +385,22 @@ def main() -> int: shutil.rmtree(tmp_output) try: + # Second traversal of the grid: xarray pulls each feature_slice + # again to compress and write it. + _t_write = time.monotonic() ds.to_zarr( tmp_output, mode="w", encoding=encoding, ) zarr.consolidate_metadata(str(tmp_output)) + phase_seconds["write"] = time.monotonic() - _t_write print() print("Round-trip integrity check...") + # Reads back from the zarr store, not the memmap — fully + # decompresses every feature to re-sum it. + _t_verify = time.monotonic() store = zarr.open(str(tmp_output), mode="r") n_checked = 0 for name in feature_names: @@ -394,8 +416,12 @@ def main() -> int: return 1 n_checked += 1 del store + phase_seconds["verify"] = time.monotonic() - _t_verify print(f" {n_checked} features verified (sums match)") + # The C-144 atomic swap. Measured so it can be ruled out + # rather than assumed cheap. + _t_swap = time.monotonic() old_output = output.parent / (output.name + ".old") if old_output.exists(): shutil.rmtree(old_output) @@ -404,6 +430,7 @@ def main() -> int: os.rename(str(tmp_output), str(output)) if old_output.exists(): shutil.rmtree(old_output) + phase_seconds["swap"] = time.monotonic() - _t_swap except Exception: if tmp_output.exists(): shutil.rmtree(tmp_output) @@ -431,12 +458,27 @@ def main() -> int: f"{n_h} lat x {n_w} lon") print(f"Time: {elapsed:.1f}s") + # Phase breakdown (#501). "other" is whatever the named phases do + # not cover — grid load, coordinate construction, digests, the + # sentinel — and is reported rather than omitted so the columns + # always account for the whole of Time above. + print("Phases:") + _denom = max(elapsed, 1e-9) + for _label, _secs in phase_seconds.items(): + print(f" {_label:<7}{_secs:>10.1f}s {100 * _secs / _denom:>5.1f}%") + _other = elapsed - sum(phase_seconds.values()) + print(f" {'other':<7}{_other:>10.1f}s {100 * _other / _denom:>5.1f}%") + append_ledger_entry(_ledger_path, { "stage": "export", "outcome": "success", "source_digest": grid_digest, "ledger_version": LEDGER_VERSION, "digest_algorithm": DIGEST_SCHEME, + "elapsed_seconds": round(elapsed, 1), + "phase_seconds": { + k: round(v, 1) for k, v in phase_seconds.items() + }, }) _ledger_written = True diff --git a/tests/test_synthetic_pipeline.py b/tests/test_synthetic_pipeline.py index 97e5fc5d..d5309258 100644 --- a/tests/test_synthetic_pipeline.py +++ b/tests/test_synthetic_pipeline.py @@ -226,6 +226,60 @@ def test_zarr_source_digest(self, pipeline: dict) -> None: def test_sentinel_cleared(self, pipeline: dict) -> None: assert not (pipeline["assembled"] / ".exports_required").exists() + # ── #501: the export's phase breakdown ───────────────── + + @staticmethod + def _export_entry(pipeline: dict) -> dict: + """The successful export entry from the run's own ledger.""" + ledger = pipeline["assembled"] / "export_ledger.jsonl" + assert ledger.exists(), "export wrote no ledger" + entries = [ + json.loads(line) + for line in ledger.read_text().splitlines() + if line.strip() + ] + successes = [ + e for e in entries + if e.get("stage") == "export" and e.get("outcome") == "success" + ] + assert successes, f"no successful export entry in {entries}" + return successes[-1] + + def test_phase_timings_recorded(self, pipeline: dict) -> None: + """Every phase #501 asks about reaches the ledger, by name. + + The export took 5h07m on 2026-10-16 with no breakdown, so the + cost could not be attributed. If a phase label is dropped here, + that blindness returns silently — hence an exact-set assertion + rather than a subset one. + """ + entry = self._export_entry(pipeline) + assert "phase_seconds" in entry, entry + assert set(entry["phase_seconds"]) == { + "scan", "write", "verify", "swap" + }, entry["phase_seconds"] + + def test_phase_timings_are_nonnegative(self, pipeline: dict) -> None: + phases = self._export_entry(pipeline)["phase_seconds"] + assert all(v >= 0 for v in phases.values()), phases + + def test_phases_do_not_exceed_the_total(self, pipeline: dict) -> None: + """The phases must fit inside the reported total. + + They need not sum to it — 'other' covers load, coordinates and + digests. But a sum ABOVE the total means a timer spans work it + does not own, which would misattribute exactly the cost #501 + exists to locate. + """ + entry = self._export_entry(pipeline) + elapsed = entry["elapsed_seconds"] + assert elapsed > 0, entry + total = sum(entry["phase_seconds"].values()) + assert total <= elapsed + 0.5, ( + f"phases sum to {total}s but total is {elapsed}s: " + f"{entry['phase_seconds']}" + ) + # ── Consumer data loading ────────────────────────────────── From 7cef97f50cb2fa3acdae6799539937b668639742 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Tue, 20 Oct 2026 02:31:09 +0200 Subject: [PATCH 6/6] =?UTF-8?q?chore:=20bump=20to=201.15.0=20=E2=80=94=20t?= =?UTF-8?q?he=20wheel=20is=20code-identical,=20the=20git=20tag=20is=20the?= =?UTF-8?q?=20whole=20point=20(#501)=20(#503)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Not one file under src/ changed since v1.14.0, and the wheel packages only the nine src/datafactory_* trees, so a consumer installing 1.15.0 from PyPI gets exactly what 1.14.0 gave them with different version metadata. That is the point rather than a defect. The server deploys by git tag (ADR-022): refresh_pipeline.sh reads ~/.views-deploy-tag and checks out that exact tag. The two things this release carries — scripts/export_zarr.py and a set of corrected guides — reach production no other way, because the wheel does not contain them. v1.11.0 and v1.12.0 were both minor bumps with zero src/ change, so the shape is house precedent, not a judgement made here. Why now: #502 merged the only instrument that can attribute the zarr export's 5h07m to a phase, and it sits on development where the server cannot see it. The monthly cron fires 00:00 on the 21st. A tag before then gets the measurement on a run that was happening anyway; missing it means a deliberate out-of-band six-hour run or waiting until 21 November. The plan edit is not housekeeping tacked on. TestF7ProductPlanCurrency asserts the product plan's title carries the current minor version, so the version bump alone turns it red — the two edits are one atomic change. Confirmed by breaking it rather than by reading it: with the title reverted to "through v1.14" and the version at 1.15.0 the test FAILED, and went green when the title was restored. uv.lock moves too: it pins the root project version (line 1701), and uv re-synced it on the first uv run after the bump. refresh_pipeline.sh:178 does git checkout -- uv.lock before the tag checkout precisely because the server mutates it locally. Noticed, deliberately not fixed: "## Version History" in the same plan file stops at v1.6 and jumps to v2.0 — eight releases happened without it. The per-release addenda at the top have been the live record since v1.7. Repairing a section stale for eight releases inside a version bump would hide a documentation change inside a release. No tag, no GitHub Release, no deploy. Each is separately gated. Verification: full suite EXIT=0, 0 FAILED, 0 ERROR (XFAILs are documented deferrals); ruff check . clean; mypy src/ clean (89 files); validate_docs.sh PASSED; deploy_v160 + _v160_r2 + merge_readiness 10 passed 1 skipped (HEAD not on a tag, expected pre-release); main is an ancestor of development. Refs #501. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) --- pyproject.toml | 2 +- reports/product_development_plan12.md | 49 ++++++++++++++++++++++++++- uv.lock | 2 +- 3 files changed, 50 insertions(+), 3 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 020ae4a2..cec4e039 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "views-datafactory" -version = "1.14.0" +version = "1.15.0" description = "Data factory for the VIEWS conflict forecasting platform — provenance, harvesting, consolidation, viewpoint construction, compilation, and assembly." authors = [ { name = "Simon Polichinel von der Maase", email = "simmaa@prio.org" }, diff --git a/reports/product_development_plan12.md b/reports/product_development_plan12.md index 31bd9cc0..b33cf83a 100644 --- a/reports/product_development_plan12.md +++ b/reports/product_development_plan12.md @@ -1,4 +1,4 @@ -# Product Development Plan v12 — current through v1.14, 11 registry sources, 79 Features, on PyPI +# Product Development Plan v12 — current through v1.15, 11 registry sources, 79 Features, on PyPI **Date:** 2026-06-29 (v1.10 addendum 2026-07-31) **Supersedes:** product_development_plan11.md (2026-05-08) @@ -7,6 +7,53 @@ --- +## v1.15 Addendum (2026-10-19) + +**v1.15.0 (the export's 5h07m gets a breakdown, and four rounds of C-343 land).** This release +contains **no library code at all** — `git diff --name-only v1.14.0 development -- src/` is empty, +so the wheel is byte-identical to v1.14.0 apart from its version metadata. That is not an oversight +and it is worth stating plainly, because it explains what the release is *for*: the server deploys +by **git tag** (ADR-022), and the two things this release carries — a script and a set of guides — +reach production no other way. A consumer installing from PyPI gets nothing new. The server gets +the only instrument that can answer #501. + +**What the server gets.** `scripts/export_zarr.py` now times four phases — `scan` (the `source_sums` +loop, one strided traversal of `grid.npy` per feature), `write` (`to_zarr` plus metadata +consolidation), `verify` (the ADR-011/C-144 round-trip integrity read) and `swap` (the `.tmp` +rename) — plus an explicit `other`, so the printed columns account for the whole of the `Time:` +line rather than a selected part of it. The breakdown prints to `logs/refresh.log` **and** persists +as `phase_seconds` in `data/assembled/export_ledger.jsonl`, which is the half that matters: logs +rotate, the ledger does not, and one measurement in a rotating log is not a trend. + +**Why it exists.** The 2026-10-16 production run took **21,007s (5h50m)**, of which the zarr export +alone was **18,427s (5h07m) — 88% of the pipeline**. ADR-041's Context had the export at ~57 +minutes. Nothing in the repo could attribute the other four hours to a phase, so nothing could be +optimised without guessing. The candidate explanation is pre-registered on #501 with a falsification +criterion committed **before** any production figure exists: `grid.npy` is C-contiguous with +features on the last axis, so `grid[:, :, :, i]` strides 316 bytes at F=79 and a 4 KB page yields +~13 useful float32s out of 1024 — **if `scan` comes in under ~25% of the export, that explanation is +refuted and abandoned.** The synthetic baseline, where 0.1 GB fits in RAM, puts `scan` at 10.1% — +the smallest named phase. The entire claim is that crossing the RAM boundary inverts that. + +**What else lands.** Four rounds of C-343 ("writing the deploy tag is not deploying"), each a +correction to the previous one's blind spot: #497 found four more copies of the one-step claim; +#498 found that `~` after `sudo` expands as the caller, so six verification commands read the wrong +home; #499 gave the procedure one home and added the tmux rule; #500 found that the authority +established in #499 carried a duration figure **nobody had ever measured** — seven places claimed a +pipeline duration, spanning 24×, and two of the wrong ones had been written two days earlier as the +fix. The through-line, now recorded: *a consolidation concentrates error as effectively as it +concentrates authority, so it must verify what it promotes, not merely absorb what it demotes.* + +**What this release does *not* fix.** The phase breakdown is *recorded* but nothing *compares* it +across runs — `check_health.py` has freshness SLOs and no duration SLO, so a second 5× regression +would still be invisible until someone read a log by hand. Flagged rather than built: one datapoint +does not justify a trend watcher, and the instrument that would have caught the first regression is +what this release ships. No change was made to chunking, compression, the slicing pattern or the +integrity check — each is a plausible remedy for a cause that is **not yet established**, and the +integrity check is an ADR-011/C-144 guarantee that must not be weakened to buy time. + +--- + ## v1.14 Addendum (2026-10-12) **v1.14.0 (ACLED pagination walks cursors — a provider-forced mechanism change).** ACLED deprecates diff --git a/uv.lock b/uv.lock index e0035d87..dab6e2a6 100644 --- a/uv.lock +++ b/uv.lock @@ -1698,7 +1698,7 @@ wheels = [ [[package]] name = "views-datafactory" -version = "1.14.0" +version = "1.15.0" source = { editable = "." } dependencies = [ { name = "aiohttp" },