Skip to content

docs: writing the deploy tag is not deploying — four more copies said it did (C-343) - #497

Merged
Polichinel merged 1 commit into
developmentfrom
docs/deploy-is-three-steps
Oct 12, 2026
Merged

Polichinel merged 1 commit into
developmentfrom
docs/deploy-is-three-steps

Conversation

@Polichinel

Copy link
Copy Markdown
Contributor

Why

Before deploying v1.14.0 the operator asked whether the one-line tag write I had given them was really the deploy procedure. It was not — it is the exact mistake C-343 (Tier 2, open) exists to prevent, and the reason the question had to be asked is that C-343's own correction was partial.

#431 registered C-343 and fixed scripts/refresh_pipeline.sh and docs/guides/server_quickref.md. Four more copies of the one-step claim were still live, and they are the ones an operator reads.

Document What it said
docs/guides/hetzner_deployment_guide.md the tag file, then "That's it." — and this is the file refresh_pipeline.sh points to "for full detail"
docs/guides/server_operations.md a local git tag && git push (cannot trigger the publish workflow, and cannot be run at all under branch protection) plus the tag file; no checkout, no uv sync
docs/ADRs/022_tag_based_deployment_gate.md "One-line rollback" listed as a benefit of the gate; a three-step deploy with neither checkout nor uv sync; a Verify block naming the tag file and git describe but not the installed distribution — the one of the three that was actually wrong in August
docs/guides/data_serving_guide.md told newcomers that writing the old tag name back puts you back on known-good code

The ADR is the worst of the four, because it is the governing document and it sold the defect as a feature.

What changed

  • server_quickref.md is the single operational home, and grew the verification block it was missing: step 4 checks all three sources — tag file, checked-out tag, and installed distribution version, the only one that reflects uv sync. Also states that step 1 alone is not a deploy, and that a rollback is a deploy.
  • The four stale documents now point there, each carrying "if they disagree, the quickref wins." No second copy of the commands anywhere — that duplication is what Release ritual documents a procedure branch protection made impossible #402 is about and what let these drift.
  • ADR-022 gets a dated amendment in the ADR-027 style rather than a rewrite. The Decision it records is sound: the gate does refuse to start without a valid tag and does check that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. The amendment states both reasons (bash has already buffered the script; uv sync is never invoked) and notes the entry is still open.
  • C-343 updated, not duplicated — same code locations, same problem type, so this merges into the existing entry per the dedup rule. The update is honest that it weakens the entry's own "corrected in the same PR as this entry" claim: the correction reached two of six copies and the stale four outlived it by two months and one release.

Verification

Check Result
uv run ruff check . clean
uv run pytest (full, unpiped) exit 0, zero FAILED lines
bash docs/validate_docs.sh PASSED
uv run pytest tests/test_docs_citations.py 7 passed — no file.sh:N citations introduced
all register-reading test files 41 passed, every one run

No source, test or dependency changes; header counts unchanged (an amendment, not a new entry).

Also worth noting for the record: the first full-suite run failed test_falsification_deploy_v160.py::TestF2ReleaseTopology::test_main_is_ancestor_of_development. Not a topology break — remote main and development are healthy; the local development ref was two commits stale at 801177d, and this gate reads local refs by design. Fast-forwarded to 47fa8b4 and it passes. The gate did its job.

What this does not fix

Nothing mechanically prevents a seventh copy, or prevents the procedure being skipped on the box. The pre-flight check that would — comparing $DEPLOY_TAG against the HEAD the script started from, failing loud per ADR-011 rather than self-correcting one run late — is still #424 and still unbuilt. C-343 stays open.

Refs #424, #402. Closes nothing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9

… it did (C-343)

C-343 was registered in #431 and corrected in scripts/refresh_pipeline.sh
and docs/guides/server_quickref.md. Four further copies of the one-step
claim were still live, and they are the ones an operator reads:

- hetzner_deployment_guide.md said the tag file, then "That's it." This is
  the file refresh_pipeline.sh itself points to "for full detail".
- server_operations.md gave a local `git tag && git push` (which cannot
  trigger the publish workflow, and cannot be run at all since branch
  protection) plus the tag file, with no checkout and no `uv sync`.
- ADR-022 listed "One-line rollback" as a benefit of the gate, gave a
  three-step deploy with neither checkout nor `uv sync`, and offered a
  Verify block naming the tag file and `git describe` but not the
  installed distribution — the one of the three that was wrong in August.
- data_serving_guide.md told newcomers that writing the old tag name back
  puts you on known-good code.

server_quickref.md is now the single operational home and grew the
verification it was missing: step 4 checks all three sources, including
the installed version, which is the only one that reflects `uv sync`. The
four others point there and say the quickref wins on disagreement.

ADR-022 carries a dated amendment in the ADR-027 style rather than a
rewrite: the Decision it records is sound, and what was wrong was the
inference that the gate's own `git checkout` makes the tag file
sufficient.

C-343 is updated rather than duplicated — same risk, new evidence that
the fix reached two of six copies. It stays open; the pre-flight
instrument that would catch a seventh copy is still #424.

Found because the operator asked, before deploying v1.14.0, whether a
one-line tag write was really the procedure. It was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9
@Polichinel
Polichinel enabled auto-merge (squash) October 12, 2026 22:32
@Polichinel
Polichinel merged commit 32f91db into development Oct 12, 2026
6 checks passed
@Polichinel
Polichinel deleted the docs/deploy-is-three-steps branch October 12, 2026 22:42
Polichinel added a commit that referenced this pull request Oct 16, 2026
…s read the wrong home (C-343) (#498)

Found by running the corrected deploy procedure on the production host.
v1.14.0 deployed cleanly, then step 4's first check failed:

    $ sudo -u views-deploy cat ~/.views-deploy-tag
    cat: /home/simmaa_prio/.views-deploy-tag: Permission denied

The calling shell expands `~` before sudo runs, so the command reads the
operator's home rather than views-deploy's. Six such commands were live
in the guides; two were added by the C-343 correction in #497. All are
now wrapped in `bash -c '...'`, where the inner shell expands `~` as the
target user.

One of the six failed green rather than loud:

    sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null \
        || echo "No recent failure"

The `||` sat outside `bash -c`, so a permission error on the wrong home
printed "No recent failure" — in the command an operator runs to ask
whether the last pipeline run broke. The fallback is now inside the
quoted command.

Step 4's third check also drops the escaped-quote python one-liner for
`ls .venv/.../site-packages | grep views_datafactory`, which needs no
quoting and prints the dist-info directory — the same evidence form
C-343 used in August.

C-343 updated with the deploy outcome: `uv sync` reported
1.13.0 -> 1.14.0 on the box, which is direct confirmation that step 3 is
load-bearing and the tag file alone would have left 1.13.0 installed.


Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Polichinel added a commit that referenced this pull request Oct 16, 2026
… carry (C-343) (#499)

The 2026-10-13 consolidation made server_quickref.md the single
operational home and told every other guide "if they disagree, the
quickref wins." hetzner_deployment_guide.md carried a rule the quickref
did not — start tmux before a pipeline run, stated at its line 30 with
the reason, and backed by a post-mortem section called "tmux saved us
from the SSH drop". Promoting the quickref silently demoted that rule.

It cost a run the same day. `client_loop: send disconnect: Broken pipe`
killed a foreground pipeline after the 37 GB assembly had completed but
before the zarr export, leaving grid.npy current and grid.zarr three
weeks stale.

Every designed defence held: assemble_grid.py renames from grid.npy.tmp
so the grid was whole, .exports_required recorded the owed export, flock
released with the process, and check_health.py reports the sentinel as
PENDING. Nothing was corrupted and nothing was silent. The cost was a
re-run.

Every interactive refresh_pipeline.sh invocation in the guides now opens
with `tmux new -s pipeline`:

- server_quickref.md §"Run the pipeline" — the rule, the dated evidence,
  detach/reattach, and the detail that catches people out: tmux does not
  change the shell prompt, so only the bottom status bar tells you
  whether you are inside it
- hetzner_deployment_guide.md §2.1, §4.5, and the rollback section added
  in #497
- server_operations.md summary table and the full-rebuild procedure

Cron invocations are deliberately unchanged — cron has no controlling
terminal and no SIGHUP to receive. The ~/.profile wrong-vs-right example
in server_operations.md is also unchanged; adding tmux would obscure the
point it exists to make.

C-343 gets the pattern rather than a new entry: a consolidation is only
safe if the new authority absorbs everything the demoted copies knew,
and checking that was not part of the 2026-10-13 work.


Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Polichinel added a commit that referenced this pull request Oct 20, 2026
…ve times in eight days (C-343) (#509)

Covers the export phase instrumentation (#501/#502), four rounds of C-343
correction (#497-#500), the release/publish/deploy of v1.15.0, a fifth C-343
round found while writing this up (#507), and a licensing contradiction found
while onboarding an external consumer (#508).

The v1.11.0 post-mortem named the spine running through everything it
reported: "a written record that was true when written and stopped being true
without anything failing." Both it and its companion observation — that the
post-mortem convention keeps failing to be applied — recurred.

Three of the four releases since are unwritten and now never will be. The
spine produced five fresh instances in eight days, all in C-343, the fifth
found inside the register entry created to track the first, including in its
Trigger: field. A sixth variant was committed by the author of the fourth
correction three days later: merging the instrumentation is not deploying the
instrumentation.

The conclusion recorded is not "try harder". Every instance in this window was
found by a human asking an unprompted question or by an author re-reading
their own work. Five rounds have produced a procedure that is right and
nothing that notices when a copy of it goes wrong.

Also records what went well rather than only the failures: block 4 of the
deploy procedure executed successfully for the first time since it was
written, the #501 falsification criterion was committed before any production
figure existed, three guards were sabotaged and confirmed red, and the clean-
venv publish check caught a defect in itself rather than a false alarm on the
release.

Verification: validate_docs.sh PASSED; all 14 register-reading test files
41 passed. Every figure quoted is cited to a file rather than recalled; the
release count was corrected against `git tag` before committing.

Refs #501, #508.


Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant