Repository navigation
docs: seven places claimed a pipeline duration, spanning 24x — the authority's figure was never measured (C-343) - #500
Merged
Conversation
2 tasks done
…thority's figure was never measured (C-343) The 2026-10-16 run took 5h50m. The guides claimed 15-30 min, 25-45 min, ~2.5h, 2-3h, 3-4h and ~6h, and the closest figure was in an ADR about something else. `server_quickref.md` was made the single operational authority on 2026-10-13 and given "3-4 hours" — a guess, with no measurement anywhere in the repo, promoted over every document that disagreed. The complement of the tmux finding: consolidation concentrates error as effectively as it concentrates authority. The direction of the error is the hazard. An under-estimate licenses intervention on a healthy run: "25-45 minutes" tells an operator two hours in that the run is hung, and killing a hung run is the reasonable response — reproducing the 2026-10-16 outage from a document instead of a dropped SSH connection. Every stale figure also blamed harvesting; harvest through compile is ~43 min and the zarr export alone is ~5h07m, 88% of the run. Corrected by document kind, not uniformly: - four live guides carry the measured figure and point at the quickref, which owns the per-step breakdown - ADR-041 and the March 2026 deployment log are NOT rewritten — both are records of what was true when written, and falsifying a record to make it current is a different defect from the one being fixed. Both carry dated amendments instead. Also corrected: ADR-041's healthcheck-grace rationale derived 48h from "2-3 hours"; the conclusion survives at ~6h, the reasoning did not. One finding escapes the docs fix and is filed separately: the zarr export was ~57 min in June 2026 and 18,427s (5h07m) on 2026-10-16, a ~5x increase against a store that grew 1.8 -> 3.1 GB. Cause unestablished and deliberately not guessed at. ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not ~103. Verification: ruff clean, validate_docs.sh PASSED, 48 register-reading and citation tests plus 38 deployment-doc guards green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9
Polichinel
force-pushed
the
docs/pipeline-duration-figures
branch
from
October 18, 2026 16:32
0efbae2 to
ccbad36
Compare
Polichinel
enabled auto-merge (squash)
October 18, 2026 16:32
This was referenced Oct 18, 2026
Merged
Polichinel
added a commit
that referenced
this pull request
Oct 20, 2026
…ee steps (C-343) (#507) v1.15.0 deployed with all four blocks and all three verification commands naming the same version, so C-343's corrected procedure completed end to end for the first time — step 4 included. That matters because on 2026-10-16 step 4's first command died on the `~`-expansion defect, meaning the verification block added on 2026-10-13 had never once executed successfully. #498's fix is now confirmed by execution rather than by reading. Writing that up prompted a grep for the step COUNT rather than the step CONTENT — a dimension none of the four previous rounds had audited. Four live copies still said three: reports/technical_risk_register.md C-343's Open Items Summary row reports/technical_risk_register.md C-343's own Trigger: field docs/guides/hetzner_deployment_guide.md release-ritual item 5 scripts/refresh_pipeline.sh header comment Two of the four were inside the entry doing the hunting, one of them its trigger — the field an operator is meant to act on. The summary row said "three steps" and then listed the three checks that constitute step 4, so it described four things while naming three. The script comment is the sharpest. Its very next sentence reads "two copies of a procedure drifting apart is what C-343 IS, and a fix that adds a third copy would be the same bug" — and it had drifted, because a count is a copy too. It carefully avoided duplicating the commands and then duplicated the number, which was the one part that changed. The comment now records that it said THREE until today, five weeks after the fourth block was added. No new register entry: same entry, same mechanism, fifth round. What the round relocates is where the mechanism hides — rounds one to four hunted copies of the procedure and found them in guides and ADRs; this one found that the procedure's cardinality is independently copyable. Also recorded on the entry, because it is the same confusion one layer up: the #501 instrumentation was merged to development and then described to the operator as something the next cron would pick up automatically. It would not have — the server runs a tag and v1.14.0 did not contain it. Merging the instrumentation is not deploying the instrumentation, committed three days after the fourth correction by whoever wrote it. Corrected on #501 before any wait was incurred. Left as written, per #500's precedent that a record is amended and never rewritten to look current: C-331's narrative, ADR-022's "previously gave deploying as three steps", and register_changelog.md's account of a past redeploy. server_quickref.md:136's "all three checks in step 4" is correct — step 4 does hold three commands, and four blocks whose fourth contains three checks is a fair part of why the count drifted. Verification: full suite EXIT=0, 0 FAILED, 0 ERROR; all 14 register-reading test files 41 passed; ruff check . clean; mypy src/ clean (89 files); validate_docs.sh PASSED; bash -n scripts/refresh_pipeline.sh OK. notebooks/acled_cursor_parity.ipynb re-executed from committed fixtures with no network: 3 slices, 106,329 events, every oracle held, 0 errors. Refs #501. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Polichinel
added a commit
that referenced
this pull request
Oct 20, 2026
…ve times in eight days (C-343) (#509) Covers the export phase instrumentation (#501/#502), four rounds of C-343 correction (#497-#500), the release/publish/deploy of v1.15.0, a fifth C-343 round found while writing this up (#507), and a licensing contradiction found while onboarding an external consumer (#508). The v1.11.0 post-mortem named the spine running through everything it reported: "a written record that was true when written and stopped being true without anything failing." Both it and its companion observation — that the post-mortem convention keeps failing to be applied — recurred. Three of the four releases since are unwritten and now never will be. The spine produced five fresh instances in eight days, all in C-343, the fifth found inside the register entry created to track the first, including in its Trigger: field. A sixth variant was committed by the author of the fourth correction three days later: merging the instrumentation is not deploying the instrumentation. The conclusion recorded is not "try harder". Every instance in this window was found by a human asking an unprompted question or by an author re-reading their own work. Five rounds have produced a procedure that is right and nothing that notices when a copy of it goes wrong. Also records what went well rather than only the failures: block 4 of the deploy procedure executed successfully for the first time since it was written, the #501 falsification criterion was committed before any production figure existed, three guards were sabotaged and confirmed red, and the clean- venv publish check caught a defect in itself rather than a false alarm on the release. Verification: validate_docs.sh PASSED; all 14 register-reading test files 41 passed. Every figure quoted is cited to a file rather than recalled; the release count was corrected against `git tag` before committing. Refs #501, #508. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The 2026-10-16 production run took 5h50m. Seven places in the repo claimed a pipeline duration, and they spanned a factor of 24.
data_serving_guide.md:177hetzner_deployment_guide.md:192hetzner_deployment_log.md:296hetzner_deployment_guide.md:466server_quickref.md:41hetzner_deployment_guide.md:30ADR-041:12The closest figure was in an ADR about something else. ADR-041 got it right incidentally, while arguing for content-addressed skipping.
Why it happened
#497 made
server_quickref.mdthe single operational authority and told every other document "if they disagree, the quickref wins." #499 then put "3-4 hours" in it. That figure was a guess — no measurement for it existed anywhere in the repo — and it was handed authority over every document that disagreed.This is the complement of #499's own finding, and it is the worse half:
Why the direction matters more than the magnitude
An over-estimate is harmless. An under-estimate licenses intervention on a healthy run.
"25-45 minutes" tells an operator who is two hours in that the run has hung, and killing a hung run is the reasonable response. That reproduces the 2026-10-16 outage from a document instead of from a dropped SSH connection.
Every stale figure also mis-attributed the cost to harvesting. It is not harvesting:
Corrected by document kind, not uniformly
This is the part worth reviewing.
Four live guides carry the measured figure and point at the quickref, which owns the per-step breakdown.
ADR-041 and the March 2026 deployment log are deliberately NOT rewritten. Both are records of what was true when written. Falsifying a record to make it current is a different defect from the one being fixed, so both carry dated amendments instead — the log's table stays as measured in March, with a note that it is March data and where current figures live.
Also corrected: the healthcheck grace rationale derived 48h from "pipeline runs 2-3 hours". The conclusion survives at ~6h; the reasoning did not, so the reasoning is fixed and the 48h stands.
One finding escapes this PR
The zarr export was ~57 min in ADR-041's June 2026 measurement and 18,427s (5h07m) on 2026-10-16 — roughly 5x, against a store that grew 1.8 → 3.1 GB. Store growth does not account for a factor of five.
The cause is unestablished and deliberately not guessed at here. It is a performance question requiring instrumentation, not a figure requiring correction. Filed as its own issue.
A consequence worth noting: ADR-041 understates its own benefit by ~3.4x. A no-change deploy avoids ~350 min, not the ~103 min it claims. The skip is the single largest runtime saving in the pipeline.
Register
C-343 addendum. No new entry — the trigger would be perpetual. What is registrable belongs on C-343: a consolidation must verify what it promotes, not only absorb what it demotes. Two rounds of this entry have now found the 2026-10-13 work sound in mechanism and unverified in content. The entry's round count is corrected from third to fourth.
Verification
uv run ruff check .— cleanbash docs/validate_docs.sh— PASSEDDocs and register only. No source, no test, no config changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9