release: v1.15.0 — the export gets a phase breakdown, and four rounds of C-343 land (#501) by Polichinel · Pull Request #504 · views-platform/views-datafactory

Polichinel · 2026-10-20T00:33:18Z

MERGE COMMIT ONLY — never squash, never rebase

Stating this first because it is the one irreversible mistake available on this page.

Promoting development → main through a PR creates a merge commit on main that does not exist
on development; the back-merge afterwards heals that by exactly one commit. A squash or rebase
rewrites the release SHAs, so main stops being an ancestor of development and the ancestry the
back-merge maintains is permanently broken — it would not surface until a later release diffed
against a base that never existed. This happened during v1.10.0 (C-340), which is why
scripts/arm_automerge.sh reads the method back instead of trusting that it was set.

tests/test_falsification_deploy_v160_r2.py::TestDF1MergeTopology asserts the invariant, and
.github/workflows/release-topology.yml ("Release hygiene") detects the breakage after the fact.
Neither can undo it.

What is being promoted

Seven commits, 13 files, +426/−100. Zero of them under src/.

PR	What
#497	four more copies said writing the deploy tag was deploying (C-343)
#498	`~` after `sudo` expands as the caller — six verification commands read the wrong home (C-343)
#499	the procedure gets one home, plus the tmux rule the quickref didn't carry (C-343)
#500	seven places claimed a pipeline duration, spanning 24× — the authority's figure had never been measured (C-343)
#502	the zarr export's four phases are timed, printed, and persisted to the export ledger (#501)
#503	bump to 1.15.0

Why this release exists, and why it is dated

The wheel is code-identical to v1.14.0. git diff --name-only v1.14.0 origin/development -- src/
is empty, and [tool.hatch.build.targets.wheel] packages only the nine src/datafactory_* trees. A
consumer installing 1.15.0 from PyPI gets exactly what 1.14.0 gave them.

That is the point. The server deploys by git tag (ADR-022) — refresh_pipeline.sh reads
~/.views-deploy-tag and checks out that exact tag. The two things this release carries,
scripts/export_zarr.py and a set of corrected guides, reach production no other way, because
the wheel does not contain them. Precedent: v1.11.0 and v1.12.0 were both minor bumps with zero
src/ change.

The 2026-10-16 production run took 21,007s (5h50m), of which the zarr export alone was 18,427s
(5h07m) — 88% of the pipeline. ADR-041's Context had the export at ~57 minutes. Nothing in the
repo could attribute the missing four hours to a phase, so nothing could be optimised without
guessing. #502 is the instrument; it is useless until it is on the box.

The monthly cron fires 00:00 on the 21st. A tag and deploy before then gets the measurement on a
run that was happening anyway. Missing it means a deliberate out-of-band six-hour run, or waiting
until 21 November — five weeks with the instrument built, merged and idle.

The pre-registered prediction this release makes measurable

Committed to #501 before any production figure exists, so the result cannot be read as agreement
after the fact:

If scan comes in under ~25% of the export, the strided-memmap explanation is refuted and I
stop advancing it.

grid.npy is C-contiguous with features on the last axis, so grid[:, :, :, i] strides 316 bytes
at F=79 and a 4 KB page yields ~13 useful float32s out of 1024. The synthetic baseline, where
0.1 GB fits in RAM, puts scan at 10.1% — the smallest named phase. The entire claim is that
crossing the RAM boundary inverts that relationship. If it does not, the explanation is wrong
however plausible the page arithmetic looks.

Verification on this tree

Check	Result
full suite, exit code captured	EXIT=0, 0 FAILED, 0 ERROR (XFAILs are documented deferrals)
`uv run ruff check .`	All checks passed
`uv run mypy src/`	Success — 89 source files
`bash docs/validate_docs.sh`	PASSED
`test_falsification_deploy_v160` + `_v160_r2` + `merge_readiness`	10 passed, 1 skipped
`git merge-base --is-ancestor origin/main origin/development`	true before this PR

Exit code captured rather than inferred from absent FAILED lines — 137 is SIGKILL/OOM and prints
no failures at all (C-345).

The one skip is the gate this PR leads to: test_version_equals_the_tag_head_is_on cannot run with
HEAD untagged. Its unskippable half is in publish_package.yml, which compares github.ref_name
against [project].version before the build, so a tag that disagrees with the bump fails the
publish rather than shipping a mislabelled wheel.

What happens after this merges, in order

Tag v1.15.0 and cut the GitHub Release from main — the Release triggers
publish_package.yml.
Verify the publish by installing the published artifact into a clean venv, not by reading a
green tick.
Deploy — all four blocks of docs/guides/server_quickref.md §"Deploy a new version". Step 1
alone is not a deploy (C-343): the pipeline's own git checkout lands a change to
refresh_pipeline.sh one run later and never runs uv sync at all. Block 4 is the verification,
and all three of its commands must name the same version.
Back-merge main → development as a merge commit — the step skipped after every release
before v1.10.0.

Refs #501.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9

chore: sync main back into development after v1.14.0

… it did (C-343) (#497) C-343 was registered in #431 and corrected in scripts/refresh_pipeline.sh and docs/guides/server_quickref.md. Four further copies of the one-step claim were still live, and they are the ones an operator reads: - hetzner_deployment_guide.md said the tag file, then "That's it." This is the file refresh_pipeline.sh itself points to "for full detail". - server_operations.md gave a local `git tag && git push` (which cannot trigger the publish workflow, and cannot be run at all since branch protection) plus the tag file, with no checkout and no `uv sync`. - ADR-022 listed "One-line rollback" as a benefit of the gate, gave a three-step deploy with neither checkout nor `uv sync`, and offered a Verify block naming the tag file and `git describe` but not the installed distribution — the one of the three that was wrong in August. - data_serving_guide.md told newcomers that writing the old tag name back puts you on known-good code. server_quickref.md is now the single operational home and grew the verification it was missing: step 4 checks all three sources, including the installed version, which is the only one that reflects `uv sync`. The four others point there and say the quickref wins on disagreement. ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite: the Decision it records is sound, and what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. C-343 is updated rather than duplicated — same risk, new evidence that the fix reached two of six copies. It stays open; the pre-flight instrument that would catch a seventh copy is still #424. Found because the operator asked, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…s read the wrong home (C-343) (#498) Found by running the corrected deploy procedure on the production host. v1.14.0 deployed cleanly, then step 4's first check failed: $ sudo -u views-deploy cat ~/.views-deploy-tag cat: /home/simmaa_prio/.views-deploy-tag: Permission denied The calling shell expands `~` before sudo runs, so the command reads the operator's home rather than views-deploy's. Six such commands were live in the guides; two were added by the C-343 correction in #497. All are now wrapped in `bash -c '...'`, where the inner shell expands `~` as the target user. One of the six failed green rather than loud: sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null \ || echo "No recent failure" The `||` sat outside `bash -c`, so a permission error on the wrong home printed "No recent failure" — in the command an operator runs to ask whether the last pipeline run broke. The fallback is now inside the quoted command. Step 4's third check also drops the escaped-quote python one-liner for `ls .venv/.../site-packages | grep views_datafactory`, which needs no quoting and prints the dist-info directory — the same evidence form C-343 used in August. C-343 updated with the deploy outcome: `uv sync` reported 1.13.0 -> 1.14.0 on the box, which is direct confirmation that step 3 is load-bearing and the tag file alone would have left 1.13.0 installed. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

… carry (C-343) (#499) The 2026-10-13 consolidation made server_quickref.md the single operational home and told every other guide "if they disagree, the quickref wins." hetzner_deployment_guide.md carried a rule the quickref did not — start tmux before a pipeline run, stated at its line 30 with the reason, and backed by a post-mortem section called "tmux saved us from the SSH drop". Promoting the quickref silently demoted that rule. It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly had completed but before the zarr export, leaving grid.npy current and grid.zarr three weeks stale. Every designed defence held: assemble_grid.py renames from grid.npy.tmp so the grid was whole, .exports_required recorded the owed export, flock released with the process, and check_health.py reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run. Every interactive refresh_pipeline.sh invocation in the guides now opens with `tmux new -s pipeline`: - server_quickref.md §"Run the pipeline" — the rule, the dated evidence, detach/reattach, and the detail that catches people out: tmux does not change the shell prompt, so only the bottom status bar tells you whether you are inside it - hetzner_deployment_guide.md §2.1, §4.5, and the rollback section added in #497 - server_operations.md summary table and the full-rebuild procedure Cron invocations are deliberately unchanged — cron has no controlling terminal and no SIGHUP to receive. The ~/.profile wrong-vs-right example in server_operations.md is also unchanged; adding tmux would obscure the point it exists to make. C-343 gets the pattern rather than a new entry: a consolidation is only safe if the new authority absorbs everything the demoted copies knew, and checking that was not part of the 2026-10-13 work. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…thority's figure was never measured (C-343) (#500) The 2026-10-16 run took 5h50m. The guides claimed 15-30 min, 25-45 min, ~2.5h, 2-3h, 3-4h and ~6h, and the closest figure was in an ADR about something else. `server_quickref.md` was made the single operational authority on 2026-10-13 and given "3-4 hours" — a guess, with no measurement anywhere in the repo, promoted over every document that disagreed. The complement of the tmux finding: consolidation concentrates error as effectively as it concentrates authority. The direction of the error is the hazard. An under-estimate licenses intervention on a healthy run: "25-45 minutes" tells an operator two hours in that the run is hung, and killing a hung run is the reasonable response — reproducing the 2026-10-16 outage from a document instead of a dropped SSH connection. Every stale figure also blamed harvesting; harvest through compile is ~43 min and the zarr export alone is ~5h07m, 88% of the run. Corrected by document kind, not uniformly: - four live guides carry the measured figure and point at the quickref, which owns the per-step breakdown - ADR-041 and the March 2026 deployment log are NOT rewritten — both are records of what was true when written, and falsifying a record to make it current is a different defect from the one being fixed. Both carry dated amendments instead. Also corrected: ADR-041's healthcheck-grace rationale derived 48h from "2-3 hours"; the conclusion survives at ~6h, the reasoning did not. One finding escapes the docs fix and is filed separately: the zarr export was ~57 min in June 2026 and 18,427s (5h07m) on 2026-10-16, a ~5x increase against a store that grew 1.8 -> 3.1 GB. Cause unestablished and deliberately not guessed at. ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not ~103. Verification: ruff clean, validate_docs.sh PASSED, 48 register-reading and citation tests plus 38 deployment-doc guards green. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…uld not be attributed (#501) (#502) The 2026-10-16 run measured the export at 18,427s, 88% of the pipeline and ~5x ADR-041's June figure of ~57 min. Nothing recorded WHERE that time went, so the regression could be stated but not located. Four phases are now timed, printed in the export's summary and written to export_ledger.jsonl as `phase_seconds`: scan the source_sums loop — one traversal of grid.npy per feature write ds.to_zarr + consolidate_metadata — the second traversal verify the round-trip integrity check — full decompression swap the C-144 .tmp -> rename dance Plus `other`, printed explicitly so the columns account for the whole of the existing `Time:` line rather than silently omitting a remainder. The ledger is the point: a print lives in logs/refresh.log, which rotates. `phase_seconds` in the ledger makes the breakdown queryable across runs, which is what turns a single measurement into a trend. Verified empirically, not inferred: - grid.npy is C_CONTIGUOUS with features as the contiguous last axis, so grid[:, :, :, i] strides F x 4 bytes between adjacent (t, h, w) — 316 bytes at F=79. A 4 KB page carries ~13 useful float32s out of 1024. - `source_sums` has exactly one consumer: the verify comparison. The scan phase's full-grid read exists solely to feed that check. NOT optimised. No change to chunking, compression, the slicing pattern, or the integrity check. Each is a plausible remedy for a cause that is not yet established, and the integrity check is an ADR-011/C-144 guarantee that must not be weakened to save time. #501 carries a pre-registered falsification criterion so the first production measurement is a test rather than a confirmation. Three guards in TestZarrExport, both substantive ones drilled by sabotage rather than assumed: delete phase_seconds["swap"] -> test_phase_timings_recorded FAILED inflate the scan timer past total -> test_phases_do_not_exceed_the_total FAILED The phase labels are asserted as an exact set; a subset assertion would let a dropped phase restore the original blindness silently. Verification: full suite exit 0 (0 FAILED, 0 ERROR), ruff clean, mypy src/ clean (89 files), validate_docs.sh PASSED, export-touching test files re-run green against the final tree. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…he whole point (#501) (#503) Not one file under src/ changed since v1.14.0, and the wheel packages only the nine src/datafactory_* trees, so a consumer installing 1.15.0 from PyPI gets exactly what 1.14.0 gave them with different version metadata. That is the point rather than a defect. The server deploys by git tag (ADR-022): refresh_pipeline.sh reads ~/.views-deploy-tag and checks out that exact tag. The two things this release carries — scripts/export_zarr.py and a set of corrected guides — reach production no other way, because the wheel does not contain them. v1.11.0 and v1.12.0 were both minor bumps with zero src/ change, so the shape is house precedent, not a judgement made here. Why now: #502 merged the only instrument that can attribute the zarr export's 5h07m to a phase, and it sits on development where the server cannot see it. The monthly cron fires 00:00 on the 21st. A tag before then gets the measurement on a run that was happening anyway; missing it means a deliberate out-of-band six-hour run or waiting until 21 November. The plan edit is not housekeeping tacked on. TestF7ProductPlanCurrency asserts the product plan's title carries the current minor version, so the version bump alone turns it red — the two edits are one atomic change. Confirmed by breaking it rather than by reading it: with the title reverted to "through v1.14" and the version at 1.15.0 the test FAILED, and went green when the title was restored. uv.lock moves too: it pins the root project version (line 1701), and uv re-synced it on the first uv run after the bump. refresh_pipeline.sh:178 does git checkout -- uv.lock before the tag checkout precisely because the server mutates it locally. Noticed, deliberately not fixed: "## Version History" in the same plan file stops at v1.6 and jumps to v2.0 — eight releases happened without it. The per-release addenda at the top have been the live record since v1.7. Repairing a section stale for eight releases inside a version bump would hide a documentation change inside a release. No tag, no GitHub Release, no deploy. Each is separately gated. Verification: full suite EXIT=0, 0 FAILED, 0 ERROR (XFAILs are documented deferrals); ruff check . clean; mypy src/ clean (89 files); validate_docs.sh PASSED; deploy_v160 + _v160_r2 + merge_readiness 10 passed 1 skipped (HEAD not on a tag, expected pre-release); main is an ancestor of development. Refs #501. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Polichinel and others added 7 commits October 12, 2026 12:52

Merge pull request #496 from views-platform/main

47fa8b4

chore: sync main back into development after v1.14.0

Polichinel merged commit b0c0b17 into main Oct 20, 2026
12 checks passed

This was referenced Oct 20, 2026

chore: sync main back into development after v1.15.0 #506

Merged

docs: post-mortem for v1.15.0 — the spine named in August recurred five times in eight days (C-343) #509

Merged

Navigation Menu

Repository navigation

release: v1.15.0 — the export gets a phase breakdown, and four rounds of C-343 land (#501) - #504

release: v1.15.0 — the export gets a phase breakdown, and four rounds of C-343 land (#501)#504
Polichinel merged 7 commits into
mainfrom
development

Polichinel commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Polichinel commented Oct 20, 2026

MERGE COMMIT ONLY — never squash, never rebase

What is being promoted

Why this release exists, and why it is dated

The pre-registered prediction this release makes measurable

Verification on this tree

What happens after this merges, in order

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant