Epic: Protect/EQL v2 to Stack/EQL v3 guide for DynamoDB and Postgres #1540

Background

Protect users need a tested upgrade to Stack for both DynamoDB and existing Postgres databases. EQL (Encrypt Query Language) v2/v3 names the stored-data generation, not the npm package major. The target is Stack 1.x's v3 API.

The current DynamoDB guide and cross-store tests are useful, but copying values into already-v3 Postgres columns does not prove a migration of an existing EQL v2 Postgres database.

Goal and delivery order

1. Deliver the tested v2-to-v3 guide for both stores first. Include executable examples, passing live CI, accurate citations, supported mappings and exclusions, plus cutover and rollback instructions.
2. Validate guide usability with the Migration Harness. Reuse the existing DynamoDB baseline, add a Postgres-v2 starting state, then use guide-only agent Runs and evidence-backed Editor revisions.

The guide milestone does not wait for the automated Loop. Migration to EQL v4 comes afterwards. PRs #1496 and #1497 cover later v4 migration work and are not dependencies of this guide.

Authoritative specification

The full architecture, requirements and acceptance contract remain in ADR PR #1575, including its approved v2-to-v3 priority amendment:

• Migration Harness ADR
• Domain glossary

This epic tracks delivery; it does not duplicate the specification. The design was accepted in merged PR #1575; implementation completion is tracked separately.

First milestone: tested guide

• Complete the tested Protect-to-Stack v3 DynamoDB guide #1604 — complete the DynamoDB procedure using feat(e2e): verify and document Protect v2 to Stack v3 migrations for both stores #1519's evidence.
• Deliver a tested Protect/EQL v2 to Stack/EQL v3 Postgres procedure #1603 — completed in merged feat(e2e): verify and document Protect v2 to Stack v3 migrations for both stores #1519: Postgres v2-to-v3 procedure, executable example and live tests.

Both can start from #1519; neither depends on harness automation or v4.

Second milestone: guide usability

• Migration harness: Run skeleton and behaviour checks on Legacy Data #1534 — original Run and behaviour baseline (PR feat(e2e): scaffold migration harness runs and isolation probes #1576).
• Migration harness: Oracle no-rewrite layer #1535 — original no-rewrite layer (PR feat(e2e): detect legacy ciphertext rewrites in migration Runs #1580).
• Migration harness: Reference Migration, static layer and selftest #1536 — original Reference Migration and self-tests (PR feat(e2e): verify the migration Oracle with reference and broken applications #1581).
• Verify the Postgres v2-to-v3 guide through real harness Runs #1605 — Postgres-v2 Run extension; blocked by Deliver a tested Protect/EQL v2 to Stack/EQL v3 Postgres procedure #1603 and Migration harness: Reference Migration, static layer and selftest #1536.
• Migration harness: real Migrator (headless, isolated, Friction Log) #1537 — real Migrator, with guide-only acceptance for both procedures.
• Migration harness: Iteration with a gated Editor #1538 — gated Editor and procedure-specific Iterations.
• Migration harness: loop with stop condition #1539 — Loop convergence for both procedures on the same final guide hash.

Existing baseline ticket acceptance is retained. Open PRs are not treated as completed issues.

Delivery status — 2026-11-12

The source guide is #1620 at a6d33c9e8f7acc65158553bbce5b7b6ff62165a2, SHA-256 6e730d48fdba975c55273f3c9d58f27c21928babe8607c862a6b0eca996bc242. It now generates the complete standalone PostgreSQL script from the executable module, verifies Protect source plaintexts independently against Stack, batches SDK operations, and resolves catalog identities without physical-column/search-path assumptions. The explicit credentials snippet and immutable links are corrected.

Current live E2E passed: 14 guide tests on each of PostgreSQL 17 and 18, including the actual pasted script, dropped-column fixture, both restricted search paths and missing-primary-key refusal. The combined PG17 suite passed 27 tests. Local verification passed 74 focused tests, strict example/snippet compilation against published packages, typechecking and the lint gate. The previous PG17 fixture failure reproduced once; the test-only follow-up explicitly targets public and checks setup before migration.

Customer documentation is synchronized in cipherstash/docs#118 at signed 5259b44995b09ab7d3cac62003e77d3b68c58c00, pinned to the byte-identical guide at e7d6a02d3. Source equivalence, types, content, links, scoped lint, production build and docs CI pass. Two pre-existing full-docs-lint formatting errors remain. Nothing is merged or deployed.

Historical guide-only evidence

One fresh sample per store passed at harness 5261d3a5c0ae6a4750a7a1102d08436a1d314725 against previous guide hash 46106cc0a71dbeb757bcb0b3889c6fc5741bac201afe807ed8cde0a035d3d649, using the existing Colima harness VM outside CI. These runs preceded the canonical-script repair and do not validate the current guide:

Runtime: Claude Code 2.1.322, resolved model claude-sonnet-5-5. Both retained complete private command/response artifacts. Independent review confirmed the actual guide-read response hash, use of the staged friction schema, no supplementary documentation supplied to the model, and no successful forbidden-access events. Filesystem README events remain conservatively marked possible; npm extraction produces those events, and command/response review is the evidence for the guide-only claim.

Report SHA-256 digests: DynamoDB 2c081ffc88daed4969d64907a87fe8fbad5f3a20f5efd7ea00d9232ca439587b; PostgreSQL 52fe3656562fda47c371e9a2a634e97de2733afa80dc7bac067f9ff0694e0ded. Raw artifacts remain private; secrets are not published.

Earlier attempts remain recorded with their original outcomes: PostgreSQL exposed the missing inline preflight, and a subsequent DynamoDB attempt passed behavior but produced an invalid section slug. The harness now supplies its exact section catalog/schema; no historical failure was relabelled.

Remaining work

These are one clean sample per store, not ADR convergence or customer staging proof. #1538/#1539 remain unimplemented. Customer schema/query inventory and actual AWS/PostgreSQL operational rehearsal, including cutover and rollback, remain required before customer-specific sign-off.

Harness integration remains pending required reviews and the acceptance-evidence follow-ups below. Deterministic CI at the schema-update heads reproduced an intermittent unmatched vfork observer continuation before behavior checks; signed repairs 62b09150f and 9ed2ff135 supply explicit PID tracing and identify the actor independently of strace's temporary capability-probe children. A red regression reproduced the helper-PID failure; independent review, 31 supervisor tests, 131 base tests and 220 combined tests pass locally (21 credentialed tests deferred to CI). The preceding heads repeatedly failed on missing actor-exit records, despite one passing retry. Current adapter head 7e12eaa0d09b8e0f28d7335e765109b975ab95cb passed current-head CI. All nine isolation and nine Test JS workflows across the five repair heads passed without retries. Both PostgreSQL branch logs show 224 passing harness tests, and both adapter logs show 241; all include the complete 21-case live matrix with no harness skips. The successful model evidence above belongs to its recorded harness revision and does not validate a later repair. No merge bypass, real-model CI workflow, paid host, VM restart or production data change was used.

Acceptance audit: four #1537 isolation criteria remain partially evidenced. Follow-up proof must cover each Protected Resource class, actual Oracle/foreign-Run/model-service targets, Docker socket absence or connection denial (the current probe only attempts a file read), and canary absence across actor-visible mounted files. These are evidence gaps, not demonstrated escapes. Startup supervision and active broken-monitoring/forbidden-access controls are evidenced at the recorded model-run revision; later tracing revisions require separate CI validation.