From cdd7b93b4f90e72f90847b91f718bf07498773f3 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 11 Nov 2026 14:07:57 +1100 Subject: [PATCH 1/7] docs(e2e): qualify in-memory migration evidence The cited compatibility assertion compares stored item objects in memory. Name that scope beside the byte-preservation claim so readers do not infer an AWS persistence guarantee. Refs #1604. --- docs/migration/protect-dynamodb-to-stack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 25fb52210..4ba1f3d85 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -44,7 +44,7 @@ Stack reads and writes the same two attributes: - It decrypts a legacy `__source` with no read option (`compat › every legacy item decrypts, one at a time, with no read option`, `compat › every legacy item decrypts in one bulk call`), and gets the same plaintext protect-dynamodb does (`compat › agrees with what protect-dynamodb itself decrypts`). - Its `encryptQuery` mints the same `__hmac` that protect-dynamodb's `createSearchTerms` did (`compat › matches protect-dynamodb createSearchTerms for the same value`), so the tested text and numeric equality queries match the expected legacy items (`compat › every equality query finds exactly the legacy rows it should`). - A new Stack write sits beside legacy items, and one query and one bulk read cover both (`compat › new Stack writes sit beside legacy items: one query and one bulk read cover both`). -- Reading and querying leave the stored items byte-identical (`compat › Stack's reads and queries left every stored item byte-identical`). +- Reading and querying leave the in-memory stored item objects byte-identical (`compat › Stack's reads and queries left every stored item byte-identical`). - A tampered ciphertext fails to decrypt; it never returns wrong plaintext (`compat › fails, never returns wrong plaintext, when the ciphertext is tampered with`). ## Steps From abae8e5ae986e311d0cd4e51579eeb66c995e3aa Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 11 Nov 2026 16:15:42 +1100 Subject: [PATCH 2/7] docs(migration): make the PostgreSQL conversion and explicit credentials self-contained The first real guide-only Runs (Claude Sonnet, isolated Migration Harness) found two gaps on the PostgreSQL path. Step 3 handed the conversion to e2e/migration-guide/postgres.ts, which a reader of the guide alone does not have, so the agent rebuilt it from the prose list. The guide never showed how to pass credentials to Encryption explicitly, so the agent read the package typings. Step 3 now carries the core of the tested conversion script and says to run it as a one-off script in the maintenance window, not at application startup (the agent had put DDL in createApp). Both step 2s show Encryption's `config` with the four credential fields. Links into this repository are now absolute GitHub URLs, so they resolve from the published guide. Refs #1603 Refs #1604 --- docs/migration/protect-dynamodb-to-stack.md | 110 +++++++++++++++++++- 1 file changed, 106 insertions(+), 4 deletions(-) diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 4ba1f3d85..1893b330c 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -30,7 +30,7 @@ pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode= The native build needs the repository's Rust toolchain and trusted mise configuration. These commands are for maintainers verifying the guide; a customer installing Stack from npm receives prebuilt binaries. The crossover task also runs the PostgreSQL upgrade and executable DynamoDB example. Credentialed tests fail if configuration is missing; a skipped or unrun suite is not migration evidence. -The [executable examples](../../e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. +The [executable examples](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. ## Why the supported scalar attributes need no re-encryption @@ -64,7 +64,21 @@ npm install --save-exact @cipherstash/stack@1.2.1 If the app's `package.json` also declares `@cipherstash/schema` or `@cipherstash/protect-ffi` directly, remove those with `npm uninstall`. Commit the resulting package manifest and lockfile. Stack retains its own newer FFI dependency; do not remove that transitive dependency. -Stack needs Node.js 22 or later. It reads the same four `CS_*` variables from the environment, so retain the same workspace and key access used to write the data. If the app passed credentials explicitly to Protect, populate these environment variables or configure the equivalent Stack credentials. If the app passed a `keyset` to `protect()`, pass the same keyset to `Encryption` as `config.keyset`. A client bound to a different keyset either fails to decrypt or, if it was granted the old keyset, decrypts but mints terms that match no existing `__hmac`. Keysets are untested by these suites. +Stack needs Node.js 22 or later. It reads the same four `CS_*` variables from the environment, so retain the same workspace and key access used to write the data. If the app passed credentials explicitly to Protect, populate these environment variables or pass the same four values to `Encryption` as `config`: + +```typescript +const client = await Encryption({ + schemas: [users], + config: { + workspaceCrn: env.CS_WORKSPACE_CRN, + clientId: env.CS_CLIENT_ID, + clientKey: env.CS_CLIENT_KEY, + accessKey: env.CS_CLIENT_ACCESS_KEY, + }, +}) +``` + +If the app passed a `keyset` to `protect()`, pass the same keyset to `Encryption` as `config.keyset`. A client bound to a different keyset either fails to decrypt or, if it was granted the old keyset, decrypts but mints terms that match no existing `__hmac`. Keysets are untested by these suites. Stack finds its service endpoints from `CS_WORKSPACE_CRN`. Protect 10.2.1 did not: outside `ap-southeast-2` it needed `CS_CTS_HOST` and `CS_ZEROKMS_HOST` set explicitly. Stack still honours both variables when they are set, so remove them from ordinary managed deployments. Keep them only for a deliberately configured private or custom endpoint, and check that endpoint before relying on it. @@ -280,13 +294,17 @@ const customers = encryptedTable('customers', { const client = await Encryption({ schemas: [customers] }) ``` +If the application receives its credentials other than through `process.env`, pass them as `config`, as shown in DynamoDB step 2. + Configure reads and writes to use the `_v3` physical columns after cutover, while passing the original logical schema above to the encryption client. Handle every SDK `{ failure }` before using `{ data }`. ### 3. Stop writes and convert This first procedure requires a **write-maintenance window**. Stop all application, background and administrative writers; a transaction's table lock alone does not stop an old writer from resuming immediately after commit. Keep writes paused through verification and the rollback decision. -The [executable migration](../../e2e/migration-guide/postgres.ts) performs these steps in one transaction: +Run the conversion as a one-off script during the window, not at application startup. An application that alters its own schema on boot repeats DDL on every deploy and takes an `ACCESS EXCLUSIVE` lock while other instances are starting. + +The conversion performs these steps in one transaction: 1. Lock the source table and reject unsupported structure/dependencies before DDL. 2. Validate each legacy envelope and plaintext against the supported mappings. Recompute supported equality terms with Stack and compare them to the stored terms; fail on disagreement. @@ -295,6 +313,90 @@ The [executable migration](../../e2e/migration-guide/postgres.ts) performs these 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. +The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. `legacy` is a Protect 10.2.1 client built from the original `csTable` declarations, so the source is decrypted by the build that wrote it; install it beside Stack for the duration of the script (the tested example does the same). The full example also runs the step 1 dependency preflight (constraints, secondary indexes, views, triggers, policies, inheritance) right after the lock; do not skip it. + +```typescript +import postgres from 'postgres' + +const sql = postgres(process.env.DATABASE_URL) +const columns = ['email', 'age', 'notes'] as const +function ok(result, step) { + if (result.failure) throw new Error(`${step}: ${result.failure.message}`) + return result.data +} + +await sql.begin(async (tx) => { + // 1. Lock, then refuse any shape other than pk plus the three v2 columns. + await tx`LOCK TABLE customers IN ACCESS EXCLUSIVE MODE` + const shape = await tx`SELECT attname, atttypid::regtype::text AS type + FROM pg_attribute WHERE attrelid = 'customers'::regclass + AND attnum > 0 AND NOT attisdropped ORDER BY attnum` + if (JSON.stringify(shape.map((a) => [a.attname, a.type])) !== JSON.stringify([ + ['pk', 'text'], ['email', 'eql_v2_encrypted'], + ['age', 'eql_v2_encrypted'], ['notes', 'eql_v2_encrypted'], + ])) throw new Error('Unsupported columns') + // (Run the dependency preflight here.) + + // 2. Validate every envelope and recompute its equality term with Stack. + const rows = await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, + notes::jsonb AS notes FROM customers ORDER BY pk` + if (rows.length === 0) throw new Error('Empty source') + for (const row of rows) { + for (const column of columns) { + const p = row[column] + if (p?.v !== 2 || p.k !== 'ct' || typeof p.c !== 'string' || + p.i?.t !== 'customers' || p.i?.c !== column || + Object.keys(p).some((k) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(k)) || + p.ob != null || p.bf != null || + (column === 'notes' ? p.hm != null : typeof p.hm !== 'string')) + throw new Error(`Unsupported legacy payload in ${column}`) + const plain = ok(await legacy.decrypt(p), 'decrypt source') + if (column === 'age' ? !Number.isSafeInteger(plain) : typeof plain !== 'string') + throw new Error(`Unsupported plaintext in ${column}`) + if (column === 'notes') continue + const term = ok(await client.encryptQuery(plain, { + table: customers, column: customers[column], + }), 'recompute term') + if (term?.hm !== p.hm) throw new Error(`Equality term mismatch in ${column}`) + } + } + + // 3. Add the target columns. + await tx`ALTER TABLE customers + ADD COLUMN email_v3 public.eql_v3_text_eq, + ADD COLUMN age_v3 public.eql_v3_integer_eq, + ADD COLUMN notes_v3 public.eql_v3_text` + + // 4. Rebuild only the envelope; ciphertext and equality term are copied unchanged. + for (const column of columns) { + await tx`UPDATE customers SET ${tx(`${column}_v3`)} = jsonb_build_object( + 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' + ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb + ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` + } + + // 5. Index the equality terms. + await tx`CREATE INDEX customers_email_v3_idx ON customers (eql_v3.eq_term(email_v3))` + await tx`CREATE INDEX customers_age_v3_idx ON customers (eql_v3.eq_term(age_v3))` + + // 6. Verify through Stack before committing; any throw rolls back the DDL too. + const converted = await tx`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, + notes_v3::jsonb AS notes FROM customers ORDER BY pk` + for (let i = 0; i < rows.length; i++) { + for (const column of columns) { + const before = rows[i][column] + const after = converted[i][column] + if (after.c !== before.c || (after.hm ?? null) !== (before.hm ?? null) || + ok(await client.decrypt(after), 'decrypt target') !== + ok(await legacy.decrypt(before), 'decrypt source')) + throw new Error(`Conversion verification failed in ${column}`) + } + } +}) +await sql`ANALYZE customers` +await sql.end() +``` + Only the envelope and database structure change for these proven mappings; the ciphertext and equality terms are copied unchanged. Do not generalize this to other capabilities or to v4 (`postgres-upgrade › migrates actual v2 columns, preserves ciphertext and queries, and rolls back before writes resume`). The example reads the small source table into memory and validates each value. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. @@ -332,7 +434,7 @@ In rehearsal, test fresh Stack writes beside migrated rows, reads, lookups and u ### Reproduce the PostgreSQL example -Use the [npm example instructions](../../e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. +Use the [npm example instructions](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. ## Optional: moving values between DynamoDB and Postgres From 8fb583930bebd79c27a785a463dcf416a6bf1889 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 11 Nov 2026 17:28:44 +1100 Subject: [PATCH 3/7] docs(migration): decrypt the PostgreSQL v2 source with Stack in the conversion script The in-guide conversion script decrypted the source with a Protect 10.2.1 client, but PostgreSQL step 2 had just uninstalled Protect. A real Claude Migrator recorded this as actionable friction and decrypted the v2 envelopes with Stack instead, a path no suite tested. A new credentialed test in protect-postgres-upgrade proves Stack 1.x decrypts every Protect-written v2 envelope read from the eql_v2_encrypted columns to the same plaintext as Protect. The guide's script now decrypts the source with client.decrypt in both the validate and verify steps and cites that test. The executable example still cross-checks with the preserved Protect build, which the guide says. Refs #1603 --- docs/migration/protect-dynamodb-to-stack.md | 6 ++-- .../protect-postgres-upgrade.e2e.test.ts | 35 +++++++++++++++++++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 1893b330c..77f71cb71 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -313,7 +313,7 @@ The conversion performs these steps in one transaction: 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. -The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. `legacy` is a Protect 10.2.1 client built from the original `csTable` declarations, so the source is decrypted by the build that wrote it; install it beside Stack for the duration of the script (the tested example does the same). The full example also runs the step 1 dependency preflight (constraints, secondary indexes, views, triggers, policies, inheritance) right after the lock; do not skip it. +The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. The script decrypts the source v2 envelopes with Stack's `client.decrypt`, which accepts Protect-written v2 payloads, so no Protect client is needed after step 2 removed it (`postgres-upgrade › decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect`). The executable example additionally cross-checks each source value with the preserved Protect build; keeping that old build for rollback (step 1) is unchanged. The full example also runs the step 1 dependency preflight (constraints, secondary indexes, views, triggers, policies, inheritance) right after the lock; do not skip it. ```typescript import postgres from 'postgres' @@ -350,7 +350,7 @@ await sql.begin(async (tx) => { p.ob != null || p.bf != null || (column === 'notes' ? p.hm != null : typeof p.hm !== 'string')) throw new Error(`Unsupported legacy payload in ${column}`) - const plain = ok(await legacy.decrypt(p), 'decrypt source') + const plain = ok(await client.decrypt(p), 'decrypt source') if (column === 'age' ? !Number.isSafeInteger(plain) : typeof plain !== 'string') throw new Error(`Unsupported plaintext in ${column}`) if (column === 'notes') continue @@ -388,7 +388,7 @@ await sql.begin(async (tx) => { const after = converted[i][column] if (after.c !== before.c || (after.hm ?? null) !== (before.hm ?? null) || ok(await client.decrypt(after), 'decrypt target') !== - ok(await legacy.decrypt(before), 'decrypt source')) + ok(await client.decrypt(before), 'decrypt source')) throw new Error(`Conversion verification failed in ${column}`) } } diff --git a/e2e/tests/protect-postgres-upgrade.e2e.test.ts b/e2e/tests/protect-postgres-upgrade.e2e.test.ts index cfeb9d4e5..11f9a1fbc 100644 --- a/e2e/tests/protect-postgres-upgrade.e2e.test.ts +++ b/e2e/tests/protect-postgres-upgrade.e2e.test.ts @@ -1,6 +1,9 @@ +import { csColumn, csTable } from '@cipherstash/protect' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' import { installEqlV3 } from '@cipherstash/test-kit/install' import postgres from 'postgres' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' +import { legacyProtect, ok } from '../migration-guide/legacy.ts' import { createExample } from '../migration-guide/postgres.ts' import { authConfigured } from './helpers/legacy-protect' @@ -74,6 +77,38 @@ describe('Protect Postgres v2 to Stack v3', () => { ).toEqual(before) }, 120_000) + // Evidence for decrypting the source without Protect during conversion. The + // envelopes are the real Protect 10.2.1 writes read back from the + // eql_v2_encrypted columns, not fixtures minted by Stack's own protect-ffi. + it('decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect', async () => { + const legacySchema = csTable('customers', { + email: csColumn('email').equality(), + age: csColumn('age').dataType('number').equality(), + notes: csColumn('notes'), + }) + const schema = encryptedTable('customers', { + email: types.TextEq('email'), + age: types.IntegerEq('age'), + notes: types.Text('notes'), + }) + const legacy = await legacyProtect([legacySchema]) + const stack = await Encryption({ schemas: [schema] }) + const rows = + await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(table)} ORDER BY pk` + expect(rows.map((row) => row.pk)).toEqual(app.corpus.map((row) => row.pk)) + for (const [i, row] of rows.entries()) { + for (const column of ['email', 'age', 'notes'] as const) { + const payload = row[column] + expect(payload.v).toBe(2) + expect(payload.k).toBe('ct') + const fromProtect = ok(await legacy.decrypt(payload), 'protect decrypt') + const fromStack = ok(await stack.decrypt(payload), 'stack decrypt') + expect(fromStack).toStrictEqual(fromProtect) + expect(fromStack).toStrictEqual(app.corpus[i][column]) + } + } + }, 120_000) + it('reads and queries fresh Stack writes beside legacy rows and refuses stale rollback', async () => { await app.migrate() const fresh = await app.write() From 0ecabc3f62ee8a117c467981ce00d312e6cfc636 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 12 Nov 2026 12:08:26 +1100 Subject: [PATCH 4/7] docs(migration): include the PostgreSQL dependency preflight Guide-only migration exposed an omitted safety check that required opening the example. Inline the tested initial-conversion checks and describe the existing refusal on a repeated conversion so the one-off procedure is self-contained without introducing new migration behavior. Refs #1540, #1605. --- docs/migration/protect-dynamodb-to-stack.md | 22 +++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 77f71cb71..5fcf71439 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -313,7 +313,7 @@ The conversion performs these steps in one transaction: 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. -The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. The script decrypts the source v2 envelopes with Stack's `client.decrypt`, which accepts Protect-written v2 payloads, so no Protect client is needed after step 2 removed it (`postgres-upgrade › decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect`). The executable example additionally cross-checks each source value with the preserved Protect build; keeping that old build for rollback (step 1) is unchanged. The full example also runs the step 1 dependency preflight (constraints, secondary indexes, views, triggers, policies, inheritance) right after the lock; do not skip it. +The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. The script decrypts the source v2 envelopes with Stack's `client.decrypt`, which accepts Protect-written v2 payloads, so no Protect client is needed after step 2 removed it (`postgres-upgrade › decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect`). The executable example additionally cross-checks each source value with the preserved Protect build; keeping that old build for rollback (step 1) is unchanged. The dependency preflight below matches the executable example's initial-migration checks: it refuses extra constraints, secondary indexes, views, triggers, policies and inheritance before changing data or adding columns. PostgreSQL 18 records the primary key column's NOT NULL constraint in `pg_constraint`; the check permits that constraint while still refusing NOT NULL on the encrypted source columns. ```typescript import postgres from 'postgres' @@ -335,7 +335,23 @@ await sql.begin(async (tx) => { ['pk', 'text'], ['email', 'eql_v2_encrypted'], ['age', 'eql_v2_encrypted'], ['notes', 'eql_v2_encrypted'], ])) throw new Error('Unsupported columns') - // (Run the dependency preflight here.) + const dependencies = await tx`SELECT + EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = 'customers'::regclass AND NOT tgisinternal) + OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = 'customers'::regclass OR confrelid = 'customers'::regclass) + -- PostgreSQL 18 also lists the primary key column's NOT NULL here. + AND NOT (contype IN ('p', 'n') AND conrelid = 'customers'::regclass AND conkey = ARRAY[1]::smallint[])) + -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint. + OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = 'customers'::regclass + AND attnum > 1 AND NOT attisdropped AND attnotnull) + -- Initial conversion allows no secondary indexes, including target indexes. + OR EXISTS (SELECT 1 FROM pg_index WHERE indrelid = 'customers'::regclass AND NOT indisprimary) + OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = 'customers'::regclass AND classid = 'pg_rewrite'::regclass) + OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = 'customers'::regclass) + OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = 'customers'::regclass) + OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = 'customers'::regclass OR inhrelid = 'customers'::regclass) + AS unsupported` + if (dependencies[0].unsupported) + throw new Error('Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first') // 2. Validate every envelope and recompute its equality term with Stack. const rows = await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, @@ -397,6 +413,8 @@ await sql`ANALYZE customers` await sql.end() ``` +**Rerunning the conversion:** after a successful commit, the table includes the three `_v3` columns. Running this conversion again fails the exact source-shape check before any DDL or data changes; the transaction rolls back without altering the completed migration. It is a one-off conversion, not an idempotent startup hook. Do not treat the mere presence of target columns as proof that a migration completed, and do not drop them to force a rerun. If a transaction failed, diagnose the failure and confirm its changes rolled back before deciding whether to retry. If the client lost the connection around commit, inspect and verify the database state first. Continue with verification and cutover after a successful migration. + Only the envelope and database structure change for these proven mappings; the ciphertext and equality terms are copied unchanged. Do not generalize this to other capabilities or to v4 (`postgres-upgrade › migrates actual v2 columns, preserves ciphertext and queries, and rolls back before writes resume`). The example reads the small source table into memory and validates each value. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. From 21631ecf68fd5356fdd4c2631550a46931123dc4 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 12 Nov 2026 15:42:50 +1100 Subject: [PATCH 5/7] fix(docs): execute the same Postgres conversion in the guide and tests A handwritten guide copy could compile incorrectly and validate Stack against itself while the executable example passed. Generate the standalone script from one typed module, retain Protect source plaintexts for independent verification, and exercise the pasted script in live tests. Resolve catalog identities without physical-column or search-path assumptions and batch SDK work under the lock. Refs #1604, #1540 --- docs/migration/protect-dynamodb-to-stack.md | 403 ++++++++++++++---- e2e/migration-guide/README.md | 4 +- e2e/migration-guide/legacy.ts | 43 +- e2e/migration-guide/postgres-conversion.ts | 294 +++++++++++++ e2e/migration-guide/postgres.ts | 199 ++------- e2e/tests/protect-guide-script.e2e.test.ts | 332 +++++++++++++++ .../protect-postgres-upgrade.e2e.test.ts | 141 ++++-- scripts/sync-migration-guide.mjs | 26 ++ 8 files changed, 1098 insertions(+), 344 deletions(-) create mode 100644 e2e/migration-guide/postgres-conversion.ts create mode 100644 e2e/tests/protect-guide-script.e2e.test.ts create mode 100644 scripts/sync-migration-guide.mjs diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 5fcf71439..c22703d20 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -30,7 +30,7 @@ pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode= The native build needs the repository's Rust toolchain and trusted mise configuration. These commands are for maintainers verifying the guide; a customer installing Stack from npm receives prebuilt binaries. The crossover task also runs the PostgreSQL upgrade and executable DynamoDB example. Credentialed tests fail if configuration is missing; a skipped or unrun suite is not migration evidence. -The [executable examples](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. +The [executable examples](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. ## Why the supported scalar attributes need no re-encryption @@ -70,10 +70,10 @@ Stack needs Node.js 22 or later. It reads the same four `CS_*` variables from th const client = await Encryption({ schemas: [users], config: { - workspaceCrn: env.CS_WORKSPACE_CRN, - clientId: env.CS_CLIENT_ID, - clientKey: env.CS_CLIENT_KEY, - accessKey: env.CS_CLIENT_ACCESS_KEY, + workspaceCrn: process.env.CS_WORKSPACE_CRN, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, }, }) ``` @@ -313,111 +313,326 @@ The conversion performs these steps in one transaction: 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. -The script below is the core of the tested [executable migration](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/postgres.ts), for the `customers` table above. `client` and `customers` are the Stack client and schema from step 2. The script decrypts the source v2 envelopes with Stack's `client.decrypt`, which accepts Protect-written v2 payloads, so no Protect client is needed after step 2 removed it (`postgres-upgrade › decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect`). The executable example additionally cross-checks each source value with the preserved Protect build; keeping that old build for rollback (step 1) is unchanged. The dependency preflight below matches the executable example's initial-migration checks: it refuses extra constraints, secondary indexes, views, triggers, policies and inheritance before changing data or adding columns. PostgreSQL 18 records the primary key column's NOT NULL constraint in `pg_constraint`; the check permits that constraint while still refusing NOT NULL on the encrypted source columns. +The complete script below is generated verbatim from the [canonical executable conversion](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/postgres-conversion.ts). The npm example and live suite use this same implementation; a hermetic check rejects snippet drift and strict TypeScript errors. The live suite also executes the pasted script itself (`postgres-upgrade › runs the standalone guide script against Protect-written source columns`). + +Prepare a **separate migration-tool directory** before the maintenance window. The new application from step 2 has removed Protect; this one-off tool deliberately retains both pinned SDKs so source validation is independent of Stack: + +```bash +npm init -y +npm pkg set type=module +npm install --save-exact @cipherstash/protect@10.2.1 @cipherstash/stack@1.2.1 postgres@3.4.9 +``` + +Save the complete snippet as `migrate.ts` in that directory. Retain its generated lockfile. Set the original four `CS_*` credentials and `DATABASE_URL`, then run `node --env-file=.env migrate.ts` with Node.js 22.18 or later. It defaults to `public.customers`; set `MIGRATION_TABLE` to another unqualified table name only if it has the same supported shape and logical `customers` identities. This managed-host example temporarily derives Protect's endpoints from the workspace region; adapt and rehearse it separately for private/custom endpoints or a non-default keyset. + +The preflight requires a primary key on `pk`, identifies its physical attribute from the catalog, and compares column type OIDs. Dropped predecessor columns and a search path without `public` are supported. Extra constraints, secondary indexes, views, triggers, policies and inheritance remain unsupported. + + ```typescript +import { pathToFileURL } from 'node:url' +import { csColumn, csTable, protect } from '@cipherstash/protect' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' import postgres from 'postgres' -const sql = postgres(process.env.DATABASE_URL) -const columns = ['email', 'age', 'notes'] as const -function ok(result, step) { - if (result.failure) throw new Error(`${step}: ${result.failure.message}`) - return result.data +/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ +export async function legacyProtect( + schemas: Parameters[0]['schemas'], +) { + const crn = process.env.CS_WORKSPACE_CRN + if (!crn) throw new Error('CS_WORKSPACE_CRN is required') + const region = crn.split(':')[1] + const saved = { + CS_CTS_HOST: process.env.CS_CTS_HOST, + CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, + } + process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` + process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` + try { + return await protect({ + schemas, + workspaceCrn: crn, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, + }) + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key] + else process.env[key] = value + } + } } -await sql.begin(async (tx) => { - // 1. Lock, then refuse any shape other than pk plus the three v2 columns. - await tx`LOCK TABLE customers IN ACCESS EXCLUSIVE MODE` - const shape = await tx`SELECT attname, atttypid::regtype::text AS type - FROM pg_attribute WHERE attrelid = 'customers'::regclass - AND attnum > 0 AND NOT attisdropped ORDER BY attnum` - if (JSON.stringify(shape.map((a) => [a.attname, a.type])) !== JSON.stringify([ - ['pk', 'text'], ['email', 'eql_v2_encrypted'], - ['age', 'eql_v2_encrypted'], ['notes', 'eql_v2_encrypted'], - ])) throw new Error('Unsupported columns') - const dependencies = await tx`SELECT - EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = 'customers'::regclass AND NOT tgisinternal) - OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = 'customers'::regclass OR confrelid = 'customers'::regclass) - -- PostgreSQL 18 also lists the primary key column's NOT NULL here. - AND NOT (contype IN ('p', 'n') AND conrelid = 'customers'::regclass AND conkey = ARRAY[1]::smallint[])) - -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint. - OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = 'customers'::regclass - AND attnum > 1 AND NOT attisdropped AND attnotnull) - -- Initial conversion allows no secondary indexes, including target indexes. - OR EXISTS (SELECT 1 FROM pg_index WHERE indrelid = 'customers'::regclass AND NOT indisprimary) - OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = 'customers'::regclass AND classid = 'pg_rewrite'::regclass) - OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = 'customers'::regclass) - OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = 'customers'::regclass) - OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = 'customers'::regclass OR inhrelid = 'customers'::regclass) - AS unsupported` - if (dependencies[0].unsupported) - throw new Error('Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first') - - // 2. Validate every envelope and recompute its equality term with Stack. - const rows = await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, - notes::jsonb AS notes FROM customers ORDER BY pk` - if (rows.length === 0) throw new Error('Empty source') - for (const row of rows) { - for (const column of columns) { - const p = row[column] - if (p?.v !== 2 || p.k !== 'ct' || typeof p.c !== 'string' || - p.i?.t !== 'customers' || p.i?.c !== column || - Object.keys(p).some((k) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(k)) || - p.ob != null || p.bf != null || - (column === 'notes' ? p.hm != null : typeof p.hm !== 'string')) - throw new Error(`Unsupported legacy payload in ${column}`) - const plain = ok(await client.decrypt(p), 'decrypt source') - if (column === 'age' ? !Number.isSafeInteger(plain) : typeof plain !== 'string') - throw new Error(`Unsupported plaintext in ${column}`) - if (column === 'notes') continue - const term = ok(await client.encryptQuery(plain, { - table: customers, column: customers[column], - }), 'recompute term') - if (term?.hm !== p.hm) throw new Error(`Equality term mismatch in ${column}`) - } +/** Check every SDK result; never continue with a failed encryption operation. */ +export function ok( + result: { data: T; failure?: undefined } | { failure: unknown }, + label: string, +): T { + if ('failure' in result && result.failure !== undefined) { + throw new Error(`${label} failed`, { cause: result.failure }) } + if ('data' in result) return result.data + throw new Error(`${label} returned no data`) +} - // 3. Add the target columns. - await tx`ALTER TABLE customers - ADD COLUMN email_v3 public.eql_v3_text_eq, - ADD COLUMN age_v3 public.eql_v3_integer_eq, - ADD COLUMN notes_v3 public.eql_v3_text` - - // 4. Rebuild only the envelope; ciphertext and equality term are copied unchanged. - for (const column of columns) { - await tx`UPDATE customers SET ${tx(`${column}_v3`)} = jsonb_build_object( - 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' - ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb - ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` +export async function createMigrationClients() { + const legacySchema = csTable('customers', { + email: csColumn('email').equality(), + age: csColumn('age').dataType('number').equality(), + notes: csColumn('notes'), + }) + const schema = encryptedTable('customers', { + email: types.TextEq('email'), + age: types.IntegerEq('age'), + notes: types.Text('notes'), + }) + const legacy = await legacyProtect([legacySchema]) + const stack = await Encryption({ schemas: [schema] }) + const columns = ['email', 'age', 'notes'] as const + return { legacySchema, schema, legacy, stack, columns } +} + +/** Bulk success is not enough: reject missing, reordered or failed items. */ +function verifiedBulk( + items: readonly ( + | { id?: string; data: unknown } + | { id?: string; error: unknown } + )[], + expected: number, + label: string, +): unknown[] { + if (items.length !== expected) throw new Error(`${label} bulk count mismatch`) + return items.map((item, i) => { + if (item.id !== String(i) || 'error' in item || !('data' in item)) + throw new Error(`${label} bulk item failed or reordered`) + return item.data + }) +} + +export async function createConversion( + sql: postgres.Sql, + table: string, + clients = createMigrationClients(), +) { + if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) + throw new Error('Invalid example table name') + const relation = `public.${table}` + const { legacy, stack, schema, columns } = await clients + async function checkDependencies( + tx: postgres.TransactionSql, + allowTargetIndexes: boolean, + ) { + // pg_get_indexdef qualifies functions only when they are not visible. + // Stabilize its rendering for our exact rollback-index check; SET LOCAL + // restores the caller's search path when this transaction ends. + await tx`SET LOCAL search_path = pg_catalog` + const dependencies = await tx`SELECT + NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = ${relation}::regclass AND contype = 'p' + AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[]) + OR EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${relation}::regclass AND NOT tgisinternal) + OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${relation}::regclass OR confrelid = ${relation}::regclass) + -- PostgreSQL 18 also lists the primary key column's NOT NULL here. + AND NOT (contype IN ('p', 'n') AND conrelid = ${relation}::regclass AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[])) + -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read + -- it from the column on every version. + OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND attname <> 'pk' AND NOT attisdropped AND attnotnull) + OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid + WHERE indrelid = ${relation}::regclass AND NOT indisprimary AND NOT ( + ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 + AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') + OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) + OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${relation}::regclass AND classid = 'pg_catalog.pg_rewrite'::regclass) + OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${relation}::regclass OR inhrelid = ${relation}::regclass) + AS unsupported` + if (dependencies[0].unsupported) + throw new Error( + 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', + ) } - // 5. Index the equality terms. - await tx`CREATE INDEX customers_email_v3_idx ON customers (eql_v3.eq_term(email_v3))` - await tx`CREATE INDEX customers_age_v3_idx ON customers (eql_v3.eq_term(age_v3))` - - // 6. Verify through Stack before committing; any throw rolls back the DDL too. - const converted = await tx`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, - notes_v3::jsonb AS notes FROM customers ORDER BY pk` - for (let i = 0; i < rows.length; i++) { - for (const column of columns) { - const before = rows[i][column] - const after = converted[i][column] - if (after.c !== before.c || (after.hm ?? null) !== (before.hm ?? null) || - ok(await client.decrypt(after), 'decrypt target') !== - ok(await client.decrypt(before), 'decrypt source')) - throw new Error(`Conversion verification failed in ${column}`) - } + async function migrate() { + await sql.begin(async (tx) => { + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` + // Fail before adding any columns. This example understands one exact shape. + const attributes = + await tx`SELECT attname, atttypid = CASE WHEN attname = 'pk' THEN 'pg_catalog.text'::regtype ELSE 'public.eql_v2_encrypted'::regtype END AS supported_type + FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND NOT attisdropped ORDER BY attnum` + if ( + JSON.stringify(attributes.map((a) => [a.attname, a.supported_type])) !== + JSON.stringify([ + ['pk', true], + ['email', true], + ['age', true], + ['notes', true], + ]) + ) + throw new Error( + 'Unsupported columns: expected pk and three EQL v2 columns', + ) + await checkDependencies(tx, false) + const rows = + await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(relation)} ORDER BY pk` + if (rows.length === 0) + throw new Error('Empty source: this example requires legacy records') + for (const row of rows) { + for (const column of columns) { + const payload = row[column] + if ( + payload?.v !== 2 || + payload.k !== 'ct' || + typeof payload.c !== 'string' || + payload.i?.t !== 'customers' || + payload.i?.c !== column || + Object.keys(payload).some( + (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), + ) || + // Protect represents unused index capabilities with null fields. + payload.ob != null || + payload.bf != null || + (column === 'notes' + ? payload.hm != null + : typeof payload.hm !== 'string') + ) { + throw new Error(`Unsupported legacy payload in ${column}`) + } + } + } + const payloads = rows.flatMap((row) => + columns.map((column) => row[column]), + ) + const sourcePlaintexts = verifiedBulk( + ok( + await legacy.bulkDecrypt( + payloads.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt source', + ), + payloads.length, + 'Source', + ) + for (const [i] of payloads.entries()) { + const column = columns[i % columns.length] + const plain = sourcePlaintexts[i] + if ( + column === 'age' + ? !Number.isSafeInteger(plain) + : typeof plain !== 'string' + ) + throw new Error(`Unsupported plaintext in ${column}`) + } + const terms = ok( + await stack.encryptQuery( + rows.flatMap((_row, i) => [ + { + value: String(sourcePlaintexts[i * columns.length]), + table: schema, + column: schema.email, + }, + { + value: Number(sourcePlaintexts[i * columns.length + 1]), + table: schema, + column: schema.age, + }, + ]), + ), + 'verify equality mappings', + ) + if (terms.length !== rows.length * 2) + throw new Error('Equality term count mismatch') + for (const [i, term] of terms.entries()) { + const column = i % 2 === 0 ? 'email' : 'age' + if ( + typeof term !== 'object' || + term === null || + !('hm' in term) || + term.hm !== rows[Math.floor(i / 2)][column].hm + ) + throw new Error(`Unsupported equality mapping in ${column}`) + } + await tx`ALTER TABLE ${tx(relation)} + ADD COLUMN email_v3 public.eql_v3_text_eq, + ADD COLUMN age_v3 public.eql_v3_integer_eq, + ADD COLUMN notes_v3 public.eql_v3_text` + // Rebuild only the envelope. Source columns and their bytes stay intact. + for (const column of columns) { + await tx`UPDATE ${tx(relation)} SET ${tx(`${column}_v3`)} = jsonb_build_object( + 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' + ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb + ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` + } + await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(email_v3))` + await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(age_v3))` + // Read through Stack before committing, so a wrong mapping rolls back DDL too. + const converted = + await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)} ORDER BY pk` + if ( + converted.length !== rows.length || + converted.some((row, i) => row.pk !== rows[i].pk) + ) + throw new Error('Conversion verification failed: row identity mismatch') + const targets = converted.flatMap((row) => + columns.map((column) => row[`${column}_v3`]), + ) + const targetPlaintexts = verifiedBulk( + ok( + await stack.bulkDecrypt( + targets.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt target', + ), + targets.length, + 'Target', + ) + for (const [i, after] of targets.entries()) { + const before = payloads[i] + if ( + after.c !== before.c || + (after.hm ?? null) !== (before.hm ?? null) || + targetPlaintexts[i] !== sourcePlaintexts[i] + ) + throw new Error( + `Conversion verification failed in ${columns[i % columns.length]}`, + ) + } + }) + await sql`ANALYZE ${sql(relation)}` } -}) -await sql`ANALYZE customers` -await sql.end() + + return { migrate, checkDependencies } +} + +// Importing this module supplies helpers; executing it runs the conversion. +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + const databaseUrl = process.env.DATABASE_URL + if (!databaseUrl) throw new Error('DATABASE_URL is required') + const sql = postgres(databaseUrl, { prepare: false }) + try { + const conversion = await createConversion( + sql, + process.env.MIGRATION_TABLE ?? 'customers', + ) + await conversion.migrate() + } finally { + await sql.end() + } +} ``` + + **Rerunning the conversion:** after a successful commit, the table includes the three `_v3` columns. Running this conversion again fails the exact source-shape check before any DDL or data changes; the transaction rolls back without altering the completed migration. It is a one-off conversion, not an idempotent startup hook. Do not treat the mere presence of target columns as proof that a migration completed, and do not drop them to force a rerun. If a transaction failed, diagnose the failure and confirm its changes rolled back before deciding whether to retry. If the client lost the connection around commit, inspect and verify the database state first. Continue with verification and cutover after a successful migration. Only the envelope and database structure change for these proven mappings; the ciphertext and equality terms are copied unchanged. Do not generalize this to other capabilities or to v4 (`postgres-upgrade › migrates actual v2 columns, preserves ciphertext and queries, and rolls back before writes resume`). -The example reads the small source table into memory and validates each value. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. +The example reads the small source table into memory. It bulk-decrypts source values once with Protect, retains those plaintexts for comparison, batches equality-term generation, and bulk-decrypts converted values once with Stack. Every bulk result is checked for item failures, count and order. These reduce repeated SDK work; they are not a measured network-call or lock-duration guarantee. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. ### 4. Verify queries, indexes and cutover @@ -452,7 +667,7 @@ In rehearsal, test fresh Stack writes beside migrated rows, reads, lookups and u ### Reproduce the PostgreSQL example -Use the [npm example instructions](https://github.com/cipherstash/stack/blob/main/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. +Use the [npm example instructions](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. ## Optional: moving values between DynamoDB and Postgres diff --git a/e2e/migration-guide/README.md b/e2e/migration-guide/README.md index 771f7b6dc..b2fbc985d 100644 --- a/e2e/migration-guide/README.md +++ b/e2e/migration-guide/README.md @@ -1,6 +1,6 @@ # Protect v2 to Stack v3 examples -These examples accompany the [two-store migration guide](../../docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. +These examples accompany the [two-store migration guide](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. Use Node.js 22.18 or later. Copy this directory to a disposable location outside the monorepo and run: @@ -56,3 +56,5 @@ The final command **must fail** because source columns do not contain the new St The example deliberately supports only an uncustomized table with a text primary key, text equality, safe whole-number equality and storage-only text under the fixed `customers` logical identity. It refuses extra columns, secondary indexes, non-primary constraints, dependent views, user triggers, row-level security and inheritance. Review those objects rather than removing them to bypass a refusal. Run the existing credentialed `test:protect-crossover` task from the repository to verify these functions against real Postgres and ZeroKMS. It also tests unsupported-input refusal, fresh writes and actual equality-index engagement. See the guide for the maintainer build commands. + +`postgres-conversion.ts` is the self-contained conversion used by the example and reproduced verbatim in the guide. To update the checked-in snippet, run `node scripts/sync-migration-guide.mjs` from the repository root; `--check` verifies it without writing. Keep Protect installed in this separate migration tool until verification and the rollback decision are complete. The new application can remove Protect independently. diff --git a/e2e/migration-guide/legacy.ts b/e2e/migration-guide/legacy.ts index fbc212104..75b9cda4f 100644 --- a/e2e/migration-guide/legacy.ts +++ b/e2e/migration-guide/legacy.ts @@ -1,42 +1 @@ -import { protect } from '@cipherstash/protect' - -/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ -export async function legacyProtect( - schemas: Parameters[0]['schemas'], -) { - const crn = process.env.CS_WORKSPACE_CRN - if (!crn) throw new Error('CS_WORKSPACE_CRN is required') - const region = crn.split(':')[1] - const saved = { - CS_CTS_HOST: process.env.CS_CTS_HOST, - CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, - } - process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` - process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` - try { - return await protect({ - schemas, - workspaceCrn: crn, - clientId: process.env.CS_CLIENT_ID, - clientKey: process.env.CS_CLIENT_KEY, - accessKey: process.env.CS_CLIENT_ACCESS_KEY, - }) - } finally { - for (const [key, value] of Object.entries(saved)) { - if (value === undefined) delete process.env[key] - else process.env[key] = value - } - } -} - -/** Check every SDK result; never continue with a failed encryption operation. */ -export function ok( - result: { data: T; failure?: undefined } | { failure: unknown }, - label: string, -): T { - if ('failure' in result && result.failure !== undefined) { - throw new Error(`${label} failed`, { cause: result.failure }) - } - if ('data' in result) return result.data - throw new Error(`${label} returned no data`) -} +export { legacyProtect, ok } from './postgres-conversion.ts' diff --git a/e2e/migration-guide/postgres-conversion.ts b/e2e/migration-guide/postgres-conversion.ts new file mode 100644 index 000000000..a04bbe879 --- /dev/null +++ b/e2e/migration-guide/postgres-conversion.ts @@ -0,0 +1,294 @@ +import { pathToFileURL } from 'node:url' +import { csColumn, csTable, protect } from '@cipherstash/protect' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +import postgres from 'postgres' + +/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ +export async function legacyProtect( + schemas: Parameters[0]['schemas'], +) { + const crn = process.env.CS_WORKSPACE_CRN + if (!crn) throw new Error('CS_WORKSPACE_CRN is required') + const region = crn.split(':')[1] + const saved = { + CS_CTS_HOST: process.env.CS_CTS_HOST, + CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, + } + process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` + process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` + try { + return await protect({ + schemas, + workspaceCrn: crn, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, + }) + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key] + else process.env[key] = value + } + } +} + +/** Check every SDK result; never continue with a failed encryption operation. */ +export function ok( + result: { data: T; failure?: undefined } | { failure: unknown }, + label: string, +): T { + if ('failure' in result && result.failure !== undefined) { + throw new Error(`${label} failed`, { cause: result.failure }) + } + if ('data' in result) return result.data + throw new Error(`${label} returned no data`) +} + +export async function createMigrationClients() { + const legacySchema = csTable('customers', { + email: csColumn('email').equality(), + age: csColumn('age').dataType('number').equality(), + notes: csColumn('notes'), + }) + const schema = encryptedTable('customers', { + email: types.TextEq('email'), + age: types.IntegerEq('age'), + notes: types.Text('notes'), + }) + const legacy = await legacyProtect([legacySchema]) + const stack = await Encryption({ schemas: [schema] }) + const columns = ['email', 'age', 'notes'] as const + return { legacySchema, schema, legacy, stack, columns } +} + +/** Bulk success is not enough: reject missing, reordered or failed items. */ +function verifiedBulk( + items: readonly ( + | { id?: string; data: unknown } + | { id?: string; error: unknown } + )[], + expected: number, + label: string, +): unknown[] { + if (items.length !== expected) throw new Error(`${label} bulk count mismatch`) + return items.map((item, i) => { + if (item.id !== String(i) || 'error' in item || !('data' in item)) + throw new Error(`${label} bulk item failed or reordered`) + return item.data + }) +} + +export async function createConversion( + sql: postgres.Sql, + table: string, + clients = createMigrationClients(), +) { + if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) + throw new Error('Invalid example table name') + const relation = `public.${table}` + const { legacy, stack, schema, columns } = await clients + async function checkDependencies( + tx: postgres.TransactionSql, + allowTargetIndexes: boolean, + ) { + // pg_get_indexdef qualifies functions only when they are not visible. + // Stabilize its rendering for our exact rollback-index check; SET LOCAL + // restores the caller's search path when this transaction ends. + await tx`SET LOCAL search_path = pg_catalog` + const dependencies = await tx`SELECT + NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = ${relation}::regclass AND contype = 'p' + AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[]) + OR EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${relation}::regclass AND NOT tgisinternal) + OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${relation}::regclass OR confrelid = ${relation}::regclass) + -- PostgreSQL 18 also lists the primary key column's NOT NULL here. + AND NOT (contype IN ('p', 'n') AND conrelid = ${relation}::regclass AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[])) + -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read + -- it from the column on every version. + OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND attname <> 'pk' AND NOT attisdropped AND attnotnull) + OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid + WHERE indrelid = ${relation}::regclass AND NOT indisprimary AND NOT ( + ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 + AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') + OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) + OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${relation}::regclass AND classid = 'pg_catalog.pg_rewrite'::regclass) + OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${relation}::regclass OR inhrelid = ${relation}::regclass) + AS unsupported` + if (dependencies[0].unsupported) + throw new Error( + 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', + ) + } + + async function migrate() { + await sql.begin(async (tx) => { + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` + // Fail before adding any columns. This example understands one exact shape. + const attributes = + await tx`SELECT attname, atttypid = CASE WHEN attname = 'pk' THEN 'pg_catalog.text'::regtype ELSE 'public.eql_v2_encrypted'::regtype END AS supported_type + FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND NOT attisdropped ORDER BY attnum` + if ( + JSON.stringify(attributes.map((a) => [a.attname, a.supported_type])) !== + JSON.stringify([ + ['pk', true], + ['email', true], + ['age', true], + ['notes', true], + ]) + ) + throw new Error( + 'Unsupported columns: expected pk and three EQL v2 columns', + ) + await checkDependencies(tx, false) + const rows = + await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(relation)} ORDER BY pk` + if (rows.length === 0) + throw new Error('Empty source: this example requires legacy records') + for (const row of rows) { + for (const column of columns) { + const payload = row[column] + if ( + payload?.v !== 2 || + payload.k !== 'ct' || + typeof payload.c !== 'string' || + payload.i?.t !== 'customers' || + payload.i?.c !== column || + Object.keys(payload).some( + (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), + ) || + // Protect represents unused index capabilities with null fields. + payload.ob != null || + payload.bf != null || + (column === 'notes' + ? payload.hm != null + : typeof payload.hm !== 'string') + ) { + throw new Error(`Unsupported legacy payload in ${column}`) + } + } + } + const payloads = rows.flatMap((row) => + columns.map((column) => row[column]), + ) + const sourcePlaintexts = verifiedBulk( + ok( + await legacy.bulkDecrypt( + payloads.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt source', + ), + payloads.length, + 'Source', + ) + for (const [i] of payloads.entries()) { + const column = columns[i % columns.length] + const plain = sourcePlaintexts[i] + if ( + column === 'age' + ? !Number.isSafeInteger(plain) + : typeof plain !== 'string' + ) + throw new Error(`Unsupported plaintext in ${column}`) + } + const terms = ok( + await stack.encryptQuery( + rows.flatMap((_row, i) => [ + { + value: String(sourcePlaintexts[i * columns.length]), + table: schema, + column: schema.email, + }, + { + value: Number(sourcePlaintexts[i * columns.length + 1]), + table: schema, + column: schema.age, + }, + ]), + ), + 'verify equality mappings', + ) + if (terms.length !== rows.length * 2) + throw new Error('Equality term count mismatch') + for (const [i, term] of terms.entries()) { + const column = i % 2 === 0 ? 'email' : 'age' + if ( + typeof term !== 'object' || + term === null || + !('hm' in term) || + term.hm !== rows[Math.floor(i / 2)][column].hm + ) + throw new Error(`Unsupported equality mapping in ${column}`) + } + await tx`ALTER TABLE ${tx(relation)} + ADD COLUMN email_v3 public.eql_v3_text_eq, + ADD COLUMN age_v3 public.eql_v3_integer_eq, + ADD COLUMN notes_v3 public.eql_v3_text` + // Rebuild only the envelope. Source columns and their bytes stay intact. + for (const column of columns) { + await tx`UPDATE ${tx(relation)} SET ${tx(`${column}_v3`)} = jsonb_build_object( + 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' + ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb + ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` + } + await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(email_v3))` + await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(age_v3))` + // Read through Stack before committing, so a wrong mapping rolls back DDL too. + const converted = + await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)} ORDER BY pk` + if ( + converted.length !== rows.length || + converted.some((row, i) => row.pk !== rows[i].pk) + ) + throw new Error('Conversion verification failed: row identity mismatch') + const targets = converted.flatMap((row) => + columns.map((column) => row[`${column}_v3`]), + ) + const targetPlaintexts = verifiedBulk( + ok( + await stack.bulkDecrypt( + targets.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt target', + ), + targets.length, + 'Target', + ) + for (const [i, after] of targets.entries()) { + const before = payloads[i] + if ( + after.c !== before.c || + (after.hm ?? null) !== (before.hm ?? null) || + targetPlaintexts[i] !== sourcePlaintexts[i] + ) + throw new Error( + `Conversion verification failed in ${columns[i % columns.length]}`, + ) + } + }) + await sql`ANALYZE ${sql(relation)}` + } + + return { migrate, checkDependencies } +} + +// Importing this module supplies helpers; executing it runs the conversion. +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + const databaseUrl = process.env.DATABASE_URL + if (!databaseUrl) throw new Error('DATABASE_URL is required') + const sql = postgres(databaseUrl, { prepare: false }) + try { + const conversion = await createConversion( + sql, + process.env.MIGRATION_TABLE ?? 'customers', + ) + await conversion.migrate() + } finally { + await sql.end() + } +} diff --git a/e2e/migration-guide/postgres.ts b/e2e/migration-guide/postgres.ts index 280429d25..3b6d82060 100644 --- a/e2e/migration-guide/postgres.ts +++ b/e2e/migration-guide/postgres.ts @@ -1,25 +1,22 @@ -import { csColumn, csTable } from '@cipherstash/protect' -import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' import type postgres from 'postgres' -import { legacyProtect, ok } from './legacy.ts' +import { + createConversion, + createMigrationClients, + ok, +} from './postgres-conversion.ts' /** A deliberately small migration example, not a general backfill service. */ export async function createExample(sql: postgres.Sql, table: string) { if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) throw new Error('Invalid example table name') - const legacySchema = csTable('customers', { - email: csColumn('email').equality(), - age: csColumn('age').dataType('number').equality(), - notes: csColumn('notes'), - }) - const schema = encryptedTable('customers', { - email: types.TextEq('email'), - age: types.IntegerEq('age'), - notes: types.Text('notes'), - }) - const legacy = await legacyProtect([legacySchema]) - const stack = await Encryption({ schemas: [schema] }) - const columns = ['email', 'age', 'notes'] as const + const clients = createMigrationClients() + const { legacySchema, schema, legacy, stack, columns } = await clients + const { migrate, checkDependencies } = await createConversion( + sql, + table, + clients, + ) + const relation = `public.${table}` const corpus = [ { pk: 'ada', email: 'ada@example.com', age: 36, notes: 'engines' }, { pk: 'alan', email: 'alan@example.com', age: 36, notes: 'machines' }, @@ -29,7 +26,7 @@ export async function createExample(sql: postgres.Sql, table: string) { async function seed() { // Never replace an existing table or seed a customer's database in place. await sql.begin(async (tx) => { - await tx`CREATE TABLE ${tx(table)} ( + await tx`CREATE TABLE ${tx(relation)} ( pk text PRIMARY KEY, email public.eql_v2_encrypted, age public.eql_v2_encrypted, @@ -57,7 +54,7 @@ export async function createExample(sql: postgres.Sql, table: string) { }), 'legacy notes', ) - await tx`INSERT INTO ${tx(table)} VALUES (${row.pk}, ${tx.json(email)}, ${tx.json(age)}, ${tx.json(notes)})` + await tx`INSERT INTO ${tx(relation)} VALUES (${row.pk}, ${tx.json(email)}, ${tx.json(age)}, ${tx.json(notes)})` } }) } @@ -67,8 +64,8 @@ export async function createExample(sql: postgres.Sql, table: string) { // base jsonb type explicitly instead of passing serialized text to the SDK. const rows = generation === 'v2' - ? await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(table)} ORDER BY pk` - : await sql`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, notes_v3::jsonb AS notes FROM ${sql(table)} ORDER BY pk` + ? await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(relation)} ORDER BY pk` + : await sql`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, notes_v3::jsonb AS notes FROM ${sql(relation)} ORDER BY pk` return Promise.all( rows.map(async (row) => { const values = await Promise.all( @@ -131,161 +128,13 @@ export async function createExample(sql: postgres.Sql, table: string) { const jsonTerm: postgres.JSONValue = JSON.parse(JSON.stringify(term)) const rows = generation === 'v2' - ? await sql`SELECT pk FROM ${sql(table)} WHERE ${sql(column)} = ${sql.json(jsonTerm)}::jsonb ORDER BY pk` + ? await sql`SELECT pk FROM ${sql(relation)} WHERE ${sql(column)} = ${sql.json(jsonTerm)}::jsonb ORDER BY pk` : column === 'email' - ? await sql`SELECT pk FROM ${sql(table)} WHERE eql_v3.eq(email_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_text_eq) ORDER BY pk` - : await sql`SELECT pk FROM ${sql(table)} WHERE eql_v3.eq(age_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_integer_eq) ORDER BY pk` + ? await sql`SELECT pk FROM ${sql(relation)} WHERE eql_v3.eq(email_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_text_eq) ORDER BY pk` + : await sql`SELECT pk FROM ${sql(relation)} WHERE eql_v3.eq(age_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_integer_eq) ORDER BY pk` return rows.map((row) => row.pk) } - async function checkDependencies( - tx: postgres.TransactionSql, - allowTargetIndexes: boolean, - ) { - const dependencies = await tx`SELECT - EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${table}::regclass AND NOT tgisinternal) - OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${table}::regclass OR confrelid = ${table}::regclass) - -- PostgreSQL 18 also lists the primary key column's NOT NULL here. - AND NOT (contype IN ('p', 'n') AND conrelid = ${table}::regclass AND conkey = ARRAY[1]::smallint[])) - -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read - -- it from the column on every version. - OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${table}::regclass - AND attnum > 1 AND NOT attisdropped AND attnotnull) - OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid - WHERE indrelid = ${table}::regclass AND NOT indisprimary AND NOT ( - ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 - AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') - OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) - OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${table}::regclass AND classid = 'pg_rewrite'::regclass) - OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${table}::regclass) - OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${table}::regclass) - OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${table}::regclass OR inhrelid = ${table}::regclass) - AS unsupported` - if (dependencies[0].unsupported) - throw new Error( - 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', - ) - } - - async function migrate() { - await sql.begin(async (tx) => { - await tx`LOCK TABLE ${tx(table)} IN ACCESS EXCLUSIVE MODE` - // Fail before adding any columns. This example understands one exact shape. - const attributes = - await tx`SELECT attname, atttypid::regtype::text AS type - FROM pg_attribute WHERE attrelid = ${table}::regclass - AND attnum > 0 AND NOT attisdropped ORDER BY attnum` - if ( - JSON.stringify(attributes.map((a) => [a.attname, a.type])) !== - JSON.stringify([ - ['pk', 'text'], - ['email', 'eql_v2_encrypted'], - ['age', 'eql_v2_encrypted'], - ['notes', 'eql_v2_encrypted'], - ]) - ) - throw new Error( - 'Unsupported columns: expected pk and three EQL v2 columns', - ) - await checkDependencies(tx, false) - const rows = - await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(table)} ORDER BY pk` - if (rows.length === 0) - throw new Error('Empty source: this example requires legacy records') - for (const row of rows) { - for (const column of columns) { - const payload = row[column] - if ( - payload?.v !== 2 || - payload.k !== 'ct' || - typeof payload.c !== 'string' || - payload.i?.t !== 'customers' || - payload.i?.c !== column || - Object.keys(payload).some( - (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), - ) || - // Protect represents unused index capabilities with null fields. - payload.ob != null || - payload.bf != null || - (column === 'notes' - ? payload.hm != null - : typeof payload.hm !== 'string') - ) { - throw new Error(`Unsupported legacy payload in ${column}`) - } - const plain = ok( - await legacy.decrypt(payload), - 'validate legacy payload', - ) - if ( - column === 'age' - ? !Number.isSafeInteger(plain) - : typeof plain !== 'string' - ) { - throw new Error(`Unsupported plaintext in ${column}`) - } - if (column !== 'notes') { - const term = - column === 'email' - ? ok( - await stack.encryptQuery(String(plain), { - table: schema, - column: schema.email, - }), - 'verify email mapping', - ) - : ok( - await stack.encryptQuery(Number(plain), { - table: schema, - column: schema.age, - }), - 'verify age mapping', - ) - if ( - typeof term !== 'object' || - term === null || - !('hm' in term) || - term.hm !== payload.hm - ) { - throw new Error(`Unsupported equality mapping in ${column}`) - } - } - } - } - await tx`ALTER TABLE ${tx(table)} - ADD COLUMN email_v3 public.eql_v3_text_eq, - ADD COLUMN age_v3 public.eql_v3_integer_eq, - ADD COLUMN notes_v3 public.eql_v3_text` - // Rebuild only the envelope. Source columns and their bytes stay intact. - for (const column of columns) { - await tx`UPDATE ${tx(table)} SET ${tx(`${column}_v3`)} = jsonb_build_object( - 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' - ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb - ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` - } - await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(table)} (eql_v3.eq_term(email_v3))` - await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(table)} (eql_v3.eq_term(age_v3))` - // Read through Stack before committing, so a wrong mapping rolls back DDL too. - const converted = - await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(table)} ORDER BY pk` - for (let i = 0; i < rows.length; i++) { - for (const column of columns) { - const before = rows[i][column] - const after = converted[i][`${column}_v3`] - if ( - after.c !== before.c || - (after.hm ?? null) !== (before.hm ?? null) || - ok(await stack.decrypt(after), 'verify target') !== - ok(await legacy.decrypt(before), 'verify source') - ) { - throw new Error(`Conversion verification failed in ${column}`) - } - } - } - }) - await sql`ANALYZE ${sql(table)}` - } - async function write() { const model = { pk: 'new', @@ -294,18 +143,18 @@ export async function createExample(sql: postgres.Sql, table: string) { notes: 'written by Stack', } const encrypted = ok(await stack.encryptModel(model, schema), 'fresh write') - await sql`INSERT INTO ${sql(table)} (pk, email_v3, age_v3, notes_v3) + await sql`INSERT INTO ${sql(relation)} (pk, email_v3, age_v3, notes_v3) VALUES (${model.pk}, ${sql.json(encrypted.email)}, ${sql.json(encrypted.age)}, ${sql.json(encrypted.notes)})` return model } async function rollback() { await sql.begin(async (tx) => { - await tx`LOCK TABLE ${tx(table)} IN ACCESS EXCLUSIVE MODE` + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` await checkDependencies(tx, true) const rows = await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes, - email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(table)}` + email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)}` for (const row of rows) { for (const column of columns) { if ( @@ -322,7 +171,7 @@ export async function createExample(sql: postgres.Sql, table: string) { } // Preflight permits only our two derived indexes. PostgreSQL otherwise // drops same-table constraints/indexes with a column even without CASCADE. - await tx`ALTER TABLE ${tx(table)} DROP COLUMN email_v3, DROP COLUMN age_v3, DROP COLUMN notes_v3` + await tx`ALTER TABLE ${tx(relation)} DROP COLUMN email_v3, DROP COLUMN age_v3, DROP COLUMN notes_v3` }) } diff --git a/e2e/tests/protect-guide-script.e2e.test.ts b/e2e/tests/protect-guide-script.e2e.test.ts new file mode 100644 index 000000000..c08321862 --- /dev/null +++ b/e2e/tests/protect-guide-script.e2e.test.ts @@ -0,0 +1,332 @@ +import { execFileSync } from 'node:child_process' +import { readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { runInNewContext } from 'node:vm' +import ts from 'typescript' +import { describe, expect, it, vi } from 'vitest' + +const guide = readFileSync( + new URL('../../docs/migration/protect-dynamodb-to-stack.md', import.meta.url), + 'utf8', +) +const fences = [...guide.matchAll(/```typescript\n([\s\S]*?)\n```/g)].map( + (match) => match[1], +) +const conversion = + fences.find((code) => code.includes('ACCESS EXCLUSIVE')) ?? '' +const credentials = fences.find((code) => code.includes('workspaceCrn:')) ?? '' +const setup = `import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +const customers = encryptedTable('customers', {email: types.TextEq('email'), age: types.IntegerEq('age'), notes: types.Text('notes')}) +const users = customers +const client = await Encryption({schemas: [customers]}) +` + +// Compile the actual customer paste, in the same module-resolution context as +// the example. Diagnostics are assertion data, never a build-error surrogate. +function diagnostics(code: string) { + const filename = fileURLToPath( + new URL('../migration-guide/pasted-guide.ts', import.meta.url), + ) + const options: ts.CompilerOptions = { + strict: true, + noEmit: true, + skipLibCheck: true, + target: ts.ScriptTarget.ESNext, + module: ts.ModuleKind.ESNext, + moduleResolution: ts.ModuleResolutionKind.Bundler, + allowImportingTsExtensions: true, + } + const host = ts.createCompilerHost(options) + const original = host.getSourceFile.bind(host) + host.getSourceFile = ( + path, + languageVersion, + onError, + shouldCreateNewSourceFile, + ) => + path === filename + ? ts.createSourceFile(path, code, languageVersion, true) + : original(path, languageVersion, onError, shouldCreateNewSourceFile) + const program = ts.createProgram([filename], options, host) + return ts + .getPreEmitDiagnostics(program) + .map((diagnostic) => + ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n'), + ) +} + +type Payload = { + v: number + k?: string + i: { t: string; c: string } + c: string + hm?: string +} +const columns = ['email', 'age', 'notes'] as const +const originalPlain = { email: 'ada@example.com', age: 36, notes: 'engines' } + +type Fault = + | 'none' + | 'wrong-plaintext' + | 'source-item' + | 'target-item' + | 'source-count' + | 'target-count' + | 'source-order' + | 'target-order' + | 'target-row' + | 'term-count' + | 'term-mismatch' + +// Execute the actual pasted implementation with only SQL/SDK boundaries faked. +// The SQL fake supplies fixed results, not a second parser or migration engine. +async function executeConversion(fault: Fault = 'none', example = false) { + const row: Record = { pk: 'ada' } + for (const column of columns) + row[column] = { + v: 2, + k: 'ct', + i: { t: 'customers', c: column }, + c: column, + ...(column === 'notes' ? {} : { hm: column }), + } + const target: Record = { + pk: fault === 'target-row' ? 'wrong-row' : 'ada', + } + for (const column of columns) + target[`${column}_v3`] = { ...(row[column] as Payload), v: 3 } + const plain = (payload: Payload) => + originalPlain[payload.i.c as keyof typeof originalPlain] + type Input = { id: string; data: Payload } + type Output = + | { id: string; data: string | number } + | { id: string; error: string } + function bulk(inputs: Input[], generation: 'source' | 'target') { + const results: Output[] = inputs.map(({ id, data }) => ({ + id, + data: + generation === 'target' && + fault === 'wrong-plaintext' && + data.i.c === 'notes' + ? 'wrong-but-consistent' + : plain(data), + })) + if (fault === `${generation}-item`) + results[1] = { id: '1', error: 'synthetic failure' } + if (fault === `${generation}-count`) results.pop() + if (fault === `${generation}-order`) results.reverse() + return { data: results } + } + const legacy = { + decrypt: vi.fn(), + bulkDecrypt: vi.fn(async (inputs: Input[]) => bulk(inputs, 'source')), + } + const stack = { + decrypt: vi.fn(), + bulkDecrypt: vi.fn(async (inputs: Input[]) => bulk(inputs, 'target')), + encryptQuery: vi.fn(async (terms: { column: string }[]) => { + const results = terms.map(({ column }) => ({ + hm: fault === 'term-mismatch' ? 'wrong' : column, + })) + if (fault === 'term-count') results.pop() + return { data: results } + }), + } + const statements: string[] = [] + const sql = Object.assign( + (parts: TemplateStringsArray | string) => { + if (typeof parts === 'string') return parts + const query = parts.join('?') + statements.push(query) + if (query.includes('AS unsupported')) return [{ unsupported: false }] + if (query.includes('SELECT attname')) + return ['pk', ...columns].map((attname) => ({ + attname, + supported_type: true, + })) + if (query.includes('SELECT pk')) + return [query.includes('email_v3::jsonb') ? target : row] + return [] + }, + { + begin: async (fn: (tx: unknown) => Promise) => { + try { + await fn(sql) + statements.push('COMMIT') + } catch (error) { + statements.push('ROLLBACK') + throw error + } + }, + end: async () => {}, + }, + ) + function executable(source: string) { + const parsed = ts.createSourceFile( + 'snippet.ts', + source, + ts.ScriptTarget.ESNext, + true, + ) + const withoutImports = parsed.statements + .filter( + (statement) => + !ts.isImportDeclaration(statement) && + !ts.isExportDeclaration(statement), + ) + .map((statement) => statement.getFullText(parsed)) + .join('\n') + return ts + .transpileModule(withoutImports, { + compilerOptions: { + target: ts.ScriptTarget.ESNext, + module: ts.ModuleKind.ESNext, + }, + }) + .outputText.replace(/\bexport /g, '') + .replace(/import.meta.url/g, "'file:///imported.ts'") + } + const exampleSource = readFileSync( + new URL('../migration-guide/postgres.ts', import.meta.url), + 'utf8', + ) + let failure: unknown + try { + await runInNewContext( + `(async () => {${executable(conversion)}; ${example ? executable(exampleSource) : ''}; await (await ${example ? 'createExample' : 'createConversion'}(sql, 'customers')).migrate()})()`, + { + sql, + csColumn: () => ({ + equality() { + return this + }, + dataType() { + return this + }, + }), + csTable: (_name: string, fields: object) => fields, + encryptedTable: (_name: string, fields: object) => fields, + types: { + TextEq: () => 'email', + IntegerEq: () => 'age', + Text: () => 'notes', + }, + protect: async () => legacy, + Encryption: async () => stack, + process: { + argv: [], + env: { CS_WORKSPACE_CRN: 'test:ap-southeast-2:test' }, + }, + }, + ) + } catch (error) { + failure = error + } + return { failure, legacy, client: stack, statements } +} + +describe('pasted migration guide', () => { + it('G1: the conversion compiles with strict TypeScript', () => { + expect(diagnostics(conversion)).toEqual([]) + }) + + it('G2: refuses a Stack interpretation that disagrees with Protect', async () => { + const result = await executeConversion('wrong-plaintext') + expect( + result.failure, + 'independent source validation must reject a consistently wrong Stack plaintext', + ).toMatchObject({ message: 'Conversion verification failed in notes' }) + expect(result.statements).toContain('ROLLBACK') + expect(result.statements.some((query) => query.includes('ANALYZE'))).toBe( + false, + ) + }) + + it('G5: retains source plaintext and decrypts converted values in bulk', async () => { + const result = await executeConversion() + expect(result.failure).toBeUndefined() + expect(result.legacy.decrypt).not.toHaveBeenCalled() + expect(result.legacy.bulkDecrypt).toHaveBeenCalledTimes(1) + expect(result.client.encryptQuery).toHaveBeenCalledTimes(1) + expect(result.statements).toContain('COMMIT') + expect(result.client.decrypt).not.toHaveBeenCalled() + expect(result.client.bulkDecrypt).toHaveBeenCalledTimes(1) + }) + + it('G5: the executable example does not decrypt each source twice under the lock', async () => { + const result = await executeConversion('none', true) + expect(result.failure).toBeUndefined() + expect(result.legacy.decrypt).not.toHaveBeenCalled() + expect(result.legacy.bulkDecrypt).toHaveBeenCalledTimes(1) + expect(result.client.encryptQuery).toHaveBeenCalledTimes(1) + expect(result.statements).toContain('COMMIT') + expect(result.client.decrypt).not.toHaveBeenCalled() + expect(result.client.bulkDecrypt).toHaveBeenCalledTimes(1) + }) + + it.each([ + ['source-item', 'Source bulk item failed or reordered'], + ['target-item', 'Target bulk item failed or reordered'], + ['source-count', 'Source bulk count mismatch'], + ['target-count', 'Target bulk count mismatch'], + ['source-order', 'Source bulk item failed or reordered'], + ['target-order', 'Target bulk item failed or reordered'], + ['target-row', 'Conversion verification failed: row identity mismatch'], + ['term-count', 'Equality term count mismatch'], + ['term-mismatch', 'Unsupported equality mapping in email'], + ] as const)('rejects %s before commit', async (fault, message) => { + const result = await executeConversion(fault) + expect(result.failure).toMatchObject({ message }) + expect(result.statements).toContain('ROLLBACK') + expect(result.statements).not.toContain('COMMIT') + expect(result.statements.some((query) => query.includes('ANALYZE'))).toBe( + false, + ) + }) + + it('G3: reproduces the complete canonical module without drift', () => { + const canonical = readFileSync( + new URL('../migration-guide/postgres-conversion.ts', import.meta.url), + 'utf8', + ) + expect(conversion).toBe(canonical.trimEnd()) + expect(() => + execFileSync(process.execPath, [ + fileURLToPath( + new URL('../../scripts/sync-migration-guide.mjs', import.meta.url), + ), + '--check', + ]), + ).not.toThrow() + }) + + it('G6: links the portable example and guide at an immutable revision', () => { + const readme = readFileSync( + new URL('../migration-guide/README.md', import.meta.url), + 'utf8', + ) + const links = [...`${guide}\n${readme}`.matchAll(/\]\(([^)]+)\)/g)] + .map((match) => match[1]) + .filter( + (url) => + url.includes('migration-guide/') || + url.includes('protect-dynamodb-to-stack.md'), + ) + expect(links.length).toBeGreaterThan(0) + for (const link of links) + expect(link).toMatch( + /^https:\/\/github\.com\/cipherstash\/stack\/blob\/[0-9a-f]{40}\//, + ) + }) + + it('G7: the explicit credentials example compiles without an undeclared env object', () => { + expect( + diagnostics( + setup.replace( + 'const client = await Encryption({schemas: [customers]})', + '', + ) + credentials, + ), + ).toEqual([]) + }) +}) diff --git a/e2e/tests/protect-postgres-upgrade.e2e.test.ts b/e2e/tests/protect-postgres-upgrade.e2e.test.ts index 11f9a1fbc..79381f5b8 100644 --- a/e2e/tests/protect-postgres-upgrade.e2e.test.ts +++ b/e2e/tests/protect-postgres-upgrade.e2e.test.ts @@ -1,9 +1,9 @@ -import { csColumn, csTable } from '@cipherstash/protect' -import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +import { spawn } from 'node:child_process' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' import { installEqlV3 } from '@cipherstash/test-kit/install' import postgres from 'postgres' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' -import { legacyProtect, ok } from '../migration-guide/legacy.ts' import { createExample } from '../migration-guide/postgres.ts' import { authConfigured } from './helpers/legacy-protect' @@ -77,35 +77,50 @@ describe('Protect Postgres v2 to Stack v3', () => { ).toEqual(before) }, 120_000) - // Evidence for decrypting the source without Protect during conversion. The - // envelopes are the real Protect 10.2.1 writes read back from the - // eql_v2_encrypted columns, not fixtures minted by Stack's own protect-ffi. - it('decrypts every Protect-written v2 source envelope with Stack to the same plaintext as Protect', async () => { - const legacySchema = csTable('customers', { - email: csColumn('email').equality(), - age: csColumn('age').dataType('number').equality(), - notes: csColumn('notes'), - }) - const schema = encryptedTable('customers', { - email: types.TextEq('email'), - age: types.IntegerEq('age'), - notes: types.Text('notes'), - }) - const legacy = await legacyProtect([legacySchema]) - const stack = await Encryption({ schemas: [schema] }) - const rows = - await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(table)} ORDER BY pk` - expect(rows.map((row) => row.pk)).toEqual(app.corpus.map((row) => row.pk)) - for (const [i, row] of rows.entries()) { - for (const column of ['email', 'age', 'notes'] as const) { - const payload = row[column] - expect(payload.v).toBe(2) - expect(payload.k).toBe('ct') - const fromProtect = ok(await legacy.decrypt(payload), 'protect decrypt') - const fromStack = ok(await stack.decrypt(payload), 'stack decrypt') - expect(fromStack).toStrictEqual(fromProtect) - expect(fromStack).toStrictEqual(app.corpus[i][column]) - } + it('runs the standalone guide script against Protect-written source columns', async () => { + const guide = await readFile( + new URL( + '../../docs/migration/protect-dynamodb-to-stack.md', + import.meta.url, + ), + 'utf8', + ) + const snippet = [...guide.matchAll(/```typescript\n([\s\S]*?)\n```/g)] + .map((match) => match[1]) + .find((code) => code.includes('ACCESS EXCLUSIVE')) + expect(snippet).toBeDefined() + if (!snippet) throw new Error('Missing standalone conversion') + // The temporary sibling resolves the same installed packages as the npm + // example. Execute the guide verbatim, including its real CLI entry point. + const directory = await mkdtemp( + fileURLToPath(new URL('../.guide-script-', import.meta.url)), + ) + try { + const script = `${directory}/migrate.ts` + await writeFile(script, snippet) + const exitCode = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, [script], { + env: { + ...process.env, + DATABASE_URL: databaseUrl, + MIGRATION_TABLE: table, + }, + stdio: 'ignore', + timeout: 110_000, + }) + child.once('error', reject) + child.once('exit', resolve) + }) + expect( + exitCode, + 'standalone guide conversion must commit successfully', + ).toBe(0) + expect(await app.read('v3')).toEqual(app.corpus) + expect(await app.find('v3', 'age', 36)).toEqual(['ada', 'alan']) + await app.rollback() + expect(await app.read('v2')).toEqual(app.corpus) + } finally { + await rm(directory, { recursive: true, force: true }) } }, 120_000) @@ -139,6 +154,68 @@ describe('Protect Postgres v2 to Stack v3', () => { expect(await app.find('v3', 'age', 36)).toEqual(['ada', 'alan']) }, 120_000) + it('accepts the supported shape after a physical column before pk was dropped', async () => { + const replacement = `${table}_replacement` + await sql.begin(async (tx) => { + await tx`CREATE TABLE ${tx(replacement)} ( + discarded text, pk text PRIMARY KEY, + email public.eql_v2_encrypted, + age public.eql_v2_encrypted, + notes public.eql_v2_encrypted + )` + await tx`INSERT INTO ${tx(replacement)} (pk, email, age, notes) + SELECT pk, email, age, notes FROM ${tx(table)}` + await tx`ALTER TABLE ${tx(replacement)} DROP COLUMN discarded` + await tx`DROP TABLE ${tx(table)}` + await tx`ALTER TABLE ${tx(replacement)} RENAME TO ${tx(table)}` + }) + await app.migrate() + expect(await app.read('v3')).toEqual(app.corpus) + await app.rollback() + expect(await app.read('v2')).toEqual(app.corpus) + }, 120_000) + + it.each(['pg_catalog', 'eql_v3,pg_catalog'])( + 'accepts the supported public table with search_path %s', + async (searchPath) => { + const restricted = postgres(databaseUrl, { + prepare: false, + max: 1, + onnotice: () => {}, + connection: { search_path: searchPath }, + }) + try { + const restrictedApp = await createExample(restricted, table) + await restrictedApp.migrate() + expect(await restrictedApp.read('v3')).toEqual(app.corpus) + expect(await restrictedApp.find('v3', 'age', 36)).toEqual([ + 'ada', + 'alan', + ]) + expect( + await restrictedApp.find('v3', 'email', 'nobody@example.com'), + ).toEqual([]) + await restrictedApp.rollback() + expect(await restrictedApp.read('v2')).toEqual(app.corpus) + } finally { + await restricted.end() + } + }, + 120_000, + ) + + it('refuses a source table without the required primary key', async () => { + await sql`ALTER TABLE ${sql(table)} DROP CONSTRAINT ${sql(`${table}_pkey`)}` + const before = await sql`SELECT * FROM ${sql(table)} ORDER BY pk` + await expect(app.migrate()).rejects.toThrow( + 'Unsupported table dependencies', + ) + expect(await sql`SELECT * FROM ${sql(table)} ORDER BY pk`).toEqual(before) + expect( + await sql`SELECT attname FROM pg_attribute WHERE attrelid = ${table}::regclass AND attname = 'email_v3'`, + ).toHaveLength(0) + }, 120_000) + it('refuses unsupported payload capabilities before changing source data or columns', async () => { await sql`UPDATE ${sql(table)} SET notes = notes::jsonb || '{"ob":"unsupported"}'::jsonb WHERE pk = 'ada'` const before = await sql`SELECT * FROM ${sql(table)} ORDER BY pk` diff --git a/scripts/sync-migration-guide.mjs b/scripts/sync-migration-guide.mjs new file mode 100644 index 000000000..a0830298d --- /dev/null +++ b/scripts/sync-migration-guide.mjs @@ -0,0 +1,26 @@ +import { readFileSync, writeFileSync } from 'node:fs' + +const source = new URL( + '../e2e/migration-guide/postgres-conversion.ts', + import.meta.url, +) +const target = new URL( + '../docs/migration/protect-dynamodb-to-stack.md', + import.meta.url, +) +const guide = readFileSync(target, 'utf8') +const start = '' +const end = '' +const region = new RegExp(`${start}[\\s\\S]*?${end}`, 'g') +if ([...guide.matchAll(region)].length !== 1) + throw new Error('Expected one migration snippet region') +const replacement = `${start}\n\n\`\`\`typescript\n${readFileSync(source, 'utf8').trimEnd()}\n\`\`\`\n\n${end}` +const updated = guide.replace(region, () => replacement) +if (process.argv.includes('--check')) { + if (guide !== updated) + throw new Error( + 'Migration snippet drift: run node scripts/sync-migration-guide.mjs', + ) +} else if (guide !== updated) { + writeFileSync(target, updated) +} From e7d6a02d3d92f466e57175711efb56e9f4bdf544 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 12 Nov 2026 15:43:31 +1100 Subject: [PATCH 6/7] docs(migration): pin guide references to the verified conversion revision Freeze executable and return links at the commit containing the shared conversion so later main changes cannot silently alter the pinned Stack 1.2.1 instructions. Every referenced path was verified and the linked conversion matches the generated snippet. Refs #1604, #1540 --- docs/migration/protect-dynamodb-to-stack.md | 6 +++--- e2e/migration-guide/README.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index c22703d20..613427142 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -30,7 +30,7 @@ pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode= The native build needs the repository's Rust toolchain and trusted mise configuration. These commands are for maintainers verifying the guide; a customer installing Stack from npm receives prebuilt binaries. The crossover task also runs the PostgreSQL upgrade and executable DynamoDB example. Credentialed tests fail if configuration is missing; a skipped or unrun suite is not migration evidence. -The [executable examples](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. +The [executable examples](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. ## Why the supported scalar attributes need no re-encryption @@ -313,7 +313,7 @@ The conversion performs these steps in one transaction: 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. -The complete script below is generated verbatim from the [canonical executable conversion](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/postgres-conversion.ts). The npm example and live suite use this same implementation; a hermetic check rejects snippet drift and strict TypeScript errors. The live suite also executes the pasted script itself (`postgres-upgrade › runs the standalone guide script against Protect-written source columns`). +The complete script below is generated verbatim from the [canonical executable conversion](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/postgres-conversion.ts). The npm example and live suite use this same implementation; a hermetic check rejects snippet drift and strict TypeScript errors. The live suite also executes the pasted script itself (`postgres-upgrade › runs the standalone guide script against Protect-written source columns`). Prepare a **separate migration-tool directory** before the maintenance window. The new application from step 2 has removed Protect; this one-off tool deliberately retains both pinned SDKs so source validation is independent of Stack: @@ -667,7 +667,7 @@ In rehearsal, test fresh Stack writes beside migrated rows, reads, lookups and u ### Reproduce the PostgreSQL example -Use the [npm example instructions](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. +Use the [npm example instructions](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. ## Optional: moving values between DynamoDB and Postgres diff --git a/e2e/migration-guide/README.md b/e2e/migration-guide/README.md index b2fbc985d..d2d8ba3cf 100644 --- a/e2e/migration-guide/README.md +++ b/e2e/migration-guide/README.md @@ -1,6 +1,6 @@ # Protect v2 to Stack v3 examples -These examples accompany the [two-store migration guide](https://github.com/cipherstash/stack/blob/0ecabc3f62ee8a117c467981ce00d312e6cfc636/docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. +These examples accompany the [two-store migration guide](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. Use Node.js 22.18 or later. Copy this directory to a disposable location outside the monorepo and run: From a6d33c9e8f7acc65158553bbce5b7b6ff62165a2 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 12 Nov 2026 17:26:41 +1100 Subject: [PATCH 7/7] test(migration): establish the dropped-column fixture in public The conversion explicitly targets public, but the regression fixture created and renamed an unqualified replacement. Bind its DDL to the same relation and assert both table presence and pk physical position before testing conversion, so a setup failure cannot masquerade as a migration failure. Refs #1604, #1540 --- .../protect-postgres-upgrade.e2e.test.ts | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/e2e/tests/protect-postgres-upgrade.e2e.test.ts b/e2e/tests/protect-postgres-upgrade.e2e.test.ts index 79381f5b8..2e0afae29 100644 --- a/e2e/tests/protect-postgres-upgrade.e2e.test.ts +++ b/e2e/tests/protect-postgres-upgrade.e2e.test.ts @@ -155,7 +155,8 @@ describe('Protect Postgres v2 to Stack v3', () => { }, 120_000) it('accepts the supported shape after a physical column before pk was dropped', async () => { - const replacement = `${table}_replacement` + const relation = `public.${table}` + const replacement = `public.${table}_replacement` await sql.begin(async (tx) => { await tx`CREATE TABLE ${tx(replacement)} ( discarded text, pk text PRIMARY KEY, @@ -164,11 +165,23 @@ describe('Protect Postgres v2 to Stack v3', () => { notes public.eql_v2_encrypted )` await tx`INSERT INTO ${tx(replacement)} (pk, email, age, notes) - SELECT pk, email, age, notes FROM ${tx(table)}` + SELECT pk, email, age, notes FROM ${tx(relation)}` await tx`ALTER TABLE ${tx(replacement)} DROP COLUMN discarded` - await tx`DROP TABLE ${tx(table)}` + await tx`DROP TABLE ${tx(relation)}` await tx`ALTER TABLE ${tx(replacement)} RENAME TO ${tx(table)}` }) + const [fixture] = + await sql`SELECT pg_catalog.to_regclass(${relation})::text AS relation` + expect( + fixture.relation, + 'fixture must create the public migration source', + ).not.toBeNull() + const attributes = await sql`SELECT attnum FROM pg_catalog.pg_attribute + WHERE attrelid = pg_catalog.to_regclass(${relation}) AND attname = 'pk' AND NOT attisdropped` + expect( + attributes.map((attribute) => attribute.attnum), + 'pk must retain physical attribute 2 after its predecessor is dropped', + ).toEqual([2]) await app.migrate() expect(await app.read('v3')).toEqual(app.corpus) await app.rollback()