diff --git a/docs/migration/protect-dynamodb-to-stack.md b/docs/migration/protect-dynamodb-to-stack.md index 25fb52210..613427142 100644 --- a/docs/migration/protect-dynamodb-to-stack.md +++ b/docs/migration/protect-dynamodb-to-stack.md @@ -30,7 +30,7 @@ pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode= The native build needs the repository's Rust toolchain and trusted mise configuration. These commands are for maintainers verifying the guide; a customer installing Stack from npm receives prebuilt binaries. The crossover task also runs the PostgreSQL upgrade and executable DynamoDB example. Credentialed tests fail if configuration is missing; a skipped or unrun suite is not migration evidence. -The [executable examples](../../e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. +The [executable examples](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/README.md) use npm and the pinned customer packages. Run them on disposable data. The DynamoDB example uses in-memory item persistence and exercises legacy reads, new writes, normalization, bigint serialization and boolean updates (`example › runs the npm DynamoDB example through legacy reads, new writes and boolean updates`). It is not an AWS deployment test. ## Why the supported scalar attributes need no re-encryption @@ -44,7 +44,7 @@ Stack reads and writes the same two attributes: - It decrypts a legacy `__source` with no read option (`compat › every legacy item decrypts, one at a time, with no read option`, `compat › every legacy item decrypts in one bulk call`), and gets the same plaintext protect-dynamodb does (`compat › agrees with what protect-dynamodb itself decrypts`). - Its `encryptQuery` mints the same `__hmac` that protect-dynamodb's `createSearchTerms` did (`compat › matches protect-dynamodb createSearchTerms for the same value`), so the tested text and numeric equality queries match the expected legacy items (`compat › every equality query finds exactly the legacy rows it should`). - A new Stack write sits beside legacy items, and one query and one bulk read cover both (`compat › new Stack writes sit beside legacy items: one query and one bulk read cover both`). -- Reading and querying leave the stored items byte-identical (`compat › Stack's reads and queries left every stored item byte-identical`). +- Reading and querying leave the in-memory stored item objects byte-identical (`compat › Stack's reads and queries left every stored item byte-identical`). - A tampered ciphertext fails to decrypt; it never returns wrong plaintext (`compat › fails, never returns wrong plaintext, when the ciphertext is tampered with`). ## Steps @@ -64,7 +64,21 @@ npm install --save-exact @cipherstash/stack@1.2.1 If the app's `package.json` also declares `@cipherstash/schema` or `@cipherstash/protect-ffi` directly, remove those with `npm uninstall`. Commit the resulting package manifest and lockfile. Stack retains its own newer FFI dependency; do not remove that transitive dependency. -Stack needs Node.js 22 or later. It reads the same four `CS_*` variables from the environment, so retain the same workspace and key access used to write the data. If the app passed credentials explicitly to Protect, populate these environment variables or configure the equivalent Stack credentials. If the app passed a `keyset` to `protect()`, pass the same keyset to `Encryption` as `config.keyset`. A client bound to a different keyset either fails to decrypt or, if it was granted the old keyset, decrypts but mints terms that match no existing `__hmac`. Keysets are untested by these suites. +Stack needs Node.js 22 or later. It reads the same four `CS_*` variables from the environment, so retain the same workspace and key access used to write the data. If the app passed credentials explicitly to Protect, populate these environment variables or pass the same four values to `Encryption` as `config`: + +```typescript +const client = await Encryption({ + schemas: [users], + config: { + workspaceCrn: process.env.CS_WORKSPACE_CRN, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, + }, +}) +``` + +If the app passed a `keyset` to `protect()`, pass the same keyset to `Encryption` as `config.keyset`. A client bound to a different keyset either fails to decrypt or, if it was granted the old keyset, decrypts but mints terms that match no existing `__hmac`. Keysets are untested by these suites. Stack finds its service endpoints from `CS_WORKSPACE_CRN`. Protect 10.2.1 did not: outside `ap-southeast-2` it needed `CS_CTS_HOST` and `CS_ZEROKMS_HOST` set explicitly. Stack still honours both variables when they are set, so remove them from ordinary managed deployments. Keep them only for a deliberately configured private or custom endpoint, and check that endpoint before relying on it. @@ -280,13 +294,17 @@ const customers = encryptedTable('customers', { const client = await Encryption({ schemas: [customers] }) ``` +If the application receives its credentials other than through `process.env`, pass them as `config`, as shown in DynamoDB step 2. + Configure reads and writes to use the `_v3` physical columns after cutover, while passing the original logical schema above to the encryption client. Handle every SDK `{ failure }` before using `{ data }`. ### 3. Stop writes and convert This first procedure requires a **write-maintenance window**. Stop all application, background and administrative writers; a transaction's table lock alone does not stop an old writer from resuming immediately after commit. Keep writes paused through verification and the rollback decision. -The [executable migration](../../e2e/migration-guide/postgres.ts) performs these steps in one transaction: +Run the conversion as a one-off script during the window, not at application startup. An application that alters its own schema on boot repeats DDL on every deploy and takes an `ACCESS EXCLUSIVE` lock while other instances are starting. + +The conversion performs these steps in one transaction: 1. Lock the source table and reject unsupported structure/dependencies before DDL. 2. Validate each legacy envelope and plaintext against the supported mappings. Recompute supported equality terms with Stack and compare them to the stored terms; fail on disagreement. @@ -295,9 +313,326 @@ The [executable migration](../../e2e/migration-guide/postgres.ts) performs these 5. Create functional equality indexes on `eql_v3.eq_term(email_v3)` and `eql_v3.eq_term(age_v3)`. 6. Read and decrypt the target values with Stack, compare plaintext and retained bytes against the source, then commit. A failed check rolls back the added columns and indexes too. +The complete script below is generated verbatim from the [canonical executable conversion](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/postgres-conversion.ts). The npm example and live suite use this same implementation; a hermetic check rejects snippet drift and strict TypeScript errors. The live suite also executes the pasted script itself (`postgres-upgrade › runs the standalone guide script against Protect-written source columns`). + +Prepare a **separate migration-tool directory** before the maintenance window. The new application from step 2 has removed Protect; this one-off tool deliberately retains both pinned SDKs so source validation is independent of Stack: + +```bash +npm init -y +npm pkg set type=module +npm install --save-exact @cipherstash/protect@10.2.1 @cipherstash/stack@1.2.1 postgres@3.4.9 +``` + +Save the complete snippet as `migrate.ts` in that directory. Retain its generated lockfile. Set the original four `CS_*` credentials and `DATABASE_URL`, then run `node --env-file=.env migrate.ts` with Node.js 22.18 or later. It defaults to `public.customers`; set `MIGRATION_TABLE` to another unqualified table name only if it has the same supported shape and logical `customers` identities. This managed-host example temporarily derives Protect's endpoints from the workspace region; adapt and rehearse it separately for private/custom endpoints or a non-default keyset. + +The preflight requires a primary key on `pk`, identifies its physical attribute from the catalog, and compares column type OIDs. Dropped predecessor columns and a search path without `public` are supported. Extra constraints, secondary indexes, views, triggers, policies and inheritance remain unsupported. + + + +```typescript +import { pathToFileURL } from 'node:url' +import { csColumn, csTable, protect } from '@cipherstash/protect' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +import postgres from 'postgres' + +/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ +export async function legacyProtect( + schemas: Parameters[0]['schemas'], +) { + const crn = process.env.CS_WORKSPACE_CRN + if (!crn) throw new Error('CS_WORKSPACE_CRN is required') + const region = crn.split(':')[1] + const saved = { + CS_CTS_HOST: process.env.CS_CTS_HOST, + CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, + } + process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` + process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` + try { + return await protect({ + schemas, + workspaceCrn: crn, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, + }) + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key] + else process.env[key] = value + } + } +} + +/** Check every SDK result; never continue with a failed encryption operation. */ +export function ok( + result: { data: T; failure?: undefined } | { failure: unknown }, + label: string, +): T { + if ('failure' in result && result.failure !== undefined) { + throw new Error(`${label} failed`, { cause: result.failure }) + } + if ('data' in result) return result.data + throw new Error(`${label} returned no data`) +} + +export async function createMigrationClients() { + const legacySchema = csTable('customers', { + email: csColumn('email').equality(), + age: csColumn('age').dataType('number').equality(), + notes: csColumn('notes'), + }) + const schema = encryptedTable('customers', { + email: types.TextEq('email'), + age: types.IntegerEq('age'), + notes: types.Text('notes'), + }) + const legacy = await legacyProtect([legacySchema]) + const stack = await Encryption({ schemas: [schema] }) + const columns = ['email', 'age', 'notes'] as const + return { legacySchema, schema, legacy, stack, columns } +} + +/** Bulk success is not enough: reject missing, reordered or failed items. */ +function verifiedBulk( + items: readonly ( + | { id?: string; data: unknown } + | { id?: string; error: unknown } + )[], + expected: number, + label: string, +): unknown[] { + if (items.length !== expected) throw new Error(`${label} bulk count mismatch`) + return items.map((item, i) => { + if (item.id !== String(i) || 'error' in item || !('data' in item)) + throw new Error(`${label} bulk item failed or reordered`) + return item.data + }) +} + +export async function createConversion( + sql: postgres.Sql, + table: string, + clients = createMigrationClients(), +) { + if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) + throw new Error('Invalid example table name') + const relation = `public.${table}` + const { legacy, stack, schema, columns } = await clients + async function checkDependencies( + tx: postgres.TransactionSql, + allowTargetIndexes: boolean, + ) { + // pg_get_indexdef qualifies functions only when they are not visible. + // Stabilize its rendering for our exact rollback-index check; SET LOCAL + // restores the caller's search path when this transaction ends. + await tx`SET LOCAL search_path = pg_catalog` + const dependencies = await tx`SELECT + NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = ${relation}::regclass AND contype = 'p' + AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[]) + OR EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${relation}::regclass AND NOT tgisinternal) + OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${relation}::regclass OR confrelid = ${relation}::regclass) + -- PostgreSQL 18 also lists the primary key column's NOT NULL here. + AND NOT (contype IN ('p', 'n') AND conrelid = ${relation}::regclass AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[])) + -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read + -- it from the column on every version. + OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND attname <> 'pk' AND NOT attisdropped AND attnotnull) + OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid + WHERE indrelid = ${relation}::regclass AND NOT indisprimary AND NOT ( + ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 + AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') + OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) + OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${relation}::regclass AND classid = 'pg_catalog.pg_rewrite'::regclass) + OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${relation}::regclass OR inhrelid = ${relation}::regclass) + AS unsupported` + if (dependencies[0].unsupported) + throw new Error( + 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', + ) + } + + async function migrate() { + await sql.begin(async (tx) => { + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` + // Fail before adding any columns. This example understands one exact shape. + const attributes = + await tx`SELECT attname, atttypid = CASE WHEN attname = 'pk' THEN 'pg_catalog.text'::regtype ELSE 'public.eql_v2_encrypted'::regtype END AS supported_type + FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND NOT attisdropped ORDER BY attnum` + if ( + JSON.stringify(attributes.map((a) => [a.attname, a.supported_type])) !== + JSON.stringify([ + ['pk', true], + ['email', true], + ['age', true], + ['notes', true], + ]) + ) + throw new Error( + 'Unsupported columns: expected pk and three EQL v2 columns', + ) + await checkDependencies(tx, false) + const rows = + await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(relation)} ORDER BY pk` + if (rows.length === 0) + throw new Error('Empty source: this example requires legacy records') + for (const row of rows) { + for (const column of columns) { + const payload = row[column] + if ( + payload?.v !== 2 || + payload.k !== 'ct' || + typeof payload.c !== 'string' || + payload.i?.t !== 'customers' || + payload.i?.c !== column || + Object.keys(payload).some( + (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), + ) || + // Protect represents unused index capabilities with null fields. + payload.ob != null || + payload.bf != null || + (column === 'notes' + ? payload.hm != null + : typeof payload.hm !== 'string') + ) { + throw new Error(`Unsupported legacy payload in ${column}`) + } + } + } + const payloads = rows.flatMap((row) => + columns.map((column) => row[column]), + ) + const sourcePlaintexts = verifiedBulk( + ok( + await legacy.bulkDecrypt( + payloads.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt source', + ), + payloads.length, + 'Source', + ) + for (const [i] of payloads.entries()) { + const column = columns[i % columns.length] + const plain = sourcePlaintexts[i] + if ( + column === 'age' + ? !Number.isSafeInteger(plain) + : typeof plain !== 'string' + ) + throw new Error(`Unsupported plaintext in ${column}`) + } + const terms = ok( + await stack.encryptQuery( + rows.flatMap((_row, i) => [ + { + value: String(sourcePlaintexts[i * columns.length]), + table: schema, + column: schema.email, + }, + { + value: Number(sourcePlaintexts[i * columns.length + 1]), + table: schema, + column: schema.age, + }, + ]), + ), + 'verify equality mappings', + ) + if (terms.length !== rows.length * 2) + throw new Error('Equality term count mismatch') + for (const [i, term] of terms.entries()) { + const column = i % 2 === 0 ? 'email' : 'age' + if ( + typeof term !== 'object' || + term === null || + !('hm' in term) || + term.hm !== rows[Math.floor(i / 2)][column].hm + ) + throw new Error(`Unsupported equality mapping in ${column}`) + } + await tx`ALTER TABLE ${tx(relation)} + ADD COLUMN email_v3 public.eql_v3_text_eq, + ADD COLUMN age_v3 public.eql_v3_integer_eq, + ADD COLUMN notes_v3 public.eql_v3_text` + // Rebuild only the envelope. Source columns and their bytes stay intact. + for (const column of columns) { + await tx`UPDATE ${tx(relation)} SET ${tx(`${column}_v3`)} = jsonb_build_object( + 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' + ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb + ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` + } + await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(email_v3))` + await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(age_v3))` + // Read through Stack before committing, so a wrong mapping rolls back DDL too. + const converted = + await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)} ORDER BY pk` + if ( + converted.length !== rows.length || + converted.some((row, i) => row.pk !== rows[i].pk) + ) + throw new Error('Conversion verification failed: row identity mismatch') + const targets = converted.flatMap((row) => + columns.map((column) => row[`${column}_v3`]), + ) + const targetPlaintexts = verifiedBulk( + ok( + await stack.bulkDecrypt( + targets.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt target', + ), + targets.length, + 'Target', + ) + for (const [i, after] of targets.entries()) { + const before = payloads[i] + if ( + after.c !== before.c || + (after.hm ?? null) !== (before.hm ?? null) || + targetPlaintexts[i] !== sourcePlaintexts[i] + ) + throw new Error( + `Conversion verification failed in ${columns[i % columns.length]}`, + ) + } + }) + await sql`ANALYZE ${sql(relation)}` + } + + return { migrate, checkDependencies } +} + +// Importing this module supplies helpers; executing it runs the conversion. +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + const databaseUrl = process.env.DATABASE_URL + if (!databaseUrl) throw new Error('DATABASE_URL is required') + const sql = postgres(databaseUrl, { prepare: false }) + try { + const conversion = await createConversion( + sql, + process.env.MIGRATION_TABLE ?? 'customers', + ) + await conversion.migrate() + } finally { + await sql.end() + } +} +``` + + + +**Rerunning the conversion:** after a successful commit, the table includes the three `_v3` columns. Running this conversion again fails the exact source-shape check before any DDL or data changes; the transaction rolls back without altering the completed migration. It is a one-off conversion, not an idempotent startup hook. Do not treat the mere presence of target columns as proof that a migration completed, and do not drop them to force a rerun. If a transaction failed, diagnose the failure and confirm its changes rolled back before deciding whether to retry. If the client lost the connection around commit, inspect and verify the database state first. Continue with verification and cutover after a successful migration. + Only the envelope and database structure change for these proven mappings; the ciphertext and equality terms are copied unchanged. Do not generalize this to other capabilities or to v4 (`postgres-upgrade › migrates actual v2 columns, preserves ciphertext and queries, and rolls back before writes resume`). -The example reads the small source table into memory and validates each value. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. +The example reads the small source table into memory. It bulk-decrypts source values once with Protect, retains those plaintexts for comparison, batches equality-term generation, and bulk-decrypts converted values once with Stack. Every bulk result is checked for item failures, count and order. These reduce repeated SDK work; they are not a measured network-call or lock-duration guarantee. It is not a resumable production backfill service. Rehearse duration, database lock time and key-service usage before choosing a maintenance window for your data size. ### 4. Verify queries, indexes and cutover @@ -332,7 +667,7 @@ In rehearsal, test fresh Stack writes beside migrated rows, reads, lookups and u ### Reproduce the PostgreSQL example -Use the [npm example instructions](../../e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. +Use the [npm example instructions](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/e2e/migration-guide/README.md) with a disposable database containing both EQL generations. `seed` creates a new table and refuses an existing one; `migrate` validates and creates targets; `verify` reads them; `rollback` demonstrates recovery before writes; `write` demonstrates why that recovery boundary closes. No step drops or reinstalls an EQL schema. ## Optional: moving values between DynamoDB and Postgres diff --git a/e2e/migration-guide/README.md b/e2e/migration-guide/README.md index 771f7b6dc..d2d8ba3cf 100644 --- a/e2e/migration-guide/README.md +++ b/e2e/migration-guide/README.md @@ -1,6 +1,6 @@ # Protect v2 to Stack v3 examples -These examples accompany the [two-store migration guide](../../docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. +These examples accompany the [two-store migration guide](https://github.com/cipherstash/stack/blob/21631ecf68fd5356fdd4c2631550a46931123dc4/docs/migration/protect-dynamodb-to-stack.md). EQL v2/v3 are data generations; the target npm package is Stack 1.2.1. Nothing here uses v4. Use Node.js 22.18 or later. Copy this directory to a disposable location outside the monorepo and run: @@ -56,3 +56,5 @@ The final command **must fail** because source columns do not contain the new St The example deliberately supports only an uncustomized table with a text primary key, text equality, safe whole-number equality and storage-only text under the fixed `customers` logical identity. It refuses extra columns, secondary indexes, non-primary constraints, dependent views, user triggers, row-level security and inheritance. Review those objects rather than removing them to bypass a refusal. Run the existing credentialed `test:protect-crossover` task from the repository to verify these functions against real Postgres and ZeroKMS. It also tests unsupported-input refusal, fresh writes and actual equality-index engagement. See the guide for the maintainer build commands. + +`postgres-conversion.ts` is the self-contained conversion used by the example and reproduced verbatim in the guide. To update the checked-in snippet, run `node scripts/sync-migration-guide.mjs` from the repository root; `--check` verifies it without writing. Keep Protect installed in this separate migration tool until verification and the rollback decision are complete. The new application can remove Protect independently. diff --git a/e2e/migration-guide/legacy.ts b/e2e/migration-guide/legacy.ts index fbc212104..75b9cda4f 100644 --- a/e2e/migration-guide/legacy.ts +++ b/e2e/migration-guide/legacy.ts @@ -1,42 +1 @@ -import { protect } from '@cipherstash/protect' - -/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ -export async function legacyProtect( - schemas: Parameters[0]['schemas'], -) { - const crn = process.env.CS_WORKSPACE_CRN - if (!crn) throw new Error('CS_WORKSPACE_CRN is required') - const region = crn.split(':')[1] - const saved = { - CS_CTS_HOST: process.env.CS_CTS_HOST, - CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, - } - process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` - process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` - try { - return await protect({ - schemas, - workspaceCrn: crn, - clientId: process.env.CS_CLIENT_ID, - clientKey: process.env.CS_CLIENT_KEY, - accessKey: process.env.CS_CLIENT_ACCESS_KEY, - }) - } finally { - for (const [key, value] of Object.entries(saved)) { - if (value === undefined) delete process.env[key] - else process.env[key] = value - } - } -} - -/** Check every SDK result; never continue with a failed encryption operation. */ -export function ok( - result: { data: T; failure?: undefined } | { failure: unknown }, - label: string, -): T { - if ('failure' in result && result.failure !== undefined) { - throw new Error(`${label} failed`, { cause: result.failure }) - } - if ('data' in result) return result.data - throw new Error(`${label} returned no data`) -} +export { legacyProtect, ok } from './postgres-conversion.ts' diff --git a/e2e/migration-guide/postgres-conversion.ts b/e2e/migration-guide/postgres-conversion.ts new file mode 100644 index 000000000..a04bbe879 --- /dev/null +++ b/e2e/migration-guide/postgres-conversion.ts @@ -0,0 +1,294 @@ +import { pathToFileURL } from 'node:url' +import { csColumn, csTable, protect } from '@cipherstash/protect' +import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +import postgres from 'postgres' + +/** Protect 10.2.1 needs explicit managed endpoints outside ap-southeast-2. */ +export async function legacyProtect( + schemas: Parameters[0]['schemas'], +) { + const crn = process.env.CS_WORKSPACE_CRN + if (!crn) throw new Error('CS_WORKSPACE_CRN is required') + const region = crn.split(':')[1] + const saved = { + CS_CTS_HOST: process.env.CS_CTS_HOST, + CS_ZEROKMS_HOST: process.env.CS_ZEROKMS_HOST, + } + process.env.CS_CTS_HOST = `https://${region}.cts.cipherstashmanaged.net` + process.env.CS_ZEROKMS_HOST = `https://${region}.viturhosted.net` + try { + return await protect({ + schemas, + workspaceCrn: crn, + clientId: process.env.CS_CLIENT_ID, + clientKey: process.env.CS_CLIENT_KEY, + accessKey: process.env.CS_CLIENT_ACCESS_KEY, + }) + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key] + else process.env[key] = value + } + } +} + +/** Check every SDK result; never continue with a failed encryption operation. */ +export function ok( + result: { data: T; failure?: undefined } | { failure: unknown }, + label: string, +): T { + if ('failure' in result && result.failure !== undefined) { + throw new Error(`${label} failed`, { cause: result.failure }) + } + if ('data' in result) return result.data + throw new Error(`${label} returned no data`) +} + +export async function createMigrationClients() { + const legacySchema = csTable('customers', { + email: csColumn('email').equality(), + age: csColumn('age').dataType('number').equality(), + notes: csColumn('notes'), + }) + const schema = encryptedTable('customers', { + email: types.TextEq('email'), + age: types.IntegerEq('age'), + notes: types.Text('notes'), + }) + const legacy = await legacyProtect([legacySchema]) + const stack = await Encryption({ schemas: [schema] }) + const columns = ['email', 'age', 'notes'] as const + return { legacySchema, schema, legacy, stack, columns } +} + +/** Bulk success is not enough: reject missing, reordered or failed items. */ +function verifiedBulk( + items: readonly ( + | { id?: string; data: unknown } + | { id?: string; error: unknown } + )[], + expected: number, + label: string, +): unknown[] { + if (items.length !== expected) throw new Error(`${label} bulk count mismatch`) + return items.map((item, i) => { + if (item.id !== String(i) || 'error' in item || !('data' in item)) + throw new Error(`${label} bulk item failed or reordered`) + return item.data + }) +} + +export async function createConversion( + sql: postgres.Sql, + table: string, + clients = createMigrationClients(), +) { + if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) + throw new Error('Invalid example table name') + const relation = `public.${table}` + const { legacy, stack, schema, columns } = await clients + async function checkDependencies( + tx: postgres.TransactionSql, + allowTargetIndexes: boolean, + ) { + // pg_get_indexdef qualifies functions only when they are not visible. + // Stabilize its rendering for our exact rollback-index check; SET LOCAL + // restores the caller's search path when this transaction ends. + await tx`SET LOCAL search_path = pg_catalog` + const dependencies = await tx`SELECT + NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = ${relation}::regclass AND contype = 'p' + AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[]) + OR EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${relation}::regclass AND NOT tgisinternal) + OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${relation}::regclass OR confrelid = ${relation}::regclass) + -- PostgreSQL 18 also lists the primary key column's NOT NULL here. + AND NOT (contype IN ('p', 'n') AND conrelid = ${relation}::regclass AND conkey = ARRAY[(SELECT attnum FROM pg_attribute WHERE attrelid = ${relation}::regclass AND attname = 'pk' AND NOT attisdropped)]::smallint[])) + -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read + -- it from the column on every version. + OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND attname <> 'pk' AND NOT attisdropped AND attnotnull) + OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid + WHERE indrelid = ${relation}::regclass AND NOT indisprimary AND NOT ( + ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 + AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') + OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) + OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${relation}::regclass AND classid = 'pg_catalog.pg_rewrite'::regclass) + OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${relation}::regclass) + OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${relation}::regclass OR inhrelid = ${relation}::regclass) + AS unsupported` + if (dependencies[0].unsupported) + throw new Error( + 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', + ) + } + + async function migrate() { + await sql.begin(async (tx) => { + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` + // Fail before adding any columns. This example understands one exact shape. + const attributes = + await tx`SELECT attname, atttypid = CASE WHEN attname = 'pk' THEN 'pg_catalog.text'::regtype ELSE 'public.eql_v2_encrypted'::regtype END AS supported_type + FROM pg_attribute WHERE attrelid = ${relation}::regclass + AND attnum > 0 AND NOT attisdropped ORDER BY attnum` + if ( + JSON.stringify(attributes.map((a) => [a.attname, a.supported_type])) !== + JSON.stringify([ + ['pk', true], + ['email', true], + ['age', true], + ['notes', true], + ]) + ) + throw new Error( + 'Unsupported columns: expected pk and three EQL v2 columns', + ) + await checkDependencies(tx, false) + const rows = + await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(relation)} ORDER BY pk` + if (rows.length === 0) + throw new Error('Empty source: this example requires legacy records') + for (const row of rows) { + for (const column of columns) { + const payload = row[column] + if ( + payload?.v !== 2 || + payload.k !== 'ct' || + typeof payload.c !== 'string' || + payload.i?.t !== 'customers' || + payload.i?.c !== column || + Object.keys(payload).some( + (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), + ) || + // Protect represents unused index capabilities with null fields. + payload.ob != null || + payload.bf != null || + (column === 'notes' + ? payload.hm != null + : typeof payload.hm !== 'string') + ) { + throw new Error(`Unsupported legacy payload in ${column}`) + } + } + } + const payloads = rows.flatMap((row) => + columns.map((column) => row[column]), + ) + const sourcePlaintexts = verifiedBulk( + ok( + await legacy.bulkDecrypt( + payloads.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt source', + ), + payloads.length, + 'Source', + ) + for (const [i] of payloads.entries()) { + const column = columns[i % columns.length] + const plain = sourcePlaintexts[i] + if ( + column === 'age' + ? !Number.isSafeInteger(plain) + : typeof plain !== 'string' + ) + throw new Error(`Unsupported plaintext in ${column}`) + } + const terms = ok( + await stack.encryptQuery( + rows.flatMap((_row, i) => [ + { + value: String(sourcePlaintexts[i * columns.length]), + table: schema, + column: schema.email, + }, + { + value: Number(sourcePlaintexts[i * columns.length + 1]), + table: schema, + column: schema.age, + }, + ]), + ), + 'verify equality mappings', + ) + if (terms.length !== rows.length * 2) + throw new Error('Equality term count mismatch') + for (const [i, term] of terms.entries()) { + const column = i % 2 === 0 ? 'email' : 'age' + if ( + typeof term !== 'object' || + term === null || + !('hm' in term) || + term.hm !== rows[Math.floor(i / 2)][column].hm + ) + throw new Error(`Unsupported equality mapping in ${column}`) + } + await tx`ALTER TABLE ${tx(relation)} + ADD COLUMN email_v3 public.eql_v3_text_eq, + ADD COLUMN age_v3 public.eql_v3_integer_eq, + ADD COLUMN notes_v3 public.eql_v3_text` + // Rebuild only the envelope. Source columns and their bytes stay intact. + for (const column of columns) { + await tx`UPDATE ${tx(relation)} SET ${tx(`${column}_v3`)} = jsonb_build_object( + 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' + ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb + ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` + } + await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(email_v3))` + await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(relation)} (eql_v3.eq_term(age_v3))` + // Read through Stack before committing, so a wrong mapping rolls back DDL too. + const converted = + await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)} ORDER BY pk` + if ( + converted.length !== rows.length || + converted.some((row, i) => row.pk !== rows[i].pk) + ) + throw new Error('Conversion verification failed: row identity mismatch') + const targets = converted.flatMap((row) => + columns.map((column) => row[`${column}_v3`]), + ) + const targetPlaintexts = verifiedBulk( + ok( + await stack.bulkDecrypt( + targets.map((data, i) => ({ id: String(i), data })), + ), + 'decrypt target', + ), + targets.length, + 'Target', + ) + for (const [i, after] of targets.entries()) { + const before = payloads[i] + if ( + after.c !== before.c || + (after.hm ?? null) !== (before.hm ?? null) || + targetPlaintexts[i] !== sourcePlaintexts[i] + ) + throw new Error( + `Conversion verification failed in ${columns[i % columns.length]}`, + ) + } + }) + await sql`ANALYZE ${sql(relation)}` + } + + return { migrate, checkDependencies } +} + +// Importing this module supplies helpers; executing it runs the conversion. +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + const databaseUrl = process.env.DATABASE_URL + if (!databaseUrl) throw new Error('DATABASE_URL is required') + const sql = postgres(databaseUrl, { prepare: false }) + try { + const conversion = await createConversion( + sql, + process.env.MIGRATION_TABLE ?? 'customers', + ) + await conversion.migrate() + } finally { + await sql.end() + } +} diff --git a/e2e/migration-guide/postgres.ts b/e2e/migration-guide/postgres.ts index 280429d25..3b6d82060 100644 --- a/e2e/migration-guide/postgres.ts +++ b/e2e/migration-guide/postgres.ts @@ -1,25 +1,22 @@ -import { csColumn, csTable } from '@cipherstash/protect' -import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' import type postgres from 'postgres' -import { legacyProtect, ok } from './legacy.ts' +import { + createConversion, + createMigrationClients, + ok, +} from './postgres-conversion.ts' /** A deliberately small migration example, not a general backfill service. */ export async function createExample(sql: postgres.Sql, table: string) { if (!/^[a-z][a-z0-9_]{0,39}$/.test(table)) throw new Error('Invalid example table name') - const legacySchema = csTable('customers', { - email: csColumn('email').equality(), - age: csColumn('age').dataType('number').equality(), - notes: csColumn('notes'), - }) - const schema = encryptedTable('customers', { - email: types.TextEq('email'), - age: types.IntegerEq('age'), - notes: types.Text('notes'), - }) - const legacy = await legacyProtect([legacySchema]) - const stack = await Encryption({ schemas: [schema] }) - const columns = ['email', 'age', 'notes'] as const + const clients = createMigrationClients() + const { legacySchema, schema, legacy, stack, columns } = await clients + const { migrate, checkDependencies } = await createConversion( + sql, + table, + clients, + ) + const relation = `public.${table}` const corpus = [ { pk: 'ada', email: 'ada@example.com', age: 36, notes: 'engines' }, { pk: 'alan', email: 'alan@example.com', age: 36, notes: 'machines' }, @@ -29,7 +26,7 @@ export async function createExample(sql: postgres.Sql, table: string) { async function seed() { // Never replace an existing table or seed a customer's database in place. await sql.begin(async (tx) => { - await tx`CREATE TABLE ${tx(table)} ( + await tx`CREATE TABLE ${tx(relation)} ( pk text PRIMARY KEY, email public.eql_v2_encrypted, age public.eql_v2_encrypted, @@ -57,7 +54,7 @@ export async function createExample(sql: postgres.Sql, table: string) { }), 'legacy notes', ) - await tx`INSERT INTO ${tx(table)} VALUES (${row.pk}, ${tx.json(email)}, ${tx.json(age)}, ${tx.json(notes)})` + await tx`INSERT INTO ${tx(relation)} VALUES (${row.pk}, ${tx.json(email)}, ${tx.json(age)}, ${tx.json(notes)})` } }) } @@ -67,8 +64,8 @@ export async function createExample(sql: postgres.Sql, table: string) { // base jsonb type explicitly instead of passing serialized text to the SDK. const rows = generation === 'v2' - ? await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(table)} ORDER BY pk` - : await sql`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, notes_v3::jsonb AS notes FROM ${sql(table)} ORDER BY pk` + ? await sql`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${sql(relation)} ORDER BY pk` + : await sql`SELECT pk, email_v3::jsonb AS email, age_v3::jsonb AS age, notes_v3::jsonb AS notes FROM ${sql(relation)} ORDER BY pk` return Promise.all( rows.map(async (row) => { const values = await Promise.all( @@ -131,161 +128,13 @@ export async function createExample(sql: postgres.Sql, table: string) { const jsonTerm: postgres.JSONValue = JSON.parse(JSON.stringify(term)) const rows = generation === 'v2' - ? await sql`SELECT pk FROM ${sql(table)} WHERE ${sql(column)} = ${sql.json(jsonTerm)}::jsonb ORDER BY pk` + ? await sql`SELECT pk FROM ${sql(relation)} WHERE ${sql(column)} = ${sql.json(jsonTerm)}::jsonb ORDER BY pk` : column === 'email' - ? await sql`SELECT pk FROM ${sql(table)} WHERE eql_v3.eq(email_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_text_eq) ORDER BY pk` - : await sql`SELECT pk FROM ${sql(table)} WHERE eql_v3.eq(age_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_integer_eq) ORDER BY pk` + ? await sql`SELECT pk FROM ${sql(relation)} WHERE eql_v3.eq(email_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_text_eq) ORDER BY pk` + : await sql`SELECT pk FROM ${sql(relation)} WHERE eql_v3.eq(age_v3, ${sql.json(jsonTerm)}::jsonb::eql_v3.query_integer_eq) ORDER BY pk` return rows.map((row) => row.pk) } - async function checkDependencies( - tx: postgres.TransactionSql, - allowTargetIndexes: boolean, - ) { - const dependencies = await tx`SELECT - EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid = ${table}::regclass AND NOT tgisinternal) - OR EXISTS (SELECT 1 FROM pg_constraint WHERE (conrelid = ${table}::regclass OR confrelid = ${table}::regclass) - -- PostgreSQL 18 also lists the primary key column's NOT NULL here. - AND NOT (contype IN ('p', 'n') AND conrelid = ${table}::regclass AND conkey = ARRAY[1]::smallint[])) - -- PostgreSQL 17 and earlier keep NOT NULL out of pg_constraint, so read - -- it from the column on every version. - OR EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = ${table}::regclass - AND attnum > 1 AND NOT attisdropped AND attnotnull) - OR EXISTS (SELECT 1 FROM pg_index ix JOIN pg_class idx ON idx.oid = ix.indexrelid - WHERE indrelid = ${table}::regclass AND NOT indisprimary AND NOT ( - ${allowTargetIndexes} AND NOT indisunique AND indpred IS NULL AND indnatts = 1 - AND ((idx.relname = ${`${table}_email_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(email_v3)') - OR (idx.relname = ${`${table}_age_v3_idx`} AND pg_get_indexdef(indexrelid, 1, true) = 'eql_v3.eq_term(age_v3)')))) - OR EXISTS (SELECT 1 FROM pg_depend WHERE refobjid = ${table}::regclass AND classid = 'pg_rewrite'::regclass) - OR (SELECT relrowsecurity OR relispartition OR relkind <> 'r' FROM pg_class WHERE oid = ${table}::regclass) - OR EXISTS (SELECT 1 FROM pg_policy WHERE polrelid = ${table}::regclass) - OR EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = ${table}::regclass OR inhrelid = ${table}::regclass) - AS unsupported` - if (dependencies[0].unsupported) - throw new Error( - 'Unsupported table dependencies: review constraints, indexes, views, triggers, policies and inheritance first', - ) - } - - async function migrate() { - await sql.begin(async (tx) => { - await tx`LOCK TABLE ${tx(table)} IN ACCESS EXCLUSIVE MODE` - // Fail before adding any columns. This example understands one exact shape. - const attributes = - await tx`SELECT attname, atttypid::regtype::text AS type - FROM pg_attribute WHERE attrelid = ${table}::regclass - AND attnum > 0 AND NOT attisdropped ORDER BY attnum` - if ( - JSON.stringify(attributes.map((a) => [a.attname, a.type])) !== - JSON.stringify([ - ['pk', 'text'], - ['email', 'eql_v2_encrypted'], - ['age', 'eql_v2_encrypted'], - ['notes', 'eql_v2_encrypted'], - ]) - ) - throw new Error( - 'Unsupported columns: expected pk and three EQL v2 columns', - ) - await checkDependencies(tx, false) - const rows = - await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes FROM ${tx(table)} ORDER BY pk` - if (rows.length === 0) - throw new Error('Empty source: this example requires legacy records') - for (const row of rows) { - for (const column of columns) { - const payload = row[column] - if ( - payload?.v !== 2 || - payload.k !== 'ct' || - typeof payload.c !== 'string' || - payload.i?.t !== 'customers' || - payload.i?.c !== column || - Object.keys(payload).some( - (key) => !['v', 'k', 'i', 'c', 'hm', 'ob', 'bf'].includes(key), - ) || - // Protect represents unused index capabilities with null fields. - payload.ob != null || - payload.bf != null || - (column === 'notes' - ? payload.hm != null - : typeof payload.hm !== 'string') - ) { - throw new Error(`Unsupported legacy payload in ${column}`) - } - const plain = ok( - await legacy.decrypt(payload), - 'validate legacy payload', - ) - if ( - column === 'age' - ? !Number.isSafeInteger(plain) - : typeof plain !== 'string' - ) { - throw new Error(`Unsupported plaintext in ${column}`) - } - if (column !== 'notes') { - const term = - column === 'email' - ? ok( - await stack.encryptQuery(String(plain), { - table: schema, - column: schema.email, - }), - 'verify email mapping', - ) - : ok( - await stack.encryptQuery(Number(plain), { - table: schema, - column: schema.age, - }), - 'verify age mapping', - ) - if ( - typeof term !== 'object' || - term === null || - !('hm' in term) || - term.hm !== payload.hm - ) { - throw new Error(`Unsupported equality mapping in ${column}`) - } - } - } - } - await tx`ALTER TABLE ${tx(table)} - ADD COLUMN email_v3 public.eql_v3_text_eq, - ADD COLUMN age_v3 public.eql_v3_integer_eq, - ADD COLUMN notes_v3 public.eql_v3_text` - // Rebuild only the envelope. Source columns and their bytes stay intact. - for (const column of columns) { - await tx`UPDATE ${tx(table)} SET ${tx(`${column}_v3`)} = jsonb_build_object( - 'v', 3, 'i', ${tx(column)}::jsonb->'i', 'c', ${tx(column)}::jsonb->'c' - ) || CASE WHEN ${column} = 'notes' THEN '{}'::jsonb - ELSE jsonb_build_object('hm', ${tx(column)}::jsonb->'hm') END` - } - await tx`CREATE INDEX ${tx(`${table}_email_v3_idx`)} ON ${tx(table)} (eql_v3.eq_term(email_v3))` - await tx`CREATE INDEX ${tx(`${table}_age_v3_idx`)} ON ${tx(table)} (eql_v3.eq_term(age_v3))` - // Read through Stack before committing, so a wrong mapping rolls back DDL too. - const converted = - await tx`SELECT pk, email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(table)} ORDER BY pk` - for (let i = 0; i < rows.length; i++) { - for (const column of columns) { - const before = rows[i][column] - const after = converted[i][`${column}_v3`] - if ( - after.c !== before.c || - (after.hm ?? null) !== (before.hm ?? null) || - ok(await stack.decrypt(after), 'verify target') !== - ok(await legacy.decrypt(before), 'verify source') - ) { - throw new Error(`Conversion verification failed in ${column}`) - } - } - } - }) - await sql`ANALYZE ${sql(table)}` - } - async function write() { const model = { pk: 'new', @@ -294,18 +143,18 @@ export async function createExample(sql: postgres.Sql, table: string) { notes: 'written by Stack', } const encrypted = ok(await stack.encryptModel(model, schema), 'fresh write') - await sql`INSERT INTO ${sql(table)} (pk, email_v3, age_v3, notes_v3) + await sql`INSERT INTO ${sql(relation)} (pk, email_v3, age_v3, notes_v3) VALUES (${model.pk}, ${sql.json(encrypted.email)}, ${sql.json(encrypted.age)}, ${sql.json(encrypted.notes)})` return model } async function rollback() { await sql.begin(async (tx) => { - await tx`LOCK TABLE ${tx(table)} IN ACCESS EXCLUSIVE MODE` + await tx`LOCK TABLE ${tx(relation)} IN ACCESS EXCLUSIVE MODE` await checkDependencies(tx, true) const rows = await tx`SELECT pk, email::jsonb AS email, age::jsonb AS age, notes::jsonb AS notes, - email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(table)}` + email_v3::jsonb AS email_v3, age_v3::jsonb AS age_v3, notes_v3::jsonb AS notes_v3 FROM ${tx(relation)}` for (const row of rows) { for (const column of columns) { if ( @@ -322,7 +171,7 @@ export async function createExample(sql: postgres.Sql, table: string) { } // Preflight permits only our two derived indexes. PostgreSQL otherwise // drops same-table constraints/indexes with a column even without CASCADE. - await tx`ALTER TABLE ${tx(table)} DROP COLUMN email_v3, DROP COLUMN age_v3, DROP COLUMN notes_v3` + await tx`ALTER TABLE ${tx(relation)} DROP COLUMN email_v3, DROP COLUMN age_v3, DROP COLUMN notes_v3` }) } diff --git a/e2e/tests/protect-guide-script.e2e.test.ts b/e2e/tests/protect-guide-script.e2e.test.ts new file mode 100644 index 000000000..c08321862 --- /dev/null +++ b/e2e/tests/protect-guide-script.e2e.test.ts @@ -0,0 +1,332 @@ +import { execFileSync } from 'node:child_process' +import { readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { runInNewContext } from 'node:vm' +import ts from 'typescript' +import { describe, expect, it, vi } from 'vitest' + +const guide = readFileSync( + new URL('../../docs/migration/protect-dynamodb-to-stack.md', import.meta.url), + 'utf8', +) +const fences = [...guide.matchAll(/```typescript\n([\s\S]*?)\n```/g)].map( + (match) => match[1], +) +const conversion = + fences.find((code) => code.includes('ACCESS EXCLUSIVE')) ?? '' +const credentials = fences.find((code) => code.includes('workspaceCrn:')) ?? '' +const setup = `import { Encryption, encryptedTable, types } from '@cipherstash/stack/v3' +const customers = encryptedTable('customers', {email: types.TextEq('email'), age: types.IntegerEq('age'), notes: types.Text('notes')}) +const users = customers +const client = await Encryption({schemas: [customers]}) +` + +// Compile the actual customer paste, in the same module-resolution context as +// the example. Diagnostics are assertion data, never a build-error surrogate. +function diagnostics(code: string) { + const filename = fileURLToPath( + new URL('../migration-guide/pasted-guide.ts', import.meta.url), + ) + const options: ts.CompilerOptions = { + strict: true, + noEmit: true, + skipLibCheck: true, + target: ts.ScriptTarget.ESNext, + module: ts.ModuleKind.ESNext, + moduleResolution: ts.ModuleResolutionKind.Bundler, + allowImportingTsExtensions: true, + } + const host = ts.createCompilerHost(options) + const original = host.getSourceFile.bind(host) + host.getSourceFile = ( + path, + languageVersion, + onError, + shouldCreateNewSourceFile, + ) => + path === filename + ? ts.createSourceFile(path, code, languageVersion, true) + : original(path, languageVersion, onError, shouldCreateNewSourceFile) + const program = ts.createProgram([filename], options, host) + return ts + .getPreEmitDiagnostics(program) + .map((diagnostic) => + ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n'), + ) +} + +type Payload = { + v: number + k?: string + i: { t: string; c: string } + c: string + hm?: string +} +const columns = ['email', 'age', 'notes'] as const +const originalPlain = { email: 'ada@example.com', age: 36, notes: 'engines' } + +type Fault = + | 'none' + | 'wrong-plaintext' + | 'source-item' + | 'target-item' + | 'source-count' + | 'target-count' + | 'source-order' + | 'target-order' + | 'target-row' + | 'term-count' + | 'term-mismatch' + +// Execute the actual pasted implementation with only SQL/SDK boundaries faked. +// The SQL fake supplies fixed results, not a second parser or migration engine. +async function executeConversion(fault: Fault = 'none', example = false) { + const row: Record = { pk: 'ada' } + for (const column of columns) + row[column] = { + v: 2, + k: 'ct', + i: { t: 'customers', c: column }, + c: column, + ...(column === 'notes' ? {} : { hm: column }), + } + const target: Record = { + pk: fault === 'target-row' ? 'wrong-row' : 'ada', + } + for (const column of columns) + target[`${column}_v3`] = { ...(row[column] as Payload), v: 3 } + const plain = (payload: Payload) => + originalPlain[payload.i.c as keyof typeof originalPlain] + type Input = { id: string; data: Payload } + type Output = + | { id: string; data: string | number } + | { id: string; error: string } + function bulk(inputs: Input[], generation: 'source' | 'target') { + const results: Output[] = inputs.map(({ id, data }) => ({ + id, + data: + generation === 'target' && + fault === 'wrong-plaintext' && + data.i.c === 'notes' + ? 'wrong-but-consistent' + : plain(data), + })) + if (fault === `${generation}-item`) + results[1] = { id: '1', error: 'synthetic failure' } + if (fault === `${generation}-count`) results.pop() + if (fault === `${generation}-order`) results.reverse() + return { data: results } + } + const legacy = { + decrypt: vi.fn(), + bulkDecrypt: vi.fn(async (inputs: Input[]) => bulk(inputs, 'source')), + } + const stack = { + decrypt: vi.fn(), + bulkDecrypt: vi.fn(async (inputs: Input[]) => bulk(inputs, 'target')), + encryptQuery: vi.fn(async (terms: { column: string }[]) => { + const results = terms.map(({ column }) => ({ + hm: fault === 'term-mismatch' ? 'wrong' : column, + })) + if (fault === 'term-count') results.pop() + return { data: results } + }), + } + const statements: string[] = [] + const sql = Object.assign( + (parts: TemplateStringsArray | string) => { + if (typeof parts === 'string') return parts + const query = parts.join('?') + statements.push(query) + if (query.includes('AS unsupported')) return [{ unsupported: false }] + if (query.includes('SELECT attname')) + return ['pk', ...columns].map((attname) => ({ + attname, + supported_type: true, + })) + if (query.includes('SELECT pk')) + return [query.includes('email_v3::jsonb') ? target : row] + return [] + }, + { + begin: async (fn: (tx: unknown) => Promise) => { + try { + await fn(sql) + statements.push('COMMIT') + } catch (error) { + statements.push('ROLLBACK') + throw error + } + }, + end: async () => {}, + }, + ) + function executable(source: string) { + const parsed = ts.createSourceFile( + 'snippet.ts', + source, + ts.ScriptTarget.ESNext, + true, + ) + const withoutImports = parsed.statements + .filter( + (statement) => + !ts.isImportDeclaration(statement) && + !ts.isExportDeclaration(statement), + ) + .map((statement) => statement.getFullText(parsed)) + .join('\n') + return ts + .transpileModule(withoutImports, { + compilerOptions: { + target: ts.ScriptTarget.ESNext, + module: ts.ModuleKind.ESNext, + }, + }) + .outputText.replace(/\bexport /g, '') + .replace(/import.meta.url/g, "'file:///imported.ts'") + } + const exampleSource = readFileSync( + new URL('../migration-guide/postgres.ts', import.meta.url), + 'utf8', + ) + let failure: unknown + try { + await runInNewContext( + `(async () => {${executable(conversion)}; ${example ? executable(exampleSource) : ''}; await (await ${example ? 'createExample' : 'createConversion'}(sql, 'customers')).migrate()})()`, + { + sql, + csColumn: () => ({ + equality() { + return this + }, + dataType() { + return this + }, + }), + csTable: (_name: string, fields: object) => fields, + encryptedTable: (_name: string, fields: object) => fields, + types: { + TextEq: () => 'email', + IntegerEq: () => 'age', + Text: () => 'notes', + }, + protect: async () => legacy, + Encryption: async () => stack, + process: { + argv: [], + env: { CS_WORKSPACE_CRN: 'test:ap-southeast-2:test' }, + }, + }, + ) + } catch (error) { + failure = error + } + return { failure, legacy, client: stack, statements } +} + +describe('pasted migration guide', () => { + it('G1: the conversion compiles with strict TypeScript', () => { + expect(diagnostics(conversion)).toEqual([]) + }) + + it('G2: refuses a Stack interpretation that disagrees with Protect', async () => { + const result = await executeConversion('wrong-plaintext') + expect( + result.failure, + 'independent source validation must reject a consistently wrong Stack plaintext', + ).toMatchObject({ message: 'Conversion verification failed in notes' }) + expect(result.statements).toContain('ROLLBACK') + expect(result.statements.some((query) => query.includes('ANALYZE'))).toBe( + false, + ) + }) + + it('G5: retains source plaintext and decrypts converted values in bulk', async () => { + const result = await executeConversion() + expect(result.failure).toBeUndefined() + expect(result.legacy.decrypt).not.toHaveBeenCalled() + expect(result.legacy.bulkDecrypt).toHaveBeenCalledTimes(1) + expect(result.client.encryptQuery).toHaveBeenCalledTimes(1) + expect(result.statements).toContain('COMMIT') + expect(result.client.decrypt).not.toHaveBeenCalled() + expect(result.client.bulkDecrypt).toHaveBeenCalledTimes(1) + }) + + it('G5: the executable example does not decrypt each source twice under the lock', async () => { + const result = await executeConversion('none', true) + expect(result.failure).toBeUndefined() + expect(result.legacy.decrypt).not.toHaveBeenCalled() + expect(result.legacy.bulkDecrypt).toHaveBeenCalledTimes(1) + expect(result.client.encryptQuery).toHaveBeenCalledTimes(1) + expect(result.statements).toContain('COMMIT') + expect(result.client.decrypt).not.toHaveBeenCalled() + expect(result.client.bulkDecrypt).toHaveBeenCalledTimes(1) + }) + + it.each([ + ['source-item', 'Source bulk item failed or reordered'], + ['target-item', 'Target bulk item failed or reordered'], + ['source-count', 'Source bulk count mismatch'], + ['target-count', 'Target bulk count mismatch'], + ['source-order', 'Source bulk item failed or reordered'], + ['target-order', 'Target bulk item failed or reordered'], + ['target-row', 'Conversion verification failed: row identity mismatch'], + ['term-count', 'Equality term count mismatch'], + ['term-mismatch', 'Unsupported equality mapping in email'], + ] as const)('rejects %s before commit', async (fault, message) => { + const result = await executeConversion(fault) + expect(result.failure).toMatchObject({ message }) + expect(result.statements).toContain('ROLLBACK') + expect(result.statements).not.toContain('COMMIT') + expect(result.statements.some((query) => query.includes('ANALYZE'))).toBe( + false, + ) + }) + + it('G3: reproduces the complete canonical module without drift', () => { + const canonical = readFileSync( + new URL('../migration-guide/postgres-conversion.ts', import.meta.url), + 'utf8', + ) + expect(conversion).toBe(canonical.trimEnd()) + expect(() => + execFileSync(process.execPath, [ + fileURLToPath( + new URL('../../scripts/sync-migration-guide.mjs', import.meta.url), + ), + '--check', + ]), + ).not.toThrow() + }) + + it('G6: links the portable example and guide at an immutable revision', () => { + const readme = readFileSync( + new URL('../migration-guide/README.md', import.meta.url), + 'utf8', + ) + const links = [...`${guide}\n${readme}`.matchAll(/\]\(([^)]+)\)/g)] + .map((match) => match[1]) + .filter( + (url) => + url.includes('migration-guide/') || + url.includes('protect-dynamodb-to-stack.md'), + ) + expect(links.length).toBeGreaterThan(0) + for (const link of links) + expect(link).toMatch( + /^https:\/\/github\.com\/cipherstash\/stack\/blob\/[0-9a-f]{40}\//, + ) + }) + + it('G7: the explicit credentials example compiles without an undeclared env object', () => { + expect( + diagnostics( + setup.replace( + 'const client = await Encryption({schemas: [customers]})', + '', + ) + credentials, + ), + ).toEqual([]) + }) +}) diff --git a/e2e/tests/protect-postgres-upgrade.e2e.test.ts b/e2e/tests/protect-postgres-upgrade.e2e.test.ts index cfeb9d4e5..2e0afae29 100644 --- a/e2e/tests/protect-postgres-upgrade.e2e.test.ts +++ b/e2e/tests/protect-postgres-upgrade.e2e.test.ts @@ -1,3 +1,6 @@ +import { spawn } from 'node:child_process' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' import { installEqlV3 } from '@cipherstash/test-kit/install' import postgres from 'postgres' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' @@ -74,6 +77,53 @@ describe('Protect Postgres v2 to Stack v3', () => { ).toEqual(before) }, 120_000) + it('runs the standalone guide script against Protect-written source columns', async () => { + const guide = await readFile( + new URL( + '../../docs/migration/protect-dynamodb-to-stack.md', + import.meta.url, + ), + 'utf8', + ) + const snippet = [...guide.matchAll(/```typescript\n([\s\S]*?)\n```/g)] + .map((match) => match[1]) + .find((code) => code.includes('ACCESS EXCLUSIVE')) + expect(snippet).toBeDefined() + if (!snippet) throw new Error('Missing standalone conversion') + // The temporary sibling resolves the same installed packages as the npm + // example. Execute the guide verbatim, including its real CLI entry point. + const directory = await mkdtemp( + fileURLToPath(new URL('../.guide-script-', import.meta.url)), + ) + try { + const script = `${directory}/migrate.ts` + await writeFile(script, snippet) + const exitCode = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, [script], { + env: { + ...process.env, + DATABASE_URL: databaseUrl, + MIGRATION_TABLE: table, + }, + stdio: 'ignore', + timeout: 110_000, + }) + child.once('error', reject) + child.once('exit', resolve) + }) + expect( + exitCode, + 'standalone guide conversion must commit successfully', + ).toBe(0) + expect(await app.read('v3')).toEqual(app.corpus) + expect(await app.find('v3', 'age', 36)).toEqual(['ada', 'alan']) + await app.rollback() + expect(await app.read('v2')).toEqual(app.corpus) + } finally { + await rm(directory, { recursive: true, force: true }) + } + }, 120_000) + it('reads and queries fresh Stack writes beside legacy rows and refuses stale rollback', async () => { await app.migrate() const fresh = await app.write() @@ -104,6 +154,81 @@ describe('Protect Postgres v2 to Stack v3', () => { expect(await app.find('v3', 'age', 36)).toEqual(['ada', 'alan']) }, 120_000) + it('accepts the supported shape after a physical column before pk was dropped', async () => { + const relation = `public.${table}` + const replacement = `public.${table}_replacement` + await sql.begin(async (tx) => { + await tx`CREATE TABLE ${tx(replacement)} ( + discarded text, pk text PRIMARY KEY, + email public.eql_v2_encrypted, + age public.eql_v2_encrypted, + notes public.eql_v2_encrypted + )` + await tx`INSERT INTO ${tx(replacement)} (pk, email, age, notes) + SELECT pk, email, age, notes FROM ${tx(relation)}` + await tx`ALTER TABLE ${tx(replacement)} DROP COLUMN discarded` + await tx`DROP TABLE ${tx(relation)}` + await tx`ALTER TABLE ${tx(replacement)} RENAME TO ${tx(table)}` + }) + const [fixture] = + await sql`SELECT pg_catalog.to_regclass(${relation})::text AS relation` + expect( + fixture.relation, + 'fixture must create the public migration source', + ).not.toBeNull() + const attributes = await sql`SELECT attnum FROM pg_catalog.pg_attribute + WHERE attrelid = pg_catalog.to_regclass(${relation}) AND attname = 'pk' AND NOT attisdropped` + expect( + attributes.map((attribute) => attribute.attnum), + 'pk must retain physical attribute 2 after its predecessor is dropped', + ).toEqual([2]) + await app.migrate() + expect(await app.read('v3')).toEqual(app.corpus) + await app.rollback() + expect(await app.read('v2')).toEqual(app.corpus) + }, 120_000) + + it.each(['pg_catalog', 'eql_v3,pg_catalog'])( + 'accepts the supported public table with search_path %s', + async (searchPath) => { + const restricted = postgres(databaseUrl, { + prepare: false, + max: 1, + onnotice: () => {}, + connection: { search_path: searchPath }, + }) + try { + const restrictedApp = await createExample(restricted, table) + await restrictedApp.migrate() + expect(await restrictedApp.read('v3')).toEqual(app.corpus) + expect(await restrictedApp.find('v3', 'age', 36)).toEqual([ + 'ada', + 'alan', + ]) + expect( + await restrictedApp.find('v3', 'email', 'nobody@example.com'), + ).toEqual([]) + await restrictedApp.rollback() + expect(await restrictedApp.read('v2')).toEqual(app.corpus) + } finally { + await restricted.end() + } + }, + 120_000, + ) + + it('refuses a source table without the required primary key', async () => { + await sql`ALTER TABLE ${sql(table)} DROP CONSTRAINT ${sql(`${table}_pkey`)}` + const before = await sql`SELECT * FROM ${sql(table)} ORDER BY pk` + await expect(app.migrate()).rejects.toThrow( + 'Unsupported table dependencies', + ) + expect(await sql`SELECT * FROM ${sql(table)} ORDER BY pk`).toEqual(before) + expect( + await sql`SELECT attname FROM pg_attribute WHERE attrelid = ${table}::regclass AND attname = 'email_v3'`, + ).toHaveLength(0) + }, 120_000) + it('refuses unsupported payload capabilities before changing source data or columns', async () => { await sql`UPDATE ${sql(table)} SET notes = notes::jsonb || '{"ob":"unsupported"}'::jsonb WHERE pk = 'ada'` const before = await sql`SELECT * FROM ${sql(table)} ORDER BY pk` diff --git a/scripts/sync-migration-guide.mjs b/scripts/sync-migration-guide.mjs new file mode 100644 index 000000000..a0830298d --- /dev/null +++ b/scripts/sync-migration-guide.mjs @@ -0,0 +1,26 @@ +import { readFileSync, writeFileSync } from 'node:fs' + +const source = new URL( + '../e2e/migration-guide/postgres-conversion.ts', + import.meta.url, +) +const target = new URL( + '../docs/migration/protect-dynamodb-to-stack.md', + import.meta.url, +) +const guide = readFileSync(target, 'utf8') +const start = '' +const end = '' +const region = new RegExp(`${start}[\\s\\S]*?${end}`, 'g') +if ([...guide.matchAll(region)].length !== 1) + throw new Error('Expected one migration snippet region') +const replacement = `${start}\n\n\`\`\`typescript\n${readFileSync(source, 'utf8').trimEnd()}\n\`\`\`\n\n${end}` +const updated = guide.replace(region, () => replacement) +if (process.argv.includes('--check')) { + if (guide !== updated) + throw new Error( + 'Migration snippet drift: run node scripts/sync-migration-guide.mjs', + ) +} else if (guide !== updated) { + writeFileSync(target, updated) +}