Repository navigation
Conversation
WalkthroughWarning Review details and warnings were omitted to fit the comment limit. |
🦋 Changeset detectedLatest commit: d2af608 The changes in this PR will be included in the next version bump. This PR includes changesets to release 27 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 436 | 0 | 203 | 0 |
cargo-mutants exited with no surviving mutants recorded. The run may not have completed cleanly; check the workflow run logs.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a05affaaa5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
freshtonic
left a comment
There was a problem hiding this comment.
I request changes for two release problems on the crates.io side. The code itself is in good shape, and every check I ran passes.
How I reviewed this
The diff is large because generated code is committed. I reviewed the generators and their templates, then I ran the drift checks to prove that the committed output matches them. I did not read each generated file.
- EQL codegen:
eql-codegenrenders the version-neutral SQL source two times (eql_v3andeql_v4) from theeql-domainscatalogue. It also generates the Rust payload types andeql_gen.go. - Go:
stashgengenerates the*_stash.gofiles and the goldens.
Checks run (in a worktree at a05affaaa)
- EQL:
cargo test -p eql-codegen(includes the Go parity test): pass.mise run types:check,codegen:parity,test:schemas:parity: pass.mise run build, with the installer completeness gate and the v4 isolation gate: pass.git statuswas clean after all of these.
- Rust and Go:
cargo fmt --check: pass.cargo clippy --workspace --all-features --all-targets -D warnings: pass.cargo test -p stack-guest-abi -p stack-encrypt --all-features: pass.
- Go guests and SDK:
- All five WASI guests build, and the import-surface checks pass.
go vet ./...andgo test ./...inlanguages/golang: pass.go generate ./...leaves no diff.
- I built the v3 bundle and compared it with the
@cipherstash/eql@3.0.6tarball. Onlylints()and comments changed, so the claim "v3 SQL unchanged" holds. git merge-treeagainst currentmain: no conflicts.cargo publish --dry-run -p stack-encrypt-derive -p stack-encryptagainst crates.io: fails (see below).
Blocking
- stack-encrypt 0.3.0 cannot publish. The root release-plz line publishes every stack-* version that crates.io does not have, so merging this starts a publish of stack-encrypt 0.3.0. stack-encrypt requires
stack-kms0.1.0, and that is the only stack-kms version on crates.io. Butsrc/lib.rsre-exportsstack_kms::{diagnostic, ErrorPayload}, and these were added after the 0.1.0 release with no version bump. The packaged build fails withE0432: unresolved imports stack_kms::diagnostic, stack_kms::ErrorPayload.- Fix: bump stack-kms (for example to 0.2.0, with a CHANGELOG entry), and make stack-encrypt require that version.
- Note that the publish dry run on
main(tests-crates.yml) packages all five crates together, so stack-encrypt resolves the tree's stack-kms there. That step does not catch a stack-kms that crates.io lacks under the same version number.
- The stack-encrypt CHANGELOG has no 0.3.0 section.
Cargo.tomlsays 0.3.0, but about 850 lines, including a### Breakingsection, are still under## [Unreleased]. The root release-plz line only publishes and never rewrites the changelog, so 0.3.0 would ship under "Unreleased", as 0.1.0 did. Please rename the section to## [0.3.0] - <date>in this PR.
Non-blocking (see inline)
SchemaVersionis now generic. Rust does not apply a default const parameter in an expression, soSchemaVersion::CURRENTandSchemaVersion::default()stop compiling (E0284). The changeset ships this as a minor bump and says "plainSchemaVersionstill means v3".- The stored and single-bound query domain templates accept
"v": null. The range template already guards against this. - stack-encrypt will publish with an exact pin on
cllw-ore =0.5.2-rc.1. - There are some small nits: stale vitaminc version numbers in comments, a Go wipe path, an ADR reference, and the position of a
use.
Before you merge
eql-bindings now requires stack-encrypt 0.3.0. Its crates.io publish (publish_all_features) succeeds only after release-crates has published stack-encrypt 0.3.0, so blocker 1 must be fixed first. Otherwise the EQL crate publish fails too.
Checked and fine
- SQL renderer: it refuses literal
eql_v3/eql_v4names, unknown placeholders and unmatched blocks. - Range domains and
between: correct, with STRICT/IMMUTABLE accessors and an inlinablebetween. On non-superuser installs, the ORE range domains are disabled with the same NOT VALID CHECK as the single-bound ORE domains. - v4
ore_block_v2SQL: the layout offsets and the GF(2^128) doubling are correct, and errors do not echo the input bytes. - stack-guest-abi
stepandvalue: no new raw reads of host lengths, handles are never reused,MAX_OPENis enforced, and secrets areZeroizingwithOpaqueDebug. - Go step loop: every early exit after a pause aborts the operation and wipes the buffers.
maxPausedis tied to the Rust constant by a test. - Go public surface:
QueryTerm,VerifyandTargetLimitfollow the existingSeal/Openpattern, andMatchQuerynever prints its text. - CI: actions are pinned by SHA, both copies of the path filters were updated, and no release workflow changed.
- Changesets: the EQL changesets are minor, which is correct apart from the
SchemaVersionpoint.chacha20-0-10-2.mdmatches the lockfile moves. - Skills: no skill update is needed.
stash eql installstill installs v3 only, and no skill covers the Go SDK or Stack Encrypt.
Not blocking. The points in this review are still open, but they need not stop the merge.
feat(stack-encrypt, eql, go): write the storage-only Json type as one ciphertext
Everything under Unreleased landed after the rc.3 tag, so it moves under a dated rc.4 heading. The section adds an upgrade note: the EQL v4 SQL has not changed since rc.3, and the rc.3 bundle already defines the Json type's column type, so a database needs nothing new; a field with two order indexes must drop one before stashgen runs again; and Extend and Extension are deprecated names for Scope and Scopes. It also records a change that had no entry: the embedded guests are built with Vitamin C 0.9, whose frame codec wipes each allocation it outgrows. Stored values do not change.
Four paths copied plaintext into a buffer that grew, or dropped the copy as it was, so a freed heap block still held it: - The match index folded text with `str::to_lowercase`, which grows its output when folding lengthens the text (`İ`). The fold now runs in two steps, each in a buffer sized before it is written: every character on its own except `Σ`, then `str::to_lowercase` over that, where only `Σ` is left to change and both its forms are its own length. The result is `str::to_lowercase`'s, checked for every character beside a `Σ`. - A JSON predicate's escaped literal, and a `JsonPath`'s escaped quoted key, were decoded through serde_json's scratch buffer. Both now use the JSON text parser's string decoder, which sizes its buffer first. A path refused part way now wipes the keys it had read. - A sealed-root JSON index document was parsed by serde_json on open. It is now parsed by `json::parse_text`. The scalar protected-borrow bridge (`ScalarRef` and `sealed::Scalar`) is removed: `&S` is already a Vitamin C scalar, so the borrow is walked as it is. `OpaqueBlob` lends its `Buffer` as a `ProtectedRef`. The owned conversion keeps `risky_ref`, with the reason beside it: the borrowed description takes `&S` so it can take a field out of a borrowed struct, which a `ProtectedRef` cannot lend. Map keys copied while a stored map is opened are left unwiped, and the reason is recorded: a sealed tree stores its keys in the clear. The heap-hygiene suites gain the cases above, cancellation before a run and with a key request in flight, and growth, refusal and cancellation for the EQL text targets. Closes #1468
… if it opens Review of the previous commit found two copies still freed unwiped and one way a stored document could stop opening. - `JsonPath::parse` took a quoted key out of its wiping wrapper before it checked for the closing `]`, so a path refused there dropped the key as it was. The key now stays wrapped until it is pushed. - A JSON predicate read its steps into a plain `Vec<Segment>`, which became a wiping `JsonPath` only at the end, so a predicate refused after a key (`.**`, a method, a subscript) dropped the key as it was. The steps are now read into the path itself. - A sealed root is opened with `json::parse_text`, which refuses a number that is neither a 64-bit integer nor a double. A `Value` holds one only with serde_json's `arbitrary_precision`, but then the index sealed a document it could not open. The root's text is now checked against the same rules before it is sealed. The heap suite also gains cancellation of a JSON index document and a `JsonPlaintext`, and the EQL suite's query case says what it runs: a query term waits on no key request. Refs #1468
A second review found more copies of the same kind, and a copy the previous commit added. - An ordering at a wildcard cloned the path's keys into a plain `Vec` and could refuse before they were wrapped. Each half is now copied straight into a `JsonPath`. - A predicate's variable names, and the path and member text of a document operator's input, were held in plain `String`s. They are now held in wiping buffers, and a member's rooted path is built in one sized before it is written. - Checking a sealed root by parsing it a second time built a second copy of the document, and wiping a parsed `Value` cannot reach the map nodes a split frees. Each number is checked on its own instead: only a number can be refused, and only its text is copied. The fold skips its second pass for text with no `Σ`, which is the only character that pass changes. The rule for where a JSON string ends is written once, in `json::text`. Refs #1468
The mutants gate replaced the refusal message for a path that is not text with an empty one and no test noticed. Read a path from text and from a number directly, and assert the refusal names what was expected. Refs #1468
…ve-transitional-api refactor(stack-encrypt)!: replace CipherScope and dynamic::Scope with one CipherRef
…ntext-copies fix(stack-encrypt): wipe the plaintext copies that were freed unwiped
ExpectedContext is the check that a record with a context field runs on decrypt. The caller never needs to name it: the paired entry points take the NonEmpty<T> the caller expects, or nothing, and convert it. The plan for issue 1488 says it stops being a public type. It cannot be private to the crate. The derives emit it as the decrypt input of a context-field record in the caller's crate, and the EQL JSON document names it in its impl. So it is now hidden from the documentation, with a note that says why it is still reachable. ValuePlan::decryption_with_context took an ExpectedContext<C>. It now takes an Option<NonEmpty<C>>, so its signature names no hidden type. The tests that named the type pass a NonEmpty or () through Into.
The previous commit put #[doc(hidden)] on the re-export of ExpectedContext only. rustc and rustdoc read the attribute on the definition, so two things still showed the type: - rustc listed ExpectedContext<T> among the implementors of AllowsNoContext and AcceptsContext in every E0277 a caller gets for a wrong context. - rustdoc for crates that name the type (eql-bindings, and any crate using the derives with a context field) linked it to a page that is never generated. On docs.rs that is a 404 on about 50 EQL type pages. The attribute is now on the struct as well. Six trybuild snapshots lose the ExpectedContext<T> lines. In eql-bindings' docs the type name is now plain text. Also: - eql-bindings: the JsonTreeDocument DecryptInto impl no longer links to the hidden type. Its doc says what the caller passes. The impl itself is unchanged, so eql-bindings' API is the same. - CHANGELOG: the upgrade note used T for both the context type and the plaintext type. It now names them C and P, says that code naming the type still compiles but is outside semver, names decrypt_from for no check, and says eql-bindings also names the type. - tests/derive_plan.rs: the empty stored context test now checks that the error is the empty-context refusal, not just any error. - ADR-0012, the API plan and attributes.md no longer say the type is public, or that the whole plan item is done.
The docs that this change added now say who passes what, and what the record checks, in short sentences. The JSON document's decrypt docs list the two cases a caller has.
The test still built a dynamic::Scope, which CipherRef replaced, so the heap_hygiene binary did not compile with --all-features on next.
…-expected-context
Resolves the conflicts that block #1229. - eql-codegen generate.rs: keeps both sides of render_query_functions_file: next's optional EQL v4 query terms and main's cross-pair blockers. next's tests pass the new siblings argument; main's tests render the placeholder output as eql_v3 before reading it. - src/v3/scalars: regenerated from the merged generator. The auto-merge had carried main's literal eql_v3 names into next's placeholder files. - src/v3/json: main's fixes (cardinality for an empty array, the boolean array flag, no ignored query_json cast, schema-qualified ord operators), written with next's placeholders. These files are confined to EQL v3. - Go encrypt/transport.go: next's stepping transport, which has no transport_send; token_get already uses main's guest.Place. - Go cmd/stashgen/main_test.go: both sides' tests. main's closingEngine returns next's Check signature.
between is unpinned LANGUAGE sql so that it inlines, which means its body is parsed under the caller's search_path. Its bare >= and <= then resolve against whatever that path finds. For the ORE range the term is a composite type whose operators live in public, so with public off the path the comparison falls back to pg_catalog's record comparison and orders the ciphertext bytes, with no error. This is the bug main fixed for the other wrappers; between arrived on next and was not covered. The body now names OPERATOR(<schema>.>=) and OPERATOR(<schema>.<=), with the schema from Term::operator_schema. The catalog-wide wrapper test now checks every generated file in each EQL version it belongs to, which is what found this.
next's gate skips the live JS jobs for a pull request that changes only docs/**. main added docs/migration/protect-dynamodb-to-stack.md and e2e/tests/protect-upgrade-guide.e2e.test.ts, which runs in e2e-tests and holds the guide's claims to the suites it describes. After the merge a pull request editing only the guide would skip the test that checks it, and tests-workflow-gate.test.mjs fails on exactly that. docs/migration/** joins the gate's direct list, so a change there runs the jobs again.
…1115 chore: merge main into next
… tags Services that load their configuration with go-envconfig or caarlos0/env had to write their own struct for the four CS_* values, then open an auth store, build an access-key strategy, wrap the key, call NewCredentials and close the strategy and store themselves after Client.Close. A hand-written struct also prints both keys under %+v or a structured logger. AccessKeyCredentials is that struct, exported, with env tags in the comma form both loaders parse. It implements Credentials itself, so WithCredentials takes it directly and NewClient owns the credential guest and the strategy, as it does for AutoCredentials. It has no fallback to the environment or the developer profile: an empty field fails NewClient with ErrEncoding, naming the field and its variable. String, GoString and LogValue show the client id and workspace only. The guest-opening code AutoCredentials and OIDCFederation share is split out of resolveWithStrategy so the new type opens the guest the same way. Closes #1689
…hen embedded Review of the first commit found two ways the keys still printed. The String, GoString and LogValue methods sat on the struct, and fmt asks String only for some verbs: %d and %t printed the raw fields, and encoding/json (so zap and logrus JSON loggers) printed both keys. The methods were also promoted into any struct that embedded the credentials, so printing a service's whole configuration showed only the credentials and hid its other fields. The redaction now lives on the key fields' own type, Secret, a string type whose Format, LogValue and MarshalJSON all write [redacted]. Nothing is promoted, so embedding is safe and the outer struct prints in full. Both go-envconfig and caarlos0/env fill a Secret as they fill any string. Also from the review: a field holding only whitespace is refused like an empty one; ExampleAccessKeyCredentials is a complete program with checked output; the doc comment uses shorter sentences, says "client access key" throughout, and names NewCredentials as the choice when the key can be held as wipeable bytes. Refs #1689
… guest loads From review of dbb272c: - A typed-nil *AccessKeyCredentials passed NewClient's nil check inside the Credentials interface and panicked when its value method ran. NewClient now checks the credentials as they were built, host-side, before it loads the guest: the existing NewCredentials nil-token check, an empty or blank AccessKeyCredentials field, and a nil pointer, each an ErrEncoding. A refusal now names the field even where no guest is built. - Secret redacted for fmt, slog and encoding/json only; encoding/xml, yaml.v3 and BurntSushi/toml wrote the keys, as they look for encoding.TextMarshaler. MarshalText replaces MarshalJSON, and covers JSON too. Secret also gets String and GoString, for Stringer APIs. - Encoding then decoding the credentials leaves the text [redacted] in each key, which passed the blank check and then failed as a bad access key. NewClient now refuses it and says the credentials were encoded. - The "check ClientAccessKey and WorkspaceCRN" hint is added only for auth.ErrConfig, as AutoCredentials does; a cancelled context or a guest failure is no longer blamed on the access key. - fmt calls no method on a value reached through an unexported field, so a Secret there prints in full. No method can change that; the docs now say to keep the credentials in an exported field. - The internal comments that listed the credentials which open the credential guest now name AccessKeyCredentials. Refs #1689
The README, the CHANGELOG entries and the doc comments added for
AccessKeyCredentials and Secret had sentences over 25 words, asides in
brackets in the middle of a sentence, "which" clauses adding a second
idea, things given a person's verb ("fields carry", "credentials that
hold it"), and no serial comma. Each is split or reworded.
The docs also used terms a new reader could not look up: CRN is now
spelled out as Cloud Resource Name, the name used elsewhere in the repo,
and the client key's base64 form names the file `stash auth login`
writes. "mints the token from an identity provider's" now says what
OIDCFederation does: it exchanges the provider's token for a CipherStash
token.
The redacted-key error now says "contains" rather than "holds". No
behaviour changes.
Refs #1689
Everything under Unreleased landed after the rc.4 tag, so it moves under a dated rc.5 heading. The section records three changes that had no entry. The EQL v4 bundle built from next differs from the one attached to rc.4: it adds 1,060 function and operator pairs that refuse a comparison between two EQL types, which PostgreSQL otherwise answered as plain jsonb, returning false for equal values (#1547). Its comparison functions also name the schema of every built-in operator they call (#1551, #1667). The crypto guest wipes more plaintext copies (#1643), and the host no longer leaves data in a guest buffer when a write fails (#1550). The upgrade note follows rc.3's: install the attached bundle, then grant access again and re-create the indexes, because the bundle drops the eql_v4 schemas with CASCADE. The uninstall script is unchanged, go.mod is unchanged, and no generated file changes.
From the review of 085d821, each checked against the rc.4 and rc.5 bundles: - The ORE comparison functions in rc.4 called a bare `<` or `=` on eql_v4_internal.ore_block_v2, a composite type whose operators are in public. With public off the search_path, PostgreSQL resolved them to its record comparison: on an rc.4 install, `<` under `search_path = pg_catalog` compared the raw bytes and returned false, where `public` on the path reaches EQL's ORE comparison. This is a wrong-result fix, so it moves to Fixed and says when it applied. - Re-running the bundle drops more than the indexes and grants. Tested on PostgreSQL 17: a view, a restrictive row-level security policy and a CHECK constraint that use EQL were all dropped without an error, and dropping the restrictive policy left the table more visible. The notice lists only the first 100 of the 4,402 objects dropped. The upgrade steps now start with a query that lists them, tested to list exactly those objects on rc.4 and rc.5 and nothing on a fresh install. - The cross-type blockers (#1547) move to Fixed too: they turn a silent false into an error. - Two credential-guest fixes from #1553 apply without workspace pinning and are now listed: an oversized expires_in no longer makes every request refresh, and profile files are read into a wiped buffer. - The upgrade note names Secret beside AccessKeyCredentials, and one long line is re-wrapped. The section is rewritten to the plain-language rules: one idea in each sentence of 25 words or fewer, a condition before its instruction, and ORE spelled out.
docs(go): an rc.5 changelog section
Summary
This is the integration branch for a release candidate of EQL v4 and of Stack Encrypt's Rust and Go support. EQL is the SQL that lets PostgreSQL store and search encrypted values; EQL v4 is a second copy of it for values written by Stack Encrypt, the Rust encryption engine behind the Go SDK. The open pull requests for both are merged here first, so they can be tested together before any of them reaches
main.The release version is not decided yet.
Why a branch, not main
Some of this work depends on vitaminc and ore.rs code that is not yet on crates.io, so it pins those crates to a git commit.
stack-encryptandeql-bindingsare published crates, andcargo publishdrops those git pins, so nothing on this branch can be released until vitaminc 0.6 is published (cipherstash/vitaminc#389, which waits on cipherstash/vitaminc#463). Once it is, the git pins become crates.io versions and this branch merges tomainas a normal release.What lands here, in order
feat/aead-remove-legacy-traitsto one commit on vitamincmain, and takes vitaminc's breaking changes since then.Rules for this branch
rev, neverbranch. A branch is deleted when its stack merges.packages/eqlandpackages/stack-encrypt/fuzz. Two commits mean two copies of vitaminc's traits, and the code stops compiling.Changes
Verification
nextwas created frommainat 2a70a47. Nothing has been built yet, because nothing has changed.Related