Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .cargo/mutants.toml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,11 @@ exclude_re = [
# `Ok(())`, so the replacement is the body. Text overrides it with
# `query::order_text`, pinned by `an_ordering_refuses_only_text_it_cannot_encode`.
'stack-encrypt/src/sem/mod\.rs:\d+:\d+: replace sealed::Sealed::check_query -> Result<\(\), super::TermError> with Ok\(\(\)\)$',
# A JSON document type's default options are the defaults: the trait's
# body is `JsonOptions::default()`. A type that overrides them is pinned
# by `a_field_that_names_no_options_queries_under_its_types_own`, and
# `Option<T>` forwarding by `a_nullable_field_queries_under_its_types_own_options`.
'stack-encrypt/src/json/target\.rs:\d+:\d+: replace JsonQueries::default_options -> JsonOptions with Default::default\(\)$',
# Both unit-context conversions explicitly return Self::default().
'stack-encrypt/src/target/context\.rs:\d+:\d+: replace <impl From<\(\)> for (DeclaredContext|ExpectedContext<T>)>::from -> Self with Default::default\(\)$',
# A caller shape with no placeholder IS `CallerShape::default()`:
Expand Down
12 changes: 12 additions & 0 deletions .changeset/eql-json-field-marker-queries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'@cipherstash/eql': minor
---

`eql-bindings`, the Rust crate released with this package: a `JsonTreeDocument` field of a derived Stack Encrypt record is queried from its field marker, and each query returns the EQL query type a parameter binds as.

- `EncryptedCustomer::PROFILE.contains(doc)` gives a `JsonTreeQuery` for `@>`, and `contained_by(doc)` one for `<@`.
- `at(path).gt(v)` and the other comparisons give a `JsonTreeLeafQuery`, `at(path).between(lo, hi)` two of them, and `at(path).selector()` the `JsonTreeSelector` that `->` takes.
- `elements(path).eq(member, v)..` gives a `JsonTreeElementsQuery`, `predicate(text)` a `JsonTreePredicateQuery`, and `children(path)` a `JsonTreeToken`.
- `#[stash(json(ordered = ["$.age"]))]` on the field names the paths the documents order. The field is sealed under those options and queried under them.

Each query is the operand a data plan naming `JsonSearch` derives for the same column and options, which is what the Go SDK sends. The stored format, the SQL bundle and the TypeScript types do not change.
2 changes: 1 addition & 1 deletion .changeset/eql-remove-structural-traits.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

- Every type keeps `EncryptTarget` and `DecryptSource`. Write with `value.encrypt_into_with_context(&keyset, column)` and read with `P::decrypt_from_with_context(stored, &keyset, column)`.
- A typed query is a field marker of a derived record, such as `EncryptedMember::EMAIL.eq("bob").encrypt(&keyset)`.
- A JSON document is a `JsonPlaintext`, not a `serde_json::Value`. A typed JSON query has no replacement yet: derive the operand through a data plan that names `JsonSearch`.
- A JSON document is a `JsonPlaintext`, not a `serde_json::Value`. A typed JSON query is a document field's marker query, such as `EncryptedCustomer::PROFILE.at("$.age").gt(30).encrypt(&keyset)`.
- `JsonTree` no longer implements `stack_encrypt::target::Layout`. `encryption()`, `index()` and `specs()` are its own methods.

The stored format, the SQL bundle, and the TypeScript types do not change.
106 changes: 61 additions & 45 deletions docs/plans/2026-10-24-stack-encrypt-api-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,30 +421,32 @@ the strict checks on JSON input. A program can also encrypt a borrowed
any [borrowed value](#encrypt-borrowed-values).

With the default options, no path has an ordering term. So a query cannot
compare the order of the values at a path. To order some paths, declare a
named type with typed Rust options:
compare the order of the values at a path. To order some paths, a record
field names them with `#[stash(json(..))]`, as Go's struct tag does
(`encrypt_into=JsonSearch(ordered=$.age)`). Decided by Dan on 16 November
2026, in place of a named type with options in a `target(..)` attribute:

```rust
use stack_encrypt::json::JsonOptions;

fn profile_options() -> Result<JsonOptions, TermError> {
JsonOptions::new().ordering(["$.age", "$.visits[*].at"])
#[derive(Zeroize)]
struct Customer {
profile: JsonPlaintext,
}

// Not on `next` yet: a JSON target with typed options.
#[derive(EncryptTarget, DecryptSource)]
#[stash(
plaintext = JsonPlaintext,
transparent,
target(column = ("customers", "profile"), options = profile_options)
)]
struct EncryptedProfile(JsonTreeDocument);

let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?;
let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?;
let opened = JsonPlaintext::decrypt_from(encrypted, &keyset).await?;
#[stash(struct = Customer, scope = "customers")]
struct EncryptedCustomer {
#[stash(json(ordered = ["$.age", "$.visits[*].at", "$.visits[*].place"]))]
profile: JsonTreeDocument,
}

let customer = Customer { profile: JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)? };
let encrypted: EncryptedCustomer = customer.encrypt_into(&keyset).await?;
let opened = Customer::decrypt_from(encrypted, &keyset).await?;
```

The field is sealed under the options, and its marker queries under them.
`sealed_root = false` in the same list turns off the sealed root.

#### Query a JSON document

A JSON query is a query on the document's field marker. The marker has
Expand All @@ -461,28 +463,27 @@ declaration:

The marker carries both, so the query names only its kind and its
values. The output type is the EQL query type, fixed by the marker.
Decided on 7 November 2026.
Decided on 7 November 2026. Built in #1459; this example runs as
`packages/eql/tests/encryption/tests/json_tree_marker.rs`.

```rust
use eql_bindings::v4::json_tree::{JsonTreeElementsQuery, JsonTreeLeafQuery, JsonTreeQuery};

// Not on `next` yet: field markers and their query methods. The method
// names are proposals.
let ada: JsonTreeQuery = EncryptedProfile::DOCUMENT
let ada: JsonTreeQuery = EncryptedCustomer::PROFILE
.contains(json!({ "name": "Ada" }))
.encrypt(&keyset)
.await?;
let older: JsonTreeLeafQuery = EncryptedProfile::DOCUMENT
let older: JsonTreeLeafQuery = EncryptedCustomer::PROFILE
.at("$.age")
.gt(30)
.encrypt(&keyset)
.await?;
let adults: JsonTreeLeafQuery = EncryptedProfile::DOCUMENT
let (low, high): (JsonTreeLeafQuery, JsonTreeLeafQuery) = EncryptedCustomer::PROFILE
.at("$.age")
.between(18, 65)
.encrypt(&keyset)
.await?;
let home: JsonTreeElementsQuery = EncryptedProfile::DOCUMENT
let home: JsonTreeElementsQuery = EncryptedCustomer::PROFILE
.elements("$.visits[*]")
.eq("place", "home")
.gt("at", 15)
Expand All @@ -508,10 +509,11 @@ The kinds of JSON query, the marker method for each, and what each finds:
| Query | Marker method | What it finds | EQL query type |
| --- | --- | --- | --- |
| Containment | `contains(doc)`, `contained_by(doc)`, and `at(path).eq(v)` | Rows whose document contains the given document, or is contained by it. Equality of a scalar at a path is containment of a one-entry document, so `at(path).eq(v)` lowers to it | `JsonTreeQuery` |
| Comparison at a path | `at(path).gt(v)`, `.lt(v)`, `.between(a, b)`, and the rest | Rows where the value at an ordered path orders against, or lies between, the given values | `JsonTreeLeafQuery` |
| Comparison at a path | `at(path).gt(v)`, `.lt(v)`, `.between(a, b)`, and the rest | Rows where the value at an ordered path orders against, or lies between, the given values | `JsonTreeLeafQuery`; two for `between`, low then high |
| A path's selector | `at(path).selector()` | The selector `->` and `ORDER BY eql_v4.ord_term(col -> $1)` take | `JsonTreeSelector` |
| Element match | `elements(path).eq(member, v).gt(member, v)` | Rows where one element of an array meets several conditions together | `JsonTreeElementsQuery` |
| Jsonpath predicate | `predicate(text)` | Rows where a SQL/JSON path predicate is true | `JsonTreePredicateQuery` |
| Children and child count | `children(path)` | The entries directly below one node, or how many there are | A node token |
| Jsonpath predicate | `predicate(text)`, with `.vars(object)` | Rows where a SQL/JSON path predicate is true | `JsonTreePredicateQuery` |
| Children and child count | `children(path)` | The entries directly below one node, or how many there are | `JsonTreeToken` |

A path is text. So an order query on a path the options do not order is
refused at run time, before any key is fetched.
Expand All @@ -525,13 +527,20 @@ page](../../packages/eql/docs/reference/json-tree.md) gives the SQL for
each query. A scope chains on a JSON
query as on a scalar one: `.contains(doc).scope(tenant)`.

The engine already has one private input type for each JSON operator,
in `stack_encrypt::dynamic`. `DocumentInput` is for containment, `AtInput`
for a comparison at a path, and `ElementsInput` for element match. The marker
methods build those same inputs, so the typed form and the dynamic form
share one set of input rules. The operand types `Contains`, `AtPath`,
The marker methods build the same inputs the dynamic interface reads a
binding's frames into (`json::inputs`, compiled without the `dynamic`
feature), so the typed form and the dynamic form share one set of input
rules and derive the same operands. The operand types `Contains`, `AtPath`,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F4, minor: the plan says these operand types are retired, but this PR adds them back as public types.

This paragraph, which the PR edits, still ends: "The operand types Contains, AtPath, ElementMatch, Predicate, and Children that the plan proposed earlier are retired, replaced by these methods, on 7 November 2026." The same claim is at line 1565-1567 ("Names. ... are retired").

This PR adds marker::json::{Selector, ElementMatch, Predicate, Children} as public type tags. MarkedJson<T, Op, S> takes op::Contains, op::ContainedBy and op::AtPath<..> as Op (Review note 3). A reader of the plan will look for types that the plan says do not exist, and find them.

Verified.

Suggested fix: say that these names now exist as the kind parameter of MarkedJson, that they carry no values, and that the methods are still the API. Give the date of that decision. Also update "Names. The method names above are proposals" at line 1565.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cf40c37. Both places in the plan now say that those names were retired as operands a caller builds. They also say that MarkedJson<T, Op, S> names the query's kind with value-free type tags: op::Contains, op::ContainedBy or op::AtPath of a comparison, plus Selector, ElementMatch, Predicate and Children, recorded on 16 November 2026. The "proposals" line now says the method names were confirmed in the naming review.

`ElementMatch`, `Predicate`, and `Children` that the plan proposed earlier
are retired, replaced by these methods, on 7 November 2026.
were retired as operands a caller builds, on 7 November 2026: a caller
writes the methods above. The query a method returns,
`marker::json::MarkedJson<T, Op, S>`, names its kind as a type with no
values, `Op`: `op::Contains`, `op::ContainedBy`, or `op::AtPath` of a
comparison, which the dynamic `Op` enum is generated from, and, for the
kinds that have no operator type, `marker::json::Selector`,
`ElementMatch`, `Predicate` and `Children`. That kind decides what
`encrypt` returns. Recorded on 16 November 2026, in #1459's pull
request.

A document encrypted straight into `JsonTreeDocument`, with no record,
has no marker. It is queried through a [query plan](#a-query-plan), as any
Expand Down Expand Up @@ -1561,19 +1570,27 @@ Base64.
returns the EQL query type. The marker supplies the column and the
options. The dynamic parser keeps reading into the same input types, so
there is one definition of each input.
- **Names.** The method names above are proposals. The operand types
`Contains`, `AtPath`, `ElementMatch`, `Predicate`, and `Children` that
this plan proposed earlier are retired, on 7 November 2026.
- **Names.** The method names above were confirmed in the naming review
of 8 November 2026. The operand types `Contains`, `AtPath`,
`ElementMatch`, `Predicate`, and `Children` that this plan proposed
earlier were retired as operands a caller builds, on 7 November 2026;
the query a method returns names its kind with value-free type tags of
those names (see [Query a JSON document](#query-a-json-document)).
- **Why.** On `next`, a Rust program needs a plan for any JSON query, and
needs the dynamic interface for element match, predicates, and children.
Go already has typed methods for these.
- **Done already.** The input types and their parsing rules, as private
types in `stack_encrypt::dynamic::json_query`. The EQL query types
exist in `eql-bindings`. Typed containment and comparison at a path
went through the chain (`op::Contains`, `op::AtPath`), which #1666
removes, so until this item is built every JSON query from Rust goes
through a query plan.
- **Tracked in.** Issue #1459 is open. It is exactly this item. Issue #1677 is open for opening one child. Issue #1486 is closed.
- **Done** in the pull request for #1459. Each method builds the input
the dynamic parser reads into (`json::inputs`, out of the `dynamic`
feature), and returns the field type's query type (`JsonQueries`):
the EQL types for `JsonTreeDocument`. A field's options are its
`#[stash(json(ordered = [..]))]`, decided on 16 November 2026. A test
holds every query to the data plan's operand, and the [JSON query
example](#query-a-json-document) runs.
- **Not done.** Typed paths checked at compile time, which #1459 asks
for: a path is text, checked before any key is fetched. Values are
typed (a number or a string for an ordering), and an element predicate
needs a condition to compile.
- **Tracked in.** Issue #1459 is open until its pull request merges. Issue #1677 is open for opening one child. Issue #1486 is closed.
- **Finished when** these are both true:
- the [JSON query example](#query-a-json-document) compiles;
- an order query on a path the options do not order is refused before
Expand Down Expand Up @@ -1605,8 +1622,7 @@ review.
indexes, a nullable field and a record field their operator methods,
and a field its query type (`#[stash(query = ..)]`). The chain query
`cipher.query(op, value).on(&plan)` is removed (#1662). It was the only
typed JSON query, so until the JSON marker methods exist (#1459) a JSON
query goes through a query plan.
typed JSON query; a document field's marker methods replace it (#1459).
- **No hand-built plan.** The review of 7 November 2026 gave a
hand-built plan the same markers, through `Plan::<Member>::scope(..)`
with `pick!`. That is withdrawn. No Rust program builds a plan by hand.
Expand Down
19 changes: 19 additions & 0 deletions languages/golang/encrypt/guest/src/status.rs
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,10 @@ pub fn status_for_error(error: &stack_encrypt::Error) -> u32 {
// children asked for as a scalar, or an altered entry. The client
// cannot tell which, so neither STATUS_AUTH nor a claim of either.
stack_encrypt::Error::JsonNotALeaf => STATUS_ENCODING,
// A refusal of the caller's input, reported through the crate's
// error by a path whose failure can only be that: the dynamic
// refusal it holds is classified as the dynamic path classifies it.
stack_encrypt::Error::Input(refusal) => status_for_dynamic(refusal),
_ => STATUS_INTERNAL,
}
}
Expand Down Expand Up @@ -422,6 +426,21 @@ mod tests {
);
}

/// A refusal of the caller's input that arrives in the crate's error is
/// classified as the dynamic path classifies it.
#[test]
fn an_input_refusal_is_the_callers_input() {
use stack_encrypt::refusal::{Error, Reason};
let refused = Error::Source {
field: None,
reason: Reason::JsonPathNoWildcard,
};
assert_eq!(
status_for_error(&stack_encrypt::Error::Input(Box::new(refused))),
STATUS_ENCODING
);
}

#[test]
fn a_context_mismatch_is_its_own_status() {
use stack_encrypt::Describe;
Expand Down
16 changes: 14 additions & 2 deletions packages/eql/crates/eql-bindings/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **A `JsonTreeDocument` field is queried from its marker** (#1459). As a
field of a derived record, a `JsonTreeDocument` (and an
`Option<JsonTreeDocument>`) has Stack Encrypt's JSON marker queries,
which return the EQL query types a parameter binds as:
`EncryptedCustomer::PROFILE.contains(doc)` gives a `JsonTreeQuery`,
`at(path).gt(v)` a `JsonTreeLeafQuery`, `elements(path)..` a
`JsonTreeElementsQuery`, `predicate(text)` a `JsonTreePredicateQuery`,
`at(path).selector()` a `JsonTreeSelector` and `children(path)` a
`JsonTreeToken`. A field marked `#[stash(json(ordered = ["$.age"]))]` is
sealed under those options, so its orderings compare. Each operand is
the one a data plan naming `JsonSearch` derives for the same column and
options. `JsonTreeDocument` implements `JsonTarget` and `JsonQueries`.

- **The parts of a `JsonSearch` document decrypt through the owned API**
(#1660). `JsonTreeExtracted`, the entry `->` returns, implements
`DecryptSource<JsonPlaintext>`. `JsonTreeSubtree`, the rows of a subtree
Expand Down Expand Up @@ -333,8 +346,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
gone.
- A JSON document is a `JsonPlaintext`, not a `serde_json::Value`.
`JsonTreeDocument`, `JsonTreeExtracted` and `JsonTreeSubtree` open to
it. A typed JSON query has no replacement yet: derive the operand
through a data plan that names `JsonSearch` (`EqlTargets`).
it. A typed JSON query is a document field's marker query (above).
- `JsonTree` no longer implements `stack_encrypt::target::Layout`, which
is removed. `JsonTree::encryption()`, `index()` and `specs()` are its
own methods, and `index()` returns the index by value.
Expand Down
66 changes: 63 additions & 3 deletions packages/eql/crates/eql-bindings/src/v4/json_tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,13 @@
//! [`EncryptTarget`] / [`DecryptSource`] impls that let a Rust program name
//! [`JsonTreeDocument`] as a target:
//! `document.encrypt_into_with_context(&keyset, column)` under an
//! [`Identifier`]. A query derives through a data plan that names the type
//! [`Identifier`]. As a field of a derived record, its marker has the
//! JSON queries (`contains`, `at`, `elements`, `predicate`, `children`),
//! which return the query types here
//! ([`JsonQueries`](stack_encrypt::json::JsonQueries)), under the options
//! the field's `#[stash(json(..))]` names
//! ([`JsonTarget`](stack_encrypt::json::JsonTarget)). A document with
//! no record is queried through a data plan that names the type
//! (`JsonSearch`) under the same column, and its operand converts into the
//! operand types here (see [`crate::encryption`]). The
//! design is `docs/plans/2026-10-17-json-tree-index.md` at
Expand All @@ -22,8 +28,9 @@ use schemars::{schema_for, JsonSchema, Schema};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use stack_encrypt::json::{
JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonPlaintext, JsonQuery,
JsonSelector, JsonSubtree, JsonSubtreeKey,
JsonDocument, JsonElementsQuery, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonPlaintext,
JsonPredicateQuery, JsonQueries, JsonQuery, JsonSelector, JsonSubtree, JsonSubtreeKey,
JsonTarget,
};
use stack_encrypt::target::{
json_document, AeadContext, Borrowed, DecryptSource, EncryptTarget, ExpectedContext, Index,
Expand Down Expand Up @@ -559,6 +566,59 @@ impl EncryptTarget<JsonPlaintext> for JsonTreeDocument {
}
}

/// A document field sealed under the options its record's
/// `#[stash(json(..))]` names, which order the paths its queries compare.
impl JsonTarget<JsonPlaintext> for JsonTreeDocument {
fn json_encryption<'s, K: 'static>(
options: JsonOptions,
) -> Encryption<'s, JsonPlaintext, Self, K, Self::ContextInput>
where
JsonPlaintext: 's,
{
JsonTree::new(options)
.sealing()
.project(JsonPlaintext::value)
}
}

/// A document field's marker queries return the EQL query types a
/// parameter binds as: `JsonTreeQuery` for `@>` and `<@`
/// (`eql_v4.query_json`), `JsonTreeLeafQuery` for an ordering at a path
/// (`eql_v4.query_json_leaf`), `JsonTreeElementsQuery` for
/// `eql_v4.json_tree_element_match`, `JsonTreePredicateQuery` for `@@`, the
/// selector `->` takes, and the token the child functions take.
impl JsonQueries for JsonTreeDocument {
type Query = JsonTreeQuery;
type LeafQuery = JsonTreeLeafQuery;
type ElementsQuery = JsonTreeElementsQuery;
type PredicateQuery = JsonTreePredicateQuery;
type Selector = JsonTreeSelector;
type Token = JsonTreeToken;
fn query(query: JsonQuery) -> Result<JsonTreeQuery, Error> {
JsonTreeQuery::try_from(query)
}
fn leaf_query(query: JsonLeafQuery) -> Result<JsonTreeLeafQuery, Error> {
JsonTreeLeafQuery::try_from(&query)
}
fn elements_query(query: JsonElementsQuery) -> Result<JsonTreeElementsQuery, Error> {
transcode(&query)
}
fn predicate_query(query: JsonPredicateQuery) -> Result<JsonTreePredicateQuery, Error> {
transcode(&query)
}
fn selector(selector: JsonSelector) -> Result<JsonTreeSelector, Error> {
transcode(&selector)
}
fn token(key: JsonSubtreeKey) -> Result<JsonTreeToken, Error> {
match key.tokens().as_slice() {
[token] => transcode(token),
_ => Err(Error::Other(
"a path with no wildcard names one node".into(),
)),
}
}
}

/// Decrypts to the plaintext document, under the stored column.
///
/// - `decrypt_from_with_context(document, &keyset, column)` checks that the
Expand Down
Loading
Loading