Repository navigation
feat(platform)!: compare typed arrays in refersTo where entries (PV14) - #5388
Conversation
A refersTo `where` entry may now compare a referenced typed array as a list:
an equality between two typed arrays (same elements, same order),
`{ "contains": b }` (a referring value of the element kind, or "$ownerId" as a
writer gate on identifier elements) and `{ "containsAll": b }` (every element
of a referring typed array). Elements compare by value, and a list a document
leaves out is the empty one.
WhereCondition gains the appended Contains and ContainsAll variants and the
new WhereContainsBound. Meta-schema v3 and parser generation 3 read the keys;
data contract reference validation 0 admits a pair of stored typed arrays of
one non-float element kind and checks the conditions (40126); document
reference validation 0 judges them against the document found or a moderated
document's removal record (40127). A pair of lists never keys a preallocated
or summableOffCountIndex index. All in place under protocol version 14.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to No actionable issue was established for the typed-array reference conditions; the change is mergeable after normal checks. Pre-merge checks ·
|
|
Waiting for bot review — coderabbitai ✓ · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
📖 Book Preview built successfully. Download the preview from the workflow artifacts. Updated at 2026-10-12T19:18:55.524Z |
|
🔍 Review in progress — actively reviewing now (commit 8296210) · triage: critical |
Review follow-ups for refersTo `where` over typed arrays (PV14): - List equality, `contains` and `containsAll` compare elements with `Value::same_scalar_data`, as a scalar equality does. `containsAll` keeps its set lookup through an order that mirrors that comparison for every value but a float, which it compares one by one. - A `where` bound whose referring path is no object on the way is refused as a mismatch, as an equality's unreadable side is, instead of reading as absent (a `containsAll` held on it). - `Index::preallocation_bindings` and `Index::count_index_derivations` bind no typed array index property, replacing the registration-only guard. - The unreachable list arm in registration is a corrupted-code error, the parse comment names the moderated `contains` writer gate, and the moderation test helpers are shared from `tests.rs`. - Tests pin the meta-schema refusal, the stored read-back and the decoded action data of the by-value write; js-evo-sdk README lists the new shapes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
The typed-array comparison feature is gated to unreleased PV14, but its new validation work is not metered, and containsAll retains an attacker-reachable quadratic float fallback. One blocking finding and one test-coverage suggestion remain after verification and deduplication. This was a static review of the exact head; the supplied CI snapshot still had Rust workspace tests, JS and Docker builds, and Kotlin build/tests pending.
🔴 1 blocking | 🟡 1 suggestion(s)
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: ffi-engineer); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 6: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 7: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); reviewer 8: muse-spark-1.3-contributor (agent: phase1-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
criticalbygpt-6.1-sol(effort low) — This is a large, intricate change to consensus-critical document-reference validation and peer-facing contract/schema deserialization, including new protocol-versioned rules and serialized enum variants. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer,muse-spark-1.3-contributor— rust-quality (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity below 15% reserve: weekly 14% left, 5h 100% left),glm-5.3-flash(not used above high effort; tier asks max) - Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— architecture-layering (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— ffi-engineer (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— platform-versioning (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— security-auditor (completed, effort xhigh); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-dpp/src/data_contract/document_type/property/reference_where.rs`:
- [BLOCKING] packages/rs-dpp/src/data_contract/document_type/property/reference_where.rs:563-579: Meter typed-array where comparisons before executing them
The new list comparisons run during document-reference state validation without recording their processing work. The state-document pair/condition loops and validate_pairs_against_removal call where_lists_equal or WhereCondition::holds without adding comparison-cost operations; StateTransitionExecutionContext::fee_cost only sums recorded operations. Existing read and hash charges do not account for repeated list scans or BTreeSet construction, so additional where entries and references multiply unmetered validator CPU.
The float fallback also defeats the intended near-linear containsAll implementation. Refusing float element schemas at registration does not exclude Value::Float(1.0) carried under an integer schema: validate_document_properties_v1 converts it to JSON, and the pinned jsonschema integer validator accepts numbers whose fractional part is zero. The create transformer preserves those original values. A held integer list of 1,023 zeros followed by one, compared against 1,024 copies of Float(1.0) with uniqueItems disabled, therefore performs 1,048,576 scalar comparisons. The field-size check examines individual scalar values rather than the aggregate array, so it does not exclude this example. The sponsor-dependent CheckTx path executes state validation before dry-run fee validation; a later condition mismatch rejects the transaction without committing fees or nonces, allowing this work to be repeated even if integer storage encoding would subsequently reject the floats.
Add deterministic PV14 processing-cost accounting for equality, contains, and containsAll in both live-document and removal-record validation. Also replace the per-float full-list scan with a numeric lookup preserving same_scalar_data's float-bit semantics, so accounting alone does not leave the unpaid CheckTx rejection path quadratic. Cover the integer-schema/float-bound case through sponsored CheckTx.
- [SUGGESTION] packages/rs-dpp/src/data_contract/document_type/property/reference_where.rs:974-995: Cover the remaining list-element normalization variants
The cross-product test is the regression guard for ListElement matching same_scalar_data, but its sample omits Bytes20, Bytes32, Bytes36, and EnumString, all of which have explicit normalization arms in the new implementation. In particular, the fixed-size byte variants should be checked against Bytes, Identifier, and U8 arrays, rather than only against themselves. Add these variants and representatives of the remaining signed and unsigned integer widths to the sample so changes to those arms cannot silently alter containsAll relative to scalar equality.
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them. |
Review follow-ups for refersTo `where` over typed arrays (PV14): - Document reference validation bills the list elements a `where` entry comparing typed arrays reads before it compares them, against a document in state and against a moderated document's removal record: `where_list_credits_per_element` (100 credits) of the PV14-only `document_reference` fee group, added in place to `FEE_DOCUMENT_REFERENCE_VERSION1`. - `containsAll` looks a wanted float up among the `f64` bits of every held number, and a wanted integer among those of every held float, instead of scanning the list for each float: a list of whole floats carried for an integer list stays near-linear, with `same_scalar_data`'s semantics kept. - Tests: the billed elements for each list form, an integer list carried as 1,024 floats and as `-0.0`, and the scalar cross-check over every value variant (fixed-size byte variants, `EnumString`, each integer width, NaN), with the whole sample held at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
…-where #5388 and #5390 are merged (squashed, with their review fixes). Resolved as v5.0-dev plus this branch's own changes: the export list keeps where_list_elements beside where_lists_elements_read, note 94 keeps the element billing and points at 95, note 92 now says preallocated and counter array indexes follow 95, and the book keeps both the commitment billing and the per-element keying.
Basic explanation
A
refersToreference can say, withwhere, that the document it points at must hold certain values: "the post this like points at must have the same hashtag as the like". Until now that only worked for single values. A list property (a typed array, such as a post'stags) was refused on either side.This PR lets
wherecompare lists, in three ways:"tags": "tags"holds when the like'stagsare exactly the post'stags, same elements, same order.contains:"tags": { "contains": "tag" }holds when the post'stagsinclude the document'stag. With"$ownerId"it becomes a writer gate:"members": { "contains": "$ownerId" }means only someone the post lists may write.containsAll:"tags": { "containsAll": "tags" }holds when the post'stagsinclude every one of the document'stags, in any order.A list a document leaves out counts as an empty list. Elements are compared by value, so an identifier equals its 32 raw bytes and an integer is the same at any width. Protocol version 14 only, which has not shipped.
Before and after
Yappr is moving a post's single
hashtagstring totags, a typed array (maxItems4,uniqueItems,immutable, listed inmoderatorAbilities.deleteKeepsFields). Its indexOnlylikebinds to the post's values throughwhere.Before, with a single hashtag (registers today):
Moving the post to
tagsand the like with it:ReferencedDocumentPropertyAgreementInvalidError(40126), "agreement properties must be single values, not typed arrays". The like has nothing to bind to.tagsare the post'stags, in order (40127 when they are not).The two conditions, before (refused by meta-schema v3, 10101, and by the parser, 10231) and after (registered):
Issue being fixed or feature implemented
Requested by pasta for Yappr's move from
hashtagtotags. The request has three parts; this PR is thewherepart only:whereover typed arrays: this PR.preallocated/summableOffCountIndexthrough an array equality: later work, on top of both. This PR keeps both refusing a pair of lists (see below).What was done?
Types (
rs-dpp,property/reference_where.rs)WhereConditiongains two appended variants,Contains(WhereContainsBound)andContainsAll(String);WhereContainsBoundis new and@append_only(Property(path),Writer). Earlier variants keep their encoding.where_lists_equalcompares two typed array values as lists, each element pair withValue::same_scalar_data, exactly as a scalar equality compares two values: an integer at any width, bytes whichever variant holds them (an identifier,Bytes, an array ofU8s), a string, a boolean.containscompares the same way;containsAlltoo.containsAlllooks each element up in aBTreeSetof the referenced elements, through an order (ListElement) that mirrorssame_scalar_datafor every value but a float; it looks a wanted float up among thef64bits of every held number (and a wanted integer among those of every held float), which is exactly howsame_scalar_datacompares a float, so no input, whole floats carried for an integer list included, makes it quadratic. Both lists can hold up tomax_typed_array_items(1024) elements, so a pairwise scan would be up to 1M comparisons per entry, per element reference, all unbilled CPU on block execution. A test checks all three forms againstsame_scalar_dataon a sample of values, floats,-0.0and integers past 2^53 included.property_agreement, keyed as today.Parse and meta-schema (
try_from_schema, meta-schema v3)parse_where_conditionreadscontains(a referring property path or"$ownerId") andcontainsAll(a referring property path). Both are refused on a referenced$ownerId,$creatorIdor$id, andcontainsAllof the writer is refused. Error messages listing the keys now name both.referenceWhereConditionin meta-schema v3 admits both keys with the same path patterns as the other bounds.Registration (
data_contract_reference_validationv0), all 40126transient), the elements share one value kind (same_value_kindonitems) and are not floats. A typed array paired with anything else is refused.contains: the referenced side is a typed array of non-float elements; the bound is a property of the element kind, or"$ownerId"for identifier elements, never the reference property itself.containsAll: the bound is a stored typed array of the same element kind, never the reference property itself.moderatedDocumentreference, acontainsof the writer is asked about on every replace, so its list must be kept by every removal record (deleteKeepsFields), the rule orderings bounded by$updatedAtalready follow. A rule reading through the reference still keeps every entry to kept fields.const,in,notEqualand the orderings still refuse a typed array.summableOffCountIndexindex is keyed through a pair of lists:Index::preallocation_bindingsandIndex::count_index_derivationsbind no index property that is a typed array (a list is no tree key). Both index kinds live only on indexOnly types, which index no typed array today, so this is inert now and keeps them refusing once indexOnly types can index typed arrays, whether the contract is registered or read back. The indexOnly typed-array index refusal is unchanged.Write time (
document_reference_validationv0)where_lists_equal, against the document in state and against a moderated document's removal record (a kept field the document held none of is the empty list). Scalar pairs are compared as before.WhereCondition::holdsevaluatescontainsandcontainsAll; mismatches are 40127 with the condition as written ({ "containsAll": "tags" }).containsAll, the referenced list for acontains, atwhere_list_credits_per_element(100 credits, about a BLAKE3 block's work), a field added in place toFEE_DOCUMENT_REFERENCE_VERSION1, the PV14-onlydocument_referencefee group every schedule carries. Two lists of 1,024 elements bill 204,800 credits.containsAllwould otherwise read it as an empty list and hold.containsof"$ownerId"is re-checked on every replace, like any writer gate; the others when their referring property changes.Other
wasm-dpp2:documentTypeReferencesrenderscontains/containsAllas the schema writes them; TS types updated.ParsedContract) do not readwhere, so nothing changes there.refers-to.mdgets a "Typed arrays" subsection, and the registration, replace and error tables are updated;typed-arrays.mdno longer says typed arrays cannot appear inwhere.v14.rs: protocol version 14 note 94.How Has This Been Tested?
Targeted runs only:
cargo test -p dpp --all-features --lib -- reference_where typed_array_where: 19 passed. List equality (order, absence, by-value elements, identifier vs bytes vsU8array, integer widths,U128pasti128::MAX),containsandcontainsAll(vacuous empty, absent bound), re-check rules, display, and parse and meta-schema accept and refuse cases.cargo test -p dpp --all-features --lib -- reference where list_element choose_first commit_reveal typed_array: 311 passed.process_raw_state_transitions: accepted fixture (equality,contains,containsof the writer,containsAll); refused element kind mismatch, list vs scalar, float elements, transient referring list,containsof another kind, of a non-list, of the reference property, of a missing property, writer on non-identifiers,containsAllof a scalar or of another kind,const/notEqualon a list; moderatedcontainsof the writer on an unkept list refused and on a kept one accepted. The oldshould_reject_agreement_on_typed_array_propertiesnow asserts the fixture registers.containsincl. the writer gate,containsAllincl. vacuous empty, a changed list re-checked on replace.BytesandU64elements as carried.summableOffCountIndexderivation through a pair of typed arrays.Float(1.0)holds against stored[0, 1], and-0.0does not equal 0.permanent_document_reference_declarations,typed_array_references,agreement,referenced_rules,moderated_document_reference,list_element_reference,preallocated,lookup_reference,owner_reference,reference_expression,deletable_document_reference,commit_reveal): 240 passed.wasm-dpp2: built,DocumentPropertyReference.spec.ts46 passing (2 new). ESLint could not start in this worktree (an unplugged Yarn package is missing without a full install).cargo clippy -p dpp --all-features --all-targetsandcargo clippy -p drive-abci -p wasm-dpp2 --all-targets,-D warnings: clean.cargo fmt --all.In-place changes to shipped generations
The validation generations below are selected only by protocol version 14, which is unreleased, and are extended in place under their existing gate. The last two rows touch code shipped versions also carry, where the change is unreachable before protocol version 14:
parse_where_condition)refersTo.where.FEE_DOCUMENT_REFERENCE_VERSION1gainswhere_list_credits_per_elementFEE_VERSION1toFEE_VERSION3, read by PV14 onlyfee_version_number, so no stored format changes.Index::preallocation_bindingsandIndex::count_index_derivationsbind no typed arrayEvery contract valid before this PR registers and validates the same way: the only shape that newly registers is a pair of typed arrays (refused with 40126 before), and
contains/containsAllwere refused by the meta-schema and the parser. Scalar pairs and the existing conditions take the same code path as before.Breaking Changes
Consensus change at protocol version 14 (unreleased): contracts with the new
whereshapes register, and writes are judged against them. Appended variants onWhereConditionand the newWhereContainsBoundchange nothing for existing encodings.Overlap with other work
chooseFirst) are merged and already in the base; no conflict.document_reference_validation/v0/mod.rs, and feat(platform)!: dpns contract v3 checks name registration with schema keywords (PV15) #5337 also meta-schema v3: textual conflicts possible, no semantic overlap.summableOffCountIndexindexes from being keyed through a list pair until the follow-up that supports it.v14.rsmay need renumbering if another PR claims it first.Possible follow-ups
containsAny(the name of the feat(sdk): query the elements of typed arrays over the wire (PV14) #5372 query operator): not needed by Yappr, left out.{ "contains": { "const": ... } }, a literal element.preallocated/summableOffCountIndexthrough a list equality (the third part of the request).Checklist:
structure.rs, regeneratedgrovedb-structure.json, and checked the structure viewer link posted on this pull requestFor repository code-owners and collaborators only
🤖 Generated with Claude Code
PR Hygiene ·
8296210/skip-botsproceeds without the ones not yet reported/self-reviewedjs-wasm-sdk(packages/js-evo-sdk/README.md) — shumkovdpp— you own itrs-drive-abci— you own itWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.Summary by CodeRabbit
containsto check for a matching element andcontainsAllto check that every referring-array element appears in the referenced array, regardless of order.