Skip to content

(Calculated fields) Add calculated-field predicates to QueryAst - #6777

Open
fulmicoton wants to merge 3 commits into
mainfrom
paul.masurel/calc-fields-integration
Open

(Calculated fields) Add calculated-field predicates to QueryAst#6777
fulmicoton wants to merge 3 commits into
mainfrom
paul.masurel/calc-fields-integration

Conversation

@fulmicoton

Copy link
Copy Markdown
Collaborator

Description

Add calculated-field predicates to Quickwit's native query AST and search preparation.

  • Add QueryAst::CalcField(CalcFieldQuery) with a jitexpr expression serialized as a string, including visitor/transformer support and user-query parsing passthrough.
  • Lower predicates directly to Tantivy's JitExprPredicate / DocPredicateQuery, keeping JIT compilation and execution in Tantivy.
  • Collect referenced fast fields for warmup, including string dictionaries through the existing whole-column warmup path. Preserve exact field paths and skip cache-hit subtrees.
  • Treat calculated predicates conservatively during tag pruning and retain required-term behavior for ordinary query clauses.
  • Enable Tantivy's jitexpr feature and pin f1413a4dcfd1386a22df7d960a754b4bcfb1c3b4 (the tip of paul.masurel/jitexpr-docpredicate), including support for . and # in expression variable names. Update dependency licenses.
  • Add focused tests and internal documentation. AST containers retain PartialEq but no longer implement Eq, matching jitexpr's expression type.

Example:

{
  "type": "calc_field",
  "expression": "(GT custom.duration 1i64)"
}

How was this PR tested?

  • cargo nextest run -p quickwit-query --locked: 172 passed, including 12 CalcField tests.
  • cargo nextest run -p quickwit-doc-mapper -E 'test(query_builder::) | test(tag_pruning::)' --locked: 28 passed.
  • Formatting, license generation, and diff whitespace checks passed during implementation.
  • The upstream parser change was tested with the jitexpr and Tantivy document-predicate suites before pushing it.

Draft follow-ups / known limitations

  • cargo check --locked still fails at quickwit-directories/src/hot_directory.rs:462: the pinned Tantivy IndexMeta does not expose list_segment_files.
  • A prior full doc-mapper test run hit the order-sensitive test_concatenate_multiple_field test with default features. A diagnostic run with serde_json/preserve_order passed; no test or feature workaround was applied.
  • Add a full remote-storage root-to-leaf regression test.
  • Calculated-field definitions, aggregation expressions, and Elasticsearch DSL syntax are outside this PR. Expression variables currently refer directly to fast fields.

@fulmicoton fulmicoton changed the title Add calculated-field predicates to QueryAst (Calculated fields) Add calculated-field predicates to QueryAst Sep 7, 2026
@fulmicoton-dd
fulmicoton-dd force-pushed the paul.masurel/calc-fields-integration branch 2 times, most recently from 5e6e6c7 to 7695065 Compare September 7, 2026 20:40
@fulmicoton
fulmicoton force-pushed the paul.masurel/calc-fields-integration branch from 7695065 to 2187a7a Compare September 8, 2026 08:40
@fulmicoton
fulmicoton requested a review from PSeitz September 8, 2026 09:12
@fulmicoton
fulmicoton marked this pull request as ready for review September 8, 2026 09:12
@fulmicoton
fulmicoton requested a review from a team as a code owner September 8, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants