feat(compiler): add opt-in RakuAST queries and isolated SQL plan caching

.github/workflows/matrix.yaml

@@ -37,6 +37,40 @@ jobs:
       - name: Run tests
         run: prove6 -lj8 t
 
+  rakuast:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    services:
+      postgres:
+        image: postgres:15
+        env:
+          POSTGRES_PASSWORD: red-ci-test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd="pg_isready -U postgres"
+          --health-interval=5s
+          --health-timeout=5s
+          --health-retries=20
+    steps:
+      - uses: actions/checkout@v4
+      - uses: Raku/setup-raku@v1
+        with:
+          raku-version: "2026.09"
+      - name: Install dependencies with the legacy frontend
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libpq-dev
+          zef install --/test --test-depends --deps-only .
+      - name: Test the optional RakuAST compiler
+        env:
+          RAKUDO_RAKUAST: "1"
+          RED_DATABASE: "Pg host=127.0.0.1 port=5432 dbname=postgres user=postgres password=red-ci-test"
+        run: |
+          raku -Ilib t/37-json.rakutest
+          raku -Ilib t/92-rakuast-compiler.rakutest
+          raku -Ilib t/93-rakuast-scope.rakutest
+
   mysql:
     name: MySQL (RakuAST=${{ matrix.rakuast }})
     strategy:

Changes

@@ -14,6 +14,14 @@ Revision history for Red
     - Retain formatter coercions, repeated separators, Unicode and embedded control characters.
     - Read generated projection getters from the receiving storage instance instead of a ResultSeq.
     - Cover independent columns, instances and updated values without issuing database queries.
+    - Preserve arithmetic grouping and parameter order in casts and SQLite JSON array paths.
+    - Add opt-in CHECK-time RakuAST query compilation with symbolic fallback and private captures.
+    - Specialize query factories and preserve argument captures, writable returns and comparison coercions.
+    - Add opt-in fallback diagnostics through :warn, $*RED-COMPILER-WARN and RED_COMPILER_WARN.
+    - Cache SQL plans and scalar predicate shapes without sharing connections, cursors or results.
+    - Bound cache growth and retain parameters and comments through uncached and cyclic values.
+    - Use portable object identity for plan traversal and predicate cache keys.
+    - Add compiler, cache, binding-isolation tests and query construction benchmarks.
     - schema: add named CHECK and PostgreSQL EXCLUDE constraints with driver selection
     - indexes: support access methods, per-key operator classes and partial predicates
     - pg: add range constructors and native range/regex expression helpers

META6.json

@@ -39,6 +39,7 @@
     "Red::AST::AddForeignKeyOnTable": "lib/Red/AST/AddForeignKeyOnTable.rakumod",
     "Red::AST::BeginTransaction": "lib/Red/AST/BeginTransaction.rakumod",
     "Red::AST::Between": "lib/Red/AST/Between.rakumod",
+    "Red::AST::BoundFilter": "lib/Red/AST/BoundFilter.rakumod",
     "Red::AST::Case": "lib/Red/AST/Case.rakumod",
     "Red::AST::Chained": "lib/Red/AST/Chained.rakumod",
     "Red::AST::ChangeColumn": "lib/Red/AST/ChangeColumn.rakumod",
@@ -102,6 +103,10 @@
     "Red::Cli::Table": "lib/Red/Cli/Table.rakumod",
     "Red::Column": "lib/Red/Column.rakumod",
     "Red::ColumnMethods": "lib/Red/ColumnMethods.rakumod",
+    "Red::Compiler::Emitter": "lib/Red/Compiler/Emitter.rakumod",
+    "Red::Compiler::RakuAST": "lib/Red/Compiler/RakuAST.rakumod",
+    "Red::Compiler::Runtime": "lib/Red/Compiler/Runtime.rakumod",
+    "Red::Compiler::Template": "lib/Red/Compiler/Template.rakumod",
     "Red::Config": "lib/Red/Config.rakumod",
     "Red::DB": "lib/Red/DB.rakumod",
     "Red::Database": "lib/Red/Database.rakumod",
@@ -134,6 +139,8 @@
     "Red::Phaser": "lib/Red/Phaser.rakumod",
     "Red::PrepareCode": "lib/Red/PrepareCode.rakumod",
     "Red::Projection": "lib/Red/Projection.rakumod",
+    "Red::QueryPlan": "lib/Red/QueryPlan.rakumod",
+    "Red::QueryPlan::Cache": "lib/Red/QueryPlan/Cache.rakumod",
     "Red::ResultAssociative": "lib/Red/ResultAssociative.rakumod",
     "Red::ResultSeq": "lib/Red/ResultSeq.rakumod",
     "Red::ResultSeq::Iterator": "lib/Red/ResultSeq/Iterator.rakumod",

docs/index.md

@@ -15,6 +15,7 @@ If you are looking for tutorials, you can visit:
 * [Special Variables](tutorials/vars.md)
 * [Red Cookbook](tutorials/cookbook.md)
 * [Experimental](tutorials/experimental.md)
+* [Optional RakuAST query compiler](tutorials/rakuast.md)
 * [Examples](https://github.com/FCO/Red/tree/master/examples)
 
 #### API documentation

docs/tutorials/rakuast.md

@@ -0,0 +1,193 @@
+# Optional RakuAST query compiler
+
+The optional compiler translates supported query callbacks during `CHECK`.
+Enable the RakuAST frontend and explicitly opt in for each compilation unit:
+
+```raku
+use Red:api<2>;
+use Red::Compiler::RakuAST;
+CHECK red-compile;
+
+my $minimum = 10;
+my $query = Product.^all
+    .map({ .price + 2 })
+    .grep({ $_ > $minimum })
+    .map({ "price: $_" });
+```
+
+Run the program with `RAKUDO_RAKUAST=1 raku -Ilib program.raku`.
+This integration is experimental and tested with Rakudo 2026.09. Loading Red
+without opting in keeps the existing frontend and callback behavior.
+
+The compiler recognizes literal blocks, pointy blocks (including tuple
+parameters), placeholder parameters, and single-Whatever expressions passed to `map`, `grep`, `first`,
+and `sort`. Supported expressions include model columns, constant indexes and
+Hash keys, dynamic JSON indexes and key existence, scalar and container projections, arithmetic, comparisons, string
+concatenation and interpolation, complete conditionals, and `defined`,
+`contains`, `elems`, `starts-with`, `ends-with`, `substr`, `index`, `lc`, `uc`,
+and `cast`. Membership predicates accept literal word lists, captured arrays and
+subqueries. Hash projections accept both fat arrows and colonpairs. Boolean operators are translated in predicates only.
+Comparisons retain the resolved Red operator, including its numeric or textual
+coercions and SQL casts.
+
+Templates bind against the current projection, so SQL aliases and SELECT
+boundaries remain the responsibility of the existing query planner. Scalar
+lexical captures are read when the query is constructed and retained if the
+planner needs to bind the template against another SELECT layer.
+
+During CHECK the compiler specializes factories that construct `Red::AST` nodes
+directly, along with the required validators. Runtime query construction invokes
+these factories rather than interpreting the expression plan. Factories are
+shared across calls; captured values and SQL subjects remain private to each
+query. CHECK time is separate from runtime query construction and execution.
+
+Map templates also record column and JSON assignments, arithmetic compound
+assignments (`+=`, `-=`, `*=`, `/=`, `%=` and `~=`), increments, decrements and
+JSON deletion. These effects use the existing update pipeline and are persisted
+with `.save`. Updates are buffered until the template evaluates successfully.
+Simple untyped scalar locals and sequential statements can be compiled too.
+Compound assignment to a projected scalar or a local variable changes the
+expression used by later statements; it does not turn that projection into a
+database update. Postfix increment/decrement retains its original return value.
+
+Expressions independent of the query element, including constructors and function
+calls, can be bound as runtime captures. Their code runs during query construction,
+never during CHECK. Ordinary native receivers still execute their original callback.
+
+An unsupported callback is left on the existing symbolic evaluation path,
+including its existing fallback policy. Named callbacks, arbitrary functions of
+the query element, loops, phasers, constrained signatures, custom operators and
+conditional updates are not compiled. Unsupported syntax is rejected as a complete
+callback before any updates are submitted.
+Translation never executes the original callback during CHECK.
+
+Instrumented calls check their receiver at runtime. A `ResultSeq` receives an
+internal template alongside the original callback. Other receivers receive the
+original arguments, preserving native dispatch, adverbs, and writable results.
+The compiler does not add SQL semantics to ordinary Arrays or Seqs.
+
+To warn whenever an executed Red callback falls back to symbolic evaluation:
+
+```raku
+CHECK red-compile(:warn);
+```
+
+Warnings include the operation, callback location, source and fallback reason.
+Unsupported syntax is diagnosed when that Red query is constructed, so native
+Array/Seq calls and callbacks that never run do not warn. Runtime template
+validation failures are reported too, once per query stage even if a SELECT
+boundary requires rebinding. The fallback still uses the original callback.
+
+You can also enable warnings dynamically or through the environment:
+
+```raku
+my $*RED-COMPILER-WARN = True;
+CHECK red-compile;
+```
+
+```sh
+RED_COMPILER_WARN=1 RAKUDO_RAKUAST=1 raku -Ilib program.raku
+```
+
+Precedence is the explicit `:warn`/`:!warn` parameter, then a defined Boolean
+`$*RED-COMPILER-WARN`, then `RED_COMPILER_WARN`. An explicit `False` disables
+warnings even when the environment enables them. Environment values `1`, `true`,
+`yes` and `on` enable warnings, ignoring case and surrounding whitespace; other
+values disable them. The default is silent.
+
+The parameter applies to instrumented calls in the current compilation unit;
+dynamic and environment settings also cover uncompiled Red callbacks elsewhere.
+If warnings are enabled only at runtime, an uninstrumented callback reports
+`No RakuAST template was supplied` with its location, instead of the precise
+CHECK-time rejection reason. Set the environment before compilation or use
+`:warn` to retain those rejection details. SQL plan-cache misses/bypasses are not
+compilation failures, and database errors retain their normal exception behavior.
+
+For compilation diagnostics, pass a reporting callback:
+
+```raku
+CHECK red-compile(:report(-> :$method, :$source, :$status, :$reason {
+    note "$method: $status ($reason)";
+}));
+```
+
+Reports distinguish translated callbacks from unsupported syntax and include the
+original callback source. A compiled template can still be inapplicable to a
+particular runtime projection. To observe successful template evaluation, set
+`$*RED-COMPILER-REPORT` to a callable; it receives `:status<used>`, `:plan`, and
+`:engine<direct>` for generated factories, or `:status<fallback>` and `:reason`.
+Manually constructed templates without a factory report `:engine<interpreted>`.
+This is useful for tests that must distinguish compilation from symbolic fallback.
+
+To measure factory runtime separately from CHECK, database execution, and the
+query planner, run `raku -Ilib tools/benchmark-compiler.raku`. The benchmark emits
+JSON samples comparing interpreted and direct factories in the same process,
+with warmup and alternating measurement order. Factory specialization happens
+before timing. `--iterations=1000` and `--rounds=7` control the sample size.
+These measurements isolate binding and AST construction; they do not represent
+the speedup of an entire query.
+
+## Runtime query-plan cache
+
+A ResultSeq automatically reuses its SQL plan when its state can be fingerprinted
+safely. Opaque mutable values bypass caching. The database
+is still queried each time, using a fresh statement and cursor. Changes to the
+query, driver, pagination or SQL metadata select a different plan. Diagnostic
+comments are attached for the current execution. `.cache` continues to materialize
+query results; it is separate from this plan cache.
+
+The optional compiler also shares predicate trees for direct comparisons between
+a model field and one captured scalar (`Int`, `Str`, `Rat` or `Num`). Each query
+reads its capture once during construction and retains that private snapshot:
+
+```raku
+sub expensive($minimum) {
+    Product.^all.grep({ .price > $minimum })
+}
+my $first = expensive(10);
+my $second = expensive(20);
+```
+
+After warming up, these calls reuse the predicate factory output and SQL while
+keeping their parameters independent. Other callbacks, mutable captures, NULL
+captures and updates retain their existing factory/fallback behavior. Comparisons
+requiring scalar coercion also use ordinary factories. Caches are bounded to 128
+entries, synchronized, and live only in the current process.
+Prepared statements are not shared between connections or iterators. The SELECT
+wrapper and current metadata are still inspected; reusing a predicate does not
+eliminate every AST allocation. Warm-up has a cost, and for small predicates the
+construction cost can remain similar. Plan reuse still hydrates every row, so end-to-end
+gains depend on how much of the workload is spent translating SQL.
+Drivers based on CommonSQL enable plan caching; other drivers retain their
+existing prepare path unless they opt in with `supports-query-plans`.
+
+`$*RED-COMPILER-REPORT` additionally receives `filter-hit`, `filter-miss`,
+`plan-hit`, `plan-miss` and `plan-fallback` events. Plan fallback includes a reason.
+For comparisons, `my $*RED-QUERY-PLAN-CACHE = False` disables both runtime caches.
+Run `raku -Ilib tools/benchmark-query-plans.raku` to compare construction,
+translation/planning (dry run), and execution with caching enabled and disabled.
+The benchmark alternates variants and reports cold-plan and warmed measurements
+in microseconds per operation as JSON. Module compilation and database setup are
+excluded. A separate `prototype-samples` series compares first specialization
+with the saved symbolic AST against the runtime factory, using empty predicate
+caches in both variants.
+For direct field/scalar comparisons, CHECK also constructs a `Red::AST` prototype
+with symbolic field and parameter slots. That object is embedded in the template
+and saved with the module in `.precomp`. Its first use resolves the model column,
+alias and parameter type from the prototype; later calls reuse the resolved
+predicate. Runtime captures, connections, query objects and SQL plans are never
+saved in the module. Other expressions continue to use CHECK-generated factories;
+this does not eliminate the runtime SELECT envelope or every AST allocation.
+
+## Measuring query construction and cache bookkeeping
+
+`raku -Ilib tools/benchmark-query-construction.raku` measures filters, arithmetic
+projections and chained projections. It checks query results before timing construction,
+warms the tasks, and alternates their order. Set `RAKUDO_RAKUAST=1` to include the
+optional CHECK compiler. `--iterations` and `--rounds` control the sample size.
+
+`raku -Ilib tools/benchmark-query-cache.raku` isolates cache lookup costs with 1, 16
+and 128 entries. It measures both a warmed working set and continual eviction, checks
+builder counts, and reports microseconds per lookup. `--iterations` and `--rounds`
+control the sample size. These cache and field-access measurements isolate individual
+operations; use the query-plan benchmark to evaluate their effect on SQL workloads.

lib/Red/AST/BoundFilter.rakumod

@@ -0,0 +1,43 @@
+use Red::AST;
+use Red::AST::Unary;
+use Red::QueryPlan::Cache;
+
+unit module Red::AST::BoundFilter;
+
+#| An immutable parameter position in a shared predicate.
+class Parameter does Red::AST {
+    has Int $.index is required;
+    has Mu:U $.type is required;
+    method returns { $!type }
+    method args { () }
+    method find-column-name { () }
+}
+
+#| A model member resolved only after the query receiver and alias are known.
+class Field does Red::AST {
+    has Str $.name is required;
+    method returns { Any }
+    method args { () }
+    method find-column-name { () }
+}
+
+#| A bind descriptor retained by a SQL plan, never a captured value.
+class Bind {
+    has Int $.group is required;
+    has Int $.index is required;
+}
+
+class Binding {}
+
+#| A private set of captures attached to a shared predicate tree.
+class Predicate does Red::AST {
+    has Binding $.binding-token .= new;
+    has Red::AST $.filter is required;
+    has List $.values is required;
+    has Red::QueryPlan::Cache $.cache is required;
+    multi method WHICH(::?CLASS:D:) { $!binding-token.WHICH }
+    method not { Red::AST::Not.new(self) }
+    method returns { Bool }
+    method args { $!filter }
+    method find-column-name { $!filter.find-column-name }
+}