Skip to content

perf(audit_trail): index a record's history and its life boundaries - #416

Merged
bexchauveto merged 1 commit into
mainfrom
perf/audit-trail-record-history-indexes
Oct 21, 2026
Merged

bexchauveto merged 1 commit into
mainfrom
perf/audit-trail-record-history-indexes

Conversation

@bexchauveto

@bexchauveto bexchauveto commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

life_boundary_of runs on every audit-trail read by a caller with a scope, on a record that still exists. On a large table it can walk the whole table. This adds the indexes it needs and rewrites one condition so Postgres can use them.

Mirror of the Node.js change on ForestAdmin/agent-nodejs#1983 (migration 003 there).

What changes

  • Migration 004-index-record-history:
    • (collection, record_id, timestamp, id), which serves every per-record read;
    • the same columns over create and delete rows only (partial index), for the life boundary lookup;
    • skipped on adapters without partial indexes (supports_partial_index?, false on MySQL/MariaDB), which get the first index only;
    • record_id keeps its 255 prefix on MySQL/MariaDB, as in 001.
  • before_row_condition now uses (timestamp, id) < (?, ?) instead of timestamp < ? OR (timestamp = ? AND id < ?). They're equivalent, but Postgres can't turn the OR into a range on the new index. past_row_condition is unchanged: it sorts id ascending whatever the timestamp direction, which a row value cannot express.

Measurements

Postgres 15, 2.16M rows: 40 collections with integer ids shared across them, plus hot records. Times are the best of 3 runs, warm cache.

record latest boundary (.first) row before it (exists?)
hot, 100k rows, created long ago — before 486 ms: walks (timestamp, id) backwards, 2.1M rows discarded 19 ms
same — after 0.04 ms (partial index) 0.07 ms (index-only scan)
hot, 50k rows, created recently — before 10 ms 13 ms
same — after 0.06 ms 0.06 ms
normal record, before / after 0.2 / 0.03 ms 0.07 / 0.05 ms

With the indexes alone, the OR form still read the record's whole history (7–8 ms); the row-value form brings it to 0.06 ms.

Tests

  • Specs for:
    • both indexes' columns and the partial index's filter;
    • the adapter without partial indexes;
    • the migration lists;
    • two life_boundary_of cases that pin the row-value comparison on a timestamp tie: a row filed before the create at the same instant, and one filed after.
  • migrator_spec's fake Postgres connection answers supports_partial_index?.
  • Full suite: 1420 examples, 0 failures; rubocop clean.
  • Run against a real Postgres 15: 004 applies after 003, and EXPLAIN shows the boundary lookup on audit_logs_life_boundaries and the exists? as an index-only scan on audit_logs_record_history.

Related: PRD-1258

Definition of Done

General

  • Write an explicit title for the Pull Request, following Conventional Commits specification
  • Test manually the implemented changes
  • Validate the code quality (indentation, syntax, style, simplicity, readability)

Security

  • Consider the security impact of the changes made

🤖 Generated with Claude Code

Note

Add record-history and life-boundary indexes to audit trail migrations

  • Adds migration 004-index-record-history, creating a composite index on collection, record_id, timestamp, and id, with a record_id length option for MySQL and MariaDB.
  • On adapters supporting partial indexes, also creates a second index with the same columns filtered to create and delete operations. Both indexes tolerate replay. See migrations.rb.
  • Rewrites Store#before_row_condition to use a row-value comparison over timestamp and id instead of an OR expression, so pagination predicates match the new indexes. See store.rb.
  • Adds tests for both indexes, partial-index fallback, and same-timestamp create-boundary ordering.
  • Behavioral Change: existing audit tables get new indexes on next migration run; adapters without partial-index support only get the unfiltered history index.

Macroscope summarized effb930.

Migration 004 adds (collection, record_id, timestamp, id), and the same columns over create
and delete rows only. Without them, life_boundary_of's latest-boundary lookup walked the whole
table's (timestamp, id) index: 486 ms on a 2.1M-row table for a record created long ago,
0.04 ms with them. Adapters without partial indexes (MySQL, MariaDB) get the first one only.

before_row_condition now compares (timestamp, id) as a row value: the OR form it replaces
cannot use the new index as a range, and read a 50k-row history whole (7 ms vs 0.06 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qltysh

qltysh Bot commented Oct 21, 2026

Copy link
Copy Markdown

Qlty


Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (2)

RatingFile% DiffUncovered Line #s
Coverage rating: A Coverage rating: A
...min_agent/lib/forest_admin_agent/audit_trail/sql/migrations.rb100.0%
Coverage rating: A Coverage rating: A
...forest_admin_agent/lib/forest_admin_agent/audit_trail/store.rb100.0%
Total100.0%
🚦 See full report on Qlty Cloud »

🛟 Help
  • Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

  • Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

  • File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

    • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

@bexchauveto
bexchauveto merged commit c33c5b9 into main Oct 21, 2026
33 checks passed
@bexchauveto
bexchauveto deleted the perf/audit-trail-record-history-indexes branch October 21, 2026 14:17
@forest-bot

Copy link
Copy Markdown
Member

🎉 This PR is included in version 1.46.3 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

forest-bot added a commit that referenced this pull request Oct 21, 2026
## [1.46.3](v1.46.2...v1.46.3) (2026-10-21)

### Performance Improvements

* **audit_trail:** index a record's history and its life boundaries ([#416](#416)) ([c33c5b9](c33c5b9))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants