Skip to content

fix: keep single writes fast as the store grows - #13

Merged
St4NNi merged 16 commits into
mainfrom
perf/receipt-trim
Sep 24, 2026
Merged

St4NNi merged 16 commits into
mainfrom
perf/receipt-trim

Conversation

@St4NNi

@St4NNi St4NNi commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Description

Makes single writes fast and flat as the store grows. Measured with the new perf_write_latency profile (release, one writer, 5,000 existing RO-Crate graphs, search on):

Operation Before p50 / p99 / max After p50 / p99 / max
create crate 0.45 / 7.6 / 26 ms 0.62 / 2.4 / 6.5 ms
apply RO-Crate update 8.4 / 19 / 43 ms 0.98 / 3.0 / 8.9 ms
set policy 0.03 / 0.05 / 37 ms 0.05 / 0.15 / 1.2 ms
delete graph 0.32 / 0.5 / 31 ms 0.62 / 2.4 / 5.9 ms

Patching one entity into a crate that already holds 2,000 files takes 4.6 ms at p50 instead of 9.2 ms, and 0.64 ms instead of 1.13 ms with 200 files.

In Aruna, seeding 10,000 documents took 639 s on 0.3.0 (51 s on 0.2).

Causes fixed:

  • Receipt trimming walked the retained receipt order backwards past 4,096 receipts and the tombstones of their updates on every write.
  • Every RO-Crate document import processed the full embedded RO-Crate context (about 2,600 to 3,000 term definitions) from scratch.
  • Every publish of a search generation cloned the graph map and rebuilt one scope entry per graph.
  • The search worker's coverage check found the oldest queued work by scanning the tombstones of all acknowledged entries while holding the queue lock that every commit needs.
  • Validation walked hasPart parents through the source order, which scans the whole graph for every parent lookup.

Changes

  • Trim receipts and batch receipts inside the commit that makes the trim authoritative. Commits that change the receipt order hold one retention lock from trim selection until the counts are updated, so the counts stay exact, overlapping batches never remove the same receipt twice, and a receipt being rewritten is never trimmed underneath. Counts are restored exactly at open.
  • Keep each trim floor as an inclusive lower bound that no committed order key sorts below. A commit that inserts a key below it, such as a delayed admission, a backward clock step or a late batch mapping, lowers the floor, so older receipts still expire first. Each write removes the oldest receipts past retention, scanning forward from the floor; retention stays exact.
  • Replace a top-level embedded context with an inline context holding only the terms and prefixes the document uses. Unused definitions cannot change expansion; a test compares the triples with full-context parsing for the RO-Crate 1.1, 1.2 and 1.3 fixtures and a synthetic document.
  • Derive active search scopes from the graph generation map instead of rebuilding a set of scopes on every change, and share graph names in that map.
  • Start search coverage checks at a remembered queue head. Every queue insert, including a stale acknowledgement that narrows work backward, lowers that head first, so the check never passes queued work.
  • Recompute orphan diagnostics from the search snapshot when the stored record is stale or missing. Diagnostics commit after their source batch, so under sustained ingest indexing could exhaust its retries on that window. The recompute stays within the item byte budget and stops when the drain is cancelled; a graph that does not fit keeps the retryable stale error.
  • Drain again instead of failing a flush when coverage is refused because a concurrent write narrowed queued work below the drained target.
  • Read the committed rows of validation's post-change view through the trusted query index when a pattern binds a predicate or object but not the subject, so parent lookups no longer scan the graph.
  • Compile search-only storage, queue and flush code only with the search feature, and gate the SHACL cache budget on shacl-core, so every feature combination builds without warnings.
  • Add regression tests for indexed parent lookups, overlapping trims, delayed receipts, stale queue clears and stale search diagnostics, and the ignored perf_write_latency profile with configurable graph count, samples and crate size, including single entity patches.

@St4NNi St4NNi changed the title fix: trim mutation receipts once per admission interval fix: keep single writes fast as the store grows Sep 24, 2026
@St4NNi
St4NNi merged commit 0f32a5a into main Sep 24, 2026
24 checks passed
@St4NNi
St4NNi deleted the perf/receipt-trim branch September 24, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant