diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 64e47275e..62a3010d2 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -43,7 +43,8 @@ Plan: `.claude/plans/dir-sim-persist-v1.md`. |---|---|---|---| | **D-DSP-0** | `StoreImage` + `VersionStore::image` / `restore` (dir-sim, no I/O) | Shipped (#1473) | P-W3-0: round trip answers every read the same; damaged image refused | | **D-DSP-1** | `VersionStore::revert(current, earlier, evidence)` — undo as a version | Shipped (#1473) | the revert's view equals `earlier`'s | -| **D-DSP-2** | Lance writer/reader crate for `StoreImage` | Queued | write → read → restore equals the in-memory image | +| **D-DSP-2** | Lance writer/reader crate for `StoredImage`: logical versions in verifiable append-only batches (one batch = one Lance commit), torn saves refused | Queued | write → read → restore equals the in-memory image; a truncated or reordered batch is refused | +| **D-DSP-4** | `StoredImage` + `RuleRegistry` (exact `(name, version)`) + `restore_stored` + read-only `History` (decision 2(a)) | In PR | unknown rule refused, never substituted; history still reads; disable runs red | | **D-DSP-3** | `ViewActors` (W-2) + `from_graph` groups (wishlist #19) in dir-sim | Shipped (#1473) | disable runs red | ## D-LBL — labels of a concept, pulled through the plug (2026-10-11) diff --git a/.claude/plans/dir-sim-persist-v1.md b/.claude/plans/dir-sim-persist-v1.md index 13e8e52ca..be8bf7d46 100644 --- a/.claude/plans/dir-sim-persist-v1.md +++ b/.claude/plans/dir-sim-persist-v1.md @@ -41,20 +41,60 @@ as a simulated version whose origin names the restored version. completeness. - [x] `VersionStore` keeps each root's `Observation` (needed for the image). - [x] `VersionStore::revert(current, earlier, evidence)` — the undo as a version, recorded under `REVERT`. -- [ ] Lance writer/reader in a separate crate (`lance-graph-dir-sim-lance`), - so `lance-graph-dir-sim` keeps its no-I/O rule: one Lance dataset per - store, tables `values`, `nodes`, `members`, `proxies`, `versions`, - `changes`, `tags`; one Lance version per dir-sim version written. +- [x] **Decision 2(a), 2026-10-11:** persisted rules resolve by exact + `(name, version)`. `StoredImage` holds every rule as text + (`StoredRule`); `RuleRegistry` resolves only to the registered rule + with the same name and version (`REVERT` always registered); + `VersionStore::restore_stored` refuses an unknown rule + (`RestoreError::UnknownRules`, every version listed); `History` reads + such an image without ever resolving or replacing the rule, and + exposes no write. Tests in `tests/persist.rs`; disable runs red + (name-only match, `REVERT` not registered). +- [ ] **D-DSP-2, Lance writer/reader** in a separate crate + (`lance-graph-dir-sim-lance`), so `lance-graph-dir-sim` keeps its + no-I/O rule. Shape, decided 2026-10-11 (3(b)): + - **Logical versions are not Lance versions.** `VersionId` is the + history's own clock. A save appends one **batch** holding every logical + version written since the previous save, in order; one batch = one + Lance commit, so a Lance version counts saves, never dir-sim versions. + - **Sparse append, resident fold** (`knowledge/split-tunnel-sparse-append-resident-fold.md`): + a batch carries only what is new since the last save (values interned + since, roots observed since, versions since, tag and verdict changes). + Nothing is updated in place or compacted; Lance is read once, at + restore, into a `StoredImage`. + - **Verifiable batches.** Each batch carries a header record: batch + sequence number, first and last logical version, record count, and a + checksum over the batch's records, written in the same commit. Restore + checks, batch by batch in sequence order: the sequence is dense, the + first version follows the previous batch's last, the record count and + checksum match. + - **Torn saves are refused, never repaired.** A batch failing any check + (missing records, a gap, a checksum mismatch, a header without its + records) refuses the restore with the batch named. No prefix of the + history is restored silently; reading up to the last good batch is a + separate, explicit call, not a fallback. + - **One writer.** The writer checks the dataset's latest version before + and after its append (stale writer refused, race detected); a race + leaves the log refused until reconciled. In-band idempotency stays + OPEN, as in the reference implementation. + - Records: `Observation`, `Change`, `Origin` and `Violation` need a + stable encoding; it lives in the Lance crate next to the reader that + must decode it, with a round-trip test per variant. +- [x] **Decision 4(b), 2026-10-11:** group-to-role grants are the + authority's versioned policy, not directory state: OGAR + `ogar_rbac::group_policy::GroupRolePolicy` (append-only, own + `PolicyVersion`, `GrantProvenance` per change). dir-sim's + `GroupRoles` is filled from a policy state; `None` (never named) and + `Some([])` (all revoked) stay distinct (5(a)). - [ ] Board: STATUS_BOARD row; HubSPO handover reply. ## Open -- Whether a Lance version per dir-sim version, or per write batch, is the - right mapping (OGAR says 1:1 "by design"; a batch import would write many - versions at once). +- ~~Whether a Lance version per dir-sim version, or per write batch, is the + right mapping.~~ Decided 2026-10-11 (3(b)): per batch; the logical clock + is independent (OGAR #354 corrects `provenance.rs`). - Encoding of `Change` / `Origin` rows: they are OGAR types; their columnar form belongs next to them in OGAR or here. -- `RuleId::name` is `&'static str` (OGAR `provenance.rs:29`). An image held in - memory restores fine, but one read back from bytes has no `'static` name to - put there. D-DSP-2 needs either a rule registry the reader maps names - through (unknown name = refusal), or an owned name in OGAR's `RuleId`. +- ~~`RuleId::name` is `&'static str`.~~ Decided 2026-10-11 (2(a)): a rule + registry, exact `(name, version)`, unknown = refusal; `History` for + read-only inspection. diff --git a/crates/lance-graph-dir-sim/src/image.rs b/crates/lance-graph-dir-sim/src/image.rs index d44efff70..83bade121 100644 --- a/crates/lance-graph-dir-sim/src/image.rs +++ b/crates/lance-graph-dir-sim/src/image.rs @@ -18,8 +18,14 @@ //! [`VersionStore`]: crate::VersionStore use crate::snapshot::{BuildError, Observation}; +use crate::store::REVERT; +use crate::store::{SimError, VersionStore}; use crate::view::ApplyError; -use ogar_dir_sim::{Version, VersionId, Violation}; +use crate::view::View; +use ogar_dir_sim::ValueId; +use ogar_dir_sim::{Change, EvidenceRef, Origin, RuleId, Version, VersionId, Violation}; +use std::collections::BTreeMap; +use std::collections::BTreeSet; /// A version history as owned data ([`crate::VersionStore::image`], /// [`crate::VersionStore::restore`]). @@ -65,4 +71,327 @@ pub enum RestoreError { IncompleteDictionary, /// A tag or verdict names a version the image does not hold. UnknownVersion(VersionId), + /// Simulated versions whose rule the registry does not know, all of + /// them, in version order. A live store needs every rule resolved. + UnknownRules(Vec), +} + +/// A rule's identity as persisted: its name as text and its version. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct StoredRule { + /// The rule's stable name. + pub name: String, + /// The rule's version. + pub version: u16, +} + +impl From for StoredRule { + fn from(r: RuleId) -> Self { + Self { + name: r.name.to_string(), + version: r.version, + } + } +} + +/// What produced a version, as persisted. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum StoredOrigin { + /// Read from reality. + Observed { + /// Which observer. + source: String, + /// When it was read (unix ms). + observed_at_ms: i64, + }, + /// Produced by a rule. + Simulated { + /// The rule. + rule: StoredRule, + /// The input it ran on. + evidence: Vec, + }, +} + +/// A version as persisted: [`Version`] with its rule as text. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct StoredVersion { + /// This version. + pub id: VersionId, + /// The version it was derived from. + pub parent: Option, + /// What produced it. + pub origin: StoredOrigin, + /// The changes it introduced, verbatim. + pub delta: Vec, +} + +impl From for StoredVersion { + fn from(v: Version) -> Self { + Self { + id: v.id, + parent: v.parent, + origin: match v.origin { + Origin::Observed { + source, + observed_at_ms, + } => StoredOrigin::Observed { + source, + observed_at_ms, + }, + Origin::Simulated { rule, evidence } => StoredOrigin::Simulated { + rule: rule.into(), + evidence, + }, + }, + delta: v.delta, + } + } +} + +/// A [`StoreImage`] as a storage crate writes it: every rule as text, so +/// nothing in it borrows from the program that wrote it. +#[derive(Clone, Debug, Default)] +pub struct StoredImage { + /// The dictionary: raw values in id order. + pub values: Vec, + /// Each root and the observation it was built from. + pub roots: Vec<(VersionId, Observation)>, + /// Every version, in id order. + pub versions: Vec, + /// Tags, sorted by name. + pub tags: Vec<(String, VersionId)>, + /// Recorded verdicts, by version id. + pub verdicts: Vec<(VersionId, Vec)>, +} + +impl From for StoredImage { + fn from(i: StoreImage) -> Self { + Self { + values: i.values, + roots: i.roots, + versions: i.versions.into_iter().map(Into::into).collect(), + tags: i.tags, + verdicts: i.verdicts, + } + } +} + +/// The rules a restore may resolve a [`StoredRule`] to. +/// +/// A stored rule resolves only to the registered rule with the same name +/// and the same version. Another version of the same rule is a different +/// behaviour, so it is never taken in its place, and nothing is guessed +/// from a name alone. [`REVERT`] is always registered: the store itself +/// records it. +#[derive(Clone, Debug)] +pub struct RuleRegistry { + rules: BTreeSet, +} + +impl Default for RuleRegistry { + fn default() -> Self { + Self::new() + } +} + +impl RuleRegistry { + /// A registry holding the store's own rule, [`REVERT`]. + pub fn new() -> Self { + Self { + rules: BTreeSet::from([REVERT]), + } + } + + /// Register `rule`. + #[must_use] + pub fn with(mut self, rule: RuleId) -> Self { + self.rules.insert(rule); + self + } + + /// The registered rule with exactly `rule`'s name and version. + pub fn resolve(&self, rule: &StoredRule) -> Option { + self.rules + .iter() + .find(|r| r.name == rule.name && r.version == rule.version) + .copied() + } +} + +/// A simulated version whose rule a [`RuleRegistry`] does not know. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct UnknownRule { + /// The version. + pub version: VersionId, + /// Its rule as stored. + pub rule: StoredRule, +} + +impl StoredImage { + /// The image with every rule resolved through `rules`, or every version + /// whose rule is unknown. + /// + /// # Errors + /// + /// [`RestoreError::UnknownRules`], listing each unresolved version. + pub fn resolve(self, rules: &RuleRegistry) -> Result { + let (image, unknown) = self.resolve_with(rules); + if unknown.is_empty() { + Ok(image) + } else { + Err(RestoreError::UnknownRules(unknown)) + } + } + + /// Resolve each rule; an unknown one is recorded as [`UNRESOLVED`] and + /// returned in the list. + fn resolve_with(self, rules: &RuleRegistry) -> (StoreImage, Vec) { + let mut unknown = Vec::new(); + let versions = self + .versions + .into_iter() + .map(|v| { + let origin = match v.origin { + StoredOrigin::Observed { + source, + observed_at_ms, + } => Origin::Observed { + source, + observed_at_ms, + }, + StoredOrigin::Simulated { rule, evidence } => { + let resolved = rules.resolve(&rule).unwrap_or_else(|| { + unknown.push(UnknownRule { + version: v.id, + rule: rule.clone(), + }); + UNRESOLVED + }); + Origin::Simulated { + rule: resolved, + evidence, + } + } + }; + Version { + id: v.id, + parent: v.parent, + origin, + delta: v.delta, + } + }) + .collect(); + ( + StoreImage { + values: self.values, + roots: self.roots, + versions, + tags: self.tags, + verdicts: self.verdicts, + }, + unknown, + ) + } +} + +/// The rule a [`History`] records for a version whose stored rule it could +/// not resolve. Never registered and never resolved to: it marks "not +/// known here", and the stored identity stays in [`History::rule`]. +const UNRESOLVED: RuleId = RuleId { + name: "Unresolved", + version: 0, +}; + +/// A restored history that can only be read. +/// +/// For inspecting a history whose rules are not all known to this program: +/// every version's delta is stored verbatim, so reading a version never +/// needs its rule. The store inside cannot be reached mutably, so nothing +/// can be observed, simulated, reverted or promoted on top of an unknown +/// rule. [`History::rule`] gives each simulated version's rule as stored. +#[derive(Debug)] +pub struct History { + store: VersionStore, + rules: BTreeMap, + unresolved: Vec, +} + +impl History { + /// Restore `image` for reading, resolving what `rules` knows. + /// + /// # Errors + /// + /// [`RestoreError`] other than `UnknownRules`: the image is damaged. + pub fn restore(image: StoredImage, rules: &RuleRegistry) -> Result { + let stored: BTreeMap = image + .versions + .iter() + .filter_map(|v| match &v.origin { + StoredOrigin::Simulated { rule, .. } => Some((v.id, rule.clone())), + StoredOrigin::Observed { .. } => None, + }) + .collect(); + let (image, unresolved) = image.resolve_with(rules); + Ok(Self { + store: VersionStore::restore(image)?, + rules: stored, + unresolved, + }) + } + + /// The versions whose rule was not resolved, in version order. + pub fn unresolved(&self) -> &[UnknownRule] { + &self.unresolved + } + + /// A simulated version's rule as stored; `None` for an observation or + /// an unknown version. + pub fn rule(&self, v: VersionId) -> Option<&StoredRule> { + self.rules.get(&v) + } + + /// The version record. For an unresolved version the origin's rule is a + /// placeholder named `"Unresolved"`; [`History::rule`] has the stored one. + pub fn version(&self, v: VersionId) -> Option<&Version> { + self.store.version(v) + } + + /// The directory as of `v`. + /// + /// # Errors + /// + /// [`SimError::UnknownVersion`]. + pub fn view(&self, v: VersionId) -> Result, SimError> { + self.store.view(v) + } + + /// The changes from `a` to `b`. + /// + /// # Errors + /// + /// As [`VersionStore::diff`]. + pub fn diff(&self, a: VersionId, b: VersionId) -> Result, SimError> { + self.store.diff(a, b) + } + + /// The version a tag names. + pub fn tag(&self, name: &str) -> Option { + self.store.tag(name) + } + + /// The verdict recorded for `v`. + pub fn verdict(&self, v: VersionId) -> Option<&[Violation]> { + self.store.verdict(v) + } + + /// The text of a dictionary value. + pub fn value(&self, v: ValueId) -> Option<&str> { + self.store.value(v) + } + + /// A value's id, if the history holds it. + pub fn lookup(&self, s: &str) -> Option { + self.store.lookup(s) + } } diff --git a/crates/lance-graph-dir-sim/src/store.rs b/crates/lance-graph-dir-sim/src/store.rs index 42e688f17..cf47628f2 100644 --- a/crates/lance-graph-dir-sim/src/store.rs +++ b/crates/lance-graph-dir-sim/src/store.rs @@ -338,6 +338,21 @@ impl VersionStore { Ok(st) } + /// A store from a persisted image whose rules are text: each resolved + /// through `rules` by exact name and version, then [`Self::restore`]. + /// + /// # Errors + /// + /// [`RestoreError::UnknownRules`] when any rule is not registered, or + /// any error of [`Self::restore`]. For reading such a history, see + /// [`crate::image::History`]. + pub fn restore_stored( + image: crate::image::StoredImage, + rules: &crate::image::RuleRegistry, + ) -> Result { + Self::restore(image.resolve(rules)?) + } + /// Version a tag points to. pub fn tag(&self, name: &str) -> Option { self.tags.get(name).copied() diff --git a/crates/lance-graph-dir-sim/tests/persist.rs b/crates/lance-graph-dir-sim/tests/persist.rs index 7d0a135dc..63139491f 100644 --- a/crates/lance-graph-dir-sim/tests/persist.rs +++ b/crates/lance-graph-dir-sim/tests/persist.rs @@ -225,3 +225,85 @@ fn a_revert_is_a_new_version_that_reads_as_the_earlier_one() { let back = VersionStore::restore(st.image()).unwrap(); assert_eq!(back.diff(r, g0).unwrap(), vec![]); } + +// ---- Persisted rules: text in, exact (name, version) out (decision 2(a)). + +use lance_graph_dir_sim::image::{History, RuleRegistry, StoredImage, StoredRule, UnknownRule}; + +fn all_rules() -> RuleRegistry { + RuleRegistry::new() + .with(EXCHANGE_ACCESS) + .with(EMPLOYEES_GET_EXCHANGE) + .with(RENAME_MAIL) +} + +// A stored image whose rules are all registered restores to a store that +// reads the same as the original. +#[test] +fn stored_rules_resolve_to_the_same_store() { + let (st, vs, _) = history(); + let stored = StoredImage::from(st.image()); + let back = VersionStore::restore_stored(stored, &all_rules()).unwrap(); + assert_eq!(reads(&back, &vs), reads(&st, &vs)); +} + +// A rule registered under another version is a different behaviour: it is +// not taken in place of the stored one, and the restore names every +// version it could not resolve. +#[test] +fn another_version_of_a_rule_is_not_substituted() { + let (st, vs, _) = history(); + let stored = StoredImage::from(st.image()); + let rules = RuleRegistry::new() + .with(RuleId { + name: "ExchangeAccess", + version: 2, + }) + .with(EMPLOYEES_GET_EXCHANGE); + let err = VersionStore::restore_stored(stored, &rules).unwrap_err(); + assert_eq!( + err, + RestoreError::UnknownRules(vec![ + UnknownRule { + version: vs[1], + rule: StoredRule::from(EXCHANGE_ACCESS), + }, + UnknownRule { + version: vs[3], + rule: StoredRule::from(RENAME_MAIL), + }, + ]) + ); +} + +// The same history whose rules are unknown can still be read: every +// version's delta is stored, so its view needs no rule. The stored rule +// stays visible as it was written. +#[test] +fn an_unknown_rule_still_reads_as_history() { + let (st, vs, _) = history(); + let stored = StoredImage::from(st.image()); + let h = History::restore(stored, &RuleRegistry::new()).unwrap(); + assert_eq!(h.unresolved().len(), 3); + assert_eq!(h.rule(vs[1]), Some(&StoredRule::from(EXCHANGE_ACCESS))); + assert_eq!(h.rule(vs[0]), None); + for &v in &vs { + assert_eq!( + h.view(v).unwrap().members_transitive(&g(EMPLOYEES)), + st.view(v).unwrap().members_transitive(&g(EMPLOYEES)), + ); + assert_eq!(h.diff(vs[0], v), st.diff(vs[0], v)); + assert_eq!(h.verdict(v), st.verdict(v)); + } + assert_eq!(h.tag(TAG_DESIRED), st.tag(TAG_DESIRED)); +} + +// The revert rule is the store's own and always resolves. +#[test] +fn a_revert_resolves_without_registering_it() { + let (mut st, vs, _) = history(); + let r = st.revert(vs[2], vs[0], &ev("UNDO-1")).unwrap(); + let stored = StoredImage::from(st.image()); + let back = VersionStore::restore_stored(stored, &all_rules()).unwrap(); + assert_eq!(back.diff(vs[0], r), st.diff(vs[0], r)); +}