Skip to content

Syncables: expose PostgreSQL collections through a read-only Atomic peer - #685

Closed
michielbdejong wants to merge 2 commits into
mainfrom
codex/syncables-atomic-peer
Closed

michielbdejong wants to merge 2 commits into
mainfrom
codex/syncables-atomic-peer

Conversation

@michielbdejong

@michielbdejong michielbdejong commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #696, the consolidated merge candidate. This PR's current head c8a1ade63c39a6c75eb8bff69b50fd1feeee647a is an ancestor of candidate 91fb1e31b7fa6c1634bba86179d3187cb43097c5; its code and review fixes are retained. All 44 checks on that candidate completed: 41 passed and three were skipped. Merge #696 rather than this source PR. The source branch is retained.


Syncables can now serve one accepted PostgreSQL collection as a native Atomic drive without Git. The new atomic-peer command follows local snapshot revisions and feeds an optional Rust companion pinned to the existing atomic_lib core. PostgreSQL remains authoritative; an embedded redb projection maintains stable identities, self-verifying genesis, signed changes, persistent Loro documents/version vectors and a durable pending journal/checkpoint.

The loopback WebSocket peer supports authentication, permission-checked retrieval, initial reconciliation, delta sync, subscriptions and reconnect. It requires explicitly selected owner signing authority and reader agents, rejects all incoming mutations (including valid owner-signed commits), and advertises only implemented capabilities. Source records and bundled schemas are native wrappers containing exact JSON-AD text, hashes, revision/OAD provenance and schema links; numbers are preserved without native numeric coercion. Membership absence retains documents and identities for reappearance.

Adds build/run and representation documentation plus a dedicated native compatibility CI job. Existing replication destination definitions remain configuration only; target-drive imports, provider writeback and typed native column mapping are outside this prototype.

Validation:

  • 1,825 TypeScript tests in 104 files passed against a separate local PostgreSQL 18 test database, including the real native process and offline catch-up.
  • Five native compatibility tests passed using the unmodified Atomic core replica and official WebSocket client: initial/unchanged/delta sync, schemas/numeric lexemes, stable identities, membership removal/reappearance, subscriptions, denied read/write access, wrong-origin/failed auth, process kill/reconnect and interrupted-journal recovery.
  • Rust formatting/Clippy, TypeScript lint/formatting/release build, changelog/whitespace checks and installed-tarball daemon smoke passed. A separate invented-data CLI smoke verified config/scope selection, native publication/listening and graceful shutdown.
  • GitHub's native compatibility job and all PostgreSQL 17 / Node 22/24/26 jobs passed on the latest commit. Shared CI is still running the issue-tracker browser lane; its Chromium installation is pending. The generic browser and shared tooling checks passed.

Related to #684. Keeping this draft and the issue open pending CI and one explicitly selected live collection/reader check. The Atomic drive skill was located, but its public identity helper reports no bootstrap configuration in this cloud workspace; the live drive selection and authorized signing access are still unavailable. No real provider records, owner keys or provider credentials were accessed.

Claude-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075
Codex-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075

Michiel de Jong added 2 commits October 23, 2026 16:19
Maintain signed native identities and Loro state, journal projection progress, and follow accepted collection revisions through an optional loopback companion. Add native replica, WebSocket and PostgreSQL acceptance coverage.

Claude-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075
Codex-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075
Keep an existing subscription only when the newly authenticated agent can still read the drive, and cover allowed and denied identity switches.

Claude-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075
Codex-Session: codex://threads/01a16979-c3bd-74c1-984d-e550ea9ff075

joepio commented Oct 26, 2026

Copy link
Copy Markdown
Member

Complexity review (head c8a1ade): the direction is right. The peer reuses atomic_lib for protocol, signatures and Loro state rather than reimplementing them. Four things to cut before this leaves draft:

  1. create_genesis (native-atomic-peer/src/lib.rs:78-97) re-implements Resource::save_as_genesis. That function exists at the pinned rev, in atomic_lib resources.rs:1434, and signs with the store's default agent, which is what this function does. The hand-built CommitOpts is a second place that decides which validations a genesis gets.
  2. Session::handle (lib.rs ~436-600) is a hand-written WS gate in front of engine::handle_frame. It covers AUTH size and parse, HELLO caps, KEEPALIVE, a write refusal for every mutating tag, and drive-scope checks on GET/GET_MANY/SUB/SYNC. A read-only replica is a general need. A read_only flag (or a refuse-writes policy) in atomic_lib's frame handler would replace most of these ~170 lines. The per-tag scope checks would then follow the protocol automatically, and adding a frame type could not open a hole here.
  3. A second 3,288-line Cargo.lock, pinned to atomic_lib 66880fc. Develop has since changed SYNC_PUSH persistence (atomic-server#2361, batch and flush on LAST) and the probe path (atomic-server#2358, hash probe). Each pin bump here is a manual compatibility pass. That cost belongs in the PR's plan, or the crate should join an existing workspace that already tracks atomic_lib.
  4. #692 (native tables per collection, +2,081/−218) stacks on this and rewrites much of the same projection. Landing the two as one PR would avoid reviewing the wrapper format twice. The wrapper format is the part SyncD: generate native collection tables in account folders #692 says needs "a fresh projection directory".

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants