Skip to content

Record chain payment receipts durably before acknowledging them - #374

Merged
ctfbruce merged 11 commits into
mainfrom
fix/durable-receipts-and-transfers
Oct 5, 2026
Merged

ctfbruce merged 11 commits into
mainfrom
fix/durable-receipts-and-transfers

Conversation

@ctfbruce

@ctfbruce ctfbruce commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Behaviour

Chain payment receipts are processed durably and acknowledged only after their effect is recorded:

  • Dispatcher schema 22 adds payment_receipts: every PaymentReceipt addressed to this dispatcher is stored under its chain transaction digest and event position (one transaction may carry several receipts, even for one nonce) with a disposition (applied, duplicate, mismatch naming the field that differed, unknown_intent, expired). Marking a transaction paid is a conditional write on its outstanding status in the same local transaction, so a duplicate, reordered or re-read receipt cannot pay twice, and a second payment for a paid intent, by another transaction or another event of the same one, is recorded as a mismatch naming the paying digest and event.
  • A receipt is validated before any effect: chain method, coin type for the currency, receiver (addresses compared in normalised form), exact amount that fits the local representation, and expiry. The expiry check never fired on main: the decoder read nothing into the timestamp field.
  • The listener's cursor advances only over proven coverage: a catch-up asks the GraphQL indexer for the last checkpoint whose events it serves and reads and stores no further than the lower of that and the full node's tip, with each events query bounded above as well. Every receipt of the covered range is recorded before the cursor moves; a local failure or an undecodable event stops the pass and leaves the cursor, so a restart re-reads the range and finds duplicates instead of skipping payments. A streamed checkpoint ahead of the cursor records its own receipts and holds the cursor at the proven bound until the gap between them has been read through the indexer; the cursor never moves backwards. An invalid stored cursor is a startup error, not a silent reset; a first start without a cursor persists the bound it begins at.
  • The schema also adds chain_transfers, the record of outbound payouts and refunds with the intended transfer, the signed transaction and its digest persisted before anything is broadcast, used by the following change.

Proof

  • Base reproduction: an expired receipt was fulfilled and an invalid stored cursor accepted (internal/dispatcher/payments/sui scratch test on main).
  • Local-fixture tests (an in-process GraphQL server serving pages and the indexer's mark, and real SQLite; no network, wallet or chain endpoint): node at 110 with the indexer at 105 keeps the cursor at 105 and applies the payment at 108 exactly once when the indexer reaches it; a streamed checkpoint 115 at indexer 105 records its receipt, holds the cursor and completes the gap when the indexer has it, and a re-read after a restart applies nothing twice; a rejected and a valid receipt event of one transaction pay in either order, replay without effect, and a second valid event is a mismatch; receipts applied once across pages, duplicates and a restart with reordered pages; each rejection recorded with its reason, including an amount beyond the signed range recorded exactly; unknown nonce recorded; a receipt to another payee not recorded; a trigger-refused write leaves the cursor and the retry applies once; undecodable events keep the cursor; a streamed checkpoint records its sequence and a failure keeps the cursor; a gap before the first streamed checkpoint is caught up first; an invalid cursor refuses to start; disabled mode writes nothing.
  • The payments, Sui, HTTP API, storage and schema packages pass under -race on the composed tree.

Upgrade

Dispatcher schema 22 is required: stop the dispatcher, back up the database and run the explicit upgrade before starting this build.

Requires #364 (schema 20) and #373 (schema 21); this branch is based on #373's head.

Limits

  • Receipts were exercised against local fixtures only; the payment-kit event layout (timestamp_ms last) and the coin-type string form are assumptions until a deployed package is checked.
  • The indexer's mark (serviceConfig.availableRange for Query.events) and the event position (Event.sequenceNumber) are taken from the published Sui GraphQL schema and have not been confirmed against a live service; the streamed event index and Event.sequenceNumber are assumed to agree, and if they did not, a re-read receipt would be recorded as a second row whose effect is a mismatch, never a second payment. Receipts read during catch-up record no checkpoint number.
  • Payout reservation and refunds outside the local transaction (Reserve exact payout amounts and reconcile uncertain transfers #80, Make refunds safe for empty orders, write failures and uncertain transfers #81) are the following change; chain payments remain disabled in every shipped configuration.

GitHub issue #79.

Candidate CI

Run 37284811811 on 2554a22: 16 of 16 lanes and required succeeded. Corrections (indexer-bounded cursor, receipt identity by event) are in f7c0b14, with #373's corrected head merged forward: run 37290240068 on f7c0b14, 16 of 16 lanes and required succeeded.

TheodorAdrienIsaak Mattli added 8 commits October 2, 2026 15:17
Compose the account recovery and provider configuration change with the
CILogon production deployment that landed on main. The merged deployment keeps
main's variable names, its production enablement, its credential preflight that
counts both entries without reading them, and its single example file, and adds
the browser-approved CLI authentication settings, the issuer and URL checks,
the configuration-update refresh of the unit and credentials, and the extended
render fixture. The duplicate example file is dropped; the installed credential
file name is unchanged.

Go, SQL and documentation changes are those already on the branch; the render
fixture passes inside the pinned provisioner against a retained package.

GitHub issues #226, #223 and #227.
The payment decision of a terminal run was applied by an order-state write
without a previous-state check and nothing recorded which run settled
which order; when the write after the terminal report failed, the order
stayed outstanding for good. Dispatcher schema 21 adds order_settlements,
one immutable row per settled order (credit or refund, amount, currency,
executor and run), and the pricing_rule column of transactions. The
decision now lives in one place: the order moves from outstanding only
through a conditional write, the settlement row and the earnings credit
or chain refund commit with it, and a repeated or concurrent delivery
leaves a settled order as it is. The maintenance loop settles terminal
runs with a recorded exit code whose inline settlement failed, a bounded
number per pass. A failed TEST run is refunded locally, moving nothing;
TEST intents are still never refunded as a whole. Every intent records
the name of the rule that priced it, and the API guide states that a USDC
intent is a local record created without a chain call.

GitHub issues #76, #77 and #78.
The receipt listener applied each receipt through a void callback whose
write failures were only logged, advanced its checkpoint cursor whatever
happened, converted the chain amount to a signed integer unchecked, only
warned about a receipt for a non-chain method and reset an invalid stored
cursor silently; its expiry check never fired because the decoder read
nothing into the timestamp. Dispatcher schema 22 adds payment_receipts,
one row per receipt addressed to this dispatcher keyed by chain digest
and nonce with its disposition, and chain_transfers, the record of
outbound payouts and refunds with the signed transaction persisted before
anything is broadcast. A receipt is validated strictly, applied by a
conditional write on an outstanding transaction and recorded in the same
local transaction, so a receipt read again after a restart is recognized
instead of applied twice; the cursor moves only after every receipt of a
range is recorded, a gap between the catch-up and the first streamed
checkpoint is read through the indexer first, an invalid stored cursor is
a startup error, and a first start persists the chain tip it begins at.

GitHub issue #79.
The executor's rejected-allocation test still asserted that the failed
run's TEST order stays outstanding, the limitation the settlement change
removes: a failed TEST run now refunds its order locally and no credit is
invented. The assertion follows the new behaviour.

GitHub issue #78.
Brings in the executor test expectation of a locally refunded TEST order
so this branch builds on the same settlement behaviour it extends.

GitHub issue #79.
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 08:48
@ctfbruce ctfbruce added the Review Implementation is ready for review and required validation. label Oct 5, 2026
@ctfbruce
ctfbruce marked this pull request as draft October 5, 2026 09:00
TheodorAdrienIsaak Mattli added 3 commits October 5, 2026 11:11
The maintenance pass retried every outstanding order of a terminal run,
including USDC refunds, which the terminal path sends inside the local
transaction: a transfer whose response was lost or whose commit failed
left the order outstanding and the next pass paid the buyer again. The
pass also always listed the oldest rows, so a page of orders that keep
failing kept every later order from being attempted. The pass now settles
only wholly local outcomes, credits of any currency and TEST refunds, and
defers a chain refund without touching the backend; chain refunds are
attempted once from the terminal path until a durable transfer record
takes them over. The listing continues after the last run the previous
pass attempted and starts over once it is exhausted, so a failing order
never blocks the ones after it.

GitHub issue #78.
The catch-up read PaymentReceipt events through the GraphQL indexer up
to the full node's checkpoint height and stored that height as the
cursor. An indexer behind the node has not served the events of the
checkpoints between its own mark and the tip yet, so a payment in that
range was passed over for good. Ask the indexer for the last checkpoint
whose events it serves (serviceConfig.availableRange) and read and
store no further than that. Bound each events query above as well, so
a pass reads exactly the range it claims. A streamed checkpoint ahead
of a lagging indexer records its own receipts and leaves the cursor at
the proven bound; the gap between them is read through the indexer
once it has ingested it, and the cursor moves on only when the run of
streamed checkpoints is joined to it, never backwards.

A payment_receipts row was keyed by transaction digest and nonce, so a
transaction carrying several receipt events for one nonce kept only the
first: a valid event after a rejected one in the same transaction was
treated as already recorded and did not pay. Key the row by transaction
digest and event position instead (GraphQL Event.sequenceNumber, the
event's index in the streamed transaction), index the nonce, and record
a second valid event of an already paid intent as a mismatch naming the
paying event. Migration 22 is edited in place; it has not shipped.

Regressions: node at 110 with the indexer at 105 keeps the cursor at
105 and applies the payment at 108 exactly once when the indexer
reaches it; a streamed checkpoint 115 at indexer 105 records its
receipt, holds the cursor and completes the gap when the indexer has it;
a rejected and a valid event of one transaction pay in either order,
replay without effect, and a second valid event is a mismatch.

Both GraphQL fields are taken from the published schema and not yet
confirmed against a live service; receipts read during catch-up still
record no checkpoint.

GitHub issue #79.
Bring the settlement ledger branch at its corrected head (local
settlement retries and a fair scan of pending orders) under the
receipts branch, so that the receipt and transfer tables build on the
order_settlements ledger as shipped.

GitHub issue #79.
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 09:37
@ctfbruce
ctfbruce merged commit be5455f into main Oct 5, 2026
16 checks passed
@vincent10400094
vincent10400094 deleted the fix/durable-receipts-and-transfers branch October 8, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Review Implementation is ready for review and required validation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant