Skip to content

Reserve payouts exactly and record chain transfers before broadcast - #381

Merged
ctfbruce merged 5 commits into
mainfrom
fix/payout-reservation-and-refunds
Oct 5, 2026
Merged

ctfbruce merged 5 commits into
mainfrom
fix/payout-reservation-and-refunds

Conversation

@ctfbruce

@ctfbruce ctfbruce commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Behaviour

Executor payouts and refunds go through one durable transfer lifecycle (internal/dispatcher/payments/transfers.go):

  • A transfer is prepared against the chain (a read), then the local decision and a reserved row of chain_transfers carrying the transaction digest, the signed transaction and its signature are committed, then the signed transaction is broadcast (bounded by two minutes), and the outcome is recorded by a conditional write on the state the caller read: sent, failed (not broadcast, or failed on chain; a failed payout releases its amount in the same write) or unknown (lost response or timeout). No SQL transaction spans a chain call.
  • Reconciliation (ReconcileTransfers, on the payout loop's one-minute ticker) resolves sent and unknown rows and reserved rows older than ten minutes with one lookup: a confirmed transaction with the expected effects becomes confirmed; a failed one failed (payout released); a transaction the chain does not know is resubmitted once from the identical stored bytes, only from reserved or unknown; an unverified or inconclusive result stays unknown with the last check in detail.
  • Payouts (Reserve exact payout amounts and reconcile uncertain transfers #80): PayoutExecutor reserves exactly the balance it read (ReservePayout, one conditional update requiring the balance and no open payout of that executor and currency); earnings credited during the transfer are kept; zero balances and non-chain currencies are skipped; executors without a wallet produce one warning per pass.
  • Refunds (Make refunds safe for empty orders, write failures and uncertain transfers #81): the transaction status change, one conditional Outstanding -> Refunded transition per order and a refund settlement row commit before the chain call; credited or refunded orders are never refunded; a repeated request sends nothing; the terminal USDC refund and RefundDebugletOrder use the same path; TEST behaviour is unchanged (bookkeeping only).
  • The maintenance pass settles chain-currency refunds of failed runs through the same lifecycle: an order with any refund transfer row is never selected again, so a lost response cannot cause a second broadcast; reconciliation resolves the row.
  • docs/operations/payments.md describes the lifecycle, the reconciliation table and what an operator sees for failed and unknown rows.

Proof

  • Real SQLite with a scripted chain, no network, wallet or endpoint: the balance is reserved exactly while earnings arrive during both chain calls (a held transaction would deadlock the one-connection pool); a second payout is refused while one is open; not-broadcast and on-chain failure restore the balance; a preparation failure reserves nothing; a lost response then a verified lookup confirms without a second broadcast; a lookup error and an unverified success stay unknown with the amount reserved; a transaction the chain does not know is resubmitted from the stored bytes exactly once; a stale reservation is left alone while fresh and resolved by digest when old; a blocked submission is recorded unknown at the deadline; disabled mode reaches no chain; refunds: one transfer of the total with one settlement per order, a failed transfer keeps the decision, a repeated request sends nothing, settled orders are skipped, a refused reservation leaves state untouched with no broadcast; the payout pass with failed, sent, zero and no-wallet balances; the maintenance pass: a refund owed by a failed USDC run is sent once with its reserved row before the broadcast, a lost response leaves the transfer unknown and the next pass selects nothing, reconciliation resolves it.
  • On the composed tree (this branch): gofmt clean, generated files match the pinned generators, vet and build pass, and the payments, Sui, dispatcher, HTTP API, storage and schema packages pass under -race.

Upgrade

Dispatcher schema 22 (from the receipts change) is required; this change adds no migration.

Stacked on #374 (receipts and the chain_transfers table) and #369 (prepared transfers); both are merged into this branch, so the diff against main shows their commits until they merge.

Limits

  • A failed refund is not re-sent as a new row; the row is the record of what is owed.
  • A resubmission whose coins were spent elsewhere stays unknown and blocks that executor's payouts until an operator acts (documented on the operations page).
  • The digest computed for a prepared transaction has been checked for self-consistency only; its agreement with a real testnet transaction is part of enabling chain payments, which remain disabled in every shipped configuration.
  • While chain payments are disabled, the maintenance pass counts chain-currency orders as deferred and does not attempt them, as the terminal path refuses them; they are settled once chain payments are enabled.
  • The pass's regressions exercise SettlePendingOrders with the scripted chain; the dispatcher's timed sweep is tested for its forwarding only.
  • A transfer whose stored bytes the chain reports as not found is resubmitted from reserved or unknown on every reconciliation pass; if the bytes can no longer execute and the node's rejection is not reported as not broadcast, the row returns to unknown and is resubmitted each minute without a bound. The digest never changes, so nothing is paid twice, but an unknown payout keeps that executor's later payouts waiting until an operator resolves it. A bound on resubmission is a following change (Bound the resubmission of a chain transfer the node reports as not found #385).
  • The refund address of a USDC order is validated when the refund is prepared, not when the intent is created; an invalid address leaves the order outstanding with the settlement pass retrying it, and the owed refund unrecorded, until the address problem is fixed at the intent API (a following change, Validate the refund address at payment intent creation #386).
  • TransferUSDC sends coins outside the lifecycle and has no production caller; it is left for the same trim.
  • SettleEarning and the backend's RefundDebuglet and TransferCoins are no longer used by payouts and refunds and are left for a later trim.

GitHub issues #80 and #81.

Candidate CI

Run 37292787301 on 4672277: 16 of 16 lanes and required succeeded. The branch then took the adapter's second correction by merge (76a70bd): run 37300726652, 16 of 16 lanes and required succeeded on attempt 2 (attempt 1 failed in the race and test lanes on a data race inside an executor test and an export deadline, both outside the diff). Corrections 66fe628: the maintenance refusal now says whether the recorded refund was sent, is pending confirmation, or could not be sent and is owed, on the deciding and on later requests, instead of "has been refunded"; each reconciliation lookup and the preparation read are bounded (30 s), a lookup that times out leaves the transfer unknown with its reservation kept and the pass continues. Run 37307615519 on 66fe628: 16 of 16 lanes and required succeeded.

TheodorAdrienIsaak Mattli added 2 commits October 5, 2026 11:30
…work

Compose the corrected receipts and transfers head with the corrected
Sui adapter head (prepared transfers with a persisted identity, coin
selection without repeated objects) as the base of the payout
reservation and refund change, which uses both.

GitHub issue #80.
Executor payouts zeroed the balance after the transfer, so earnings
credited in between were lost; refunds and payouts held a SQL
transaction across the chain call, and a transaction-wide refund wrote
order states unconditionally and refunded credited orders.

Give payouts and refunds one lifecycle: prepare the transfer (a chain
read), commit the local decision together with a reserved chain
transfer row carrying the digest, the signed transaction and its
signature, broadcast the signed transaction under a two-minute bound,
and record the outcome by a conditional write: sent, failed (a failed
payout releases its amount in the same write) or unknown. Reconcile
sent, unknown and stale reserved rows on the payout loop's ticker with
one lookup: confirmed with the expected effects, failed, or resubmitted
once from the identical stored bytes when the chain does not know the
transaction; anything unverified stays unknown. A payout reserves
exactly the balance it read and is refused while another is open;
refunds change the transaction status and each outstanding order
conditionally, with a refund settlement row, before the chain call.
The maintenance pass settles chain refunds of failed runs through the
same lifecycle and never selects an order with a refund transfer again.

GitHub issues #80 and #81.
@ctfbruce ctfbruce added the In progress Implementation is active. label Oct 5, 2026
@ctfbruce ctfbruce added Review Implementation is ready for review and required validation. and removed In progress Implementation is active. labels Oct 5, 2026
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 10:01
Bring the adapter branch at its corrected head, which validates a
listed coin's object id before normalizing it, under the payout and
refund lifecycle, so that the composed tree carries the correction the
transfer tests run against.

GitHub issues #80 and #81.
@ctfbruce
ctfbruce marked this pull request as draft October 5, 2026 11:07
@ctfbruce ctfbruce added In progress Implementation is active. Review Implementation is ready for review and required validation. and removed Review Implementation is ready for review and required validation. In progress Implementation is active. labels Oct 5, 2026
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 11:21
@ctfbruce
ctfbruce marked this pull request as draft October 5, 2026 11:58
@ctfbruce ctfbruce added In progress Implementation is active. and removed Review Implementation is ready for review and required validation. labels Oct 5, 2026
After a refund decision was committed, the transfer lifecycle returned
no outcome to its caller, so a buyer refused during maintenance was told
the payment had been refunded even when the transfer failed or its
outcome was unknown, and later requests repeated it. Return the
recorded outcome and word the refusal as sent, pending confirmation, or
recorded but not sent and owed; a repeated request reads the refund
transfer's state. The order stays unspendable after the decision.

A reconciliation lookup ran under the daemon's lifetime context on an
SDK path with no deadline, so one lookup that never completed stopped
every later row and all payouts. Bound each lookup; a timeout is
recorded as an unknown outcome with the reservation kept, and the pass
continues with the next row.

GitHub issues #80 and #81.
@ctfbruce ctfbruce added Review Implementation is ready for review and required validation. and removed In progress Implementation is active. labels Oct 5, 2026
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 12:32
Include the accepted runtime, deployment, monitoring and SDK updates alongside payout reservation and refund reconciliation.

GitHub issues #80 and #81.
@ctfbruce
ctfbruce marked this pull request as draft October 5, 2026 13:53
@ctfbruce
ctfbruce marked this pull request as ready for review October 5, 2026 14:03
@ctfbruce
ctfbruce merged commit 58683c2 into main Oct 5, 2026
16 checks passed
@vincent10400094
vincent10400094 deleted the fix/payout-reservation-and-refunds branch October 8, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Review Implementation is ready for review and required validation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant