Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
356 changes: 355 additions & 1 deletion api/openapi.yaml

Large diffs are not rendered by default.

6 changes: 4 additions & 2 deletions api/spec.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ var OpenAPI []byte
const (
// Version is the HTTP API contract version as major.minor. It equals the
// info.version field of OpenAPI.
Version = "1.14"
Version = "1.15"
// Major is the compatibility number of Version. Clients that require a
// different major version are answered with an explicit incompatibility
// error instead of a best-effort response.
Expand All @@ -35,7 +35,9 @@ const (
// 1.12 added measurement workflows, payload deletion and structured request field failures.
// 1.13 added linked provider identities and scoped browser-approved credentials.
// 1.14 added local allocation reclamation to recovery inspection.
Minor = 14
// 1.15 added payment quotes, the account's order history and the
// account's payment features.
Minor = 15
// VersionHeader carries the contract version a client requires on requests
// and the version the dispatcher implements on responses.
VersionHeader = "Debuglet-API-Version"
Expand Down
10 changes: 10 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,16 @@ A `TEST` intent has no payment backend: its transaction row is the intent, writt

Every intent records the name of the rule that priced it in the `pricing_rule` column of its transaction; the rule above is `bw-s-ceil-ms-v1`, and a transaction written before the column existed carries an empty name and was priced by the rule of the release that wrote it. A `USDC` intent is a local record of the price, the auth key and the addresses the payer uses, created without any chain call, so an intent that fails leaves nothing external to reconcile.

## Quotes and economics (API 1.15)

`POST /payment/quote` takes the body of `PUT /payment/intent`, under the same authentication and account request quota, and prices it with the rule described in [Pricing and TEST bookkeeping](#pricing-and-test-bookkeeping), named `bw-s-ceil-ms-v1`. Only `floor_bw` enters the price; `ceil_bw` does not. A `timeout_ms` of 0 is refused like any timeout outside its range: `timeout_ms must be positive and at most 9223372036854`. A quote writes nothing: no transaction, order, auth key or run, and no capacity is reserved. It answers `{pricing_rule, currency, unit, total, orders, errors}`, where every amount is a decimal string of integer base units of the currency and `unit` names them (`TEST units`, `micro-USDC`, otherwise `base units`). A batch the intent would refuse is still answered 200: each order lists its own problems in `errors` with an empty `price`, including a `start_time` outside its range and a policy the executor cannot currently serve (for example `require_icmp` on an executor without ICMP), reported with the field and reason submission would use; problems of the whole batch (an empty batch, a total above the signed 64-bit maximum) appear in the top-level `errors`, and `total` is empty unless every order was priced. A malformed body, an unsupported payment method, a chain method while blockchain payments are disabled and a missing credential keep the intent's status. `retry` is not evaluated.

The intent prices the batch again when it is created and returns the result as `quote` in its response, in the same shape, so a client can compare it with an earlier quote; a changed `pricing_rule` means the server's rule changed in between. An intent recovered through an explicit retry does not repeat its quote.

`GET /me` adds `economics`: `payment_methods` lists the methods the intent admits now (`TEST`, and `USDC` when blockchain payments are enabled), `chain_payments` whether blockchain payments are enabled and `allowances` whether usage allowances are offered, which is always false in this version.

`GET /me/orders` pages the caller's own payment intents, newest first. An intent has no stored creation time, so the order is that of its expiry time, which is creation plus five minutes, then of its ID. `limit` is 1 to 100 (default 25); pass the `next` value of a page as `before` to read the following one, and `next` is empty on the last page. Each intent reports `id`, `method`, `status` (`outstanding`, `paid`, `refunded`, `expired`, `aborted`, or `unknown`), its recorded `price` and `currency`, `pricing_rule` (empty when the dispatcher did not record one) and `expires_at`, and each of its orders `order_id`, `executor_id`, `price`, `currency`, `settlement` (`pending`, `credited` or `refunded`, as in run detail costs) and, once admitted, `run_id`. Another account's intents are never listed; a `before` value that is not one of your intents returns an empty page.

## Executor listing

API 1.9 also adds `admission`, operator `display` metadata, the executor-reported `scion_isd_as` and `listeners` to `GET /executors`. Each value carries a source label; see [executor discovery](operations/executor-discovery.md#vantage-point-metadata). It also adds the executor's host probes: `capabilities.icmp`, `capabilities.enforcement_reason` and `clock`. Host platform detail is operator-only and is never listed; it appears only in result provenance ([host probes](operations/executor-discovery.md#host-probes)).
Expand Down
8 changes: 6 additions & 2 deletions docs/operations/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,12 @@ whether it was applied, a duplicate, a mismatch, expired or for an unknown
intent), and
`chain_transfers`, the outbound payout and refund transfers and their states,
including each signed transaction and its signature, stored before broadcast.
Both are kept indefinitely for reconciliation; they stay empty while chain
payments are disabled.
A transfer row is `reserved` before the chain is called, then `sent`,
`confirmed`, `failed` (nothing moved; a failed payout's amount is back in the
balance) or `unknown` (the outcome is not known yet; reconciliation keeps
checking it). Both are kept indefinitely for reconciliation; they stay empty
while chain payments are disabled. See [chain payments](payments.md) for the
transfer lifecycle and what each state means for an operator.

## State and upgrades

Expand Down
84 changes: 84 additions & 0 deletions docs/operations/payments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Chain payments

This page describes how a dispatcher with chain payments enabled sends money:
executor payouts and buyer refunds. TEST payments move no funds and create none
of the records below. While `[sui] disabled = true`, nothing here runs.

## Transfer lifecycle

Every outbound transfer, payout or refund, follows the same steps and leaves
one row in the dispatcher's `chain_transfers` table.

1. **Prepare.** The dispatcher selects coins and gas, builds and signs the
transaction and computes its digest. This only reads from the chain. If it
fails, nothing is recorded and nothing is sent.
2. **Reserve.** One database transaction records the local decision and the
transfer row in state `reserved`, with the digest, the signed transaction
and its signature. For a payout the decision is subtracting exactly the
amount being paid from the executor's balance; earnings credited later stay
in the balance for the next payout. For a refund it is marking the orders
(and, for a transaction refund, the transaction) refunded with one refund
settlement record per order. The signed transaction is stored before it is
ever submitted, and no database transaction is held open while the chain is
called.
3. **Submit** the stored transaction.
4. **Record** the outcome:
- `sent`: the chain accepted the transaction and reported success.
- `failed`: the transaction was not submitted, or the chain executed it
with a failure status. No coins moved. A failed payout returns its amount
to the executor's balance in the same database transaction.
- `unknown`: the transaction may have been submitted, but the response was
lost or incomplete. The row keeps the amount reserved until it is
resolved.

A payout is not started while the same executor and currency has a `reserved`,
`sent` or `unknown` payout. A balance without a payout wallet is not paid out;
each payout pass logs how many such balances it skipped.

A failed run's refund is attempted when the run ends. If that attempt never
reserved its transfer, for example because the dispatcher stopped first, the
periodic settlement pass sends it. The pass never takes an order that already
has a refund transfer, whatever that transfer's state, so one order is
refunded by at most one transfer; an uncertain outcome is resolved by
reconciliation below. While chain payments are disabled, the pass leaves
chain-currency orders as they are.

## Reconciliation

Once a minute the dispatcher looks up, by digest, every `sent` and `unknown`
transfer and every `reserved` transfer older than ten minutes (left by a stop
between the reservation and the submission):

| Lookup result | New state |
| --- | --- |
| Executed successfully, and the receiver's credit matches the amount and coin | `confirmed` |
| Executed successfully, but the node returned no balance changes to check the credit against | `unknown` |
| Executed with failure status | `failed` (a payout's amount returns to the balance) |
| Not found, row `reserved` or `unknown` | the stored signed transaction is submitted again and the outcome recorded as above |
| Not found, row `sent` | `unknown` |
| Lookup error, or a credit that does not match | `unknown` |

Each lookup waits at most 30 seconds; one that does not answer leaves the
transfer `unknown` and the pass continues with the next transfer.

A resubmission sends the identical signed bytes, which have the same digest,
so the chain executes the transfer at most once. A transfer is never rebuilt
with new bytes because a response was lost.

## What an operator sees

The `detail` column of a `failed` or `unknown` row says why, and `updated_at`
is the time of the last check.

- **`failed` payout**: nothing was paid and the amount is back in the
executor's balance; the next payout pass tries again with a new transfer.
- **`failed` refund**: the orders stay refunded and the amount is owed to the
buyer's refund address. It is not retried automatically; the row is the
record of what is owed. A buyer whose paid order is refunded because
submissions are paused is told whether the refund was sent, is pending
confirmation, or could not be sent and is owed.
- **`unknown`**: reconciliation keeps checking it every minute. A payout in
this state blocks further payouts of that executor and currency, so the
amount cannot be paid twice. If it stays `unknown`, for example because the
node never returns balance changes or the stored transaction can no longer
be executed, check the digest on a chain explorer before changing anything.
2 changes: 2 additions & 0 deletions docs/versions.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,5 @@ API 1.12 adds measurement profiles, batch details, explicit retry lineage, liste
Dispatcher schema 21 records one settlement per order and the pricing rule of each transaction; this build refuses an older dispatcher database, so upgrade it explicitly before starting this build.

Dispatcher schema 22 records chain payment receipts and outbound chain transfers. The receipt listener and the transfer code depend on it, so this build requires schema 22; upgrade explicitly before starting it. The HTTP API is unchanged.

API 1.15 adds `POST /payment/quote`, the `quote` field of the payment intent response, the `economics` field of `GET /me` and `GET /me/orders`. The quote and order history require 1.15 from the Go client; no dispatcher schema change is involved.
15 changes: 15 additions & 0 deletions internal/dispatcher/capability_admission.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,21 @@ type CapabilityError struct {
func (e *CapabilityError) Error() string { return e.Message }
func (e *CapabilityError) Unwrap() error { return ErrInvalidPolicy }

// CheckExecutorCapabilities applies the capability check of admission to an
// order that is not being admitted: it reports whether the available executor
// can serve what policy requires of it now, with the refusal admission would
// return. It reads the registry only. An executor that is not registered or not
// available is ErrUnknownExecutor.
func (d *Dispatcher) CheckExecutorCapabilities(executorID string, orderID int64, policy models.DebugletPolicy) error {
d.mu.RLock()
defer d.mu.RUnlock()
exec, exists := d.executors[executorID]
if d.closed || !exists || !exec.owner.Available() {
return ErrUnknownExecutor
}
return validateExecutorCapabilities(&models.DebugletSpec{OrderID: orderID, Policy: policy}, exec, d.now())
}

func validateExecutorCapabilities(spec *models.DebugletSpec, exec *executorEntry, now time.Time) error {
refuse := func(field, code, message string) error {
return &CapabilityError{Field: "policy." + field, Code: code, Message: message, OrderID: spec.OrderID}
Expand Down
126 changes: 126 additions & 0 deletions internal/dispatcher/database/economics.sql.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 24 additions & 0 deletions internal/dispatcher/database/queries/economics.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
-- name: ListAccountIntents :many
-- One page of an account's payment intents, newest expiry first. A cursor
-- names the last intent of the previous page; one that is not the account's
-- own intent selects nothing.
SELECT t.id, t.method, t.status, t.price, t.currency, t.expires_at, t.pricing_rule
FROM transactions t
JOIN transaction_users tu ON tu.transaction_id = t.id
JOIN users u ON u.id = tu.user_id
WHERE u.uuid = sqlc.arg(user_uuid)
AND (CAST(sqlc.arg(before) AS TEXT) = '' OR EXISTS (
SELECT 1 FROM transactions c
JOIN transaction_users cu ON cu.transaction_id = c.id
WHERE c.id = sqlc.arg(before) AND cu.user_id = tu.user_id
AND (t.expires_at < c.expires_at OR (t.expires_at = c.expires_at AND t.id < c.id))))
ORDER BY t.expires_at DESC, t.id DESC
LIMIT sqlc.arg(page_limit);

-- name: ListIntentOrders :many
-- The orders of one payment intent with the run each one admitted, if any.
SELECT o.order_id, o.executor_id, o.price, o.currency, o.state, d.uuid AS run_id
FROM debuglet_order o
LEFT JOIN debuglets d ON d.id = o.debuglet_id
WHERE o.transaction_id = ?
ORDER BY o.order_id;
43 changes: 43 additions & 0 deletions internal/dispatcher/database/queries/settlement.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,46 @@ ORDER BY observed_at, tx_digest, event_seq;
UPDATE transactions
SET status = sqlc.arg(paid)
WHERE id = sqlc.arg(id) AND status = sqlc.arg(outstanding);

-- name: ReservePayout :execrows
UPDATE earnings
SET current_balance = current_balance - sqlc.arg(amount)
WHERE earnings.executor_id = sqlc.arg(executor_id) AND earnings.currency = sqlc.arg(currency)
AND earnings.current_balance >= sqlc.arg(amount)
AND NOT EXISTS (
SELECT 1 FROM chain_transfers t
WHERE t.kind = 'payout' AND t.executor_id = sqlc.arg(executor_id) AND t.currency = sqlc.arg(currency)
AND t.state IN ('reserved', 'sent', 'unknown')
);

-- name: ReleasePayout :exec
UPDATE earnings
SET current_balance = current_balance + sqlc.arg(amount)
WHERE executor_id = sqlc.arg(executor_id) AND currency = sqlc.arg(currency);

-- name: InsertChainTransfer :one
INSERT INTO chain_transfers (kind, executor_id, transaction_id, order_id, amount, currency, receiver, state, digest, signed_transaction, signature, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
RETURNING id;

-- name: UpdateChainTransfer :execrows
UPDATE chain_transfers
SET state = sqlc.arg(state), digest = sqlc.arg(digest), detail = sqlc.arg(detail), updated_at = sqlc.arg(updated_at)
WHERE id = sqlc.arg(id) AND state = sqlc.arg(from_state);

-- name: ListUnresolvedChainTransfers :many
SELECT * FROM chain_transfers
WHERE state IN ('sent', 'unknown') OR (state = 'reserved' AND updated_at < sqlc.arg(reserved_before))
ORDER BY updated_at, id
LIMIT sqlc.arg(row_limit);

-- name: RefundPaidTransaction :execrows
UPDATE transactions
SET status = sqlc.arg(refunded)
WHERE id = sqlc.arg(id) AND status = sqlc.arg(paid);

-- name: GetLatestRefundTransferState :one
SELECT state FROM chain_transfers
WHERE kind = 'refund' AND transaction_id = ?
ORDER BY id DESC
LIMIT 1;
10 changes: 8 additions & 2 deletions internal/dispatcher/database/queries/transactions.sql
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,19 @@ SELECT * FROM order_settlements
WHERE transaction_id = ? AND order_id = ?;

-- name: ListPendingSettlements :many
-- Only wholly local settlements: credits of any currency and TEST refunds.
-- An order covered by a refund transfer in any state is never listed: the
-- transfer's own reconciliation resolves it, so a pass cannot send it twice.
SELECT sqlc.embed(d), e.exit_code, o.currency FROM debuglet_order o
JOIN debuglets d ON d.id = o.debuglet_id
AND d.transaction_id = o.transaction_id AND d.order_id = o.order_id
JOIN measurement_execution e ON e.debuglet_id = d.id
WHERE o.state = sqlc.arg(outstanding_state) AND d.state = sqlc.arg(exited_state)
AND e.exit_code IS NOT NULL AND (e.exit_code = 0 OR o.currency = 'TEST')
AND e.exit_code IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM chain_transfers t
WHERE t.kind = 'refund' AND t.transaction_id = o.transaction_id
AND (t.order_id IS NULL OR t.order_id = o.order_id)
)
AND d.id > sqlc.arg(after_id)
ORDER BY d.id
LIMIT sqlc.arg(row_limit);
Expand Down
Loading