From 483c0fe63c18f3b1f4ec85265aae9e98c6c02467 Mon Sep 17 00:00:00 2001 From: rob Date: Fri, 25 Sep 2026 20:03:41 +0200 Subject: [PATCH 1/3] slipstream: take txs from anyone, mine them without relaying A new service, slipstream/, alongside payout/. It takes a raw tx over HTTP and hands it straight to the enforcer's block template server -- the one the proxy mines from -- which keeps it in its template mempool and never relays it (LayerTwo-Labs/bip300301_enforcer#642). That is how the pool mines BIP300/301 txs the network will not carry, or any other consensus-valid tx. The fee rule is Slipstream's: the higher of a floor (1 sat/vB) and the current mineable rate, read off the template being mined. A tx that pays less is taken back out of the enforcer before the refusal returns. Every submission is kept, refused ones included, and each accepted tx is followed from template to block: pending, in_template, mined, confirmed, or dropped/expired with a reason. The enforcer forgets a tx on restart or reorg; this side remembers, so it resubmits. It keeps its own slipstream.db and only reads shares.db, for pool_meta and blocks_found. info.json is served here: facts from pool_meta, with mode the exact pool_mode, presentation from env. tests/test_slipstream_regtest.sh runs it against a real enforcer built with --enable-slipstream. It is not in CI until an enforcer release ships the flag; the unit tests are. --- .github/workflows/build_docker.yaml | 6 +- .github/workflows/integration_tests.yaml | 8 + README.md | 17 +- deploy/docker/Dockerfile.slipstream | 21 + deploy/docker/docker-compose.yml | 41 +- scripts/regtest/start.sh | 6 +- slipstream/.gitignore | 1 + slipstream/README.md | 128 +++++++ slipstream/index.js | 68 ++++ slipstream/lib/config.js | 107 ++++++ slipstream/lib/fees.js | 46 +++ slipstream/lib/http.js | 170 +++++++++ slipstream/lib/info.js | 54 +++ slipstream/lib/pool.js | 61 +++ slipstream/lib/rpc.js | 121 ++++++ slipstream/lib/slipstream.js | 292 ++++++++++++++ slipstream/lib/store.js | 189 +++++++++ slipstream/package-lock.json | 467 +++++++++++++++++++++++ slipstream/package.json | 17 + slipstream/test/fees.test.js | 28 ++ slipstream/test/helpers.js | 108 ++++++ slipstream/test/http.test.js | 79 ++++ slipstream/test/info.test.js | 73 ++++ slipstream/test/submit.test.js | 66 ++++ slipstream/test/tracker.test.js | 147 +++++++ tests/README.md | 11 + tests/test_slipstream_regtest.sh | 250 ++++++++++++ 27 files changed, 2577 insertions(+), 5 deletions(-) create mode 100644 deploy/docker/Dockerfile.slipstream create mode 100644 slipstream/.gitignore create mode 100644 slipstream/README.md create mode 100644 slipstream/index.js create mode 100644 slipstream/lib/config.js create mode 100644 slipstream/lib/fees.js create mode 100644 slipstream/lib/http.js create mode 100644 slipstream/lib/info.js create mode 100644 slipstream/lib/pool.js create mode 100644 slipstream/lib/rpc.js create mode 100644 slipstream/lib/slipstream.js create mode 100644 slipstream/lib/store.js create mode 100644 slipstream/package-lock.json create mode 100644 slipstream/package.json create mode 100644 slipstream/test/fees.test.js create mode 100644 slipstream/test/helpers.js create mode 100644 slipstream/test/http.test.js create mode 100644 slipstream/test/info.test.js create mode 100644 slipstream/test/submit.test.js create mode 100644 slipstream/test/tracker.test.js create mode 100755 tests/test_slipstream_regtest.sh diff --git a/.github/workflows/build_docker.yaml b/.github/workflows/build_docker.yaml index 44fb9a8..a54de89 100644 --- a/.github/workflows/build_docker.yaml +++ b/.github/workflows/build_docker.yaml @@ -1,4 +1,4 @@ -# Build the three simplepool container images and push them to the GitHub +# Build the four simplepool container images and push them to the GitHub # Container Registry (ghcr.io) — the same registry coinshift-rs publishes to. # # This job runs only in the canonical LayerTwo-Labs repo (see the `if:` guard @@ -47,6 +47,8 @@ jobs: dockerfile: deploy/docker/Dockerfile.dashboard - image: simplepool-payout dockerfile: deploy/docker/Dockerfile.payout + - image: simplepool-slipstream + dockerfile: deploy/docker/Dockerfile.slipstream steps: - name: Checkout uses: actions/checkout@v4 @@ -88,6 +90,6 @@ jobs: github.event.pull_request.head.repo.full_name == github.repository }} tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} - # Per-image GitHub Actions cache so the three legs don't collide. + # Per-image GitHub Actions cache so the legs don't collide. cache-from: type=gha,scope=${{ matrix.image }} cache-to: type=gha,mode=max,scope=${{ matrix.image }} diff --git a/.github/workflows/integration_tests.yaml b/.github/workflows/integration_tests.yaml index b6fab09..acaa230 100644 --- a/.github/workflows/integration_tests.yaml +++ b/.github/workflows/integration_tests.yaml @@ -126,6 +126,14 @@ jobs: working-directory: payout run: npm test + - name: Install slipstream dependencies + working-directory: slipstream + run: npm ci || npm install + + - name: Run slipstream service tests + working-directory: slipstream + run: npm test + # The payout path the coinbase e2e skips: wallet-enabled enforcer + # thunder, a real deposit, and one payout tick broadcasting a Thunder # transaction (tests/test_payout_regtest.sh). Separate job purely for diff --git a/README.md b/README.md index 4ba0f82..dbf9ba6 100644 --- a/README.md +++ b/README.md @@ -579,7 +579,7 @@ you can commit + push from here), use ### Docker An alternative to the bare-metal script: containerized builds of the -three services (stratum proxy, dashboard, payout worker) under +four services (stratum proxy, dashboard, payout worker, slipstream) under [`deploy/docker/`](deploy/docker/). One `docker compose up -d --build` gets the whole app stack running against a Thunder daemon and Bitcoin Core that live on the host (or wherever you point them). Shared bind @@ -592,6 +592,19 @@ Note: the drivechain infrastructure (`bitcoind`, Thunder, the those daemons have their own lifecycles and typically run bare-metal on the same host. +### Slipstream + +[`slipstream/`](slipstream/) is an optional service that takes a raw tx from +anyone and gets it into the pool's blocks **without relaying it** — for BIP300/ +301 txs (deposits, withdrawal bundles, BMM requests) the network will not relay, +or any other consensus-valid tx. It hands each one to the enforcer's block +template server, the same one the proxy mines from, which needs +`--enable-slipstream` (LayerTwo-Labs/bip300301_enforcer#642). The fee rule is +Slipstream's: the higher of a 1 sat/vB floor and the current mineable rate. +Every submission is kept, and each accepted tx is followed from template to +block. It also serves the pool's `info.json` for pool directories. See +[`slipstream/README.md`](slipstream/README.md). + ## Config keys ``` @@ -748,6 +761,8 @@ scripts/ dashboard/ # Node/Express read-only stats UI payout/ # payout worker: Thunder rail (pps-classic, pplns-thunder) # and L1 rail via the enforcer wallet (pplns-btc) +slipstream/ # slipstream service: takes txs from anyone and mines them + # without relaying, via the enforcer; serves info.json docs/simplepool.html # single-file explainer: every mode, end to end ``` diff --git a/deploy/docker/Dockerfile.slipstream b/deploy/docker/Dockerfile.slipstream new file mode 100644 index 0000000..3a48757 --- /dev/null +++ b/deploy/docker/Dockerfile.slipstream @@ -0,0 +1,21 @@ +# ----------------------------------------------------------------------------- +# simplepool slipstream service — private tx submission (Node.js) +# ----------------------------------------------------------------------------- + +FROM node:20-bookworm-slim AS deps +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 make g++ ca-certificates \ + && rm -rf /var/lib/apt/lists/* +WORKDIR /app +COPY slipstream/package.json slipstream/package-lock.json ./ +RUN npm ci --omit=dev + +# ----------------------------------------------------------------------------- +FROM node:20-bookworm-slim +RUN apt-get update && apt-get install -y --no-install-recommends tini \ + && rm -rf /var/lib/apt/lists/* +WORKDIR /app +COPY --from=deps /app/node_modules ./node_modules +COPY slipstream/ ./ +USER node +ENTRYPOINT ["/usr/bin/tini", "--", "node", "index.js"] diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml index ae8285a..f2d0b0b 100644 --- a/deploy/docker/docker-compose.yml +++ b/deploy/docker/docker-compose.yml @@ -1,7 +1,7 @@ # ----------------------------------------------------------------------------- # simplepool — docker-compose orchestration # ----------------------------------------------------------------------------- -# Runs the three simplepool services against a Thunder daemon that lives on +# Runs the four simplepool services against a Thunder daemon that lives on # the host (or wherever THUNDER_RPC_URL points). Bitcoin Core is likewise # expected to be reachable from the simplepool container. # @@ -102,3 +102,42 @@ services: extra_hosts: - "host.docker.internal:host-gateway" stop_grace_period: 30s + + slipstream: + build: + context: ../.. + dockerfile: deploy/docker/Dockerfile.slipstream + image: simplepool-slipstream:latest + container_name: simplepool-slipstream + restart: unless-stopped + environment: + # The enforcer's block template server -- the proxy's bitcoind_url -- + # running with --enable-slipstream. + ENFORCER_GBT_URL: ${ENFORCER_GBT_URL:-http://host.docker.internal:8122} + BITCOIND_RPC_URL: ${BITCOIND_RPC_URL:-} + BITCOIND_RPC_USER: ${BITCOIND_RPC_USER:-} + BITCOIND_RPC_PASS: ${BITCOIND_RPC_PASS:-} + SLIPSTREAM_DB_PATH: /data/slipstream.db + PROXY_DB_PATH: /data/shares.db + SLIPSTREAM_BIND: 0.0.0.0 + SLIPSTREAM_PORT: "8124" + SLIPSTREAM_TRUST_PROXY: ${SLIPSTREAM_TRUST_PROXY:-0} + SLIPSTREAM_MIN_FEE_RATE: ${SLIPSTREAM_MIN_FEE_RATE:-1} + POOL_NAME: ${POOL_NAME:-} + POOL_OPERATOR: ${POOL_OPERATOR:-} + POOL_CHAIN: ${POOL_CHAIN:-} + POOL_LOGO: ${POOL_LOGO:-} + POOL_CONTACT: ${POOL_CONTACT:-} + PUBLIC_STRATUM_URL: ${PUBLIC_STRATUM_URL:-} + PUBLIC_DASHBOARD_URL: ${PUBLIC_DASHBOARD_URL:-} + PUBLIC_SLIPSTREAM_URL: ${PUBLIC_SLIPSTREAM_URL:-} + ports: + # Loopback on the host: publish through nginx, which is also what + # SLIPSTREAM_TRUST_PROXY=1 assumes sets X-Forwarded-For. + - "127.0.0.1:${SLIPSTREAM_PORT:-8124}:8124" + volumes: + # Writes slipstream.db; reads shares.db. + - ../../data:/data + extra_hosts: + - "host.docker.internal:host-gateway" + stop_grace_period: 10s diff --git a/scripts/regtest/start.sh b/scripts/regtest/start.sh index 6d09c8a..0cdc3c6 100755 --- a/scripts/regtest/start.sh +++ b/scripts/regtest/start.sh @@ -15,6 +15,9 @@ # CreateDepositTransaction) are unavailable. # Mine with MiningService/GenerateToAddress # (enforcer PR #477). +# REGTEST_ENFORCER_EXTRA_ARGS +# further enforcer flags, space-separated, e.g. +# --enable-slipstream (tests/test_slipstream_regtest.sh) set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" @@ -119,7 +122,8 @@ start_if_dead bip300301_enforcer \ --node-zmq-addr-sequence=tcp://127.0.0.1:$BITCOIND_ZMQ_PORT \ --enable-block-template-server \ --serve-rpc-addr=127.0.0.1:$ENFORCER_RPC_PORT \ - --serve-grpc-addr=127.0.0.1:$ENFORCER_GRPC_PORT + --serve-grpc-addr=127.0.0.1:$ENFORCER_GRPC_PORT \ + ${REGTEST_ENFORCER_EXTRA_ARGS:-} wait_for bip300301_enforcer "nc -z 127.0.0.1 $ENFORCER_RPC_PORT" 30 # Thunder connects to the enforcer's gRPC — make sure that's actually diff --git a/slipstream/.gitignore b/slipstream/.gitignore new file mode 100644 index 0000000..c2658d7 --- /dev/null +++ b/slipstream/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/slipstream/README.md b/slipstream/README.md new file mode 100644 index 0000000..e89d8b3 --- /dev/null +++ b/slipstream/README.md @@ -0,0 +1,128 @@ +# simplepool-slipstream + +Takes a raw transaction from anyone and gets it into the pool's blocks +**without relaying it**. The first use is BIP300/301 transactions that the +network will not relay, such as deposits, withdrawal bundles and BMM requests, +but any consensus-valid transaction qualifies. + +``` +submitter ──POST /api/tx──▶ slipstream ──submitslipstreamtx──▶ enforcer :8122 ◀── simplepool proxy + │ (template mempool) (mines it) + └── follows it: template → block → confirmed +``` + +The transaction goes straight to the enforcer's block template server, the +same one the proxy mines from. The enforcer has the node check it for +consensus validity and keeps it in its template mempool. It is never handed to +the node's mempool and never relayed. The node's relay policy does not apply, +but consensus does, and so do the enforcer's own BIP300 rules: a deposit that +skips the sidechain's CTIP is refused. + +**Requires** `bip300301_enforcer` running with `--enable-block-template-server +--enable-slipstream` (LayerTwo-Labs/bip300301_enforcer#642). The enforcer's +template server has no authentication, so keep `:8122` on loopback. This +service is the public face. + +## The fee rule + +It follows Slipstream's rule: a transaction must pay **the higher of the +minimum submission rate** (`SLIPSTREAM_MIN_FEE_RATE`, default 1 sat/vB) **and +the current mineable rate**. + +- **Mineable rate.** This is read from the template the proxy is mining now. + While the template has room, it equals the floor. Once the template is full, + it is the fee rate of the cheapest transaction in it. +- **When it's checked.** The rate is checked once, at submission. Accepted + transactions then compete for the block by fee rate like any other. A + transaction that loses its place stays pending until it is mined or expires. + +A transaction that passes the enforcer but pays too little is removed from the +enforcer again before the refusal is returned, so it cannot be mined for less +than was asked. + +## API + +Every endpoint is readable cross-origin. + +| | | +|---|---| +| `GET /info.json` (also `/api/info`) | The pool, for pool directories. | +| `GET /api/fees` | `{min_submission_rate, mineable_rate, required_rate, template_height, template_weight, updated_at}` | +| `POST /api/tx` (also `/tx`) | The raw transaction as hex, either as a text body or as JSON `{"hex": "..."}`. Returns `{accepted: true, txid, status, ...}` or `{accepted: false, reject_reason}`. Reject reasons are the node's (`missing-inputs`, `bad-txns-...`, `mandatory-script-verify-flag-failed ...`) or this service's (`fee-rate-too-low`, `invalid-hex`, `tx-size`, `rate-limited`, `rejected-by-enforcer`). | +| `GET /api/tx/:txid` | `{tx, events}`: where it stands now, and every status change it went through. | +| `GET /api/txs?status=&limit=` | Recent transactions, newest first. | +| `GET /healthz` | `503` while the enforcer is unreachable. | + +``` +curl -s -X POST --data-binary "$RAW_TX_HEX" https://slipstream.example/api/tx +``` + +## What "propagated" means here + +Nothing is ever relayed, so the statuses track only two things: whether the +transaction reached a template, and whether a block carried it. + +| status | | +|---|---| +| `pending` | In the enforcer's mempool, but not in the latest template. | +| `in_template` | In the latest template, so miners are working on it now. | +| `mined` | In a block, fewer than `SLIPSTREAM_CONFIRMATIONS` (default 6) deep. `mined_by_pool` says whether the block was ours (from `blocks_found`). | +| `confirmed` | That deep. | +| `dropped` | Will not be mined. `status_reason` says why: `conflict_mined` (a block spent one of its inputs), `parent_removed`, `rejected_by_enforcer`, `withdrawn`, or the node's reason for refusing a resubmission. | +| `expired` | Waited `SLIPSTREAM_EXPIRY_BLOCKS` (default 144) without being mined, and was withdrawn. | + +This service keeps the record, and the enforcer keeps only its mempool. When +the enforcer loses a transaction, the service resubmits it and counts it in +`resubmissions`. That covers: +- an enforcer restart +- a reorg, because the enforcer evicts every slipstream transaction when a block disconnects +- the transaction's block being orphaned + +## Storage + +The service keeps its own database, `SLIPSTREAM_DB_PATH` (default +`../data/slipstream.db`). It **never writes to `shares.db`**, which it reads +only for `pool_meta` and `blocks_found`. + +- `slipstream_submissions`: every POST exactly as it arrived, accepted or not. +- `slipstream_txs`: one row per accepted transaction, including the raw + transaction so it can be resubmitted. +- `slipstream_events`: every status change, append-only. + +## info.json + +Facts about the pool are read from `pool_meta`, which the proxy writes: `mode` +(the exact `pool_mode`), `fee_bps`, `coinbase_tag`, `operator_address` and +`pool_btc_address`. They are never configured here, so they cannot disagree +with what the coinbase actually does. Presentation fields come from +environment variables: +- `POOL_NAME`, `POOL_OPERATOR`, `POOL_LOGO`, `POOL_CONTACT` +- `POOL_CHAIN`, for a name like `betanet` that `pool_meta.network` cannot express +- `POOL_PAYOUT_TEXT`, which overrides the default sentence for each mode +- `PUBLIC_STRATUM_URL`, `PUBLIC_DASHBOARD_URL`, `PUBLIC_SLIPSTREAM_URL` + +`status_url` is the dashboard's `/api/status`. + +## Run + +``` +cd slipstream && npm ci +ENFORCER_GBT_URL=http://127.0.0.1:8122 \ +BITCOIND_RPC_URL=http://127.0.0.1:8332 BITCOIND_RPC_COOKIE_FILE=/var/lib/bitcoind/.cookie \ +PROXY_DB_PATH=../data/shares.db \ +PUBLIC_SLIPSTREAM_URL=https://slipstream.example \ +node index.js +``` + +Without `BITCOIND_RPC_URL`, a mined transaction's depth is only estimated +from the template height, and an orphaned block goes unnoticed. The full list +of variables is in [`lib/config.js`](lib/config.js). The service listens on +`127.0.0.1:8124`, and is published through nginx with +`SLIPSTREAM_TRUST_PROXY=1`, so the rate limit applies per client. + +## Tests + +- `npm test`: unit tests. These run against a fake enforcer and bitcoind, and + cover every status transition. +- `tests/test_slipstream_regtest.sh`: end to end, against a real enforcer + (see [`tests/README.md`](../tests/README.md)). diff --git a/slipstream/index.js b/slipstream/index.js new file mode 100644 index 0000000..cd65d13 --- /dev/null +++ b/slipstream/index.js @@ -0,0 +1,68 @@ +/* simplepool-slipstream — take txs from anyone, mine them without relaying. + * + * Submissions go straight to the enforcer's block template server, the one + * the proxy mines from, which keeps them in its template mempool and never + * hands them to the node's. This service records every submission, holds + * each one to the fee rule, and follows it from template to block. + * + * Config is environment-only — see lib/config.js for the full list. + * + * Run: + * ENFORCER_GBT_URL=http://127.0.0.1:8122 \ + * BITCOIND_RPC_URL=http://127.0.0.1:8332 BITCOIND_RPC_COOKIE_FILE=... \ + * PROXY_DB_PATH=../data/shares.db \ + * node index.js + */ + +import { readFileSync } from 'node:fs'; + +import { loadConfig } from './lib/config.js'; +import { openStore } from './lib/store.js'; +import { openPoolDb } from './lib/pool.js'; +import { EnforcerClient, BitcoindClient } from './lib/rpc.js'; +import { Slipstream } from './lib/slipstream.js'; +import { startHttp } from './lib/http.js'; + +const cfg = loadConfig(); +const { version } = JSON.parse(readFileSync(new URL('./package.json', import.meta.url), 'utf8')); + +const log = { + debug: (m) => process.env.SLIPSTREAM_DEBUG === '1' && console.log(`[debug] ${m}`), + info: (m) => console.log(`[info] ${m}`), + warn: (m) => console.warn(`[warn] ${m}`), + error: (m) => console.error(`[error] ${m}`), +}; + +log.info(`simplepool-slipstream ${version} starting (enforcer=${cfg.enforcerUrl} db=${cfg.dbPath})`); +log.info(` fee floor ${cfg.minFeeRate} sat/vB, confirmed at ${cfg.confirmations}, ` + + `expiry ${cfg.expiryBlocks} blocks, poll ${cfg.pollMs}ms`); +if (!cfg.bitcoind) { + log.warn('BITCOIND_RPC_URL unset: mined txs are never checked for orphaning, ' + + 'and their depth is only estimated from the template height'); +} + +const slipstream = new Slipstream({ + store: openStore(cfg.dbPath), + enforcer: new EnforcerClient({ url: cfg.enforcerUrl }), + bitcoind: cfg.bitcoind ? new BitcoindClient(cfg.bitcoind) : null, + pool: openPoolDb(cfg.proxyDbPath), + cfg, + log, +}); + +let lastTickError = null; +async function loop() { + try { + await slipstream.tick(); + if (lastTickError) log.info('enforcer reachable again'); + lastTickError = null; + } catch (e) { + // Logged once per distinct failure, not once per poll + if (e.message !== lastTickError) log.warn(`tick failed: ${e.message}`); + lastTickError = e.message; + } + setTimeout(loop, cfg.pollMs); +} + +startHttp({ slipstream, pool: slipstream.pool, cfg, version, log }); +loop(); diff --git a/slipstream/lib/config.js b/slipstream/lib/config.js new file mode 100644 index 0000000..4b67b02 --- /dev/null +++ b/slipstream/lib/config.js @@ -0,0 +1,107 @@ +/* Slipstream service config, loaded from environment variables. + * + * Required: + * ENFORCER_GBT_URL the enforcer's block template server, the same + * endpoint the proxy mines from (bitcoind_url in + * proxy.conf), e.g. http://127.0.0.1:8122. The + * enforcer must run with --enable-slipstream. + * + * Optional: + * SLIPSTREAM_DB_PATH this service's own SQLite file (default + * ../data/slipstream.db). Every submission and + * every status change is kept here. Not shares.db: + * this service never writes to the proxy's + * database, whose writer is the proxy's share + * path, and nothing here is worth contending + * with it for. + * PROXY_DB_PATH shares.db, opened read-only (default + * ../data/shares.db). Pool identity for info.json + * comes from pool_meta, and blocks_found says + * whether the block that mined a tx was ours. + * Without it both are reported as unknown. + * BITCOIND_RPC_URL bitcoind JSON-RPC, read-only use (getblockheader, + * getrawtransaction). Without it a mined tx is + * never confirmed or noticed orphaned: the + * enforcer's template server cannot answer either. + * BITCOIND_RPC_USER / BITCOIND_RPC_PASS, or BITCOIND_RPC_COOKIE_FILE + * SLIPSTREAM_PORT HTTP port (default 8124) + * SLIPSTREAM_BIND bind address (default 127.0.0.1; put nginx in + * front to publish it) + * SLIPSTREAM_TRUST_PROXY '1' = take the client address from + * X-Forwarded-For, for rate limiting. Only behind a + * proxy that sets it, or anyone can pick theirs. + * SLIPSTREAM_MIN_FEE_RATE floor in sat/vB (default 1). The required rate is + * the higher of this and the current mineable rate, + * as Slipstream states it. + * SLIPSTREAM_CONFIRMATIONS depth at which a mined tx counts as confirmed + * (default 6) + * SLIPSTREAM_EXPIRY_BLOCKS blocks a tx may wait unmined before it is + * withdrawn (default 144, about a day) + * SLIPSTREAM_POLL_MS how often to read the template and follow every + * open tx (default 5000) + * SLIPSTREAM_RATE_LIMIT_PER_MIN + * submissions per client address per minute + * (default 30) + * + * info.json presentation fields — facts about the pool (mode, fee, coinbase + * tag, addresses) are never configured here, they are read from pool_meta, + * which the proxy writes: + * POOL_NAME, POOL_OPERATOR, POOL_LOGO, POOL_CONTACT + * POOL_CHAIN chain name as the pool advertises it, e.g. + * 'betanet'. pool_meta.network can only say main, + * test, signet or regtest. Defaults to that. + * POOL_PAYOUT_TEXT overrides the per-mode default payout sentence + * PUBLIC_STRATUM_URL, PUBLIC_DASHBOARD_URL, PUBLIC_SLIPSTREAM_URL + */ + +function num(name, dflt, { min = -Infinity } = {}) { + const raw = process.env[name]; + if (raw === undefined || raw === '') return dflt; + const v = Number(raw); + if (!Number.isFinite(v) || v < min) { + throw new Error(`${name}=${raw}: expected a number >= ${min}`); + } + return v; +} + +function str(name) { + const v = process.env[name]; + return v === undefined || v === '' ? null : v; +} + +export function loadConfig() { + const enforcerUrl = str('ENFORCER_GBT_URL'); + if (!enforcerUrl) { + throw new Error('ENFORCER_GBT_URL is required (the enforcer block template server)'); + } + return { + enforcerUrl, + dbPath: str('SLIPSTREAM_DB_PATH') || '../data/slipstream.db', + proxyDbPath: str('PROXY_DB_PATH') || '../data/shares.db', + bitcoind: str('BITCOIND_RPC_URL') ? { + url: str('BITCOIND_RPC_URL'), + user: str('BITCOIND_RPC_USER'), + pass: str('BITCOIND_RPC_PASS'), + cookieFile: str('BITCOIND_RPC_COOKIE_FILE'), + } : null, + port: num('SLIPSTREAM_PORT', 8124, { min: 1 }), + bind: str('SLIPSTREAM_BIND') || '127.0.0.1', + trustProxy: process.env.SLIPSTREAM_TRUST_PROXY === '1', + minFeeRate: num('SLIPSTREAM_MIN_FEE_RATE', 1, { min: 0 }), + confirmations: num('SLIPSTREAM_CONFIRMATIONS', 6, { min: 1 }), + expiryBlocks: num('SLIPSTREAM_EXPIRY_BLOCKS', 144, { min: 1 }), + pollMs: num('SLIPSTREAM_POLL_MS', 5000, { min: 100 }), + rateLimitPerMin: num('SLIPSTREAM_RATE_LIMIT_PER_MIN', 30, { min: 1 }), + presentation: { + name: str('POOL_NAME'), + operator: str('POOL_OPERATOR'), + logo: str('POOL_LOGO'), + contact: str('POOL_CONTACT'), + chain: str('POOL_CHAIN'), + payoutText: str('POOL_PAYOUT_TEXT'), + stratumUrl: str('PUBLIC_STRATUM_URL'), + dashboardUrl: str('PUBLIC_DASHBOARD_URL'), + slipstreamUrl: str('PUBLIC_SLIPSTREAM_URL'), + }, + }; +} diff --git a/slipstream/lib/fees.js b/slipstream/lib/fees.js new file mode 100644 index 0000000..c6edd80 --- /dev/null +++ b/slipstream/lib/fees.js @@ -0,0 +1,46 @@ +/* The fee rule, as Slipstream states it: a tx must pay the higher of the + * minimum submission rate and the current mineable rate. + * + * The mineable rate is read off the template the proxy is mining now. While + * the template has room, anything over the floor gets in, so the floor is the + * mineable rate. Once it is full, the cheapest tx it carries is what a new tx + * has to beat. Per tx, not per package: a cheap parent carried by its child + * reads lower than it was really priced at, which only ever errs towards + * accepting. + */ + +/* Weight a template can carry: 4M less what the header and a coinbase take. + * The coinbase is the one piece that varies, and a pplns-coinbase one paying + * a window of miners is the heaviest, so reserve for that. */ +export const TEMPLATE_WEIGHT_CAPACITY = 4_000_000 - 8_000; + +/* A template this close to capacity is full: nothing typical fits after it. */ +export const FULL_MARGIN_WEIGHT = 4_000; + +export const vsizeOf = (weight) => Math.ceil(weight / 4); + +/* Rate in sat/vB, rounded to 3 places so a stored rate reads back as it was + * shown. */ +export const feeRate = (feeSats, vsize) => Math.round((feeSats / vsize) * 1000) / 1000; + +export function mineableRate(template, floor) { + const txs = template?.transactions ?? []; + const used = txs.reduce((w, tx) => w + (tx.weight ?? 0), 0); + if (txs.length === 0 || used < TEMPLATE_WEIGHT_CAPACITY - FULL_MARGIN_WEIGHT) { + return floor; + } + const cheapest = Math.min(...txs.map(tx => feeRate(tx.fee ?? 0, vsizeOf(tx.weight)))); + return Math.max(floor, cheapest); +} + +/* What /api/fees reports, and what a submission is held to. */ +export function feeSnapshot(template, floor) { + const mineable = mineableRate(template, floor); + return { + min_submission_rate: floor, + mineable_rate: mineable, + required_rate: Math.max(floor, mineable), + template_height: template?.height ?? null, + template_weight: (template?.transactions ?? []).reduce((w, tx) => w + (tx.weight ?? 0), 0), + }; +} diff --git a/slipstream/lib/http.js b/slipstream/lib/http.js new file mode 100644 index 0000000..8d4088d --- /dev/null +++ b/slipstream/lib/http.js @@ -0,0 +1,170 @@ +/* The public HTTP surface. + * + * GET /info.json the pool, for pool directories (also /api/info) + * GET /api/fees minimum submission rate and current mineable rate + * POST /api/tx submit a raw tx, as hex: a text/plain body, or + * JSON {"hex": "..."} (also POST /tx) + * GET /api/tx/:txid where a tx stands, with its full history + * GET /api/txs recent txs; ?status= and ?limit= (max 500) + * GET /healthz + * + * Plain node:http, like the payout worker's admin surface: a handful of + * routes does not need a framework. Everything is readable cross-origin, + * since pool directories fetch info.json from their own pages. + */ + +import { createServer } from 'node:http'; + +import { buildInfo } from './info.js'; +import { summary } from './slipstream.js'; + +/* Hex doubles the size, and JSON wraps it. */ +const MAX_BODY_BYTES = 2 * 1_000_000 + 1024; + +const TXID_RE = /^[0-9a-f]{64}$/; +const STATUSES = ['pending', 'in_template', 'mined', 'confirmed', 'dropped', 'expired']; + +/* Fixed-window per-address counter. Crude, and enough: a submission costs + * the enforcer a node round trip, so the point is only to bound that. */ +export function rateLimiter(perMinute) { + const windows = new Map(); + return (key) => { + const minute = Math.floor(Date.now() / 60_000); + const w = windows.get(key); + if (!w || w.minute !== minute) { + windows.set(key, { minute, count: 1 }); + if (windows.size > 10_000) { + for (const [k, v] of windows) if (v.minute !== minute) windows.delete(k); + } + return true; + } + w.count += 1; + return w.count <= perMinute; + }; +} + +function clientAddress(req, trustProxy) { + if (trustProxy) { + const fwd = req.headers['x-forwarded-for']; + if (typeof fwd === 'string' && fwd.length > 0) return fwd.split(',')[0].trim(); + } + return req.socket.remoteAddress ?? 'unknown'; +} + +function readBody(req) { + return new Promise((resolve, reject) => { + let size = 0; + const chunks = []; + req.on('data', (chunk) => { + size += chunk.length; + if (size > MAX_BODY_BYTES) { + reject(Object.assign(new Error('body too large'), { tooLarge: true })); + req.destroy(); + return; + } + chunks.push(chunk); + }); + req.on('end', () => resolve(Buffer.concat(chunks).toString('utf8'))); + req.on('error', reject); + }); +} + +function txHexFrom(body, contentType) { + if ((contentType ?? '').includes('application/json')) { + const parsed = JSON.parse(body); + return typeof parsed === 'string' ? parsed : (parsed?.hex ?? parsed?.tx ?? ''); + } + return body; +} + +export function createHandler({ slipstream, pool, cfg, version, log }) { + const allow = rateLimiter(cfg.rateLimitPerMin); + return async (req, res) => { + const send = (status, body) => { + res.statusCode = status; + res.setHeader('content-type', 'application/json'); + res.setHeader('access-control-allow-origin', '*'); + res.setHeader('cache-control', 'no-store'); + res.end(JSON.stringify(body)); + }; + const url = new URL(req.url, 'http://localhost'); + const path = url.pathname.replace(/\/+$/, '') || '/'; + try { + if (req.method === 'OPTIONS') { + res.statusCode = 204; + res.setHeader('access-control-allow-origin', '*'); + res.setHeader('access-control-allow-methods', 'GET, POST, OPTIONS'); + res.setHeader('access-control-allow-headers', 'content-type'); + return res.end(); + } + if (req.method === 'GET' && (path === '/info.json' || path === '/api/info')) { + res.setHeader('cache-control', 'public, max-age=60'); + const body = buildInfo(pool.meta(), cfg.presentation, { version }); + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.setHeader('access-control-allow-origin', '*'); + return res.end(JSON.stringify(body, null, 2)); + } + if (req.method === 'GET' && path === '/api/fees') { + return send(200, slipstream.fees()); + } + if (req.method === 'POST' && (path === '/api/tx' || path === '/tx')) { + const who = clientAddress(req, cfg.trustProxy); + if (!allow(who)) return send(429, { accepted: false, reject_reason: 'rate-limited' }); + let hex; + try { + hex = txHexFrom(await readBody(req), req.headers['content-type']); + } catch (e) { + if (e.tooLarge) return send(413, { accepted: false, reject_reason: 'tx-size' }); + return send(400, { accepted: false, reject_reason: 'invalid-body' }); + } + const { httpStatus, body } = await slipstream.submit(hex, who); + return send(httpStatus, body); + } + const txMatch = path.match(/^\/(?:api\/)?tx\/([0-9a-fA-F]+)$/); + if (req.method === 'GET' && txMatch) { + const txid = txMatch[1].toLowerCase(); + if (!TXID_RE.test(txid)) return send(400, { error: 'invalid txid' }); + const row = slipstream.store.get(txid); + if (!row) return send(404, { error: 'unknown txid' }); + return send(200, { tx: summary(row), events: slipstream.store.events(txid) }); + } + if (req.method === 'GET' && path === '/api/txs') { + const status = url.searchParams.get('status'); + if (status && !STATUSES.includes(status)) { + return send(400, { error: `status must be one of ${STATUSES.join(', ')}` }); + } + const limit = Math.min(Math.max(Number(url.searchParams.get('limit')) || 50, 1), 500); + return send(200, { txs: slipstream.store.recent({ status, limit }) }); + } + if (req.method === 'GET' && path === '/healthz') { + const age = slipstream.templateAt === null + ? null : Math.floor(Date.now() / 1000) - slipstream.templateAt; + const ok = slipstream.enforcerError === null && age !== null; + return send(ok ? 200 : 503, { + ok, template_age_s: age, enforcer_error: slipstream.enforcerError, + }); + } + if (req.method === 'GET' && path === '/') { + return send(200, { + service: 'simplepool slipstream', + endpoints: ['GET /info.json', 'GET /api/fees', 'POST /api/tx', + 'GET /api/tx/:txid', 'GET /api/txs', 'GET /healthz'], + }); + } + return send(404, { error: 'not found' }); + } catch (e) { + log.error(`${req.method} ${path}: ${e.stack ?? e.message}`); + return send(500, { error: 'internal error' }); + } + }; +} + +export function startHttp(opts) { + const { cfg, log } = opts; + const server = createServer(createHandler(opts)); + server.listen(cfg.port, cfg.bind, () => { + log.info(`slipstream http listening on ${cfg.bind}:${cfg.port}`); + }); + return server; +} diff --git a/slipstream/lib/info.js b/slipstream/lib/info.js new file mode 100644 index 0000000..f418483 --- /dev/null +++ b/slipstream/lib/info.js @@ -0,0 +1,54 @@ +/* info.json: what a pool directory needs to list this pool. + * + * Facts about the pool are read from pool_meta, which the proxy writes from + * its own config, and never configured here: a second copy of the fee or the + * mode is a copy that can silently disagree with what the coinbase actually + * does. Only presentation (name, logo, URLs, contact) comes from env. + * + * `mode` is the exact pool_mode. Three modes are PPLNS and they differ in + * what a miner has to trust the pool with, so "pplns" alone would hide the + * one thing a miner choosing a pool most needs to know. + */ + +/* One sentence per mode on how a miner gets paid. */ +export const PAYOUT_TEXT = { + 'solo': + 'The stratum username is your BTC address; each block\'s coinbase pays the miner who found it.', + 'pps-classic': + 'The stratum username is your Thunder address; every accepted share is credited at a rate ' + + 'derived from the current block template, and balances are paid over Thunder in a daily batch.', + 'pplns-thunder': + 'The stratum username is your Thunder address; each block is split across the last window of ' + + 'shares once it is 100 blocks deep, and balances are paid over Thunder.', + 'pplns-btc': + 'The stratum username is your BTC address; each block is split across the last window of ' + + 'shares once it is 100 blocks deep, and balances are paid on Bitcoin.', + 'pplns-coinbase': + 'The stratum username is your BTC address; the coinbase pays every miner in the PPLNS window ' + + 'directly, so the pool never holds funds.', +}; + +const trimSlash = (url) => url.replace(/\/+$/, ''); + +export function buildInfo(meta, p, { version = null } = {}) { + const mode = meta?.pool_mode ?? null; + return { + name: p.name, + operator: p.operator, + chain: p.chain ?? meta?.network ?? null, + mode, + fee_bps: meta?.fee_bps ?? null, + coinbase_tag: meta?.coinbase_tag ?? null, + stratum_url: p.stratumUrl, + dashboard_url: p.dashboardUrl, + status_url: p.dashboardUrl ? `${trimSlash(p.dashboardUrl)}/api/status` : null, + slipstream_url: p.slipstreamUrl, + operator_address: meta?.operator_address ?? null, + pool_btc_address: meta?.pool_btc_address ?? null, + payout: p.payoutText ?? PAYOUT_TEXT[mode] ?? null, + software: 'simplepool', + version, + logo: p.logo, + contact: p.contact, + }; +} diff --git a/slipstream/lib/pool.js b/slipstream/lib/pool.js new file mode 100644 index 0000000..e7e43a6 --- /dev/null +++ b/slipstream/lib/pool.js @@ -0,0 +1,61 @@ +/* Read-only view of the proxy's shares.db. + * + * Opened lazily and re-tried, like the dashboard's: the proxy may not have + * created it yet. Every read degrades to "unknown" rather than failing, so + * slipstream keeps accepting txs through a proxy restart. + */ + +import Database from 'better-sqlite3'; +import fs from 'node:fs'; + +export function openPoolDb(path) { + let db = null; + let lastTryMs = 0; + + function get() { + if (db) return db; + const nowMs = Date.now(); + if (!path || nowMs - lastTryMs < 1000) return null; + lastTryMs = nowMs; + if (!fs.existsSync(path)) return null; + try { + db = new Database(path, { readonly: true, fileMustExist: true }); + db.pragma('busy_timeout = 2000'); + } catch (err) { + console.error(`[warn] pool db open failed: ${err.message}`); + db = null; + } + return db; + } + + return { + /* pool_meta's identity columns, or null if there is no pool_meta + * yet. Read per call: a proxy restarted onto another mode or network + * shows up on the next request. */ + meta() { + const d = get(); + if (!d) return null; + try { + return d.prepare(` + SELECT pool_mode, fee_bps, network, coinbase_tag, + operator_address, pool_btc_address + FROM pool_meta WHERE id = 1 + `).get() ?? null; + } catch { + return null; // a DB predating the identity columns + } + }, + + /* Whether the pool found `blockHash`: true, false, or null when + * shares.db cannot say. */ + foundBlock(blockHash) { + const d = get(); + if (!d) return null; + try { + return d.prepare('SELECT 1 FROM blocks_found WHERE hash = ?').get(blockHash) !== undefined; + } catch { + return null; + } + }, + }; +} diff --git a/slipstream/lib/rpc.js b/slipstream/lib/rpc.js new file mode 100644 index 0000000..ab73efc --- /dev/null +++ b/slipstream/lib/rpc.js @@ -0,0 +1,121 @@ +/* JSON-RPC clients for the enforcer's block template server and bitcoind. + * + * Both speak the same wire format; they differ only in auth. The template + * server has none, bitcoind takes basic auth from user/pass or its cookie + * file. A cookie is re-read on every call, because bitcoind rewrites it on + * each restart and a stale one fails every request until this restarts too. + */ + +import fs from 'node:fs'; + +export class RpcError extends Error { + constructor(method, code, message) { + super(`${method}: ${code} ${message}`); + this.method = method; + this.code = code; + this.rpcMessage = message; + } +} + +export class RpcClient { + constructor({ url, user = null, pass = null, cookieFile = null, timeoutMs = 30000 }) { + this.url = url; + this.user = user; + this.pass = pass; + this.cookieFile = cookieFile; + this.timeoutMs = timeoutMs; + this._id = 0; + } + + _auth() { + if (this.cookieFile) { + const cookie = fs.readFileSync(this.cookieFile, 'utf8').trim(); + return 'Basic ' + Buffer.from(cookie).toString('base64'); + } + if (this.user && this.pass) { + return 'Basic ' + Buffer.from(`${this.user}:${this.pass}`).toString('base64'); + } + return null; + } + + async call(method, params = []) { + const headers = { 'Content-Type': 'application/json' }; + const auth = this._auth(); + if (auth) headers.Authorization = auth; + const ctrl = new AbortController(); + const t = setTimeout(() => ctrl.abort(), this.timeoutMs); + let res; + try { + res = await fetch(this.url, { + method: 'POST', + headers, + body: JSON.stringify({ jsonrpc: '2.0', id: ++this._id, method, params }), + signal: ctrl.signal, + }); + } finally { + clearTimeout(t); + } + // bitcoind answers RPC errors with HTTP 500 and a JSON body, so read + // the body before deciding the status means transport failure. + const text = await res.text(); + let body; + try { + body = JSON.parse(text); + } catch { + throw new Error(`${method}: HTTP ${res.status} ${res.statusText}`); + } + if (body.error) throw new RpcError(method, body.error.code, body.error.message); + return body.result; + } +} + +/* The enforcer's block template server, as far as slipstream needs it. */ +export class EnforcerClient { + constructor(opts) { + this.rpc = new RpcClient(opts); + } + + /* The enforcer serves only coinbasetxn templates, and says so rather + * than falling back. */ + getBlockTemplate() { + return this.rpc.call('getblocktemplate', [{ + rules: ['segwit'], + capabilities: ['coinbasetxn'], + }]); + } + + submit(txHex) { return this.rpc.call('submitslipstreamtx', [txHex]); } + status(txid) { return this.rpc.call('getslipstreamtx', [txid]); } + remove(txid) { return this.rpc.call('removeslipstreamtx', [txid]); } +} + +/* bitcoind, read-only. */ +export class BitcoindClient { + constructor(opts) { + this.rpc = new RpcClient(opts); + } + + /* Confirmations of `blockHash`, -1 once it has left the main chain, or + * null if bitcoind does not know it. */ + async blockConfirmations(blockHash) { + try { + const header = await this.rpc.call('getblockheader', [blockHash, true]); + return { confirmations: header.confirmations, height: header.height }; + } catch (e) { + if (e instanceof RpcError && e.code === -5) return null; + throw e; + } + } + + /* The block a tx is confirmed in, if it is confirmed at all. Needs + * txindex, which the enforcer already requires of this node. */ + async txBlock(txid) { + try { + const tx = await this.rpc.call('getrawtransaction', [txid, true]); + return tx.blockhash && tx.confirmations > 0 ? tx.blockhash : null; + } catch (e) { + if (e instanceof RpcError && e.code === -5) return null; + throw e; + } + } +} diff --git a/slipstream/lib/slipstream.js b/slipstream/lib/slipstream.js new file mode 100644 index 0000000..85d14f2 --- /dev/null +++ b/slipstream/lib/slipstream.js @@ -0,0 +1,292 @@ +/* Submission and tracking. + * + * The enforcer's block template server holds the txs; this keeps the record + * of them. A submission is handed straight to the enforcer, which has the + * node check it for consensus validity and then keeps it in the template + * mempool, never relaying it. After that, every poll asks the enforcer where + * each open tx stands and reads the template the proxy is mining, and moves + * the row accordingly. + * + * The enforcer forgets a tx when it leaves its mempool, and forgets all of + * them when it restarts or a block is disconnected. This side remembers, so + * that is where resubmission lives. + */ + +import { RpcError } from './rpc.js'; +import { OPEN_STATUSES } from './store.js'; +import { feeRate, feeSnapshot } from './fees.js'; + +const now = () => Math.floor(Date.now() / 1000); + +/* A serialized tx is at most the 1M wu the enforcer accepts, i.e. 1MB. */ +export const MAX_TX_HEX_LEN = 2 * 1_000_000; + +export class Slipstream { + constructor({ store, enforcer, bitcoind = null, pool, cfg, log }) { + this.store = store; + this.enforcer = enforcer; + this.bitcoind = bitcoind; + this.pool = pool; + this.cfg = cfg; + this.log = log; + this.template = null; + this.templateAt = null; + this.enforcerError = null; + } + + fees() { + return { + ...feeSnapshot(this.template, this.cfg.minFeeRate), + updated_at: this.templateAt, + }; + } + + async refreshTemplate() { + try { + this.template = await this.enforcer.getBlockTemplate(); + this.templateAt = now(); + this.enforcerError = null; + } catch (e) { + this.enforcerError = e.message; + throw e; + } + return this.template; + } + + /* Hand one tx to the enforcer and record the outcome. Returns + * `{ httpStatus, body }`. Every call is logged as a submission, whatever + * becomes of it. */ + async submit(rawHex, submitter) { + const hex = typeof rawHex === 'string' ? rawHex.trim() : ''; + const record = (fields) => this.store.logSubmission({ submitter, rawHex: hex, ...fields }); + const refuse = (httpStatus, reason, extra = {}) => { + record({ txid: extra.txid ?? null, accepted: false, rejectReason: reason }); + return { httpStatus, body: { accepted: false, reject_reason: reason, ...extra } }; + }; + + if (hex.length === 0 || hex.length % 2 !== 0 || !/^[0-9a-fA-F]+$/.test(hex)) { + return refuse(400, 'invalid-hex'); + } + if (hex.length > MAX_TX_HEX_LEN) return refuse(400, 'tx-size'); + + let resp; + try { + resp = await this.enforcer.submit(hex); + } catch (e) { + if (e instanceof RpcError) { + if (e.code === -22) return refuse(400, 'tx-decode-failed'); + if (e.code === -32601) return refuse(503, 'slipstream-disabled'); + if (e.code === -10) return refuse(503, 'enforcer-syncing'); + } + this.log.error(`submitslipstreamtx failed: ${e.message}`); + return refuse(502, 'enforcer-unavailable'); + } + const { txid } = resp; + if (!resp.accepted) { + return refuse(200, resp.reject_reason ?? 'rejected', { txid }); + } + + const existing = this.store.get(txid); + if (existing && !['dropped', 'expired'].includes(existing.status)) { + record({ txid, accepted: true }); + return { httpStatus: 200, body: { accepted: true, already_tracked: true, ...summary(existing) } }; + } + + if (!this.template) { + try { await this.refreshTemplate(); } catch { /* the floor still applies */ } + } + const fees = this.fees(); + const rate = feeRate(resp.fee_sat, resp.vsize); + if (rate < fees.required_rate) { + // Already in the enforcer's mempool by now: take it back out, or + // it is mined for less than was asked. + try { + await this.enforcer.remove(txid); + } catch (e) { + this.log.error(`removeslipstreamtx ${txid} after a low fee failed: ${e.message}`); + } + return refuse(200, 'fee-rate-too-low', { + txid, fee_rate: rate, required_fee_rate: fees.required_rate, + }); + } + + const row = { + txid, + wtxid: resp.wtxid, + raw_hex: hex, + vsize: resp.vsize, + weight: resp.weight, + fee_sats: resp.fee_sat, + fee_rate: rate, + required_fee_rate: fees.required_rate, + submitted_height: this.template?.height ?? null, + submitter: submitter ?? null, + }; + if (existing) { + // Dropped or expired before, and valid again now + this.store.touch(txid, { + raw_hex: hex, fee_sats: row.fee_sats, fee_rate: rate, + required_fee_rate: row.required_fee_rate, submitted_at: now(), + submitted_height: row.submitted_height, + }); + this.store.setStatus(txid, 'pending', { detail: { resubmitted_by: 'submitter' } }); + } else { + this.store.insertAccepted(row); + } + record({ txid, accepted: true }); + this.log.info(`accepted ${txid} at ${rate} sat/vB (required ${fees.required_rate})`); + return { httpStatus: 200, body: { accepted: true, ...summary(this.store.get(txid)) } }; + } + + /* One pass over every tx still in play. Errors reaching the enforcer end + * the pass early; the next one picks up where it stood. */ + async tick() { + const template = await this.refreshTemplate(); + const inTemplate = new Set((template.transactions ?? []).map(tx => tx.txid)); + for (const row of this.store.byStatus(OPEN_STATUSES)) { + await this._followOpen(row, template, inTemplate); + } + for (const row of this.store.byStatus(['mined'])) { + await this._followMined(row, template); + } + } + + async _followOpen(row, template, inTemplate) { + const status = await this.enforcer.status(row.txid); + switch (status.status) { + case 'pending': { + if (row.submitted_height !== null + && template.height - row.submitted_height >= this.cfg.expiryBlocks) { + await this.enforcer.remove(row.txid); + this.store.setStatus(row.txid, 'expired', { + detail: { submitted_height: row.submitted_height, height: template.height }, + }); + this.log.info(`expired ${row.txid} after ${this.cfg.expiryBlocks} blocks`); + return; + } + const ts = now(); + if (inTemplate.has(row.txid)) { + this.store.setStatus(row.txid, 'in_template', { + fields: row.first_in_template_at === null ? { first_in_template_at: ts } : {}, + }); + this.store.touch(row.txid, { last_in_template_at: ts }); + } else { + this.store.setStatus(row.txid, 'pending'); + } + return; + } + case 'removed': + switch (status.reason) { + case 'mined': + return this._markMined(row, status.block_hash, template); + case 'reorged': + return this._resubmit(row, 'reorged'); + default: + // conflict_mined, parent_removed, rejected_by_enforcer, or a + // withdrawal that was not ours + this.store.setStatus(row.txid, 'dropped', { + reason: status.reason, + detail: { block_hash: status.block_hash, spent_by: status.spent_by, parent: status.parent }, + }); + this.log.info(`dropped ${row.txid}: ${status.reason}`); + return; + } + case 'unknown': { + // The enforcer restarted and lost it. It may have been mined + // while it was down. + const block = this.bitcoind ? await this.bitcoind.txBlock(row.txid) : null; + if (block) return this._markMined(row, block, template); + return this._resubmit(row, 'enforcer-forgot'); + } + default: + this.log.warn(`getslipstreamtx ${row.txid}: unexpected status ${JSON.stringify(status)}`); + } + } + + async _markMined(row, blockHash, template) { + let height = null; + let confirmations = null; + if (this.bitcoind) { + const header = await this.bitcoind.blockConfirmations(blockHash); + if (header) ({ height, confirmations } = header); + } else if (blockHash === template.previousblockhash) { + height = template.height - 1; + confirmations = 1; + } + this.store.setStatus(row.txid, 'mined', { + fields: { + mined_block_hash: blockHash, + mined_height: height, + mined_at: now(), + mined_by_pool: boolOrNull(this.pool.foundBlock(blockHash)), + confirmations, + }, + detail: { block_hash: blockHash, height }, + }); + this.log.info(`mined ${row.txid} in ${blockHash}`); + } + + async _followMined(row, template) { + let confirmations; + if (this.bitcoind) { + const header = await this.bitcoind.blockConfirmations(row.mined_block_hash); + if (!header || header.confirmations < 0) { + // Its block left the main chain. Mined again elsewhere, or + // waiting to be. + const block = await this.bitcoind.txBlock(row.txid); + if (block && block !== row.mined_block_hash) { + return this._markMined(row, block, template); + } + return this._resubmit(row, 'orphaned'); + } + confirmations = header.confirmations; + } else if (row.mined_height !== null) { + // Without the node an orphan cannot be seen, only depth + confirmations = template.height - row.mined_height; + } else { + return; + } + if (confirmations >= this.cfg.confirmations) { + this.store.setStatus(row.txid, 'confirmed', { + fields: { confirmations, confirmed_at: now() }, + }); + this.log.info(`confirmed ${row.txid} (${confirmations} deep)`); + } else { + this.store.touch(row.txid, { confirmations }); + } + } + + async _resubmit(row, why) { + let resp; + try { + resp = await this.enforcer.submit(row.raw_hex); + } catch (e) { + this.log.warn(`resubmitting ${row.txid} (${why}) failed: ${e.message}`); + return; + } + if (resp.accepted) { + this.store.touch(row.txid, { resubmissions: row.resubmissions + 1 }); + this.store.setStatus(row.txid, 'pending', { + fields: { mined_block_hash: null, mined_height: null, mined_at: null, + mined_by_pool: null, confirmations: null }, + detail: { resubmitted_after: why }, + }); + this.log.info(`resubmitted ${row.txid} after ${why}`); + } else { + this.store.setStatus(row.txid, 'dropped', { + reason: resp.reject_reason ?? 'rejected', + detail: { resubmitted_after: why }, + }); + this.log.info(`dropped ${row.txid}: resubmission after ${why} refused (${resp.reject_reason})`); + } + } +} + +const boolOrNull = (v) => (v === null || v === undefined ? null : v ? 1 : 0); + +/* A row as the API shows it: everything but the raw tx. */ +export function summary(row) { + if (!row) return null; + const { raw_hex: _raw, ...rest } = row; + return rest; +} diff --git a/slipstream/lib/store.js b/slipstream/lib/store.js new file mode 100644 index 0000000..d7f915e --- /dev/null +++ b/slipstream/lib/store.js @@ -0,0 +1,189 @@ +/* The slipstream service's own database. + * + * Three tables, each with one job: + * + * slipstream_submissions every POST, accepted or not, exactly as it + * arrived. The record of what was asked of the + * pool, kept even when the tx was refused. + * slipstream_txs one row per accepted tx, holding where it stands + * now: pending, in_template, mined, confirmed, + * dropped or expired. + * slipstream_events every status change, append-only, so a row's + * history can be read back rather than inferred. + * + * Status meanings: + * pending in the enforcer's mempool, not in the latest template + * in_template in the latest template, i.e. being mined on now + * mined in a block, fewer than the configured confirmations deep + * confirmed that deep + * dropped will not be mined: status_reason says why + * expired waited past the expiry and was withdrawn + * + * Nothing is ever relayed, so "propagated" here means only this: did the tx + * reach a template, and did a block carry it. + */ + +import Database from 'better-sqlite3'; + +export const OPEN_STATUSES = ['pending', 'in_template']; + +const SCHEMA = ` +CREATE TABLE IF NOT EXISTS slipstream_submissions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + ts INTEGER NOT NULL, + submitter TEXT, + txid TEXT, + raw_hex TEXT NOT NULL, + accepted INTEGER NOT NULL, + reject_reason TEXT +); +CREATE INDEX IF NOT EXISTS slipstream_submissions_ts_idx ON slipstream_submissions(ts); +CREATE INDEX IF NOT EXISTS slipstream_submissions_txid_idx ON slipstream_submissions(txid); + +CREATE TABLE IF NOT EXISTS slipstream_txs ( + txid TEXT PRIMARY KEY, + wtxid TEXT, + raw_hex TEXT NOT NULL, + vsize INTEGER NOT NULL, + weight INTEGER NOT NULL, + fee_sats INTEGER NOT NULL, + fee_rate REAL NOT NULL, -- sat/vB + required_fee_rate REAL NOT NULL, -- what was asked of it, sat/vB + submitted_at INTEGER NOT NULL, + submitted_height INTEGER, -- template height at submission + submitter TEXT, + status TEXT NOT NULL, + status_reason TEXT, + status_at INTEGER NOT NULL, + first_in_template_at INTEGER, + last_in_template_at INTEGER, + mined_block_hash TEXT, + mined_height INTEGER, + mined_at INTEGER, + mined_by_pool INTEGER, -- 1 ours, 0 not, NULL unknown + confirmations INTEGER, + confirmed_at INTEGER, + resubmissions INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX IF NOT EXISTS slipstream_txs_status_idx ON slipstream_txs(status); +CREATE INDEX IF NOT EXISTS slipstream_txs_submitted_idx ON slipstream_txs(submitted_at); + +CREATE TABLE IF NOT EXISTS slipstream_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + txid TEXT NOT NULL, + ts INTEGER NOT NULL, + event TEXT NOT NULL, + detail TEXT +); +CREATE INDEX IF NOT EXISTS slipstream_events_txid_idx ON slipstream_events(txid, id); +`; + +const now = () => Math.floor(Date.now() / 1000); + +export function openStore(path) { + const db = new Database(path); + db.pragma('journal_mode = WAL'); + db.pragma('synchronous = NORMAL'); + db.pragma('busy_timeout = 5000'); + db.exec(SCHEMA); + return new Store(db); +} + +export class Store { + constructor(db) { + this.db = db; + this._event = db.prepare( + 'INSERT INTO slipstream_events (txid, ts, event, detail) VALUES (?, ?, ?, ?)'); + } + + logSubmission({ submitter, txid, rawHex, accepted, rejectReason }) { + this.db.prepare(` + INSERT INTO slipstream_submissions (ts, submitter, txid, raw_hex, accepted, reject_reason) + VALUES (?, ?, ?, ?, ?, ?) + `).run(now(), submitter ?? null, txid ?? null, rawHex, accepted ? 1 : 0, rejectReason ?? null); + } + + event(txid, event, detail = null) { + this._event.run(txid, now(), event, + detail === null || typeof detail === 'string' ? detail : JSON.stringify(detail)); + } + + get(txid) { + return this.db.prepare('SELECT * FROM slipstream_txs WHERE txid = ?').get(txid) ?? null; + } + + events(txid) { + return this.db.prepare( + 'SELECT ts, event, detail FROM slipstream_events WHERE txid = ? ORDER BY id').all(txid); + } + + insertAccepted(row) { + const ts = now(); + this.db.transaction(() => { + this.db.prepare(` + INSERT INTO slipstream_txs + (txid, wtxid, raw_hex, vsize, weight, fee_sats, fee_rate, required_fee_rate, + submitted_at, submitted_height, submitter, status, status_at) + VALUES (@txid, @wtxid, @raw_hex, @vsize, @weight, @fee_sats, @fee_rate, + @required_fee_rate, @ts, @submitted_height, @submitter, 'pending', @ts) + `).run({ ...row, ts }); + this._event.run(row.txid, ts, 'accepted', JSON.stringify({ + fee_sats: row.fee_sats, fee_rate: row.fee_rate, + required_fee_rate: row.required_fee_rate, + })); + })(); + } + + /* Move a row to `status`, recording the change as an event. `fields` + * are further columns to set alongside. A no-op when nothing changes, so + * a poll that finds a tx where it already was writes nothing. */ + setStatus(txid, status, { reason = null, fields = {}, detail = null } = {}) { + const row = this.get(txid); + if (!row) return false; + const changed = row.status !== status || (row.status_reason ?? null) !== reason; + const extra = Object.entries(fields).filter(([k, v]) => row[k] !== v); + if (!changed && extra.length === 0) return false; + const ts = now(); + this.db.transaction(() => { + const sets = ['status = @status', 'status_reason = @reason']; + if (changed) sets.push('status_at = @ts'); + for (const [k] of extra) sets.push(`${k} = @${k}`); + this.db.prepare(`UPDATE slipstream_txs SET ${sets.join(', ')} WHERE txid = @txid`) + .run({ txid, status, reason, ts, ...Object.fromEntries(extra) }); + if (changed) { + this._event.run(txid, ts, status, + detail === null ? reason : JSON.stringify({ reason, ...detail })); + } + })(); + return true; + } + + /* Columns that change with every poll and would drown the event log: + * set without an event. */ + touch(txid, fields) { + const keys = Object.keys(fields); + if (keys.length === 0) return; + this.db.prepare( + `UPDATE slipstream_txs SET ${keys.map(k => `${k} = @${k}`).join(', ')} WHERE txid = @txid`, + ).run({ txid, ...fields }); + } + + byStatus(statuses) { + const marks = statuses.map(() => '?').join(', '); + return this.db.prepare( + `SELECT * FROM slipstream_txs WHERE status IN (${marks}) ORDER BY submitted_at`, + ).all(...statuses); + } + + recent({ status = null, limit = 50 } = {}) { + const cols = `txid, wtxid, vsize, weight, fee_sats, fee_rate, required_fee_rate, + submitted_at, status, status_reason, status_at, first_in_template_at, + last_in_template_at, mined_block_hash, mined_height, mined_at, + mined_by_pool, confirmations, confirmed_at, resubmissions`; + return status + ? this.db.prepare(`SELECT ${cols} FROM slipstream_txs WHERE status = ? + ORDER BY submitted_at DESC LIMIT ?`).all(status, limit) + : this.db.prepare(`SELECT ${cols} FROM slipstream_txs + ORDER BY submitted_at DESC LIMIT ?`).all(limit); + } +} diff --git a/slipstream/package-lock.json b/slipstream/package-lock.json new file mode 100644 index 0000000..b8991b3 --- /dev/null +++ b/slipstream/package-lock.json @@ -0,0 +1,467 @@ +{ + "name": "simplepool-slipstream", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "simplepool-slipstream", + "version": "0.1.0", + "dependencies": { + "better-sqlite3": "^11.5.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/base64-js": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", + "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/better-sqlite3": { + "version": "11.10.0", + "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz", + "integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==", + "hasInstallScript": true, + "license": "MIT", + "dependencies": { + "bindings": "^1.5.0", + "prebuild-install": "^7.1.1" + } + }, + "node_modules/bindings": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz", + "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==", + "license": "MIT", + "dependencies": { + "file-uri-to-path": "1.0.0" + } + }, + "node_modules/bl": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", + "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", + "license": "MIT", + "dependencies": { + "buffer": "^5.5.0", + "inherits": "^2.0.4", + "readable-stream": "^3.4.0" + } + }, + "node_modules/buffer": { + "version": "5.7.1", + "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", + "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "dependencies": { + "base64-js": "^1.3.1", + "ieee754": "^1.1.13" + } + }, + "node_modules/chownr": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", + "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", + "license": "ISC" + }, + "node_modules/decompress-response": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", + "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", + "license": "MIT", + "dependencies": { + "mimic-response": "^3.1.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/deep-extend": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", + "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", + "license": "MIT", + "engines": { + "node": ">=4.0.0" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=8" + } + }, + "node_modules/end-of-stream": { + "version": "1.4.5", + "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", + "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", + "license": "MIT", + "dependencies": { + "once": "^1.4.0" + } + }, + "node_modules/expand-template": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", + "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", + "license": "(MIT OR WTFPL)", + "engines": { + "node": ">=6" + } + }, + "node_modules/file-uri-to-path": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", + "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==", + "license": "MIT" + }, + "node_modules/fs-constants": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", + "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", + "license": "MIT" + }, + "node_modules/github-from-package": { + "version": "0.0.0", + "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", + "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", + "license": "MIT" + }, + "node_modules/ieee754": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", + "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "BSD-3-Clause" + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ini": { + "version": "1.3.8", + "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", + "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", + "license": "ISC" + }, + "node_modules/mimic-response": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", + "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/minimist": { + "version": "1.2.8", + "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", + "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/mkdirp-classic": { + "version": "0.5.3", + "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", + "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", + "license": "MIT" + }, + "node_modules/napi-build-utils": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", + "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", + "license": "MIT" + }, + "node_modules/node-abi": { + "version": "3.96.0", + "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz", + "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==", + "license": "MIT", + "dependencies": { + "semver": "^7.3.5" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "license": "ISC", + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/prebuild-install": { + "version": "7.1.3", + "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", + "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", + "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.", + "license": "MIT", + "dependencies": { + "detect-libc": "^2.0.0", + "expand-template": "^2.0.3", + "github-from-package": "0.0.0", + "minimist": "^1.2.3", + "mkdirp-classic": "^0.5.3", + "napi-build-utils": "^2.0.0", + "node-abi": "^3.3.0", + "pump": "^3.0.0", + "rc": "^1.2.7", + "simple-get": "^4.0.0", + "tar-fs": "^2.0.0", + "tunnel-agent": "^0.6.0" + }, + "bin": { + "prebuild-install": "bin.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/pump": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz", + "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==", + "license": "MIT", + "dependencies": { + "end-of-stream": "^1.1.0", + "once": "^1.3.1" + } + }, + "node_modules/rc": { + "version": "1.2.8", + "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", + "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", + "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", + "dependencies": { + "deep-extend": "^0.6.0", + "ini": "~1.3.0", + "minimist": "^1.2.0", + "strip-json-comments": "~2.0.1" + }, + "bin": { + "rc": "cli.js" + } + }, + "node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "license": "MIT", + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/safe-buffer": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/semver": { + "version": "7.8.5", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/simple-concat": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", + "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/simple-get": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", + "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "dependencies": { + "decompress-response": "^6.0.0", + "once": "^1.3.1", + "simple-concat": "^1.0.0" + } + }, + "node_modules/string_decoder": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", + "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", + "license": "MIT", + "dependencies": { + "safe-buffer": "~5.2.0" + } + }, + "node_modules/strip-json-comments": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", + "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/tar-fs": { + "version": "2.1.5", + "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz", + "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==", + "license": "MIT", + "dependencies": { + "chownr": "^1.1.1", + "mkdirp-classic": "^0.5.2", + "pump": "^3.0.0", + "tar-stream": "^2.1.4" + } + }, + "node_modules/tar-stream": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", + "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", + "license": "MIT", + "dependencies": { + "bl": "^4.0.3", + "end-of-stream": "^1.4.1", + "fs-constants": "^1.0.0", + "inherits": "^2.0.3", + "readable-stream": "^3.1.1" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/tunnel-agent": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", + "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", + "license": "Apache-2.0", + "dependencies": { + "safe-buffer": "^5.0.1" + }, + "engines": { + "node": "*" + } + }, + "node_modules/util-deprecate": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", + "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", + "license": "MIT" + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "license": "ISC" + } + } +} diff --git a/slipstream/package.json b/slipstream/package.json new file mode 100644 index 0000000..5c1ffe4 --- /dev/null +++ b/slipstream/package.json @@ -0,0 +1,17 @@ +{ + "name": "simplepool-slipstream", + "version": "0.1.0", + "private": true, + "type": "module", + "scripts": { + "start": "node index.js", + "dev": "node --watch index.js", + "test": "node --test test/*.test.js" + }, + "engines": { + "node": ">=20" + }, + "dependencies": { + "better-sqlite3": "^11.5.0" + } +} diff --git a/slipstream/test/fees.test.js b/slipstream/test/fees.test.js new file mode 100644 index 0000000..63bc437 --- /dev/null +++ b/slipstream/test/fees.test.js @@ -0,0 +1,28 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { feeSnapshot, mineableRate } from '../lib/fees.js'; +import { fullTemplate } from './helpers.js'; + +test('a template with room is mineable at the floor', () => { + const template = fullTemplate({ feeRate: 50, count: 2 }); + assert.equal(mineableRate(template, 1), 1); + assert.equal(mineableRate({ transactions: [] }, 1), 1); + assert.equal(mineableRate(null, 1), 1); +}); + +test('a full template is mineable at its cheapest tx', () => { + const template = fullTemplate({ feeRate: 3 }); + template.transactions[4].fee = 5 * 100_000; // 5 sat/vB, dearer than the rest + assert.equal(mineableRate(template, 1), 3); +}); + +test('the required rate is the higher of floor and mineable', () => { + assert.equal(feeSnapshot(fullTemplate({ feeRate: 3 }), 1).required_rate, 3); + assert.equal(feeSnapshot(fullTemplate({ feeRate: 3 }), 10).required_rate, 10); + const snap = feeSnapshot(fullTemplate({ feeRate: 50, count: 1 }), 1); + assert.deepEqual( + [snap.min_submission_rate, snap.mineable_rate, snap.required_rate], + [1, 1, 1], + ); +}); diff --git a/slipstream/test/helpers.js b/slipstream/test/helpers.js new file mode 100644 index 0000000..f285247 --- /dev/null +++ b/slipstream/test/helpers.js @@ -0,0 +1,108 @@ +/* A fake enforcer and bitcoind, scripted per test, around a real in-memory + * store: the store is what the tests are about, so it is not faked. */ + +import { openStore } from '../lib/store.js'; +import { Slipstream } from '../lib/slipstream.js'; +import { RpcError } from '../lib/rpc.js'; + +export const TXID_A = 'a'.repeat(64); +export const TXID_B = 'b'.repeat(64); +export const BLOCK_1 = '1'.repeat(64); +export const BLOCK_2 = '2'.repeat(64); +export const TIP = 'f'.repeat(64); + +export const quietLog = { debug() {}, info() {}, warn() {}, error() {} }; + +export function baseCfg(overrides = {}) { + return { + minFeeRate: 1, + confirmations: 6, + expiryBlocks: 144, + rateLimitPerMin: 30, + trustProxy: false, + presentation: {}, + ...overrides, + }; +} + +/* An enforcer whose mempool is a Map of txid -> status, and whose + * template is whatever the test sets. */ +export class FakeEnforcer { + constructor() { + this.template = { height: 100, previousblockhash: TIP, transactions: [] }; + this.statuses = new Map(); + this.submitted = []; + this.removed = []; + this.nextSubmit = null; + } + + async getBlockTemplate() { + if (this.down) throw new Error('connect ECONNREFUSED'); + return this.template; + } + + /* `nextSubmit` scripts the next answer; by default any tx is accepted as + * `txid` at 2 sat/vB. */ + async submit(hex) { + this.submitted.push(hex); + const next = this.nextSubmit ?? { txid: TXID_A, accepted: true, fee_sat: 200, vsize: 100 }; + this.nextSubmit = null; + if (next instanceof RpcError) throw next; + const resp = { wtxid: next.txid, weight: (next.vsize ?? 100) * 4, ...next }; + if (resp.accepted) this.statuses.set(resp.txid, { status: 'pending', txid: resp.txid }); + return resp; + } + + async status(txid) { + return this.statuses.get(txid) ?? { status: 'unknown', txid }; + } + + async remove(txid) { + this.removed.push(txid); + this.statuses.set(txid, { status: 'removed', txid, reason: 'withdrawn' }); + return [txid]; + } +} + +export class FakeBitcoind { + constructor() { + this.headers = new Map(); // block hash -> { confirmations, height } + this.txBlocks = new Map(); // txid -> block hash + } + + async blockConfirmations(hash) { return this.headers.get(hash) ?? null; } + async txBlock(txid) { return this.txBlocks.get(txid) ?? null; } +} + +export function fakePool({ meta = null, ours = [] } = {}) { + return { + meta: () => meta, + foundBlock: (hash) => ours.includes(hash), + }; +} + +export function makeSlipstream({ cfg = {}, bitcoind = null, pool = fakePool() } = {}) { + const enforcer = new FakeEnforcer(); + const slipstream = new Slipstream({ + store: openStore(':memory:'), + enforcer, + bitcoind, + pool, + cfg: baseCfg(cfg), + log: quietLog, + }); + return { slipstream, enforcer, store: slipstream.store }; +} + +/* A template carrying `count` txs of `weight` wu each, paying `feeRate`. */ +export function fullTemplate({ feeRate, weight = 400_000, count = 10 }) { + return { + height: 100, + previousblockhash: TIP, + transactions: Array.from({ length: count }, (_, i) => ({ + txid: String(i).padStart(64, '0'), + weight, + fee: Math.round(feeRate * (weight / 4)), + })), + }; +} diff --git a/slipstream/test/http.test.js b/slipstream/test/http.test.js new file mode 100644 index 0000000..0ee3a30 --- /dev/null +++ b/slipstream/test/http.test.js @@ -0,0 +1,79 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createServer } from 'node:http'; + +import { createHandler } from '../lib/http.js'; +import { TXID_A, fakePool, makeSlipstream, quietLog } from './helpers.js'; + +const HEX = '0200000001' + '00'.repeat(60); + +async function serve(cfgOverrides = {}, meta = { pool_mode: 'solo', fee_bps: 0 }) { + const ctx = makeSlipstream({ cfg: cfgOverrides }); + const pool = fakePool({ meta }); + const server = createServer(createHandler({ + slipstream: ctx.slipstream, pool, cfg: ctx.slipstream.cfg, version: '0.1.0', log: quietLog, + })); + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); + const base = `http://127.0.0.1:${server.address().port}`; + return { ...ctx, base, close: () => new Promise(resolve => server.close(resolve)) }; +} + +test('info.json is served cross-origin', async () => { + const { base, close } = await serve({ presentation: { slipstreamUrl: 'https://s.example' } }); + try { + const res = await fetch(`${base}/info.json`); + assert.equal(res.status, 200); + assert.equal(res.headers.get('access-control-allow-origin'), '*'); + const info = await res.json(); + assert.equal(info.mode, 'solo'); + assert.equal(info.slipstream_url, 'https://s.example'); + assert.equal(info.version, '0.1.0'); + } finally { + await close(); + } +}); + +test('submit as text or JSON, then read the tx back with its history', async () => { + const { base, close } = await serve(); + try { + let res = await fetch(`${base}/api/tx`, { method: 'POST', body: HEX }); + assert.equal(res.status, 200); + assert.equal((await res.json()).accepted, true); + + res = await fetch(`${base}/tx`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ hex: HEX }), + }); + assert.equal((await res.json()).already_tracked, true); + + res = await fetch(`${base}/api/tx/${TXID_A}`); + const { tx, events } = await res.json(); + assert.equal(tx.status, 'pending'); + assert.equal(tx.raw_hex, undefined); + assert.deepEqual(events.map(e => e.event), ['accepted']); + + res = await fetch(`${base}/api/txs?status=pending`); + assert.equal((await res.json()).txs.length, 1); + assert.equal((await fetch(`${base}/api/txs?status=bogus`)).status, 400); + assert.equal((await fetch(`${base}/api/tx/${'c'.repeat(64)}`)).status, 404); + } finally { + await close(); + } +}); + +test('fees are reported, and submissions are rate limited per address', async () => { + const { base, close } = await serve({ rateLimitPerMin: 2 }); + try { + const fees = await (await fetch(`${base}/api/fees`)).json(); + assert.equal(fees.min_submission_rate, 1); + assert.equal(fees.required_rate, 1); + const statuses = []; + for (let i = 0; i < 3; i++) { + statuses.push((await fetch(`${base}/api/tx`, { method: 'POST', body: HEX })).status); + } + assert.deepEqual(statuses, [200, 200, 429]); + } finally { + await close(); + } +}); diff --git a/slipstream/test/info.test.js b/slipstream/test/info.test.js new file mode 100644 index 0000000..3021a46 --- /dev/null +++ b/slipstream/test/info.test.js @@ -0,0 +1,73 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { buildInfo, PAYOUT_TEXT } from '../lib/info.js'; + +const META = { + pool_mode: 'pplns-coinbase', + fee_bps: 100, + network: 'main', + coinbase_tag: '/bip300xyz/', + operator_address: 'bc1qljvzxk0tp6qtrunt590z5rtdhs8jhkkn4ny4rm', + pool_btc_address: null, +}; + +const PRESENTATION = { + name: 'bip300.xyz', + operator: 'bip300.xyz', + chain: 'betanet', + logo: 'mining-pool-logos/bip300xyz.svg', + contact: null, + payoutText: null, + stratumUrl: 'stratum+tcp://stratum.beta.bip300.xyz:3334', + dashboardUrl: 'https://pool.beta.bip300.xyz/', + slipstreamUrl: 'https://slipstream.beta.bip300.xyz', +}; + +test('matches the shape pool directories expect', () => { + const info = buildInfo(META, PRESENTATION); + assert.deepEqual(info, { + name: 'bip300.xyz', + operator: 'bip300.xyz', + chain: 'betanet', + mode: 'pplns-coinbase', + fee_bps: 100, + coinbase_tag: '/bip300xyz/', + stratum_url: 'stratum+tcp://stratum.beta.bip300.xyz:3334', + dashboard_url: 'https://pool.beta.bip300.xyz/', + status_url: 'https://pool.beta.bip300.xyz/api/status', + slipstream_url: 'https://slipstream.beta.bip300.xyz', + operator_address: 'bc1qljvzxk0tp6qtrunt590z5rtdhs8jhkkn4ny4rm', + pool_btc_address: null, + payout: PAYOUT_TEXT['pplns-coinbase'], + software: 'simplepool', + version: null, + logo: 'mining-pool-logos/bip300xyz.svg', + contact: null, + }); +}); + +test('facts come from pool_meta, the exact mode included', () => { + const info = buildInfo({ ...META, pool_mode: 'pplns-btc', fee_bps: 250 }, PRESENTATION); + assert.equal(info.mode, 'pplns-btc'); + assert.equal(info.fee_bps, 250); + assert.equal(info.payout, PAYOUT_TEXT['pplns-btc']); +}); + +test('every mode has a payout sentence, and it can be overridden', () => { + for (const mode of ['solo', 'pps-classic', 'pplns-thunder', 'pplns-btc', 'pplns-coinbase']) { + assert.ok(buildInfo({ ...META, pool_mode: mode }, PRESENTATION).payout, mode); + } + const info = buildInfo(META, { ...PRESENTATION, payoutText: 'custom' }); + assert.equal(info.payout, 'custom'); +}); + +test('chain falls back to the network, and missing pool_meta reads as unknown', () => { + assert.equal(buildInfo(META, { ...PRESENTATION, chain: null }).chain, 'main'); + const info = buildInfo(null, { ...PRESENTATION, chain: null, dashboardUrl: null }); + assert.equal(info.mode, null); + assert.equal(info.fee_bps, null); + assert.equal(info.chain, null); + assert.equal(info.status_url, null); + assert.equal(info.payout, null); +}); diff --git a/slipstream/test/submit.test.js b/slipstream/test/submit.test.js new file mode 100644 index 0000000..a4dba58 --- /dev/null +++ b/slipstream/test/submit.test.js @@ -0,0 +1,66 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { RpcError } from '../lib/rpc.js'; +import { TXID_A, fullTemplate, makeSlipstream } from './helpers.js'; + +const HEX = '0200000001' + '00'.repeat(60); + +const submissions = (store) => + store.db.prepare('SELECT txid, accepted, reject_reason FROM slipstream_submissions ORDER BY id').all(); + +test('an accepted tx is tracked as pending, and the submission logged', async () => { + const { slipstream, store } = makeSlipstream(); + const { httpStatus, body } = await slipstream.submit(HEX, '203.0.113.5'); + assert.equal(httpStatus, 200); + assert.equal(body.accepted, true); + assert.equal(body.status, 'pending'); + assert.equal(body.fee_rate, 2); + assert.equal(body.raw_hex, undefined, 'the raw tx is not echoed'); + const row = store.get(TXID_A); + assert.equal(row.raw_hex, HEX); + assert.equal(row.submitter, '203.0.113.5'); + assert.equal(row.submitted_height, 100); + assert.deepEqual(submissions(store), [{ txid: TXID_A, accepted: 1, reject_reason: null }]); + assert.deepEqual(store.events(TXID_A).map(e => e.event), ['accepted']); +}); + +test('under the mineable rate: taken back out of the enforcer, and refused', async () => { + const { slipstream, enforcer, store } = makeSlipstream(); + enforcer.template = fullTemplate({ feeRate: 3 }); + const { body } = await slipstream.submit(HEX, 'x'); + assert.equal(body.accepted, false); + assert.equal(body.reject_reason, 'fee-rate-too-low'); + assert.equal(body.required_fee_rate, 3); + assert.deepEqual(enforcer.removed, [TXID_A]); + assert.equal(store.get(TXID_A), null); + assert.deepEqual(submissions(store), [{ txid: TXID_A, accepted: 0, reject_reason: 'fee-rate-too-low' }]); +}); + +test('refusals are logged too, with the reason', async () => { + const { slipstream, enforcer, store } = makeSlipstream(); + + assert.equal((await slipstream.submit('zz', 'x')).httpStatus, 400); + + enforcer.nextSubmit = { txid: TXID_A, accepted: false, reject_reason: 'missing-inputs' }; + const refused = await slipstream.submit(HEX, 'x'); + assert.equal(refused.httpStatus, 200); + assert.equal(refused.body.reject_reason, 'missing-inputs'); + + enforcer.nextSubmit = new RpcError('submitslipstreamtx', -32601, 'slipstream is disabled'); + assert.equal((await slipstream.submit(HEX, 'x')).httpStatus, 503); + + assert.deepEqual(submissions(store).map(s => s.reject_reason), + ['invalid-hex', 'missing-inputs', 'slipstream-disabled']); + assert.equal(store.get(TXID_A), null); +}); + +test('resubmitting a tracked tx returns where it stands', async () => { + const { slipstream, enforcer } = makeSlipstream(); + await slipstream.submit(HEX, 'x'); + enforcer.nextSubmit = { txid: TXID_A, accepted: true, already_present: true, fee_sat: 200, vsize: 100 }; + const { body } = await slipstream.submit(HEX, 'y'); + assert.equal(body.accepted, true); + assert.equal(body.already_tracked, true); + assert.equal(body.submitter, 'x'); +}); diff --git a/slipstream/test/tracker.test.js b/slipstream/test/tracker.test.js new file mode 100644 index 0000000..89a35f5 --- /dev/null +++ b/slipstream/test/tracker.test.js @@ -0,0 +1,147 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { + BLOCK_1, BLOCK_2, TIP, TXID_A, FakeBitcoind, fakePool, makeSlipstream, +} from './helpers.js'; + +const HEX = '0200000001' + '00'.repeat(60); + +async function submitted(opts = {}) { + const ctx = makeSlipstream(opts); + await ctx.slipstream.submit(HEX, 'x'); + return ctx; +} + +const eventsOf = (store) => store.events(TXID_A).map(e => e.event); + +test('in and out of the template, without an event per poll', async () => { + const { slipstream, enforcer, store } = await submitted(); + enforcer.template.transactions = [{ txid: TXID_A, weight: 400, fee: 200 }]; + await slipstream.tick(); + await slipstream.tick(); + const row = store.get(TXID_A); + assert.equal(row.status, 'in_template'); + assert.ok(row.first_in_template_at !== null && row.last_in_template_at !== null); + + enforcer.template.transactions = []; + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'pending'); + assert.deepEqual(eventsOf(store), ['accepted', 'in_template', 'pending']); +}); + +test('mined by the pool, then confirmed', async () => { + const bitcoind = new FakeBitcoind(); + const { slipstream, enforcer, store } = await submitted({ + bitcoind, pool: fakePool({ ours: [BLOCK_1] }), + }); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); + bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); + await slipstream.tick(); + let row = store.get(TXID_A); + assert.equal(row.status, 'mined'); + assert.equal(row.mined_block_hash, BLOCK_1); + assert.equal(row.mined_height, 100); + assert.equal(row.mined_by_pool, 1); + + bitcoind.headers.set(BLOCK_1, { confirmations: 6, height: 100 }); + await slipstream.tick(); + row = store.get(TXID_A); + assert.equal(row.status, 'confirmed'); + assert.equal(row.confirmations, 6); + assert.deepEqual(eventsOf(store), ['accepted', 'mined', 'confirmed']); +}); + +test('without bitcoind, depth comes from the template height', async () => { + const { slipstream, enforcer, store } = await submitted(); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: TIP }); + await slipstream.tick(); + assert.equal(store.get(TXID_A).mined_height, 99); + assert.equal(store.get(TXID_A).mined_by_pool, 0); + enforcer.template = { ...enforcer.template, height: 105 }; + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'confirmed'); +}); + +test('an orphaned block puts the tx back in the enforcer', async () => { + const bitcoind = new FakeBitcoind(); + const { slipstream, enforcer, store } = await submitted({ bitcoind }); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); + bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); + await slipstream.tick(); + + bitcoind.headers.set(BLOCK_1, { confirmations: -1, height: 100 }); + await slipstream.tick(); + const row = store.get(TXID_A); + assert.equal(row.status, 'pending'); + assert.equal(row.mined_block_hash, null); + assert.equal(row.resubmissions, 1); + assert.equal(enforcer.submitted.length, 2); +}); + +test('an orphaned tx mined again elsewhere follows its new block', async () => { + const bitcoind = new FakeBitcoind(); + const { slipstream, enforcer, store } = await submitted({ bitcoind }); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); + bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); + await slipstream.tick(); + + bitcoind.headers.set(BLOCK_1, { confirmations: -1, height: 100 }); + bitcoind.headers.set(BLOCK_2, { confirmations: 1, height: 100 }); + bitcoind.txBlocks.set(TXID_A, BLOCK_2); + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'mined'); + assert.equal(store.get(TXID_A).mined_block_hash, BLOCK_2); + assert.equal(enforcer.submitted.length, 1, 'not resubmitted'); +}); + +test('a reorg, or an enforcer restart, resubmits', async () => { + for (const status of [ + { status: 'removed', txid: TXID_A, reason: 'reorged', block_hash: BLOCK_1 }, + { status: 'unknown', txid: TXID_A }, + ]) { + const { slipstream, enforcer, store } = await submitted(); + enforcer.statuses.set(TXID_A, status); + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'pending', status.status); + assert.equal(store.get(TXID_A).resubmissions, 1, status.status); + } +}); + +test('a resubmission the enforcer refuses drops the tx, with its reason', async () => { + const { slipstream, enforcer, store } = await submitted(); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'reorged', block_hash: BLOCK_1 }); + enforcer.nextSubmit = { txid: TXID_A, accepted: false, reject_reason: 'bad-txns-inputs-missingorspent' }; + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'dropped'); + assert.equal(store.get(TXID_A).status_reason, 'bad-txns-inputs-missingorspent'); +}); + +test('a conflict or a lost parent drops the tx', async () => { + for (const reason of ['conflict_mined', 'parent_removed', 'rejected_by_enforcer']) { + const { slipstream, enforcer, store } = await submitted(); + enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason }); + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'dropped', reason); + assert.equal(store.get(TXID_A).status_reason, reason); + } +}); + +test('a tx that waits past the expiry is withdrawn', async () => { + const { slipstream, enforcer, store } = await submitted({ cfg: { expiryBlocks: 10 } }); + enforcer.template = { ...enforcer.template, height: 110 }; + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'expired'); + assert.deepEqual(enforcer.removed, [TXID_A]); + // The withdrawal the expiry caused is not mistaken for someone else's + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'expired'); +}); + +test('an unreachable enforcer fails the tick and changes nothing', async () => { + const { slipstream, enforcer, store } = await submitted(); + enforcer.down = true; + await assert.rejects(slipstream.tick()); + assert.equal(store.get(TXID_A).status, 'pending'); + assert.ok(slipstream.enforcerError); +}); diff --git a/tests/README.md b/tests/README.md index caa5df5..6f1fb0d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -96,6 +96,17 @@ the version last validated against. batched transaction through the enforcer's wallet, with a shared address summed. Own `.regtest-btcpay/` dir. +- `test_slipstream_regtest.sh` — the slipstream service against a real + enforcer: a tx the node refuses to relay is accepted, never enters the + node's mempool, reaches the template, and is followed through mined to + confirmed; a tx under the fee floor is refused and taken back out of the + enforcer. Needs `SLIPSTREAM_ENFORCER_BIN`, an enforcer built with + `--enable-slipstream`, until a release ships it — so it is not in CI yet. + Own `.regtest/slipstream-e2e/` dir. + + SLIPSTREAM_ENFORCER_BIN=../bip300301_enforcer/target/debug/bip300301_enforcer \ + bash tests/test_slipstream_regtest.sh + - `test_pplns_coinbase_regtest.sh` — `pool_mode=pplns-coinbase`, which has no ledger step at all: the payment IS the block. Asserts the coinbase pays the window on chain, that no output pays anything the pool controls beyond its diff --git a/tests/test_slipstream_regtest.sh b/tests/test_slipstream_regtest.sh new file mode 100755 index 0000000..89126bc --- /dev/null +++ b/tests/test_slipstream_regtest.sh @@ -0,0 +1,250 @@ +#!/usr/bin/env bash +# End-to-end proof of the slipstream service, against a real chain. +# +# bitcoind-patched <-> bip300301_enforcer --enable-slipstream <-> slipstream/ +# +# What only a chain can prove, and what this asserts: +# +# 1. a tx the node refuses to relay (non-standard: a dust output) is +# accepted through slipstream, and never enters the node's mempool. +# Nothing is relayed, which is the point of the service. +# 2. it reaches the template the enforcer serves -- the one the proxy +# mines -- and the service sees it there (in_template). +# 3. mining that template confirms it, and the service follows it through +# mined to confirmed, with bitcoind supplying the depth. +# 4. the fee rule holds: a tx under the floor is refused, and taken back +# out of the enforcer so it cannot be mined for less than was asked. +# 5. every submission is kept, refused ones included. +# 6. info.json names the slipstream URL and the software. +# +# The proxy is not run: the service talks only to the enforcer and bitcoind, +# and the template is mined directly with generateblock, which accepts a +# block only if the node finds the whole of it valid. +# +# Env: +# SLIPSTREAM_ENFORCER_BIN REQUIRED until an enforcer release ships +# --enable-slipstream (LayerTwo-Labs/ +# bip300301_enforcer#642): a bip300301_enforcer +# binary built from that branch. +# REGTEST_DIR data dir, WIPED each run (default: /.regtest/slipstream-e2e) +# REGTEST_BIN_DIR binary cache for bitcoind (default: /.regtest/bin) +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$HERE/.." && pwd)" +export REGTEST_DIR="${REGTEST_DIR:-$ROOT/.regtest/slipstream-e2e}" +CACHE_BIN_DIR="${REGTEST_BIN_DIR:-$ROOT/.regtest/bin}" +export REGTEST_SKIP_THUNDER=1 +export REGTEST_WALLETLESS=1 +export REGTEST_ENFORCER_EXTRA_ARGS="--enable-slipstream" + +: "${SLIPSTREAM_ENFORCER_BIN:?set SLIPSTREAM_ENFORCER_BIN to an enforcer built with --enable-slipstream}" +[ -x "$SLIPSTREAM_ENFORCER_BIN" ] || { echo "not executable: $SLIPSTREAM_ENFORCER_BIN" >&2; exit 1; } + +SVC_LOG="$REGTEST_DIR/slipstream.log" +SVC_DB="$REGTEST_DIR/slipstream.db" +SVC_PID="" +BIN="$REGTEST_DIR/bin" + +cli() { "$BIN/bitcoin-cli" -datadir="$REGTEST_DIR/data/bitcoind" -regtest \ + -rpcuser=user -rpcpassword=password "$@"; } +wcli() { cli -rpcwallet=miner "$@"; } +stage() { echo; echo "=== slipstream-e2e: $1"; } +fail() { echo "FAIL: $*" >&2; exit 1; } +api() { curl -sf "http://127.0.0.1:$SVC_PORT$1"; } +post() { curl -s -X POST --data-binary "$1" "http://127.0.0.1:$SVC_PORT/api/tx"; } + +dump_logs() { + echo "!!! slipstream-e2e FAILED — recent logs:" >&2 + for f in "$REGTEST_DIR"/logs/*.log "$SVC_LOG"; do + [ -f "$f" ] || continue + echo "--- tail $f" >&2 + tail -40 "$f" >&2 + done +} + +cleanup() { + [ -n "$SVC_PID" ] && kill "$SVC_PID" 2>/dev/null || true + REGTEST_BIN_DIR="$BIN" "$ROOT/scripts/regtest/stop.sh" || true + rm -rf "$LOCK" +} + +mkdir -p "$(dirname "$REGTEST_DIR")" +LOCK="$REGTEST_DIR.lock" +if ! mkdir "$LOCK" 2>/dev/null; then + echo "FAIL: $LOCK exists — another run of this suite is active." >&2 + echo " REGTEST_DIR=$REGTEST_DIR scripts/regtest/stop.sh && rm -rf $LOCK" >&2 + exit 1 +fi +trap 'code=$?; [ "$code" -ne 0 ] && dump_logs; cleanup; exit $code' EXIT +trap 'exit 130' INT TERM + +for dep in sqlite3 jq node nc curl python3; do + command -v "$dep" >/dev/null 2>&1 || { echo "$dep not installed" >&2; exit 1; } +done + +PICKED="" +pick_port() { + local p + while :; do + p=$(( (RANDOM % 20000) + 20001 )) + [[ " $PICKED " == *" $p "* ]] && continue + nc -z 127.0.0.1 "$p" 2>/dev/null && continue + PICKED="$PICKED $p" + printf -v "$1" '%s' "$p" + return + done +} + +stage "allocate stack ports" +pick_port REGTEST_BITCOIND_RPC_PORT +pick_port REGTEST_BITCOIND_ZMQ_PORT +pick_port REGTEST_ENFORCER_RPC_PORT +pick_port REGTEST_ENFORCER_GRPC_PORT +pick_port SVC_PORT +export REGTEST_BITCOIND_RPC_PORT REGTEST_BITCOIND_ZMQ_PORT \ + REGTEST_ENFORCER_RPC_PORT REGTEST_ENFORCER_GRPC_PORT + +stage "wipe data dir (fresh chain every run)" +rm -rf "$REGTEST_DIR/data" "$REGTEST_DIR/logs" "$REGTEST_DIR/run" "$BIN" +rm -f "$SVC_DB" "$SVC_DB-wal" "$SVC_DB-shm" "$SVC_LOG" + +stage "binaries: cached bitcoind, slipstream enforcer" +REGTEST_BIN_DIR="$CACHE_BIN_DIR" "$ROOT/scripts/regtest/setup.sh" >/dev/null +mkdir -p "$BIN" +ln -sf "$CACHE_BIN_DIR/bitcoind" "$BIN/bitcoind" +ln -sf "$CACHE_BIN_DIR/bitcoin-cli" "$BIN/bitcoin-cli" +ln -sf "$SLIPSTREAM_ENFORCER_BIN" "$BIN/bip300301_enforcer" +"$BIN/bip300301_enforcer" --help | grep -q -- '--enable-slipstream' \ + || fail "$SLIPSTREAM_ENFORCER_BIN has no --enable-slipstream" + +stage "start bitcoind-patched + walletless enforcer with slipstream" +REGTEST_BIN_DIR="$BIN" "$ROOT/scripts/regtest/start.sh" >/dev/null + +stage "mature coins for the node wallet" +MINER_ADDR=$(wcli getnewaddress) +cli generatetoaddress 110 "$MINER_ADDR" >/dev/null + +gbt() { + curl -s --data-binary \ + '{"jsonrpc":"2.0","id":"t","method":"getblocktemplate","params":[{"rules":["segwit"],"capabilities":["coinbasetxn"]}]}' \ + -H 'content-type: application/json' "http://127.0.0.1:$REGTEST_ENFORCER_RPC_PORT" +} +wait_template_on_tip() { + local tip + tip=$(cli getbestblockhash) + for _ in $(seq 1 60); do + [ "$(gbt | jq -r '.result.previousblockhash // empty')" = "$tip" ] && return + sleep 1 + done + fail "the enforcer's template never reached tip $tip" +} +wait_template_on_tip + +stage "start the slipstream service" +( cd "$ROOT/slipstream" && [ -d node_modules ] || npm ci --silent --no-audit --no-fund ) +( + cd "$ROOT/slipstream" + ENFORCER_GBT_URL="http://127.0.0.1:$REGTEST_ENFORCER_RPC_PORT" \ + BITCOIND_RPC_URL="http://127.0.0.1:$REGTEST_BITCOIND_RPC_PORT" \ + BITCOIND_RPC_USER=user BITCOIND_RPC_PASS=password \ + SLIPSTREAM_DB_PATH="$SVC_DB" \ + PROXY_DB_PATH="$REGTEST_DIR/no-proxy.db" \ + SLIPSTREAM_PORT="$SVC_PORT" \ + SLIPSTREAM_POLL_MS=500 \ + SLIPSTREAM_CONFIRMATIONS=3 \ + PUBLIC_SLIPSTREAM_URL="http://127.0.0.1:$SVC_PORT" \ + exec node index.js +) > "$SVC_LOG" 2>&1 & +SVC_PID=$! +for _ in $(seq 1 30); do api /healthz >/dev/null 2>&1 && break; sleep 1; done +api /healthz >/dev/null || fail "the service never became healthy" + +# A spend of one wallet UTXO at `fee_rate` sat/vB paying `extra` as a second +# output. A 1-sat `extra` is dust, which the node refuses to relay. +build_tx() { + local fee_sats="$1" extra_btc="$2" + local utxo txid vout amount pay dust raw + utxo=$(wcli listunspent 100 | jq -c '[.[] | select(.amount >= 1)][0]') + txid=$(jq -r .txid <<<"$utxo"); vout=$(jq -r .vout <<<"$utxo") + amount=$(jq -r .amount <<<"$utxo") + wcli lockunspent false "[{\"txid\":\"$txid\",\"vout\":$vout}]" >/dev/null + pay=$(wcli getnewaddress); dust=$(wcli getnewaddress) + local send + send=$(python3 -c "print(f'{$amount - $fee_sats/1e8 - $extra_btc:.8f}')") + raw=$(wcli createrawtransaction "[{\"txid\":\"$txid\",\"vout\":$vout}]" \ + "[{\"$pay\":$send},{\"$dust\":$extra_btc}]") + wcli signrawtransactionwithwallet "$raw" | jq -r .hex +} + +stage "1. a non-standard tx is accepted, and the node never sees it" +# ~141 vB for a P2WPKH one-in, two-out: 1_000 sat is ~7 sat/vB +TX_HEX=$(build_tx 1000 0.00000001) +[ "$(cli testmempoolaccept "[\"$TX_HEX\"]" | jq -r '.[0].allowed')" = "false" ] \ + || fail "the fixture tx is supposed to be refused by the node's policy" +RESP=$(post "$TX_HEX") +echo " $RESP" +[ "$(jq -r .accepted <<<"$RESP")" = "true" ] || fail "slipstream refused the tx: $RESP" +TXID=$(jq -r .txid <<<"$RESP") +cli getrawmempool | jq -e --arg t "$TXID" 'index($t) == null' >/dev/null \ + || fail "the slipstream tx reached the node's mempool" + +stage "2. it reaches the template" +for _ in $(seq 1 30); do + [ "$(api "/api/tx/$TXID" | jq -r .tx.status)" = "in_template" ] && break + sleep 1 +done +[ "$(api "/api/tx/$TXID" | jq -r .tx.status)" = "in_template" ] \ + || fail "never seen in the template: $(api "/api/tx/$TXID")" +gbt | jq -e --arg t "$TXID" '[.result.transactions[].txid] | index($t) != null' >/dev/null \ + || fail "not in the enforcer's template" + +stage "3. mining the template confirms it" +TEMPLATE_TXS=$(gbt | jq -c '[.result.transactions[].data]') +BLOCK=$(cli generateblock "$MINER_ADDR" "$TEMPLATE_TXS" | jq -r .hash) +echo " mined $BLOCK" +for _ in $(seq 1 30); do + [ "$(api "/api/tx/$TXID" | jq -r .tx.status)" = "mined" ] && break + sleep 1 +done +TX_JSON=$(api "/api/tx/$TXID") +[ "$(jq -r .tx.status <<<"$TX_JSON")" = "mined" ] || fail "never seen mined: $TX_JSON" +[ "$(jq -r .tx.mined_block_hash <<<"$TX_JSON")" = "$BLOCK" ] || fail "wrong block: $TX_JSON" +cli generatetoaddress 2 "$MINER_ADDR" >/dev/null +for _ in $(seq 1 30); do + [ "$(api "/api/tx/$TXID" | jq -r .tx.status)" = "confirmed" ] && break + sleep 1 +done +TX_JSON=$(api "/api/tx/$TXID") +[ "$(jq -r .tx.status <<<"$TX_JSON")" = "confirmed" ] || fail "never confirmed: $TX_JSON" +jq -r '.events[].event' <<<"$TX_JSON" | tr '\n' ' '; echo +# A poll can land between the block connecting and the enforcer reporting +# it, and see the tx out of the template but not yet mined: a `pending` in +# between is correct, so it is not part of what is asserted. +[ "$(jq -r '[.events[].event | select(. != "pending")] | join(",")' <<<"$TX_JSON")" \ + = "accepted,in_template,mined,confirmed" ] || fail "unexpected history: $TX_JSON" + +stage "4. under the fee floor: refused, and not left in the enforcer" +wait_template_on_tip +# 20 sat for ~141 vB is ~0.14 sat/vB: over the node's relay floor, under ours +LOW_HEX=$(build_tx 20 0.00001000) +RESP=$(post "$LOW_HEX") +echo " $RESP" +[ "$(jq -r .reject_reason <<<"$RESP")" = "fee-rate-too-low" ] || fail "low fee not refused: $RESP" +LOW_TXID=$(jq -r .txid <<<"$RESP") +gbt | jq -e --arg t "$LOW_TXID" '[.result.transactions[].txid] | index($t) == null' >/dev/null \ + || fail "the refused tx is still in the enforcer's template" + +stage "5. every submission is kept" +SUBS=$(sqlite3 "$SVC_DB" "SELECT accepted || ':' || COALESCE(reject_reason, '') FROM slipstream_submissions ORDER BY id" | tr '\n' ' ') +echo " $SUBS" +[ "$SUBS" = "1: 0:fee-rate-too-low " ] || fail "unexpected submission log: $SUBS" + +stage "6. info.json" +INFO=$(api /info.json) +[ "$(jq -r .software <<<"$INFO")" = "simplepool" ] || fail "info.json: $INFO" +[ "$(jq -r .slipstream_url <<<"$INFO")" = "http://127.0.0.1:$SVC_PORT" ] || fail "info.json: $INFO" +api /api/fees | jq -c . + +echo +echo "=== slipstream-e2e: PASS" From ed1c4b4bb154c2867add4b7e19c571dc61aac3da Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 28 Sep 2026 11:28:43 +0200 Subject: [PATCH 2/3] slipstream: submit to the pool's bitcoind, not an enforcer-side pool Suggested by the enforcer's maintainer, and simpler by a wide margin: the enforcer's template mempool mirrors the pool's bitcoind over ZMQ, so a tx that node accepts reaches the templates the proxy mines with no enforcer change at all. The enforcer-side pool this replaces (cusf-enforcer-mempool#130, bip300301_enforcer#642) is closed. A tx is now checked with testmempoolaccept and broadcast with sendrawtransaction. Nothing can be taken back out of a mempool, so the fee rule runs between the two: a tx that pays too little is refused before it is ever sent. Non-standard txs need the node to run -acceptnonstdtxn, which Core allows only off mainnet; BIP300 deposits are standard on a patched node already. The trade: a tx is relayed like any other, so another pool may mine it. mined_by_pool says whose block it was. Tracking follows the node. A tx that leaves the mempool unmined is sent again, and the node's answer decides between pending and dropped (with its reason); an orphaned block's tx comes back to pending when the node restores it. There is no expiry and no withdrawal: Core has its own mempool expiry and no RPC to remove a tx. tests/test_slipstream_regtest.sh now runs against the stock release enforcer, so it runs in CI. scripts/regtest/start.sh is back as it was. --- .github/workflows/integration_tests.yaml | 9 + README.md | 23 +- deploy/docker/docker-compose.yml | 11 +- scripts/regtest/start.sh | 6 +- slipstream/README.md | 100 ++++----- slipstream/index.js | 24 +-- slipstream/lib/config.js | 35 +-- slipstream/lib/http.js | 2 +- slipstream/lib/rpc.js | 88 ++++---- slipstream/lib/slipstream.js | 261 +++++++++++------------ slipstream/lib/store.js | 16 +- slipstream/test/helpers.js | 83 +++---- slipstream/test/submit.test.js | 43 ++-- slipstream/test/tracker.test.js | 102 ++++----- tests/README.md | 17 +- tests/test_slipstream_regtest.sh | 106 ++++----- 16 files changed, 470 insertions(+), 456 deletions(-) diff --git a/.github/workflows/integration_tests.yaml b/.github/workflows/integration_tests.yaml index acaa230..e02c352 100644 --- a/.github/workflows/integration_tests.yaml +++ b/.github/workflows/integration_tests.yaml @@ -71,6 +71,13 @@ jobs: - name: Run coinbase-direct PPLNS end-to-end regtest test run: bash tests/test_pplns_coinbase_regtest.sh + # Slipstream: a tx submitted to the service is checked and broadcast to + # the node, reaches the enforcer's template through its mempool mirror, + # and is followed to confirmed. No proxy; the template is mined with + # generateblock. + - name: Run slipstream end-to-end regtest test + run: bash tests/test_slipstream_regtest.sh + - name: Upload logs if: failure() uses: actions/upload-artifact@v4 @@ -81,6 +88,8 @@ jobs: .regtest-e2e/logs/ .regtest-pplns/logs/ .regtest-cbwin/logs/ + .regtest/slipstream-e2e/logs/ + .regtest/slipstream-e2e/slipstream.log /tmp/simplepool-e2e.log /tmp/simplepool-e2e.conf /tmp/simplepool-int.log diff --git a/README.md b/README.md index dbf9ba6..44c82b4 100644 --- a/README.md +++ b/README.md @@ -595,15 +595,16 @@ the same host. ### Slipstream [`slipstream/`](slipstream/) is an optional service that takes a raw tx from -anyone and gets it into the pool's blocks **without relaying it** — for BIP300/ -301 txs (deposits, withdrawal bundles, BMM requests) the network will not relay, -or any other consensus-valid tx. It hands each one to the enforcer's block -template server, the same one the proxy mines from, which needs -`--enable-slipstream` (LayerTwo-Labs/bip300301_enforcer#642). The fee rule is -Slipstream's: the higher of a 1 sat/vB floor and the current mineable rate. -Every submission is kept, and each accepted tx is followed from template to -block. It also serves the pool's `info.json` for pool directories. See -[`slipstream/README.md`](slipstream/README.md). +anyone and gets it into the pool's blocks, including txs the network will not +relay: BIP300/301 txs, or any other consensus-valid tx. It checks each one +against the pool's own bitcoind (`testmempoolaccept`) and the fee rule, then +broadcasts it there; the enforcer's template mempool mirrors that node's, so it +reaches the templates the proxy mines with no enforcer change. Non-standard txs +need the node to run `-acceptnonstdtxn` (which Core allows only off mainnet). +The fee rule is Slipstream's: the higher of a 1 sat/vB floor and the current +mineable rate. Every submission is kept, and each accepted tx is followed from +template to block. It also serves the pool's `info.json` for pool directories. +See [`slipstream/README.md`](slipstream/README.md). ## Config keys @@ -761,8 +762,8 @@ scripts/ dashboard/ # Node/Express read-only stats UI payout/ # payout worker: Thunder rail (pps-classic, pplns-thunder) # and L1 rail via the enforcer wallet (pplns-btc) -slipstream/ # slipstream service: takes txs from anyone and mines them - # without relaying, via the enforcer; serves info.json +slipstream/ # slipstream service: takes txs from anyone and gets them + # into the pool's blocks via its bitcoind; serves info.json docs/simplepool.html # single-file explainer: every mode, end to end ``` diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml index f2d0b0b..7c2fb47 100644 --- a/deploy/docker/docker-compose.yml +++ b/deploy/docker/docker-compose.yml @@ -111,12 +111,15 @@ services: container_name: simplepool-slipstream restart: unless-stopped environment: - # The enforcer's block template server -- the proxy's bitcoind_url -- - # running with --enable-slipstream. - ENFORCER_GBT_URL: ${ENFORCER_GBT_URL:-http://host.docker.internal:8122} - BITCOIND_RPC_URL: ${BITCOIND_RPC_URL:-} + # The pool's bitcoind -- the node the enforcer mirrors its mempool + # from. Txs are checked and broadcast here. Run it with + # -acceptnonstdtxn to take non-standard txs (not allowed on mainnet). + BITCOIND_RPC_URL: ${BITCOIND_RPC_URL:-http://host.docker.internal:8332} BITCOIND_RPC_USER: ${BITCOIND_RPC_USER:-} BITCOIND_RPC_PASS: ${BITCOIND_RPC_PASS:-} + # The enforcer's block template server -- the proxy's bitcoind_url. + # Read only: the template is where the mineable rate comes from. + ENFORCER_GBT_URL: ${ENFORCER_GBT_URL:-http://host.docker.internal:8122} SLIPSTREAM_DB_PATH: /data/slipstream.db PROXY_DB_PATH: /data/shares.db SLIPSTREAM_BIND: 0.0.0.0 diff --git a/scripts/regtest/start.sh b/scripts/regtest/start.sh index 0cdc3c6..6d09c8a 100755 --- a/scripts/regtest/start.sh +++ b/scripts/regtest/start.sh @@ -15,9 +15,6 @@ # CreateDepositTransaction) are unavailable. # Mine with MiningService/GenerateToAddress # (enforcer PR #477). -# REGTEST_ENFORCER_EXTRA_ARGS -# further enforcer flags, space-separated, e.g. -# --enable-slipstream (tests/test_slipstream_regtest.sh) set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" @@ -122,8 +119,7 @@ start_if_dead bip300301_enforcer \ --node-zmq-addr-sequence=tcp://127.0.0.1:$BITCOIND_ZMQ_PORT \ --enable-block-template-server \ --serve-rpc-addr=127.0.0.1:$ENFORCER_RPC_PORT \ - --serve-grpc-addr=127.0.0.1:$ENFORCER_GRPC_PORT \ - ${REGTEST_ENFORCER_EXTRA_ARGS:-} + --serve-grpc-addr=127.0.0.1:$ENFORCER_GRPC_PORT wait_for bip300301_enforcer "nc -z 127.0.0.1 $ENFORCER_RPC_PORT" 30 # Thunder connects to the enforcer's gRPC — make sure that's actually diff --git a/slipstream/README.md b/slipstream/README.md index e89d8b3..3356ae1 100644 --- a/slipstream/README.md +++ b/slipstream/README.md @@ -1,27 +1,37 @@ # simplepool-slipstream -Takes a raw transaction from anyone and gets it into the pool's blocks -**without relaying it**. The first use is BIP300/301 transactions that the -network will not relay, such as deposits, withdrawal bundles and BMM requests, -but any consensus-valid transaction qualifies. +Takes a raw transaction from anyone and gets it into the pool's blocks. That +includes transactions the network will not relay: BIP300/301 transactions +(deposits, withdrawal bundles, BMM requests) or any other consensus-valid +transaction. ``` -submitter ──POST /api/tx──▶ slipstream ──submitslipstreamtx──▶ enforcer :8122 ◀── simplepool proxy - │ (template mempool) (mines it) - └── follows it: template → block → confirmed +submitter ──POST /api/tx──▶ slipstream ──testmempoolaccept, sendrawtransaction──▶ bitcoind + │ │ ZMQ mempool mirror + │ ▼ + └── reads the template ◀──────────────── enforcer :8122 ◀── simplepool proxy ``` -The transaction goes straight to the enforcer's block template server, the -same one the proxy mines from. The enforcer has the node check it for -consensus validity and keeps it in its template mempool. It is never handed to -the node's mempool and never relayed. The node's relay policy does not apply, -but consensus does, and so do the enforcer's own BIP300 rules: a deposit that -skips the sidechain's CTIP is refused. - -**Requires** `bip300301_enforcer` running with `--enable-block-template-server ---enable-slipstream` (LayerTwo-Labs/bip300301_enforcer#642). The enforcer's -template server has no authentication, so keep `:8122` on loopback. This -service is the public face. +Each transaction goes to the pool's own bitcoind, the node the enforcer syncs +its template mempool from. Once the node accepts it, it reaches the templates +the proxy mines with no further step and **no enforcer change**. The +enforcer's own BIP300 rules still apply to it on the way in. + +## The node + +- **Non-standard transactions** need the node to run **`-acceptnonstdtxn=1`**. + Core refuses that flag on mainnet with `acceptnonstdtxn is not currently + supported for main chain`. This was checked against the drivechain-patched, + ecash betanet and stock builds. It works on signet and regtest. + - drynet3 reports itself as `main`, so non-standard transactions there need a + patched Core that lifts the check. +- **BIP300 deposits** are standard on a drivechain-patched node already, so + they need no flag. +- **Transactions are relayed.** The node broadcasts an accepted transaction + like any other, so another pool can mine it. `mined_by_pool` records whose + block it was. +- **Nothing can be withdrawn.** Core has no RPC to remove a transaction from + its mempool, which is why the fee rule is applied *before* broadcast. ## The fee rule @@ -32,13 +42,9 @@ the current mineable rate**. - **Mineable rate.** This is read from the template the proxy is mining now. While the template has room, it equals the floor. Once the template is full, it is the fee rate of the cheapest transaction in it. -- **When it's checked.** The rate is checked once, at submission. Accepted - transactions then compete for the block by fee rate like any other. A - transaction that loses its place stays pending until it is mined or expires. - -A transaction that passes the enforcer but pays too little is removed from the -enforcer again before the refusal is returned, so it cannot be mined for less -than was asked. +- **Checked before anything is sent.** `testmempoolaccept` reports the fee and + size without broadcasting, and only a transaction that passes both the node + and the rule reaches `sendrawtransaction`. ## API @@ -48,7 +54,7 @@ Every endpoint is readable cross-origin. |---|---| | `GET /info.json` (also `/api/info`) | The pool, for pool directories. | | `GET /api/fees` | `{min_submission_rate, mineable_rate, required_rate, template_height, template_weight, updated_at}` | -| `POST /api/tx` (also `/tx`) | The raw transaction as hex, either as a text body or as JSON `{"hex": "..."}`. Returns `{accepted: true, txid, status, ...}` or `{accepted: false, reject_reason}`. Reject reasons are the node's (`missing-inputs`, `bad-txns-...`, `mandatory-script-verify-flag-failed ...`) or this service's (`fee-rate-too-low`, `invalid-hex`, `tx-size`, `rate-limited`, `rejected-by-enforcer`). | +| `POST /api/tx` (also `/tx`) | The raw transaction as hex, either as a text body or as JSON `{"hex": "..."}`. Returns `{accepted: true, txid, status, ...}` or `{accepted: false, reject_reason}`. Reject reasons are the node's (`missing-inputs`, `bad-txns-...`, `mandatory-script-verify-flag-failed ...`, `scriptpubkey` when the node does not run `-acceptnonstdtxn`) or this service's (`fee-rate-too-low`, `invalid-hex`, `tx-size`, `rate-limited`). | | `GET /api/tx/:txid` | `{tx, events}`: where it stands now, and every status change it went through. | | `GET /api/txs?status=&limit=` | Recent transactions, newest first. | | `GET /healthz` | `503` while the enforcer is unreachable. | @@ -57,26 +63,22 @@ Every endpoint is readable cross-origin. curl -s -X POST --data-binary "$RAW_TX_HEX" https://slipstream.example/api/tx ``` -## What "propagated" means here - -Nothing is ever relayed, so the statuses track only two things: whether the -transaction reached a template, and whether a block carried it. +## Statuses | status | | |---|---| -| `pending` | In the enforcer's mempool, but not in the latest template. | -| `in_template` | In the latest template, so miners are working on it now. | -| `mined` | In a block, fewer than `SLIPSTREAM_CONFIRMATIONS` (default 6) deep. `mined_by_pool` says whether the block was ours (from `blocks_found`). | +| `pending` | In the node's mempool, but not in the latest template. | +| `in_template` | In the latest template, so the pool's miners are working on it now. | +| `mined` | In a block, fewer than `SLIPSTREAM_CONFIRMATIONS` (default 6) deep. `mined_by_pool` says whose block it was (from `blocks_found`). | | `confirmed` | That deep. | -| `dropped` | Will not be mined. `status_reason` says why: `conflict_mined` (a block spent one of its inputs), `parent_removed`, `rejected_by_enforcer`, `withdrawn`, or the node's reason for refusing a resubmission. | -| `expired` | Waited `SLIPSTREAM_EXPIRY_BLOCKS` (default 144) without being mined, and was withdrawn. | +| `dropped` | Left the mempool unmined, and the node refused it when it was sent again. `status_reason` is the node's reason, e.g. `insufficient fee, rejecting replacement ...` or `bad-txns-inputs-missingorspent`. | -This service keeps the record, and the enforcer keeps only its mempool. When -the enforcer loses a transaction, the service resubmits it and counts it in -`resubmissions`. That covers: -- an enforcer restart -- a reorg, because the enforcer evicts every slipstream transaction when a block disconnects -- the transaction's block being orphaned +- **Leaving the mempool.** A transaction can leave the mempool without being + mined: evicted, expired, replaced, or its input spent by a block. The + service then sends it again, and the node's answer decides between `pending` + and `dropped`. +- **Orphaned blocks.** When a transaction's block is orphaned, the node puts + it back in its mempool itself, and the transaction becomes `pending` again. ## Storage @@ -86,7 +88,7 @@ only for `pool_meta` and `blocks_found`. - `slipstream_submissions`: every POST exactly as it arrived, accepted or not. - `slipstream_txs`: one row per accepted transaction, including the raw - transaction so it can be resubmitted. + transaction so it can be sent again. - `slipstream_events`: every status change, append-only. ## info.json @@ -107,22 +109,20 @@ environment variables: ``` cd slipstream && npm ci -ENFORCER_GBT_URL=http://127.0.0.1:8122 \ BITCOIND_RPC_URL=http://127.0.0.1:8332 BITCOIND_RPC_COOKIE_FILE=/var/lib/bitcoind/.cookie \ +ENFORCER_GBT_URL=http://127.0.0.1:8122 \ PROXY_DB_PATH=../data/shares.db \ PUBLIC_SLIPSTREAM_URL=https://slipstream.example \ node index.js ``` -Without `BITCOIND_RPC_URL`, a mined transaction's depth is only estimated -from the template height, and an orphaned block goes unnoticed. The full list -of variables is in [`lib/config.js`](lib/config.js). The service listens on -`127.0.0.1:8124`, and is published through nginx with +The full list of variables is in [`lib/config.js`](lib/config.js). The service +listens on `127.0.0.1:8124`, and is published through nginx with `SLIPSTREAM_TRUST_PROXY=1`, so the rate limit applies per client. ## Tests -- `npm test`: unit tests. These run against a fake enforcer and bitcoind, and +- `npm test`: unit tests. These run against a fake bitcoind and enforcer, and cover every status transition. -- `tests/test_slipstream_regtest.sh`: end to end, against a real enforcer - (see [`tests/README.md`](../tests/README.md)). +- `tests/test_slipstream_regtest.sh`: end to end, against a real node and the + stock enforcer. It runs in CI. diff --git a/slipstream/index.js b/slipstream/index.js index cd65d13..9ca4ab0 100644 --- a/slipstream/index.js +++ b/slipstream/index.js @@ -1,15 +1,16 @@ -/* simplepool-slipstream — take txs from anyone, mine them without relaying. +/* simplepool-slipstream — take txs from anyone, mine them in the pool's blocks. * - * Submissions go straight to the enforcer's block template server, the one - * the proxy mines from, which keeps them in its template mempool and never - * hands them to the node's. This service records every submission, holds - * each one to the fee rule, and follows it from template to block. + * Submissions are checked against the pool's own bitcoind and the fee rule, + * then broadcast to it. The enforcer's template mempool mirrors that node's, + * so an accepted tx reaches the templates the proxy mines. This service + * records every submission and follows each accepted tx from template to + * block. * * Config is environment-only — see lib/config.js for the full list. * * Run: - * ENFORCER_GBT_URL=http://127.0.0.1:8122 \ * BITCOIND_RPC_URL=http://127.0.0.1:8332 BITCOIND_RPC_COOKIE_FILE=... \ + * ENFORCER_GBT_URL=http://127.0.0.1:8122 \ * PROXY_DB_PATH=../data/shares.db \ * node index.js */ @@ -33,18 +34,15 @@ const log = { error: (m) => console.error(`[error] ${m}`), }; -log.info(`simplepool-slipstream ${version} starting (enforcer=${cfg.enforcerUrl} db=${cfg.dbPath})`); +log.info(`simplepool-slipstream ${version} starting ` + + `(node=${cfg.bitcoind.url} enforcer=${cfg.enforcerUrl} db=${cfg.dbPath})`); log.info(` fee floor ${cfg.minFeeRate} sat/vB, confirmed at ${cfg.confirmations}, ` + - `expiry ${cfg.expiryBlocks} blocks, poll ${cfg.pollMs}ms`); -if (!cfg.bitcoind) { - log.warn('BITCOIND_RPC_URL unset: mined txs are never checked for orphaning, ' + - 'and their depth is only estimated from the template height'); -} + `poll ${cfg.pollMs}ms`); const slipstream = new Slipstream({ store: openStore(cfg.dbPath), + bitcoind: new BitcoindClient(cfg.bitcoind), enforcer: new EnforcerClient({ url: cfg.enforcerUrl }), - bitcoind: cfg.bitcoind ? new BitcoindClient(cfg.bitcoind) : null, pool: openPoolDb(cfg.proxyDbPath), cfg, log, diff --git a/slipstream/lib/config.js b/slipstream/lib/config.js index 4b67b02..26b7606 100644 --- a/slipstream/lib/config.js +++ b/slipstream/lib/config.js @@ -1,10 +1,22 @@ /* Slipstream service config, loaded from environment variables. * * Required: + * BITCOIND_RPC_URL the pool's own bitcoind, the node the enforcer + * syncs its mempool from. Txs are checked + * (testmempoolaccept) and broadcast + * (sendrawtransaction) here, and followed with + * getmempoolentry, getrawtransaction and + * getblockheader. For non-standard txs it must run + * -acceptnonstdtxn, which Core allows only off + * mainnet; BIP300 deposits are standard on a + * drivechain-patched node without it. + * BITCOIND_RPC_USER / BITCOIND_RPC_PASS, or BITCOIND_RPC_COOKIE_FILE * ENFORCER_GBT_URL the enforcer's block template server, the same * endpoint the proxy mines from (bitcoind_url in - * proxy.conf), e.g. http://127.0.0.1:8122. The - * enforcer must run with --enable-slipstream. + * proxy.conf), e.g. http://127.0.0.1:8122. Read + * only: the template it serves is what the pool is + * mining, so it is where the mineable rate comes + * from and how a tx is seen to be in play. * * Optional: * SLIPSTREAM_DB_PATH this service's own SQLite file (default @@ -19,11 +31,6 @@ * comes from pool_meta, and blocks_found says * whether the block that mined a tx was ours. * Without it both are reported as unknown. - * BITCOIND_RPC_URL bitcoind JSON-RPC, read-only use (getblockheader, - * getrawtransaction). Without it a mined tx is - * never confirmed or noticed orphaned: the - * enforcer's template server cannot answer either. - * BITCOIND_RPC_USER / BITCOIND_RPC_PASS, or BITCOIND_RPC_COOKIE_FILE * SLIPSTREAM_PORT HTTP port (default 8124) * SLIPSTREAM_BIND bind address (default 127.0.0.1; put nginx in * front to publish it) @@ -35,8 +42,6 @@ * as Slipstream states it. * SLIPSTREAM_CONFIRMATIONS depth at which a mined tx counts as confirmed * (default 6) - * SLIPSTREAM_EXPIRY_BLOCKS blocks a tx may wait unmined before it is - * withdrawn (default 144, about a day) * SLIPSTREAM_POLL_MS how often to read the template and follow every * open tx (default 5000) * SLIPSTREAM_RATE_LIMIT_PER_MIN @@ -70,26 +75,24 @@ function str(name) { } export function loadConfig() { - const enforcerUrl = str('ENFORCER_GBT_URL'); - if (!enforcerUrl) { - throw new Error('ENFORCER_GBT_URL is required (the enforcer block template server)'); + for (const name of ['BITCOIND_RPC_URL', 'ENFORCER_GBT_URL']) { + if (!str(name)) throw new Error(`${name} is required`); } return { - enforcerUrl, + enforcerUrl: str('ENFORCER_GBT_URL'), dbPath: str('SLIPSTREAM_DB_PATH') || '../data/slipstream.db', proxyDbPath: str('PROXY_DB_PATH') || '../data/shares.db', - bitcoind: str('BITCOIND_RPC_URL') ? { + bitcoind: { url: str('BITCOIND_RPC_URL'), user: str('BITCOIND_RPC_USER'), pass: str('BITCOIND_RPC_PASS'), cookieFile: str('BITCOIND_RPC_COOKIE_FILE'), - } : null, + }, port: num('SLIPSTREAM_PORT', 8124, { min: 1 }), bind: str('SLIPSTREAM_BIND') || '127.0.0.1', trustProxy: process.env.SLIPSTREAM_TRUST_PROXY === '1', minFeeRate: num('SLIPSTREAM_MIN_FEE_RATE', 1, { min: 0 }), confirmations: num('SLIPSTREAM_CONFIRMATIONS', 6, { min: 1 }), - expiryBlocks: num('SLIPSTREAM_EXPIRY_BLOCKS', 144, { min: 1 }), pollMs: num('SLIPSTREAM_POLL_MS', 5000, { min: 100 }), rateLimitPerMin: num('SLIPSTREAM_RATE_LIMIT_PER_MIN', 30, { min: 1 }), presentation: { diff --git a/slipstream/lib/http.js b/slipstream/lib/http.js index 8d4088d..154d4bb 100644 --- a/slipstream/lib/http.js +++ b/slipstream/lib/http.js @@ -22,7 +22,7 @@ import { summary } from './slipstream.js'; const MAX_BODY_BYTES = 2 * 1_000_000 + 1024; const TXID_RE = /^[0-9a-f]{64}$/; -const STATUSES = ['pending', 'in_template', 'mined', 'confirmed', 'dropped', 'expired']; +const STATUSES = ['pending', 'in_template', 'mined', 'confirmed', 'dropped']; /* Fixed-window per-address counter. Crude, and enough: a submission costs * the enforcer a node round trip, so the point is only to bound that. */ diff --git a/slipstream/lib/rpc.js b/slipstream/lib/rpc.js index ab73efc..c79c141 100644 --- a/slipstream/lib/rpc.js +++ b/slipstream/lib/rpc.js @@ -1,4 +1,4 @@ -/* JSON-RPC clients for the enforcer's block template server and bitcoind. +/* JSON-RPC clients for bitcoind and the enforcer's block template server. * * Both speak the same wire format; they differ only in auth. The template * server has none, bitcoind takes basic auth from user/pass or its cookie @@ -69,53 +69,69 @@ export class RpcClient { } } -/* The enforcer's block template server, as far as slipstream needs it. */ -export class EnforcerClient { - constructor(opts) { - this.rpc = new RpcClient(opts); - } +/* RPC_INVALID_ADDRESS_OR_KEY: what bitcoind answers for an unknown txid or + * block hash. */ +const NOT_FOUND = -5; - /* The enforcer serves only coinbasetxn templates, and says so rather - * than falling back. */ - getBlockTemplate() { - return this.rpc.call('getblocktemplate', [{ - rules: ['segwit'], - capabilities: ['coinbasetxn'], - }]); +const orNull = async (promise) => { + try { + return await promise; + } catch (e) { + if (e instanceof RpcError && e.code === NOT_FOUND) return null; + throw e; } +}; - submit(txHex) { return this.rpc.call('submitslipstreamtx', [txHex]); } - status(txid) { return this.rpc.call('getslipstreamtx', [txid]); } - remove(txid) { return this.rpc.call('removeslipstreamtx', [txid]); } -} - -/* bitcoind, read-only. */ +/* The pool's own bitcoind: where a slipstream tx is checked, broadcast and + * followed. The enforcer's template mempool mirrors this node's mempool, so + * a tx accepted here reaches the templates the proxy mines. */ export class BitcoindClient { constructor(opts) { this.rpc = new RpcClient(opts); } - /* Confirmations of `blockHash`, -1 once it has left the main chain, or - * null if bitcoind does not know it. */ - async blockConfirmations(blockHash) { - try { - const header = await this.rpc.call('getblockheader', [blockHash, true]); - return { confirmations: header.confirmations, height: header.height }; - } catch (e) { - if (e instanceof RpcError && e.code === -5) return null; - throw e; - } + /* testmempoolaccept for one tx: { txid, wtxid, allowed, vsize, fees, + * 'reject-reason' }. Checks everything sendrawtransaction would, and + * broadcasts nothing. */ + async testAccept(txHex) { + const [result] = await this.rpc.call('testmempoolaccept', [[txHex]]); + return result; } + send(txHex) { return this.rpc.call('sendrawtransaction', [txHex]); } + + /* { vsize, weight, fees: { base }, ... }, or null outside the mempool */ + mempoolEntry(txid) { return orNull(this.rpc.call('getmempoolentry', [txid])); } + /* The block a tx is confirmed in, if it is confirmed at all. Needs * txindex, which the enforcer already requires of this node. */ async txBlock(txid) { - try { - const tx = await this.rpc.call('getrawtransaction', [txid, true]); - return tx.blockhash && tx.confirmations > 0 ? tx.blockhash : null; - } catch (e) { - if (e instanceof RpcError && e.code === -5) return null; - throw e; - } + const tx = await orNull(this.rpc.call('getrawtransaction', [txid, true])); + return tx?.blockhash && tx.confirmations > 0 ? tx.blockhash : null; + } + + /* { confirmations, height } of a block, confirmations -1 once it has left + * the main chain; null if the node does not know it. */ + async blockConfirmations(blockHash) { + const header = await orNull(this.rpc.call('getblockheader', [blockHash, true])); + return header ? { confirmations: header.confirmations, height: header.height } : null; + } +} + +/* The enforcer's block template server: read only for the template the + * proxy is mining, which is where the mineable rate comes from and how a tx + * is seen to be in play. */ +export class EnforcerClient { + constructor(opts) { + this.rpc = new RpcClient(opts); + } + + /* The enforcer serves only coinbasetxn templates, and says so rather + * than falling back. */ + getBlockTemplate() { + return this.rpc.call('getblocktemplate', [{ + rules: ['segwit'], + capabilities: ['coinbasetxn'], + }]); } } diff --git a/slipstream/lib/slipstream.js b/slipstream/lib/slipstream.js index 85d14f2..61c12de 100644 --- a/slipstream/lib/slipstream.js +++ b/slipstream/lib/slipstream.js @@ -1,15 +1,18 @@ /* Submission and tracking. * - * The enforcer's block template server holds the txs; this keeps the record - * of them. A submission is handed straight to the enforcer, which has the - * node check it for consensus validity and then keeps it in the template - * mempool, never relaying it. After that, every poll asks the enforcer where - * each open tx stands and reads the template the proxy is mining, and moves - * the row accordingly. + * A submission goes to the pool's own bitcoind. The enforcer's template + * mempool mirrors that node's mempool, so a tx the node accepts reaches the + * templates the proxy mines with no further step. The node needs + * `-acceptnonstdtxn` for it to take non-standard txs; BIP300 deposits are + * standard on a drivechain-patched node already. * - * The enforcer forgets a tx when it leaves its mempool, and forgets all of - * them when it restarts or a block is disconnected. This side remembers, so - * that is where resubmission lives. + * Nothing can be taken back out of a node's mempool, so the fee rule is + * applied BEFORE anything is broadcast: `testmempoolaccept` reports the fee + * and size without broadcasting, and only a tx that passes both the node and + * the rule is sent. Once sent it is relayed like any other. + * + * After that, every poll reads the template the proxy is mining and asks the + * node where each open tx stands, and moves the row accordingly. */ import { RpcError } from './rpc.js'; @@ -17,15 +20,20 @@ import { OPEN_STATUSES } from './store.js'; import { feeRate, feeSnapshot } from './fees.js'; const now = () => Math.floor(Date.now() / 1000); +const btcToSats = (btc) => Math.round(btc * 1e8); -/* A serialized tx is at most the 1M wu the enforcer accepts, i.e. 1MB. */ +/* A serialized tx is at most a block, 4M wu, i.e. 4MB; nothing near that is + * useful, and a POST body has to be bounded somewhere. 1MB of tx. */ export const MAX_TX_HEX_LEN = 2 * 1_000_000; +/* testmempoolaccept's reasons for a tx the node already has */ +const ALREADY_KNOWN = ['txn-already-in-mempool', 'txn-already-known']; + export class Slipstream { - constructor({ store, enforcer, bitcoind = null, pool, cfg, log }) { + constructor({ store, bitcoind, enforcer, pool, cfg, log }) { this.store = store; - this.enforcer = enforcer; this.bitcoind = bitcoind; + this.enforcer = enforcer; this.pool = pool; this.cfg = cfg; this.log = log; @@ -53,7 +61,7 @@ export class Slipstream { return this.template; } - /* Hand one tx to the enforcer and record the outcome. Returns + /* Check one tx, and broadcast it if it passes. Returns * `{ httpStatus, body }`. Every call is logged as a submission, whatever * becomes of it. */ async submit(rawHex, submitter) { @@ -69,63 +77,79 @@ export class Slipstream { } if (hex.length > MAX_TX_HEX_LEN) return refuse(400, 'tx-size'); - let resp; + let check; try { - resp = await this.enforcer.submit(hex); + check = await this.bitcoind.testAccept(hex); } catch (e) { - if (e instanceof RpcError) { - if (e.code === -22) return refuse(400, 'tx-decode-failed'); - if (e.code === -32601) return refuse(503, 'slipstream-disabled'); - if (e.code === -10) return refuse(503, 'enforcer-syncing'); - } - this.log.error(`submitslipstreamtx failed: ${e.message}`); - return refuse(502, 'enforcer-unavailable'); - } - const { txid } = resp; - if (!resp.accepted) { - return refuse(200, resp.reject_reason ?? 'rejected', { txid }); + if (e instanceof RpcError && e.code === -22) return refuse(400, 'tx-decode-failed'); + this.log.error(`testmempoolaccept failed: ${e.message}`); + return refuse(502, 'node-unavailable'); } + const { txid } = check; const existing = this.store.get(txid); - if (existing && !['dropped', 'expired'].includes(existing.status)) { + if (existing && !['dropped'].includes(existing.status)) { record({ txid, accepted: true }); return { httpStatus: 200, body: { accepted: true, already_tracked: true, ...summary(existing) } }; } + // Fee and size: from the check, or from the mempool for a tx the node + // already has, which is then taken on and followed like any other. + let feeSats; + let vsize; + let alreadyInMempool = false; + if (check.allowed) { + feeSats = btcToSats(check.fees.base); + vsize = check.vsize; + } else if (ALREADY_KNOWN.includes(check['reject-reason'])) { + const entry = await this.bitcoind.mempoolEntry(txid); + if (!entry) return refuse(200, 'already-confirmed', { txid }); + feeSats = btcToSats(entry.fees.base); + vsize = entry.vsize; + alreadyInMempool = true; + } else { + return refuse(200, check['reject-reason'] ?? 'rejected', { txid }); + } + if (!this.template) { try { await this.refreshTemplate(); } catch { /* the floor still applies */ } } const fees = this.fees(); - const rate = feeRate(resp.fee_sat, resp.vsize); + const rate = feeRate(feeSats, vsize); if (rate < fees.required_rate) { - // Already in the enforcer's mempool by now: take it back out, or - // it is mined for less than was asked. - try { - await this.enforcer.remove(txid); - } catch (e) { - this.log.error(`removeslipstreamtx ${txid} after a low fee failed: ${e.message}`); - } return refuse(200, 'fee-rate-too-low', { txid, fee_rate: rate, required_fee_rate: fees.required_rate, }); } + if (!alreadyInMempool) { + try { + await this.bitcoind.send(hex); + } catch (e) { + // Lost a race with something the check did not see, e.g. a + // conflicting tx arriving in between. + if (e instanceof RpcError) return refuse(200, e.rpcMessage, { txid }); + this.log.error(`sendrawtransaction ${txid} failed: ${e.message}`); + return refuse(502, 'node-unavailable', { txid }); + } + } + const entry = await this.bitcoind.mempoolEntry(txid); const row = { txid, - wtxid: resp.wtxid, + wtxid: check.wtxid, raw_hex: hex, - vsize: resp.vsize, - weight: resp.weight, - fee_sats: resp.fee_sat, + vsize, + weight: entry?.weight ?? vsize * 4, + fee_sats: feeSats, fee_rate: rate, required_fee_rate: fees.required_rate, submitted_height: this.template?.height ?? null, submitter: submitter ?? null, }; if (existing) { - // Dropped or expired before, and valid again now + // Dropped before, and valid again now this.store.touch(txid, { - raw_hex: hex, fee_sats: row.fee_sats, fee_rate: rate, + raw_hex: hex, fee_sats: feeSats, fee_rate: rate, required_fee_rate: row.required_fee_rate, submitted_at: now(), submitted_height: row.submitted_height, }); @@ -134,36 +158,26 @@ export class Slipstream { this.store.insertAccepted(row); } record({ txid, accepted: true }); - this.log.info(`accepted ${txid} at ${rate} sat/vB (required ${fees.required_rate})`); + this.log.info(`accepted ${txid} at ${rate} sat/vB (required ${fees.required_rate})` + + (alreadyInMempool ? ', already in the mempool' : '')); return { httpStatus: 200, body: { accepted: true, ...summary(this.store.get(txid)) } }; } - /* One pass over every tx still in play. Errors reaching the enforcer end - * the pass early; the next one picks up where it stood. */ + /* One pass over every tx still in play. An unreachable enforcer ends the + * pass before anything moves; the next one picks up where it stood. */ async tick() { const template = await this.refreshTemplate(); const inTemplate = new Set((template.transactions ?? []).map(tx => tx.txid)); for (const row of this.store.byStatus(OPEN_STATUSES)) { - await this._followOpen(row, template, inTemplate); + await this._followOpen(row, inTemplate); } for (const row of this.store.byStatus(['mined'])) { - await this._followMined(row, template); + await this._followMined(row); } } - async _followOpen(row, template, inTemplate) { - const status = await this.enforcer.status(row.txid); - switch (status.status) { - case 'pending': { - if (row.submitted_height !== null - && template.height - row.submitted_height >= this.cfg.expiryBlocks) { - await this.enforcer.remove(row.txid); - this.store.setStatus(row.txid, 'expired', { - detail: { submitted_height: row.submitted_height, height: template.height }, - }); - this.log.info(`expired ${row.txid} after ${this.cfg.expiryBlocks} blocks`); - return; - } + async _followOpen(row, inTemplate) { + if (await this.bitcoind.mempoolEntry(row.txid)) { const ts = now(); if (inTemplate.has(row.txid)) { this.store.setStatus(row.txid, 'in_template', { @@ -175,110 +189,81 @@ export class Slipstream { } return; } - case 'removed': - switch (status.reason) { - case 'mined': - return this._markMined(row, status.block_hash, template); - case 'reorged': - return this._resubmit(row, 'reorged'); - default: - // conflict_mined, parent_removed, rejected_by_enforcer, or a - // withdrawal that was not ours - this.store.setStatus(row.txid, 'dropped', { - reason: status.reason, - detail: { block_hash: status.block_hash, spent_by: status.spent_by, parent: status.parent }, - }); - this.log.info(`dropped ${row.txid}: ${status.reason}`); - return; - } - case 'unknown': { - // The enforcer restarted and lost it. It may have been mined - // while it was down. - const block = this.bitcoind ? await this.bitcoind.txBlock(row.txid) : null; - if (block) return this._markMined(row, block, template); - return this._resubmit(row, 'enforcer-forgot'); - } - default: - this.log.warn(`getslipstreamtx ${row.txid}: unexpected status ${JSON.stringify(status)}`); - } + const block = await this.bitcoind.txBlock(row.txid); + if (block) return this._markMined(row, block); + // Out of the mempool and not mined: evicted, expired, replaced, or its + // input spent by a block. Sending it again tells which. + return this._rebroadcast(row, 'left-mempool'); } - async _markMined(row, blockHash, template) { - let height = null; - let confirmations = null; - if (this.bitcoind) { - const header = await this.bitcoind.blockConfirmations(blockHash); - if (header) ({ height, confirmations } = header); - } else if (blockHash === template.previousblockhash) { - height = template.height - 1; - confirmations = 1; - } + async _markMined(row, blockHash) { + const header = await this.bitcoind.blockConfirmations(blockHash); this.store.setStatus(row.txid, 'mined', { fields: { mined_block_hash: blockHash, - mined_height: height, + mined_height: header?.height ?? null, mined_at: now(), mined_by_pool: boolOrNull(this.pool.foundBlock(blockHash)), - confirmations, + confirmations: header?.confirmations ?? null, }, - detail: { block_hash: blockHash, height }, + detail: { block_hash: blockHash, height: header?.height ?? null }, }); this.log.info(`mined ${row.txid} in ${blockHash}`); } - async _followMined(row, template) { - let confirmations; - if (this.bitcoind) { - const header = await this.bitcoind.blockConfirmations(row.mined_block_hash); - if (!header || header.confirmations < 0) { - // Its block left the main chain. Mined again elsewhere, or - // waiting to be. - const block = await this.bitcoind.txBlock(row.txid); - if (block && block !== row.mined_block_hash) { - return this._markMined(row, block, template); - } - return this._resubmit(row, 'orphaned'); + async _followMined(row) { + const header = await this.bitcoind.blockConfirmations(row.mined_block_hash); + if (!header || header.confirmations < 0) { + // Its block left the main chain. The node puts a disconnected + // block's txs back in its mempool when it can, so it may already + // be waiting again, or mined in the block that replaced it. + const block = await this.bitcoind.txBlock(row.txid); + if (block && block !== row.mined_block_hash) return this._markMined(row, block); + if (await this.bitcoind.mempoolEntry(row.txid)) { + return this._backToPending(row, 'orphaned'); } - confirmations = header.confirmations; - } else if (row.mined_height !== null) { - // Without the node an orphan cannot be seen, only depth - confirmations = template.height - row.mined_height; - } else { - return; + return this._rebroadcast(row, 'orphaned'); } - if (confirmations >= this.cfg.confirmations) { + if (header.confirmations >= this.cfg.confirmations) { this.store.setStatus(row.txid, 'confirmed', { - fields: { confirmations, confirmed_at: now() }, + fields: { confirmations: header.confirmations, confirmed_at: now() }, }); - this.log.info(`confirmed ${row.txid} (${confirmations} deep)`); + this.log.info(`confirmed ${row.txid} (${header.confirmations} deep)`); } else { - this.store.touch(row.txid, { confirmations }); + this.store.touch(row.txid, { confirmations: header.confirmations }); } } - async _resubmit(row, why) { - let resp; + _backToPending(row, why) { + this.store.setStatus(row.txid, 'pending', { + fields: { mined_block_hash: null, mined_height: null, mined_at: null, + mined_by_pool: null, confirmations: null }, + detail: { after: why }, + }); + this.log.info(`${row.txid} back in the mempool after ${why}`); + } + + async _rebroadcast(row, why) { try { - resp = await this.enforcer.submit(row.raw_hex); + await this.bitcoind.send(row.raw_hex); } catch (e) { - this.log.warn(`resubmitting ${row.txid} (${why}) failed: ${e.message}`); - return; - } - if (resp.accepted) { - this.store.touch(row.txid, { resubmissions: row.resubmissions + 1 }); - this.store.setStatus(row.txid, 'pending', { - fields: { mined_block_hash: null, mined_height: null, mined_at: null, - mined_by_pool: null, confirmations: null }, - detail: { resubmitted_after: why }, - }); - this.log.info(`resubmitted ${row.txid} after ${why}`); - } else { + if (!(e instanceof RpcError)) { + this.log.warn(`rebroadcasting ${row.txid} (${why}) failed: ${e.message}`); + return; + } + // -27: "Transaction already in block chain" -- mined, and the + // lookup above raced the block. The next poll finds its block. + if (e.code === -27) return; this.store.setStatus(row.txid, 'dropped', { - reason: resp.reject_reason ?? 'rejected', - detail: { resubmitted_after: why }, + reason: e.rpcMessage, + detail: { after: why }, }); - this.log.info(`dropped ${row.txid}: resubmission after ${why} refused (${resp.reject_reason})`); + this.log.info(`dropped ${row.txid}: the node refused it again after ${why} (${e.rpcMessage})`); + return; } + this.store.touch(row.txid, { resubmissions: row.resubmissions + 1 }); + this.store.event(row.txid, 'rebroadcast', { after: why }); + this._backToPending(row, why); } } diff --git a/slipstream/lib/store.js b/slipstream/lib/store.js index d7f915e..0c9e230 100644 --- a/slipstream/lib/store.js +++ b/slipstream/lib/store.js @@ -6,21 +6,21 @@ * arrived. The record of what was asked of the * pool, kept even when the tx was refused. * slipstream_txs one row per accepted tx, holding where it stands - * now: pending, in_template, mined, confirmed, - * dropped or expired. + * now: pending, in_template, mined, confirmed or + * dropped. * slipstream_events every status change, append-only, so a row's * history can be read back rather than inferred. * * Status meanings: - * pending in the enforcer's mempool, not in the latest template + * pending in the node's mempool, not in the latest template * in_template in the latest template, i.e. being mined on now * mined in a block, fewer than the configured confirmations deep * confirmed that deep - * dropped will not be mined: status_reason says why - * expired waited past the expiry and was withdrawn + * dropped left the mempool unmined, and the node refused it when it + * was sent again: status_reason is the node's reason * - * Nothing is ever relayed, so "propagated" here means only this: did the tx - * reach a template, and did a block carry it. + * A tx is relayed once the node accepts it, so another pool may mine it: + * mined_by_pool says whose block it was. */ import Database from 'better-sqlite3'; @@ -63,7 +63,7 @@ CREATE TABLE IF NOT EXISTS slipstream_txs ( mined_by_pool INTEGER, -- 1 ours, 0 not, NULL unknown confirmations INTEGER, confirmed_at INTEGER, - resubmissions INTEGER NOT NULL DEFAULT 0 + resubmissions INTEGER NOT NULL DEFAULT 0 -- rebroadcasts after leaving the mempool ); CREATE INDEX IF NOT EXISTS slipstream_txs_status_idx ON slipstream_txs(status); CREATE INDEX IF NOT EXISTS slipstream_txs_submitted_idx ON slipstream_txs(submitted_at); diff --git a/slipstream/test/helpers.js b/slipstream/test/helpers.js index f285247..b2c7a40 100644 --- a/slipstream/test/helpers.js +++ b/slipstream/test/helpers.js @@ -1,4 +1,4 @@ -/* A fake enforcer and bitcoind, scripted per test, around a real in-memory +/* A fake bitcoind and enforcer, scripted per test, around a real in-memory * store: the store is what the tests are about, so it is not faked. */ import { openStore } from '../lib/store.js'; @@ -17,7 +17,6 @@ export function baseCfg(overrides = {}) { return { minFeeRate: 1, confirmations: 6, - expiryBlocks: 144, rateLimitPerMin: 30, trustProxy: false, presentation: {}, @@ -25,53 +24,60 @@ export function baseCfg(overrides = {}) { }; } -/* An enforcer whose mempool is a Map of txid -> status, and whose - * template is whatever the test sets. */ -export class FakeEnforcer { +/* A node with a mempool (txid -> entry) and a chain (txid -> block hash, + * block hash -> header). */ +export class FakeBitcoind { constructor() { - this.template = { height: 100, previousblockhash: TIP, transactions: [] }; - this.statuses = new Map(); - this.submitted = []; - this.removed = []; - this.nextSubmit = null; + this.mempool = new Map(); + this.txBlocks = new Map(); + this.headers = new Map(); + this.sent = []; + this.nextTest = null; + this.sendError = null; } - async getBlockTemplate() { - if (this.down) throw new Error('connect ECONNREFUSED'); - return this.template; - } - - /* `nextSubmit` scripts the next answer; by default any tx is accepted as - * `txid` at 2 sat/vB. */ - async submit(hex) { - this.submitted.push(hex); - const next = this.nextSubmit ?? { txid: TXID_A, accepted: true, fee_sat: 200, vsize: 100 }; - this.nextSubmit = null; + /* By default any tx checks out as TXID_A, 100 vB paying 200 sat. */ + async testAccept(_hex) { + const next = this.nextTest ?? { txid: TXID_A, allowed: true, vsize: 100, fees: { base: 200e-8 } }; + this.nextTest = null; if (next instanceof RpcError) throw next; - const resp = { wtxid: next.txid, weight: (next.vsize ?? 100) * 4, ...next }; - if (resp.accepted) this.statuses.set(resp.txid, { status: 'pending', txid: resp.txid }); - return resp; + this._lastTest = { wtxid: next.txid, ...next }; + return this._lastTest; } - async status(txid) { - return this.statuses.get(txid) ?? { status: 'unknown', txid }; + async send(hex) { + this.sent.push(hex); + if (this.sendError) { + const e = this.sendError; + this.sendError = null; + throw e; + } + const { txid, vsize = 100, fees = { base: 200e-8 } } = this._lastTest ?? { txid: TXID_A }; + this.mempool.set(txid, { vsize, weight: vsize * 4, fees }); + return txid; } - async remove(txid) { - this.removed.push(txid); - this.statuses.set(txid, { status: 'removed', txid, reason: 'withdrawn' }); - return [txid]; + async mempoolEntry(txid) { return this.mempool.get(txid) ?? null; } + async txBlock(txid) { return this.txBlocks.get(txid) ?? null; } + async blockConfirmations(hash) { return this.headers.get(hash) ?? null; } + + /* Move a tx from the mempool into `block`, `confirmations` deep. */ + mine(txid, block, { height = 100, confirmations = 1 } = {}) { + this.mempool.delete(txid); + this.txBlocks.set(txid, block); + this.headers.set(block, { height, confirmations }); } } -export class FakeBitcoind { +export class FakeEnforcer { constructor() { - this.headers = new Map(); // block hash -> { confirmations, height } - this.txBlocks = new Map(); // txid -> block hash + this.template = { height: 100, previousblockhash: TIP, transactions: [] }; } - async blockConfirmations(hash) { return this.headers.get(hash) ?? null; } - async txBlock(txid) { return this.txBlocks.get(txid) ?? null; } + async getBlockTemplate() { + if (this.down) throw new Error('connect ECONNREFUSED'); + return this.template; + } } export function fakePool({ meta = null, ours = [] } = {}) { @@ -81,17 +87,18 @@ export function fakePool({ meta = null, ours = [] } = {}) { }; } -export function makeSlipstream({ cfg = {}, bitcoind = null, pool = fakePool() } = {}) { +export function makeSlipstream({ cfg = {}, pool = fakePool() } = {}) { + const bitcoind = new FakeBitcoind(); const enforcer = new FakeEnforcer(); const slipstream = new Slipstream({ store: openStore(':memory:'), - enforcer, bitcoind, + enforcer, pool, cfg: baseCfg(cfg), log: quietLog, }); - return { slipstream, enforcer, store: slipstream.store }; + return { slipstream, bitcoind, enforcer, store: slipstream.store }; } /* A template carrying `count` txs of `weight` wu each, paying `feeRate`. */ diff --git a/slipstream/test/submit.test.js b/slipstream/test/submit.test.js index a4dba58..517e674 100644 --- a/slipstream/test/submit.test.js +++ b/slipstream/test/submit.test.js @@ -9,56 +9,73 @@ const HEX = '0200000001' + '00'.repeat(60); const submissions = (store) => store.db.prepare('SELECT txid, accepted, reject_reason FROM slipstream_submissions ORDER BY id').all(); -test('an accepted tx is tracked as pending, and the submission logged', async () => { - const { slipstream, store } = makeSlipstream(); +test('an accepted tx is broadcast, tracked as pending, and the submission logged', async () => { + const { slipstream, bitcoind, store } = makeSlipstream(); const { httpStatus, body } = await slipstream.submit(HEX, '203.0.113.5'); assert.equal(httpStatus, 200); assert.equal(body.accepted, true); assert.equal(body.status, 'pending'); + assert.equal(body.fee_sats, 200); assert.equal(body.fee_rate, 2); assert.equal(body.raw_hex, undefined, 'the raw tx is not echoed'); + assert.deepEqual(bitcoind.sent, [HEX]); const row = store.get(TXID_A); assert.equal(row.raw_hex, HEX); + assert.equal(row.weight, 400); assert.equal(row.submitter, '203.0.113.5'); assert.equal(row.submitted_height, 100); assert.deepEqual(submissions(store), [{ txid: TXID_A, accepted: 1, reject_reason: null }]); assert.deepEqual(store.events(TXID_A).map(e => e.event), ['accepted']); }); -test('under the mineable rate: taken back out of the enforcer, and refused', async () => { - const { slipstream, enforcer, store } = makeSlipstream(); +test('under the mineable rate: refused before anything is broadcast', async () => { + const { slipstream, bitcoind, enforcer, store } = makeSlipstream(); enforcer.template = fullTemplate({ feeRate: 3 }); const { body } = await slipstream.submit(HEX, 'x'); assert.equal(body.accepted, false); assert.equal(body.reject_reason, 'fee-rate-too-low'); assert.equal(body.required_fee_rate, 3); - assert.deepEqual(enforcer.removed, [TXID_A]); + assert.deepEqual(bitcoind.sent, [], 'nothing can be taken back once sent, so nothing is sent'); assert.equal(store.get(TXID_A), null); assert.deepEqual(submissions(store), [{ txid: TXID_A, accepted: 0, reject_reason: 'fee-rate-too-low' }]); }); -test('refusals are logged too, with the reason', async () => { - const { slipstream, enforcer, store } = makeSlipstream(); +test("the node's refusals are passed on, and logged", async () => { + const { slipstream, bitcoind, store } = makeSlipstream(); assert.equal((await slipstream.submit('zz', 'x')).httpStatus, 400); - enforcer.nextSubmit = { txid: TXID_A, accepted: false, reject_reason: 'missing-inputs' }; + bitcoind.nextTest = { txid: TXID_A, allowed: false, 'reject-reason': 'missing-inputs' }; const refused = await slipstream.submit(HEX, 'x'); assert.equal(refused.httpStatus, 200); assert.equal(refused.body.reject_reason, 'missing-inputs'); - enforcer.nextSubmit = new RpcError('submitslipstreamtx', -32601, 'slipstream is disabled'); - assert.equal((await slipstream.submit(HEX, 'x')).httpStatus, 503); + bitcoind.nextTest = new RpcError('testmempoolaccept', -22, 'TX decode failed'); + assert.equal((await slipstream.submit(HEX, 'x')).httpStatus, 400); + + // Passed the check, lost a race before the broadcast + bitcoind.sendError = new RpcError('sendrawtransaction', -26, 'txn-mempool-conflict'); + assert.equal((await slipstream.submit(HEX, 'x')).body.reject_reason, 'txn-mempool-conflict'); assert.deepEqual(submissions(store).map(s => s.reject_reason), - ['invalid-hex', 'missing-inputs', 'slipstream-disabled']); + ['invalid-hex', 'missing-inputs', 'tx-decode-failed', 'txn-mempool-conflict']); assert.equal(store.get(TXID_A), null); }); +test('a tx the node already has is taken on without sending it again', async () => { + const { slipstream, bitcoind, store } = makeSlipstream(); + bitcoind.mempool.set(TXID_A, { vsize: 100, weight: 400, fees: { base: 500e-8 } }); + bitcoind.nextTest = { txid: TXID_A, allowed: false, 'reject-reason': 'txn-already-in-mempool' }; + const { body } = await slipstream.submit(HEX, 'x'); + assert.equal(body.accepted, true); + assert.equal(body.fee_sats, 500); + assert.deepEqual(bitcoind.sent, []); + assert.equal(store.get(TXID_A).status, 'pending'); +}); + test('resubmitting a tracked tx returns where it stands', async () => { - const { slipstream, enforcer } = makeSlipstream(); + const { slipstream } = makeSlipstream(); await slipstream.submit(HEX, 'x'); - enforcer.nextSubmit = { txid: TXID_A, accepted: true, already_present: true, fee_sat: 200, vsize: 100 }; const { body } = await slipstream.submit(HEX, 'y'); assert.equal(body.accepted, true); assert.equal(body.already_tracked, true); diff --git a/slipstream/test/tracker.test.js b/slipstream/test/tracker.test.js index 89a35f5..8f6f056 100644 --- a/slipstream/test/tracker.test.js +++ b/slipstream/test/tracker.test.js @@ -1,9 +1,8 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { - BLOCK_1, BLOCK_2, TIP, TXID_A, FakeBitcoind, fakePool, makeSlipstream, -} from './helpers.js'; +import { RpcError } from '../lib/rpc.js'; +import { BLOCK_1, BLOCK_2, TXID_A, fakePool, makeSlipstream } from './helpers.js'; const HEX = '0200000001' + '00'.repeat(60); @@ -31,12 +30,8 @@ test('in and out of the template, without an event per poll', async () => { }); test('mined by the pool, then confirmed', async () => { - const bitcoind = new FakeBitcoind(); - const { slipstream, enforcer, store } = await submitted({ - bitcoind, pool: fakePool({ ours: [BLOCK_1] }), - }); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); - bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); + const { slipstream, bitcoind, store } = await submitted({ pool: fakePool({ ours: [BLOCK_1] }) }); + bitcoind.mine(TXID_A, BLOCK_1, { height: 100, confirmations: 1 }); await slipstream.tick(); let row = store.get(TXID_A); assert.equal(row.status, 'mined'); @@ -44,7 +39,7 @@ test('mined by the pool, then confirmed', async () => { assert.equal(row.mined_height, 100); assert.equal(row.mined_by_pool, 1); - bitcoind.headers.set(BLOCK_1, { confirmations: 6, height: 100 }); + bitcoind.headers.set(BLOCK_1, { height: 100, confirmations: 6 }); await slipstream.tick(); row = store.get(TXID_A); assert.equal(row.status, 'confirmed'); @@ -52,90 +47,65 @@ test('mined by the pool, then confirmed', async () => { assert.deepEqual(eventsOf(store), ['accepted', 'mined', 'confirmed']); }); -test('without bitcoind, depth comes from the template height', async () => { - const { slipstream, enforcer, store } = await submitted(); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: TIP }); +test('mined by another pool: it was relayed, so that is recorded, not assumed away', async () => { + const { slipstream, bitcoind, store } = await submitted({ pool: fakePool({ ours: [] }) }); + bitcoind.mine(TXID_A, BLOCK_1); await slipstream.tick(); - assert.equal(store.get(TXID_A).mined_height, 99); assert.equal(store.get(TXID_A).mined_by_pool, 0); - enforcer.template = { ...enforcer.template, height: 105 }; - await slipstream.tick(); - assert.equal(store.get(TXID_A).status, 'confirmed'); }); -test('an orphaned block puts the tx back in the enforcer', async () => { - const bitcoind = new FakeBitcoind(); - const { slipstream, enforcer, store } = await submitted({ bitcoind }); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); - bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); +test('an orphaned block the node put the tx back from: pending again', async () => { + const { slipstream, bitcoind, store } = await submitted(); + bitcoind.mine(TXID_A, BLOCK_1); await slipstream.tick(); - bitcoind.headers.set(BLOCK_1, { confirmations: -1, height: 100 }); + bitcoind.headers.set(BLOCK_1, { height: 100, confirmations: -1 }); + bitcoind.txBlocks.delete(TXID_A); + bitcoind.mempool.set(TXID_A, { vsize: 100, weight: 400, fees: { base: 200e-8 } }); await slipstream.tick(); const row = store.get(TXID_A); assert.equal(row.status, 'pending'); assert.equal(row.mined_block_hash, null); - assert.equal(row.resubmissions, 1); - assert.equal(enforcer.submitted.length, 2); + assert.equal(row.resubmissions, 0, 'the node put it back, nothing was sent'); }); test('an orphaned tx mined again elsewhere follows its new block', async () => { - const bitcoind = new FakeBitcoind(); - const { slipstream, enforcer, store } = await submitted({ bitcoind }); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'mined', block_hash: BLOCK_1 }); - bitcoind.headers.set(BLOCK_1, { confirmations: 1, height: 100 }); + const { slipstream, bitcoind, store } = await submitted(); + bitcoind.mine(TXID_A, BLOCK_1); await slipstream.tick(); - bitcoind.headers.set(BLOCK_1, { confirmations: -1, height: 100 }); - bitcoind.headers.set(BLOCK_2, { confirmations: 1, height: 100 }); - bitcoind.txBlocks.set(TXID_A, BLOCK_2); + bitcoind.headers.set(BLOCK_1, { height: 100, confirmations: -1 }); + bitcoind.mine(TXID_A, BLOCK_2); await slipstream.tick(); assert.equal(store.get(TXID_A).status, 'mined'); assert.equal(store.get(TXID_A).mined_block_hash, BLOCK_2); - assert.equal(enforcer.submitted.length, 1, 'not resubmitted'); }); -test('a reorg, or an enforcer restart, resubmits', async () => { - for (const status of [ - { status: 'removed', txid: TXID_A, reason: 'reorged', block_hash: BLOCK_1 }, - { status: 'unknown', txid: TXID_A }, - ]) { - const { slipstream, enforcer, store } = await submitted(); - enforcer.statuses.set(TXID_A, status); - await slipstream.tick(); - assert.equal(store.get(TXID_A).status, 'pending', status.status); - assert.equal(store.get(TXID_A).resubmissions, 1, status.status); - } +test('out of the mempool, not mined: sent again, and pending if the node takes it', async () => { + const { slipstream, bitcoind, store } = await submitted(); + bitcoind.mempool.delete(TXID_A); // evicted, or a node restart without mempool.dat + await slipstream.tick(); + assert.equal(store.get(TXID_A).status, 'pending'); + assert.equal(store.get(TXID_A).resubmissions, 1); + assert.equal(bitcoind.sent.length, 2); + assert.ok(eventsOf(store).includes('rebroadcast')); }); -test('a resubmission the enforcer refuses drops the tx, with its reason', async () => { - const { slipstream, enforcer, store } = await submitted(); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason: 'reorged', block_hash: BLOCK_1 }); - enforcer.nextSubmit = { txid: TXID_A, accepted: false, reject_reason: 'bad-txns-inputs-missingorspent' }; +test("out of the mempool and refused again: dropped, with the node's reason", async () => { + const { slipstream, bitcoind, store } = await submitted(); + bitcoind.mempool.delete(TXID_A); // replaced, or its input spent by a block + bitcoind.sendError = new RpcError('sendrawtransaction', -25, 'bad-txns-inputs-missingorspent'); await slipstream.tick(); assert.equal(store.get(TXID_A).status, 'dropped'); assert.equal(store.get(TXID_A).status_reason, 'bad-txns-inputs-missingorspent'); }); -test('a conflict or a lost parent drops the tx', async () => { - for (const reason of ['conflict_mined', 'parent_removed', 'rejected_by_enforcer']) { - const { slipstream, enforcer, store } = await submitted(); - enforcer.statuses.set(TXID_A, { status: 'removed', txid: TXID_A, reason }); - await slipstream.tick(); - assert.equal(store.get(TXID_A).status, 'dropped', reason); - assert.equal(store.get(TXID_A).status_reason, reason); - } -}); - -test('a tx that waits past the expiry is withdrawn', async () => { - const { slipstream, enforcer, store } = await submitted({ cfg: { expiryBlocks: 10 } }); - enforcer.template = { ...enforcer.template, height: 110 }; +test('"already in block chain" on a rebroadcast waits for the block, rather than dropping', async () => { + const { slipstream, bitcoind, store } = await submitted(); + bitcoind.mempool.delete(TXID_A); + bitcoind.sendError = new RpcError('sendrawtransaction', -27, 'Transaction already in block chain'); await slipstream.tick(); - assert.equal(store.get(TXID_A).status, 'expired'); - assert.deepEqual(enforcer.removed, [TXID_A]); - // The withdrawal the expiry caused is not mistaken for someone else's - await slipstream.tick(); - assert.equal(store.get(TXID_A).status, 'expired'); + assert.equal(store.get(TXID_A).status, 'pending'); }); test('an unreachable enforcer fails the tick and changes nothing', async () => { diff --git a/tests/README.md b/tests/README.md index 6f1fb0d..6588acc 100644 --- a/tests/README.md +++ b/tests/README.md @@ -97,15 +97,14 @@ the version last validated against. summed. Own `.regtest-btcpay/` dir. - `test_slipstream_regtest.sh` — the slipstream service against a real - enforcer: a tx the node refuses to relay is accepted, never enters the - node's mempool, reaches the template, and is followed through mined to - confirmed; a tx under the fee floor is refused and taken back out of the - enforcer. Needs `SLIPSTREAM_ENFORCER_BIN`, an enforcer built with - `--enable-slipstream`, until a release ships it — so it is not in CI yet. - Own `.regtest/slipstream-e2e/` dir. - - SLIPSTREAM_ENFORCER_BIN=../bip300301_enforcer/target/debug/bip300301_enforcer \ - bash tests/test_slipstream_regtest.sh + node and the stock enforcer: a non-standard tx is accepted by a node + running `-acceptnonstdtxn`, reaches the enforcer's template through its + mempool mirror, and is followed through mined to confirmed; a tx under the + fee floor is refused before it is broadcast; a tx replaced in the node's + mempool is recorded as dropped with the node's reason. Own + `.regtest/slipstream-e2e/` dir. + + bash tests/test_slipstream_regtest.sh - `test_pplns_coinbase_regtest.sh` — `pool_mode=pplns-coinbase`, which has no ledger step at all: the payment IS the block. Asserts the coinbase pays the diff --git a/tests/test_slipstream_regtest.sh b/tests/test_slipstream_regtest.sh index 89126bc..c13706c 100755 --- a/tests/test_slipstream_regtest.sh +++ b/tests/test_slipstream_regtest.sh @@ -1,50 +1,49 @@ #!/usr/bin/env bash # End-to-end proof of the slipstream service, against a real chain. # -# bitcoind-patched <-> bip300301_enforcer --enable-slipstream <-> slipstream/ +# bitcoind-patched (-acceptnonstdtxn) <-> bip300301_enforcer <-> (proxy) +# ^ +# +-- slipstream/ (testmempoolaccept, sendrawtransaction) +# +# The enforcer is the stock release: it mirrors the node's mempool over ZMQ, +# so whatever the node accepts reaches its templates with no enforcer change. # # What only a chain can prove, and what this asserts: # -# 1. a tx the node refuses to relay (non-standard: a dust output) is -# accepted through slipstream, and never enters the node's mempool. -# Nothing is relayed, which is the point of the service. +# 1. a non-standard tx (a dust output) is accepted through slipstream by a +# node running -acceptnonstdtxn, and lands in the node's mempool. # 2. it reaches the template the enforcer serves -- the one the proxy # mines -- and the service sees it there (in_template). # 3. mining that template confirms it, and the service follows it through -# mined to confirmed, with bitcoind supplying the depth. -# 4. the fee rule holds: a tx under the floor is refused, and taken back -# out of the enforcer so it cannot be mined for less than was asked. -# 5. every submission is kept, refused ones included. -# 6. info.json names the slipstream URL and the software. +# mined to confirmed. +# 4. the fee rule holds, and holds BEFORE broadcast: a tx under the floor +# is refused and never reaches the node's mempool. Nothing can be taken +# back out of a mempool, so refusing after sending would be too late. +# 5. a tx replaced in the node's mempool (RBF) leaves it, is refused when +# sent again, and is recorded as dropped with the node's reason. +# 6. every submission is kept, refused ones included. +# 7. info.json names the slipstream URL and the software. # -# The proxy is not run: the service talks only to the enforcer and bitcoind, +# The proxy is not run: the service talks only to the node and the enforcer, # and the template is mined directly with generateblock, which accepts a # block only if the node finds the whole of it valid. # # Env: -# SLIPSTREAM_ENFORCER_BIN REQUIRED until an enforcer release ships -# --enable-slipstream (LayerTwo-Labs/ -# bip300301_enforcer#642): a bip300301_enforcer -# binary built from that branch. # REGTEST_DIR data dir, WIPED each run (default: /.regtest/slipstream-e2e) -# REGTEST_BIN_DIR binary cache for bitcoind (default: /.regtest/bin) +# REGTEST_BIN_DIR binary cache, kept across runs (default: /.regtest/bin) set -euo pipefail HERE="$(cd "$(dirname "$0")" && pwd)" ROOT="$(cd "$HERE/.." && pwd)" export REGTEST_DIR="${REGTEST_DIR:-$ROOT/.regtest/slipstream-e2e}" -CACHE_BIN_DIR="${REGTEST_BIN_DIR:-$ROOT/.regtest/bin}" +export REGTEST_BIN_DIR="${REGTEST_BIN_DIR:-$ROOT/.regtest/bin}" export REGTEST_SKIP_THUNDER=1 export REGTEST_WALLETLESS=1 -export REGTEST_ENFORCER_EXTRA_ARGS="--enable-slipstream" - -: "${SLIPSTREAM_ENFORCER_BIN:?set SLIPSTREAM_ENFORCER_BIN to an enforcer built with --enable-slipstream}" -[ -x "$SLIPSTREAM_ENFORCER_BIN" ] || { echo "not executable: $SLIPSTREAM_ENFORCER_BIN" >&2; exit 1; } SVC_LOG="$REGTEST_DIR/slipstream.log" SVC_DB="$REGTEST_DIR/slipstream.db" SVC_PID="" -BIN="$REGTEST_DIR/bin" +BIN="$REGTEST_BIN_DIR" cli() { "$BIN/bitcoin-cli" -datadir="$REGTEST_DIR/data/bitcoind" -regtest \ -rpcuser=user -rpcpassword=password "$@"; } @@ -65,7 +64,7 @@ dump_logs() { cleanup() { [ -n "$SVC_PID" ] && kill "$SVC_PID" 2>/dev/null || true - REGTEST_BIN_DIR="$BIN" "$ROOT/scripts/regtest/stop.sh" || true + "$ROOT/scripts/regtest/stop.sh" || true rm -rf "$LOCK" } @@ -106,20 +105,17 @@ export REGTEST_BITCOIND_RPC_PORT REGTEST_BITCOIND_ZMQ_PORT \ REGTEST_ENFORCER_RPC_PORT REGTEST_ENFORCER_GRPC_PORT stage "wipe data dir (fresh chain every run)" -rm -rf "$REGTEST_DIR/data" "$REGTEST_DIR/logs" "$REGTEST_DIR/run" "$BIN" +rm -rf "$REGTEST_DIR/data" "$REGTEST_DIR/logs" "$REGTEST_DIR/run" rm -f "$SVC_DB" "$SVC_DB-wal" "$SVC_DB-shm" "$SVC_LOG" -stage "binaries: cached bitcoind, slipstream enforcer" -REGTEST_BIN_DIR="$CACHE_BIN_DIR" "$ROOT/scripts/regtest/setup.sh" >/dev/null -mkdir -p "$BIN" -ln -sf "$CACHE_BIN_DIR/bitcoind" "$BIN/bitcoind" -ln -sf "$CACHE_BIN_DIR/bitcoin-cli" "$BIN/bitcoin-cli" -ln -sf "$SLIPSTREAM_ENFORCER_BIN" "$BIN/bip300301_enforcer" -"$BIN/bip300301_enforcer" --help | grep -q -- '--enable-slipstream' \ - || fail "$SLIPSTREAM_ENFORCER_BIN has no --enable-slipstream" +stage "download prebuilt binaries" +"$ROOT/scripts/regtest/setup.sh" >/dev/null -stage "start bitcoind-patched + walletless enforcer with slipstream" -REGTEST_BIN_DIR="$BIN" "$ROOT/scripts/regtest/start.sh" >/dev/null +stage "start bitcoind-patched (-acceptnonstdtxn) + walletless enforcer" +# The one node setting slipstream needs, for non-standard txs. Core allows it +# only off mainnet; on regtest it is simply a config line. +echo 'acceptnonstdtxn=1' >> "$REGTEST_DIR/data/bitcoind/bitcoin.conf" +"$ROOT/scripts/regtest/start.sh" >/dev/null stage "mature coins for the node wallet" MINER_ADDR=$(wcli getnewaddress) @@ -145,9 +141,9 @@ stage "start the slipstream service" ( cd "$ROOT/slipstream" && [ -d node_modules ] || npm ci --silent --no-audit --no-fund ) ( cd "$ROOT/slipstream" - ENFORCER_GBT_URL="http://127.0.0.1:$REGTEST_ENFORCER_RPC_PORT" \ BITCOIND_RPC_URL="http://127.0.0.1:$REGTEST_BITCOIND_RPC_PORT" \ BITCOIND_RPC_USER=user BITCOIND_RPC_PASS=password \ + ENFORCER_GBT_URL="http://127.0.0.1:$REGTEST_ENFORCER_RPC_PORT" \ SLIPSTREAM_DB_PATH="$SVC_DB" \ PROXY_DB_PATH="$REGTEST_DIR/no-proxy.db" \ SLIPSTREAM_PORT="$SVC_PORT" \ @@ -160,8 +156,9 @@ SVC_PID=$! for _ in $(seq 1 30); do api /healthz >/dev/null 2>&1 && break; sleep 1; done api /healthz >/dev/null || fail "the service never became healthy" -# A spend of one wallet UTXO at `fee_rate` sat/vB paying `extra` as a second -# output. A 1-sat `extra` is dust, which the node refuses to relay. +# A spend of one wallet UTXO paying `fee_sats`, with `extra` as a second +# output, signalling RBF. A 1-sat `extra` is dust: non-standard, so only a +# node running -acceptnonstdtxn takes it. build_tx() { local fee_sats="$1" extra_btc="$2" local utxo txid vout amount pay dust raw @@ -173,21 +170,20 @@ build_tx() { local send send=$(python3 -c "print(f'{$amount - $fee_sats/1e8 - $extra_btc:.8f}')") raw=$(wcli createrawtransaction "[{\"txid\":\"$txid\",\"vout\":$vout}]" \ - "[{\"$pay\":$send},{\"$dust\":$extra_btc}]") + "[{\"$pay\":$send},{\"$dust\":$extra_btc}]" 0 true) wcli signrawtransactionwithwallet "$raw" | jq -r .hex } -stage "1. a non-standard tx is accepted, and the node never sees it" +stage "1. a non-standard tx is accepted, and broadcast to the node" # ~141 vB for a P2WPKH one-in, two-out: 1_000 sat is ~7 sat/vB TX_HEX=$(build_tx 1000 0.00000001) -[ "$(cli testmempoolaccept "[\"$TX_HEX\"]" | jq -r '.[0].allowed')" = "false" ] \ - || fail "the fixture tx is supposed to be refused by the node's policy" RESP=$(post "$TX_HEX") echo " $RESP" [ "$(jq -r .accepted <<<"$RESP")" = "true" ] || fail "slipstream refused the tx: $RESP" TXID=$(jq -r .txid <<<"$RESP") -cli getrawmempool | jq -e --arg t "$TXID" 'index($t) == null' >/dev/null \ - || fail "the slipstream tx reached the node's mempool" +cli getrawmempool | jq -e --arg t "$TXID" 'index($t) != null' >/dev/null \ + || fail "the accepted tx is not in the node's mempool" +cli getmempoolentry "$TXID" >/dev/null stage "2. it reaches the template" for _ in $(seq 1 30); do @@ -224,7 +220,7 @@ jq -r '.events[].event' <<<"$TX_JSON" | tr '\n' ' '; echo [ "$(jq -r '[.events[].event | select(. != "pending")] | join(",")' <<<"$TX_JSON")" \ = "accepted,in_template,mined,confirmed" ] || fail "unexpected history: $TX_JSON" -stage "4. under the fee floor: refused, and not left in the enforcer" +stage "4. under the fee floor: refused before broadcast" wait_template_on_tip # 20 sat for ~141 vB is ~0.14 sat/vB: over the node's relay floor, under ours LOW_HEX=$(build_tx 20 0.00001000) @@ -232,15 +228,29 @@ RESP=$(post "$LOW_HEX") echo " $RESP" [ "$(jq -r .reject_reason <<<"$RESP")" = "fee-rate-too-low" ] || fail "low fee not refused: $RESP" LOW_TXID=$(jq -r .txid <<<"$RESP") -gbt | jq -e --arg t "$LOW_TXID" '[.result.transactions[].txid] | index($t) == null' >/dev/null \ - || fail "the refused tx is still in the enforcer's template" +cli getrawmempool | jq -e --arg t "$LOW_TXID" 'index($t) == null' >/dev/null \ + || fail "the refused tx was broadcast anyway" + +stage "5. replaced in the node's mempool: dropped, with the node's reason" +RBF_HEX=$(build_tx 2000 0.00001000) +RESP=$(post "$RBF_HEX") +[ "$(jq -r .accepted <<<"$RESP")" = "true" ] || fail "slipstream refused the tx: $RESP" +RBF_TXID=$(jq -r .txid <<<"$RESP") +wcli bumpfee "$RBF_TXID" >/dev/null +for _ in $(seq 1 30); do + [ "$(api "/api/tx/$RBF_TXID" | jq -r .tx.status)" = "dropped" ] && break + sleep 1 +done +RBF_JSON=$(api "/api/tx/$RBF_TXID") +echo " $(jq -c '{status: .tx.status, reason: .tx.status_reason}' <<<"$RBF_JSON")" +[ "$(jq -r .tx.status <<<"$RBF_JSON")" = "dropped" ] || fail "replacement not noticed: $RBF_JSON" -stage "5. every submission is kept" +stage "6. every submission is kept" SUBS=$(sqlite3 "$SVC_DB" "SELECT accepted || ':' || COALESCE(reject_reason, '') FROM slipstream_submissions ORDER BY id" | tr '\n' ' ') echo " $SUBS" -[ "$SUBS" = "1: 0:fee-rate-too-low " ] || fail "unexpected submission log: $SUBS" +[ "$SUBS" = "1: 0:fee-rate-too-low 1: " ] || fail "unexpected submission log: $SUBS" -stage "6. info.json" +stage "7. info.json" INFO=$(api /info.json) [ "$(jq -r .software <<<"$INFO")" = "simplepool" ] || fail "info.json: $INFO" [ "$(jq -r .slipstream_url <<<"$INFO")" = "http://127.0.0.1:$SVC_PORT" ] || fail "info.json: $INFO" From 3f75201db3ecffd3e037c2867d5387352564d870 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 28 Sep 2026 11:43:47 +0200 Subject: [PATCH 3/3] slipstream: deploy templates, docs/simplepool.html, and a spoofable rate-limit key - deploy/systemd/simplepool-slipstream.service and deploy/nginx/slipstream.conf, templates in the form install.sh already renders (@USER@ / @ROOT@), with the install steps in slipstream/README.md. install.sh does not set slipstream up yet. - docs/simplepool.html gains a Slipstream section (with its sequence diagram, generated like the others) and a Dashboard section listing every page and endpoint, plus slipstream.db in the data model. Also brought up to date where it had fallen behind: the proxy long-polls the enforcer, the payout worker runs in three modes, not one, and the footer still called this a solo and PPS pool. - The rate limit behind a proxy was keyed on the FIRST X-Forwarded-For hop, which the client writes itself, so any client could pick its own key. It is now the last hop, the one the proxy added, and the vhost template overwrites the header rather than appending to it. --- README.md | 5 +- deploy/nginx/slipstream.conf | 44 ++++ deploy/systemd/simplepool-slipstream.service | 55 ++++ docs/sequence-diagrams.py | 29 +++ docs/simplepool.html | 259 ++++++++++++++++++- slipstream/README.md | 49 +++- slipstream/lib/http.js | 11 +- slipstream/test/http.test.js | 10 +- 8 files changed, 443 insertions(+), 19 deletions(-) create mode 100644 deploy/nginx/slipstream.conf create mode 100644 deploy/systemd/simplepool-slipstream.service diff --git a/README.md b/README.md index 44c82b4..76ea9e0 100644 --- a/README.md +++ b/README.md @@ -604,7 +604,10 @@ need the node to run `-acceptnonstdtxn` (which Core allows only off mainnet). The fee rule is Slipstream's: the higher of a 1 sat/vB floor and the current mineable rate. Every submission is kept, and each accepted tx is followed from template to block. It also serves the pool's `info.json` for pool directories. -See [`slipstream/README.md`](slipstream/README.md). +It is not set up by `install.sh` yet: the systemd unit and nginx vhost are +templates in [`deploy/`](deploy/), and [`slipstream/README.md`](slipstream/README.md) +walks through installing them. [`docs/simplepool.html`](docs/simplepool.html) +explains it end to end. ## Config keys diff --git a/deploy/nginx/slipstream.conf b/deploy/nginx/slipstream.conf new file mode 100644 index 0000000..e13dd5b --- /dev/null +++ b/deploy/nginx/slipstream.conf @@ -0,0 +1,44 @@ +# slipstream.example — reverse proxy for the simplepool slipstream service. +# +# The service is bound to 127.0.0.1:8124 (see +# deploy/systemd/simplepool-slipstream.service) and keys its per-client +# submission limit on the X-Forwarded-For hop set here. Replace +# slipstream.example with your hostname, then: +# +# sudo cp deploy/nginx/slipstream.conf /etc/nginx/sites-available/ +# sudo ln -s /etc/nginx/sites-available/ /etc/nginx/sites-enabled/ +# sudo nginx -t && sudo systemctl reload nginx +# sudo certbot --nginx -d # TLS, edits this file in place + +server { + listen 80; + listen [::]:80; + server_name slipstream.example; + + # Reuses the dashboard's zone (deploy/nginx/pool-ratelimit.conf) for + # reads. Submissions have their own, stricter limit inside the service. + limit_req zone=pool_dash burst=20 nodelay; + + # A raw tx arrives as hex: up to 1MB of tx is 2MB of body. + client_max_body_size 3m; + + access_log /var/log/nginx/slipstream.example.access.log; + error_log /var/log/nginx/slipstream.example.error.log warn; + + location / { + proxy_pass http://127.0.0.1:8124; + proxy_http_version 1.1; + proxy_set_header Host $host; + # Overwritten, not appended: the service trusts this hop, so the + # client must not be able to put its own in front of it. + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 30s; + proxy_connect_timeout 5s; + } + + location = /healthz { + proxy_pass http://127.0.0.1:8124/healthz; + access_log off; + } +} diff --git a/deploy/systemd/simplepool-slipstream.service b/deploy/systemd/simplepool-slipstream.service new file mode 100644 index 0000000..38e5124 --- /dev/null +++ b/deploy/systemd/simplepool-slipstream.service @@ -0,0 +1,55 @@ +[Unit] +Description=simplepool slipstream (tx submission) +Documentation=https://github.com/LayerTwo-Labs/simplepool/blob/main/slipstream/README.md +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=@USER@ +Group=@USER@ +WorkingDirectory=@ROOT@/slipstream +# The pool's bitcoind: the node the enforcer mirrors its mempool from. Txs +# are checked (testmempoolaccept) and broadcast (sendrawtransaction) here. +# For non-standard txs it must run -acceptnonstdtxn, which Core allows only +# off mainnet. Credentials: user/pass, or BITCOIND_RPC_COOKIE_FILE instead. +Environment=BITCOIND_RPC_URL=http://127.0.0.1:8332 +# Environment=BITCOIND_RPC_USER= +# Environment=BITCOIND_RPC_PASS= +# Environment=BITCOIND_RPC_COOKIE_FILE=/var/lib/bitcoind/.cookie +# The enforcer's block template server -- the same URL as bitcoind_url in +# proxy.conf. Read only: the template it serves is where the mineable rate +# comes from and how a tx is seen to be in play. +Environment=ENFORCER_GBT_URL=http://127.0.0.1:8122 +Environment=SLIPSTREAM_DB_PATH=@ROOT@/data/slipstream.db +Environment=PROXY_DB_PATH=@ROOT@/data/shares.db +# Loopback only; nginx publishes it (deploy/nginx/slipstream.conf), and +# sets the X-Forwarded-For hop the rate limit is keyed on. +Environment=SLIPSTREAM_BIND=127.0.0.1 +Environment=SLIPSTREAM_PORT=8124 +Environment=SLIPSTREAM_TRUST_PROXY=1 +Environment=SLIPSTREAM_MIN_FEE_RATE=1 +# info.json presentation. Facts about the pool (mode, fee, coinbase tag, +# addresses) are read from pool_meta and are NOT set here. +# Environment=PUBLIC_SLIPSTREAM_URL=https://slipstream.example +# Environment=PUBLIC_STRATUM_URL=stratum+tcp://stratum.example:3334 +# Environment=PUBLIC_DASHBOARD_URL=https://pool.example +# Environment=POOL_NAME= +# Environment=POOL_OPERATOR= +# Environment=POOL_CHAIN= +# Environment=POOL_LOGO= +# Environment=POOL_CONTACT= +ExecStart=/usr/bin/node @ROOT@/slipstream/index.js +Restart=on-failure +RestartSec=5 +StandardOutput=journal +StandardError=journal + +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=full +ProtectHome=read-only +ReadWritePaths=@ROOT@/data + +[Install] +WantedBy=multi-user.target diff --git a/docs/sequence-diagrams.py b/docs/sequence-diagrams.py index 943deb2..bc4a3a6 100755 --- a/docs/sequence-diagrams.py +++ b/docs/sequence-diagrams.py @@ -274,6 +274,35 @@ def build(title, desc, actors, steps, accent="pplns", min_width=0): +# ---------------------------------------------------------- slipstream ----- +DIAGRAMS['slipstream'] = build( + 'slipstream: a tx from anyone, into the pool\'s blocks', + 'A submitter posts a raw tx. The slipstream service checks it against the ' + 'pool\'s bitcoind and the fee rule without broadcasting, then broadcasts it. ' + 'The enforcer mirrors that node\'s mempool, so the tx reaches the template ' + 'the pool mines; the service follows it to a block.', + [('sub', 'Submitter', 'any wallet'), + ('slip', 'slipstream', ':8124'), + ('node', 'bitcoind', '-acceptnonstdtxn'), + ('enf', 'enforcer', 'template server'), + POOL], + [('msg', 'sub', 'slip', 'POST /api/tx '), + ('msg', 'slip', 'node', 'testmempoolaccept'), + ('msg', 'node', 'slip', 'allowed? fee, vsize', 'dashed'), + ('self', 'slip', 'fee rule: max(floor, mineable)'), + ('note', 'Checked BEFORE it is sent: nothing can be taken back out of a mempool, so refusing afterwards would be too late.', 'warn'), + ('msg', 'slip', 'node', 'sendrawtransaction'), + ('msg', 'node', 'enf', 'mempool mirror (ZMQ)'), + ('note', 'From here it is an ordinary mempool tx: relayed to peers, and minable by any pool whose node took it.'), + ('msg', 'pool', 'enf', 'getblocktemplate'), + ('msg', 'enf', 'pool', 'a template carrying the tx', 'dashed'), + ('msg', 'slip', 'enf', 'poll: in the template?'), + ('msg', 'slip', 'node', 'poll: mined? how deep?'), + ('note', 'Every submission is kept, refusals included. Each accepted tx is followed to confirmed, or to dropped with the node\'s own reason.', 'win'), + ], min_width=600) + + + # ---- splicing ------------------------------------------------------------- # # Each figure sits between HTML comment markers so a regeneration replaces diff --git a/docs/simplepool.html b/docs/simplepool.html index fb29ed2..0dd9cae 100644 --- a/docs/simplepool.html +++ b/docs/simplepool.html @@ -308,6 +308,8 @@

simplepool

  • Auditing every number
  • The data model
  • Connect a miner
    • +
  • The dashboard
    • +
  • Slipstream
  • Configuration
  • Running one
  • What it can't do
    • @@ -710,6 +712,20 @@

    The stack

    pool:tip, pool:credits). SQLite stays authoritative; the publish is fire-and-forget and a Redis outage cannot cost you a share.

    + +

    + The proxy takes templates from the enforcer's template server, never from + bitcoind directly: only the enforcer's carry the BIP300/301 commitments a + sidechain needs. It long-polls (BIP22), so a new tip reaches miners as soon + as the enforcer has it rather than on the next poll. +

    + +

    + Optionally, the slipstream service sits beside the + proxy and takes transactions from anyone. It hands them to the pool's + bitcoind; the enforcer's template mempool mirrors that node's, so they reach + the pool's templates without any change to the enforcer. +

    @@ -2067,6 +2083,27 @@

    The data model

    it is stated here rather than enforced by a trigger that would hide it.

    + +

    The slipstream service keeps its own

    +

    + data/slipstream.db, written only by the + slipstream service, which opens + shares.db read-only for pool_meta and + blocks_found and never writes to it. +

    +
    + + + + + + + + + + +
    TableWhat it holds
    slipstream_submissionsevery POST exactly as it arrived, accepted or refused, with the reason — the record of what was asked of the pool
    slipstream_txsone row per accepted tx: fee and the rate it was held to, where it stands now, when it was first and last in a template, the block that mined it and whether that block was the pool's, and the raw tx, so it can be sent again
    slipstream_eventsevery status change, append-only, so a tx's history can be read back rather than inferred
    +
    @@ -2190,7 +2227,181 @@

    What will not disconnect you

    - + +
    +

    The dashboard

    +

    + Everything the proxy writes down, readable by anyone. The dashboard reads + shares.db and never keeps a second copy of the pool's config: + the mode, the fee, the rate and the ports it shows are the ones the proxy + wrote into pool_meta, so the page cannot disagree with the + process that did the work. +

    + +

    Public pages

    +
    + + + + + + + + + + +
    PathWhat it shows
    /hashrate, the leaderboard (per worker and per address), recent blocks, how to connect with each port's difficulty policy, and a card explaining every number on the page for this pool's mode
    /worker/:nameone worker: shares, hashrate, credits and payouts
    /blocksevery block the pool found, paginated, with its status and what it paid
    /templatesthe work being handed out: height, block value, fees, the sidechain commitments it carries, and the template history
    /slipstreamonly when slipstream runs: its fees, how to submit, and each recent submission with where it stands
    /healththe hard-failure checks — ledger arithmetic, rate, duplicates — with a 503 while any is failing, so an uptime monitor can watch it
    +
    + +

    JSON, for monitors and other tools

    +
    + + + + + + + + +
    PathWhat it returns
    /api/statuseverything at once: the pool's identity and totals, the node, the health checks, and which commit of each component is running. Always 200 — read health.ok
    /api/overview, /api/leaderboard, /api/worker/:name, /api/blocks, /api/templates, /api/nodethe data behind each page
    /api/versionsbuild provenance of simplepool, the enforcer, Thunder and bitcoind
    /healthzliveness only
    +
    + +

    Admin

    +

    + /admin, behind basic auth, is where the operator acts: balances + and what is owed, per-worker audits, deposits into the Thunder reserve, + payouts and a "pay now" trigger, and a tools page for the stuck cases. Every + write action is CSRF-gated, and every broadcast attempt is logged in + tx_attempts, successful or not. +

    +
    + + +
    +

    Slipstream

    +

    + An optional service that takes a raw transaction from anyone and gets it + into the pool's blocks — including the ones the network will not relay: + BIP300/301 deposits, withdrawal bundles and BMM requests, or anything else + that is consensus-valid. It borrows the fee rule of Marathon's Slipstream. +

    + +

    How a transaction gets in

    +

    + The service checks the transaction against the pool's own bitcoind with + testmempoolaccept, which reports the fee and size and + broadcasts nothing, applies the fee rule, and only then sends it with + sendrawtransaction. The enforcer's template mempool mirrors + that node's mempool, so the transaction reaches the template the proxy is + mining with no enforcer change — and with the enforcer's own BIP300 rules + still applied on the way in. +

    + + +
    +
    + + slipstream: a tx from anyone, into the pool's blocks + A submitter posts a raw tx. The slipstream service checks it against the pool's bitcoind and the fee rule without broadcasting, then broadcasts it. The enforcer mirrors that node's mempool, so the tx reaches the template the pool mines; the service follows it to a block. + + + + + + + Submitterany walletslipstream:8124bitcoind-acceptnonstdtxnenforcertemplate serversimplepool:3334 + + POST /api/tx <raw hex>testmempoolacceptallowed? fee, vsizefee rule: max(floor, mineable)Checked BEFORE it is sent: nothing can be taken back out of a mempool, so refusing afterwards would be too late.sendrawtransactionmempool mirror (ZMQ)From here it is an ordinary mempool tx: relayed to peers, and minable by any pool whose node took it.getblocktemplatea template carrying the txpoll: in the template?poll: mined? how deep?Every submission is kept, refusals included. Each accepted tx is followed to confirmed, or to dropped with the node's own reason. + +
    +
    + + +

    The fee rule

    +

    + A transaction must pay the higher of the minimum submission rate + and the current mineable rate. The minimum is + SLIPSTREAM_MIN_FEE_RATE, 1 sat/vB by default. The mineable + rate is read off the template being mined: while it has room, anything over + the minimum gets in, so the two are equal; once it is full, it is the rate + of the cheapest transaction it carries. The rule is checked once, at + submission; after that a transaction competes by fee rate like any other. +

    +
    + Why the order matters +

    + Nothing can be taken back out of a node's mempool. A transaction that + paid too little and was refused after it was sent would be + relayed and mined anyway, for less than was asked. So the rule runs + between the check and the broadcast, never after. +

    +
    + +

    Following it to a block

    +
    + + + + + + + + + +
    StatusMeans
    pendingin the node's mempool, not in the latest template
    in_templatein the latest template: the pool's miners are working on it now
    minedin a block, recorded as the pool's or another pool's from blocks_found
    confirmedSLIPSTREAM_CONFIRMATIONS deep, 6 by default
    droppedleft the mempool unmined, and the node refused it when it was sent again; the reason is the node's own, e.g. a replacement or a spent input
    +
    +

    + A transaction whose block is orphaned goes back to pending when + the node restores it to its mempool, which it does by itself. Every + submission is kept, refusals included, and every status change is logged. +

    + +

    info.json

    +

    + The service also answers GET /info.json, the listing a pool + directory reads. Its facts — mode (the exact + pool_mode), fee_bps, coinbase_tag, + operator_address, pool_btc_address — come from + pool_meta, never from the service's config. Name, chain, logo, + contact and the public URLs, slipstream_url among them, come + from its environment. +

    + +

    What it needs

    +
      +
    • + A node that takes non-standard transactions: + acceptnonstdtxn=1. Core refuses that setting on a chain that + reports itself as mainnet — drivechain-patched builds included — so a + mainnet-fork chain needs a Core patch that lifts the check before it can + take them. BIP300 deposits are standard on a drivechain-patched node + already and need no setting at all. +
      • +
    • + Acceptance of relay. An accepted transaction is broadcast + like any other, so another pool may mine it first. That is the price of + needing nothing from the enforcer, and the status table records whose + block it was. +
      • +
    +
    + +

    Configuration

    @@ -2244,7 +2455,7 @@

    Configuration

    bitcoind_user / bitcoind_pass— Optional — omit both for an unauthenticated backend and the call goes out with no auth header. Cookie auth is not supported. bitcoind_poll_interval_ms30000 - Template refresh. On a drivechain pool this is also the worst-case delay before a sidechain's BMM request can reach a job — lower it to 5000–10000 if sidechains need to merge-mine reliably. + Template refresh when the backend does not long-poll. The enforcer does (BIP22), and then the proxy wakes on each new tip instead and this only paces retries. Against a backend that does not, it is also the worst-case delay before a sidechain's BMM request can reach a job — lower it to 5000–10000 if sidechains need to merge-mine reliably. coinbase_tag/simplepool/ Short string baked into the coinbase scriptSig. db_path./data/shares.db @@ -2266,11 +2477,16 @@

    Configuration

    PAYOUT_SETTLE_INTERVAL_MS, PAYOUT_MIN_SATS, THUNDER_FROM_ADDRESS and friends. The Thunder reserve address is deliberately not a proxy key: the coinbase never touches Thunder, so - only the dashboard and the payout worker have any business knowing it. + only the dashboard and the payout worker have any business knowing it. The + slipstream service is configured the same way — + BITCOIND_RPC_URL, ENFORCER_GBT_URL, + SLIPSTREAM_MIN_FEE_RATE and the POOL_* / + PUBLIC_* presentation fields; slipstream/lib/config.js + has the full list.

    - +

    Running one

    @@ -2303,13 +2519,22 @@

    What runs

    - - - + + + +
    UnitModeJob
    simplepool.serviceboththe stratum proxy — the only thing that is strictly required
    simplepool-dashboard.servicebothread-only public stats on :8081, plus /admin behind basic auth
    simplepool-payout.servicepps-classicthe daily Thunder payout batch
    simplepool.serviceallthe stratum proxy — the only thing that is strictly required
    simplepool-dashboard.serviceallread-only public stats on :8081, plus /admin behind basic auth
    simplepool-payout.servicepps-classic, pplns-thunder, pplns-btcthe daily payout batch, over Thunder or on L1 through the enforcer's wallet
    simplepool-slipstream.serviceany, optionalthe slipstream service on :8124, published through nginx
    +

    + Slipstream is not set up by the installer yet. Its unit and nginx vhost are + templates in the repository — deploy/systemd/simplepool-slipstream.service + and deploy/nginx/slipstream.conf — and + slipstream/README.md walks through installing them. The + docker-compose stack runs it as a fourth container. +

    +

    Which commit is running

    The build commit is compiled into the binary, so simplepool @@ -2322,7 +2547,7 @@

    Which commit is running

    - +

    What it can't do

    @@ -2376,6 +2601,14 @@

    What it can't do

    dashboard says so plainly, rather than letting either half be found out when an order is cancelled. +
  • + Slipstream does not keep transactions private. It + broadcasts through the pool's bitcoind, so another pool can mine a + submitted transaction first, and it cannot withdraw one once sent. On a + chain that reports itself as mainnet the node cannot run + acceptnonstdtxn without a Core patch, so there it takes only + standard transactions — which includes BIP300 deposits. +
  • The submit ceiling bounds a flood, it does not end one. A mismatched connection is refused cheaply instead of validated, but it is @@ -2392,8 +2625,8 @@

    What it can't do