feat: add venue catalogues and binary downloads

Author: mrmx

.github/workflows/publish.yml

@@ -71,7 +71,7 @@ jobs:
           GH_TOKEN: ${{ github.token }}
         run: |
           VERSION="${GITHUB_REF_NAME#v}"
-          awk "/^## \[${VERSION}\]/{found=1; print; next} found && /^## \[/{exit} found{print}" \
+          awk "/^## \[${VERSION}\]/{found=1; next} found && /^## \[/{exit} found{print}" \
             CHANGELOG.md > /tmp/release-notes.md
           if [ ! -s /tmp/release-notes.md ]; then
             echo "No CHANGELOG section for ${VERSION}; refusing to publish an empty release." >&2

CHANGELOG.md

@@ -6,6 +6,35 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+## [0.128.36] — 2026-10-17
+
+### Added ✨
+
+- Discover CEX instruments and DEX pools through both instrument routes, with typed venue,
+  segment and live-segment metadata. `InstrumentCatalogueResponse` names their response union.
+- Import centralized-exchange ticker or kline history with the `cex` source block and fifteen
+  cadence labels; retain typed `ResponseError.reason` and `next_` recovery instructions.
+- Public `api.exchange.download_tickers` and `download_klines` adapters return successful
+  binary bytes for sync and async calls, including strict-status clients. Detailed calls preserve
+  provisional-hour metadata, timestamp units, and retry delays.
+
+### Changed 🔄
+
+- Token scopes are a `KeyScopes` resource-to-actions map, also required on account responses.
+  The API reports grants but does not yet enforce them.
+- Live sources use `cex` / `dex` enums. DEX runs require one pool address and `kline` data;
+  discover valid exchange / live-segment combinations before starting a run.
+- Exchange metadata and instrument `symbol` are required. Unknown market values are absent
+  (`UNSET`), never null. Instrument segment requests accept discovered protocol strings.
+- Hourly downloads cover seven days. Lastra `ts` is microseconds with MIME type
+  `application/vnd.qtsurfer.lastra`; Parquet remains milliseconds. Handle provisional `200`,
+  `404 NOT_YET_AVAILABLE` / `NO_DATA`, `400 OUTSIDE_WINDOW` and retryable `503`.
+
+### Fixed 🐛
+
+- Public download adapters encode quote currencies without colliding with the URL encoder.
+  Use the public `api.exchange` imports rather than private generated download modules.
+
 ## [0.128.25] — 2026-10-09
 
 ### Added ✨

README.md

@@ -55,7 +55,7 @@ for ex in exchanges or []:
     print(ex.id, ex.name)
 
 instruments = list_instruments.sync(client=client, exchange_id="binance")
-print(f"{len(instruments.data)} instruments on binance ({instruments.meta.segment.value})")
+print(f"{len(instruments.data)} instruments on binance ({instruments.meta.segment})")
 ```
 
 ### API key → JWT
@@ -82,6 +82,18 @@ token_response = authenticate.sync(client=apikey_client)
 jwt = token_response.access_token  # use this in subsequent calls
 ```
 
+`token_response.scopes`, when present, is a `KeyScopes` map from resource to actions,
+such as `{"backtests": ["read", "write", "execute"]}`, not an array. `get_account`
+returns the same required map. The API reports grants but does not yet enforce them.
+
+Exchange discovery returns required `venue_type`, `segments`, and `live_segments`.
+Both instrument routes return `InstrumentCatalogueResponse`: either `InstrumentListResponse`
+(`meta.venue_type == "cex"`) or `PoolListResponse` (`"dex"`). Pool ids are live-run
+addresses; symbols name signals. Missing market values are `UNSET`, never null.
+Live sources use `LiveSourceRequestVenueType.CEX` / `.DEX`; DEX needs exactly one
+pool address, `kline`, and a discovered live protocol. CEX imports use
+`DatasetImportRequestType.CEX` with a `DatasetImportCexRequest` block.
+
 For production use, prefer the [`qtsurfer-sdk`](https://github.com/QTSurfer/sdk-python)
 `auth(apikey)` helper — it handles token refresh, env-var pickup
 (`QTSURFER_APIKEY`), and pluggable token storage so callers don't reinvent any
@@ -240,7 +252,7 @@ responses also expose optional `reason` text when a run stops or fails.
 
 > Exact module/function names are produced from `operationId` in the OpenAPI spec. Run `scripts/regenerate.sh` to refresh and check `src/qtsurfer/api/client/_generated/api/` for the authoritative listing.
 
-All generated model types (`Exchange`, `InstrumentDetail`, `InstrumentCoverage`, `CoverageWindow`, `JobState`, `PrepareJobState`, `StrategyState`, `StrategyLinks`, `BacktestJobResult`, `ResultMap`, `ResponseError`, `Dataset`, `DatasetWithLinks`, `DatasetCreated`, `DatasetVersion`, `DatasetUploadState`, …) live under `qtsurfer.api.client.models`. `list_instruments`/`list_segment_instruments` return an `InstrumentListResponse` (HAL envelope: `data` + `meta` + `_links`), not a bare list — each `InstrumentDetail.coverage` carries per-data-type `CoverageWindow`s instead of flat `dataFrom`/`dataTo`. A single-instrument `get_prepare_status` returns a `PrepareJobState` — always terminal (`status: Completed`), with a `coverage_ratio` and a per-hour `hours_without_data` breakdown to act on instead of polling. Against a dataset-backed prepare (`exchangeId: user`), `PrepareJobState` reports coverage on the dataset's own cadence grid instead — `cadence`/`gaps`/`largest_gap_steps` — with `total_hours`/`hours_with_data`/`hours_without_data` absent in that case. `get_strategy` returns a `StrategyState`, whose `validation` field (`not_validated` / `pending` / `passed` / `failed`) reports the outcome of the most recent `validate_strategy` check rather than a compile job status.
+All generated model types (`Exchange`, `InstrumentDetail`, `InstrumentCoverage`, `CoverageWindow`, `JobState`, `PrepareJobState`, `StrategyState`, `StrategyLinks`, `BacktestJobResult`, `ResultMap`, `ResponseError`, `Dataset`, `DatasetWithLinks`, `DatasetCreated`, `DatasetVersion`, `DatasetUploadState`, …) live under `qtsurfer.api.client.models`. `list_instruments`/`list_segment_instruments` return `InstrumentCatalogueResponse`, either `InstrumentListResponse` for CEX or `PoolListResponse` for DEX (HAL envelope: `data` + `meta` + `_links`) — each `InstrumentDetail.coverage` carries per-data-type `CoverageWindow`s instead of flat `dataFrom`/`dataTo`. A single-instrument `get_prepare_status` returns a `PrepareJobState` — always terminal (`status: Completed`), with a `coverage_ratio` and a per-hour `hours_without_data` breakdown to act on instead of polling. Against a dataset-backed prepare (`exchangeId: user`), `PrepareJobState` reports coverage on the dataset's own cadence grid instead — `cadence`/`gaps`/`largest_gap_steps` — with `total_hours`/`hours_with_data`/`hours_without_data` absent in that case. `get_strategy` returns a `StrategyState`, whose `validation` field (`not_validated` / `pending` / `passed` / `failed`) reports the outcome of the most recent `validate_strategy` check rather than a compile job status.
 
 A full `StrategyState` (from `get_strategy`, and from `validate_strategy`'s already-validated `200`) carries an optional `field_links` (`_links` on the wire) — a `StrategyLinks` with a `code: HalLink` pointing at `get_strategy_code`. `validate_strategy`'s `202` omits it, since a check that just started has nothing to link to yet. `list_strategies` returns every strategy you've registered and not deleted, most recently compiled first, but deliberately without each one's `validation` state — check that per strategy with `get_strategy`. `delete_strategy` removes a strategy from both `get_strategy` and `list_strategies`; it doesn't touch backtests already run against it, and re-submitting the same source afterwards registers a new strategy under a new id. `get_strategy_code`'s `404` covers two indistinguishable cases: an id never registered by you, or one that resolves only through a shared/marketplace reference with no source of its own.
 
@@ -259,10 +271,14 @@ server-applied size guard, so treat it as authoritative over requested defaults.
 
 ### Binary downloads (`/exchange/{ex}/tickers|klines/{base}/{quote}`)
 
-These endpoints return raw [Lastra](https://github.com/QTSurfer/lastra-java) bytes (default) or Parquet (`format=parquet`). The generated `sync()` helpers parse the response body as JSON and will raise on binary payloads; use `sync_detailed()` and read `response.content` directly:
+These endpoints return raw [Lastra](https://github.com/QTSurfer/lastra-java) bytes (default) or
+Parquet (`format=parquet`). Import the public `api.exchange` adapters: generated binary
+success responses are omitted by the generator. Public `sync()` / `asyncio()` return bytes
+or `ResponseError`; detailed variants retain headers, status, parsed value, and content.
 
 ```python
 from qtsurfer.api.client import AuthenticatedClient
+from datetime import UTC, datetime, timedelta
 from qtsurfer.api.client.api.exchange import download_tickers
 
 client = AuthenticatedClient(base_url="https://api.qtsurfer.net/v1", token=token)
@@ -272,19 +288,31 @@ response = download_tickers.sync_detailed(
     exchange_id="binance",
     base="BTC",
     quote="USDT",
-    hour="2026-01-15T10",
+    hour=(datetime.now(UTC) - timedelta(hours=2)).strftime("%Y-%m-%dT%H"),
 )
-with open("BTC_USDT_2026-01-15_h10.lastra", "wb") as f:
-    f.write(response.content)
+if response.status_code == 200:
+    with open("BTC_USDT.lastra", "wb") as f:
+        f.write(response.content)
+else:
+    print(response.parsed.reason, response.headers.get("Retry-After"))
 ```
 
+Hourly downloads cover seven days. Lastra MIME is `application/vnd.qtsurfer.lastra`,
+and `ts` uses microseconds (`X-Ts-Unit: micros` and column metadata `ts_unit: micros`);
+Parquet timestamps remain milliseconds. Recent hours may be provisional (`X-Data-Status`
+and ISO-8601 `X-Data-Through`) or unavailable (`404 NOT_YET_AVAILABLE` with `Retry-After`).
+Do not archive provisional data as a complete hour. `404 NO_DATA` is a stored empty hour;
+`503 TEMPORARILY_UNAVAILABLE` is retryable after `Retry-After`. `400 OUTSIDE_WINDOW`
+includes `ResponseError.next_`, a CEX import request for the hour; add a dataset name.
+Unknown reasons should be handled like absent reasons, using status and message.
+
 For very large segments, drop down to the underlying `httpx.Client` (`client.get_httpx_client()`) and stream:
 
 ```python
 with client.get_httpx_client().stream(
     "GET",
     "/exchange/binance/klines/BTC/USDT",
-    params={"hour": "2026-01-15T10", "format": "parquet"},
+    params={"hour": "2026-10-15T10", "format": "parquet"},
 ) as r:
     r.raise_for_status()
     with open("out.parquet", "wb") as f:

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "qtsurfer-api-client"
-version = "0.128.25"
+version = "0.128.36"
 description = "Auto-generated Python client for the QTSurfer API."
 readme = "README.md"
 license = "Apache-2.0"

src/qtsurfer/api/client/_generated/api/dataset/import_dataset.py

@@ -83,8 +83,18 @@ def sync_detailed(
     as a dataset version indistinguishable from an uploaded one once it's ready. Poll
     `GET /datasets/{datasetId}/imports/{importId}` for progress.
 
-    `type` selects the source. `dex` — history over a pool/pair's own on-chain market — is the
-    only value today; other source types join this same endpoint later.
+    `type` selects the kind of venue the history comes from, with the same two values as the
+    `venueType` of `GET /exchanges`: `dex` — history over a pool/pair's own on-chain market —
+    and `cex` — the history of a centralized exchange's market, kept by the platform. Each
+    has its own block (`dex`, `cex`). The exchange id of a `cex` import has to be a centralized
+    exchange of `GET /exchanges`; a decentralized one (`robinhood`, say) is `400`, and is imported
+    with `type: dex`.
+
+    A `cex` import covers one instrument over a range: the API reads that range of the
+    exchange's history, builds the bars of the requested `cadence` from the finest data it has
+    for it, and stores the result as a dataset version, exactly as for an upload. `cex.dataType`
+    picks tickers or klines and is also the resulting dataset's `type`. The range is capped by
+    your tier like any import (`maxImportRangeHours` of `GET /account`).
 
     A `dex` import has two data shapes, chosen by the top-level `cadence`:
 
@@ -100,7 +110,8 @@ def sync_detailed(
     Args:
         body (DatasetImportRequest): `POST /datasets/imports`'s request body. A common block plus
             one type-specific block,
-            selected by `type` — `dex` is the only value today.
+            selected by `type`: `dex` (block `dex`) or `cex` (block `cex`), the two `venueType`s of
+            `GET /exchanges`.
 
     Raises:
         errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
@@ -134,8 +145,18 @@ def sync(
     as a dataset version indistinguishable from an uploaded one once it's ready. Poll
     `GET /datasets/{datasetId}/imports/{importId}` for progress.
 
-    `type` selects the source. `dex` — history over a pool/pair's own on-chain market — is the
-    only value today; other source types join this same endpoint later.
+    `type` selects the kind of venue the history comes from, with the same two values as the
+    `venueType` of `GET /exchanges`: `dex` — history over a pool/pair's own on-chain market —
+    and `cex` — the history of a centralized exchange's market, kept by the platform. Each
+    has its own block (`dex`, `cex`). The exchange id of a `cex` import has to be a centralized
+    exchange of `GET /exchanges`; a decentralized one (`robinhood`, say) is `400`, and is imported
+    with `type: dex`.
+
+    A `cex` import covers one instrument over a range: the API reads that range of the
+    exchange's history, builds the bars of the requested `cadence` from the finest data it has
+    for it, and stores the result as a dataset version, exactly as for an upload. `cex.dataType`
+    picks tickers or klines and is also the resulting dataset's `type`. The range is capped by
+    your tier like any import (`maxImportRangeHours` of `GET /account`).
 
     A `dex` import has two data shapes, chosen by the top-level `cadence`:
 
@@ -151,7 +172,8 @@ def sync(
     Args:
         body (DatasetImportRequest): `POST /datasets/imports`'s request body. A common block plus
             one type-specific block,
-            selected by `type` — `dex` is the only value today.
+            selected by `type`: `dex` (block `dex`) or `cex` (block `cex`), the two `venueType`s of
+            `GET /exchanges`.
 
     Raises:
         errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
@@ -180,8 +202,18 @@ async def asyncio_detailed(
     as a dataset version indistinguishable from an uploaded one once it's ready. Poll
     `GET /datasets/{datasetId}/imports/{importId}` for progress.
 
-    `type` selects the source. `dex` — history over a pool/pair's own on-chain market — is the
-    only value today; other source types join this same endpoint later.
+    `type` selects the kind of venue the history comes from, with the same two values as the
+    `venueType` of `GET /exchanges`: `dex` — history over a pool/pair's own on-chain market —
+    and `cex` — the history of a centralized exchange's market, kept by the platform. Each
+    has its own block (`dex`, `cex`). The exchange id of a `cex` import has to be a centralized
+    exchange of `GET /exchanges`; a decentralized one (`robinhood`, say) is `400`, and is imported
+    with `type: dex`.
+
+    A `cex` import covers one instrument over a range: the API reads that range of the
+    exchange's history, builds the bars of the requested `cadence` from the finest data it has
+    for it, and stores the result as a dataset version, exactly as for an upload. `cex.dataType`
+    picks tickers or klines and is also the resulting dataset's `type`. The range is capped by
+    your tier like any import (`maxImportRangeHours` of `GET /account`).
 
     A `dex` import has two data shapes, chosen by the top-level `cadence`:
 
@@ -197,7 +229,8 @@ async def asyncio_detailed(
     Args:
         body (DatasetImportRequest): `POST /datasets/imports`'s request body. A common block plus
             one type-specific block,
-            selected by `type` — `dex` is the only value today.
+            selected by `type`: `dex` (block `dex`) or `cex` (block `cex`), the two `venueType`s of
+            `GET /exchanges`.
 
     Raises:
         errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
@@ -229,8 +262,18 @@ async def asyncio(
     as a dataset version indistinguishable from an uploaded one once it's ready. Poll
     `GET /datasets/{datasetId}/imports/{importId}` for progress.
 
-    `type` selects the source. `dex` — history over a pool/pair's own on-chain market — is the
-    only value today; other source types join this same endpoint later.
+    `type` selects the kind of venue the history comes from, with the same two values as the
+    `venueType` of `GET /exchanges`: `dex` — history over a pool/pair's own on-chain market —
+    and `cex` — the history of a centralized exchange's market, kept by the platform. Each
+    has its own block (`dex`, `cex`). The exchange id of a `cex` import has to be a centralized
+    exchange of `GET /exchanges`; a decentralized one (`robinhood`, say) is `400`, and is imported
+    with `type: dex`.
+
+    A `cex` import covers one instrument over a range: the API reads that range of the
+    exchange's history, builds the bars of the requested `cadence` from the finest data it has
+    for it, and stores the result as a dataset version, exactly as for an upload. `cex.dataType`
+    picks tickers or klines and is also the resulting dataset's `type`. The range is capped by
+    your tier like any import (`maxImportRangeHours` of `GET /account`).
 
     A `dex` import has two data shapes, chosen by the top-level `cadence`:
 
@@ -246,7 +289,8 @@ async def asyncio(
     Args:
         body (DatasetImportRequest): `POST /datasets/imports`'s request body. A common block plus
             one type-specific block,
-            selected by `type` — `dex` is the only value today.
+            selected by `type`: `dex` (block `dex`) or `cex` (block `cex`), the two `venueType`s of
+            `GET /exchanges`.
 
     Raises:
         errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.