Skip to content

Add AsyncTinybirdClient/AsyncTinybirdApi async client variant - #35

Open
sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2260-async-client-variant
Open

sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2260-async-client-variant

Conversation

@sandshoes

Copy link
Copy Markdown
Contributor

Summary

Closes PROTM-2260. The entire HTTP layer was synchronous (urllib-based), blocking the event loop when called from async frameworks (FastAPI, aiohttp). This adds a full async counterpart:

  • AsyncTinybirdApi (api/async_api.py) — same method surface as TinybirdApi (query, ingest, ingest_batch, sql, append_datasource, delete_datasource, truncate_datasource, create_token, request/request_json), as async def.
  • AsyncTinybirdClient + AsyncTokensNamespace (client/async_base.py, client/async_tokens.py) — same datasources.*/tokens.create_jwt/query/ingest/ingest_batch/sql surface as TinybirdClient.
  • AsyncTinybird (schema/project.py) — async counterpart to the Tinybird facade.

create_async_client/create_async_tinybird_api mirror the existing create_client/create_tinybird_api factory functions. All four are exported from the package root alongside their sync counterparts.

Design decisions

1. Transport — kept the existing _http.py's sync urllib-based tinybird_fetch untouched (zero behavior-change risk to the proven sync path) and added tinybird_fetch_async alongside it, backed by httpx.AsyncClient. Both share all the pure request-building/serialization helpers already in _http.py (with_tinybird_from_param, create_multipart_body, serialize_event_value, detect_data_format, to_query_value) and both return the same HTTPResponse shape, so error/response handling code is identical either way. AsyncTinybirdApi owns a single lazily-created httpx.AsyncClient for connection pooling across calls (aclose() / async context manager to clean it up), rather than opening a new connection per request.

Retry/error logic (429/503 backoff, header parsing, error-body parsing) was extracted out of TinybirdApi into pure functions in a new api/_shared.py, and TinybirdApi was refactored to call them instead of its private methods — a behavior-preserving refactor (covered by the existing, unmodified test_api_parity.py/test_api_core.py, all still green) that lets AsyncTinybirdApi reuse the exact same semantics instead of the two drifting apart over time.

2. API shape — a separate explicit AsyncTinybirdApi/AsyncTinybirdClient pair (every method as async def, identical names/signatures to the sync versions), rather than a hybrid sync-or-async interface. This is the standard pattern for Python SDKs (e.g. openai's/anthropic's Client/AsyncClient) and keeps the mental model a straight await away from the sync code.

3. Facade interaction — added AsyncTinybird directly alongside Tinybird in schema/project.py (it's hand-written, not templated, so this was cheap). The code generator (generator/client.py) still emits tinybird = Tinybird({...}) by default; a project can opt into async today by constructing AsyncTinybird({...}) directly with the same datasources/pipes dicts the generated .datasources/.pipes modules already export. Wiring a --async codegen flag to emit AsyncTinybird(...) instead is a natural fast-follow, scoped out here to keep this PR to the client layer itself.

Branch resolution (dev_mode=True) — get_or_create_branch (branch creation/polling, up to ~2 minutes) stays as the existing sync implementation, called via asyncio.to_thread so it doesn't block the event loop. Reimplementing the whole branch-management module (api/branches.py) as async would be a large scope expansion for what is an infrequent, one-time setup operation, not a per-request hot path — the per-request methods (query/ingest/sql/etc.) are the ones that get genuine async I/O via httpx.AsyncClient.

Scoped out

  • Codegen --async flag to generate AsyncTinybird(...) directly (noted above).
  • Async versions of the other 9 not-yet-merged PROTM-2255 PRs' methods (jobs/secrets/token-lifecycle namespaces, GCS connector/sink variants, schema inference) — those don't exist on main yet; this PR only covers the method surface currently on main. Each should get async coverage as a small fast-follow once merged, following the exact pattern established here.
  • api/branches.py/api/workspaces.py/api/build.py/api/deploy.py and the CLI modules stay sync-only — they're inherently one-shot CLI/admin operations, not called from request-handling code, so there's no async use case for them.

Changed

  • pyproject.toml — added httpx>=0.27,<1 as a runtime dependency.
  • src/tinybird_sdk/_http.py — added tinybird_fetch_async.
  • src/tinybird_sdk/api/_shared.py (new) — shared retry/error pure helpers.
  • src/tinybird_sdk/api/api.py — refactored to use the shared helpers (no behavior change).
  • src/tinybird_sdk/api/async_api.py (new) — AsyncTinybirdApi.
  • src/tinybird_sdk/api/tokens.py — added create_jwt_async.
  • src/tinybird_sdk/client/async_base.py (new) — AsyncTinybirdClient.
  • src/tinybird_sdk/client/async_tokens.py (new) — AsyncTokensNamespace.
  • src/tinybird_sdk/schema/project.py — added AsyncTinybird.
  • __init__.py files — exported the new symbols.
  • tests/fixtures/parity_contract/root_exports.json — regenerated to include the 6 new root exports (AsyncTinybird, AsyncTinybirdApi, AsyncTinybirdClient, create_async_client, create_async_tinybird_api, create_jwt_async).
  • tests/test_async_api.py, tests/test_async_client.py, tests/test_async_facade.py (new) — 21 tests covering the async transport (via httpx.MockTransport, including a real 429-retry-then-succeed flow), the async client's datasource/token namespaces and error mapping, the dev_mode/branch-context asyncio.to_thread offload, and the AsyncTinybird facade.
  • README.md, CHANGELOG.md — documented the async client with a FastAPI usage example.

Verification

ruff check, ruff format --check, mypy, and the full pytest suite are all green (184 tests, including the 21 new ones; zero regressions in the existing suite). make check's secrets/gitleaks target fails in this sandbox on an unrelated SSL/network error fetching its pre-commit environment — the same known environment limitation noted in the other PROTM-2255 PRs in this batch, not something introduced here.

Checklist

  • CI is green (lint, typecheck, test, secrets*)
  • pre-commit run --all-files passes locally (*except gitleaks, sandbox-only SSL issue)
  • Tests were added or updated when behavior changed
  • Public API / typing changes were reviewed
  • Documentation was updated (README.md)
  • Breaking changes are clearly documented — none; this is purely additive
  • CHANGELOG.md was updated when user-facing behavior changed

Adds an async counterpart to the whole sync client layer so the SDK
no longer blocks the event loop when used from async frameworks
(FastAPI, aiohttp):

- AsyncTinybirdApi (api/async_api.py): same method surface as
  TinybirdApi, backed by httpx.AsyncClient instead of blocking
  urllib. Retry/error logic is extracted into api/_shared.py so both
  clients share identical semantics instead of drifting; TinybirdApi
  itself is refactored to call the shared helpers with no behavior
  change (covered by its existing tests).
- AsyncTinybirdClient (client/async_base.py) and AsyncTokensNamespace
  (client/async_tokens.py): async counterparts to TinybirdClient/
  TokensNamespace with the same datasources/tokens/query/ingest/sql
  surface. Branch-token resolution (dev_mode=True) still calls the
  existing sync branch-management API, offloaded via
  asyncio.to_thread, since provisioning/polling a branch is a slow,
  infrequent one-time setup operation rather than a per-request path -
  reimplementing that whole module as async would be a large scope
  expansion for no real benefit.
- AsyncTinybird (schema/project.py): async counterpart to the Tinybird
  facade. The code generator still emits the sync Tinybird(...) by
  default; wiring an --async codegen flag is a natural fast-follow,
  scoped out here to keep this change to the client layer.

Adds httpx as a runtime dependency.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant