Conversation
Adds an async counterpart to the whole sync client layer so the SDK no longer blocks the event loop when used from async frameworks (FastAPI, aiohttp): - AsyncTinybirdApi (api/async_api.py): same method surface as TinybirdApi, backed by httpx.AsyncClient instead of blocking urllib. Retry/error logic is extracted into api/_shared.py so both clients share identical semantics instead of drifting; TinybirdApi itself is refactored to call the shared helpers with no behavior change (covered by its existing tests). - AsyncTinybirdClient (client/async_base.py) and AsyncTokensNamespace (client/async_tokens.py): async counterparts to TinybirdClient/ TokensNamespace with the same datasources/tokens/query/ingest/sql surface. Branch-token resolution (dev_mode=True) still calls the existing sync branch-management API, offloaded via asyncio.to_thread, since provisioning/polling a branch is a slow, infrequent one-time setup operation rather than a per-request path - reimplementing that whole module as async would be a large scope expansion for no real benefit. - AsyncTinybird (schema/project.py): async counterpart to the Tinybird facade. The code generator still emits the sync Tinybird(...) by default; wiring an --async codegen flag is a natural fast-follow, scoped out here to keep this change to the client layer. Adds httpx as a runtime dependency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes PROTM-2260. The entire HTTP layer was synchronous (
urllib-based), blocking the event loop when called from async frameworks (FastAPI, aiohttp). This adds a full async counterpart:AsyncTinybirdApi(api/async_api.py) — same method surface asTinybirdApi(query,ingest,ingest_batch,sql,append_datasource,delete_datasource,truncate_datasource,create_token,request/request_json), asasync def.AsyncTinybirdClient+AsyncTokensNamespace(client/async_base.py,client/async_tokens.py) — samedatasources.*/tokens.create_jwt/query/ingest/ingest_batch/sqlsurface asTinybirdClient.AsyncTinybird(schema/project.py) — async counterpart to theTinybirdfacade.create_async_client/create_async_tinybird_apimirror the existingcreate_client/create_tinybird_apifactory functions. All four are exported from the package root alongside their sync counterparts.Design decisions
1. Transport — kept the existing
_http.py's syncurllib-basedtinybird_fetchuntouched (zero behavior-change risk to the proven sync path) and addedtinybird_fetch_asyncalongside it, backed byhttpx.AsyncClient. Both share all the pure request-building/serialization helpers already in_http.py(with_tinybird_from_param,create_multipart_body,serialize_event_value,detect_data_format,to_query_value) and both return the sameHTTPResponseshape, so error/response handling code is identical either way.AsyncTinybirdApiowns a single lazily-createdhttpx.AsyncClientfor connection pooling across calls (aclose()/ async context manager to clean it up), rather than opening a new connection per request.Retry/error logic (429/503 backoff, header parsing, error-body parsing) was extracted out of
TinybirdApiinto pure functions in a newapi/_shared.py, andTinybirdApiwas refactored to call them instead of its private methods — a behavior-preserving refactor (covered by the existing, unmodifiedtest_api_parity.py/test_api_core.py, all still green) that letsAsyncTinybirdApireuse the exact same semantics instead of the two drifting apart over time.2. API shape — a separate explicit
AsyncTinybirdApi/AsyncTinybirdClientpair (every method asasync def, identical names/signatures to the sync versions), rather than a hybrid sync-or-async interface. This is the standard pattern for Python SDKs (e.g.openai's/anthropic'sClient/AsyncClient) and keeps the mental model a straightawaitaway from the sync code.3. Facade interaction — added
AsyncTinybirddirectly alongsideTinybirdinschema/project.py(it's hand-written, not templated, so this was cheap). The code generator (generator/client.py) still emitstinybird = Tinybird({...})by default; a project can opt into async today by constructingAsyncTinybird({...})directly with the samedatasources/pipesdicts the generated.datasources/.pipesmodules already export. Wiring a--asynccodegen flag to emitAsyncTinybird(...)instead is a natural fast-follow, scoped out here to keep this PR to the client layer itself.Branch resolution (
dev_mode=True) —get_or_create_branch(branch creation/polling, up to ~2 minutes) stays as the existing sync implementation, called viaasyncio.to_threadso it doesn't block the event loop. Reimplementing the whole branch-management module (api/branches.py) as async would be a large scope expansion for what is an infrequent, one-time setup operation, not a per-request hot path — the per-request methods (query/ingest/sql/etc.) are the ones that get genuine async I/O viahttpx.AsyncClient.Scoped out
--asyncflag to generateAsyncTinybird(...)directly (noted above).mainyet; this PR only covers the method surface currently onmain. Each should get async coverage as a small fast-follow once merged, following the exact pattern established here.api/branches.py/api/workspaces.py/api/build.py/api/deploy.pyand the CLI modules stay sync-only — they're inherently one-shot CLI/admin operations, not called from request-handling code, so there's no async use case for them.Changed
pyproject.toml— addedhttpx>=0.27,<1as a runtime dependency.src/tinybird_sdk/_http.py— addedtinybird_fetch_async.src/tinybird_sdk/api/_shared.py(new) — shared retry/error pure helpers.src/tinybird_sdk/api/api.py— refactored to use the shared helpers (no behavior change).src/tinybird_sdk/api/async_api.py(new) —AsyncTinybirdApi.src/tinybird_sdk/api/tokens.py— addedcreate_jwt_async.src/tinybird_sdk/client/async_base.py(new) —AsyncTinybirdClient.src/tinybird_sdk/client/async_tokens.py(new) —AsyncTokensNamespace.src/tinybird_sdk/schema/project.py— addedAsyncTinybird.__init__.pyfiles — exported the new symbols.tests/fixtures/parity_contract/root_exports.json— regenerated to include the 6 new root exports (AsyncTinybird,AsyncTinybirdApi,AsyncTinybirdClient,create_async_client,create_async_tinybird_api,create_jwt_async).tests/test_async_api.py,tests/test_async_client.py,tests/test_async_facade.py(new) — 21 tests covering the async transport (viahttpx.MockTransport, including a real 429-retry-then-succeed flow), the async client's datasource/token namespaces and error mapping, thedev_mode/branch-contextasyncio.to_threadoffload, and theAsyncTinybirdfacade.README.md,CHANGELOG.md— documented the async client with a FastAPI usage example.Verification
ruff check,ruff format --check,mypy, and the fullpytestsuite are all green (184 tests, including the 21 new ones; zero regressions in the existing suite).make check'ssecrets/gitleaks target fails in this sandbox on an unrelated SSL/network error fetching its pre-commit environment — the same known environment limitation noted in the other PROTM-2255 PRs in this batch, not something introduced here.Checklist
lint,typecheck,test,secrets*)pre-commit run --all-filespasses locally (*except gitleaks, sandbox-only SSL issue)README.md)CHANGELOG.mdwas updated when user-facing behavior changed