Skip to content

Add schema inference from real sample data (tb datasource analyze parity) - #31

Open
sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2263-schema-inference
Open

sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2263-schema-inference

Conversation

@sandshoes

Copy link
Copy Markdown
Contributor

Summary

  • TinybirdApi.analyze() wraps the Tinybird Analyze API (POST /v0/analyze), inferring column names, ClickHouse types, and (for CSV) dialect from a local file or URL — matching tb datasource analyze <url_or_file>. Follows the same url/file (form-encoded vs. multipart) convention as the existing append_datasource().
  • tinybird_sdk.infer.parse_analyze_response() parses the raw API response into a structured AnalyzedSchema, tolerating two known response shapes: an analysis.columns list of {name, recommended_type} entries, or an analysis.schema/schema ClickHouse DDL-like string as a fallback.
  • tinybird_sdk.infer.generate_schema_code() turns an AnalyzedSchema into ready-to-paste define_datasource(...) source with t.* validators, reusing the existing codegen.clickhouse_type_to_validator type-mapping (the same one used for reverse codegen of live workspace resources), so inferred and reverse-codegen'd datasources read identically.

Closes https://linear.app/tinybird/issue/PROTM-2263/python-sdk-add-schema-inference-from-real-sample-data

Design notes

  • This PR does not add a new tinybird analyze CLI subcommand — it ships the library-level helper (API method + parsing + codegen) the ticket asks for. A thin CLI wrapper mirroring tb datasource analyze would be a natural, separable follow-up once this API shape is validated against the real Analyze API contract in a live environment.
  • The exact response shape of /v0/analyze could not be verified against live docs/API from this sandbox, so parse_analyze_response() is intentionally tolerant of both a columns list and a DDL-style schema string, and raises a clear ValueError if neither is present rather than silently guessing. Worth a quick confirmation against a real analyze response before relying on this in production.

Checklist

  • CI is green (lint, typecheck, test)
  • pre-commit run --all-files passes locally (secrets/gitleaks fails in this sandbox on an unrelated SSL cert error fetching its pre-commit env, not something introduced by this change)
  • Tests were added or updated when behavior changed
  • Public API / typing changes were reviewed
  • Documentation was updated (README.md) — new "Schema Inference Helpers" example
  • Breaking changes are clearly documented (none — purely additive)
  • CHANGELOG.md was updated when user-facing behavior changed

…ity)

TinybirdApi.analyze() wraps POST /v0/analyze, inferring column names,
ClickHouse types, and CSV dialect from a local file or URL, matching
`tb datasource analyze <url_or_file>`. tinybird_sdk.infer.parse_analyze_response()
turns the response into a structured AnalyzedSchema, and
generate_schema_code() emits ready-to-use define_datasource(...) source
with t.* validators, reusing the same type-mapping codegen already used
for reverse codegen (live workspace -> Python).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant