Skip to content

Add GCS sink destination support to define_sink_pipe - #30

Open
sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2262-gcs-sink-destination
Open

sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2262-gcs-sink-destination

Conversation

@sandshoes

Copy link
Copy Markdown
Contributor

Summary

  • define_sink_pipe currently only supports Kafka and S3 destinations. Adds GCS as a third sink destination, matching the Forward CLI's gcs_hmac export service, for https://linear.app/tinybird/issue/PROTM-2262/python-sdk-add-gcs-sink-destination-support
  • New GCSSinkConfig mirrors S3SinkConfig's field set (bucket_uri, file_template, format, schedule, strategy, compression) but references a GCSConnectionDefinition instead of S3ConnectionDefinition.
  • Since GCS and S3 sinks share an identical EXPORT_* directive shape in .pipe datafiles, the generator emits an explicit EXPORT_SERVICE gcs_hmac directive for GCS sinks so the migration parser can disambiguate them from S3 sinks on reverse-parse (S3 remains the default when EXPORT_SERVICE is omitted, preserving existing behavior/datafiles).
  • Updated the migrate runner's sink/connection compatibility check, which previously compared pipe.sink.service directly against the connection's _connectionType — this would have incorrectly rejected a valid gcs_hmac sink pointed at a gcs connection. It now maps gcs_hmac → gcs before comparing.
  • Also wired SinkGCSModel through migrate/emit_ts.py for TypeScript re-emission parity with S3.

Changes

  • src/tinybird_sdk/schema/pipe.py: GCSSinkConfig dataclass, SinkConfig union, _normalize_sink_config GCS branch.
  • src/tinybird_sdk/generator/pipe.py: emit EXPORT_SERVICE gcs_hmac + shared export directives for GCSSinkConfig.
  • src/tinybird_sdk/migrate/types.py: SinkGCSModel.
  • src/tinybird_sdk/migrate/parse_pipe.py: accept gcs_hmac as an EXPORT_SERVICE value; build SinkGCSModel when explicitly set, default to s3 otherwise (unchanged behavior for existing datafiles).
  • src/tinybird_sdk/migrate/runner.py: fix sink/connection type compatibility check for gcs_hmac.
  • src/tinybird_sdk/migrate/emit_ts.py, schema/__init__.py, migrate/__init__.py: wiring/exports.
  • README.md, CHANGELOG.md: documented the new sink option.
  • tests/test_gcs_sink_destination.py (new, 8 tests): schema validation, datafile generation, parse round-trip (explicit service + default-to-s3 fallback + unsupported service rejection), runner compatibility (accept/reject), and TS emission.

Checklist

  • CI is green (lint, typecheck, test)
  • pre-commit run --all-files passes locally (gitleaks step fails in this sandbox on an unrelated SSL/network error fetching its pre-commit environment, not a code issue)
  • Tests were added or updated when behavior changed
  • Public API / typing changes were reviewed
  • Documentation was updated (README.md)
  • Breaking changes are clearly documented (none — purely additive; default inference for ambiguous existing sinks is unchanged)
  • CHANGELOG.md was updated when user-facing behavior changed

Matches the Forward CLI's gcs_hmac export service: GCSSinkConfig
mirrors S3SinkConfig's fields (bucket_uri/file_template/format/
schedule/strategy/compression) but references a GCSConnectionDefinition.
Since GCS and S3 sinks share identical EXPORT_* directives, the
generator emits an explicit EXPORT_SERVICE gcs_hmac line to
disambiguate on reverse parse, and the migrate runner's sink/connection
compatibility check now maps gcs_hmac sinks to "gcs" connections.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sandshoes
sandshoes requested a review from jalbertsr October 2, 2026 09:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant