Conversation
Matches the Forward CLI's gcs_hmac export service: GCSSinkConfig mirrors S3SinkConfig's fields (bucket_uri/file_template/format/ schedule/strategy/compression) but references a GCSConnectionDefinition. Since GCS and S3 sinks share identical EXPORT_* directives, the generator emits an explicit EXPORT_SERVICE gcs_hmac line to disambiguate on reverse parse, and the migrate runner's sink/connection compatibility check now maps gcs_hmac sinks to "gcs" connections. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
define_sink_pipecurrently only supports Kafka and S3 destinations. Adds GCS as a third sink destination, matching the Forward CLI'sgcs_hmacexport service, for https://linear.app/tinybird/issue/PROTM-2262/python-sdk-add-gcs-sink-destination-supportGCSSinkConfigmirrorsS3SinkConfig's field set (bucket_uri,file_template,format,schedule,strategy,compression) but references aGCSConnectionDefinitioninstead ofS3ConnectionDefinition.EXPORT_*directive shape in.pipedatafiles, the generator emits an explicitEXPORT_SERVICE gcs_hmacdirective for GCS sinks so the migration parser can disambiguate them from S3 sinks on reverse-parse (S3 remains the default whenEXPORT_SERVICEis omitted, preserving existing behavior/datafiles).pipe.sink.servicedirectly against the connection's_connectionType— this would have incorrectly rejected a validgcs_hmacsink pointed at agcsconnection. It now mapsgcs_hmac→gcsbefore comparing.SinkGCSModelthroughmigrate/emit_ts.pyfor TypeScript re-emission parity with S3.Changes
src/tinybird_sdk/schema/pipe.py:GCSSinkConfigdataclass,SinkConfigunion,_normalize_sink_configGCS branch.src/tinybird_sdk/generator/pipe.py: emitEXPORT_SERVICE gcs_hmac+ shared export directives forGCSSinkConfig.src/tinybird_sdk/migrate/types.py:SinkGCSModel.src/tinybird_sdk/migrate/parse_pipe.py: acceptgcs_hmacas anEXPORT_SERVICEvalue; buildSinkGCSModelwhen explicitly set, default tos3otherwise (unchanged behavior for existing datafiles).src/tinybird_sdk/migrate/runner.py: fix sink/connection type compatibility check forgcs_hmac.src/tinybird_sdk/migrate/emit_ts.py,schema/__init__.py,migrate/__init__.py: wiring/exports.README.md,CHANGELOG.md: documented the new sink option.tests/test_gcs_sink_destination.py(new, 8 tests): schema validation, datafile generation, parse round-trip (explicit service + default-to-s3 fallback + unsupported service rejection), runner compatibility (accept/reject), and TS emission.Checklist
lint,typecheck,test)pre-commit run --all-filespasses locally (gitleaks step fails in this sandbox on an unrelated SSL/network error fetching its pre-commit environment, not a code issue)README.md)CHANGELOG.mdwas updated when user-facing behavior changed