Repository navigation
chore: the Google Pub/Sub, Cloud Storage and BigQuery Kamelets take ack, ordering, move and insert ID options (#3170) - #3227
Merged
Conversation
…ck, ordering, move and insert ID options (apache#3170) New optional options, all mapped with optional placeholders, so that with them unset the endpoint URIs are the same as before: - google-pubsub-sink: messageOrderingEnabled and pubsubEndpoint, so that the messages with the same CamelGooglePubsubOrderingKey header keep their order. - google-pubsub-source: ackMode, maxDeliveryAttempts and maxAckExtensionPeriod. - google-storage-event-based-source: ackMode and maxDeliveryAttempts. - google-storage-source: moveAfterRead with destinationBucket, includeBody, includeFolders and delay, and Workload Identity Federation (useWorkloadIdentityFederation, workloadIdentityConfig, impersonatedServiceAccount). - google-storage-sink: objectName and storageLocation, and the same Workload Identity Federation options. - google-bigquery-sink: useAsInsertId, so that BigQuery can drop a row that a retried message sends again. With includeBody false, the component puts the Blob object of the Google client in the body, which the binary output data type of the source cannot convert, so every file failed and was read again forever. A precondition choice, resolved when the route is created, empties the body in that case only. storageClass is left out of google-storage-sink: the component has no type converter from text to com.google.cloud.storage.StorageClass, so any value fails the route at start. It would also only set the class of a bucket that autoCreateBucket creates, not of each file. The descriptions, the doc partials and the header counts of KameletsCatalogTest are updated, and the library copies refreshed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
davsclaus
force-pushed
the
ci-issue-3170-google-data
branch
from
November 18, 2026 20:01
6743be1 to
b7b897f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170: the Google Pub/Sub Kamelets can leave the acknowledgement to the route, stop processing a message that keeps failing, and publish in order;
google-storage-sourcecan move the files it reads to another bucket, skip their content or the folder markers, and poll less often; the Cloud Storage Kamelets can log in with Workload Identity Federation;google-storage-sinkcan write to a fixed object name and choose where a bucket it creates is located; andgoogle-bigquery-sinkcan send an insert ID, so that BigQuery can drop the rows a retried message sends again.The options
google-pubsub-sinkmessageOrderingEnabledmessageOrderingEnabled={{?messageOrderingEnabled}}CamelGooglePubsubOrderingKeyheader in order. Without it, a message with that header fails.pubsubEndpointpubsubEndpoint={{?pubsubEndpoint}}europe-west1-pubsub.googleapis.com:443. Ordering needs a regional endpoint.google-pubsub-sourceackModeackMode={{?ackMode}}AUTO(unset) orNONE: the route acknowledges each message itself through theCamelGooglePubsubAcknowledgeheader.maxDeliveryAttemptsmaxDeliveryAttempts={{?maxDeliveryAttempts}}maxAckExtensionPeriodmaxAckExtensionPeriod={{?maxAckExtensionPeriod}}google-storage-event-based-sourceackMode,maxDeliveryAttemptsgoogle-pubsub-sourcegoogle-storage-sourcemoveAfterRead,destinationBucketmoveAfterRead={{?moveAfterRead}},destinationBucket={{?destinationBucket}}destinationBucketonce processed, then deletes it (unlessdeleteAfterReadis false).includeBodyincludeBody={{?includeBody}}, plus a precondition step (below)false: the content is not downloaded; the body is empty, the headers describe the file.includeFoldersincludeFolders={{?includeFolders}}false: skips folder markers (objects whose name ends with/).delaydelay={{?delay}}google-storage-source,google-storage-sinkuseWorkloadIdentityFederation,workloadIdentityConfig,impersonatedServiceAccount{{?...}}eachexternal_accountconfiguration file, optionally impersonating a service account.google-storage-sinkobjectNameobjectName=RAW({{?objectName}})fileandce-fileheaders are then not used.storageLocationstorageLocation={{?storageLocation}}autoCreateBucketcreates;US-EAST1when unset.google-bigquery-sinkuseAsInsertIduseAsInsertId: "{{?useAsInsertId}}"default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are identical tomain's.CamelGooglePubsubOrderingKey(sink, input),CamelGooglePubsubDeliveryAttemptandCamelGooglePubsubAcknowledge(sources, output) are documented indataTypes.KameletsCatalogTestcounts:google-pubsub-sink1 → 2,google-pubsub-source3 → 5.includeBody: falseneeds a template step. The component then puts thecom.google.cloud.storage.Blobobject of the Google client in the body. Thebinaryoutput data type ofgoogle-storage-sourcecannot convert it to bytes, so every file failed and, not being deleted, was read again on every poll. The template now has achoicewithprecondition: trueon${properties:includeBody:true} == 'false', which empties the body. Being a precondition, it is resolved once when the route is created, and drops out entirely whenincludeBodyis unset or true.objectNameinRAW(): a name such asstatus/latest 100%+x.jsonis stored as written.pubsubEndpoint(host:port),destinationBucket(bucket name),storageLocation,objectName(no line breaks),workloadIdentityConfig(file:,classpath:,http:,https:),impersonatedServiceAccount(an email),useAsInsertId(a column name).Left out
storageClassongoogle-storage-sink. The component option is typedcom.google.cloud.storage.StorageClass, and Camel has no type converter from text to it:storageClass=NEARLINEfails the route at start withNo type converter available to convert from type: java.lang.String to the required type: com.google.cloud.storage.StorageClass. It needs a Camel change first, such as a type converter incamel-google-storage. Also, the component only uses it for a bucket thatautoCreateBucketcreates: the upload request sets no storage class, so each file gets the default class of its bucket. A storage class per file would need a producer change too.camel.component.google-pubsub.endpointandcamel.component.google-pubsub.authenticate=false. The Java Cloud Storage and BigQuery clients do not readSTORAGE_EMULATOR_HOSTor a similar variable, and the components have no host option; both autowire a client bean (Storage,GoogleBigQueryConnectionFactory), which is how they were pointed at the emulators below.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT:
gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators(gcloud beta emulators pubsub start), topics and subscriptions made with its REST API, some with a dead-letter policy (maxDeliveryAttempts: 5).fsouza/fake-gcs-server(memory backend), through an autowiredStoragebean.ghcr.io/goccy/bigquery-emulator, through an autowiredGoogleBigQueryConnectionFactorybean.external_accountfile with a made-up pool and subject token, against the real Google STS (no Google account involved).mainand this branchcamel get endpoint --jsonshows the same six endpoint URIsgoogle-pubsub-source, one message, unset /ackMode: AUTO/synchronousPullackMode: NONE, streaming pullmaxAckExtensionPeriod3600 s)ackMode: NONE,maxAckExtensionPeriod: 5ackMode: NONE,synchronousPull: true, subscription deadline 10 sackMode: NONE, route calls${header.CamelGooglePubsubAcknowledge.ack(${exchange})}(Simple) orrequest.headers.CamelGooglePubsubAcknowledge.ack(exchange)(Groovy)maxDeliveryAttemptsunsetAuto-fetched maxDeliveryAttempts=5; attempts 1–4 processed (CamelGooglePubsubDeliveryAttempt1–4), attempt 5 nacked unprocessed; message on the dead-letter topicmaxDeliveryAttempts: 2maxDeliveryAttempts: 0CamelGooglePubsubDeliveryAttemptheadermaxDeliveryAttemptsunset, policy cannot be read (subscription missing)Failed to auto-fetch maxDeliveryAttempts ... enforcement will be disabledgoogle-storage-event-based-source,ackModeunset /NONE+synchronousPullmaxDeliveryAttempts: 2google-pubsub-sink,CamelGooglePubsubOrderingKeyset, options unsetCannot publish a message with an ordering key when message ordering is not enabledmessageOrderingEnabled: trueIn conjunction with enabling message ordering the pubsubEndpoint should be setmessageOrderingEnabled: true,pubsubEndpoint: localhost:<emulator>pubsubEndpoint: localhost:<emulator>without the component's emulatorendpointgoogle-storage-source,moveAfterRead: true,destinationBucket: archivea.txtgone frominbox, present inarchivedeleteAfterRead: falsearchive, kept in the source bucketmoveAfterRead, failing routemoveAfterRead, destination bucket missingError occurred during moving or deleting object(404); file kept, read again every pollincludeBody: false, before the template stepNo type converter available to convert from type: com.google.cloud.storage.Blob to the required type: byte[]; read again every 500 msincludeBody: false, with itCamelGoogleCloudStorageContentLength: 7; file deletedincludeBody: false+moveAfterReadincludeBody: trueincludeFoldersunset /false, bucket withdocs/anddocs/x.txtdocs/x.txtdeleteAfterRead: false,delayunset /5000google-storage-sink,objectName: "status/latest 100%+x.json",fileheader setstatus/latest 100%+x.json, as writtenautoCreateBucket: true,storageLocation: europe-west1/ unset"location":"europe-west1"/"location":"US-EAST1"(both"storageClass":"STANDARD"; the upload request sets no class)storageClass: NEARLINE(option tried, then left out)StorageClassuseWorkloadIdentityFederation: true,workloadIdentityConfig: file:cred.jsoninvalid_targetfor the made-up pool: the file is loaded and the token exchange attempted. A localtoken_urlis refused by the Google client (The provided token URL is invalid)impersonatedServiceAccountUnable to refresh sourceCredentialsfrom the impersonating credentials, caused by the same STS answergoogle-bigquery-sink,useAsInsertId: orderId, same row twice{"rows":[{"insertId":"o-1","json":{...}}]}both timesuseAsInsertIdunset{"rows":[{"json":{...}}]}, noinsertIdinsertId(o-3,o-4)insertIduseAsInsertId: num(a number field)ClassCastException: Integer cannot be cast to String; nothing sentThe BigQuery emulator stores every row, also a repeated insert ID: the de-duplication is done by BigQuery on a best-effort basis, so only the request was checked, not the de-duplication itself.
./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root, the latter after the header counts ofKameletsCatalogTestwere updated. Before building, the library copies of the six Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code