Skip to content

chore: the Google Pub/Sub, Cloud Storage and BigQuery Kamelets take ack, ordering, move and insert ID options (#3170) - #3227

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-google-data
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-google-data

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: the Google Pub/Sub Kamelets can leave the acknowledgement to the route, stop processing a message that keeps failing, and publish in order; google-storage-source can move the files it reads to another bucket, skip their content or the folder markers, and poll less often; the Cloud Storage Kamelets can log in with Workload Identity Federation; google-storage-sink can write to a fixed object name and choose where a bucket it creates is located; and google-bigquery-sink can send an insert ID, so that BigQuery can drop the rows a retried message sends again.

The options

Kamelet Option Template mapping What it does
google-pubsub-sink messageOrderingEnabled messageOrderingEnabled={{?messageOrderingEnabled}} Publishes the messages with the same CamelGooglePubsubOrderingKey header in order. Without it, a message with that header fails.
pubsubEndpoint pubsubEndpoint={{?pubsubEndpoint}} The Pub/Sub service to publish to, such as europe-west1-pubsub.googleapis.com:443. Ordering needs a regional endpoint.
google-pubsub-source ackMode ackMode={{?ackMode}} AUTO (unset) or NONE: the route acknowledges each message itself through the CamelGooglePubsubAcknowledge header.
maxDeliveryAttempts maxDeliveryAttempts={{?maxDeliveryAttempts}} A message delivered this many times is given back (nack) without being processed.
maxAckExtensionPeriod maxAckExtensionPeriod={{?maxAckExtensionPeriod}} How long, in seconds, an unacknowledged message is kept before Pub/Sub delivers it again (streaming pull).
google-storage-event-based-source ackMode, maxDeliveryAttempts as for google-pubsub-source The same, for the bucket notifications.
google-storage-source moveAfterRead, destinationBucket moveAfterRead={{?moveAfterRead}}, destinationBucket={{?destinationBucket}} Copies each file to destinationBucket once processed, then deletes it (unless deleteAfterRead is false).
includeBody includeBody={{?includeBody}}, plus a precondition step (below) false: the content is not downloaded; the body is empty, the headers describe the file.
includeFolders includeFolders={{?includeFolders}} false: skips folder markers (objects whose name ends with /).
delay delay={{?delay}} Milliseconds between two polls; 500 when unset.
google-storage-source, google-storage-sink useWorkloadIdentityFederation, workloadIdentityConfig, impersonatedServiceAccount {{?...}} each Log in with Workload Identity Federation: the external_account configuration file, optionally impersonating a service account.
google-storage-sink objectName objectName=RAW({{?objectName}}) Every message is written to this object; the file and ce-file headers are then not used.
storageLocation storageLocation={{?storageLocation}} Location of a bucket that autoCreateBucket creates; US-EAST1 when unset.
google-bigquery-sink useAsInsertId useAsInsertId: "{{?useAsInsertId}}" The field of each row whose value is sent as the insert ID of the row.
  • Defaults: every option is optional, with no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are identical to main's.
  • Headers: the new headers CamelGooglePubsubOrderingKey (sink, input), CamelGooglePubsubDeliveryAttempt and CamelGooglePubsubAcknowledge (sources, output) are documented in dataTypes. KameletsCatalogTest counts: google-pubsub-sink 1 → 2, google-pubsub-source 3 → 5.
  • includeBody: false needs a template step. The component then puts the com.google.cloud.storage.Blob object of the Google client in the body. The binary output data type of google-storage-source cannot convert it to bytes, so every file failed and, not being deleted, was read again on every poll. The template now has a choice with precondition: true on ${properties:includeBody:true} == 'false', which empties the body. Being a precondition, it is resolved once when the route is created, and drops out entirely when includeBody is unset or true.
  • objectName in RAW(): a name such as status/latest 100%+x.json is stored as written.
  • Patterns: pubsubEndpoint (host:port), destinationBucket (bucket name), storageLocation, objectName (no line breaks), workloadIdentityConfig (file:, classpath:, http:, https:), impersonatedServiceAccount (an email), useAsInsertId (a column name).
  • Descriptions: "Good to know" of each Kamelet names the new options. The partial pages have new sections with examples: Message Ordering; Acknowledging Messages and Dead-Letter Topic; Acknowledging Notifications; Moving Files to Another Bucket and Workload Identity Federation; A Fixed Object Name and Creating the Bucket; Insert IDs.

Left out

  • storageClass on google-storage-sink. The component option is typed com.google.cloud.storage.StorageClass, and Camel has no type converter from text to it: storageClass=NEARLINE fails the route at start with No type converter available to convert from type: java.lang.String to the required type: com.google.cloud.storage.StorageClass. It needs a Camel change first, such as a type converter in camel-google-storage. Also, the component only uses it for a bucket that autoCreateBucket creates: the upload request sets no storage class, so each file gets the default class of its bucket. A storage class per file would need a producer change too.
  • No emulator option on the Kamelets. None is needed for testing: Pub/Sub is pointed at an emulator with the component options camel.component.google-pubsub.endpoint and camel.component.google-pubsub.authenticate=false. The Java Cloud Storage and BigQuery clients do not read STORAGE_EMULATOR_HOST or a similar variable, and the components have no host option; both autowire a client bean (Storage, GoogleBigQueryConnectionFactory), which is how they were pointed at the emulators below.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT:

  • Pub/Sub: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators (gcloud beta emulators pubsub start), topics and subscriptions made with its REST API, some with a dead-letter policy (maxDeliveryAttempts: 5).
  • Cloud Storage: fsouza/fake-gcs-server (memory backend), through an autowired Storage bean.
  • BigQuery: ghcr.io/goccy/bigquery-emulator, through an autowired GoogleBigQueryConnectionFactory bean.
  • A small logging HTTP proxy in front of fake-gcs-server and the BigQuery emulator captured the request bodies.
  • Workload Identity Federation: an external_account file with a made-up pool and subject token, against the real Google STS (no Google account involved).
Case Result
All six Kamelets with the new options unset, main and this branch camel get endpoint --json shows the same six endpoint URIs
google-pubsub-source, one message, unset / ackMode: AUTO / synchronousPull Received once in 100 s
ackMode: NONE, streaming pull Received once in 100 s: the client keeps extending the deadline (maxAckExtensionPeriod 3600 s)
ackMode: NONE, maxAckExtensionPeriod: 5 Received again every ~5 s (22 times in 100 s)
ackMode: NONE, synchronousPull: true, subscription deadline 10 s Received again every 10 s (11 times in 100 s)
ackMode: NONE, route calls ${header.CamelGooglePubsubAcknowledge.ack(${exchange})} (Simple) or request.headers.CamelGooglePubsubAcknowledge.ack(exchange) (Groovy) Received once, not again after the 10 s deadline
Failing route, dead-letter subscription (max 5), maxDeliveryAttempts unset Auto-fetched maxDeliveryAttempts=5; attempts 1–4 processed (CamelGooglePubsubDeliveryAttempt 1–4), attempt 5 nacked unprocessed; message on the dead-letter topic
The same with maxDeliveryAttempts: 2 Attempt 1 processed, attempts 2–5 nacked unprocessed; still moved to the dead-letter topic after 5 (the subscription's limit)
The same with maxDeliveryAttempts: 0 All 5 attempts processed
Subscription without a dead-letter policy No CamelGooglePubsubDeliveryAttempt header
maxDeliveryAttempts unset, policy cannot be read (subscription missing) WARN Failed to auto-fetch maxDeliveryAttempts ... enforcement will be disabled
google-storage-event-based-source, ackMode unset / NONE + synchronousPull Notification received once / again after 10 s
The same, failing route, maxDeliveryAttempts: 2 Attempt 1 processed, attempts 2–5 nacked unprocessed
google-pubsub-sink, CamelGooglePubsubOrderingKey set, options unset Every message fails: Cannot publish a message with an ordering key when message ordering is not enabled
messageOrderingEnabled: true Published; WARN In conjunction with enabling message ordering the pubsubEndpoint should be set
messageOrderingEnabled: true, pubsubEndpoint: localhost:<emulator> Published, no warning. Ten messages 20 ms apart arrive on an ordered subscription in publish order
pubsubEndpoint: localhost:<emulator> without the component's emulator endpoint Published to the emulator (message IDs from its sequence): the option alone chooses the service
google-storage-source, moveAfterRead: true, destinationBucket: archive a.txt gone from inbox, present in archive
The same with deleteAfterRead: false Copied to archive, kept in the source bucket
moveAfterRead, failing route Not moved, not deleted; read again
moveAfterRead, destination bucket missing WARN Error occurred during moving or deleting object (404); file kept, read again every poll
includeBody: false, before the template step Every file fails: No type converter available to convert from type: com.google.cloud.storage.Blob to the required type: byte[]; read again every 500 ms
includeBody: false, with it Body empty, CamelGoogleCloudStorageContentLength: 7; file deleted
includeBody: false + moveAfterRead Moved without downloading; body empty
includeBody: true Body is the content, as when unset
includeFolders unset / false, bucket with docs/ and docs/x.txt Both read / only docs/x.txt
deleteAfterRead: false, delay unset / 5000 Same file read every ~500 ms / every 5 s
google-storage-sink, objectName: "status/latest 100%+x.json", file header set Both messages written to status/latest 100%+x.json, as written
autoCreateBucket: true, storageLocation: europe-west1 / unset Captured create request: "location":"europe-west1" / "location":"US-EAST1" (both "storageClass":"STANDARD"; the upload request sets no class)
storageClass: NEARLINE (option tried, then left out) Route fails at start: no type converter from String to StorageClass
Source and sink, useWorkloadIdentityFederation: true, workloadIdentityConfig: file:cred.json Google STS answers invalid_target for the made-up pool: the file is loaded and the token exchange attempted. A local token_url is refused by the Google client (The provided token URL is invalid)
The same plus impersonatedServiceAccount Fails with Unable to refresh sourceCredentials from the impersonating credentials, caused by the same STS answer
google-bigquery-sink, useAsInsertId: orderId, same row twice Captured: {"rows":[{"insertId":"o-1","json":{...}}]} both times
useAsInsertId unset Captured: {"rows":[{"json":{...}}]}, no insertId
JSON array of two rows Each row has its own insertId (o-3, o-4)
Row without the field Sent without insertId
useAsInsertId: num (a number field) Message fails with ClassCastException: Integer cannot be cast to String; nothing sent

The BigQuery emulator stores every row, also a repeated insert ID: the de-duplication is done by BigQuery on a best-effort basis, so only the request was checked, not the de-duplication itself.

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root, the latter after the header counts of KameletsCatalogTest were updated. Before building, the library copies of the six Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…ck, ordering, move and insert ID options (apache#3170)

New optional options, all mapped with optional placeholders, so that with
them unset the endpoint URIs are the same as before:

- google-pubsub-sink: messageOrderingEnabled and pubsubEndpoint, so that the
  messages with the same CamelGooglePubsubOrderingKey header keep their order.
- google-pubsub-source: ackMode, maxDeliveryAttempts and
  maxAckExtensionPeriod.
- google-storage-event-based-source: ackMode and maxDeliveryAttempts.
- google-storage-source: moveAfterRead with destinationBucket, includeBody,
  includeFolders and delay, and Workload Identity Federation
  (useWorkloadIdentityFederation, workloadIdentityConfig,
  impersonatedServiceAccount).
- google-storage-sink: objectName and storageLocation, and the same Workload
  Identity Federation options.
- google-bigquery-sink: useAsInsertId, so that BigQuery can drop a row that a
  retried message sends again.

With includeBody false, the component puts the Blob object of the Google
client in the body, which the binary output data type of the source cannot
convert, so every file failed and was read again forever. A precondition
choice, resolved when the route is created, empties the body in that case
only.

storageClass is left out of google-storage-sink: the component has no type
converter from text to com.google.cloud.storage.StorageClass, so any value
fails the route at start. It would also only set the class of a bucket that
autoCreateBucket creates, not of each file.

The descriptions, the doc partials and the header counts of
KameletsCatalogTest are updated, and the library copies refreshed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus force-pushed the ci-issue-3170-google-data branch from 6743be1 to b7b897f Compare November 18, 2026 20:01
@davsclaus
davsclaus merged commit dd07214 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-google-data branch November 18, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant