Repository navigation
chore: the SQS Kamelets can process in parallel, delay messages and skip ListQueues, the SNS sinks can set a subject (#3170) - #3234
Merged
Conversation
…kip ListQueues, the SNS sinks can set a subject (apache#3170) aws-sqs-source gets concurrentConsumers, extendMessageVisibility (which needs visibilityTimeout) and queueOwnerAWSAccountId. aws-sqs-sink, aws-sqs-batch-sink and aws-sqs-fifo-sink get queueURL and queueOwnerAWSAccountId, so that the queue is not looked up with ListQueues, messageHeaderExceededLimit, and serverSideEncryptionEnabled with kmsMasterKeyId, which change the queue itself: an existing queue is updated with SetQueueAttributes when the integration starts, and a queue that autoCreateQueue creates is created with the key. aws-sqs-sink and aws-sqs-batch-sink also get delaySeconds; aws-sqs-fifo-sink does not, as a FIFO queue takes no delay for each message and aws2-sqs ignores it. aws-sns-sink and aws-sns-fifo-sink get subject (the subject header still takes precedence), messageStructure, and serverSideEncryptionEnabled with kmsMasterKeyId, which apply only to a topic that autoCreateTopic creates. All the options are optional, without a default: when they are unset the endpoint URIs are the same as before. Tested with Camel JBang 4.24.0-SNAPSHOT against floci. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170: the SQS and SNS Kamelets get the options the issue lists.
aws-sqs-sourcecan process messages in parallel and keep a slow message hidden while it is processed. The SQS sinks can send without listing the queues, delay messages, and choose what happens when a message has too many headers. The SNS sinks can set a subject and a text for each kind of subscriber.The options
aws-sqs-source:concurrentConsumersconcurrentConsumers: "{{?concurrentConsumers}}"extendMessageVisibilityextendMessageVisibility: "{{?extendMessageVisibility}}"visibilityTimeout: every half of it, the timeout is extended byvisibilityTimeoutseconds. WithoutvisibilityTimeout, the integration does not startqueueOwnerAWSAccountIdqueueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"aws-sqs-sink,aws-sqs-batch-sink,aws-sqs-fifo-sink:queueURLqueueUrl: "{{?queueURL}}"aws-sqs-sourcequeueOwnerAWSAccountIdqueueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"delaySeconds(not in the FIFO sink)delaySeconds: "{{?delaySeconds}}"messageHeaderExceededLimitmessageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}"WARN(the component default),WARN_ONCE,IGNOREorFAIL, for a message with more than 10 headers to send as message attributesserverSideEncryptionEnabledserverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"kmsMasterKeyId, sets the queue to encrypt its messages with that KMS keykmsMasterKeyIdkmsMasterKeyId: "{{?kmsMasterKeyId}}"aws-sns-sink,aws-sns-fifo-sink:subjectsubject: "RAW({{?subject}})"subject/ce-subjectheader still takes precedencemessageStructuremessageStructure: "{{?messageStructure}}"json: the body is a JSON object with adefaulttext and a text for each protocolserverSideEncryptionEnabledserverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"kmsMasterKeyId, a topic thatautoCreateTopiccreates is encrypted with that KMS keykmsMasterKeyIdkmsMasterKeyId: "{{?kmsMasterKeyId}}"default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are the same as onmain. In the FIFO sinks, the options are added to bothaws2-sqs/aws2-snssteps.aws2-sqscalls ListQueues with the queue name as prefix when the endpoint starts, so the credentials needsqs:ListQueues. It does not call it whenqueueUrlis set. It also does not call it when the account is known, fromqueueOwnerAWSAccountIdor from an ARN inqueueNameOrArn: it then buildshttps://sqs.<region>.amazonaws.com/<account>/<name>. The ARN case already worked onmain, and the descriptions now say so.serverSideEncryptionEnabledandkmsMasterKeyId, SQS: they apply to an existing queue too, not only toautoCreateQueue. When the endpoint starts,aws2-sqssendsSetQueueAttributeswithKmsMasterKeyIdto an existing queue (this needssqs:SetQueueAttributes), orCreateQueuewith it whenautoCreateQueuecreates the queue. They change the queue itself, and the descriptions say so.serverSideEncryptionEnabledwithout a key does nothing;kmsMasterKeyIdwithoutserverSideEncryptionEnableddoes nothing either.serverSideEncryptionEnabledandkmsMasterKeyId, SNS: they apply only toCreateTopic, whenautoCreateTopicis true and the topic does not exist. An existing topic is not changed.serverSideEncryptionEnabledwithout a key fails the start when the topic must be created.subjectorce-subjectheader and setCamelAwsSnsSubjectfrom it. The newsubjectoption maps to thesubjectendpoint option, whichaws2-snsuses only when that header is missing, so the header still wins. It is inRAW()so that+,%,&and#are kept. Thepattern:^[ -~]+$(printable ASCII, no line breaks) follows the rules SNS has for a subject.aws2-snscuts a subject longer than 100 characters to 100.delaySeconds: SQS takes no delay for each message on a FIFO queue, only a delay for the whole queue.aws2-sqsskips the per-message delay for a queue whose name ends in.fifo. The description says so.messageHeaderExceededLimit: the 10 headers that are sent are the first 10 in the order they were set on the message, not in alphabetical order. In the batch sink the headers go with each part, soWARN_ONCElogs once for each part.pattern:on the new free-form options:queueURL(^https?://[^\s]+$),queueOwnerAWSAccountId(12 digits),kmsMasterKeyId(asawsKMSKeyIdin the S3 Kamelets),subject.subjectheader descriptions are updated. Each partial page has new sections with an example: "Slow Messages and Parallel Processing" and "Without Permission to List the Queues" (source); "Delaying the Messages", "Without Permission to List the Queues", "Too Many Headers" and "Encrypting the Queue" (SQS sinks); "The Subject", "A Text for Each Kind of Subscriber" and "Encrypting a New Topic" (SNS sinks).KameletsCatalogTestcounts the options ofaws-sqs-source: now 29.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against
floci/floci. To see the AWS calls, the Kamelets connected to a small reverse proxy in front of floci. It logged the operation and the body of each request. In the ListQueues cases, it answered ListQueues withAccessDenied.mainand this branchcamel get endpoint: the 8aws2-sqs/aws2-snsURIs are identical; the messages are sent and receivedaws-sqs-sinkwithoutqueueURLListQueuesis called and denied (not authorized to perform ListQueues)queueURLonaws-sqs-sink,aws-sqs-batch-sink,aws-sqs-fifo-sinkSendMessage/SendMessageBatchto that URL, with noListQueuesqueueOwnerAWSAccountId: "000000000000"onaws-sqs-sinkandaws-sqs-sourceListQueues. Queue URLhttps://sqs.us-east-1.amazonaws.com/000000000000/qbqueueNameOrArn,aws-sqs-sink(no new option)ListQueuesdelaySeconds: 10onaws-sqs-sinkSentTimestampdelaySeconds: 6onaws-sqs-batch-sink, 2 partsSentTimestampdelaySeconds, same queueSentTimestampaws-sqs-source, 6 messages, each takes 2 s (blocking delay),concurrentConsumersunsetconcurrentConsumers: 3visibilityTimeout: 4, a message that takes 12 s, 2 threads, noextendMessageVisibilityApproximateReceiveCount2, and processed twiceextendMessageVisibility: trueChangeMessageVisibilityBatchevery 2 s until it ended, thenDeleteMessageextendMessageVisibility: truewithoutvisibilityTimeoutextendMessageVisibility requires visibilityTimeout to be set on the EndpointmessageHeaderExceededLimitunset,aws-sqs-sinkzeta,Beta,alpha,h04…h10, the first 10 set); a warning each forh11andh12WARN_ONCE/IGNOREWARN_ONCE,aws-sqs-batch-sink, 2 partsFAIL,aws-sqs-sinkandaws-sqs-fifo-sinkIllegalArgumentException: Number of message headers exceeded; noSendMessageserverSideEncryptionEnabled: true,kmsMasterKeyId: alias/aws/sqs, existing queue,aws-sqs-sinkSetQueueAttributes {"KmsMasterKeyId":"alias/aws/sqs"}at start; the queue has the keykmsMasterKeyIdonly, existing queueSetQueueAttributes; the queue has no keyautoCreateQueue: true, new queue,aws-sqs-batch-sinkandaws-sqs-fifo-sinkCreateQueuewithKmsMasterKeyId(andFifoQueuefor the FIFO one); the queues have the keyaws-sns-sink,subject: "Order #42 + 50% off & more", SQS queue subscribed"Subject":"Order #42 + 50% off & more"subjectoption andsubjectheaderFrom the header"Subject":"From the header"Subjectin the notificationmessageStructure: json, body{"default": …, "sqs": "case4 text for sqs", "email": …}Publish … MessageStructure=json; the queue gets"Message":"case4 text for sqs"messageStructure: json, body withoutdefaultInvalidParameterException: Message Structure - No default entry in JSON message bodyaws-sns-fifo-sink,subject, FIFO queue subscribed"Subject":"Fifo subject"aws-sns-fifo-sink,messageStructure: jsonsqstextserverSideEncryptionEnabled+kmsMasterKeyId,autoCreateTopic, new topic,aws-sns-sink/aws-sns-fifo-sinkCreateTopicwithKmsMasterKeyId; the topics have the keyCreateTopicorSetTopicAttributes; the topic has no keyserverSideEncryptionEnabledwithoutkmsMasterKeyId, new topicThe option kmsMasterKeyId is required when serverSideEncryptionEnabled is true./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root. Before building, the library copies of the six Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code