Skip to content

chore: the SQS Kamelets can process in parallel, delay messages and skip ListQueues, the SNS sinks can set a subject (#3170) - #3234

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-aws-sqs-sns
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-aws-sqs-sns

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: the SQS and SNS Kamelets get the options the issue lists. aws-sqs-source can process messages in parallel and keep a slow message hidden while it is processed. The SQS sinks can send without listing the queues, delay messages, and choose what happens when a message has too many headers. The SNS sinks can set a subject and a text for each kind of subscriber.

The options

aws-sqs-source:

Option Template mapping What it does
concurrentConsumers concurrentConsumers: "{{?concurrentConsumers}}" How many threads poll the queue and process messages at the same time
extendMessageVisibility extendMessageVisibility: "{{?extendMessageVisibility}}" Keeps a message hidden for as long as it is processed. It needs visibilityTimeout: every half of it, the timeout is extended by visibilityTimeout seconds. Without visibilityTimeout, the integration does not start
queueOwnerAWSAccountId queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}" The queue URL is built from the account, the region and the name, without ListQueues. Use it for a queue of another account too

aws-sqs-sink, aws-sqs-batch-sink, aws-sqs-fifo-sink:

Option Template mapping What it does
queueURL queueUrl: "{{?queueURL}}" Sends to this URL, without ListQueues. Same name as the existing option of aws-sqs-source
queueOwnerAWSAccountId queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}" As in the source
delaySeconds (not in the FIFO sink) delaySeconds: "{{?delaySeconds}}" Keeps each message (each part, in the batch sink) hidden for 0 to 900 seconds after it is sent
messageHeaderExceededLimit messageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}" WARN (the component default), WARN_ONCE, IGNORE or FAIL, for a message with more than 10 headers to send as message attributes
serverSideEncryptionEnabled serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}" With kmsMasterKeyId, sets the queue to encrypt its messages with that KMS key
kmsMasterKeyId kmsMasterKeyId: "{{?kmsMasterKeyId}}" The KMS key ID, alias or ARN

aws-sns-sink, aws-sns-fifo-sink:

Option Template mapping What it does
subject subject: "RAW({{?subject}})" The subject of every notification. The subject / ce-subject header still takes precedence
messageStructure messageStructure: "{{?messageStructure}}" json: the body is a JSON object with a default text and a text for each protocol
serverSideEncryptionEnabled serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}" With kmsMasterKeyId, a topic that autoCreateTopic creates is encrypted with that KMS key
kmsMasterKeyId kmsMasterKeyId: "{{?kmsMasterKeyId}}" The KMS key ID, alias or ARN
  • Defaults: all the options are optional and have no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are the same as on main. In the FIFO sinks, the options are added to both aws2-sqs / aws2-sns steps.
  • How the queue is found today: without a queue URL, aws2-sqs calls ListQueues with the queue name as prefix when the endpoint starts, so the credentials need sqs:ListQueues. It does not call it when queueUrl is set. It also does not call it when the account is known, from queueOwnerAWSAccountId or from an ARN in queueNameOrArn: it then builds https://sqs.<region>.amazonaws.com/<account>/<name>. The ARN case already worked on main, and the descriptions now say so.
  • serverSideEncryptionEnabled and kmsMasterKeyId, SQS: they apply to an existing queue too, not only to autoCreateQueue. When the endpoint starts, aws2-sqs sends SetQueueAttributes with KmsMasterKeyId to an existing queue (this needs sqs:SetQueueAttributes), or CreateQueue with it when autoCreateQueue creates the queue. They change the queue itself, and the descriptions say so. serverSideEncryptionEnabled without a key does nothing; kmsMasterKeyId without serverSideEncryptionEnabled does nothing either.
  • serverSideEncryptionEnabled and kmsMasterKeyId, SNS: they apply only to CreateTopic, when autoCreateTopic is true and the topic does not exist. An existing topic is not changed. serverSideEncryptionEnabled without a key fails the start when the topic must be created.
  • The subject: the Kamelets already took the subject from the subject or ce-subject header and set CamelAwsSnsSubject from it. The new subject option maps to the subject endpoint option, which aws2-sns uses only when that header is missing, so the header still wins. It is in RAW() so that +, %, & and # are kept. The pattern: ^[ -~]+$ (printable ASCII, no line breaks) follows the rules SNS has for a subject. aws2-sns cuts a subject longer than 100 characters to 100.
  • Why the FIFO SQS sink has no delaySeconds: SQS takes no delay for each message on a FIFO queue, only a delay for the whole queue. aws2-sqs skips the per-message delay for a queue whose name ends in .fifo. The description says so.
  • messageHeaderExceededLimit: the 10 headers that are sent are the first 10 in the order they were set on the message, not in alphabetical order. In the batch sink the headers go with each part, so WARN_ONCE logs once for each part.
  • pattern: on the new free-form options: queueURL (^https?://[^\s]+$), queueOwnerAWSAccountId (12 digits), kmsMasterKeyId (as awsKMSKeyId in the S3 Kamelets), subject.
  • Descriptions: the "Good to know" paragraphs, the option descriptions and the subject header descriptions are updated. Each partial page has new sections with an example: "Slow Messages and Parallel Processing" and "Without Permission to List the Queues" (source); "Delaying the Messages", "Without Permission to List the Queues", "Too Many Headers" and "Encrypting the Queue" (SQS sinks); "The Subject", "A Text for Each Kind of Subscriber" and "Encrypting a New Topic" (SNS sinks).
  • KameletsCatalogTest counts the options of aws-sqs-source: now 29.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT against floci/floci. To see the AWS calls, the Kamelets connected to a small reverse proxy in front of floci. It logged the operation and the body of each request. In the ListQueues cases, it answered ListQueues with AccessDenied.

Case Result
Options unset, all six Kamelets, main and this branch camel get endpoint: the 8 aws2-sqs / aws2-sns URIs are identical; the messages are sent and received
ListQueues denied, aws-sqs-sink without queueURL Fails to start: ListQueues is called and denied (not authorized to perform ListQueues)
ListQueues denied, queueURL on aws-sqs-sink, aws-sqs-batch-sink, aws-sqs-fifo-sink All three send; the proxy logs only SendMessage / SendMessageBatch to that URL, with no ListQueues
ListQueues denied, queueOwnerAWSAccountId: "000000000000" on aws-sqs-sink and aws-sqs-source The sink sends and the source receives and deletes, with no ListQueues. Queue URL https://sqs.us-east-1.amazonaws.com/000000000000/qb
ListQueues denied, ARN in queueNameOrArn, aws-sqs-sink (no new option) Sends, with no ListQueues
delaySeconds: 10 on aws-sqs-sink Received 10.27 s after SentTimestamp
delaySeconds: 6 on aws-sqs-batch-sink, 2 parts Both parts received 6.17 s after SentTimestamp
No delaySeconds, same queue Received 12 ms after SentTimestamp
aws-sqs-source, 6 messages, each takes 2 s (blocking delay), concurrentConsumers unset One thread, one message after the other: 14.6 s in all
The same with concurrentConsumers: 3 Three threads, three messages at a time: 4.5 s in all
visibilityTimeout: 4, a message that takes 12 s, 2 threads, no extendMessageVisibility Received twice: by the second thread 4.1 s later, with ApproximateReceiveCount 2, and processed twice
The same with extendMessageVisibility: true Received once; ChangeMessageVisibilityBatch every 2 s until it ended, then DeleteMessage
extendMessageVisibility: true without visibilityTimeout Does not start: extendMessageVisibility requires visibilityTimeout to be set on the Endpoint
12 headers, messageHeaderExceededLimit unset, aws-sqs-sink Sent with 10 attributes (zeta, Beta, alpha, h04…h10, the first 10 set); a warning each for h11 and h12
The same, WARN_ONCE / IGNORE The same 10 attributes; one warning / no warning
The same, WARN_ONCE, aws-sqs-batch-sink, 2 parts One warning for each part
The same, FAIL, aws-sqs-sink and aws-sqs-fifo-sink IllegalArgumentException: Number of message headers exceeded; no SendMessage
serverSideEncryptionEnabled: true, kmsMasterKeyId: alias/aws/sqs, existing queue, aws-sqs-sink SetQueueAttributes {"KmsMasterKeyId":"alias/aws/sqs"} at start; the queue has the key
kmsMasterKeyId only, existing queue No SetQueueAttributes; the queue has no key
Both, autoCreateQueue: true, new queue, aws-sqs-batch-sink and aws-sqs-fifo-sink CreateQueue with KmsMasterKeyId (and FifoQueue for the FIFO one); the queues have the key
aws-sns-sink, subject: "Order #42 + 50% off & more", SQS queue subscribed The notification in the queue has "Subject":"Order #42 + 50% off & more"
subject option and subject header From the header "Subject":"From the header"
Neither No Subject in the notification
messageStructure: json, body {"default": …, "sqs": "case4 text for sqs", "email": …} Publish … MessageStructure=json; the queue gets "Message":"case4 text for sqs"
messageStructure: json, body without default InvalidParameterException: Message Structure - No default entry in JSON message body
aws-sns-fifo-sink, subject, FIFO queue subscribed "Subject":"Fifo subject"
aws-sns-fifo-sink, messageStructure: json The queue gets the sqs text
serverSideEncryptionEnabled + kmsMasterKeyId, autoCreateTopic, new topic, aws-sns-sink / aws-sns-fifo-sink CreateTopic with KmsMasterKeyId; the topics have the key
The same, existing topic No CreateTopic or SetTopicAttributes; the topic has no key
serverSideEncryptionEnabled without kmsMasterKeyId, new topic Does not start: The option kmsMasterKeyId is required when serverSideEncryptionEnabled is true

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root. Before building, the library copies of the six Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…kip ListQueues, the SNS sinks can set a subject (apache#3170)

aws-sqs-source gets concurrentConsumers, extendMessageVisibility (which
needs visibilityTimeout) and queueOwnerAWSAccountId.

aws-sqs-sink, aws-sqs-batch-sink and aws-sqs-fifo-sink get queueURL and
queueOwnerAWSAccountId, so that the queue is not looked up with
ListQueues, messageHeaderExceededLimit, and serverSideEncryptionEnabled
with kmsMasterKeyId, which change the queue itself: an existing queue is
updated with SetQueueAttributes when the integration starts, and a queue
that autoCreateQueue creates is created with the key. aws-sqs-sink and
aws-sqs-batch-sink also get delaySeconds; aws-sqs-fifo-sink does not, as
a FIFO queue takes no delay for each message and aws2-sqs ignores it.

aws-sns-sink and aws-sns-fifo-sink get subject (the subject header still
takes precedence), messageStructure, and serverSideEncryptionEnabled with
kmsMasterKeyId, which apply only to a topic that autoCreateTopic creates.

All the options are optional, without a default: when they are unset the
endpoint URIs are the same as before. Tested with Camel JBang
4.24.0-SNAPSHOT against floci.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit 16bad22 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-aws-sqs-sns branch November 18, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant