Repository navigation
chore: the RabbitMQ Kamelets can scale, retry and dead-letter, and the sink can set the content type (#3170) - #3238
Merged
Conversation
…e sink can set the content type (apache#3170) spring-rabbitmq-source takes new optional options: - exchangeType: the type of the exchange that autoDeclare creates (direct, fanout, headers, topic); with topic, routingKey is a pattern. - concurrentConsumers and prefetchCount: how many consumers receive in parallel, each on its own channel, and how many messages RabbitMQ sends ahead to each. - acknowledgeMode: AUTO or NONE. MANUAL is left out: a Kamelet cannot acknowledge, so the messages would stay unacknowledged. - maximumRetryAttempts and retryDelay: how many times a failing message is tried again, and how long apart. - deadLetterExchange, deadLetterRoutingKey, deadLetterQueue and deadLetterExchangeType: with autoDeclare, create a dead letter exchange and queue and set them on the queue, so that a message that still fails after the retries moves to the dead letter queue. spring-rabbitmq-sink takes allowNullBody, to send a message with no body as an empty message, and contentType, to send every message with a content type such as application/json instead of text/plain or application/octet-stream. maximumRetryAttempts counts the retries, not the tries: a message is tried 6 times when it is not set, so the description of the source, which said 5, is corrected. rejectAndDontRequeue and maxConcurrentConsumers are left out: with false, the first drops the message after the retries instead of requeuing it, and the second works only with the SMLC container, which the Kamelet does not use. With the new options unset, the endpoint URIs are the same as before. The connection factory and the trust store options are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170:
spring-rabbitmq-sourcecan run several consumers in parallel, set the prefetch count and the acknowledge mode, choose the exchange type, tune the retries and set up a dead letter queue;spring-rabbitmq-sinkcan send a message with no body and set the content type.The options
spring-rabbitmq-source, all added to the endpoint URI with optional placeholders:exchangeTypeexchangeType={{?exchangeType}}autoDeclarecreates:direct,fanout,headersortopic. Withtopic,routingKeyis a pattern (orders.*,orders.#).concurrentConsumersconcurrentConsumers={{?concurrentConsumers}}prefetchCountprefetchCount={{?prefetchCount}}acknowledgeModeacknowledgeMode={{?acknowledgeMode}}AUTO(as today) orNONE: removed from the queue on delivery, so a failing message is lost.maximumRetryAttemptsmaximumRetryAttempts={{?maximumRetryAttempts}}retryDelayretryDelay={{?retryDelay}}deadLetterExchangedeadLetterExchange={{?deadLetterExchange}}autoDeclare, creates this exchange and sets it as the dead letter exchange of the queue.deadLetterRoutingKeydeadLetterRoutingKey={{?deadLetterRoutingKey}}deadLetterQueue.deadLetterQueuedeadLetterQueue={{?deadLetterQueue}}autoDeclare, creates this queue and binds it todeadLetterExchange.deadLetterExchangeTypedeadLetterExchangeType={{?deadLetterExchangeType}}directwhen unset).spring-rabbitmq-sink:allowNullBodyallowNullBody={{?allowNullBody}}in the endpoint URIcontentTypechoiceon'{{?contentType}}' != ''that sets the headerCamelSpringRabbitmqContentTypeto{{?contentType}}application/json.default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them and the endpoint URIs are the same as onmain; thechoiceof the sink takes no branch.Stringbody astext/plainand anything else asapplication/octet-stream, unless the headerCamelSpringRabbitmqContentTypesays otherwise.contentTypesets that header, so JSON text can go out asapplication/json. It is not a URI option of the component, hence the header.^[A-Za-z0-9_.:-]+$;contentTypetakes a media type with optional parameters.maximumRetryAttemptscounts the retries, not the tries, with the Spring AMQP that Camel 4.24 uses: unset, a message is tried 6 times; 3 gives 4 tries, 0 gives 1. The Good to know of the source said "tried 5 times"; it now says 6. (The Camel option description, "the number of attempts includes the initial try", is no longer right; that is forapache/camel.)autoDeclare: the component sets the dead letter exchange only on the queues it declares. WithautoDeclarefalse, the options do nothing (verified), and the descriptions say to set the dead letter exchange on the queue in RabbitMQ, for example with a policy.deadLetterQueuewithoutdeadLetterRoutingKeyfails to start ('routingKey' must be non-null), also with afanoutdead letter exchange; the description says so.acknowledgeMode: MANUAL: a Kamelet has no way to acknowledge, so every processed message stays unacknowledged (verified:unacked 1after processing). The enum offersAUTOandNONEonly.rejectAndDontRequeue: withfalsethe message is not requeued, as the component docs suggest; after the retries, Spring AMQP logsMessage(s) dropped on recoveryand acknowledges it, so it is lost, also when the queue has a dead letter exchange. The option would only turn the dead letter exchange off.maxConcurrentConsumers: only used by theSMLClistener container; the Kamelet uses the defaultDMLC, whereconcurrentConsumersis a fixed count. Adding it would also needmessageListenerContainerType.SpringRabbitmqKameletTlsTestpasses.exchangeType, the start failure when the exchange or queue exists with other settings, the retry count, the dead letter options and parallel processing. The Input of the sink namescontentTypeandallowNullBody, and no longer says that an "empty body" fails: an empty string is sent fine; only a message with no body fails.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against
rabbitmq:management(RabbitMQ 4.4.1), reading the broker state from the management API.mainand this branchcamel get endpointshows identicalspring-rabbitmqURIs; the message arrives withCamelSpringRabbitmqContentType=text/plainas beforeconcurrentConsumers: 3,prefetchCount: 7concurrentConsumers=3&prefetchCount=7;/api/consumers: 3 consumers on the queue, on channels (1), (2), (3) of one connection, each with prefetch 7concurrentConsumers: 3, 6 messages, 2 s of work eachmaximumRetryAttempts: 3,retryDelay: 500,deadLetterExchange,deadLetterRoutingKey: failed,deadLetterQueue, failing routex-dead-letter-exchangeandx-dead-letter-routing-key: failed; the dead letter queue holds the message, routing keyfailed, withx-death(reasonrejected, queue and original routing key)maximumRetryAttempts: 1/0deadLetterExchangeType: fanoutfanout; message reaches the dead letter queuedeadLetterExchangeonly, no routing key or queue; a queue bound to it by hand with the original keyworkand reaches that queuedeadLetterQueuewithoutdeadLetterRoutingKey'routingKey' must be non-nullautoDeclare: false, queue created by handacknowledgeMode: NONE, failing route with dead letter options/api/consumersshowsack_required: false; 3 tries, then the message is gone, and the dead letter queue stays emptyacknowledgeMode: MANUAL(not offered)unacked 1rejectAndDontRequeue: false(not offered)Message(s) dropped on recovery; not requeued, not dead-letteredexchangeType: topic, two sources bound withorders.*andorders.#; messages with keysorders.new,orders.eu.new,invoices.newtopic;orders.*gotorders.newonly,orders.#gotorders.newandorders.eu.new, nobody gotinvoices.newexchangeType: topicon an existingdirectexchangePRECONDITION_FAILED - inequivalent arg 'type'failed, started withotherPRECONDITION_FAILED - inequivalent arg 'x-dead-letter-routing-key'allowNullBody: trueapplication/octet-streamallowNullBody: false; and onmainCannot send message as message body is null, and option allowNullBody is false.text/plaincontentType: "application/json; charset=utf-8", text bodyapplication/json; charset=utf-8contentTypeunsetapplication/octet-streamcontentType: application/xml, message header sayingtext/plainapplication/xmlallowNullBody: true,contentType: application/jsonapplication/jsonwebhook-source→ sink, source with a step that fails)curlPOST of JSON; 4 consumers with prefetch 10; 3 tries 2 s apart; the message inorders.dlqwith content typeapplication/jsonandx-death./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root (includingSpringRabbitmqKameletTlsTestandKameletsCatalogTest). Before building, the library copies of both Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code