Skip to content

chore: azure-servicebus-source can process messages in parallel and dead-letter failures, the sink can send bytes and delay messages (#3170) - #3239

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-servicebus-extras
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-servicebus-extras

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: azure-servicebus-source can process several messages at the same time, and move a message whose processing fails to the dead-letter queue straight away. azure-servicebus-sink can send the body as bytes, and have Service Bus hold each message for a delay.

The options

Kamelet Option Template mapping What it does
source maxConcurrentCalls maxConcurrentCalls: "{{?maxConcurrentCalls}}" How many messages are processed at the same time. When unset, the component default of 1 applies.
source enableDeadLettering enableDeadLettering: "{{?enableDeadLettering}}" true dead-letters a message whose processing fails, instead of abandoning it.
sink binary binary: "{{?binary}}" true sends the body as bytes instead of text.
sink scheduledEnqueueDelay When set: a choice sets the CamelAzureServiceBusProducerOperation header to scheduleMessages, and CamelAzureServiceBusScheduledEnqueueTime to ${bean:type:java.time.OffsetDateTime?method=now()} plus {{?scheduledEnqueueDelay}} seconds, before the send; a second choice removes both headers after it Service Bus holds each message for that many seconds.
  • Defaults: all four options are optional and deliberately have no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them and the choice does nothing, so the endpoint URIs are identical to main's.
  • maxConcurrentCalls:
    • Above 1, messages are processed in no particular order.
    • With sessionEnabled, it counts for each session. The messages of one session are then processed at the same time too, and lose their order. The descriptions say so.
  • enableDeadLettering:
    • The dead letter gets the exception as its reason (java.lang.IllegalStateException: …) and the stack trace as its error description. The component does this.
    • It works only with PEEK_LOCK.
    • It does nothing while reading a dead-letter queue with subQueue: the component abandons the message there, as before.
  • binary:
    • Without it, the producer reads the body as a String. Bytes that are not valid UTF-8 come out damaged.
    • With it, a Map body fails (NullPointerException: 'content' cannot be null). The descriptions say to turn it into JSON first.
  • scheduledEnqueueDelay is a delay, not the component's scheduledEnqueueTime. The component has a scheduledEnqueueTime endpoint option, but it cannot be used from a Kamelet:
    • It is a fixed OffsetDateTime, and Camel has no type converter from a String. scheduledEnqueueTime=2026-11-18T23:00:00+01:00 fails with No type converter available to convert from type: java.lang.String to the required type: java.time.OffsetDateTime.
    • It is read only by the scheduleMessages operation. The Kamelet uses sendMessages, which ignores it.
    • A fixed instant would also be of little use for a stream of messages.
  • How the delay is set: the template computes now plus the delay in seconds, and sets it in the CamelAzureServiceBusScheduledEnqueueTime header. It also sets the operation header for that send, so the endpoint stays producerOperation=sendMessages, and no second sender client is created.
    • Both headers are set only when the option is set, so the delay is fixed for the Kamelet.
    • Both are removed after the send, so the message leaves the Kamelet unchanged ("Output: the message is unchanged."). Without that removal, a test showed the caller getting CamelAzureServiceBusProducerOperation=scheduleMessages back, which would schedule its next send to Service Bus as well.
    • The computation uses the bean language (${bean:type:…} and an OGNL call), so the sink now declares camel:bean. The catalog SBOM was regenerated by the build.
  • Descriptions and partials:
    • Source: "Example Processing Messages in Parallel" and "Example Dead-Lettering Failed Messages".
    • Sink: "Example Sending Binary Data" and "Example with a Delay".
    • The Good-to-know paragraphs cover ordering, sessions, the dead-letter reason, and the binary and delay behaviour.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT against the Azure Service Bus emulator (mcr.microsoft.com/azure-messaging/servicebus-emulator, with mcr.microsoft.com/mssql/server:2022-latest). It was configured through Config.json with:

  • plain queues with MaxDeliveryCount: 10;
  • two queues with RequiresSession: true;
  • a topic with a subscription.

Processing was failed on purpose with throwException. Received bytes were compared through base64.

Case Result
Options unset, both Kamelets, main and this branch camel get endpoint --json: identical URIs (azure-servicebus://plainq?connectionString=…&credentialType=CONNECTION_STRING&producerOperation=sendMessages&serviceBusType=queue, and …&serviceBusReceiveMode=PEEK_LOCK&serviceBusType=queue); message sent and received
All four options set Sink endpoint has binary=true (still producerOperation=sendMessages). Source endpoint has enableDeadLettering=true&maxConcurrentCalls=4. Message received 1 s later (scheduledEnqueueDelay: 1)
10 messages, synchronous 1 s step, maxConcurrentCalls unset P-1 … P-10 in order, one thread, one per second: 10 s
The same, maxConcurrentCalls: 5 P-1 … P-5 at 21:48:47.015 on 5 threads, then P-6 … P-10 at 21:48:48.049: 2 s, out of order (P-7 before P-6)
Session queue, 6 messages of session A, sessionEnabled: true, maxConcurrentCalls: 3, 1 s step A-1, A-2, A-3 at the same time, then A-4, A-5, A-6: three at once within one session
Plain queue, E-1 fails, enableDeadLettering unset E-1 received 10 times (delivery count 0 … 9), then dead-lettered: MaxDeliveryCountExceeded
The same with enableDeadLettering: true D-1 received once, then D-2 processed. In the dead-letter queue (subQueue: DEAD_LETTER_QUEUE): D-1 with reason java.lang.IllegalStateException: processing failed on purpose and the stack trace as description
enableDeadLettering: true on a session queue, S-2 fails S-1, S-2 (once), S-3; S-2 is in the dead-letter queue with the exception as reason
enableDeadLettering: true on a topic subscription, T-1 fails T-1 once, T-2 processed; T-1 is in the subscription's dead-letter queue with the exception as reason
enableDeadLettering: true with RECEIVE_AND_DELETE, R-1 fails Received once; neither in the queue nor in the dead-letter queue afterwards (lost, as before)
enableDeadLettering: true with subQueue: DEAD_LETTER_QUEUE, processing fails The dead letter is received again 501 times in 12 s: not dead-lettered again
Sink, body bytes 0x00 … 0xFF (256 bytes), binary unset Received 512 bytes: each byte ≥ 0x80 became U+FFFD (efbfbd)
The same with binary: true Received 256 bytes, identical
file: route sending a 1 KiB binary file, binary: true (the partial's example) Received file identical (cmp)
binary: true, text blåbærgrød Received as 13 bytes, blåbærgrød (UTF-8)
binary: true, Map body Fails: NullPointerException: 'content' cannot be null
scheduledEnqueueDelay: 20, 2 messages, reader running Sent at 21:56:32.365. Received at 21:56:52.367 with enqueued time 20:56:52.340Z (UTC): 20 s later. After the sink, CamelAzureServiceBusProducerOperation and CamelAzureServiceBusScheduledEnqueueTime are empty
scheduledEnqueueDelay: 10 with sessionId: B to a session queue Received 10 s later, with session B
scheduledEnqueueDelay set, template without the removal step The caller sees op=scheduleMessages and the scheduled time after the sink, which is why the step is there
Sink with no option set, message carries only a CamelAzureServiceBusScheduledEnqueueTime header (OffsetDateTime, now + 15 s) Received at once: the header is not used with sendMessages

Not verified:

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root:

  • Before building, the library copies of the two Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one.
  • KameletsCatalogTest needed no change.
  • The build regenerated kamelets-index.json and the catalog SBOM (camel-bean for the sink).

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…ead-letter failures, the sink can send bytes and delay messages (apache#3170)

azure-servicebus-source takes maxConcurrentCalls, how many messages are
processed at the same time (with sessions, for each session), and
enableDeadLettering, which moves a message whose processing fails to the
dead-letter queue straight away, with the exception as the reason,
instead of giving it back until the maximum delivery count.

azure-servicebus-sink takes binary, to send the body as bytes rather than
as text, and scheduledEnqueueDelay, a number of seconds Service Bus holds
each message before readers can receive it.

camel-azure-servicebus has a scheduledEnqueueTime endpoint option, but it
is a fixed OffsetDateTime that cannot be set from a string (no type
converter), and it is read only by the scheduleMessages operation. So,
when scheduledEnqueueDelay is set, the template sets the
CamelAzureServiceBusProducerOperation and
CamelAzureServiceBusScheduledEnqueueTime headers to now plus the delay
for the send, and removes them afterwards, so the message leaves the
Kamelet unchanged. The time is computed with the bean language, so the
sink now declares camel:bean.

All the new options are optional placeholders without a default: when
they are unset, the endpoint URIs are identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit 1627074 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-servicebus-extras branch November 18, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant