Skip to content

chore: nats-source can use a durable JetStream consumer that carries on after a restart (#3170) - #3236

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-nats-jetstream
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-nats-jetstream

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: nats-source can use a durable JetStream consumer, which carries on after a restart where it left off instead of reading the stream again from the beginning, and which several copies of an integration can share. It can also choose how messages are acknowledged and how many times a failing message is delivered. The descriptions of the existing JetStream options of both Kamelets are corrected after running them.

The options

Option Kamelet Template mapping
durableName (pattern: ^[A-Za-z0-9_-]+$) nats-source durableName: "{{?durableName}}"
ackPolicy (enum: Explicit, All, None) nats-source ackPolicy: "{{?ackPolicy}}"
maxDeliver (integer) nats-source maxDeliver: "{{?maxDeliver}}"
  • durableName. camel-nats creates a durable pull consumer. The server keeps it and remembers which messages were acknowledged, so after a restart the source carries on with the first unacknowledged message. Copies of the integration with the same durableName share the messages, which queueName cannot do with JetStream (chore: the NATS Kamelets can use TLS and a credentials file, and nats-source a queue group (#3170) #3205). camel-nats names the consumer <durableName>-durable on the server, and the description says so, because that is the name nats consumer rm needs. jnats refuses ., *, >, / and \ in the name, so the pattern allows only letters, digits, - and _.
  • ackPolicy and maxDeliver are plain endpoint options of camel-nats, so they need no ConsumerConfiguration bean. They apply to ephemeral consumers too. A durable consumer keeps the values it was created with: if they change later, jnats fails with Existing consumer cannot be modified, logs it, and the source receives nothing until the consumer is deleted. The description says this.
  • requestTimeout is not added to nats-sink. The sink is InOnly since chore: messaging sinks send without waiting for a reply; nats-source and mqtt source fixes #3156, so it never makes a request and waits for a reply. The only other place camel-nats uses requestTimeout is the wait for the JetStream publish acknowledgement with jetstreamAsync=true. jnats already cancels that publish after its own request timeout, which is 2 seconds (the connectionTimeout) and is applied at the next cleanup run, every 5 seconds. So the option could only shorten the wait, never lengthen it. With the server paused, requestTimeout: 500 failed after 0.5 s, the default (20000) after 6 to 7.5 s, 60000 after 7 s, and with jetstreamAsync=false the option is ignored (2 s). For a longer wait, camel-nats would need to pass requestTimeout to the JetStream options of jnats. That would be a Camel change.
  • Corrected descriptions of the existing JetStream options, all checked by running them:
    • nats-sink does not use jetstreamName. The message is stored by whichever stream takes in the subject, whatever its name. The jetstreamName value is used only in dead code (jsm.jetStream() is never null).
    • With jetstreamEnabled, the sink waits until the stream confirms that it stored the message. It fails with 503 No Responders Available For Request when no stream takes in the subject.
    • A stream also stores what the sink publishes without jetstreamEnabled.
    • On a server without JetStream, the sink ignores jetstreamEnabled and publishes as plain NATS.
    • nats-source does not use jetstreamAsync.
    • The sink's jetstreamAsync text said "without blocking a thread". In fact it waits for the acknowledgement either way.
  • In the "Good to know" paragraph of the source: what is acknowledged and when, the redelivery of a failed message 5 seconds later, the 30 seconds after an abrupt stop, and sharing with durableName. The source partial page has a durable consumer section, an ackPolicy table and an example. The sink partial page has a JetStream section and an example. Both examples were run against the test server.
  • None of the options has a default, so with them unset the endpoint URI is exactly as on main.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT (camel run --local-kamelet-dir=kamelets) against nats:2 (nats-server 2.15.3) started with -js, in Docker. natsio/nats-box (nats CLI) published, read streams and inspected consumers. The routes logged each body and the CamelNatsDeliveryCounter header. A route failed on purpose with throwException for bodies that start with bad.

Case Result
Options unset, endpoint URIs from main and from this PR (camel get endpoint --json): plain source, plain sink, JetStream source and sink Same, e.g. nats://ep.js?jetstreamAsync=true&jetstreamEnabled=true&jetstreamName=EPJS&servers=...
All three set nats://dur.x?ackPolicy=All&durableName=allopts&...&maxDeliver=4&...; nats consumer info shows ack_policy: all, max_deliver: 4
durableName: proc: 5 messages, stop, 5 more published while it is stopped, start again First run m1–m5; second run n1–n5 only: nothing lost, nothing repeated. Consumer proc-durable on the server
The same stream without durableName (as on main) Receives all 10 again at start
durableName, slow route, kill -9 while 10 messages are in flight Ack floor at k4. After restart k5–k10 arrive after the 30 s ack wait: k5, which was in progress, twice; nothing lost
Two copies with durableName: workers, 40 messages 19 and 21, no duplicate, all 40
Failing message, defaults (Explicit, no maxDeliver) Delivered again about every 5 s, 7 times in 40 s, with no end
Failing message, maxDeliver: 3 Delivered 3 times, then not again
Failing message, ackPolicy: None Delivered once
Failing message then a good one, ackPolicy: All Bad delivered once: the good one's ack acknowledged it
Failing message alone, ackPolicy: All Delivered again (3 times in 16 s)
maxDeliver: 2 without durableName (ephemeral consumer) Delivered 2 times
Existing durable consumer started again with maxDeliver: 5 instead of 3 Integration starts; WARN [SUB-90016] Existing consumer cannot be modified. Changed fields: [maxDeliver]; a new message is not received
The same after nats consumer rm FAILmd fmd-durable New consumer reads the stream from the beginning; bad message delivered 5 times
durableName: a.b WARN Durable must be in the printable ASCII range and cannot include '*', '.', '>' ...: hence the pattern
durableName, ackPolicy, maxDeliver without jetstreamEnabled Ignored: plain NATS, message received, no stream or consumer created
Source jetstreamAsync: false Receives as with true
Sink jetstreamEnabled, stream SINK takes in the subject; with jetstreamName: SINK, with jetstreamName: NOPE, and without jetstreamEnabled All 3×5 messages stored in SINK (read back with nats stream get, with the Content-Type header); no stream NOPE created
Sink jetstreamEnabled, no stream for the subject Each send fails: 503 No Responders Available For Request
Sink jetstreamEnabled on a server without -js Sent with no error; a nats sub subscriber received all 5
Server paused (docker pause): requestTimeout 500 / unset / 60000 / 500 with jetstreamAsync=false (tested, then left out) Fails after 0.5 s / 6–7.5 s / 7 s / 2 s
Partial-page examples (durable source, JetStream sink), address replaced Source created ORDERS and received the sink's messages; nats consumer rm ORDERS order-processor-durable worked

./mvnw clean verify -DskipTests and ./mvnw clean verify from the root both pass. The generated kamelets-index.json and the library/camel-kamelets copies of the Kamelets were regenerated and are committed. The copies were made with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. No Citrus test was added.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…on after a restart (apache#3170)

nats-source gets three optional JetStream options, mapped with optional
placeholders, so that with them unset the endpoint is the same as before:

- durableName: a durable JetStream consumer. After a restart the source
  carries on with the first message that was not acknowledged, instead of
  reading the stream from the beginning, and copies of the integration with
  the same durableName share the messages. camel-nats names the consumer
  on the server <durableName>-durable. The pattern allows letters, digits,
  '-' and '_'; jnats refuses '.', '*', '>', '/' and '\'.
- ackPolicy (Explicit, All, None): how the messages are acknowledged, and so
  whether a message that failed is delivered again.
- maxDeliver: how many times at most a message is delivered.

requestTimeout is not added to nats-sink: the sink is InOnly, so it never
sends a request and waits for a reply. camel-nats uses requestTimeout only
as the wait for the JetStream publish acknowledgement with jetstreamAsync,
where jnats cancels the publish after its own 2 second request timeout
anyway, so the option can only shorten the wait, not lengthen it.

The descriptions of the existing JetStream options are corrected after
running them: nats-sink does not use jetstreamName (the message goes to the
stream that takes in the subject, whatever its name), nats-source does not
use jetstreamAsync, a stream also stores what the sink publishes without
jetstreamEnabled, and on a server without JetStream jetstreamEnabled is
ignored by the sink. The partial pages get a durable consumer section, an
ackPolicy table, a sink JetStream section and examples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit b4c66b7 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-nats-jetstream branch November 18, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant