Repository navigation
chore: nats-source can use a durable JetStream consumer that carries on after a restart (#3170) - #3236
Merged
Conversation
…on after a restart (apache#3170) nats-source gets three optional JetStream options, mapped with optional placeholders, so that with them unset the endpoint is the same as before: - durableName: a durable JetStream consumer. After a restart the source carries on with the first message that was not acknowledged, instead of reading the stream from the beginning, and copies of the integration with the same durableName share the messages. camel-nats names the consumer on the server <durableName>-durable. The pattern allows letters, digits, '-' and '_'; jnats refuses '.', '*', '>', '/' and '\'. - ackPolicy (Explicit, All, None): how the messages are acknowledged, and so whether a message that failed is delivered again. - maxDeliver: how many times at most a message is delivered. requestTimeout is not added to nats-sink: the sink is InOnly, so it never sends a request and waits for a reply. camel-nats uses requestTimeout only as the wait for the JetStream publish acknowledgement with jetstreamAsync, where jnats cancels the publish after its own 2 second request timeout anyway, so the option can only shorten the wait, not lengthen it. The descriptions of the existing JetStream options are corrected after running them: nats-sink does not use jetstreamName (the message goes to the stream that takes in the subject, whatever its name), nats-source does not use jetstreamAsync, a stream also stores what the sink publishes without jetstreamEnabled, and on a server without JetStream jetstreamEnabled is ignored by the sink. The partial pages get a durable consumer section, an ackPolicy table, a sink JetStream section and examples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170:
nats-sourcecan use a durable JetStream consumer, which carries on after a restart where it left off instead of reading the stream again from the beginning, and which several copies of an integration can share. It can also choose how messages are acknowledged and how many times a failing message is delivered. The descriptions of the existing JetStream options of both Kamelets are corrected after running them.The options
durableName(pattern: ^[A-Za-z0-9_-]+$)nats-sourcedurableName: "{{?durableName}}"ackPolicy(enum: Explicit, All, None)nats-sourceackPolicy: "{{?ackPolicy}}"maxDeliver(integer)nats-sourcemaxDeliver: "{{?maxDeliver}}"durableName. camel-nats creates a durable pull consumer. The server keeps it and remembers which messages were acknowledged, so after a restart the source carries on with the first unacknowledged message. Copies of the integration with the samedurableNameshare the messages, whichqueueNamecannot do with JetStream (chore: the NATS Kamelets can use TLS and a credentials file, and nats-source a queue group (#3170) #3205). camel-nats names the consumer<durableName>-durableon the server, and the description says so, because that is the namenats consumer rmneeds. jnats refuses.,*,>,/and\in the name, so thepatternallows only letters, digits,-and_.ackPolicyandmaxDeliverare plain endpoint options of camel-nats, so they need noConsumerConfigurationbean. They apply to ephemeral consumers too. A durable consumer keeps the values it was created with: if they change later, jnats fails withExisting consumer cannot be modified, logs it, and the source receives nothing until the consumer is deleted. The description says this.requestTimeoutis not added tonats-sink. The sink isInOnlysince chore: messaging sinks send without waiting for a reply; nats-source and mqtt source fixes #3156, so it never makes a request and waits for a reply. The only other place camel-nats usesrequestTimeoutis the wait for the JetStream publish acknowledgement withjetstreamAsync=true. jnats already cancels that publish after its own request timeout, which is 2 seconds (theconnectionTimeout) and is applied at the next cleanup run, every 5 seconds. So the option could only shorten the wait, never lengthen it. With the server paused,requestTimeout: 500failed after 0.5 s, the default (20000) after 6 to 7.5 s,60000after 7 s, and withjetstreamAsync=falsethe option is ignored (2 s). For a longer wait, camel-nats would need to passrequestTimeoutto the JetStream options of jnats. That would be a Camel change.nats-sinkdoes not usejetstreamName. The message is stored by whichever stream takes in the subject, whatever its name. ThejetstreamNamevalue is used only in dead code (jsm.jetStream()is never null).jetstreamEnabled, the sink waits until the stream confirms that it stored the message. It fails with503 No Responders Available For Requestwhen no stream takes in the subject.jetstreamEnabled.jetstreamEnabledand publishes as plain NATS.nats-sourcedoes not usejetstreamAsync.jetstreamAsynctext said "without blocking a thread". In fact it waits for the acknowledgement either way.durableName. The source partial page has a durable consumer section, anackPolicytable and an example. The sink partial page has a JetStream section and an example. Both examples were run against the test server.default, so with them unset the endpoint URI is exactly as onmain.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT (
camel run --local-kamelet-dir=kamelets) againstnats:2(nats-server 2.15.3) started with-js, in Docker.natsio/nats-box(natsCLI) published, read streams and inspected consumers. The routes logged each body and theCamelNatsDeliveryCounterheader. A route failed on purpose withthrowExceptionfor bodies that start withbad.mainand from this PR (camel get endpoint --json): plain source, plain sink, JetStream source and sinknats://ep.js?jetstreamAsync=true&jetstreamEnabled=true&jetstreamName=EPJS&servers=...nats://dur.x?ackPolicy=All&durableName=allopts&...&maxDeliver=4&...;nats consumer infoshowsack_policy: all,max_deliver: 4durableName: proc: 5 messages, stop, 5 more published while it is stopped, start againproc-durableon the serverdurableName(as onmain)durableName, slow route,kill -9while 10 messages are in flightdurableName: workers, 40 messagesExplicit, nomaxDeliver)maxDeliver: 3ackPolicy: NoneackPolicy: AllackPolicy: AllmaxDeliver: 2withoutdurableName(ephemeral consumer)maxDeliver: 5instead of 3[SUB-90016] Existing consumer cannot be modified. Changed fields: [maxDeliver]; a new message is not receivednats consumer rm FAILmd fmd-durabledurableName: a.bDurable must be in the printable ASCII range and cannot include '*', '.', '>' ...: hence thepatterndurableName,ackPolicy,maxDeliverwithoutjetstreamEnabledjetstreamAsync: falsetruejetstreamEnabled, streamSINKtakes in the subject; withjetstreamName: SINK, withjetstreamName: NOPE, and withoutjetstreamEnabledSINK(read back withnats stream get, with theContent-Typeheader); no streamNOPEcreatedjetstreamEnabled, no stream for the subject503 No Responders Available For RequestjetstreamEnabledon a server without-jsnats subsubscriber received all 5docker pause):requestTimeout500 / unset / 60000 / 500 withjetstreamAsync=false(tested, then left out)ORDERSand received the sink's messages;nats consumer rm ORDERS order-processor-durableworked./mvnw clean verify -DskipTestsand./mvnw clean verifyfrom the root both pass. The generatedkamelets-index.jsonand thelibrary/camel-kameletscopies of the Kamelets were regenerated and are committed. The copies were made with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. No Citrus test was added.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code