Repository navigation
chore: the Couchbase Kamelets work with a scope and a collection, and couchbase-sink can delete (#3170) - #3220
Merged
Conversation
… couchbase-sink can delete (apache#3170) couchbase-sink and couchbase-source take optional scope and collection, so documents are written to and read from a collection other than the default one of the bucket. couchbase-sink also takes: - key: the document ID for a message without the CamelCouchbaseId header, when autoStartId is false; - operation: CCB_PUT (store, as before) or CCB_DEL (delete); - persistTo and replicateTo, the durability of a write; - connectTimeout, and retryAttempts, retryPause and operationTimeout, mapped to the endpoint options producerRetryAttempts, producerRetryPause and queryTimeout. A write or delete fails after retryAttempts x retryPause ms, or operationTimeout when that is longer. couchbase-source also takes processedStrategy: with delete, each document is deleted once its message has been processed, so it is sent only once. All new options are optional with no default; with them unset the endpoint URIs are the same as before. scope, collection and key are passed in RAW() and have a pattern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
davsclaus
force-pushed
the
ci-issue-3170-couchbase
branch
from
November 18, 2026 20:01
0eea618 to
355bf21
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170:
couchbase-sinkandcouchbase-sourcework with a scope and a collection,couchbase-sinkcan delete documents and takes durability and timeout settings, andcouchbase-sourcecan delete each document once it has been sent.The problem
Both Kamelets only worked on the default collection of the bucket. The sink could only store documents, with the Couchbase defaults for durability and timeouts, and the source sent the same documents again on every poll.
The options
couchbase-sink:scopescope: "RAW({{?scope}})"collection. Used only together withcollection.collectioncollection: "RAW({{?collection}})"keykey: "RAW({{?key}})"CamelCouchbaseIdheader, whenautoStartIdis false.operationoperation: "{{?operation}}"CCB_PUTstores the body (as before),CCB_DELdeletes the document.persistTopersistTo: "{{?persistTo}}"replicateToreplicateTo: "{{?replicateTo}}"connectTimeoutconnectTimeout: "{{?connectTimeout}}"retryAttemptsproducerRetryAttempts: "{{?retryAttempts}}"retryPause, how long a write or delete may take.retryPauseproducerRetryPause: "{{?retryPause}}"operationTimeoutqueryTimeout: "{{?operationTimeout}}"retryAttempts×retryPause.couchbase-source:scopescope: "RAW({{?scope}})"collectioncollection: "RAW({{?collection}})"statement, and wherefullDocumentanddeleteread and delete the documents.processedStrategyconsumerProcessedStrategy: "{{?processedStrategy}}"none(as before) ordelete: each document is deleted once its message has been processed.default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the endpoint URIs are the same as onmain.RAW()andpattern::scope,collectionandkeyare passed inRAW(), so a+or a%is kept. A collection namedp%2B-x_1and a keylatest status+1%were written as named.scopeandcollectionhave the pattern of Couchbase names,^[A-Za-z0-9_%-]{1,251}$, andkeyhas^[^\r\n]{1,250}$.timeoutis reserved by thekamelet:component. The Kamelet usesoperationTimeout,retryAttemptsandretryPauseinstead of the component namesqueryTimeout,producerRetryAttemptsandproducerRetryPause. For the sink,queryTimeoutis not a query timeout: the component uses max(producerRetryAttempts×producerRetryPause,queryTimeout) as the timeout of a write or delete, and retries everyproducerRetryPausems. The descriptions say how the three options work together.operation: the enum isCCB_PUTandCCB_DEL.CCB_GETis left out because it puts an object of the Couchbase SDK in the body, which is of no use in a sink.scopewithoutcollectionis ignored, and the default collection is used.AmbiguousTimeoutException.CCB_DELwithautoStartIdtrue, the default, deletes the documents1,2, and so on.Document with the given id not found.persistToandreplicateToabove what the bucket has fail the message withNot enough replicas configured on the bucket, but the document is stored first.persistToandreplicateToare not used byCCB_DEL.fullDocumentordelete, the source reads or deletes the documents incollection, even when astatementreads another collection.connectTimeoutandqueryTimeout. The component appliesqueryTimeoutto SQL++ queries only when it is not 2500 ms. Otherwise the SDK uses its own 75 s, so the option would behave in a surprising way.rangeStartKey,rangeEndKey,consumerRetryPauseand the scheduler options were also left out;delayandlimitalready cover the polling.processedStrategy: delete.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against
couchbase/server:community-7.6.2in Docker. The setup was a single node with data, index and query services and the bucketshopwith 0 replicas. It had the scopesaleswith the collectionsorders,inboxandp%2B-x_1, all with primary indexes, set up withcouchbase-cliand SQL++. The results were checked with SQL++ queries on the server.mainand this branchcamel get endpointshows identicalcouchbase:URIsscope: sales,collection: orders,autoStartId: false, headerCamelCouchbaseIdorder-1..3order-1,order-2,order-3inshop.sales.orders, default collection emptyoperation: CCB_DEL, headerorder-2order-2removed;order-1andorder-3kept. Body afterwardsMutationResult{cas=..., ...}CCB_DELof a document that does not existDocumentNotFoundException: Document with the given id not foundCCB_DELwithautoStartIdunset (true)1of the default collection;2keptoperation: CCB_PUTkey: "latest status+1%"latest status+1%storedA-101andkey: fallback(header fromjq: .orderId, as in the partial)A-101; the header winskey,autoStartId: falseNo document id: set the CamelCouchbaseId header or the key option of the endpointscope: salesonlycollection: ordersonly (no such collection in_default)AmbiguousTimeoutException, retry reasonCOLLECTION_NOT_FOUNDcollection: "p%2B-x_1"RAW()persistTo: 1persistTo: 2/replicateTo: 1, bucket without replicasReplicaNotConfiguredException: Not enough replicas configured on the bucket; the document is storedCCB_DELwithpersistTo: 2,replicateTo: 1timeoutMs10000, retried 2 timesretryAttempts: 1,retryPause: 1000timeoutMs2500, retried 3 timesoperationTimeout: 4000timeoutMs4000, retried 4 timesretryAttempts: 3,retryPause: 2000timeoutMs6000, retried 3 timesconnectTimeoutunset /2000,operationTimeout: 25000EndpointConnectionFailedEvent); both messages fail after 25 sscope: sales,collection: orders, no statementsales.ordersas{"orders":{...},"__id":"..."}scope: sales,statement: SELECT ... FROM orders o WHERE ...sales; matching rows sentscope,collection,fullDocument: true(with and without a statement)sales.ordersscope: sales, statement oninbox,fullDocument: true, nocollectionDocumentNotFoundException: the document is read from the default collectioncollection: inbox,processedStrategy: delete, a route that fails forin-badin-1,in-2sent once and deleted;in-badsent on every poll and kept./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root. Before building, the library copies of the two Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json. The Citrus testcouchbase-source-routewas not run; it uses none of the new options.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code