Skip to content

chore: the Couchbase Kamelets work with a scope and a collection, and couchbase-sink can delete (#3170) - #3220

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-couchbase
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-couchbase

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: couchbase-sink and couchbase-source work with a scope and a collection, couchbase-sink can delete documents and takes durability and timeout settings, and couchbase-source can delete each document once it has been sent.

The problem

Both Kamelets only worked on the default collection of the bucket. The sink could only store documents, with the Couchbase defaults for durability and timeouts, and the source sent the same documents again on every poll.

The options

couchbase-sink:

Option Template mapping What it does
scope scope: "RAW({{?scope}})" The scope that holds collection. Used only together with collection.
collection collection: "RAW({{?collection}})" The collection to store the documents in, or delete them from.
key key: "RAW({{?key}})" The document ID for a message without the CamelCouchbaseId header, when autoStartId is false.
operation operation: "{{?operation}}" CCB_PUT stores the body (as before), CCB_DEL deletes the document.
persistTo persistTo: "{{?persistTo}}" On how many nodes a stored document must be on disk, 0 to 4.
replicateTo replicateTo: "{{?replicateTo}}" To how many replicas a stored document must be copied, 0 to 3.
connectTimeout connectTimeout: "{{?connectTimeout}}" How long one attempt to connect to a node may take, in ms.
retryAttempts producerRetryAttempts: "{{?retryAttempts}}" With retryPause, how long a write or delete may take.
retryPause producerRetryPause: "{{?retryPause}}" The pause between two tries of a write or delete, in ms.
operationTimeout queryTimeout: "{{?operationTimeout}}" How long a write or delete may take, when that is longer than retryAttempts × retryPause.

couchbase-source:

Option Template mapping What it does
scope scope: "RAW({{?scope}})" The scope the query runs in.
collection collection: "RAW({{?collection}})" The collection to read when there is no statement, and where fullDocument and delete read and delete the documents.
processedStrategy consumerProcessedStrategy: "{{?processedStrategy}}" none (as before) or delete: each document is deleted once its message has been processed.
  • Defaults: all options are optional and have no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the endpoint URIs are the same as on main.
  • RAW() and pattern:: scope, collection and key are passed in RAW(), so a + or a % is kept. A collection named p%2B-x_1 and a key latest status+1% were written as named. scope and collection have the pattern of Couchbase names, ^[A-Za-z0-9_%-]{1,251}$, and key has ^[^\r\n]{1,250}$.
  • Names: timeout is reserved by the kamelet: component. The Kamelet uses operationTimeout, retryAttempts and retryPause instead of the component names queryTimeout, producerRetryAttempts and producerRetryPause. For the sink, queryTimeout is not a query timeout: the component uses max(producerRetryAttempts × producerRetryPause, queryTimeout) as the timeout of a write or delete, and retries every producerRetryPause ms. The descriptions say how the three options work together.
  • operation: the enum is CCB_PUT and CCB_DEL. CCB_GET is left out because it puts an object of the Couchbase SDK in the body, which is of no use in a sink.
  • Things the descriptions now say, found in the tests:
    • scope without collection is ignored, and the default collection is used.
    • A collection that does not exist makes each message wait until the write timeout, 10 s by default, and fail with AmbiguousTimeoutException.
    • CCB_DEL with autoStartId true, the default, deletes the documents 1, 2, and so on.
    • Deleting a document that does not exist fails with Document with the given id not found.
    • persistTo and replicateTo above what the bucket has fail the message with Not enough replicas configured on the bucket, but the document is stored first.
    • persistTo and replicateTo are not used by CCB_DEL.
    • With fullDocument or delete, the source reads or deletes the documents in collection, even when a statement reads another collection.
  • Left out of the source: connectTimeout and queryTimeout. The component applies queryTimeout to SQL++ queries only when it is not 2500 ms. Otherwise the SDK uses its own 75 s, so the option would behave in a surprising way. rangeStartKey, rangeEndKey, consumerRetryPause and the scheduler options were also left out; delay and limit already cover the polling.
  • Partial pages: the sink page has examples for a scope and a collection and for deleting, and a section on durability and timeouts. The source page has an example for a scope, a collection and processedStrategy: delete.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT against couchbase/server:community-7.6.2 in Docker. The setup was a single node with data, index and query services and the bucket shop with 0 replicas. It had the scope sales with the collections orders, inbox and p%2B-x_1, all with primary indexes, set up with couchbase-cli and SQL++. The results were checked with SQL++ queries on the server.

Case Result
Options unset, both Kamelets, main and this branch camel get endpoint shows identical couchbase: URIs
Sink, scope: sales, collection: orders, autoStartId: false, header CamelCouchbaseId order-1..3 order-1, order-2, order-3 in shop.sales.orders, default collection empty
Sink, operation: CCB_DEL, header order-2 order-2 removed; order-1 and order-3 kept. Body afterwards MutationResult{cas=..., ...}
Sink, CCB_DEL of a document that does not exist Fails: DocumentNotFoundException: Document with the given id not found
Sink, CCB_DEL with autoStartId unset (true) Deletes document 1 of the default collection; 2 kept
Sink, operation: CCB_PUT Stored as without the option
Sink, no header, key: "latest status+1%" Document latest status+1% stored
Sink, header A-101 and key: fallback (header from jq: .orderId, as in the partial) Stored as A-101; the header wins
Sink, no header, no key, autoStartId: false Fails: No document id: set the CamelCouchbaseId header or the key option of the endpoint
Sink, scope: sales only Stored in the default collection
Sink, collection: orders only (no such collection in _default) Fails after 10 s: AmbiguousTimeoutException, retry reason COLLECTION_NOT_FOUND
Sink, collection: "p%2B-x_1" Stored in that collection; the name is kept by RAW()
Sink, persistTo: 1 Stored, message OK
Sink, persistTo: 2 / replicateTo: 1, bucket without replicas Fails: ReplicaNotConfiguredException: Not enough replicas configured on the bucket; the document is stored
Sink, CCB_DEL with persistTo: 2, replicateTo: 1 Deleted, message OK (not used by delete)
Sink, missing collection, options unset Fails after 10.0 s, timeoutMs 10000, retried 2 times
The same, retryAttempts: 1, retryPause: 1000 Fails after 2.5 s, timeoutMs 2500, retried 3 times
The same plus operationTimeout: 4000 Fails after 4.0 s, timeoutMs 4000, retried 4 times
The same, retryAttempts: 3, retryPause: 2000 Fails after 6.0 s, timeoutMs 6000, retried 3 times
Sink to an unreachable host, connectTimeout unset / 2000, operationTimeout: 25000 Connect attempts time out after 30000 ms / 2000 ms (EndpointConnectionFailedEvent); both messages fail after 25 s
Source, scope: sales, collection: orders, no statement Sends each document of sales.orders as {"orders":{...},"__id":"..."}
Source, scope: sales, statement: SELECT ... FROM orders o WHERE ... The statement runs in sales; matching rows sent
Source, scope, collection, fullDocument: true (with and without a statement) Body is the document read from sales.orders
Source, scope: sales, statement on inbox, fullDocument: true, no collection Poll fails with DocumentNotFoundException: the document is read from the default collection
Source, collection: inbox, processedStrategy: delete, a route that fails for in-bad in-1, in-2 sent once and deleted; in-bad sent on every poll and kept

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root. Before building, the library copies of the two Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json. The Citrus test couchbase-source-route was not run; it uses none of the new options.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

… couchbase-sink can delete (apache#3170)

couchbase-sink and couchbase-source take optional scope and collection,
so documents are written to and read from a collection other than the
default one of the bucket.

couchbase-sink also takes:
- key: the document ID for a message without the CamelCouchbaseId
  header, when autoStartId is false;
- operation: CCB_PUT (store, as before) or CCB_DEL (delete);
- persistTo and replicateTo, the durability of a write;
- connectTimeout, and retryAttempts, retryPause and operationTimeout,
  mapped to the endpoint options producerRetryAttempts,
  producerRetryPause and queryTimeout. A write or delete fails after
  retryAttempts x retryPause ms, or operationTimeout when that is longer.

couchbase-source also takes processedStrategy: with delete, each document
is deleted once its message has been processed, so it is sent only once.

All new options are optional with no default; with them unset the
endpoint URIs are the same as before. scope, collection and key are
passed in RAW() and have a pattern.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus force-pushed the ci-issue-3170-couchbase branch from 0eea618 to 355bf21 Compare November 18, 2026 20:01
@davsclaus
davsclaus merged commit 51acba2 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-couchbase branch November 18, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant