Repository navigation
chore: the SQL Kamelets can page and mark the rows of a poll, write a JSON array of rows, and size the connection pool (#3170) - #3215
Merged
Conversation
… JSON array of rows, and size the connection pool (apache#3170) The PostgreSQL, MySQL, MariaDB, SQL Server and Oracle Database Kamelets take new optional options. With them unset, the endpoint URIs are the same as before, apart from oracle-database-source (see below). Sources (postgresql, mysql, mariadb, sqlserver, oracle-database): - maxMessagesPerPoll: the most rows one poll sends. - onConsumeFailed: an SQL statement run for a row whose processing failed, such as marking it as failed. - onConsumeBatchComplete: an SQL statement run once after the rows of a poll are sent. - initialDelay: how long to wait before the first poll. - useIterator: false sends all the rows of a poll as one JSON array. - routeEmptyResultSet: with useIterator false, sends [] when no rows are found. oracle-database-source gets the delay option the other sources have, with the same default of 500 ms. Its endpoint now carries delay=500, which is the delay it had before. Sinks: batch accepts a JSON array of objects and writes one row for each, in one transaction. A single object still works. All ten: maxTotal, maxIdle and maxWaitMillis set the commons-dbcp2 connection pool. transacted is left out: without a Spring transaction manager it starts no transaction, and a failed row stops the source for good. Snowflake is left out, as apache#3208 changes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
davsclaus
force-pushed
the
ci-issue-3170-sql-options
branch
from
November 18, 2026 19:58
9bd9a3d to
79ef55e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170: the SQL sources can limit the rows of a poll, mark a failed row, run a statement after each poll, and send a whole poll as one JSON array. The SQL sinks can write a JSON array of rows from one message. All of them can size their connection pool.
Kamelets:
postgresql-*,mysql-*,mariadb-*,sqlserver-*andoracle-database-*.snowflake-*is left out, because #3208 changes it.The options
Sources:
maxMessagesPerPoll&maxMessagesPerPoll={{?maxMessagesPerPoll}}consumedQuerymarks the rows that were sent.onConsumeFailed&onConsumeFailed=RAW({{?onConsumeFailed}})consumedQuerywhen the next step fails for a row, such asUPDATE accounts SET failed = 1 WHERE id = :#id.onConsumeBatchComplete&onConsumeBatchComplete=RAW({{?onConsumeBatchComplete}})initialDelay&initialDelay={{?initialDelay}}useIterator&useIterator={{?useIterator}}falsesends all the rows of a poll as one message, a JSON array.routeEmptyResultSet&routeEmptyResultSet={{?routeEmptyResultSet}}useIterator: false, also sends[]from a poll that finds no rows.delay(oracle-database-sourceonly)&delay={{delay}},default: 500Sinks:
batch&batch={{?batch}}queryruns once for each object, in one transaction: if one row fails, none is written. A single object still works.CamelSqlUpdateCountholds the total.All ten, on the commons-dbcp2
BasicDataSourcebean, next toconnectionPropertiesfrom #3176:maxTotalmaxTotal: "{{?maxTotal}}"maxIdlemaxIdle: "{{?maxIdle}}"maxWaitMillismaxWaitMillis: "{{?maxWaitMillis}}"default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When unset, the optional placeholders drop them from the URI, and an empty bean property is not set. The endpoint URIs are then byte-identical tomain's.oracle-database-source. It getsdelaywithdefault: 500, like the other sources, so its URI now hasdelay=500. That was already its delay.sqlcomponent, so the Camel documentation still applies. The titles say what they do: "Failed Row Query", "Poll Complete Query", "One Message Per Row", "Send Empty Result".RAW(): the two SQL options are wrapped inRAW(), asconsumedQueryalready is.useIterator: false.onConsumeFailedandmaxMessagesPerPoll.batch, and its all-or-nothing transaction.maxTotalwarns that the source holds one connection while it sends the rows of a poll.consumedQuery,onConsumeFailedandonConsumeBatchCompleteeach need a second one.onConsumeBatchCompletewarns that a row added after the query ran is not in the poll, so the statement must not mark it.useIteratorsays thatconsumedQueryandonConsumeFailedcannot use:#columnin that mode.delay.Left out
transacted: without a Spring transaction manager bean and a transacted route, which a Kamelet cannot take, the consumer starts no transaction. A test withtransacted=trueon PostgreSQL showed:consumedQueryof earlier rows was committed.onConsumeFailedwas not run.outputType:SelectOneandStreamListdo not fit a Kamelet that sends JSON.useIteratorcovers "the whole result as one message".minIdle: commons-dbcp2 keeps it only when the evictor runs (timeBetweenEvictionRuns), which is off by default. It would add an option that does nothing on its own.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against PostgreSQL 17, MySQL 8.4, MariaDB 11, SQL Server 2022 (
mcr.microsoft.com/mssql/server:2022-latest, amd64 under emulation) and Oracle 23ai Free (gvenzl/oracle-free:23-slim-faststart, arm64). Each test reached the endpoint through the Kamelet, andcamel get endpointshowed the URI.mainand this branch:camel get endpoint, source and sinkdelay=500maxMessagesPerPoll: 3, 10 rowsCamelBatchSize)onConsumeFailed, the step throws for rows 2 and 5failed=1, the other 8processed=1onConsumeBatchCompleteinserting into a log tableinitialDelay: 4000delay: 1000useIterator: false, 3 rows[{"id":1,...},{"id":2,...},{"id":3,...}],Content-Type: application/jsonIDandUSERNAMErouteEmptyResultSet: true, then no rows[]on every later pollrouteEmptyResultSet, no rowsbatchunset, one objectbatchunset, JSON arrayCannot find key [username](as onmain)batch: true, array of 5CamelSqlUpdateCount=5batch: true, one objectbatch: true, row 2 of 3 too long for its columnbatch: true, row 2 of 2 lacks a fieldbatch: true,[]batch: false, one objectmaxTotal: 1,maxWaitMillis: 500, the same 4 messagesFailed to obtain JDBC Connection; 1 OK after 2 sPostgreSQL only:
pg_stat_activityduring the burst; 8 messages finish at 2 s, the other 2 at 4 smaxIdle: 1, 4 messages that finish one second apartpg_stat_activityshows 1 idle connection; withmaxIdleunset, 1, 2, 3, then 4 idlemaxIdle: 0maxTotal: 1,maxWaitMillis: 1000,consumedQueryconsumedQueryfails for each row withTimeout waiting for idle object; no row is markedmaxWaitMillisuseIterator: falsewithconsumedQuery ... :#idconsumedQueryfails on every poll withCannot find key [id]; the rows are sent againuseIterator: false,onConsumeBatchComplete: UPDATE ... WHERE processed = 0, a row inserted while the poll's message is processedtransacted=true(template copy, not in this PR), a row fails./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root. The first run caught lower-case "per" in two titles, which are now "Max Rows Per Poll" and "One Message Per Row". Before building, the library copies of the ten Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.KameletsCatalogTestpassed with no count changes.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code