Repository navigation
chore: the Snowflake Kamelets can choose the warehouse, schema and role of the session (#3170) - #3237
Merged
davsclaus merged 1 commit intoNov 18, 2026
Conversation
…le of the session (apache#3170) snowflake-source and snowflake-sink take three new optional options, warehouse, schema and role. They are set on the Snowflake data source bean (setWarehouse, setSchema, setRole of snowflake-jdbc 4.3.4), and the driver sends them with each login as the warehouse, schemaName and roleName parameters of the login request. When they are unset, the login request is the same as on main, and the defaults of the Snowflake user apply. The pool settings of the other SQL Kamelets (maxTotal, maxIdle, maxWaitMillis) are left out: the Snowflake bean is the driver's SnowflakeBasicDataSource, not a commons-dbcp2 BasicDataSource, and does not pool. Each message of the sink and each poll of the source logs in again; the descriptions now say so. mongodb-changes-stream-source gets no new option. camel-mongodb resumes a change stream after a restart only through changeStreamTokenRepository, a StateRepository object, which a Kamelet cannot create only when asked. persistentTailTracking is used only by the tailable cursor consumer, and changeStreamToken is a fixed start that replays the same changes on every start. The description and the partial page now say that changes made while the Kamelet is stopped are never sent, and that the collection must exist when it starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170:
snowflake-sourceandsnowflake-sinkcan choose the warehouse, the schema and the role of the session.mongodb-changes-stream-sourcegets no new option: it cannot resume after a restart without a bean. Its docs now say that.The options
snowflake-sourceandsnowflake-sink. They are set on the data source beandsBean, next todatabaseName. The bean isSnowflakeDataSourceFactory.createDataSource(), as before #3208 and after it.dsBean)warehousewarehouse: "{{?warehouse}}"schemaschema: "{{?schema}}"databaseNamethat holds the tables.rolerole: "{{?role}}"properties[...]: snowflake-jdbc 4.3.4 has the setterssetWarehouse,setSchemaandsetRoleonSnowflakeBasicDataSource. Each one puts itsSFSessionPropertykey into the samepropertiesmap that chore: the Snowflake Kamelets can log in with a key pair (#3170) #3208 fills with the key options. So these map likedatabaseNamedoes. chore: the Snowflake Kamelets can log in with a key pair (#3170) #3208 usedproperties[...]only because the key setters take two arguments.warehouse,schemaNameandroleNamequery parameters ofPOST /session/v1/login-request, next todatabaseName.default: with{{?x}}unset, the setter is not called. The login request and the endpoint URI are then identical tomain's; see the table below. Snowflake then uses the user'sDEFAULT_WAREHOUSE,DEFAULT_NAMESPACEandDEFAULT_ROLE.pattern:: none of them goes into a URI. The bean gets them as they are, and the driver encodes them as query parameters.warehouse,schemaandroleare the driver's names (the JDBC connection parameters, and theSnowflakeDataSourcesetters). None of them is an option name that thekamelet:component takes.Left out
Pool settings (
maxTotal,maxIdle,maxWaitMillis), as chore: the SQL Kamelets can page and mark the rows of a poll, write a JSON array of rows, and size the connection pool (#3170) #3215 adds them to the other SQL Kamelets. There, the bean is a commons-dbcp2BasicDataSource. Here it is the driver'sSnowflakeBasicDataSource, which does not pool: eachgetConnection()logs in and opens a new Snowflake session, and closing the connection ends it. In the tests below, the sink logged in once per message (2 messages, 2 logins) and the source once per poll (4 polls, 4 logins). The "Good to know" paragraphs and the partial pages now say so.BasicDataSourcewith the Snowflake driver class, with the key options passed throughconnectionProperties. Long-lived pooled sessions also raise Snowflake-specific questions, such as session expiry and keep-alive, and that needs its own PR and its own tests.commons-dbcp2is still inspec.dependenciesfrom before Fix #3054: move snowflake-jdbc to 4.3.4 and onto the 4.x DataSource factory #3055, but nothing uses it.Resuming
mongodb-changes-stream-sourceafter a restart. camel-mongodb 4.24.0-SNAPSHOT (the same in 4.23.0) has three options that look related. None of them works in this Kamelet without a bean:changeStreamTokenRepository: this is what resumes. After each exchange, the consumer saves the change's resume token in aStateRepository, keyed by route ID and collection, and starts after it on the next start. It takes an object, such as aFileStateRepository. A Kamelet can declare a bean, but it creates the bean every time, and the endpoint must name it every time. An optional placeholder drops a URI parameter only when the value starts with the placeholder, so#class:...FileStateRepository#fileStateRepository('{{?file}}')is not dropped when unset. The option cannot be made optional with the endpoint unchanged frommain.changeStreamToken: a fixed token to start after. It is read on every start, and it wins over the repository, so each restart sends the same changes again (see the table). The Kamelet does not send the tokens as text either: the only header that carries one,CamelOffset, holds a Java object. So a user has no token to set.persistentTailTracking(withpersistentId): only the tailable cursor consumer (mongodb-source) uses it. WithconsumerType=changeStreams, it does nothing (see the table).Instead, the "Good to know" paragraph and a new "Restarts" section of the partial page say that the changes made while the Kamelet is stopped are never sent. They also say that the collection must exist when the Kamelet starts; otherwise it fails with
Collection orders does not exist on the database and createCollection is false, which happened in a test. Supporting resume would need a Camel change first, such as a camel-mongodb option that takes a file path or that stores the tokens in a MongoDB collection.Verification
Tested with Camel JBang 4.24.0-SNAPSHOT,
camel run ... --local-kamelet-dir.Snowflake. There is no local Snowflake, and nothing was run against a real Snowflake account. Two checks instead:
localhost:18099that saves eachPOST /session/v1/login-request(URL and gunzipped JSON) and answers with Snowflake's login-failure JSON.instanceUrlwaslocalhost:18099/?ssl=off&account=MYACCOUNT; these are driver connect-string parameters, used for the test only. To compare requests, the request IDs and the driver'sCLIENT_ENVIRONMENTblock (timings, JVM details) were left out.w6dummyorg-noaccount.snowflakecomputing.com, with the driver'stracing=ALL). The driver's log shows the request it sent before the 404.mainand this branchlogin-request?databaseName=SHOP,PASSWORDsent.camel get endpoint --jsonshows the same URIs on both:local-sql-snowflake-sink-1://INSERT...?dataSource=%23bean%3AdsBean-2andlocal-sql-snowflake-source-1://SELECT...?dataSource=%23bean%3AdsBean-2&delay=2000schema: SALES,warehouse: COMPUTE_WH,role: LOADER/READERlogin-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER(sink) and...&roleName=READER(source), on every login. The login JSON is otherwise unchanged, and the endpoint URI is the same as when they are unsetprivateKeyFile, nodatabaseNamelogin-request?schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER,AUTHENTICATOR=SNOWFLAKE_JWT, noPASSWORD: the options and the key share the bean'spropertiesmap without clashingPOST https://w6dummyorg-noaccount.snowflakecomputing.com:443/session/v1/login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER&requestId=..., answeredHTTP Response code: 404. The message fails withCannotGetJdbcConnectionException. The password is shown asxxxxxxin the Camel startup log, and nowhere elsedelay: 2000, about 8 s: 4 polls, 4 login requestsMongoDB. Tested against
mongo:8as a one-member replica set (--replSet rs0), with collectionshop.orders. In each case: start the route and insertA; stop the route and insertB; start the route again and insertC.mongodb-changes-stream-source(this branch; the template is unchanged frommain)AC:Bis lostCollection orders does not exist on the database and createCollection is falsemongodb:route,consumerType=changeStreams&persistentTailTracking=true&persistentId=w6snowAC:Bis lost, the option does nothingchangeStreamTokenRepository=#class:...FileStateRepository#fileStateRepository('.../tokens.dat')AB,C: resumed. The file holdscs-filerepo/orders={"_data": "826AFE10..."}changeStreamTokenset to the token of an earlier changeAAagain,B,C: it starts after the same token each timeBuild
The library copies were refreshed with
./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. Then./mvnw clean verify -DskipTestsand./mvnw clean verifypassed from the root. The build regeneratedkamelets-index.json.KameletsCatalogTestneeded no change.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code