Skip to content

chore: the Snowflake Kamelets can choose the warehouse, schema and role of the session (#3170) - #3237

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-snowflake-mongo-changes
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-snowflake-mongo-changes

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: snowflake-source and snowflake-sink can choose the warehouse, the schema and the role of the session. mongodb-changes-stream-source gets no new option: it cannot resume after a restart without a bean. Its docs now say that.

The options

snowflake-source and snowflake-sink. They are set on the data source bean dsBean, next to databaseName. The bean is SnowflakeDataSourceFactory.createDataSource(), as before #3208 and after it.

Option Template mapping (bean dsBean) What it does
warehouse warehouse: "{{?warehouse}}" The virtual warehouse that runs the queries or statements.
schema schema: "{{?schema}}" The schema of databaseName that holds the tables.
role role: "{{?role}}" The role to run as, which must be granted to the user.
  • Bean setters, not properties[...]: snowflake-jdbc 4.3.4 has the setters setWarehouse, setSchema and setRole on SnowflakeBasicDataSource. Each one puts its SFSessionProperty key into the same properties map that chore: the Snowflake Kamelets can log in with a key pair (#3170) #3208 fills with the key options. So these map like databaseName does. chore: the Snowflake Kamelets can log in with a key pair (#3170) #3208 used properties[...] only because the key setters take two arguments.
  • What reaches Snowflake: the driver sends them as the warehouse, schemaName and roleName query parameters of POST /session/v1/login-request, next to databaseName.
  • Optional, no default: with {{?x}} unset, the setter is not called. The login request and the endpoint URI are then identical to main's; see the table below. Snowflake then uses the user's DEFAULT_WAREHOUSE, DEFAULT_NAMESPACE and DEFAULT_ROLE.
  • No pattern:: none of them goes into a URI. The bean gets them as they are, and the driver encodes them as query parameters.
  • Names: warehouse, schema and role are the driver's names (the JDBC connection parameters, and the SnowflakeDataSource setters). None of them is an option name that the kamelet: component takes.
  • Descriptions:
    • "Good to know" of both Kamelets names the three options and says that the user's defaults apply when they are not set.
    • Each option says what happens when it is not set.
    • Both partial pages have a "Warehouse, Schema and Role" section with an example, and a "Connections" section (see below).
    • The statements about Snowflake's defaults and about a statement failing with no warehouse come from Snowflake's JDBC connection parameters and user documentation. They were not run against Snowflake.

Left out

  • Pool settings (maxTotal, maxIdle, maxWaitMillis), as chore: the SQL Kamelets can page and mark the rows of a poll, write a JSON array of rows, and size the connection pool (#3170) #3215 adds them to the other SQL Kamelets. There, the bean is a commons-dbcp2 BasicDataSource. Here it is the driver's SnowflakeBasicDataSource, which does not pool: each getConnection() logs in and opens a new Snowflake session, and closing the connection ends it. In the tests below, the sink logged in once per message (2 messages, 2 logins) and the source once per poll (4 polls, 4 logins). The "Good to know" paragraphs and the partial pages now say so.

    • A pool would mean a different bean: a commons-dbcp2 BasicDataSource with the Snowflake driver class, with the key options passed through connectionProperties. Long-lived pooled sessions also raise Snowflake-specific questions, such as session expiry and keep-alive, and that needs its own PR and its own tests. commons-dbcp2 is still in spec.dependencies from before Fix #3054: move snowflake-jdbc to 4.3.4 and onto the 4.x DataSource factory #3055, but nothing uses it.
  • Resuming mongodb-changes-stream-source after a restart. camel-mongodb 4.24.0-SNAPSHOT (the same in 4.23.0) has three options that look related. None of them works in this Kamelet without a bean:

    • changeStreamTokenRepository: this is what resumes. After each exchange, the consumer saves the change's resume token in a StateRepository, keyed by route ID and collection, and starts after it on the next start. It takes an object, such as a FileStateRepository. A Kamelet can declare a bean, but it creates the bean every time, and the endpoint must name it every time. An optional placeholder drops a URI parameter only when the value starts with the placeholder, so #class:...FileStateRepository#fileStateRepository('{{?file}}') is not dropped when unset. The option cannot be made optional with the endpoint unchanged from main.
    • changeStreamToken: a fixed token to start after. It is read on every start, and it wins over the repository, so each restart sends the same changes again (see the table). The Kamelet does not send the tokens as text either: the only header that carries one, CamelOffset, holds a Java object. So a user has no token to set.
    • persistentTailTracking (with persistentId): only the tailable cursor consumer (mongodb-source) uses it. With consumerType=changeStreams, it does nothing (see the table).

    Instead, the "Good to know" paragraph and a new "Restarts" section of the partial page say that the changes made while the Kamelet is stopped are never sent. They also say that the collection must exist when the Kamelet starts; otherwise it fails with Collection orders does not exist on the database and createCollection is false, which happened in a test. Supporting resume would need a Camel change first, such as a camel-mongodb option that takes a file path or that stores the tokens in a MongoDB collection.

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT, camel run ... --local-kamelet-dir.

Snowflake. There is no local Snowflake, and nothing was run against a real Snowflake account. Two checks instead:

  • A stand-in, as in chore: the Snowflake Kamelets can log in with a key pair (#3170) #3208: a small Python HTTP server on localhost:18099 that saves each POST /session/v1/login-request (URL and gunzipped JSON) and answers with Snowflake's login-failure JSON. instanceUrl was localhost:18099/?ssl=off&account=MYACCOUNT; these are driver connect-string parameters, used for the test only. To compare requests, the request IDs and the driver's CLIENT_ENVIRONMENT block (timings, JVM details) were left out.
  • A login against Snowflake's real front end, for an account that does not exist (w6dummyorg-noaccount.snowflakecomputing.com, with the driver's tracing=ALL). The driver's log shows the request it sent before the 404.
Case Kamelet Result
Options unset, main and this branch sink, source Login requests identical: login-request?databaseName=SHOP, PASSWORD sent. camel get endpoint --json shows the same URIs on both: local-sql-snowflake-sink-1://INSERT...?dataSource=%23bean%3AdsBean-2 and local-sql-snowflake-source-1://SELECT...?dataSource=%23bean%3AdsBean-2&delay=2000
schema: SALES, warehouse: COMPUTE_WH, role: LOADER / READER sink, source login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER (sink) and ...&roleName=READER (source), on every login. The login JSON is otherwise unchanged, and the endpoint URI is the same as when they are unset
The three options with privateKeyFile, no databaseName sink login-request?schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER, AUTHENTICATOR=SNOWFLAKE_JWT, no PASSWORD: the options and the key share the bean's properties map without clashing
The three options, real Snowflake front end, account that does not exist sink Driver log: POST https://w6dummyorg-noaccount.snowflakecomputing.com:443/session/v1/login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER&requestId=..., answered HTTP Response code: 404. The message fails with CannotGetJdbcConnectionException. The password is shown as xxxxxx in the Camel startup log, and nowhere else
Connections per message sink 2 messages, 2 login requests
Connections per poll source delay: 2000, about 8 s: 4 polls, 4 login requests

MongoDB. Tested against mongo:8 as a one-member replica set (--replSet rs0), with collection shop.orders. In each case: start the route and insert A; stop the route and insert B; start the route again and insert C.

Case Run 1 sent Run 2 sent
mongodb-changes-stream-source (this branch; the template is unchanged from main) A C: B is lost
Collection does not exist yet Fails to start: Collection orders does not exist on the database and createCollection is false
Plain mongodb: route, consumerType=changeStreams&persistentTailTracking=true&persistentId=w6snow A C: B is lost, the option does nothing
Plain route, changeStreamTokenRepository=#class:...FileStateRepository#fileStateRepository('.../tokens.dat') A B, C: resumed. The file holds cs-filerepo/orders={"_data": "826AFE10..."}
Plain route, changeStreamToken set to the token of an earlier change A A again, B, C: it starts after the same token each time

Build

The library copies were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. Then ./mvnw clean verify -DskipTests and ./mvnw clean verify passed from the root. The build regenerated kamelets-index.json. KameletsCatalogTest needed no change.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…le of the session (apache#3170)

snowflake-source and snowflake-sink take three new optional options,
warehouse, schema and role. They are set on the Snowflake data source bean
(setWarehouse, setSchema, setRole of snowflake-jdbc 4.3.4), and the driver
sends them with each login as the warehouse, schemaName and roleName
parameters of the login request. When they are unset, the login request is
the same as on main, and the defaults of the Snowflake user apply.

The pool settings of the other SQL Kamelets (maxTotal, maxIdle,
maxWaitMillis) are left out: the Snowflake bean is the driver's
SnowflakeBasicDataSource, not a commons-dbcp2 BasicDataSource, and does not
pool. Each message of the sink and each poll of the source logs in again;
the descriptions now say so.

mongodb-changes-stream-source gets no new option. camel-mongodb resumes a
change stream after a restart only through changeStreamTokenRepository, a
StateRepository object, which a Kamelet cannot create only when asked.
persistentTailTracking is used only by the tailable cursor consumer, and
changeStreamToken is a fixed start that replays the same changes on every
start. The description and the partial page now say that changes made while
the Kamelet is stopped are never sent, and that the collection must exist
when it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit f9d7186 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-snowflake-mongo-changes branch November 18, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant