From 9a28dfaeeb7949967e25ce5b1d89752d102bb693 Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Fri, 6 Nov 2026 09:51:58 +0200 Subject: [PATCH 1/7] Add the RDI 2.4.0 upgrade notes and draft release notes for Debezium 3.7 --- .../installation/upgrade.md | 72 ++++++++++++++++++- .../release-notes/rdi-2-4-0.md | 27 +++++++ 2 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index da2f726f99..cde4929a8a 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -17,7 +17,8 @@ weight: 30 --- > [!NOTE] -> Before you upgrade, review [Upgrading to RDI 2.2.0](#upgrading-to-rdi-220). If you upgrade +> Before you upgrade, review [Upgrading to RDI 2.4.0](#upgrading-to-rdi-240). If you upgrade +> from RDI 2.0.x, also review [Upgrading to RDI 2.2.0](#upgrading-to-rdi-220), and if you upgrade > from RDI 1.19.x or earlier, also review [Upgrading to RDI 2.0.0](#upgrading-to-rdi-200). > [!WARNING] @@ -181,6 +182,75 @@ your saved `rdi-values.yaml` for that version. This will restore your previous w > described in [Uninstall RDI](/content/integrate/redis-data-integration/installation/install-k8s.md#uninstall-rdi), > and then install the old version. +## Upgrading to RDI 2.4.0 + +RDI 2.4.0 upgrades the Debezium collector from Debezium 3.5 to Debezium 3.7. Check the following +changes before you upgrade from RDI 2.2.x. If you upgrade from RDI 2.0.x, also review +[Upgrading to RDI 2.2.0](#upgrading-to-rdi-220). + +### Replace the Oracle LogMiner batch and sleep properties + +Debezium 3.7 no longer sizes each LogMiner iteration by a range of SCNs. Instead, while the +collector catches up, each iteration reads `log.mining.log.count.min` (default `2`) logs per redo +thread, and up to `log.mining.log.count.growth.max` (default `4`) while a long-running transaction +holds back the mining. RDI 2.4.0 rejects the properties that Debezium removed. This example shows where to remove them from your configuration: + +```yaml +sources: + : + advanced: + source: + log.mining.batch.size.min: # Rejected since RDI 2.4.0 + log.mining.batch.size.max: # Rejected since RDI 2.4.0 + log.mining.batch.size.default: # Rejected since RDI 2.4.0 + log.mining.batch.size.increment: # Rejected since RDI 2.4.0 + log.mining.sleep.time.min.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.max.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.default.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.increment.ms: # Rejected since RDI 2.4.0 + log.mining.scn.gap.detection.gap.size.min: # Rejected since RDI 2.4.0 + log.mining.scn.gap.detection.time.interval.max.ms: # Rejected since RDI 2.4.0 + log.mining.continuous.mine: # Rejected since RDI 2.4.0 + log.mining.buffer.infinispan.cache.rollbacks: # Rejected since RDI 2.4.0 + log.mining.buffer.ehcache.rollbacks.config: # Rejected since RDI 2.4.0 +``` + +A collector that is already running ignores these properties after the upgrade, so it mines with +the default log counts until you redeploy. If you raised the batch size because the collector +could not keep up with the volume of your redo logs, raise `log.mining.log.count.min` instead. +A higher count clears a backlog in fewer iterations, but each iteration takes longer, so check the +collector lag after you redeploy. + +`log.mining.strategy: redo_log_catalog` still works, but Debezium has deprecated it. Use the +default `online_catalog` or `hybrid` instead. + +### Check MySQL and MariaDB sources + +RDI 2.4.0 rejects `snapshot.mode: never` for MySQL and MariaDB sources, and a source that uses it +fails at startup. Use `no_data`, which also skips the initial snapshot, if it suits your case. + +In addition, these sources parse DDL statements with a new parser. If a source fails on a DDL +statement that it parsed before, set `ddl.parser.type: legacy` in the `advanced.source` section +of the source. + +### Update the database.ssl.mode values of MariaDB sources + +MariaDB sources now use the MariaDB connector, which takes MariaDB values for `database.ssl.mode`. +Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with +`verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. + +### Check changes in the captured data + +The following changes affect the data or the metadata that RDI reports, without any change to +your configuration: + +- PostgreSQL user defined types are reported without their schema, for example `geometry` + instead of `"inventory"."geometry"`. +- SQL Server sources write the value of `unavailable.value.placeholder` instead of `NULL` for + unchanged `VARCHAR(MAX)`, `NVARCHAR(MAX)`, and `VARBINARY(MAX)` columns in updates. +- A `poll.interval.ms` value above 5 seconds is now applied as set. Earlier versions capped it at + 5 seconds. + ## Upgrading to RDI 2.2.0 Check the following changes before you upgrade from RDI 2.0.0. If you upgrade from RDI 1.19.x or diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md new file mode 100644 index 0000000000..841141ef79 --- /dev/null +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md @@ -0,0 +1,27 @@ +--- +Title: Redis Data Integration release notes 2.4.0 (TBD 2026) +alwaysopen: false +categories: +- docs +- operate +- rs +description: | + Debezium 3.7 for the Debezium collector. +linkTitle: 2.4.0 (TBD 2026) +toc: 'true' +weight: 964 +--- + +## What's New in 2.4.0 + +### Breaking Changes + +See [Upgrading to RDI 2.4.0](/content/integrate/redis-data-integration/installation/upgrade.md#upgrading-to-rdi-240) for the breaking changes and the steps to take before and after you upgrade. + +- **Oracle LogMiner batch and sleep properties are no longer supported**: Configuration validation now rejects `log.mining.batch.size.*`, `log.mining.sleep.time.*`, `log.mining.scn.gap.detection.*`, `log.mining.continuous.mine`, `log.mining.buffer.infinispan.cache.rollbacks`, and `log.mining.buffer.ehcache.rollbacks.config` in `sources..advanced.source`, and running collectors ignore them. Remove them before redeploying the pipeline configuration, and tune `log.mining.log.count.min` and `log.mining.log.count.growth.max` instead. +- **MySQL and MariaDB sources no longer support `snapshot.mode: never`, and parse DDL statements with a new parser**: A source with `snapshot.mode: never` fails at startup, so check whether `no_data`, which skips the initial snapshot, works for your case. In addition, if a source fails on a DDL statement that it parsed before, set `ddl.parser.type: legacy` in `sources..advanced.source`. +- **MariaDB sources take MariaDB `database.ssl.mode` values**: Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with `verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. +- **PostgreSQL user defined types are reported without their schema**: For example, a column type is now `geometry` instead of `"inventory"."geometry"`. +### Improvements + +- **Debezium upgrade**: Upgraded Debezium to version 3.7.0. From 44dc033f38d5bd29dce3ab84f816c70047481a30 Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Fri, 6 Nov 2026 10:03:10 +0200 Subject: [PATCH 2/7] Fix the Oracle archive destination property and mining strategy advice --- .../data-pipelines/prepare-dbs/oracle.md | 4 ++-- .../redis-data-integration/data-pipelines/supported-types.md | 4 ++-- .../integrate/redis-data-integration/installation/upgrade.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index e9e7f25a36..234706757c 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -178,7 +178,7 @@ If you have more than one destination with these settings, then you should consult your database administrator about which one to choose for Debezium. -Use the `log.mining.archive.destination.name` property in the connector configuration +Use the `archive.destination.name` property in the connector configuration to select the archive log destination for Debezium. For example, suppose you have two archive destinations, `LOG_ARCHIVE_DEST_2` and @@ -189,7 +189,7 @@ use the following setting: ```json { - "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_3" + "archive.destination.name": "LOG_ARCHIVE_DEST_3" } ``` diff --git a/content/integrate/redis-data-integration/data-pipelines/supported-types.md b/content/integrate/redis-data-integration/data-pipelines/supported-types.md index e80f7b6ead..bb65166edd 100644 --- a/content/integrate/redis-data-integration/data-pipelines/supported-types.md +++ b/content/integrate/redis-data-integration/data-pipelines/supported-types.md @@ -256,12 +256,12 @@ for the required supplemental-logging setup. - **LOBs**: set `lob.enabled: true` (default `false`) to capture `CLOB`, `NCLOB`, `BLOB`, and `XMLTYPE`. You cannot use the *hybrid* mining strategy with - `lob.enabled: true` — use `online_catalog` or `redo_log_catalog` instead. + `lob.enabled: true` — use `online_catalog` instead. - **Extended strings**: if the database parameter `max_string_size` is `EXTENDED`, set `lob.enabled: true` to capture `VARCHAR2`/`NVARCHAR2` values over 4000 bytes and `RAW` values over 2000 bytes. - **XMLTYPE**: requires `lob.enabled: true` and a non-hybrid mining strategy - (`online_catalog` or `redo_log_catalog`). The connector emits the XML as text + (`online_catalog`). The connector emits the XML as text (`STRING`). XMLTYPE support also requires the Oracle **XDB library** and the **`xmlparserv2`** dependency in addition to the standard `ojdbc11.jar` driver. If the runtime selects Oracle's `xmlparserv2` SAX parser, you may need to set the JVM diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index cde4929a8a..4d88ea3543 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -222,7 +222,7 @@ A higher count clears a backlog in fewer iterations, but each iteration takes lo collector lag after you redeploy. `log.mining.strategy: redo_log_catalog` still works, but Debezium has deprecated it. Use the -default `online_catalog` or `hybrid` instead. +default `online_catalog` instead, or `hybrid` if `lob.enabled` is not set to `true`. ### Check MySQL and MariaDB sources From 5308dcc0c98010df4873bd8027df32061f2a166a Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Mon, 9 Nov 2026 09:42:15 +0200 Subject: [PATCH 3/7] RDSC-6347: Update Oracle custom images, snapshot-mode notes, and MariaDB TLS values for Debezium 3.7 --- content/embeds/rdi-snapshot-modes.md | 18 ------- .../1.19.1/data-pipelines/pipeline-config.md | 18 +++++++ .../data-pipelines/pipeline-config.md | 18 +++++++ .../prepare-dbs/my-sql-mariadb.md | 15 +++++- .../data-pipelines/prepare-dbs/oracle.md | 53 +++++++++++++------ .../installation/install-k8s.md | 4 +- .../installation/upgrade.md | 12 +++-- .../release-notes/rdi-2-4-0.md | 7 +++ 8 files changed, 105 insertions(+), 40 deletions(-) diff --git a/content/embeds/rdi-snapshot-modes.md b/content/embeds/rdi-snapshot-modes.md index e7a42995c1..af1dce0848 100644 --- a/content/embeds/rdi-snapshot-modes.md +++ b/content/embeds/rdi-snapshot-modes.md @@ -51,21 +51,3 @@ monitoring. Check snapshot completion separately from collector connectivity; An interrupted initial snapshot can restart when the collector starts again, including with `initial` and `initial_only`. - -For Debezium 3.5, note these connector differences: - -- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence - number (LSN), or from the replication slot's creation position when no LSN is saved. - This requires the relevant write-ahead log (WAL) history to remain available. -- Other connectors can capture schema or initialize a position without copying - existing rows in `no_data` mode. -- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and - MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server - do not offer `never`; use `no_data` to skip existing rows. - -See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html), -[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html), -[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html), -[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html), -[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html), -and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html). diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md index fb962db493..6ac7274e15 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md @@ -199,6 +199,24 @@ section apply to that release. {{< embed-md "rdi-snapshot-modes.md" >}} +For Debezium 3.5, note these connector differences: + +- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence + number (LSN), or from the replication slot's creation position when no LSN is saved. + This requires the relevant write-ahead log (WAL) history to remain available. +- Other connectors can capture schema or initialize a position without copying + existing rows in `no_data` mode. +- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and + MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server + do not offer `never`; use `no_data` to skip existing rows. + +See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html), +[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html), +[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html), +[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html), +[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html), +and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html). + A reset clears saved positions without changing `snapshot.mode` or flushing the target Redis database. If you need to reload existing rows, select a mode that takes a data snapshot. A snapshot can overwrite target records, but it does not remove stale records diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index e94382c752..9449c2395d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -211,6 +211,24 @@ Debezium 3.7. {{< embed-md "rdi-snapshot-modes.md" >}} +For Debezium 3.7, note these connector differences: + +- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence + number (LSN), or from the replication slot's creation position when no LSN is saved. + This requires the relevant write-ahead log (WAL) history to remain available. +- Other connectors can capture schema or initialize a position without copying + existing rows in `no_data` mode. +- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and + MariaDB no longer offer `never`, and RDI rejects it for these sources. Oracle and + SQL Server do not offer `never`. Use `no_data` to skip existing rows. + +See the Debezium 3.7 connector references for [MariaDB](https://debezium.io/documentation/reference/3.7/connectors/mariadb.html), +[MongoDB](https://debezium.io/documentation/reference/3.7/connectors/mongodb.html), +[MySQL](https://debezium.io/documentation/reference/3.7/connectors/mysql.html), +[Oracle](https://debezium.io/documentation/reference/3.7/connectors/oracle.html), +[PostgreSQL](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html), +and [SQL Server](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html). + A reset clears saved positions without changing `snapshot.mode` or flushing the target Redis database. If you need to reload existing rows, select a mode that takes a data snapshot. A snapshot can overwrite target records, but it does not remove stale records diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 2a52612416..219ac2273b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -245,7 +245,9 @@ certificate. For mutual TLS, also set the source's `CERT` and `KEY` secrets. See [Set secrets](/content/integrate/redis-data-integration/data-pipelines/deploy.md#set-secrets) for the full list of source database TLS and mTLS secrets. -Select the TLS mode in the source `advanced.source` section, for example: +Select the TLS mode in the source `advanced.source` section. MySQL and MariaDB sources +take different values for `database.ssl.mode`. For example, to verify the server +certificate against the CA certificate, use the following for a MySQL source: ```yaml advanced: @@ -253,6 +255,17 @@ advanced: database.ssl.mode: verify_ca ``` +and the following for a MariaDB source: + +```yaml +advanced: + source: + database.ssl.mode: verify-ca +``` + +MySQL sources take `disabled`, `preferred`, `required`, `verify_ca`, and `verify_identity`. +MariaDB sources take `disable`, `trust`, `verify-ca`, and `verify-full`. + RDI builds the keystore that presents the client certificate to the source database from the source's `CERT` and `KEY` secrets, and configures the collector to use it. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index 234706757c..eb09d0989b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -953,6 +953,18 @@ PROMPT ===================================================== To support XStream connector, you must create a custom [Docker](https://www.docker.com/) image that includes the required Instant Client package libraries for Linux x64 from the Oracle website. +Build the custom image from the Debezium Server image that your RDI version uses, so that +the collector keeps the Debezium version and the RDI modifications that your configuration +is validated against. To find the image, check the collector section of the `rdi-operator` +configmap: + +```bash +kubectl get configmap rdi-operator -n rdi -o yaml | grep -A 5 'collector:' +``` + +The examples below use `redislabs/debezium-server:3.7.0.Final-rdi.2`, the image that +RDI 2.4.0 uses. Replace it with the repository and tag from your configmap. + 1. On the Docker machine, download the Instant Client package for Linux x64 from the Oracle website ```bash @@ -968,7 +980,7 @@ To support XStream connector, you must create a custom [Docker](https://www.dock 1. Create a `Dockerfile` in the `./dbz-ora` directory with the following contents: ```docker - FROM debezium/server:3.0.8.Final + FROM redislabs/debezium-server:3.7.0.Final-rdi.2 USER root @@ -996,9 +1008,9 @@ To support XStream connector, you must create a custom [Docker](https://www.dock 1. Add the image to the K3s image registry using the following commands: ```bash - docker tag dbz-ora quay.io/debezium/server:3.0.8.Final - docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xstream-linux-amd.tar - sudo k3s ctr images import dbz3.0.8-xstream-linux-amd.tar all + docker tag dbz-ora redislabs/debezium-server:3.7.0.Final-rdi.2-xstream + docker image save redislabs/debezium-server:3.7.0.Final-rdi.2-xstream -o dbz-xstream-linux-amd.tar + sudo k3s ctr images import dbz-xstream-linux-amd.tar all ``` ### 8. Make RDI use the custom image (VM installation) @@ -1009,22 +1021,24 @@ Edit the `rdi-operator` configmap: kubectl edit configmap rdi-operator -n rdi ``` -In the editor, find the collector section and change the image settings: +In the editor, find the collector section and change the image tag to the tag of your +custom image: ```yaml collector: image: pullPolicy: IfNotPresent - registry: docker.io # change this to `quay.io` - repository: redislabs/debezium-server # Change this to `debezium/server` - tag: 3.0.8.Final-rdi.1 # Change this to `3.0.8.Final` + registry: docker.io + repository: redislabs/debezium-server + tag: 3.7.0.Final-rdi.2 # Change this to `3.7.0.Final-rdi.2-xstream` ``` Save the configmap. Once it is saved, the operator will restart automatically and will apply the changes. > [!NOTE] > After upgrading to another RDI version, -> the changes to the configmap will be lost. You must repeat the above steps after each upgrade. +> the changes to the configmap will be lost. You must repeat the above steps after each upgrade, +> and rebuild the custom image from the Debezium Server image of the new RDI version. ### 9. Enable the Oracle configuration in RDI @@ -1048,7 +1062,7 @@ sources: database.dbname: ORCLCDB database.pdb.name: ORCLPDB1 database.connection.adapter: xstream - database.out.server.name: dbzxout + xstream.out.server.name: dbzxout ``` > [!NOTE] @@ -1072,7 +1086,10 @@ you must configure additional libraries for Debezium Server to process these col ### Create a custom Debezium Server image To support `XMLTYPE` columns, you must create a custom [Docker](https://www.docker.com/) image -that includes the required Oracle XML libraries. +that includes the required Oracle XML libraries. As with the +[XStream image](#7-add-a-custom-docker-image-for-the-debezium-server), build it from the +Debezium Server image that your RDI version uses. The examples below use +`redislabs/debezium-server:3.7.0.Final-rdi.2`, the image that RDI 2.4.0 uses. 1. Download the required libraries from Maven Central: @@ -1088,12 +1105,14 @@ that includes the required Oracle XML libraries. 2. Create a `Dockerfile` in the same directory: ```dockerfile - FROM quay.io/debezium/server:3.0.8.Final + FROM redislabs/debezium-server:3.7.0.Final-rdi.2 USER root COPY xdb.jar /debezium/lib COPY xmlparserv2.jar /debezium/lib + + USER jboss ``` 3. Build the custom image: @@ -1101,16 +1120,20 @@ that includes the required Oracle XML libraries. ```bash cd .. docker build -t dbz-xml xml - docker tag dbz-xml quay.io/debezium/server:3.0.8.Final - docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xml-linux-amd.tar + docker tag dbz-xml redislabs/debezium-server:3.7.0.Final-rdi.2-xml + docker image save redislabs/debezium-server:3.7.0.Final-rdi.2-xml -o dbz-xml-linux-amd.tar ``` 4. Add the image to your K3s image repository: ```bash - sudo k3s ctr images import dbz3.0.8-xml-linux-amd.tar all + sudo k3s ctr images import dbz-xml-linux-amd.tar all ``` +5. Change the collector image tag in the `rdi-operator` configmap to + `3.7.0.Final-rdi.2-xml`, as described in + [Make RDI use the custom image](#8-make-rdi-use-the-custom-image-vm-installation). + ### Configure RDI for XMLTYPE support In your RDI configuration file, set the `lob.enabled` property to `true` in the diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 6e82598097..0af5b1e602 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -106,8 +106,8 @@ If you plan to use Snowflake as a source for any of your pipelines, you'll also In addition, the RDI Helm chart uses the following 3rd party images: -- [redislabs/debezium-server:3.5.0.Final-rdi.3](https://hub.docker.com/r/redislabs/debezium-server), - based on `quay.io/debezium/server/3.5.0.Final` with minor modifications: +- [redislabs/debezium-server:3.7.0.Final-rdi.2](https://hub.docker.com/r/redislabs/debezium-server), + based on `quay.io/debezium/server/3.7.0.Final` with minor modifications: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. - [redis/reloader:v1.4.13](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.4.13`: [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 4d88ea3543..1ff56b4d6e 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -193,7 +193,8 @@ changes before you upgrade from RDI 2.2.x. If you upgrade from RDI 2.0.x, also r Debezium 3.7 no longer sizes each LogMiner iteration by a range of SCNs. Instead, while the collector catches up, each iteration reads `log.mining.log.count.min` (default `2`) logs per redo thread, and up to `log.mining.log.count.growth.max` (default `4`) while a long-running transaction -holds back the mining. RDI 2.4.0 rejects the properties that Debezium removed. This example shows where to remove them from your configuration: +holds back the mining. RDI 2.4.0 rejects the properties that Debezium removed. This example +shows where to remove them from your configuration: ```yaml sources: @@ -219,10 +220,13 @@ A collector that is already running ignores these properties after the upgrade, the default log counts until you redeploy. If you raised the batch size because the collector could not keep up with the volume of your redo logs, raise `log.mining.log.count.min` instead. A higher count clears a backlog in fewer iterations, but each iteration takes longer, so check the -collector lag after you redeploy. +collector lag after you redeploy. If you set `log.mining.log.count.min` above +`log.mining.log.count.growth.max`, the minimum takes precedence. Set `log.mining.log.count.min` +to `0` to remove the limit and mine all available logs in one iteration. -`log.mining.strategy: redo_log_catalog` still works, but Debezium has deprecated it. Use the -default `online_catalog` instead, or `hybrid` if `lob.enabled` is not set to `true`. +`log.mining.strategy: redo_log_catalog` still works in Debezium 3.7, but Debezium has deprecated +it and plans to remove it. Use the default `online_catalog` instead, or `hybrid` if `lob.enabled` +is not set to `true`. ### Check MySQL and MariaDB sources diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md index 841141ef79..6ed9a63dbd 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md @@ -22,6 +22,13 @@ See [Upgrading to RDI 2.4.0](/content/integrate/redis-data-integration/installat - **MySQL and MariaDB sources no longer support `snapshot.mode: never`, and parse DDL statements with a new parser**: A source with `snapshot.mode: never` fails at startup, so check whether `no_data`, which skips the initial snapshot, works for your case. In addition, if a source fails on a DDL statement that it parsed before, set `ddl.parser.type: legacy` in `sources..advanced.source`. - **MariaDB sources take MariaDB `database.ssl.mode` values**: Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with `verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. - **PostgreSQL user defined types are reported without their schema**: For example, a column type is now `geometry` instead of `"inventory"."geometry"`. +- **SQL Server sources report unchanged large columns with a placeholder**: In update events, unchanged `VARCHAR(MAX)`, `NVARCHAR(MAX)`, and `VARBINARY(MAX)` columns now contain the value of `unavailable.value.placeholder` instead of `NULL`. +- **`poll.interval.ms` values above 5 seconds are applied as set**: Earlier versions capped the poll interval at 5 seconds. + +### Compatibility Notes + +- **Oracle `redo_log_catalog` mining strategy is deprecated**: Debezium 3.7 still accepts `log.mining.strategy: redo_log_catalog`, but plans to remove it. Use the default `online_catalog`, or `hybrid` if `lob.enabled` is not set to `true`. + ### Improvements - **Debezium upgrade**: Upgraded Debezium to version 3.7.0. From 00ce6337d7b4d68021463b1272fb610aa062ac4d Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Mon, 9 Nov 2026 09:54:23 +0200 Subject: [PATCH 4/7] RDSC-6347: Remove SQL Server and poll interval bullets from the 2.4.0 release notes --- .../integrate/redis-data-integration/release-notes/rdi-2-4-0.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md index 6ed9a63dbd..5c19ad0f29 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md @@ -22,8 +22,6 @@ See [Upgrading to RDI 2.4.0](/content/integrate/redis-data-integration/installat - **MySQL and MariaDB sources no longer support `snapshot.mode: never`, and parse DDL statements with a new parser**: A source with `snapshot.mode: never` fails at startup, so check whether `no_data`, which skips the initial snapshot, works for your case. In addition, if a source fails on a DDL statement that it parsed before, set `ddl.parser.type: legacy` in `sources..advanced.source`. - **MariaDB sources take MariaDB `database.ssl.mode` values**: Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with `verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. - **PostgreSQL user defined types are reported without their schema**: For example, a column type is now `geometry` instead of `"inventory"."geometry"`. -- **SQL Server sources report unchanged large columns with a placeholder**: In update events, unchanged `VARCHAR(MAX)`, `NVARCHAR(MAX)`, and `VARBINARY(MAX)` columns now contain the value of `unavailable.value.placeholder` instead of `NULL`. -- **`poll.interval.ms` values above 5 seconds are applied as set**: Earlier versions capped the poll interval at 5 seconds. ### Compatibility Notes From 06949f8d7a3e1f43644064a35a9aeb094064389e Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Mon, 9 Nov 2026 14:22:23 +0200 Subject: [PATCH 5/7] RDSC-6347: Narrow the 2.4.0 docs to the Debezium 3.7 upgrade --- .../data-pipelines/prepare-dbs/oracle.md | 8 +++----- .../data-pipelines/supported-types.md | 4 ++-- .../installation/upgrade.md | 18 +++++------------- .../release-notes/rdi-2-4-0.md | 12 ++++-------- 4 files changed, 14 insertions(+), 28 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index eb09d0989b..984aee9eb2 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -178,7 +178,7 @@ If you have more than one destination with these settings, then you should consult your database administrator about which one to choose for Debezium. -Use the `archive.destination.name` property in the connector configuration +Use the `log.mining.archive.destination.name` property in the connector configuration to select the archive log destination for Debezium. For example, suppose you have two archive destinations, `LOG_ARCHIVE_DEST_2` and @@ -189,7 +189,7 @@ use the following setting: ```json { - "archive.destination.name": "LOG_ARCHIVE_DEST_3" + "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_3" } ``` @@ -1062,7 +1062,7 @@ sources: database.dbname: ORCLCDB database.pdb.name: ORCLPDB1 database.connection.adapter: xstream - xstream.out.server.name: dbzxout + database.out.server.name: dbzxout ``` > [!NOTE] @@ -1111,8 +1111,6 @@ Debezium Server image that your RDI version uses. The examples below use COPY xdb.jar /debezium/lib COPY xmlparserv2.jar /debezium/lib - - USER jboss ``` 3. Build the custom image: diff --git a/content/integrate/redis-data-integration/data-pipelines/supported-types.md b/content/integrate/redis-data-integration/data-pipelines/supported-types.md index bb65166edd..e80f7b6ead 100644 --- a/content/integrate/redis-data-integration/data-pipelines/supported-types.md +++ b/content/integrate/redis-data-integration/data-pipelines/supported-types.md @@ -256,12 +256,12 @@ for the required supplemental-logging setup. - **LOBs**: set `lob.enabled: true` (default `false`) to capture `CLOB`, `NCLOB`, `BLOB`, and `XMLTYPE`. You cannot use the *hybrid* mining strategy with - `lob.enabled: true` — use `online_catalog` instead. + `lob.enabled: true` — use `online_catalog` or `redo_log_catalog` instead. - **Extended strings**: if the database parameter `max_string_size` is `EXTENDED`, set `lob.enabled: true` to capture `VARCHAR2`/`NVARCHAR2` values over 4000 bytes and `RAW` values over 2000 bytes. - **XMLTYPE**: requires `lob.enabled: true` and a non-hybrid mining strategy - (`online_catalog`). The connector emits the XML as text + (`online_catalog` or `redo_log_catalog`). The connector emits the XML as text (`STRING`). XMLTYPE support also requires the Oracle **XDB library** and the **`xmlparserv2`** dependency in addition to the standard `ojdbc11.jar` driver. If the runtime selects Oracle's `xmlparserv2` SAX parser, you may need to set the JVM diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 1ff56b4d6e..9850ba1b1a 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -219,10 +219,6 @@ sources: A collector that is already running ignores these properties after the upgrade, so it mines with the default log counts until you redeploy. If you raised the batch size because the collector could not keep up with the volume of your redo logs, raise `log.mining.log.count.min` instead. -A higher count clears a backlog in fewer iterations, but each iteration takes longer, so check the -collector lag after you redeploy. If you set `log.mining.log.count.min` above -`log.mining.log.count.growth.max`, the minimum takes precedence. Set `log.mining.log.count.min` -to `0` to remove the limit and mine all available logs in one iteration. `log.mining.strategy: redo_log_catalog` still works in Debezium 3.7, but Debezium has deprecated it and plans to remove it. Use the default `online_catalog` instead, or `hybrid` if `lob.enabled` @@ -242,18 +238,14 @@ of the source. MariaDB sources now use the MariaDB connector, which takes MariaDB values for `database.ssl.mode`. Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with `verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. +`disable` turns off encryption. `trust` requires encryption but doesn't verify the server +certificate. ### Check changes in the captured data -The following changes affect the data or the metadata that RDI reports, without any change to -your configuration: - -- PostgreSQL user defined types are reported without their schema, for example `geometry` - instead of `"inventory"."geometry"`. -- SQL Server sources write the value of `unavailable.value.placeholder` instead of `NULL` for - unchanged `VARCHAR(MAX)`, `NVARCHAR(MAX)`, and `VARBINARY(MAX)` columns in updates. -- A `poll.interval.ms` value above 5 seconds is now applied as set. Earlier versions capped it at - 5 seconds. +PostgreSQL user defined types are reported without their schema, for example `geometry` +instead of `"inventory"."geometry"`. This changes the metadata that RDI reports, without any +change to your configuration. ## Upgrading to RDI 2.2.0 diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md index 5c19ad0f29..0802eac348 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md @@ -18,14 +18,10 @@ weight: 964 See [Upgrading to RDI 2.4.0](/content/integrate/redis-data-integration/installation/upgrade.md#upgrading-to-rdi-240) for the breaking changes and the steps to take before and after you upgrade. -- **Oracle LogMiner batch and sleep properties are no longer supported**: Configuration validation now rejects `log.mining.batch.size.*`, `log.mining.sleep.time.*`, `log.mining.scn.gap.detection.*`, `log.mining.continuous.mine`, `log.mining.buffer.infinispan.cache.rollbacks`, and `log.mining.buffer.ehcache.rollbacks.config` in `sources..advanced.source`, and running collectors ignore them. Remove them before redeploying the pipeline configuration, and tune `log.mining.log.count.min` and `log.mining.log.count.growth.max` instead. -- **MySQL and MariaDB sources no longer support `snapshot.mode: never`, and parse DDL statements with a new parser**: A source with `snapshot.mode: never` fails at startup, so check whether `no_data`, which skips the initial snapshot, works for your case. In addition, if a source fails on a DDL statement that it parsed before, set `ddl.parser.type: legacy` in `sources..advanced.source`. -- **MariaDB sources take MariaDB `database.ssl.mode` values**: Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with `verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. -- **PostgreSQL user defined types are reported without their schema**: For example, a column type is now `geometry` instead of `"inventory"."geometry"`. - -### Compatibility Notes - -- **Oracle `redo_log_catalog` mining strategy is deprecated**: Debezium 3.7 still accepts `log.mining.strategy: redo_log_catalog`, but plans to remove it. Use the default `online_catalog`, or `hybrid` if `lob.enabled` is not set to `true`. +- Oracle sources reject the LogMiner batch, sleep, and SCN gap detection properties that Debezium 3.7 removed. +- MySQL and MariaDB sources no longer accept `snapshot.mode: never`, and parse DDL statements with a new parser. +- MariaDB sources take MariaDB values for `database.ssl.mode`. +- PostgreSQL user defined types are reported without their schema. ### Improvements From c9ee26d37796011a0ab3152b449a3e002fdaad19 Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Mon, 9 Nov 2026 14:41:48 +0200 Subject: [PATCH 6/7] RDSC-6347: Restore the Oracle archive destination property fix --- .../data-pipelines/prepare-dbs/oracle.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index 984aee9eb2..8d962ba383 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -178,7 +178,7 @@ If you have more than one destination with these settings, then you should consult your database administrator about which one to choose for Debezium. -Use the `log.mining.archive.destination.name` property in the connector configuration +Use the `archive.destination.name` property in the connector configuration to select the archive log destination for Debezium. For example, suppose you have two archive destinations, `LOG_ARCHIVE_DEST_2` and @@ -189,7 +189,7 @@ use the following setting: ```json { - "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_3" + "archive.destination.name": "LOG_ARCHIVE_DEST_3" } ``` From dea58cd0236066bc33422403b7a5738ae941b9a3 Mon Sep 17 00:00:00 2001 From: Orhan Abdula Date: Tue, 10 Nov 2026 09:13:19 +0200 Subject: [PATCH 7/7] RDSC-6347: Address review comments on snapshot modes, MariaDB TLS values, and Oracle LogMiner properties --- .../data-pipelines/pipeline-config.md | 26 +++++++------------ .../prepare-dbs/my-sql-mariadb.md | 5 ++-- .../installation/upgrade.md | 17 ++++++++++-- 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 9449c2395d..e9b9a6a1e1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -211,23 +211,15 @@ Debezium 3.7. {{< embed-md "rdi-snapshot-modes.md" >}} -For Debezium 3.7, note these connector differences: - -- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence - number (LSN), or from the replication slot's creation position when no LSN is saved. - This requires the relevant write-ahead log (WAL) history to remain available. -- Other connectors can capture schema or initialize a position without copying - existing rows in `no_data` mode. -- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and - MariaDB no longer offer `never`, and RDI rejects it for these sources. Oracle and - SQL Server do not offer `never`. Use `no_data` to skip existing rows. - -See the Debezium 3.7 connector references for [MariaDB](https://debezium.io/documentation/reference/3.7/connectors/mariadb.html), -[MongoDB](https://debezium.io/documentation/reference/3.7/connectors/mongodb.html), -[MySQL](https://debezium.io/documentation/reference/3.7/connectors/mysql.html), -[Oracle](https://debezium.io/documentation/reference/3.7/connectors/oracle.html), -[PostgreSQL](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html), -and [SQL Server](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html). +Use the default `initial` mode unless you have a specific reason to choose another. +The supported modes and their behavior differ by connector. For the modes your source +supports, see `snapshot.mode` in the Debezium 3.7 reference for +[MariaDB](https://debezium.io/documentation/reference/3.7/connectors/mariadb.html#mariadb-property-snapshot-mode), +[MongoDB](https://debezium.io/documentation/reference/3.7/connectors/mongodb.html#mongodb-property-snapshot-mode), +[MySQL](https://debezium.io/documentation/reference/3.7/connectors/mysql.html#mysql-property-snapshot-mode), +[Oracle](https://debezium.io/documentation/reference/3.7/connectors/oracle.html#oracle-property-snapshot-mode), +[PostgreSQL](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html#postgresql-property-snapshot-mode), +and [SQL Server](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html#sqlserver-property-snapshot-mode). A reset clears saved positions without changing `snapshot.mode` or flushing the target Redis database. If you need to reload existing rows, select a mode that takes a data diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 219ac2273b..788bc6e636 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -255,7 +255,9 @@ advanced: database.ssl.mode: verify_ca ``` -and the following for a MariaDB source: +MySQL sources take `disabled`, `preferred`, `required`, `verify_ca`, and `verify_identity`. + +For a MariaDB source, use the following: ```yaml advanced: @@ -263,7 +265,6 @@ advanced: database.ssl.mode: verify-ca ``` -MySQL sources take `disabled`, `preferred`, `required`, `verify_ca`, and `verify_identity`. MariaDB sources take `disable`, `trust`, `verify-ca`, and `verify-full`. RDI builds the keystore that presents the client certificate to the source database from the diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 9850ba1b1a..39c28fc486 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -217,8 +217,21 @@ sources: ``` A collector that is already running ignores these properties after the upgrade, so it mines with -the default log counts until you redeploy. If you raised the batch size because the collector -could not keep up with the volume of your redo logs, raise `log.mining.log.count.min` instead. +the default log counts until you redeploy. + +If you set custom values for the removed properties, replace them as follows: + +| Removed properties | What to do | +| :-- | :-- | +| `log.mining.batch.size.*`, raised so the collector keeps up with the volume of your redo logs | Remove them and raise `log.mining.log.count.min`. | +| `log.mining.batch.size.*`, raised so mining advances past long-running transactions | Remove them and raise `log.mining.log.count.growth.max`. | +| `log.mining.sleep.time.*` | Remove them. Debezium 3.7 has no replacement. | +| `log.mining.scn.gap.detection.*` | Remove them. Debezium 3.7 has no replacement. | +| `log.mining.continuous.mine`, `log.mining.buffer.infinispan.cache.rollbacks`, and `log.mining.buffer.ehcache.rollbacks.config` | Remove them. Debezium 3.7 has no replacement. | + +The new properties count redo logs, not SCNs, so you can't convert an old batch size to a log +count. Start with the defaults, and raise the values in small steps if the collector falls behind. +The log count properties don't apply when `log.mining.strategy` is set to `redo_log_catalog`. `log.mining.strategy: redo_log_catalog` still works in Debezium 3.7, but Debezium has deprecated it and plans to remove it. Use the default `online_catalog` instead, or `hybrid` if `lob.enabled`