diff --git a/content/embeds/rdi-snapshot-modes.md b/content/embeds/rdi-snapshot-modes.md index e7a42995c1..af1dce0848 100644 --- a/content/embeds/rdi-snapshot-modes.md +++ b/content/embeds/rdi-snapshot-modes.md @@ -51,21 +51,3 @@ monitoring. Check snapshot completion separately from collector connectivity; An interrupted initial snapshot can restart when the collector starts again, including with `initial` and `initial_only`. - -For Debezium 3.5, note these connector differences: - -- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence - number (LSN), or from the replication slot's creation position when no LSN is saved. - This requires the relevant write-ahead log (WAL) history to remain available. -- Other connectors can capture schema or initialize a position without copying - existing rows in `no_data` mode. -- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and - MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server - do not offer `never`; use `no_data` to skip existing rows. - -See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html), -[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html), -[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html), -[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html), -[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html), -and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html). diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md index fb962db493..6ac7274e15 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md @@ -199,6 +199,24 @@ section apply to that release. {{< embed-md "rdi-snapshot-modes.md" >}} +For Debezium 3.5, note these connector differences: + +- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence + number (LSN), or from the replication slot's creation position when no LSN is saved. + This requires the relevant write-ahead log (WAL) history to remain available. +- Other connectors can capture schema or initialize a position without copying + existing rows in `no_data` mode. +- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and + MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server + do not offer `never`; use `no_data` to skip existing rows. + +See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html), +[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html), +[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html), +[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html), +[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html), +and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html). + A reset clears saved positions without changing `snapshot.mode` or flushing the target Redis database. If you need to reload existing rows, select a mode that takes a data snapshot. A snapshot can overwrite target records, but it does not remove stale records diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index e94382c752..e9b9a6a1e1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -211,6 +211,16 @@ Debezium 3.7. {{< embed-md "rdi-snapshot-modes.md" >}} +Use the default `initial` mode unless you have a specific reason to choose another. +The supported modes and their behavior differ by connector. For the modes your source +supports, see `snapshot.mode` in the Debezium 3.7 reference for +[MariaDB](https://debezium.io/documentation/reference/3.7/connectors/mariadb.html#mariadb-property-snapshot-mode), +[MongoDB](https://debezium.io/documentation/reference/3.7/connectors/mongodb.html#mongodb-property-snapshot-mode), +[MySQL](https://debezium.io/documentation/reference/3.7/connectors/mysql.html#mysql-property-snapshot-mode), +[Oracle](https://debezium.io/documentation/reference/3.7/connectors/oracle.html#oracle-property-snapshot-mode), +[PostgreSQL](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html#postgresql-property-snapshot-mode), +and [SQL Server](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html#sqlserver-property-snapshot-mode). + A reset clears saved positions without changing `snapshot.mode` or flushing the target Redis database. If you need to reload existing rows, select a mode that takes a data snapshot. A snapshot can overwrite target records, but it does not remove stale records diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 2a52612416..788bc6e636 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -245,7 +245,9 @@ certificate. For mutual TLS, also set the source's `CERT` and `KEY` secrets. See [Set secrets](/content/integrate/redis-data-integration/data-pipelines/deploy.md#set-secrets) for the full list of source database TLS and mTLS secrets. -Select the TLS mode in the source `advanced.source` section, for example: +Select the TLS mode in the source `advanced.source` section. MySQL and MariaDB sources +take different values for `database.ssl.mode`. For example, to verify the server +certificate against the CA certificate, use the following for a MySQL source: ```yaml advanced: @@ -253,6 +255,18 @@ advanced: database.ssl.mode: verify_ca ``` +MySQL sources take `disabled`, `preferred`, `required`, `verify_ca`, and `verify_identity`. + +For a MariaDB source, use the following: + +```yaml +advanced: + source: + database.ssl.mode: verify-ca +``` + +MariaDB sources take `disable`, `trust`, `verify-ca`, and `verify-full`. + RDI builds the keystore that presents the client certificate to the source database from the source's `CERT` and `KEY` secrets, and configures the collector to use it. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index e9e7f25a36..8d962ba383 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -178,7 +178,7 @@ If you have more than one destination with these settings, then you should consult your database administrator about which one to choose for Debezium. -Use the `log.mining.archive.destination.name` property in the connector configuration +Use the `archive.destination.name` property in the connector configuration to select the archive log destination for Debezium. For example, suppose you have two archive destinations, `LOG_ARCHIVE_DEST_2` and @@ -189,7 +189,7 @@ use the following setting: ```json { - "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_3" + "archive.destination.name": "LOG_ARCHIVE_DEST_3" } ``` @@ -953,6 +953,18 @@ PROMPT ===================================================== To support XStream connector, you must create a custom [Docker](https://www.docker.com/) image that includes the required Instant Client package libraries for Linux x64 from the Oracle website. +Build the custom image from the Debezium Server image that your RDI version uses, so that +the collector keeps the Debezium version and the RDI modifications that your configuration +is validated against. To find the image, check the collector section of the `rdi-operator` +configmap: + +```bash +kubectl get configmap rdi-operator -n rdi -o yaml | grep -A 5 'collector:' +``` + +The examples below use `redislabs/debezium-server:3.7.0.Final-rdi.2`, the image that +RDI 2.4.0 uses. Replace it with the repository and tag from your configmap. + 1. On the Docker machine, download the Instant Client package for Linux x64 from the Oracle website ```bash @@ -968,7 +980,7 @@ To support XStream connector, you must create a custom [Docker](https://www.dock 1. Create a `Dockerfile` in the `./dbz-ora` directory with the following contents: ```docker - FROM debezium/server:3.0.8.Final + FROM redislabs/debezium-server:3.7.0.Final-rdi.2 USER root @@ -996,9 +1008,9 @@ To support XStream connector, you must create a custom [Docker](https://www.dock 1. Add the image to the K3s image registry using the following commands: ```bash - docker tag dbz-ora quay.io/debezium/server:3.0.8.Final - docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xstream-linux-amd.tar - sudo k3s ctr images import dbz3.0.8-xstream-linux-amd.tar all + docker tag dbz-ora redislabs/debezium-server:3.7.0.Final-rdi.2-xstream + docker image save redislabs/debezium-server:3.7.0.Final-rdi.2-xstream -o dbz-xstream-linux-amd.tar + sudo k3s ctr images import dbz-xstream-linux-amd.tar all ``` ### 8. Make RDI use the custom image (VM installation) @@ -1009,22 +1021,24 @@ Edit the `rdi-operator` configmap: kubectl edit configmap rdi-operator -n rdi ``` -In the editor, find the collector section and change the image settings: +In the editor, find the collector section and change the image tag to the tag of your +custom image: ```yaml collector: image: pullPolicy: IfNotPresent - registry: docker.io # change this to `quay.io` - repository: redislabs/debezium-server # Change this to `debezium/server` - tag: 3.0.8.Final-rdi.1 # Change this to `3.0.8.Final` + registry: docker.io + repository: redislabs/debezium-server + tag: 3.7.0.Final-rdi.2 # Change this to `3.7.0.Final-rdi.2-xstream` ``` Save the configmap. Once it is saved, the operator will restart automatically and will apply the changes. > [!NOTE] > After upgrading to another RDI version, -> the changes to the configmap will be lost. You must repeat the above steps after each upgrade. +> the changes to the configmap will be lost. You must repeat the above steps after each upgrade, +> and rebuild the custom image from the Debezium Server image of the new RDI version. ### 9. Enable the Oracle configuration in RDI @@ -1072,7 +1086,10 @@ you must configure additional libraries for Debezium Server to process these col ### Create a custom Debezium Server image To support `XMLTYPE` columns, you must create a custom [Docker](https://www.docker.com/) image -that includes the required Oracle XML libraries. +that includes the required Oracle XML libraries. As with the +[XStream image](#7-add-a-custom-docker-image-for-the-debezium-server), build it from the +Debezium Server image that your RDI version uses. The examples below use +`redislabs/debezium-server:3.7.0.Final-rdi.2`, the image that RDI 2.4.0 uses. 1. Download the required libraries from Maven Central: @@ -1088,7 +1105,7 @@ that includes the required Oracle XML libraries. 2. Create a `Dockerfile` in the same directory: ```dockerfile - FROM quay.io/debezium/server:3.0.8.Final + FROM redislabs/debezium-server:3.7.0.Final-rdi.2 USER root @@ -1101,16 +1118,20 @@ that includes the required Oracle XML libraries. ```bash cd .. docker build -t dbz-xml xml - docker tag dbz-xml quay.io/debezium/server:3.0.8.Final - docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xml-linux-amd.tar + docker tag dbz-xml redislabs/debezium-server:3.7.0.Final-rdi.2-xml + docker image save redislabs/debezium-server:3.7.0.Final-rdi.2-xml -o dbz-xml-linux-amd.tar ``` 4. Add the image to your K3s image repository: ```bash - sudo k3s ctr images import dbz3.0.8-xml-linux-amd.tar all + sudo k3s ctr images import dbz-xml-linux-amd.tar all ``` +5. Change the collector image tag in the `rdi-operator` configmap to + `3.7.0.Final-rdi.2-xml`, as described in + [Make RDI use the custom image](#8-make-rdi-use-the-custom-image-vm-installation). + ### Configure RDI for XMLTYPE support In your RDI configuration file, set the `lob.enabled` property to `true` in the diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 6e82598097..0af5b1e602 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -106,8 +106,8 @@ If you plan to use Snowflake as a source for any of your pipelines, you'll also In addition, the RDI Helm chart uses the following 3rd party images: -- [redislabs/debezium-server:3.5.0.Final-rdi.3](https://hub.docker.com/r/redislabs/debezium-server), - based on `quay.io/debezium/server/3.5.0.Final` with minor modifications: +- [redislabs/debezium-server:3.7.0.Final-rdi.2](https://hub.docker.com/r/redislabs/debezium-server), + based on `quay.io/debezium/server/3.7.0.Final` with minor modifications: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. - [redis/reloader:v1.4.13](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.4.13`: [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index da2f726f99..39c28fc486 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -17,7 +17,8 @@ weight: 30 --- > [!NOTE] -> Before you upgrade, review [Upgrading to RDI 2.2.0](#upgrading-to-rdi-220). If you upgrade +> Before you upgrade, review [Upgrading to RDI 2.4.0](#upgrading-to-rdi-240). If you upgrade +> from RDI 2.0.x, also review [Upgrading to RDI 2.2.0](#upgrading-to-rdi-220), and if you upgrade > from RDI 1.19.x or earlier, also review [Upgrading to RDI 2.0.0](#upgrading-to-rdi-200). > [!WARNING] @@ -181,6 +182,84 @@ your saved `rdi-values.yaml` for that version. This will restore your previous w > described in [Uninstall RDI](/content/integrate/redis-data-integration/installation/install-k8s.md#uninstall-rdi), > and then install the old version. +## Upgrading to RDI 2.4.0 + +RDI 2.4.0 upgrades the Debezium collector from Debezium 3.5 to Debezium 3.7. Check the following +changes before you upgrade from RDI 2.2.x. If you upgrade from RDI 2.0.x, also review +[Upgrading to RDI 2.2.0](#upgrading-to-rdi-220). + +### Replace the Oracle LogMiner batch and sleep properties + +Debezium 3.7 no longer sizes each LogMiner iteration by a range of SCNs. Instead, while the +collector catches up, each iteration reads `log.mining.log.count.min` (default `2`) logs per redo +thread, and up to `log.mining.log.count.growth.max` (default `4`) while a long-running transaction +holds back the mining. RDI 2.4.0 rejects the properties that Debezium removed. This example +shows where to remove them from your configuration: + +```yaml +sources: + : + advanced: + source: + log.mining.batch.size.min: # Rejected since RDI 2.4.0 + log.mining.batch.size.max: # Rejected since RDI 2.4.0 + log.mining.batch.size.default: # Rejected since RDI 2.4.0 + log.mining.batch.size.increment: # Rejected since RDI 2.4.0 + log.mining.sleep.time.min.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.max.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.default.ms: # Rejected since RDI 2.4.0 + log.mining.sleep.time.increment.ms: # Rejected since RDI 2.4.0 + log.mining.scn.gap.detection.gap.size.min: # Rejected since RDI 2.4.0 + log.mining.scn.gap.detection.time.interval.max.ms: # Rejected since RDI 2.4.0 + log.mining.continuous.mine: # Rejected since RDI 2.4.0 + log.mining.buffer.infinispan.cache.rollbacks: # Rejected since RDI 2.4.0 + log.mining.buffer.ehcache.rollbacks.config: # Rejected since RDI 2.4.0 +``` + +A collector that is already running ignores these properties after the upgrade, so it mines with +the default log counts until you redeploy. + +If you set custom values for the removed properties, replace them as follows: + +| Removed properties | What to do | +| :-- | :-- | +| `log.mining.batch.size.*`, raised so the collector keeps up with the volume of your redo logs | Remove them and raise `log.mining.log.count.min`. | +| `log.mining.batch.size.*`, raised so mining advances past long-running transactions | Remove them and raise `log.mining.log.count.growth.max`. | +| `log.mining.sleep.time.*` | Remove them. Debezium 3.7 has no replacement. | +| `log.mining.scn.gap.detection.*` | Remove them. Debezium 3.7 has no replacement. | +| `log.mining.continuous.mine`, `log.mining.buffer.infinispan.cache.rollbacks`, and `log.mining.buffer.ehcache.rollbacks.config` | Remove them. Debezium 3.7 has no replacement. | + +The new properties count redo logs, not SCNs, so you can't convert an old batch size to a log +count. Start with the defaults, and raise the values in small steps if the collector falls behind. +The log count properties don't apply when `log.mining.strategy` is set to `redo_log_catalog`. + +`log.mining.strategy: redo_log_catalog` still works in Debezium 3.7, but Debezium has deprecated +it and plans to remove it. Use the default `online_catalog` instead, or `hybrid` if `lob.enabled` +is not set to `true`. + +### Check MySQL and MariaDB sources + +RDI 2.4.0 rejects `snapshot.mode: never` for MySQL and MariaDB sources, and a source that uses it +fails at startup. Use `no_data`, which also skips the initial snapshot, if it suits your case. + +In addition, these sources parse DDL statements with a new parser. If a source fails on a DDL +statement that it parsed before, set `ddl.parser.type: legacy` in the `advanced.source` section +of the source. + +### Update the database.ssl.mode values of MariaDB sources + +MariaDB sources now use the MariaDB connector, which takes MariaDB values for `database.ssl.mode`. +Replace `disabled` and `preferred` with `disable`, `required` with `trust`, `verify_ca` with +`verify-ca`, and `verify_identity` with `verify-full`. The default is now `disable`. +`disable` turns off encryption. `trust` requires encryption but doesn't verify the server +certificate. + +### Check changes in the captured data + +PostgreSQL user defined types are reported without their schema, for example `geometry` +instead of `"inventory"."geometry"`. This changes the metadata that RDI reports, without any +change to your configuration. + ## Upgrading to RDI 2.2.0 Check the following changes before you upgrade from RDI 2.0.0. If you upgrade from RDI 1.19.x or diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md new file mode 100644 index 0000000000..0802eac348 --- /dev/null +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-4-0.md @@ -0,0 +1,28 @@ +--- +Title: Redis Data Integration release notes 2.4.0 (TBD 2026) +alwaysopen: false +categories: +- docs +- operate +- rs +description: | + Debezium 3.7 for the Debezium collector. +linkTitle: 2.4.0 (TBD 2026) +toc: 'true' +weight: 964 +--- + +## What's New in 2.4.0 + +### Breaking Changes + +See [Upgrading to RDI 2.4.0](/content/integrate/redis-data-integration/installation/upgrade.md#upgrading-to-rdi-240) for the breaking changes and the steps to take before and after you upgrade. + +- Oracle sources reject the LogMiner batch, sleep, and SCN gap detection properties that Debezium 3.7 removed. +- MySQL and MariaDB sources no longer accept `snapshot.mode: never`, and parse DDL statements with a new parser. +- MariaDB sources take MariaDB values for `database.ssl.mode`. +- PostgreSQL user defined types are reported without their schema. + +### Improvements + +- **Debezium upgrade**: Upgraded Debezium to version 3.7.0.