Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions servicecontrol/upgrades/6.19to6.20.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
---
title: Upgrade ServiceControl from Version 6.19 to Version 6.20
summary: Instructions on how to upgrade ServiceControl from version 6.19 to 6.20
reviewed: 2026-10-05
reviewed: 2026-10-17
isUpgradeGuide: true
component: ServiceControl
---

> [!WARNING]
> **Upgrading audit instances rebuilds the audit message index.** Version 6.20 changes the definition of the audit instance index `MessagesViewIndex`, or `MessagesViewIndexWithFullTextSearch` when full-text search on message bodies is enabled. At the first start-up after the upgrade, RavenDB rebuilds the index. On large audit databases the rebuild can take a long time and degrades ingestion and indexing rates while it runs. Read [Audit message index definition changed](#audit-message-index-definition-changed) and plan the upgrade before upgrading audit instances.

ServiceControl version 6.20 changes the default RavenDB search engine from Corax to Lucene to address performance and stability issues on large database instances. Databases created by version 6.20 use Lucene. Existing error and audit databases are not modified by the upgrade and continue to use Corax. **Migrating existing databases to Lucene is recommended**, see [Should existing databases be migrated?](#should-existing-databases-be-migrated). Monitoring instances are not affected.

## New databases use the Lucene search engine
Expand Down Expand Up @@ -36,7 +39,7 @@ Instances without these symptoms should be migrated in the next planned maintena

## Audit message index definition changed

Version 6.20 changes the definition of the audit instance index `MessagesViewIndex`. When [full-text search on message bodies](/servicecontrol/audit-instances/configuration.md#performance-tuning-servicecontrol-auditenablefulltextsearchonbodies) is enabled, the index is `MessagesViewIndexWithFullTextSearch`. At the first start-up after the upgrade, RavenDB builds the new definition as a replacement index next to the existing index. When the replacement is no longer stale, RavenDB swaps it in. On a large audit database, the rebuild takes time. Ingestion and indexing rates are lower during the rebuild. Plan the upgrade of audit instances in a maintenance window.
Version 6.20 changes the definition of the audit instance index `MessagesViewIndex`. When [full-text search on message bodies](/servicecontrol/audit-instances/configuration.md#performance-tuning-servicecontrol-auditenablefulltextsearchonbodies) is enabled, the index is `MessagesViewIndexWithFullTextSearch`. At the first start-up after the upgrade, RavenDB builds the new definition as a replacement index next to the existing index. When the replacement is no longer stale, RavenDB swaps it in. On a large audit database, the rebuild takes time. Ingestion and indexing rates are lower during the rebuild. Plan the upgrade of audit instances in a maintenance window. To keep the environment running during the rebuild, see [upgrades that rebuild indexes](/servicecontrol/upgrades/#upgrades-that-rebuild-indexes).

On versions 6.20.0 and 6.21.0, the rebuilt index uses the search engine of the database. For databases created before version 6.20, this is Corax, also for an unlocked index that was migrated to Lucene. From version 6.21.1, the rebuilt index keeps the search engine of the existing index.

Expand Down
42 changes: 41 additions & 1 deletion servicecontrol/upgrades/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Upgrade Tips
summary: Tips for upgrading to newer versions of ServiceControl.
reviewed: 2023-07-07
reviewed: 2026-10-17
related:
---

Expand All @@ -17,6 +17,46 @@ related:
> [!NOTE]
> Upgrading the ServiceControl will not upgrade each of the individual error, monitoring or audit instances. Each of those instances will need to be upgraded separately if desired.

## Upgrades that rebuild indexes

Some ServiceControl versions change the definition of one or more RavenDB indexes. The [upgrade guide](/servicecontrol/upgrades/) for the version notes when this happens. At the first start-up after the upgrade, ServiceControl deploys the new definition and RavenDB builds it as a side-by-side replacement index, listed as `ReplacementOf/<index>` in RavenDB Studio. The existing index keeps serving queries while the replacement is built. Once the replacement has caught up with all documents and is no longer stale, RavenDB swaps it in and deletes the old index.

The rebuild has to process every document in the database. On large databases it can take a long time and it competes with message ingestion for CPU, RAM, and storage I/O. While ingestion continues, the rebuild takes longer, ingestion and indexing rates are degraded, and the replacement index might never be reported as non-stale, which prevents the automatic swap.

> [!WARNING]
> Until the swap, the database stores both the existing index and its replacement. The replacement usually requires at least as much storage as the index it replaces. Before upgrading, check the index sizes in RavenDB Studio and ensure the database drive has enough free space for both. If the drive runs low on storage, ServiceControl [stops ingesting messages](/servicecontrol/troubleshooting.md#low-on-storage-space).

### Upgrading audit instances one at a time

When audit messages are spread across [multiple audit instances](/servicecontrol/servicecontrol-instances/remotes.md#overview-sharding-audit-messages-with-competing-consumers), upgrade them one at a time so that the rebuild does not have to compete with ingestion and the environment keeps running:

1. Pick an audit instance and [disable audit message ingestion](/servicecontrol/audit-instances/configuration.md#recoverability-servicecontrol-auditingestauditmessages) by setting `ServiceControl.Audit/IngestAuditMessages` to `false`. The instance keeps serving its data to ServicePulse but no longer adds documents to its database and indexes. With competing consumers, the other audit instances keep ingesting from the audit queue.
1. Upgrade the instance. At start-up, RavenDB starts building the replacement indexes.
1. Wait until the rebuild completes. In RavenDB Studio, the **List of Indexes** must no longer show `ReplacementOf/<index>` entries, and the [stale indexes](/servicecontrol/troubleshooting.md#stale-indexes) custom check must pass. See [accessing the ServiceControl database](/servicecontrol/ravendb/accessing-database.md) for how to open RavenDB Studio.
1. Enable ingestion again by setting `ServiceControl.Audit/IngestAuditMessages` to `true`, or by removing the setting, and restart the instance.
1. Repeat with the next audit instance.

Start with the smallest instance to estimate how long the rebuild takes on the larger ones.

### Reducing the database size before rebuilding a locked index

When the index is [locked](/servicecontrol/ravendb/search-engine.md#migrating-existing-indexes-to-lucene-lock-the-index), the upgrade does not apply the new definition and the index is not rebuilt until it is unlocked. As an alternative to the procedure above, the rebuild can be postponed until a large part of the database has been purged, which makes the rebuild shorter. For each audit instance, one at a time:

1. [Disable audit message ingestion](/servicecontrol/audit-instances/configuration.md#recoverability-servicecontrol-auditingestauditmessages) by setting `ServiceControl.Audit/IngestAuditMessages` to `false`, and upgrade the instance.
1. Wait until a significant amount of audit messages has been purged because their [retention period](/servicecontrol/audit-instances/configuration.md#data-retention-servicecontrol-auditauditretentionperiod) expired. Expired messages are deleted while ingestion is disabled.
1. Unlock the index in RavenDB Studio by changing `🔒 Locked (ignore)` to `🔓 Unlocked`.
1. Restart the instance. At start-up, RavenDB starts building the replacement index.
1. Wait until the rebuild completes, as described in step 3 of the procedure above.
1. Enable ingestion again by setting `ServiceControl.Audit/IngestAuditMessages` to `true`, or by removing the setting, and restart the instance.
1. Repeat with the next audit instance.

The instance is not ingesting for a longer time, up to the full retention period. Messages that have been purged are no longer available in ServicePulse. On versions 6.20.0 and 6.21.0, the rebuild resets an index that was migrated to Lucene to Corax, see [Index reset to Corax after unlocking](/servicecontrol/ravendb/search-engine.md#index-reset-to-corax-after-unlocking).

> [!NOTE]
> When each audit instance ingests from its own audit queue, for example with [split audit queues](/servicecontrol/servicecontrol-instances/remotes.md#overview-sharding-audit-messages-with-split-audit-queues), messages build up in the queue of the instance that has ingestion disabled until ingestion is enabled again. Ensure the queue does not run out of storage space. The same applies to an environment with a single audit instance.

For error instances, the equivalent setting is [`ServiceControl/IngestErrorMessages`](/servicecontrol/servicecontrol-instances/configuration.md#recoverability-servicecontrolingesterrormessages). While error ingestion is disabled, failed messages build up in the error queue.

## Downgrading ServiceControl

This section contains information about downgrading ServiceControl instances and the ServiceControl Management Utility.
Expand Down
Loading