diff --git a/docs/migration/ravendb-to-sql-migration-instructions.md b/docs/migration/ravendb-to-sql-migration-instructions.md index bf7a425abc..88be796b03 100644 --- a/docs/migration/ravendb-to-sql-migration-instructions.md +++ b/docs/migration/ravendb-to-sql-migration-instructions.md @@ -31,13 +31,13 @@ Make sure that your setup is supported: Get the data ready: -- Resolve or archive as many unresolved failed messages as you can. They never age out, and ServiceControl stays closed until they are all copied. **A large backlog makes the outage long**. +- Resolve or archive as many unresolved failed messages as you can. They never age out, and ServiceControl stays closed until they are all copied. **A large backlog makes the outage long**. The copy reads archived and resolved messages only after ServiceControl opens. The exception is a RavenDB index that is broken or leaves out some failed messages: then the copy reads every failed message during the outage, logs a warning, and a large archive makes the outage longer too. - Run `--import-failed-errors` while the instance is still on RavenDB. A failed error import is a message that ServiceControl took off the error queue but failed to store. The migration copies these either way, but imported ones arrive as ordinary failed messages. Keep these settings as they are: - Keep `ServiceControl/RetryHistoryDepth` above 0. If it is 0 or less, the migration refuses to start. At that value, the first completed retry after the move deletes all the copied retry history. -- Do not change `ServiceControl/ErrorRetentionPeriod` or `ServiceControl/EventRetentionPeriod` until the migration is finished. The migration uses them to decide which rows are too old to copy. If one changes after the copy starts, ServiceControl refuses to start. +- Do not change `ServiceControl/ErrorRetentionPeriod` or `ServiceControl/EventRetentionPeriod` until the migration is finished. The migration uses them to decide which rows are too old to copy. If one changes after an optional category has started copying with its default window, ServiceControl refuses to start and names the category. ## Keep the RavenDB settings @@ -110,9 +110,9 @@ By default, the migration copies all the optional data that RavenDB still holds ## Step 4: Run the dry run -The dry run reads RavenDB and tells you what the migration will do, including a range for how long ServiceControl will be closed. It does not write anything. Run it with the `--migration-dry-run` argument, in the same way as the source report. [The overview](ravendb-to-sql-migration-overview.md#the-dry-run) lists what it reports. +The dry run reads RavenDB and tells you what the migration will do, including an estimate of how long ServiceControl will be closed. It does not write anything. Run it with the `--migration-dry-run` argument, in the same way as the source report. [The overview](ravendb-to-sql-migration-overview.md#the-dry-run) lists what it reports. -Book the outage from the range in the report. The range is the shortest time the outage can take, not a promise. If the report says that a required category will end Failed and that a retry cannot fix it, plan to abandon that category after it fails. +Book the outage from the estimate in the report, and add a margin. It is an estimate, not a promise. If the report says that a required category will end Failed and that a retry cannot fix it, plan to abandon that category after it fails. ## Step 5: Back up RavenDB diff --git a/docs/migration/ravendb-to-sql-migration-overview.md b/docs/migration/ravendb-to-sql-migration-overview.md index 613dca1680..38061deaae 100644 --- a/docs/migration/ravendb-to-sql-migration-overview.md +++ b/docs/migration/ravendb-to-sql-migration-overview.md @@ -20,7 +20,8 @@ This page describes the migration as designed, not what is built today. [The ins - **What the instance needs to open is copied while it is closed.** This is the *required* data. Most of it is small. - **Everything else is copied in the background while it runs.** This is the *optional* data: the event log and archived and resolved messages. - **Anything not copied ages out of RavenDB**, and the customer deletes the old database when ready. -- **The main thing that makes the outage long is a big backlog of unresolved failures.** They never age out, they are required, and they can legitimately be months old. The strategy assumes you keep them low by resolving and archiving. The *dry run* tells you which case you are in. +- **The main thing that makes the outage long is a big backlog of unresolved failures.** They almost never age out (see the [6.18 exception](#skipped-rows)), they are required, and they can legitimately be months old. The strategy assumes you keep them low by resolving and archiving. The *dry run* tells you which case you are in. +- **A big archive does not make the outage longer, as long as RavenDB has indexed every failed message.** The copy finds the unresolved failures through a RavenDB index, so while ServiceControl is closed it reads only them. If the index is still catching up, the copy waits for it first. If the index is broken, or leaves out some failed messages, the copy reads every failed message instead, archived and resolved ones included, and logs a warning saying so. Then a big archive does make the outage longer. ```mermaid flowchart LR @@ -184,7 +185,7 @@ There are eighteen *categories*: sixteen required and two optional. ### Data that changes shape -- **Attempt history collapses to the newest attempt.** SQL has no attempts table, so a message that failed five times arrives showing one attempt. This applies to every failed message, whatever its status. +- **Attempt history collapses to the newest attempt.** SQL has no attempts table, so it keeps the newest attempt and the number of attempts. A message that failed five times still shows five attempts and still shows as a repeated failure, but only the newest attempt keeps its headers and exception. This applies to every failed message, whatever its status. - **Subscriptions that differ only in message-type version merge into one row**, because the SQL key leaves the version out. - **On SQL Server, endpoint settings whose names differ only in case merge into one row.** The name column compares without case by default. PostgreSQL keeps both. - The name column's own collation decides, not the database default. A case-sensitive database whose name column was given a case-insensitive collation still merges. @@ -207,22 +208,28 @@ Every skipped row is counted under a reason, and its id is written to the log. | Reason | Kind | Can a retry fix it? | | --- | --- | --- | -| A message body that cannot be read after three tries, or that RavenDB does not hold at all. The whole message is skipped, because a message with no body is worse than none | Fault | Yes, once the cause is fixed | +| A message body that cannot be read after three tries. The whole message is skipped, because a message with no body is worse than none | Fault | Yes, once the cause is fixed | +| A message body that RavenDB does not hold at all, counted as `BodyMissing`. The whole message is skipped, for the same reason | Fault | No. A retry meets the same missing body, so only abandoning gets past it. The one exception is a message that expired during the copy (see below) | | An error from the SQL target | Fault | Yes, once the cause is fixed | | A `UniqueMessageId` that is not a GUID. It is the key, the ServicePulse URL and the retry key, so it is never regenerated | Fault | No | | A failed message with no processing attempts, so nothing to fill its required failure time, failing endpoint and exception from | Fault | No | | A subscription whose message type or address is over 200 characters, the SQL key limit | Fault | No | -| A row missing a value SQL requires, such as a known endpoint with no name | Fault | No | +| A row missing a value SQL requires, such as a known endpoint with no name, or a failed message with no failing endpoint address to retry to | Fault | No | +| A text value too long for its SQL column, such as a failed message id or failing endpoint address over 450 characters. The skip names the column and the length, counted the way the database counts (PostgreSQL by characters, SQL Server by UTF-16 units). A message type is not skipped: it is cut to its last 450 characters, as ingestion does | Fault | No | +| A value of the wrong type, such as a `TimeSent` that is not a date. On PostgreSQL, also a text value holding the character U+0000, which PostgreSQL cannot store | Fault | No | | A blank group comment, which SQL never stores | Harmless | Not needed | | An archived or resolved message, or event log item, already past its retention period | Harmless | Not needed | - **Any fault skip makes the category [Failed](#failed-categories)**, even one. Harmless skips leave it Done. +- **A malformed document is skipped and counted, not a reason to stop the copy on its own.** The rest of the category carries on, so abandoning a category that read to the end and ended Failed this way gives up only those rows. But if the skips pass the [early-stop threshold](#failed-categories), the category stops where it is, and abandoning it then also gives up every row it had not yet read. In that case raise the threshold or fix the cause, and retry. - **Endpoint settings for an endpoint ServiceControl does not know are copied, not skipped.** ServiceControl removes them shortly after it opens, by the same rule it always applies, so the migration does not repeat that rule. - **Rows outside a window are not skips.** They are simply not part of the copy. - **Rows RavenDB expires during the copy are not skips either.** A deleted row is never read, so it is an absence, not a loss. - Expiration only deletes a document carrying `@expires`, and only two kinds ever get one: archived or resolved failed messages, and event log items (`ExpirationManager.cs`). - A failed message loses its expiry whenever it becomes unresolved or retry-issued again: when it fails again, is unarchived, or is retried. - The exception: a message that failed again after being archived or resolved on version 6.18 or earlier kept its old expiry, and upgrading does not remove it. So a few unresolved messages can still expire during the copy. + - A failed message that expires after the copy reads it, but before the copy reads its body, is a skip. It counts as a body RavenDB does not hold, so it is a fault skip and the category ends Failed. For unresolved messages this happens only with a source last written by version 6.18 or earlier. A retry gets past it, because the expired message is no longer there to read. + - If the copy stops, and the message it stopped at expires before the next start, that start cannot carry on. The category stops with an error saying RavenDB holds no document with that id. If RavenDB is the right database, `--migration-retry` gets past it, because a retry reads the category from the start. - Both shrinking categories copy in the background, so the copy has hours to race the sweep. A window shorter than the retention period keeps the copy clear of it, by the difference between the two. ### Startup checks @@ -269,13 +276,14 @@ flowchart LR ### Writing to SQL -- **A failed message is written whole**, with its status unchanged. +- **A failed message is written as one row**, with its status unchanged. Its older attempts are dropped, as described in [Data that changes shape](#data-that-changes-shape). +- **A failed message keeps its failure groups and its last-modified time.** `LastModified` is RavenDB's own `@last-modified`, copied as it is, so failure groups, and lists filtered by when a message last changed, show the same messages as before. - **`UniqueMessageId` keeps its value** but changes type, from a string in RavenDB to a `uniqueidentifier` column in SQL. It is the primary key, the ServicePulse URL, the retry correlation key and the body lookup key at once. - **`StatusChangedAt` is rebuilt.** - - For archived and resolved messages it comes from `@expires`, the only place RavenDB records it. + - For archived and resolved messages it comes from `@expires`, the only place RavenDB records it. `@expires` is the status-change time plus `ServiceControl/ErrorRetentionPeriod`, so the copy subtracts the current value. If you changed that setting since RavenDB stamped a message, its rebuilt time is off by the difference. It still ages out of SQL when RavenDB would have deleted it, because SQL adds the same current value back when it checks the row against the retention period. - For unresolved and retry-issued messages it is the newest attempt's time, because the column is `NOT NULL` and cannot be left empty. That is harmless, because the retention sweep only considers archived and resolved rows. - **Deciding whether a row is past retention needs two retention periods.** The source's turns `@expires` back into the status-change time. The target's current one decides whether that time is past the cutoff. -- **Bodies go through `IBodyStoragePersistence`**, which owns compression and the choice of filesystem, Azure Blob or S3. The copier applies the inline threshold itself, because that threshold lives on the ingestion path. It defaults to 102,400 bytes and is set by `ServiceControl/MaxBodySizeToStore`. +- **Bodies go through `IBodyStoragePersistence`**, which owns compression and the choice of filesystem, Azure Blob or S3. The copier decides inline or external storage with the same rule ingestion uses, because `IBodyStoragePersistence` does not apply it. The size limit for inline storage defaults to 102,400 bytes and is set by `ServiceControl/MaxBodySizeToStore`. - **Throughput rows are written directly, not through the collector, and each day's count is set, not added.** - Throughput is required, so it copies while ServiceControl is closed, before any collector has written to SQL. - Setting makes the category safe to resume after a crash, where adding would double-count. The collector's own path does add (`LicensingDataStore.cs:198`). @@ -372,7 +380,7 @@ Stop ServiceControl, run one of these, then start it again. In a container, run - **A required category can be abandoned once it is Failed or has started copying.** One that never ran cannot, so nothing required is given up blind. If RavenDB is gone before it starts, point back or start over: ServiceControl has not opened, so nothing being served is lost. - **An optional category can be abandoned at any time.** That is how to stop a big optional copy short, or get out when RavenDB is gone. - **Abandoning cascades to categories whose rows hang off it**, and the command names each: throughput history (it points at licensing endpoint records) and endpoint settings (it belongs to known endpoints). Group comments only waits for unresolved failed messages, so it is not abandoned with them. -- **Some skips no retry can fix**: a non-GUID id, a failed message with no processing attempts, a key over 200 characters, a missing required value. The refusal and the dry run say which, so abandon after the first failure rather than retrying. +- **Some skips no retry can fix**: a non-GUID id, a failed message with no processing attempts, a key over 200 characters, a value too long for its column, a value of the wrong type, a missing required value, a body RavenDB does not hold. The refusal and the dry run say which, so abandon after the first failure rather than retrying. The exception is a category that stopped early at the threshold: abandoning it also gives up every row it had not yet read, so raise the threshold and retry first. > [!IMPORTANT] > Abandon is final. Nothing goes back for an abandoned category, so it is right when the data is not worth the outage, and wrong if picked by accident. @@ -422,11 +430,11 @@ The dry run never writes to RavenDB, and can run again later against whatever is - Whether the source is embedded or external, and which server. - Both RavenDB database names, and the setting each came from. - Rows and body volume per category, counting only what is inside each window. -- **A range for how long ServiceControl will be closed. The range is a floor, not a promise:** it times counting the rows and sizing the bodies, not reading every row and body or writing to SQL, so the required copy can take longer. +- An estimate of how long ServiceControl will be closed. - **A duration for each optional category**, timed from a sample of its rows and bodies plus the pause between batches. ServiceControl is open while these copy. - The result of the same startup checks a real start runs, so a missing setting surfaces before anyone books an outage. - How many failed error imports there are, with advice to run `--import-failed-errors` on RavenDB first. They are copied either way, but ones imported first arrive as ordinary failed messages. -- Rows that would be skipped or merged, by reason, and which of those no retry can fix. If a required category holds any, it will end Failed, so plan to abandon it after its first failure. +- Rows that would be skipped or merged, by reason, and which of those no retry can fix. If a required category holds any, it will end Failed, so plan to abandon it after its first failure, unless that failure stopped it early at the threshold. It says nothing about load on the source. @@ -493,4 +501,4 @@ It says nothing about load on the source. ## Further reading - [The instructions](ravendb-to-sql-migration-instructions.md): how an operator runs the migration. -- [How the migration is put together](ravendb-to-sql-migration-system-design.md): which class does what, and what is built so far. +- [How the migration is put together](ravendb-to-sql-migration-system-design.md): which class does what, as it will be when it ships. diff --git a/docs/migration/ravendb-to-sql-migration-system-design.md b/docs/migration/ravendb-to-sql-migration-system-design.md index 2e8d087297..e57db50044 100644 --- a/docs/migration/ravendb-to-sql-migration-system-design.md +++ b/docs/migration/ravendb-to-sql-migration-system-design.md @@ -164,8 +164,8 @@ Windowing: `MigrationEngineOptions.WindowFor` supplies the window and the reader | Ordering | `MigrationCategory.MustFollow`, from `MigrationCategoryRegistry.All` | Data, not code. A follower whose predecessor is not `IsFinished` is upserted `Blocked` with `LastError` naming it | | Start or resume | Engine: `NotStarted`, `Blocked`, or any row with `LastError` moves to `InProgress`, cursor kept | `Halted` never resumes by itself; only `--migration-retry` resets it | | Batch size | `EFCoreMigrationTarget.BatchSizeFor` via the category's `IMigrationCategoryWriter` | Writers use `RowsPerStatementFor` from the dialect's parameter budget; `StoreKeyedInsert`-only writers use a fixed 500 | -| Read after cursor | `IMigrationCategoryReader`, usually via `RavenDocumentStream.ByPrefix`; single-document readers load by id; licensing settings readers use `LicensingSettingsSource` | No-tracking session. A cursor whose document no longer exists is refused, because RavenDB would otherwise resume after the missing id and skip rows. The cursor advances on every document seen | -| Fetch bodies | `MigrationBodyLoader` over `IMigrationSource.ReadBody` | Rows with a null body only, 8 concurrent, `MigrationEngine.MaxBodyReadAttempts` (3) with `MigrationEngineOptions.BodyRetryBackoff` (200 ms). `IsDefect` exceptions such as `NotSupportedException` are not retried and reach the batch catch. A missing attachment is skipped as `BodyUnreadable` after one attempt, since ingestion writes an attachment for every failed message. Body skips stay off the checkpoint until the write commits | +| Read after cursor | `IMigrationCategoryReader`, usually via `RavenDocumentStream.ByPrefix`; `FailedMessagesReader` streams a query on `FailedMessageViewIndex` ordered by `id()`, after waiting for the index to be up to date, and falls back to `ByPrefix` when the index is errored, disabled or holds fewer failed messages with a copied status than the collection does; single-document readers load by id; licensing settings readers use `LicensingSettingsSource` | No-tracking session. A cursor whose document no longer exists is refused, because RavenDB would otherwise resume after the missing id and skip rows. The cursor advances on every document seen. Both unresolved paths resume on the same document id cursor | +| Fetch bodies | `MigrationBodyLoader` over `IMigrationSource.ReadBody` | Rows with a null body only, 8 concurrent, `MigrationEngine.MaxBodyReadAttempts` (3) with `MigrationEngineOptions.BodyRetryBackoff` (200 ms). `IsDefect` exceptions such as `NotSupportedException` are not retried and reach the batch catch. A missing attachment is skipped as `BodyMissing` after one attempt, a permanent fault no retry fixes, since ingestion writes an attachment for every failed message. Body skips stay off the checkpoint until the write commits | | Map documents to rows | `IMigrationCategoryWriter.Prepare`, returning `PreparedBatch` | Only the writer knows `NOT NULL` columns, key caps and what the product deletes anyway. Rows are inserted later inside the target transaction. Body-carrying writers write external bodies here, before the transaction, only for messages the target does not hold, so a crash leaves orphan bodies, never dangling references | | Insert and checkpoint | `EFCoreMigrationTarget.Write`: `AccountForEveryRow`, then in one transaction under the execution strategy `PreparedBatch.Insert`, `AlreadyPresentIn`, `MigrationCheckpoint.Extend`, `MigrationCheckpointExtensions.UpsertCheckpoint` | Most writers use the dialect's `InsertMissing` built on `MigrationInsert`. Store-keyed rows use `StoreKeyedInsert`, because `MigrationInsert` rejects store-generated keys. Throughput days use `SetLicensingEndpointThroughput`. Each attempt clears the change tracker. Skips and merges are logged after commit, and `Extend` adds the batch's `PreparedBatch.Merges` to `MergedCount`. See [progress and checkpoints](ravendb-to-sql-migration-overview.md#progress-and-checkpoints) | | Post-commit reconciliation | Engine compares `Write`'s saved checkpoint with the counts it restated | Mismatch settles `Halted`, either kind | diff --git a/src/ServiceControl.Migration.AcceptanceTests/FailedMessageBodyMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/FailedMessageBodyMigrationTests.cs new file mode 100644 index 0000000000..b0ed4e2e08 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/FailedMessageBodyMigrationTests.cs @@ -0,0 +1,185 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Operations.BodyStorage; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.Tests; + +[TestFixture] +[NonParallelizable] +class FailedMessageBodyMigrationTests : MigrationAcceptanceTest +{ + [Test] + public async Task A_small_text_body_stays_inline_and_is_never_written_externally() + { + var failure = new IngestedFailure { ContentType = "text/xml" }; + var content = Encoding.UTF8.GetBytes("1"); + await SeedSourceWithBody(failure.ToFailedMessage(), content, "text/xml"); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + var fetched = await BodyStorage.TryFetch(failure.UniqueMessageId.ToString()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyText, Is.EqualTo("1")); + Assert.That(row.BodyStoredExternally, Is.False); + Assert.That(row.BodySize, Is.EqualTo(content.Length)); + Assert.That(row.BodyContentType, Is.EqualTo("text/xml")); + Assert.That(RecordedBodies.Written, Is.Empty); + Assert.That(fetched.State, Is.EqualTo(MessageBodyState.Available)); + Assert.That(fetched.Content.ContentType, Is.EqualTo("text/xml")); + Assert.That(await ReadAll(fetched.Content.Stream), Is.EqualTo(content)); + } + } + + [Test] + public async Task A_large_text_body_leaves_a_search_prefix_inline_and_writes_the_whole_body_externally() + { + var failure = new IngestedFailure { ContentType = "application/json" }; + var text = new string('a', EFSettings.BodyStorage.MaxBodySizeToStore * 2); + var content = Encoding.UTF8.GetBytes(text); + await SeedSourceWithBody(failure.ToFailedMessage(), content, "application/json"); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyText, Is.Not.Null, "an over-cap text body still keeps a prefix inline so that message search can find it"); + Assert.That(text, Does.StartWith(row.BodyText)); + Assert.That(row.BodyText.Length, Is.EqualTo(EFSettings.BodyStorage.MaxBodySizeToStore)); + Assert.That(row.BodyStoredExternally, Is.True); + Assert.That(row.BodySize, Is.EqualTo(content.Length)); + } + + var written = RecordedBodies.Written.Single(); + var fetched = await BodyStorage.TryFetch(failure.UniqueMessageId.ToString()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(written.BodyId, Is.EqualTo(failure.UniqueMessageId.ToString())); + Assert.That(written.Body, Is.EqualTo(content)); + Assert.That(written.ContentType, Is.EqualTo("application/json")); + Assert.That(fetched.State, Is.EqualTo(MessageBodyState.Available)); + Assert.That(fetched.Content.ContentType, Is.EqualTo("application/json")); + Assert.That(await ReadAll(fetched.Content.Stream), Is.EqualTo(content)); + } + } + + [Test] + public async Task A_binary_body_goes_external_with_nothing_left_inline() + { + var failure = new IngestedFailure { ContentType = "application/x-protobuf" }; + var content = new byte[] { 0x08, 0x00, 0x96, 0x01, 0xFF }; + await SeedSourceWithBody(failure.ToFailedMessage(), content, "application/x-protobuf"); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + var written = RecordedBodies.Written.Single(); + var fetched = await BodyStorage.TryFetch(failure.UniqueMessageId.ToString()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyText, Is.Null, "there is nothing in a binary payload for message search to index"); + Assert.That(row.BodyStoredExternally, Is.True); + Assert.That(row.BodySize, Is.EqualTo(content.Length)); + Assert.That(row.BodyContentType, Is.EqualTo("application/x-protobuf")); + Assert.That(written.BodyId, Is.EqualTo(failure.UniqueMessageId.ToString())); + Assert.That(written.Body, Is.EqualTo(content)); + Assert.That(written.ContentType, Is.EqualTo("application/x-protobuf")); + Assert.That(fetched.State, Is.EqualTo(MessageBodyState.Available)); + Assert.That(fetched.Content.ContentType, Is.EqualTo("application/x-protobuf")); + Assert.That(await ReadAll(fetched.Content.Stream), Is.EqualTo(content)); + } + } + + // Covers the missing-attachment path (BodyMissing, one read) only; the retried BodyUnreadable path is covered by the loader unit tests. + [Test] + public async Task A_message_with_no_body_attachment_is_left_out_entirely() + { + var readable = new IngestedFailure(); + var missing = new IngestedFailure(); + + await SeedSourceWithBody(readable.ToFailedMessage(), Encoding.UTF8.GetBytes("1"), "text/xml"); + await SeedSource(($"FailedMessages/{missing.UniqueMessageId}", missing.ToFailedMessage())); + + var result = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var checkpoint = await ReadCheckpoint(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await FindFailedMessage(readable.UniqueMessageId), Is.Not.Null); + Assert.That(await FindFailedMessage(missing.UniqueMessageId), Is.Null, "a row with no body would dispatch an empty message if anyone retried it"); + Assert.That(result.Skipped, Is.EqualTo(1)); + Assert.That(checkpoint.SkipReasons, Contains.Key(MigrationSkipReason.BodyMissing), "the skip is counted under the reason that tells the operator no retry can fix it"); + Assert.That(checkpoint.SkipReasons, Does.Not.ContainKey(MigrationSkipReason.BodyUnreadable)); + } + } + + [Test] + public async Task A_zero_length_body_is_copied_rather_than_treated_as_missing() + { + var failure = new IngestedFailure(); + await SeedSourceWithBody(failure.ToFailedMessage(), [], "text/plain"); + + var result = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Skipped, Is.Zero); + Assert.That(row.BodySize, Is.Zero); + Assert.That(row.BodyText, Is.Null, "an empty body is stored nowhere, which is not the same as being stored externally"); + Assert.That(row.BodyStoredExternally, Is.False); + Assert.That(RecordedBodies.Written, Is.Empty); + } + } + + [Test] + public async Task The_copied_body_reads_back_through_the_products_own_body_storage() + { + var inline = new IngestedFailure { ContentType = "text/xml" }; + var external = new IngestedFailure { ContentType = "application/x-protobuf" }; + var inlineContent = Encoding.UTF8.GetBytes("1"); + var externalContent = new byte[] { 0x08, 0x00, 0x96, 0x01, 0xFF }; + + await SeedSourceWithBodies( + (inline.ToFailedMessage(), inlineContent, "text/xml"), + (external.ToFailedMessage(), externalContent, "application/x-protobuf")); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + Assert.That(await Fetch(inline.UniqueMessageId), Is.EqualTo(inlineContent)); + Assert.That(await Fetch(external.UniqueMessageId), Is.EqualTo(externalContent), "the body was stored under an id TryFetch does not resolve"); + } + + async Task Fetch(Guid uniqueMessageId) + { + var result = await BodyStorage.TryFetch(uniqueMessageId.ToString()); + + Assert.That(result.State, Is.EqualTo(MessageBodyState.Available)); + + return await ReadAll(result.Content.Stream); + } + + static async Task ReadAll(Stream stream) + { + await using (stream) + { + using var buffer = new MemoryStream(); + await stream.CopyToAsync(buffer); + return buffer.ToArray(); + } + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs b/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs index adf724538b..6574e6eced 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Migration.AcceptanceTests; using System; using System.Collections.Generic; using System.Globalization; +using System.IO; using System.Linq; using System.Net.Http; using System.Net.Http.Json; @@ -22,8 +23,10 @@ namespace ServiceControl.Migration.AcceptanceTests; using ServiceControl.AcceptanceTesting.InfrastructureConfig; using ServiceControl.Hosting.Commands; using ServiceControl.Infrastructure.WebApi; +using ServiceControl.MessageFailures; using ServiceControl.Migration.Checks; using ServiceControl.Operations; +using ServiceControl.Operations.BodyStorage; using ServiceControl.Persistence; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.EFCore.Abstractions; @@ -70,6 +73,10 @@ public async Task SetUp() SetSourceVariable("LICENSINGCOMPONENT_RAVENDB_THROUGHPUTDATABASENAME", Source.ThroughputDatabase); SetSourceVariable("SERVICECONTROL_ERRORRETENTIONPERIOD", "10.00:00:00"); SetSourceVariable("SERVICECONTROL_MIGRATION_ENABLED", "true"); +#if DEBUG + // Cleared because a developer who set it to smoke test the migration would let every test past the release check, which only the test host's own AllowUnreleasedMigration registration may do. + SetSourceVariable(RunCommand.AllowUnreleasedMigrationVariable, null); +#endif // Settings.Port has no public setter, so the port has to come in as the environment variable a customer would set. SetSourceVariable("SERVICECONTROL_PORT", PortUtility.GetAssignedOrAvailablePort(33500).ToString(CultureInfo.InvariantCulture)); @@ -149,6 +156,33 @@ protected async Task SeedSource(string database, params (string Id, object Docum protected Task SeedSource(params (string Id, object Document)[] documents) => SeedSource(Source.PrimaryDatabase, documents); + protected Task SeedSourceWithBody(FailedMessage document, byte[] body, string contentType) => + SeedSourceWithBodies((document, body, contentType)); + + protected async Task SeedSourceWithBodies(params (FailedMessage Document, byte[] Body, string ContentType)[] messages) + { + using (var session = SourceStore.OpenAsyncSession(Source.PrimaryDatabase)) + { + foreach (var (document, _, _) in messages) + { + await session.StoreAsync(document, $"FailedMessages/{document.UniqueMessageId}"); + } + + await session.SaveChangesAsync(); + } + + using (var session = SourceStore.OpenAsyncSession(Source.PrimaryDatabase)) + { + foreach (var (document, body, contentType) in messages) + { + // "body" is the attachment name the RavenDB persister reads, retyped because this project cannot reference that persister. + session.Advanced.Attachments.Store($"FailedMessages/{document.UniqueMessageId}", "body", new MemoryStream(body), contentType); + } + + await session.SaveChangesAsync(); + } + } + protected Task SeedSourceEndpointSettings(params (string Name, bool TrackInstances)[] settings) => SeedSource([.. settings.Select(setting => ($"EndpointSettings/{DeterministicGuid.MakeId(setting.Name)}", (object)new EndpointSettings { Name = setting.Name, TrackInstances = setting.TrackInstances }))]); @@ -402,6 +436,8 @@ protected Task FindFailedMessage(Guid uniqueMessageId) => protected EFPersisterSettings EFSettings => targetServices.GetRequiredService(); + protected IBodyStorage BodyStorage => targetServices.GetRequiredService(); + protected Task Ingest(params IngestedFailure[] failures) => InBatch(async unitOfWork => { diff --git a/src/ServiceControl.Migration.AcceptanceTests/When_a_batch_exceeds_one_statement.cs b/src/ServiceControl.Migration.AcceptanceTests/When_a_batch_exceeds_one_statement.cs new file mode 100644 index 0000000000..c3d218938d --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/When_a_batch_exceeds_one_statement.cs @@ -0,0 +1,32 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.Tests; + +// Proves a copy larger than one batch, where one batch is as many rows as one insert statement takes. +[TestFixture] +[NonParallelizable] +class When_a_batch_exceeds_one_statement : MigrationAcceptanceTest +{ + [Test] + public async Task Copies_more_messages_than_fit_in_one_statement() + { + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + var batchSize = await Target.BatchSizeFor(category); + var failures = Enumerable.Range(0, (batchSize * 2) + 1).Select(_ => new IngestedFailure()).ToArray(); + + await SeedSourceWithBodies([.. failures.Select(failure => (failure.ToFailedMessage(), failure.Body, failure.ContentType))]); + + var result = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + Assert.That(result.Copied, Is.EqualTo(failures.Length)); + + foreach (var failure in failures) + { + Assert.That(await FindFailedMessage(failure.UniqueMessageId), Is.Not.Null); + } + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs b/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs new file mode 100644 index 0000000000..88650bf499 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs @@ -0,0 +1,339 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net.Http.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using NUnit.Framework; +using Raven.Client.Documents.Operations; +using ServiceControl.Hosting.Commands; +using ServiceControl.Infrastructure.WebApi; +using ServiceControl.MessageFailures; +using ServiceControl.MessageFailures.Api; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.Tests; + +[TestFixture] +[NonParallelizable] +class When_unresolved_failed_messages_are_migrated : MigrationAcceptanceTest +{ + [Test] + public async Task Fills_every_column_the_newest_attempt_supplies() + { + var failure = new IngestedFailure { FailingEndpointAddress = "Sales@MACHINE" }; + await SeedFailure(failure); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.MessageId, Is.EqualTo(failure.MessageId)); + Assert.That(row.MessageType, Is.EqualTo(failure.MessageType)); + Assert.That(row.TimeSent, Is.EqualTo(failure.TimeSent)); + Assert.That(row.ConversationId, Is.EqualTo(failure.ConversationId)); + Assert.That(row.IsSystemMessage, Is.False); + Assert.That(row.SendingEndpointName, Is.EqualTo(failure.SendingEndpoint.Name)); + Assert.That(row.SendingEndpointHostId, Is.EqualTo(failure.SendingEndpoint.HostId)); + Assert.That(row.SendingEndpointHost, Is.EqualTo(failure.SendingEndpoint.Host)); + Assert.That(row.ReceivingEndpointName, Is.EqualTo(failure.ReceivingEndpoint.Name)); + Assert.That(row.ReceivingEndpointHostId, Is.EqualTo(failure.ReceivingEndpoint.HostId)); + Assert.That(row.ReceivingEndpointHost, Is.EqualTo(failure.ReceivingEndpoint.Host)); + Assert.That(row.ExceptionType, Is.EqualTo(failure.ExceptionType)); + Assert.That(row.ExceptionMessage, Is.EqualTo(failure.ExceptionMessage)); + Assert.That(row.FailingEndpointAddress, Is.EqualTo("Sales@MACHINE")); + Assert.That(row.HeadersJson, Does.Contain(failure.MessageId)); + } + } + + [Test] + public async Task Collapses_attempt_history_to_the_newest_attempt() + { + var failure = new IngestedFailure(); + await SeedFailure(failure, numberOfAttempts: 5); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.NumberOfProcessingAttempts, Is.EqualTo(5), "the count survives even though the attempts do not"); + Assert.That(row.LastAttemptedAt, Is.EqualTo(failure.AttemptedAt)); + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(failure.AttemptedAt.AddMinutes(-4))); + Assert.That(row.LastTimeOfFailure, Is.EqualTo(failure.TimeOfFailure)); + } + } + + [Test] + public async Task A_resumed_copy_leaves_the_rows_the_first_run_wrote() + { + var failure = new IngestedFailure(); + await SeedFailure(failure); + + var first = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + await QueryTarget(async db => + { + var existing = await db.FailedMessages.SingleAsync(row => row.UniqueMessageId == failure.UniqueMessageId); + existing.ExceptionMessage = "Edited after the first copy, the way a live instance would have"; + await db.SaveChangesAsync(); + return existing; + }); + + await DeleteCheckpoint(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var second = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var row = await GetFailedMessage(failure.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(first.Copied, Is.EqualTo(1)); + Assert.That(row.ExceptionMessage, Is.EqualTo("Edited after the first copy, the way a live instance would have"), "the copier overwrote a row that was already in the target"); + Assert.That(second.Copied, Is.Zero); + Assert.That(second.AlreadyPresent, Is.EqualTo(1), "a row the target already holds is a third outcome"); + Assert.That(second.Skipped, Is.Zero, "counting it as a skip would push a healthy category toward the halt threshold"); + Assert.That(second.SkippedIds, Is.Empty); + } + } + + [Test] + public async Task Counts_only_the_statuses_this_category_owns() + { + var unresolved = new IngestedFailure(); + var retryIssued = new IngestedFailure(); + await SeedFailure(unresolved); + await SeedFailure(retryIssued, FailedMessageStatus.RetryIssued); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + // Written straight into the target, because this build has no category that copies archived messages. + await QueryTarget(async db => + { + db.FailedMessages.Add(new FailedMessageEntity + { + UniqueMessageId = Guid.NewGuid(), + Status = FailedMessageStatus.Archived, + HeadersJson = "{}", + FailingEndpointAddress = "error" + }); + + return await db.SaveChangesAsync(); + }); + + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await Target.Count(category), Is.EqualTo(2), "the category's count takes in the archive's rows unless it honours the category's statuses"); + Assert.That(await QueryTarget(db => db.FailedMessages.CountAsync()), Is.EqualTo(3), "the filter is doing the work, not an empty table"); + } + } + + [Test] + public async Task A_copied_message_keeps_its_failure_groups() + { + var failure = new IngestedFailure(); + await SeedFailure(failure); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var groups = await QueryTarget(db => db.FailedMessageGroups + .AsNoTracking() + .Where(group => group.FailedMessageUniqueId == failure.UniqueMessageId) + .ToListAsync()); + var expected = failure.Groups.Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(groups, Has.Count.EqualTo(1), "a message copied without its group rows is missing from every group view"); + Assert.That(groups.Single().GroupId, Is.EqualTo(expected.Id)); + Assert.That(groups.Single().Title, Is.EqualTo(expected.Title)); + Assert.That(groups.Single().Type, Is.EqualTo(expected.Type)); + } + } + + [Test] + public async Task A_message_already_in_the_target_gets_no_group_rows_from_the_copy() + { + var failure = new IngestedFailure(); + await SeedFailure(failure); + + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + await QueryTarget(db => db.FailedMessageGroups.Where(group => group.FailedMessageUniqueId == failure.UniqueMessageId).ExecuteDeleteAsync()); + await DeleteCheckpoint(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var second = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var groups = await QueryTarget(db => db.FailedMessageGroups.AsNoTracking().CountAsync(group => group.FailedMessageUniqueId == failure.UniqueMessageId)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(second.AlreadyPresent, Is.EqualTo(1)); + Assert.That(groups, Is.Zero, "the copy wrote RavenDB's groups onto a message it did not insert"); + } + } + + [Test] + public async Task Answers_for_a_copied_message_on_the_URL_it_had_on_RavenDB() + { + var failure = new IngestedFailure(); + await SeedFailure(failure); + + await RunHostUntilTheApiAnswers(); + + var response = await HttpClient.GetAsync($"api/errors/last/{failure.UniqueMessageIdString}"); + + Assert.That(response.IsSuccessStatusCode, Is.True, "the bookmark a customer had on RavenDB stopped resolving"); + + var fromApi = await response.Content.ReadFromJsonAsync(SerializerOptions.Default); + + using (Assert.EnterMultipleScope()) + { + Assert.That(fromApi.Id, Is.EqualTo(failure.UniqueMessageIdString)); + Assert.That(fromApi.Status, Is.EqualTo(FailedMessageStatus.Unresolved)); + Assert.That(fromApi.MessageType, Is.EqualTo(failure.MessageType)); + Assert.That(fromApi.ReceivingEndpoint.Name, Is.EqualTo(failure.ReceivingEndpoint.Name)); + Assert.That(fromApi.QueueAddress, Is.EqualTo(failure.QueueAddress)); + Assert.That(fromApi.NumberOfProcessingAttempts, Is.EqualTo(1)); + } + } + + [Test] + public async Task The_required_copy_completes_the_category_before_the_host_opens() + { + var failures = Enumerable.Range(0, 3).Select(_ => new IngestedFailure()).ToArray(); + + foreach (var failure in failures) + { + await SeedFailure(failure); + } + + await RunHostUntilTheApiAnswers(); + + var checkpoint = await ReadCheckpoint("UnresolvedAndRetryIssuedFailedMessages"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(checkpoint.State, Is.EqualTo(MigrationCategoryState.Complete)); + Assert.That(checkpoint.CopiedCount, Is.EqualTo(3)); + Assert.That(checkpoint.SkippedCount, Is.Zero); + Assert.That(checkpoint.SourceTotal, Is.Null, "the engine no longer counts the source before copying"); + } + } + + [Test] + public async Task Records_each_skip_under_the_reason_that_caused_it() + { + var good = new IngestedFailure(); + await SeedFailure(good); + + var unparseable = new IngestedFailure(); + var unparseableDocument = unparseable.ToFailedMessage(); + unparseableDocument.UniqueMessageId = "not-a-guid"; + await SeedSourceWithBody(unparseableDocument, unparseable.Body, unparseable.ContentType); + + var attemptless = new IngestedFailure(); + var attemptlessDocument = attemptless.ToFailedMessage(); + attemptlessDocument.ProcessingAttempts.Clear(); + await SeedSourceWithBody(attemptlessDocument, attemptless.Body, attemptless.ContentType); + + using var cancellation = new CancellationTokenSource(TimeSpan.FromMinutes(3)); + + var refused = Assert.ThrowsAsync(async () => await RunCommand.Run(Settings, AllowingAnUnreleasedMigration(), cancellation.Token)); + + var checkpoint = await ReadCheckpoint("UnresolvedAndRetryIssuedFailedMessages"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(refused.Message, Does.Contain("UnresolvedAndRetryIssuedFailedMessages").And.Contain("--migration-retry").And.Contain("--migration-abandon"), "a fault skip leaves a required category Failed, so the host stays closed and names both commands"); + Assert.That(checkpoint.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors)); + Assert.That(checkpoint.CopiedCount, Is.EqualTo(1)); + Assert.That(checkpoint.SkippedCount, Is.EqualTo(2)); + Assert.That(checkpoint.SkipReasons, Is.EqualTo(new Dictionary + { + [MigrationSkipReason.UnparseableUniqueMessageId] = 1, + [MigrationSkipReason.NoProcessingAttempts] = 1 + })); + } + } + + [Test] + public async Task A_document_whose_TimeSent_cannot_be_read_is_skipped_as_malformed_and_one_whose_host_id_is_not_a_GUID_copies_with_the_empty_host_id() + { + var good = new IngestedFailure(); + await SeedFailure(good); + + var badTimeSent = new IngestedFailure(); + var badTimeSentDocument = badTimeSent.ToFailedMessage(); + badTimeSentDocument.ProcessingAttempts[^1].MessageMetadata["TimeSent"] = "garbage"; + + var badHostId = new IngestedFailure(); + var badHostIdDocument = badHostId.ToFailedMessage(); + var attempt = badHostIdDocument.ProcessingAttempts[^1]; + attempt.Headers[NServiceBus.Headers.HostId] = "not-a-guid"; + attempt.Headers["NServiceBus.ProcessingMachine"] = "MACHINE"; + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + await SeedSourceWithBodies( + (badTimeSentDocument, badTimeSent.Body, badTimeSent.ContentType), + (badHostIdDocument, badHostId.Body, badHostId.ContentType)); + + var result = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors), "the copy reaches the end of the source instead of halting at the first bad document"); + Assert.That(result.Copied, Is.EqualTo(2)); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { $"FailedMessages/{badTimeSent.UniqueMessageIdString}" })); + Assert.That(result.Saved.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.MalformedValue] = 1 })); + Assert.That(await QueryTarget(db => db.FailedMessages.Select(row => row.UniqueMessageId).ToListAsync()), Is.EquivalentTo(new[] { good.UniqueMessageId, badHostId.UniqueMessageId })); + var copied = await QueryTarget(db => db.FailedMessages.Where(row => row.UniqueMessageId == badHostId.UniqueMessageId).SingleAsync()); + Assert.That(copied.ReceivingEndpointHostId, Is.EqualTo(Guid.Empty), "ingestion stores an unparseable host id as the empty GUID, so the message copies with it"); + Assert.That(copied.ReceivingEndpointName, Is.EqualTo("Sales"), "the endpoint is rebuilt from the headers, not taken from the stored dictionary"); + Assert.That(copied.ReceivingEndpointHost, Is.EqualTo("MACHINE")); + } + } + + [Test] + public async Task A_failure_stored_without_its_time_is_skipped_when_newest_and_left_out_of_the_failure_times_when_older() + { + var newestHasNoTime = new IngestedFailure(); + var olderHasNoTime = new IngestedFailure(); + await SeedFailure(newestHasNoTime); + await SeedFailure(olderHasNoTime, numberOfAttempts: 2); + await DeleteStoredTimeOfFailure(newestHasNoTime, attempt: 0); + await DeleteStoredTimeOfFailure(olderHasNoTime, attempt: 0); + + var result = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var copied = await GetFailedMessage(olderHasNoTime.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { $"FailedMessages/{newestHasNoTime.UniqueMessageIdString}" }), "a missing time read back as the copy's own clock, so the message copied with a made-up failure time"); + Assert.That(result.Saved.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.RequiredValueMissing] = 1 })); + Assert.That(copied.FirstTimeOfFailure, Is.EqualTo(olderHasNoTime.TimeOfFailure), "the older attempt has no time, so only the newest one counts"); + Assert.That(copied.LastTimeOfFailure, Is.EqualTo(olderHasNoTime.TimeOfFailure), "the copy's own clock would become the last failure time"); + } + } + + // Removes the key itself, because a stored default date is a different case that the source already reads correctly. + Task DeleteStoredTimeOfFailure(IngestedFailure failure, int attempt) => + SourceStore.Operations.ForDatabase(Source.PrimaryDatabase).SendAsync(new PatchOperation( + $"FailedMessages/{failure.UniqueMessageIdString}", + null, + new PatchRequest { Script = $"delete this.ProcessingAttempts[{attempt}].FailureDetails.TimeOfFailure;" })); + + // Seeded with its body, because the copy leaves out a message whose body the source does not hold. + Task SeedFailure(IngestedFailure failure, FailedMessageStatus status = FailedMessageStatus.Unresolved, int numberOfAttempts = 1) => + SeedSourceWithBody(failure.ToFailedMessage(status, numberOfAttempts), failure.Body, failure.ContentType); +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs index fdac0674c0..20c1542231 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs @@ -17,6 +17,12 @@ class PostgreSqlMigrationSqlDialect : PostgreSqlDialect, IMigrationSqlDialect // A fixed chunk, because PostgreSQL's parameter limit is far away and the row width does not bring it closer. public int RowsPerStatement(int parametersPerRow) => MaxRowsPerStatement; + // A varchar(n) column counts characters, which are code points, so a pair of UTF-16 surrogates is one. + public int TextLength(string value) => value.EnumerateRunes().Count(); + + // PostgreSQL text types cannot hold U+0000 in any encoding, and refuse the statement rather than drop it. + public char? RejectedCharacterIn(string value) => value.Contains('\0') ? '\0' : null; + public async Task> InsertMissing(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) where TEntity : class { var insert = MigrationInsert.For(dbContext); @@ -35,6 +41,7 @@ ON CONFLICT ({keyColumns}) DO NOTHING RETURNING {keyColumns} """, chunk, + configureParameter: null, cancellationToken)); } diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs index 45d8ea8454..5de111af7b 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.SqlServer; +using System.Data.Common; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Metadata; @@ -67,6 +68,12 @@ public IEqualityComparer KeyComparer(Type entityType, string propertyNam public int RowsPerStatement(int parametersPerRow) => MaxRowsPerStatement(parametersPerRow); + // An nvarchar(n) column counts byte-pairs, which are UTF-16 units, so a surrogate pair takes two. + public int TextLength(string value) => value.Length; + + // An nvarchar column stores any UTF-16 unit, U+0000 included. + public char? RejectedCharacterIn(string value) => null; + public async Task> InsertMissing(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) where TEntity : class { var insert = MigrationInsert.For(dbContext); @@ -79,10 +86,11 @@ public async Task> InsertMissing(ServiceControlD foreach (var chunk in rows.Chunk(RowsPerStatement(insert.Columns.Count))) { // HOLDLOCK, or two writers can both find a key missing and both insert it. + // FORCESEEK, because on a small table SQL Server scans a whole index instead of seeking the primary key, and HOLDLOCK then locks every row the scan reads. inserted.AddRange(await insert.Execute( dbContext, $""" - MERGE {insert.Table} WITH (HOLDLOCK) AS t + MERGE {insert.Table} WITH (HOLDLOCK, FORCESEEK) AS t USING ( SELECT {columns} FROM ( @@ -98,12 +106,23 @@ WHEN NOT MATCHED THEN INSERT ({columns}) VALUES ({string.Join(", ", insert.Colum OUTPUT {string.Join(", ", insert.KeyColumns.Select(column => $"inserted.{column}"))}; """, chunk, + DeclareUnboundedTextAsMax, cancellationToken)); } return inserted; } + // EF declares unbounded text as nvarchar(4000) or nvarchar(max) by each value's length, and each new mix compiles its own plan. + // Size -1 declares every such value as the nvarchar(max) its column already is. + static void DeclareUnboundedTextAsMax(IProperty property, DbParameter parameter) + { + if (property.ClrType == typeof(string) && property.GetMaxLength() is null) + { + parameter.Size = -1; + } + } + OpenedKeys Keys => OpenedState ?? throw new InvalidOperationException($"The SQL Server migration dialect is not open. Call {nameof(Open)} first."); sealed record OpenedKeys( diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/BodyStorageIsWritableCheck.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/BodyStorageIsWritableCheck.cs index 5a5c116bdf..025cb1fe22 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/BodyStorageIsWritableCheck.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/BodyStorageIsWritableCheck.cs @@ -5,9 +5,9 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; using ServiceControl.Persistence.DataMigration; /// -/// Writes, reads and deletes one probe body, so a body store that is misconfigured stops the copy before it -/// starts. Bodies can live on a file share, in Azure Blob Storage or in S3, and without this the first category -/// that carries bodies would find out part way through and skip every message it could not write. +/// Writes, reads and deletes one probe body, so a misconfigured body store stops the copy before it starts. +/// Bodies can live on a file share, in Azure Blob Storage or in S3, and without this check the first category +/// that carries bodies would only find out part way through, when a body it cannot write fails the batch and stops the category. /// sealed class BodyStorageIsWritableCheck(IBodyStoragePersistence bodyStorage) : IMigrationStartupCheck { diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs index 09510292ff..85c4bea2be 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; using System.Collections.Frozen; +using Abstractions; using Implementation; using Infrastructure; using Microsoft.EntityFrameworkCore; @@ -14,12 +15,18 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; /// any of them beyond that, so a category this build cannot write is simply absent from /// and never reaches the engine. /// -sealed class EFCoreMigrationTarget(IServiceScopeFactory scopeFactory, IMigrationSqlDialect migrationDialect, ILogger logger) : DataStoreBase(scopeFactory), IMigrationTarget +sealed class EFCoreMigrationTarget( + IServiceScopeFactory scopeFactory, + IMigrationSqlDialect migrationDialect, + IBodyStoragePersistence storagePersistence, + EFPersisterSettings settings, + ILogger logger) : DataStoreBase(scopeFactory), IMigrationTarget { readonly FrozenDictionary writers = new IMigrationCategoryWriter[] { new KnownEndpointsWriter(migrationDialect), - new EndpointSettingsWriter(migrationDialect) + new EndpointSettingsWriter(migrationDialect), + new FailedMessagesWriter(migrationDialect, storagePersistence, settings) }.ToFrozenDictionary(writer => writer.CategoryId, StringComparer.Ordinal); public Task Open(CancellationToken cancellationToken = default) => diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapper.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapper.cs new file mode 100644 index 0000000000..3f3346c617 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapper.cs @@ -0,0 +1,198 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using System.Text.Json; +using ServiceControl.MessageFailures; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.EntityConfigurations; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; +using ServiceControl.Persistence.EFCore.Infrastructure; + +static class FailedMessageRowMapper +{ + public static FailedMessageRowMapping Map(MigrationRow source) + { + var document = (FailedMessage)source.Document; + + if (!Guid.TryParse(document.UniqueMessageId, out var uniqueMessageId)) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.UnparseableUniqueMessageId, + $"UniqueMessageId '{document.UniqueMessageId}' on document '{source.SourceId}' is not a GUID and the target column is a uniqueidentifier"); + } + + if (document.ProcessingAttempts is not { Count: > 0 }) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.NoProcessingAttempts, + $"Document '{source.SourceId}' has no processing attempts, so no attempt supplies the required columns"); + } + + if (document.ProcessingAttempts[^1] is not { } newest) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.RequiredValueMissing, + $"The newest attempt on document '{source.SourceId}' is null, so no attempt supplies the required columns"); + } + + if (MissingFrom(newest) is { } missing) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.RequiredValueMissing, + $"The newest attempt on document '{source.SourceId}' has no {missing}, which the row's columns are built from"); + } + + if (MalformedIn(newest.MessageMetadata) is { } malformed) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.MalformedValue, + $"The newest attempt on document '{source.SourceId}' holds a {malformed.Found.FullName} as its {malformed.Key} metadata, where the row needs a {malformed.Expected.FullName}: '{Shown(malformed.Value)}'"); + } + + // An attempt stored without FailureDetails reads back as the default object, whose address is empty. + if (string.IsNullOrEmpty(newest.FailureDetails.AddressOfFailingEndpoint)) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.RequiredValueMissing, + $"FailedMessages.{nameof(FailedMessageEntity.FailingEndpointAddress)} is NOT NULL and the newest attempt on document '{source.SourceId}' has no AddressOfFailingEndpoint"); + } + + if (newest.FailureDetails.TimeOfFailure == default || newest.AttemptedAt == default) + { + return new FailedMessageRowMapping.Skipped( + MigrationSkipReason.RequiredValueMissing, + $"The newest attempt on document '{source.SourceId}' has no {(newest.FailureDetails.TimeOfFailure == default ? nameof(newest.FailureDetails.TimeOfFailure) : nameof(newest.AttemptedAt))}, which the row's failure times are built from"); + } + + var metadata = newest.MessageMetadata; + // The newest attempt passed the checks above, so the list always holds at least one time for Min and Max. + var timesOfFailure = document.ProcessingAttempts + .Where(HasRecordedFailure) + .Select(attempt => attempt.FailureDetails.TimeOfFailure) + .ToList(); + var sendingEndpoint = (EndpointDetails?)metadata.GetValueOrDefault("SendingEndpoint"); + var receivingEndpoint = (EndpointDetails?)metadata.GetValueOrDefault("ReceivingEndpoint"); + + var row = new FailedMessageEntity + { + UniqueMessageId = uniqueMessageId, + Status = document.Status, + StatusChangedAt = ReadResolvedDate(source, MigrationMetadataKeys.StatusChangedAt), + LastModified = ReadResolvedDate(source, MigrationMetadataKeys.LastModified), + NumberOfProcessingAttempts = document.ProcessingAttempts.Count, + FirstTimeOfFailure = AsUtc(timesOfFailure.Min()), + LastTimeOfFailure = AsUtc(timesOfFailure.Max()), + LastAttemptedAt = AsUtc(newest.AttemptedAt), + MessageId = newest.MessageId, + MessageType = LastCharactersThatFit(metadata.GetValueOrDefault("MessageType") as string), + TimeSent = metadata.GetValueOrDefault("TimeSent") is DateTime timeSent ? AsUtc(timeSent) : null, + ConversationId = metadata.GetValueOrDefault("ConversationId") as string, + SendingEndpointName = sendingEndpoint?.Name, + SendingEndpointHostId = sendingEndpoint?.HostId, + SendingEndpointHost = sendingEndpoint?.Host, + ReceivingEndpointName = receivingEndpoint?.Name, + ReceivingEndpointHostId = receivingEndpoint?.HostId, + ReceivingEndpointHost = receivingEndpoint?.Host, + ExceptionType = newest.FailureDetails.Exception?.ExceptionType, + ExceptionMessage = newest.FailureDetails.Exception?.Message, + IsSystemMessage = metadata.GetValueOrDefault("IsSystemMessage") is true, + HeadersJson = MessageHeaders.Write(newest.Headers), + FailingEndpointAddress = newest.FailureDetails.AddressOfFailingEndpoint + }; + + // Null groups and groups with no id are dropped, because GroupId is part of a NOT NULL key. + IReadOnlyList groups = + [ + .. (document.FailureGroups ?? []) + .Where(failureGroup => failureGroup is { Id: not null }) + .Select(failureGroup => new FailedMessageGroupEntity + { + FailedMessageUniqueId = uniqueMessageId, + GroupId = failureGroup.Id, + Title = failureGroup.Title ?? string.Empty, + Type = failureGroup.Type ?? string.Empty + }) + ]; + + return new FailedMessageRowMapping.Mapped(row, groups); + } + + /// + /// Fills the four body columns of a mapped row the way ingestion does, letting + /// decide whether the body stays inline or goes to external storage. Returns true when the body goes to external + /// storage. It changes nothing else on the row and writes no body itself. + /// + public static bool SetBody(FailedMessageEntity row, IReadOnlyDictionary headers, MigrationBody body, int maxBodySizeToStore) + { + var (bodyText, storeExternally) = MessageBodyClassifier.Classify(headers, body.Content, maxBodySizeToStore); + + row.BodyText = bodyText; + row.BodyStoredExternally = storeExternally; + row.BodySize = body.Content.Length; + row.BodyContentType = body.ContentType; + + return storeExternally; + } + + static string? MissingFrom(FailedMessage.ProcessingAttempt attempt) => + attempt.FailureDetails is null ? nameof(attempt.FailureDetails) + : attempt.MessageMetadata is null ? nameof(attempt.MessageMetadata) + : attempt.Headers is null ? nameof(attempt.Headers) + : null; + + // An attempt stored without FailureDetails reads back as a default object with an empty address and a time of failure set to when it was read. + static bool HasRecordedFailure(FailedMessage.ProcessingAttempt? attempt) => + attempt is { FailureDetails: { AddressOfFailingEndpoint: { Length: > 0 } } details } && details.TimeOfFailure != default; + + // A missing value becomes null, just as RavenDB's own read path shows it, so only a value of the wrong type needs a skip. + static (string Key, Type Found, Type Expected, object Value)? MalformedIn(Dictionary metadata) + { + foreach (var (key, expected) in TypedMetadata) + { + if (metadata.GetValueOrDefault(key) is { } value && !expected.IsInstanceOfType(value)) + { + return (key, value.GetType(), expected, value); + } + } + + return null; + } + + // Every metadata value the row reads by type, so a value of another type is skipped rather than copied as null or false. + static readonly (string Key, Type Expected)[] TypedMetadata = + [ + ("TimeSent", typeof(DateTime)), + ("MessageType", typeof(string)), + ("ConversationId", typeof(string)), + ("IsSystemMessage", typeof(bool)) + ]; + + const int MaxShownValueLength = 100; + + // Cut short because a stored value can be any length and the skip detail is one log line. + static string Shown(object value) + { + var text = value as string ?? JsonSerializer.Serialize(value); + + return text.Length > MaxShownValueLength ? $"{text[..MaxShownValueLength]}..." : text; + } + + // Throws rather than make up a date, because a made-up StatusChangedAt or LastModified would give the row a retention or modification time it never had. + static DateTime ReadResolvedDate(MigrationRow source, string key) => + source.Metadata.GetValueOrDefault(key) switch + { + DateTime value => AsUtc(value), + null => throw new InvalidOperationException($"Row '{source.SourceId}' arrived without a {key} value, which the source resolves before a failed message crosses the seam"), + var other => throw new InvalidOperationException($"Row '{source.SourceId}' carries a {other.GetType().FullName} as its {key} value, where the source resolves a {typeof(DateTime).FullName} before a failed message crosses the seam") + }; + + // Npgsql only writes UTC to a timestamp with time zone column, and a value whose kind is Unspecified is already UTC, so it must not be shifted. + // Kept apart from the RavenDB source's copy because this project does not reference it. + static DateTime AsUtc(DateTime value) => + value.Kind == DateTimeKind.Unspecified ? DateTime.SpecifyKind(value, DateTimeKind.Utc) : value.ToUniversalTime(); + + // Uses the same length limit that ingestion applies to the MessageType column. + static string? LastCharactersThatFit(string? value) => + value is { Length: > ColumnLengths.ShortTextLength } ? value[^ColumnLengths.ShortTextLength..] : value; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapping.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapping.cs new file mode 100644 index 0000000000..ee9d6957a5 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageRowMapping.cs @@ -0,0 +1,24 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Entities; + +/// +/// What one failed message document became: either a row with its failure group rows, or a skip with the reason. +/// +abstract record FailedMessageRowMapping +{ + FailedMessageRowMapping() + { + } + + /// + /// The document maps to a failed message row and the group rows keyed by it. + /// + public sealed record Mapped(FailedMessageEntity Row, IReadOnlyList Groups) : FailedMessageRowMapping; + + /// + /// The document cannot become a row, for the reason given, and says which document and why. + /// + public sealed record Skipped(MigrationSkipReason Reason, string Detail) : FailedMessageRowMapping; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/IMigrationCategoryWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/IMigrationCategoryWriter.cs index 3349a45a38..82f6d827c1 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/IMigrationCategoryWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/IMigrationCategoryWriter.cs @@ -9,7 +9,9 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; /// interface IMigrationCategoryWriter { - /// The category this writer handles, named as in . + /// + /// The category this writer handles, named as in . + /// string CategoryId { get; } /// @@ -25,7 +27,8 @@ interface IMigrationCategoryWriter int BatchSize(ServiceControlDbContext dbContext); /// - /// Turns one batch of source documents into the rows to insert and the rows to skip, writing nothing: the insert it returns runs later, inside the target's transaction. + /// Turns one batch of source documents into the rows to insert and the rows to skip, writing nothing to the database: the insert it returns runs later, inside the target's transaction. + /// A writer may store data its rows point at outside the database, such as message bodies, because that data has to exist before the insert commits. /// Every batch row produces exactly one row or exactly one skip, never both and never neither, because the target works out how many were already there by subtracting both from the batch size. /// Task Prepare(ServiceControlDbContext dbContext, MigrationBatch batch, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/MigrationCategoryWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/MigrationCategoryWriter.cs index 4550acc0bb..959c40ff35 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/MigrationCategoryWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/MigrationCategoryWriter.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; using DbContexts; +using Infrastructure; using ServiceControl.Persistence.DataMigration; /// @@ -23,12 +24,20 @@ public Task Prepare(ServiceControlDbContext dbContext, MigrationB /// /// Does what promises, for rows already cast. /// - /// The context to read the target through. Nothing may be written through it here. - /// Every row in the batch, in source order, each with its document. - /// Cancels any read of the target. - /// One prepared row or one skip for every entry in . protected abstract Task PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, TDocument Document)> documents, CancellationToken cancellationToken = default); + /// + /// Returns a skip for a row whose entity holds a text value the database would refuse, because one such value fails the whole statement the row goes into, + /// or null when every value can be stored. The reason is for a value longer than its column allows, which is checked first, + /// or for a value holding a character the provider cannot store. + /// + protected static (string SourceId, MigrationSkipReason Reason, string Detail)? UnstorableValueSkip(MigrationInsert insert, IMigrationSqlDialect dialect, MigrationRow row, TEntity entity) where TEntity : class => + insert.ValueTooLongIn(entity, dialect) is { } tooLong + ? (row.SourceId, MigrationSkipReason.ValueTooLong, $"{tooLong}, on document '{row.SourceId}'") + : insert.RejectedCharacterIn(entity, dialect) is { } rejected + ? (row.SourceId, MigrationSkipReason.MalformedValue, $"{rejected}, on document '{row.SourceId}'") + : null; + TDocument DocumentOf(MigrationRow row) => row.Document as TDocument ?? throw new InvalidCastException($"The {CategoryId} writer takes {typeof(TDocument).FullName} documents, but row {row.SourceId} holds {row.Document?.GetType().FullName ?? "no document"}. The reader and writer for one category must agree on the document type."); diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs index 485518ca67..72db9db495 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs @@ -16,21 +16,37 @@ sealed class EndpointSettingsWriter(IMigrationSqlDialect migrationDialect) : Mig { public override string CategoryId => MigrationCategoryIds.EndpointSettings; - public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatement(MigrationInsert.For(dbContext).Columns.Count); + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor(dbContext); public override Task Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => dbContext.EndpointSettings.LongCountAsync(cancellationToken); protected override async Task PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, EndpointSettings Document)> documents, CancellationToken cancellationToken = default) { - List rows = [.. documents.Select(document => new EndpointSettingsEntity { Name = document.Document.Name, TrackInstances = document.Document.TrackInstances })]; - List rowSourceIds = [.. documents.Select(document => document.Row.SourceId)]; + var insert = MigrationInsert.For(dbContext); + List rows = []; + List rowSourceIds = []; + List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; + + foreach (var (row, settings) in documents) + { + var entity = new EndpointSettingsEntity { Name = settings.Name, TrackInstances = settings.TrackInstances }; + + if (UnstorableValueSkip(insert, migrationDialect, row, entity) is { } unstorable) + { + skips.Add(unstorable); + continue; + } + + rows.Add(entity); + rowSourceIds.Add(row.SourceId); + } // The source yields document-id order, and the statement keeps the first of any keys the database treats as one. return new PreparedBatch( async (context, token) => (await migrationDialect.InsertMissing(context, rows, token)).Count, rows.Count, - [], + skips, await MergesIn(dbContext, rows, rowSourceIds, cancellationToken)); } diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs new file mode 100644 index 0000000000..558401d972 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs @@ -0,0 +1,158 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using Abstractions; +using DbContexts; +using Entities; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; + +/// +/// Writes failed messages and their failure group rows, inserting only messages the target does not already hold +/// and never updating one it does. Each category that copies failed messages derives from this and names its +/// category id and the statuses it copies. +/// +abstract class FailedMessagesWriterBase(IMigrationSqlDialect migrationDialect, IBodyStoragePersistence storagePersistence, EFPersisterSettings persisterSettings) : MigrationCategoryWriter +{ + /// + /// The statuses this category copies. counts only rows with one of these statuses, because + /// both failed message categories share one table. + /// + protected abstract IReadOnlyCollection Statuses { get; } + + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor(dbContext); + + public override Task Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.FailedMessages + .Where(message => Statuses.Contains(message.Status)) + .LongCountAsync(cancellationToken); + + protected override async Task PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, FailedMessage Document)> documents, CancellationToken cancellationToken = default) + { + var messageInsert = MigrationInsert.For(dbContext); + var groupInsert = MigrationInsert.For(dbContext); + List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; + List mapped = []; + HashSet mappedIds = []; + List<(string SourceId, FailedMessageEntity Row, MigrationBody Body)> externalBodies = []; + + foreach (var (row, document) in documents) + { + switch (NamingTheDocument(row, () => FailedMessageRowMapper.Map(row))) + { + case FailedMessageRowMapping.Mapped mapping: + var body = row.Body ?? throw new InvalidOperationException($"Row '{row.SourceId}' in category {CategoryId} reached the writer without a body. The engine attaches a body to every row of a category that carries bodies, or skips the row."); + // Set before the length check, because the stored content type is a text column with a limit. + var storeExternally = NamingTheDocument(row, () => FailedMessageRowMapper.SetBody(mapping.Row, document.ProcessingAttempts[^1].Headers, body, persisterSettings.BodyStorage.MaxBodySizeToStore)); + + // A message copied without one of its groups would be missing from that group, so a group that cannot be stored skips the message. + var unstorable = UnstorableValueSkip(messageInsert, migrationDialect, row, mapping.Row) + ?? mapping.Groups.Select(group => UnstorableValueSkip(groupInsert, migrationDialect, row, group)).FirstOrDefault(groupSkip => groupSkip is not null); + + if (unstorable is { } unstorableSkip) + { + skips.Add(unstorableSkip); + } + else + { + mapped.Add(mapping); + + // The insert keeps only the first document with an id, so only its body may be stored under that id. + if (mappedIds.Add(mapping.Row.UniqueMessageId) && storeExternally) + { + externalBodies.Add((row.SourceId, mapping.Row, body)); + } + } + + break; + case FailedMessageRowMapping.Skipped skip: + skips.Add((row.SourceId, skip.Reason, skip.Detail)); + break; + default: + throw new InvalidOperationException($"The mapper returned a mapping the writer does not know for document '{row.SourceId}'."); + } + } + + await WriteBodiesTheTargetLacks(dbContext, externalBodies, cancellationToken); + + return new PreparedBatch((context, token) => InsertFailedMessages(context, mapped, token), mapped.Count, skips, Merges: []); + } + + // All bodies are written before the transaction opens, so a crash leaves a body with no row rather than a row pointing at a body that was never stored. + async Task WriteBodiesTheTargetLacks(ServiceControlDbContext dbContext, IReadOnlyList<(string SourceId, FailedMessageEntity Row, MigrationBody Body)> externalBodies, CancellationToken cancellationToken) + { + if (externalBodies.Count == 0) + { + return; + } + + var ids = externalBodies.Select(external => external.Row.UniqueMessageId).ToArray(); + + // This only skips writing bodies for messages the target already holds; InsertMissing still decides which rows go in. + var present = (await dbContext.FailedMessages + .AsNoTracking() + .Where(message => ids.Contains(message.UniqueMessageId)) + .Select(message => message.UniqueMessageId) + .ToListAsync(cancellationToken)).ToHashSet(); + + await Task.WhenAll(externalBodies + .Where(external => !present.Contains(external.Row.UniqueMessageId)) + .Select(external => WriteBody(external, cancellationToken))); + } + + async Task WriteBody((string SourceId, FailedMessageEntity Row, MigrationBody Body) external, CancellationToken cancellationToken) + { + var bodyId = external.Row.UniqueMessageId.ToString(); + + try + { + await storagePersistence.WriteBody(bodyId, external.Body.Content, external.Body.ContentType, cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + // The storage SDK errors do not name the body, so without this the error that stops the copy would not say which message failed. + catch (Exception ex) + { + throw new InvalidOperationException($"Writing the body of document '{external.SourceId}' in category {CategoryId} to external body storage as '{bodyId}' ({external.Body.Content.Length} bytes) failed with {ex.GetType().Name}: {ex.Message}", ex); + } + } + + // Adds the document id to the error, because the bare error names no document and the source can hold millions. + T NamingTheDocument(MigrationRow row, Func step) + { + try + { + return step(); + } + catch (Exception ex) + { + throw new InvalidOperationException($"Preparing document '{row.SourceId}' in category {CategoryId} failed with {ex.GetType().Name}: {ex.Message}", ex); + } + } + + async Task InsertFailedMessages(ServiceControlDbContext dbContext, IReadOnlyList mapped, CancellationToken cancellationToken) + { + var inserted = (await migrationDialect.InsertMissing(dbContext, [.. mapped.Select(mapping => mapping.Row)], cancellationToken)) + .ToHashSet(ReferenceEqualityComparer.Instance); + + // Matched by reference, because InsertMissing returns the row objects it kept, and a document it turned away can share a kept row's id. + // A group goes in only when the message its foreign key points at went in, so a message already in the target keeps the groups it had. + await migrationDialect.InsertMissing(dbContext, [.. mapped.Where(mapping => inserted.Contains(mapping.Row)).SelectMany(mapping => mapping.Groups)], cancellationToken); + + return inserted.Count; + } +} + +/// +/// Writes the unresolved and retry-issued failed messages. +/// +sealed class FailedMessagesWriter(IMigrationSqlDialect migrationDialect, IBodyStoragePersistence storagePersistence, EFPersisterSettings persisterSettings) + : FailedMessagesWriterBase(migrationDialect, storagePersistence, persisterSettings) +{ + public override string CategoryId => MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages; + + protected override IReadOnlyCollection Statuses => MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KnownEndpointsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KnownEndpointsWriter.cs index a0ec9ff796..fb7e845a67 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KnownEndpointsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KnownEndpointsWriter.cs @@ -14,13 +14,14 @@ sealed class KnownEndpointsWriter(IMigrationSqlDialect migrationDialect) : Migra { public override string CategoryId => MigrationCategoryIds.KnownEndpoints; - public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatement(MigrationInsert.For(dbContext).Columns.Count); + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor(dbContext); public override Task Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => dbContext.KnownEndpoints.LongCountAsync(cancellationToken); protected override Task PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, KnownEndpoint Document)> documents, CancellationToken cancellationToken = default) { + var insert = MigrationInsert.For(dbContext); var rows = new List(documents.Count); var skips = new List<(string SourceId, MigrationSkipReason Reason, string Detail)>(); @@ -33,14 +34,22 @@ protected override Task PrepareDocuments(ServiceControlDbContext continue; } - rows.Add(new KnownEndpointEntity + var entity = new KnownEndpointEntity { Id = endpoint.EndpointDetails.GetDeterministicId(), Name = endpoint.EndpointDetails.Name, HostId = endpoint.EndpointDetails.HostId, Host = endpoint.EndpointDetails.Host, Monitored = endpoint.Monitored - }); + }; + + if (UnstorableValueSkip(insert, migrationDialect, row, entity) is { } unstorable) + { + skips.Add(unstorable); + continue; + } + + rows.Add(entity); } return Task.FromResult(new PreparedBatch( diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs index 864562c443..67f875e037 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs @@ -24,6 +24,17 @@ public interface IMigrationSqlDialect /// int RowsPerStatement(int parametersPerRow); + /// + /// The length of a text value as the database counts it when checking a column's maximum length. + /// + int TextLength(string value); + + /// + /// Finds a character the database refuses to store in any text column, which would fail the whole statement the value goes into. + /// Returns that character, or null when the value has none. + /// + char? RejectedCharacterIn(string value); + /// /// Inserts the rows whose key the table does not hold, merges any whose keys the database treats as one, and returns the rows it inserted. /// diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationInsert.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationInsert.cs index 37ac476786..947e20afec 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationInsert.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationInsert.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; +using System.Data.Common; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Metadata; @@ -15,29 +16,43 @@ public sealed class MigrationInsert where TEntity : class readonly IReadOnlyList properties; readonly IReadOnlyList keyProperties; + readonly IReadOnlyList<(IProperty Property, string TableAndColumn, int MaxLength)> boundedText; + readonly IReadOnlyList<(IProperty Property, string TableAndColumn)> text; - MigrationInsert(string table, IReadOnlyList properties, IReadOnlyList keyProperties, Func columnName) + MigrationInsert(string tableName, string quotedTable, IReadOnlyList properties, IReadOnlyList keyProperties, Func columnName, Func quote) { - Table = table; + Table = quotedTable; this.properties = properties; this.keyProperties = keyProperties; - Columns = [.. properties.Select(columnName)]; - KeyColumns = [.. keyProperties.Select(columnName)]; + Columns = [.. properties.Select(property => quote(columnName(property)))]; + KeyColumns = [.. keyProperties.Select(property => quote(columnName(property)))]; + text = [.. properties.Where(property => property.ClrType == typeof(string)).Select(property => (property, $"{tableName}.{columnName(property)}"))]; + boundedText = + [ + .. text + .Where(column => column.Property.GetMaxLength() is not null) + .Select(column => (column.Property, column.TableAndColumn, column.Property.GetMaxLength()!.Value)) + ]; } - /// The table name, quoted for this provider and carrying its schema. + /// + /// The table name, quoted for this provider and carrying its schema. + /// public string Table { get; } - /// Every column the insert writes, quoted, in the order the parameters of one row follow. + /// + /// Every column the insert writes, quoted, in the order the parameters of one row follow. + /// public IReadOnlyList Columns { get; } - /// The primary key columns, quoted, in the order the statement has to return them. + /// + /// The primary key columns, quoted, in the order the statement has to return them. + /// public IReadOnlyList KeyColumns { get; } /// /// Reads the table, columns and key out of the EF Core model for this entity. /// - /// The entity is not mapped, has no primary key, or has a column this insert cannot write, such as one the database generates. Rows like that go through EF in the caller's context instead. public static MigrationInsert For(ServiceControlDbContext dbContext) { var entityType = dbContext.Model.FindEntityType(typeof(TEntity)) @@ -59,19 +74,57 @@ public static MigrationInsert For(ServiceControlDbContext dbContext) var sql = dbContext.GetService(); return new MigrationInsert( + tableName, sql.DelimitIdentifier(tableName, entityType.GetSchema()), properties, key.Properties, - property => sql.DelimitIdentifier(property.GetColumnName(storeObject) - ?? throw new InvalidOperationException($"{typeof(TEntity).Name}.{property.Name} has no column in {tableName}."))); + property => property.GetColumnName(storeObject) + ?? throw new InvalidOperationException($"{typeof(TEntity).Name}.{property.Name} has no column in {tableName}."), + sql.DelimitIdentifier); + } + + /// + /// Finds the first text value in a row that is longer than its column allows, which would fail the whole statement the row goes into. + /// Returns a message naming the column and both lengths, or null when every value fits. + /// + public string? ValueTooLongIn(TEntity row, IMigrationSqlDialect dialect) + { + foreach (var (property, tableAndColumn, maxLength) in boundedText) + { + // No provider counts a string as longer than its UTF-16 length, so a value within that length fits and skips the dialect's own count. + if (property.GetGetter().GetClrValue(row) is string { Length: var units } value && units > maxLength && dialect.TextLength(value) is var length && length > maxLength) + { + return $"{tableAndColumn} takes at most {maxLength} characters and the value holds {length}"; + } + } + + return null; + } + + /// + /// Finds the first text value in a row holding a character the provider refuses to store, which would fail the whole statement the row goes into. + /// Returns a message naming the column and the character, or null when there is none. + /// + public string? RejectedCharacterIn(TEntity row, IMigrationSqlDialect dialect) + { + foreach (var (property, tableAndColumn) in text) + { + if (property.GetGetter().GetClrValue(row) is string value && dialect.RejectedCharacterIn(value) is { } rejected) + { + return $"{tableAndColumn} cannot hold the character U+{(int)rejected:X4}, which the value contains"; + } + } + + return null; } /// /// Runs one insert statement over a chunk of rows and returns the rows of that chunk it inserted. /// The statement reads parameters named @p0 upward, row after row in order, and must return the of each row it inserted, in that order. /// It runs on the caller's open transaction, and throws when there is none or when a returned key matches no row of the chunk. + /// Each parameter is built from EF Core's type mapping and then passed, with its property, to when one is given, so a provider can change how it is declared. /// - public async Task> Execute(ServiceControlDbContext dbContext, string sql, TEntity[] chunk, CancellationToken cancellationToken = default) + public async Task> Execute(ServiceControlDbContext dbContext, string sql, TEntity[] chunk, Action? configureParameter, CancellationToken cancellationToken = default) { await using var command = dbContext.Database.GetDbConnection().CreateCommand(); command.Transaction = (dbContext.Database.CurrentTransaction @@ -86,7 +139,9 @@ public async Task> Execute(ServiceControlDbContext dbCont foreach (var property in properties) { // The mapping applies the value converter and the provider type, as EF's own inserts do. - command.Parameters.Add(property.GetRelationalTypeMapping().CreateParameter(command, $"@p{index++}", property.GetGetter().GetClrValue(row), property.IsNullable)); + var parameter = property.GetRelationalTypeMapping().CreateParameter(command, $"@p{index++}", property.GetGetter().GetClrValue(row), property.IsNullable); + configureParameter?.Invoke(property, parameter); + command.Parameters.Add(parameter); } } diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationSqlDialectExtensions.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationSqlDialectExtensions.cs new file mode 100644 index 0000000000..5ebc22df43 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/MigrationSqlDialectExtensions.cs @@ -0,0 +1,12 @@ +namespace ServiceControl.Persistence.EFCore.Infrastructure; + +using ServiceControl.Persistence.EFCore.DbContexts; + +static class MigrationSqlDialectExtensions +{ + /// + /// How many rows fit in one of 's statements, sized from the same column list the statement chunks by. + /// + public static int RowsPerStatementFor(this IMigrationSqlDialect dialect, ServiceControlDbContext dbContext) where TEntity : class => + dialect.RowsPerStatement(MigrationInsert.For(dbContext).Columns.Count); +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs new file mode 100644 index 0000000000..5efdc67bcf --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs @@ -0,0 +1,105 @@ +#nullable enable +namespace ServiceControl.Persistence.RavenDB.DataMigration; + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using Raven.Client; +using Raven.Client.Documents.Commands; +using ServiceControl.MessageFailures; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; + +/// +/// Turns the failed message documents RavenDB streams into migration rows. +/// +class FailedMessageStream(RavenReadOnlySourceLifecycle lifecycle) +{ + /// + /// Builds the function that turns each streamed failed message into a row. The function returns null for a message whose + /// status is not in , and throws for a status that neither + /// failed message category copies. + /// + public Func, MigrationRow?> Project(IReadOnlyCollection statuses) => + result => + { + var status = result.Document.Status; + + if (!MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses.Contains(status) && !MigrationCategoryRegistry.ArchivedAndResolvedStatuses.Contains(status)) + { + throw new InvalidOperationException($"The failed message document '{result.Id}' has the status value {(int)status}, which neither failed message category copies, so it would be left behind without a skip to show for it."); + } + + if (!statuses.Contains(status)) + { + return null; + } + + RestoreMetadataTypes(result.Document); + + return new MigrationRow( + result.Id, + result.Document, + new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = StatusChangedAt(result.Document, ReadMetadataDate(result, Constants.Documents.Metadata.Expires), lifecycle.Settings.ErrorRetentionPeriod), + [MigrationMetadataKeys.LastModified] = ReadMetadataDate(result, Constants.Documents.Metadata.LastModified) + }); + }; + + // The mapper checks TimeSent, SendingEndpoint and ReceivingEndpoint by exact type, but RavenDB can hand them back as text or untyped objects. + // Only the newest attempt is fixed because the mapper reads no other, and a TimeSent that is not a date is left as it is for the mapper to skip. + internal static void RestoreMetadataTypes(FailedMessage document) + { + if (document.ProcessingAttempts is not { Count: > 0 } || document.ProcessingAttempts[^1] is not { MessageMetadata: { } metadata } newest) + { + return; + } + + if (metadata.GetValueOrDefault("TimeSent") is string timeSent + && DateTime.TryParse(timeSent, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed)) + { + metadata["TimeSent"] = parsed; + } + + if (newest.Headers is not { } headers) + { + return; + } + + RestoreEndpoint(metadata, "SendingEndpoint", () => MigrationEndpointDetailsParser.SendingEndpoint(headers)); + RestoreEndpoint(metadata, "ReceivingEndpoint", () => MigrationEndpointDetailsParser.ReceivingEndpoint(headers)); + } + + static void RestoreEndpoint(Dictionary metadata, string key, Func parseHeaders) + { + if (metadata.GetValueOrDefault(key) is null or EndpointDetails) + { + return; + } + + // Storing null is deliberate, because the mapper reads it as no endpoint, the same as when ingestion stored none. + metadata[key] = parseHeaders()!; + } + + // For an archived or resolved message RavenDB stores only when it expires, so the time its status + // changed is that expiry less the ErrorRetentionPeriod the source set it with. + internal static DateTime? StatusChangedAt(FailedMessage document, DateTime? expires, TimeSpan sourceErrorRetentionPeriod) + { + ArgumentOutOfRangeException.ThrowIfLessThan(sourceErrorRetentionPeriod, TimeSpan.Zero, "ErrorRetentionPeriod"); + + return document.Status is FailedMessageStatus.Resolved or FailedMessageStatus.Archived && expires is { } deadline + ? AsUtc(deadline) - sourceErrorRetentionPeriod + : document.ProcessingAttempts?.Where(attempt => attempt is not null).MaxBy(attempt => attempt.AttemptedAt) is { } newest ? AsUtc(newest.AttemptedAt) : null; + } + + // RavenDB stores times in UTC, so a time read back with no kind is already UTC and must not be shifted by the local time zone. + static DateTime AsUtc(DateTime value) => + value.Kind == DateTimeKind.Unspecified ? DateTime.SpecifyKind(value, DateTimeKind.Utc) : value.ToUniversalTime(); + + static DateTime? ReadMetadataDate(StreamResult result, string key) => + result.Metadata.TryGetValue(key, out var value) && value != null + ? AsUtc(DateTime.Parse(value.ToString()!, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind)) + : null; +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationCategoryReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationCategoryReader.cs index 590e534e18..bd26b24b74 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationCategoryReader.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationCategoryReader.cs @@ -5,6 +5,7 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System; using System.Collections.Generic; using System.Threading; +using Raven.Client.Documents.Commands; using ServiceControl.Persistence.DataMigration; /// @@ -28,13 +29,22 @@ abstract class MigrationCategoryReader(RavenReadOnlySourceLifecycle l /// Streams the primary database's documents whose id starts with , each one whole as a row. /// protected IAsyncEnumerable WholeDocuments(string prefix, string? resumeAfter, int batchSize, CancellationToken cancellationToken = default) => - RavenDocumentStream.ByPrefix( + Documents(prefix, resumeAfter, batchSize, RavenDocumentStream.WholeDocument, cancellationToken); + + /// + /// Streams the primary database's documents whose id starts with , in id order, through + /// with this category's id and the walk limit every reader shares, + /// . + /// + protected IAsyncEnumerable Documents(string prefix, string? resumeAfter, int batchSize, Func, MigrationRow?> project, CancellationToken cancellationToken = default) => + RavenDocumentStream.ByPrefix( Lifecycle, CategoryId, Lifecycle.Settings.DatabaseName, prefix, resumeAfter, batchSize, - RavenDocumentStream.WholeDocument, + RavenDocumentStream.MaxDocumentsWalkedPerBatch, + project, cancellationToken); } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationEndpointDetailsParser.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationEndpointDetailsParser.cs new file mode 100644 index 0000000000..37a67848cb --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/MigrationEndpointDetailsParser.cs @@ -0,0 +1,95 @@ +#nullable enable +namespace ServiceControl.Persistence.RavenDB.DataMigration; + +using System; +using System.Collections.Generic; +using NServiceBus; +using NServiceBus.Faults; +using ServiceControl.Operations; +using ServiceControl.Persistence.Infrastructure; + +/// +/// Rebuilds the sending and receiving endpoint of a failed message from its headers exactly as ingestion would have stored them, +/// so a message ingested with an odd header copies with the same endpoint details it would have had. +/// +static class MigrationEndpointDetailsParser +{ + public static EndpointDetails? SendingEndpoint(IReadOnlyDictionary headers) + { + var endpoint = new EndpointDetails + { + Name = ValueOf(headers, Headers.OriginatingEndpoint) ?? string.Empty, + Host = ValueOf(headers, Headers.OriginatingMachine) ?? string.Empty, + HostId = Guid.TryParse(ValueOf(headers, Headers.OriginatingHostId), out var hostId) ? hostId : Guid.Empty + }; + + if (!string.IsNullOrEmpty(endpoint.Name) && !string.IsNullOrEmpty(endpoint.Host)) + { + return endpoint; + } + + if (ValueOf(headers, Headers.OriginatingAddress) is not { } address) + { + return null; + } + + var (queue, machine) = ExtractQueueAndMachineName(address); + endpoint.Name = queue; + endpoint.Host = machine!; + return endpoint; + } + + public static EndpointDetails? ReceivingEndpoint(IReadOnlyDictionary headers) + { + var endpoint = new EndpointDetails + { + Name = ValueOf(headers, Headers.ProcessingEndpoint) ?? string.Empty, + Host = ValueOf(headers, Headers.HostDisplayName) ?? ValueOf(headers, Headers.ProcessingMachine) ?? string.Empty, + HostId = Guid.TryParse(ValueOf(headers, Headers.HostId), out var hostId) ? hostId : Guid.Empty + }; + + if (!string.IsNullOrEmpty(endpoint.Name) && !string.IsNullOrEmpty(endpoint.Host)) + { + return endpoint; + } + + if (ValueOf(headers, FaultsHeaderKeys.FailedQ) is not { } address) + { + return null; + } + + var (queue, machine) = ExtractQueueAndMachineName(address); + + if (string.IsNullOrEmpty(endpoint.Name)) + { + endpoint.Name = queue; + } + + if (string.IsNullOrEmpty(endpoint.Host)) + { + endpoint.Host = machine ?? "unknown"; + } + + if (string.IsNullOrEmpty(endpoint.Name)) + { + return null; + } + + if (endpoint.HostId == Guid.Empty) + { + endpoint.HostId = DeterministicGuid.MakeId(endpoint.Name, endpoint.Host); + } + + return endpoint; + } + + static string? ValueOf(IReadOnlyDictionary headers, string key) => + headers.TryGetValue(key, out var value) ? value : null; + + static (string Queue, string? Machine) ExtractQueueAndMachineName(string address) + { + var atIndex = address.IndexOf('@', StringComparison.InvariantCulture); + + return atIndex > -1 ? (address[..atIndex], address[(atIndex + 1)..]) : (address, null); + } +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenDocumentStream.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenDocumentStream.cs index 892641c1f5..cfa5726b66 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenDocumentStream.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenDocumentStream.cs @@ -7,23 +7,28 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System.Collections.ObjectModel; using System.Runtime.CompilerServices; using System.Threading; +using System.Threading.Tasks; using Raven.Client.Documents.Commands; +using Raven.Client.Documents.Session; using ServiceControl.Persistence.DataMigration; /// -/// The one way every category reads RavenDB: stream the documents whose id starts with a prefix, in id order, -/// and hand them back in batches. Id order is what makes the cursor work, because a resume asks RavenDB to -/// start after an id rather than to skip a count. +/// Reads RavenDB documents for a migration category in id order and hands them back in batches, most often every +/// document whose id starts with a prefix. Id order is what lets a copy resume, because a resume asks RavenDB to +/// start after the last id it copied rather than to skip a count. /// static class RavenDocumentStream { /// - /// Streams one collection and yields it in batches of at most . + /// The most documents one batch reads before it is handed back, even if it holds few rows or none. Every batch is + /// one commit, so a long run of documents that make no row costs one commit per this many documents rather than one + /// per batch size, while the cursor is still saved every few seconds and a resume reads at most this many documents again. + /// + public const int MaxDocumentsWalkedPerBatch = 10_000; + + /// + /// Streams the documents whose id starts with , after , and yields a batch as soon as it holds rows, or once it has read documents, even if it holds few rows or none. A batch never holds more than rows, but when returns null for some documents, one batch can cover more documents than that. If any documents were read after the last such batch, it yields one more batch for them, even an empty one, so the cursor past them is saved. When never returns null and is no more than , every batch but the last holds exactly rows. Throws when the source holds no document with the id in . /// - /// The document id prefix that selects the collection, such as "KnownEndpoints/". - /// The cursor a previous run saved, or null to start at the beginning. - /// Turns one document into a row, or into null for a document this category does not copy. The cursor still moves past a document that becomes null. - /// The source holds no document with the id in , which means the cursor and the database no longer belong together. public static async IAsyncEnumerable ByPrefix( RavenReadOnlySourceLifecycle lifecycle, string categoryId, @@ -31,40 +36,80 @@ public static async IAsyncEnumerable ByPrefix( string prefix, string? resumeAfter, int batchSize, + int maxDocumentsWalkedPerBatch, Func, MigrationRow?> project, [EnumeratorCancellation] CancellationToken cancellationToken = default) where TDocument : class { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(maxDocumentsWalkedPerBatch); + using var session = lifecycle.OpenSession(databaseName); // RavenDB starts a stream after whatever id it is given, even one that does not exist, so a stale cursor would skip rows nothing has copied. + await RefuseMissingCursor(session, categoryId, databaseName, resumeAfter, cancellationToken); + + await using var enumerator = await session.Advanced.StreamAsync( + prefix, startAfter: resumeAfter, token: cancellationToken); + + await foreach (var batch in Batches(enumerator, resumeAfter, batchSize, maxDocumentsWalkedPerBatch, project)) + { + yield return batch; + } + } + + /// + /// Checks that the cursor names a document the source still holds. The cursor comes from the target's checkpoint, so + /// if the source has no document with that id, either the copy is now reading a different database or the document was + /// deleted since, and resuming after it could skip rows that were never copied. Throws + /// when is not null and the source holds no document with that id. + /// + public static async Task RefuseMissingCursor(IAsyncDocumentSession session, string categoryId, string databaseName, string? resumeAfter, CancellationToken cancellationToken = default) + { if (resumeAfter is not null && !await session.Advanced.ExistsAsync(resumeAfter, cancellationToken)) { throw new InvalidOperationException( - $"The migration cannot resume the '{categoryId}' category after '{resumeAfter}', because the RavenDB database '{databaseName}' holds no document with that id. The cursor is saved in the target database and names a source document, so restoring RavenDB from a backup or changing which database it reads separates the two. Point the migration back at the RavenDB database this copy started from and restart."); + $"The migration cannot resume the '{categoryId}' category after '{resumeAfter}', because the RavenDB database '{databaseName}' no longer holds a document with that id. The likeliest cause is that the document expired and RavenDB deleted it while the copy was interrupted, which can happen to any archived or resolved message. Run --migration-retry {categoryId} to copy the category again from the start. If RavenDB was instead restored from a backup or the migration now reads a different database, point it back at the database this copy started from and restart."); } + } - await using var enumerator = await session.Advanced.StreamAsync( - prefix, startAfter: resumeAfter, token: cancellationToken); - + /// + /// Groups the documents from a RavenDB stream into batches, by the rules describes: + /// a batch goes out once it holds rows or has read + /// documents, and one last batch goes out for any documents read after that, even if it holds no rows. The stream must return + /// documents in id order, or a resume would start in the wrong place. is the id the stream + /// started after, or null. turns a document into a row, or returns null for a document the category + /// does not copy, and the cursor still moves past that document. Each batch's cursor is the id of the last document it read. + /// An empty stream yields nothing. + /// + public static async IAsyncEnumerable Batches( + IAsyncEnumerator> enumerator, + string? resumeAfter, + int batchSize, + int maxDocumentsWalkedPerBatch, + Func, MigrationRow?> project) + { var rows = new List(batchSize); var cursor = resumeAfter; var lastYielded = resumeAfter; + var walkedSinceYield = 0; while (await enumerator.MoveNextAsync()) { // The cursor moves on every document the stream hands back, even one that becomes no row, so a resume never walks it again. cursor = enumerator.Current.Id; + walkedSinceYield++; if (project(enumerator.Current) is { } row) { rows.Add(row); } - if (rows.Count == batchSize) + // Handing back a batch at the document cap saves progress through a long run of documents that make no row. + if (rows.Count == batchSize || walkedSinceYield == maxDocumentsWalkedPerBatch) { yield return new MigrationBatch(rows, cursor); rows = new List(batchSize); lastYielded = cursor; + walkedSinceYield = 0; } } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs index 8cb44a7cf7..22eefa3bbf 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs @@ -5,12 +5,14 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System; using System.Collections.Frozen; using System.Collections.Generic; +using System.IO; using System.Linq; using System.Threading; using System.Threading.Tasks; using Particular.LicensingComponent.Contracts; using Raven.Client.Documents.Operations; using Raven.Client.ServerWide.Operations; +using ServiceControl.Operations.BodyStorage.RavenAttachments; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.RavenDB.DataMigration.Readers; @@ -24,7 +26,8 @@ sealed class RavenMigrationSource(RavenReadOnlySourceLifecycle lifecycle) : IMig readonly FrozenDictionary readers = new IMigrationCategoryReader[] { new KnownEndpointsReader(lifecycle), - new EndpointSettingsReader(lifecycle) + new EndpointSettingsReader(lifecycle), + new FailedMessagesReader(lifecycle) }.ToFrozenDictionary(reader => reader.CategoryId, StringComparer.Ordinal); public Task Open(CancellationToken cancellationToken = default) => lifecycle.Open(cancellationToken); @@ -84,8 +87,30 @@ public IAsyncEnumerable Read( ? reader.Read(resumeAfter, batchSize, cancellationToken) : throw new NotSupportedException($"The migration source cannot yet read the '{category.Id}' category."); - public Task ReadBody(MigrationCategory category, string sourceId, CancellationToken cancellationToken = default) => - throw new NotSupportedException($"The RavenDB migration source cannot read bodies for category {category.Id} yet"); + public async Task ReadBody(MigrationCategory category, string sourceId, CancellationToken cancellationToken = default) + { + // Another category may keep its body inside the document, where a body attachment read would find nothing and skip every row as BodyMissing. + if (category.Id != MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages) + { + throw new NotSupportedException($"The migration source does not read bodies as attachments for the '{category.Id}' category."); + } + + using var session = lifecycle.OpenSession(lifecycle.Settings.DatabaseName); + + var attachment = await session.Advanced.Attachments.GetAsync(sourceId, RavenAttachmentsBodyStorage.AttachmentName, cancellationToken); + + if (attachment == null) + { + return null; + } + + await using var stream = attachment.Stream; + + var content = new MemoryStream((int)attachment.Details.Size); + await stream.CopyToAsync(content, cancellationToken); + + return new MigrationBody(content.ToArray(), attachment.Details.ContentType); + } public IReadOnlyCollection SupportedCategoryIds => readers.Keys; diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs index 901c76f6ac..b3092c4096 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs @@ -9,6 +9,7 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Hosting; +using Newtonsoft.Json; using Particular.LicensingComponent.Contracts; using Raven.Client.Documents; using Raven.Client.Documents.Conventions; @@ -16,7 +17,9 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using Raven.Client.Documents.Session; using Raven.Client.Exceptions.Database; using Raven.Client.Exceptions.Security; +using Raven.Client.Json.Serialization.NewtonsoftJson; using ServiceControl.Configuration; +using ServiceControl.Contracts.Operations; using ServiceControl.RavenDB; /// @@ -142,7 +145,11 @@ IDocumentStore Connect(string serverUrl) { Database = settings.DatabaseName, Urls = [serverUrl], - Conventions = new DocumentConventions { SaveEnumsAsIntegers = true } + Conventions = new DocumentConventions + { + SaveEnumsAsIntegers = true, + Serialization = new NewtonsoftJsonSerializationConventions { CustomizeJsonDeserializer = serializer => serializer.Converters.Add(new FailureDetailsConverter()) } + } }; if (!settings.UseEmbeddedServer) @@ -237,4 +244,24 @@ public void StopApplication() { } } + + // The FailureDetails constructor sets TimeOfFailure to now. Starting from the default instead lets the mapper see a stored attempt that has no time. + sealed class FailureDetailsConverter : JsonConverter + { + public override FailureDetails? ReadJson(JsonReader reader, Type objectType, FailureDetails? existingValue, bool hasExistingValue, JsonSerializer serializer) + { + if (reader.TokenType == JsonToken.Null) + { + return null; + } + + var details = new FailureDetails { AddressOfFailingEndpoint = string.Empty, TimeOfFailure = default }; + serializer.Populate(reader, details); + + return details; + } + + public override void WriteJson(JsonWriter writer, FailureDetails? value, JsonSerializer serializer) => + throw new NotSupportedException("The migration source only reads, so it never writes a FailureDetails."); + } } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs new file mode 100644 index 0000000000..637625354d --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs @@ -0,0 +1,179 @@ +#nullable enable + +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Raven.Client.Documents.Commands; +using Raven.Client.Documents.Indexes; +using Raven.Client.Documents.Operations; +using Raven.Client.Documents.Operations.Indexes; +using Raven.Client.Documents.Session; +using ServiceControl.Infrastructure; +using ServiceControl.MessageFailures; +using ServiceControl.MessageFailures.Api; +using ServiceControl.Persistence.DataMigration; + +/// +/// Reads the failed messages that are still waiting on somebody: unresolved, or with a retry issued. This category is +/// copied while ServiceControl is closed, and the archived and resolved messages in the same collection can outnumber it +/// many times, so it queries for just these statuses instead of reading them all. +/// +sealed class FailedMessagesReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader(lifecycle) +{ + public override string CategoryId => MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages; + + /// + /// Waits for to catch up, then streams the unresolved and retry issued messages in id + /// order, starting after . When the index is missing, errored or disabled, or does not hold + /// every failed message with a status some category copies, it logs a warning and reads the whole collection instead. + /// Both ways use the document id as the cursor, so a copy can switch from one to the other between runs. + /// Throws when the source holds no document with the id in , + /// or when a failed message has a status no category copies. + /// + public override async IAsyncEnumerable Read(string? resumeAfter, int batchSize, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + var databaseName = Lifecycle.Settings.DatabaseName; + var project = new FailedMessageStream(Lifecycle).Project(MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses); + + var batches = await IndexHoldsEveryCopiedFailedMessage(databaseName, cancellationToken) + ? ThroughIndex(databaseName, resumeAfter, batchSize, project, cancellationToken) + : Documents($"{FailedMessageIdGenerator.CollectionName}/", resumeAfter, batchSize, project, cancellationToken); + + await foreach (var batch in batches) + { + yield return batch; + } + } + + async IAsyncEnumerable ThroughIndex( + string databaseName, + string? resumeAfter, + int batchSize, + Func, MigrationRow?> project, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + using var session = Lifecycle.OpenSession(databaseName); + + await RavenDocumentStream.RefuseMissingCursor(session, CategoryId, databaseName, resumeAfter, cancellationToken); + + var query = Query(session).WhereIn(nameof(FailedMessage.Status), StatusValues(MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses)); + + if (resumeAfter is not null) + { + query = query.AndAlso().WhereGreaterThan(DocumentId, resumeAfter); + } + + await using var enumerator = await session.Advanced.StreamAsync(query.OrderBy(DocumentId).OfType(), cancellationToken); + + await foreach (var batch in RavenDocumentStream.Batches(enumerator, resumeAfter, batchSize, RavenDocumentStream.MaxDocumentsWalkedPerBatch, project)) + { + yield return batch; + } + } + + // A full read passes a document RavenDB could not index to the target to skip, and refuses a status no category copies, + // but a query on the index sees neither, so either kind of document sends the copy back to a full read. + async Task IndexHoldsEveryCopiedFailedMessage(string databaseName, CancellationToken cancellationToken) + { + using var session = Lifecycle.OpenSession(databaseName); + + while (true) + { + var index = await WaitForIndex(databaseName, cancellationToken); + + if (index is null || index.State is IndexState.Error or IndexState.Disabled) + { + logger.LogWarning("The RavenDB index {IndexName} in {DatabaseName} is {IndexState}, so copying {CategoryId} reads every failed message instead, archived and resolved ones included, which keeps ServiceControl closed for longer", IndexName, databaseName, index?.State.ToString() ?? "missing", CategoryId); + return false; + } + + // Expiration keeps deleting archived and resolved messages during the copy, so the collection is counted before and + // after the index count, because a deletion in between could hide a document the index left out. A stale index count + // means something changed after the wait, so it waits and counts again. + var storedBefore = await StoredFailedMessages(databaseName, cancellationToken); + var indexed = await Query(session).Statistics(out var statistics).WhereIn(nameof(FailedMessage.Status), StatusValues(CopiedStatuses)).CountAsync(cancellationToken); + var storedAfter = await StoredFailedMessages(databaseName, cancellationToken); + + if (statistics.IsStale) + { + continue; + } + + if (indexed == storedBefore && indexed == storedAfter) + { + return true; + } + + logger.LogWarning("The RavenDB database {DatabaseName} held {StoredBefore} and then {StoredAfter} failed messages while its index {IndexName} held {Indexed} with a status a category copies, so copying {CategoryId} reads every failed message instead, archived and resolved ones included, which keeps ServiceControl closed for longer. The counts differ when RavenDB could not index a document, which the index errors in RavenDB Studio name, when a document has a status no category copies, or when expiration deleted documents during the count", databaseName, storedBefore, storedAfter, IndexName, indexed, CategoryId); + return false; + } + } + + async Task StoredFailedMessages(string databaseName, CancellationToken cancellationToken) => + (await Lifecycle.DocumentStore.Maintenance.ForDatabase(databaseName).SendAsync(new GetCollectionStatisticsOperation(), cancellationToken)) + .Collections.GetValueOrDefault(FailedMessageIdGenerator.CollectionName); + + // A query on an index that is behind leaves out what it has not indexed yet, which here means unresolved messages nothing + // would copy. It waits rather than halts, because halting would need an operator to retry for a wait that is normally seconds. + async Task WaitForIndex(string databaseName, CancellationToken cancellationToken) + { + var waiting = Stopwatch.StartNew(); + var nextLog = TimeSpan.Zero; + + while (true) + { + var statistics = await Lifecycle.DocumentStore.Maintenance.ForDatabase(databaseName).SendAsync(new GetStatisticsOperation(), cancellationToken); + var index = statistics.Indexes.SingleOrDefault(candidate => candidate.Name == IndexName); + + // A missing, errored or disabled index never catches up, so waiting for it would keep ServiceControl closed for good. + if (index is null || !index.IsStale || index.State is IndexState.Error or IndexState.Disabled) + { + return index; + } + + if (waiting.Elapsed >= nextLog) + { + await LogWait(databaseName, waiting.Elapsed, cancellationToken); + nextLog += LogEvery; + } + + await Task.Delay(PollEvery, cancellationToken); + } + } + + // A paused index stays stale with its state still Normal, so without this the log reads exactly like an index that is merely slow. + async Task LogWait(string databaseName, TimeSpan waited, CancellationToken cancellationToken) + { + var indexing = await Lifecycle.DocumentStore.Maintenance.ForDatabase(databaseName).SendAsync(new GetIndexingStatusOperation(), cancellationToken); + var paused = indexing.Status == IndexRunningStatus.Paused + || indexing.Indexes.Any(index => index.Name == IndexName && index.Status == IndexRunningStatus.Paused); + + if (paused) + { + logger.LogWarning("Copying {CategoryId} is waiting for the RavenDB index {IndexName} in {DatabaseName}, but indexing is paused, so ServiceControl stays closed until it is resumed in RavenDB Studio. Waited {Waited} so far", CategoryId, IndexName, databaseName, waited); + return; + } + + logger.LogInformation("Waiting for the RavenDB index {IndexName} in {DatabaseName} to catch up before copying {CategoryId}, waited {Waited} so far", IndexName, databaseName, CategoryId, waited); + } + + static IAsyncDocumentQuery Query(IAsyncDocumentSession session) => + session.Advanced.AsyncDocumentQuery(); + + static object[] StatusValues(IEnumerable statuses) => statuses.Select(status => (object)(int)status).ToArray(); + + static readonly FailedMessageStatus[] CopiedStatuses = [.. MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses, .. MigrationCategoryRegistry.ArchivedAndResolvedStatuses]; + static readonly string IndexName = new FailedMessageViewIndex().IndexName; + static readonly TimeSpan PollEvery = TimeSpan.FromSeconds(1); + static readonly TimeSpan LogEvery = TimeSpan.FromSeconds(30); + static readonly ILogger logger = LoggerUtil.CreateStaticLogger(); + + const string DocumentId = "id()"; +} diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs new file mode 100644 index 0000000000..07c39bd01b --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs @@ -0,0 +1,81 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class MigrationValueLengthTests : PersistenceTestBase +{ + static readonly string OutsideTheBasicPlane = char.ConvertFromUtf32(0x1F600); + + static readonly MigrationCategory KnownEndpointsCategory = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.KnownEndpoints); + + IMigrationTarget Target => ServiceProvider.GetRequiredService(); + + [SetUp] + public Task OpenTarget() => Target.Open(); + + [Test] + public async Task A_value_is_measured_in_code_points_as_varchar_counts_them() + { + var fits = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = Repeat(450), HostId = Guid.NewGuid(), Host = "SALES01" } }; + var tooLong = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = Repeat(451), HostId = Guid.NewGuid(), Host = "SALES02" } }; + var batch = new MigrationBatch( + [ + new MigrationRow("KnownEndpoints/1", fits, new Dictionary()), + new MigrationRow("KnownEndpoints/2", tooLong, new Dictionary()) + ], + "KnownEndpoints/2"); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new KnownEndpointsWriter(ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(KnownEndpointsCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.KnownEndpoints, MigrationCategoryState.InProgress, "KnownEndpoints/2", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "450 code points fit a varchar(450) column though they are 900 UTF-16 units, and a skip here would be permanent"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { "KnownEndpoints/2" })); + Assert.That((await MonitoringDataStore.GetAllKnownEndpoints()).Select(endpoint => endpoint.EndpointDetails.Name), Is.EqualTo(new[] { Repeat(450) })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(KnownEndpointEntity.Name))} takes at most 450 characters and the value holds 451, on document 'KnownEndpoints/2'" + }), "the length reported is the one PostgreSQL counts"); + Assert.That(ServiceProvider.GetRequiredService().TextLength(Repeat(451)), Is.LessThanOrEqualTo(Repeat(451).Length), "the check skips measuring a value whose UTF-16 length fits, which is safe only while no count exceeds it"); + } + } + + [Test] + public async Task A_value_holding_a_NUL_character_is_skipped_as_malformed_naming_the_column() + { + var clean = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = "Sales", HostId = Guid.NewGuid(), Host = "SALES01" } }; + var withNul = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = "Sales", HostId = Guid.NewGuid(), Host = "SAL\u0000ES" } }; + var batch = new MigrationBatch( + [ + new MigrationRow("KnownEndpoints/1", clean, new Dictionary()), + new MigrationRow("KnownEndpoints/2", withNul, new Dictionary()) + ], + "KnownEndpoints/2"); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new KnownEndpointsWriter(ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(KnownEndpointsCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.KnownEndpoints, MigrationCategoryState.InProgress, "KnownEndpoints/2", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "PostgreSQL refuses U+0000 in text, which would fail the whole insert and halt the category"); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.MalformedValue] = 1 })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(KnownEndpointEntity.Host))} cannot hold the character U+0000, which the value contains, on document 'KnownEndpoints/2'" + })); + } + } + + static string Repeat(int codePoints) => string.Concat(Enumerable.Repeat(OutsideTheBasicPlane, codePoints)); +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageBodySourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageBodySourceTests.cs new file mode 100644 index 0000000000..beaee964fe --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageBodySourceTests.cs @@ -0,0 +1,94 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; + +[TestFixture] +class FailedMessageBodySourceTests : RavenMigrationSourceTestBase +{ + static readonly MigrationCategory FailedMessages = + MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + [Test] + public async Task Reads_the_attachment_and_its_own_content_type() + { + var content = Encoding.UTF8.GetBytes("1"); + var documentId = await SeedWithAttachment(content, "application/json"); + + await using var source = await OpenMigrationSource(); + var body = await source.ReadBody(FailedMessages, documentId); + + Assert.That(body, Is.Not.Null); + + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Content.ToArray(), Is.EqualTo(content)); + Assert.That(body.ContentType, Is.EqualTo("application/json"), "the message header says text/xml, the attachment's own type wins"); + } + } + + [Test] + public async Task Reads_a_zero_length_attachment_as_an_empty_body() + { + var documentId = await SeedWithAttachment([], "text/plain"); + + await using var source = await OpenMigrationSource(); + var body = await source.ReadBody(FailedMessages, documentId); + + Assert.That(body, Is.Not.Null, "an empty payload is a body, and is not the same as a missing one"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Content.Length, Is.Zero); + Assert.That(body.ContentType, Is.EqualTo("text/plain")); + } + } + + [Test] + public async Task Returns_null_when_the_document_has_no_attachment() + { + var message = await SeedFailedMessage(new IngestedFailure().ToFailedMessage()); + + await using var source = await OpenMigrationSource(); + + Assert.That(await source.ReadBody(FailedMessages, message.Id), Is.Null); + } + + [Test] + public async Task Returns_null_for_a_document_that_does_not_exist() + { + await using var source = await OpenMigrationSource(); + + Assert.That(await source.ReadBody(FailedMessages, "FailedMessages/no-such-message"), Is.Null, + "a message RavenDB expired during the copy has to be a BodyMissing skip, not three failed reads and a BodyUnreadable one"); + } + + [Test] + public async Task Refuses_a_category_it_does_not_read_body_attachments_for() + { + var message = await SeedFailedMessage(new IngestedFailure().ToFailedMessage()); + var failedErrorImports = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.FailedErrorImports); + + await using var source = await OpenMigrationSource(); + + var exception = Assert.ThrowsAsync(() => source.ReadBody(failedErrorImports, message.Id), + "a failed error import keeps its body inside the document, so reading an attachment would skip every one of them as BodyMissing"); + Assert.That(exception.Message, Does.Contain(MigrationCategoryIds.FailedErrorImports)); + } + + async Task SeedWithAttachment(byte[] content, string contentType) + { + var message = await SeedFailedMessage(new IngestedFailure().ToFailedMessage()); + + using var session = await SessionProvider.OpenSession(); + session.Advanced.Attachments.Store(message.Id, "body", new MemoryStream(content), contentType); + await session.SaveChangesAsync(); + + return message.Id; + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageStatusChangedAtTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageStatusChangedAtTests.cs new file mode 100644 index 0000000000..4842997d31 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedMessageStatusChangedAtTests.cs @@ -0,0 +1,114 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration +{ + using System; + using NUnit.Framework; + using ServiceControl.MessageFailures; + using ServiceControl.Persistence.RavenDB.DataMigration; + + [TestFixture] + class FailedMessageStatusChangedAtTests + { + static readonly TimeSpan ErrorRetentionPeriod = TimeSpan.FromDays(10); + + [TestCase(FailedMessageStatus.Resolved)] + [TestCase(FailedMessageStatus.Archived)] + public void Reconstructs_the_status_change_instant_from_the_expiry_stamp(FailedMessageStatus status) + { + var statusChangedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc); + + var resolved = FailedMessageStream.StatusChangedAt(new IngestedFailure().ToFailedMessage(status), statusChangedAt + ErrorRetentionPeriod, ErrorRetentionPeriod); + + Assert.That(resolved, Is.EqualTo(statusChangedAt)); + } + + [TestCase(FailedMessageStatus.Unresolved)] + [TestCase(FailedMessageStatus.RetryIssued)] + public void Records_the_newest_attempt_for_a_status_that_never_expires(FailedMessageStatus status) + { + var failure = new IngestedFailure(); + var stamp = new DateTime(2026, 10, 1, 0, 0, 0, DateTimeKind.Utc); + + var resolved = FailedMessageStream.StatusChangedAt(failure.ToFailedMessage(status), stamp, ErrorRetentionPeriod); + + Assert.That(resolved, Is.EqualTo(failure.AttemptedAt), "a stamp on a status that never expires is not its status change"); + } + + [Test] + public void Falls_back_to_the_newest_attempt_for_an_archived_message_with_no_stamp() + { + var failure = new IngestedFailure(); + + var resolved = FailedMessageStream.StatusChangedAt(failure.ToFailedMessage(FailedMessageStatus.Archived), null, ErrorRetentionPeriod); + + Assert.That(resolved, Is.EqualTo(failure.AttemptedAt)); + } + + [Test] + public void Returns_null_for_a_document_with_no_attempts_and_no_stamp() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts.Clear(); + + Assert.That(FailedMessageStream.StatusChangedAt(document, null, ErrorRetentionPeriod), Is.Null); + } + + [Test] + public void Returns_null_for_a_document_whose_attempt_list_is_null_and_has_no_stamp() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts = null; + + Assert.That(FailedMessageStream.StatusChangedAt(document, null, ErrorRetentionPeriod), Is.Null, "the mapper skips a message with no attempts, so this must not throw and halt the category"); + } + + [Test] + public void Reconstructs_an_instant_already_past_the_retention_cutoff_faithfully() + { + var expires = new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + var resolved = FailedMessageStream.StatusChangedAt(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived), expires, ErrorRetentionPeriod); + + Assert.That(resolved, Is.EqualTo(expires - ErrorRetentionPeriod), "the value the source implies, not one adjusted to survive the sweep"); + } + + [TestCase(DateTimeKind.Unspecified)] + [TestCase(DateTimeKind.Utc)] + public void Returns_a_UTC_instant_whatever_kind_the_attempt_was_read_with(DateTimeKind kind) + { + var document = new IngestedFailure().ToFailedMessage(); + var attemptedAt = new DateTime(2026, 9, 1, 8, 0, 0, kind); + document.ProcessingAttempts[^1].AttemptedAt = attemptedAt; + + var resolved = FailedMessageStream.StatusChangedAt(document, null, ErrorRetentionPeriod); + + using (Assert.EnterMultipleScope()) + { + Assert.That(resolved.Value.Kind, Is.EqualTo(DateTimeKind.Utc), "MigrationMetadataKeys promises a UTC DateTime and DateTime equality ignores Kind"); + Assert.That(resolved.Value.Ticks, Is.EqualTo(attemptedAt.Ticks)); + } + } + + [Test] + public void Reads_an_expiry_stamp_with_no_kind_as_UTC() + { + var statusChangedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Unspecified); + + var resolved = FailedMessageStream.StatusChangedAt(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived), statusChangedAt + ErrorRetentionPeriod, ErrorRetentionPeriod); + + using (Assert.EnterMultipleScope()) + { + Assert.That(resolved.Value.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(resolved.Value.Ticks, Is.EqualTo(statusChangedAt.Ticks), "an unspecified instant is already UTC and must not be shifted by the local offset"); + } + } + + [Test] + public void Refuses_a_negative_source_retention_period() + { + var ex = Assert.Throws(() => + FailedMessageStream.StatusChangedAt(new IngestedFailure().ToFailedMessage(), null, TimeSpan.FromDays(-1))); + + Assert.That(ex.Message, Does.Contain("ErrorRetentionPeriod")); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/MigrationEndpointDetailsParserParityTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/MigrationEndpointDetailsParserParityTests.cs new file mode 100644 index 0000000000..0cbf6a8e4d --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/MigrationEndpointDetailsParserParityTests.cs @@ -0,0 +1,106 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System; +using System.Collections.Generic; +using NServiceBus; +using NServiceBus.Faults; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.RavenDB.DataMigration; +// Aliased because the RavenDB persister's own EndpointDetailsParser would otherwise win from this namespace. +using IngestionParser = Contracts.Operations.EndpointDetailsParser; + +[TestFixture] +class MigrationEndpointDetailsParserParityTests +{ + [TestCaseSource(nameof(HeaderSets))] + public void Rebuilds_both_endpoints_exactly_as_ingestion_does(Dictionary headers) + { + using (Assert.EnterMultipleScope()) + { + Assert.That(Shape(MigrationEndpointDetailsParser.SendingEndpoint(headers)), Is.EqualTo(Shape(IngestionParser.SendingEndpoint(headers))), + "a migrated message whose sending endpoint differs from a newly ingested one is grouped apart from it in ServicePulse"); + Assert.That(Shape(MigrationEndpointDetailsParser.ReceivingEndpoint(headers)), Is.EqualTo(Shape(IngestionParser.ReceivingEndpoint(headers))), + "a migrated message whose receiving endpoint differs from a newly ingested one is grouped apart from it in ServicePulse"); + } + } + + static (string Name, string Host, Guid HostId)? Shape(EndpointDetails endpoint) => + endpoint is null ? null : (endpoint.Name, endpoint.Host, endpoint.HostId); + + static IEnumerable HeaderSets() + { + var sendingHostId = Guid.Parse("6a3a9b1c-2d4e-4f50-8a61-7b8c9d0e1f20").ToString(); + var receivingHostId = Guid.Parse("0f1e2d3c-4b5a-4968-8776-655443322110").ToString(); + + yield return Case("full headers", new() + { + [Headers.OriginatingEndpoint] = "Sales", + [Headers.OriginatingMachine] = "SALES-01", + [Headers.OriginatingHostId] = sendingHostId, + [Headers.ProcessingEndpoint] = "Billing", + [Headers.HostDisplayName] = "Billing host", + [Headers.ProcessingMachine] = "BILLING-01", + [Headers.HostId] = receivingHostId + }); + + yield return Case("non-GUID host ids", new() + { + [Headers.OriginatingEndpoint] = "Sales", + [Headers.OriginatingMachine] = "SALES-01", + [Headers.OriginatingHostId] = "not-a-guid", + [Headers.ProcessingEndpoint] = "Billing", + [Headers.ProcessingMachine] = "BILLING-01", + [Headers.HostId] = "not-a-guid" + }); + + yield return Case("null-valued host ids", new() + { + [Headers.OriginatingEndpoint] = "Sales", + [Headers.OriginatingMachine] = "SALES-01", + [Headers.OriginatingHostId] = null, + [Headers.ProcessingEndpoint] = "Billing", + [Headers.ProcessingMachine] = "BILLING-01", + [Headers.HostId] = null + }); + + yield return Case("null-valued host display name", new() + { + [Headers.ProcessingEndpoint] = "Billing", + [Headers.HostDisplayName] = null, + [Headers.ProcessingMachine] = "BILLING-01", + [Headers.HostId] = receivingHostId + }); + + yield return Case("addresses with a machine", new() + { + [Headers.OriginatingAddress] = "Sales@SALES-01", + [FaultsHeaderKeys.FailedQ] = "Billing@BILLING-01" + }); + + yield return Case("addresses without a machine", new() + { + [Headers.OriginatingAddress] = "Sales", + [FaultsHeaderKeys.FailedQ] = "Billing" + }); + + yield return Case("failed queue with a non-GUID host id", new() + { + [FaultsHeaderKeys.FailedQ] = "Billing@BILLING-01", + [Headers.HostId] = "not-a-guid" + }); + + yield return Case("null-valued addresses", new() + { + [Headers.OriginatingAddress] = null, + [FaultsHeaderKeys.FailedQ] = null + }); + + yield return Case("no endpoint headers", new() + { + [Headers.MessageId] = "message-1" + }); + } + + static TestCaseData Case(string name, Dictionary headers) => new TestCaseData(headers).SetArgDisplayNames(name); +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs index c34641d535..e97f52a466 100644 --- a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs @@ -70,6 +70,8 @@ public async Task Resuming_after_a_cursor_this_source_never_issued_refuses_inste Assert.That(refusal.Message, Does.Contain(EndpointSettingsCategory.Id)); Assert.That(refusal.Message, Does.Contain("EndpointSettings/zzz-from-another-database")); Assert.That(refusal.Message, Does.Contain(DatabaseName), "the operator has to be told which database the cursor was looked for in"); + Assert.That(refusal.Message, Does.Contain("expired"), "an archived or resolved document can expire during an interrupted copy, on any source"); + Assert.That(refusal.Message, Does.Contain($"--migration-retry {EndpointSettingsCategory.Id}"), "a retry re-reads the category from the start, so it is the way out"); }); } @@ -123,7 +125,8 @@ public async Task Reading_every_category_creates_no_index() static readonly Dictionary> Seeds = new() { [MigrationCategoryIds.EndpointSettings] = tests => tests.SeedEndpointSettings(2), - [MigrationCategoryIds.KnownEndpoints] = tests => tests.MonitoringDataStore.CreateIfNotExists(new EndpointDetails { Name = "Sales.Orders", HostId = Guid.NewGuid(), Host = "HOST01" }) + [MigrationCategoryIds.KnownEndpoints] = tests => tests.MonitoringDataStore.CreateIfNotExists(new EndpointDetails { Name = "Sales.Orders", HostId = Guid.NewGuid(), Host = "HOST01" }), + [MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages] = tests => tests.SeedFailedMessage(new IngestedFailure().ToFailedMessage()) }; string DatabaseName => ((RavenPersisterSettings)PersistenceSettings).DatabaseName; diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs new file mode 100644 index 0000000000..9c25b6a528 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs @@ -0,0 +1,901 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using Raven.Client.Documents.Indexes; +using Raven.Client.Documents.Operations; +using Raven.Client.Documents.Operations.Indexes; +using ServiceControl.Configuration; +using ServiceControl.MessageFailures; +using ServiceControl.MessageFailures.Api; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.RavenDB.DataMigration; + +[TestFixture] +class UnresolvedFailedMessageSourceTests : RavenMigrationSourceTestBase +{ + [Test] + public async Task Reads_both_unresolved_statuses_and_neither_of_the_others() + { + var unresolved = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + var retryIssued = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued)); + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived)); + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Resolved)); + await CompleteDatabaseOperation(); + + var rows = await ReadAll(); + + Assert.That( + rows.Select(row => ((FailedMessage)row.Document).UniqueMessageId), + Is.EquivalentTo(new[] { unresolved.UniqueMessageId, retryIssued.UniqueMessageId })); + } + + [Test] + public async Task Counts_only_the_documents_this_category_copies_not_the_whole_collection() + { + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived)); + await CompleteDatabaseOperation(); + + await using var source = await OpenMigrationSource(); + + Assert.That(await source.Count(Category), Is.EqualTo(2), "the archived message shares the collection but belongs to another category"); + } + + [Test] + public async Task Refuses_a_document_whose_status_neither_failed_message_category_copies() + { + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage((FailedMessageStatus)99)); + await CompleteDatabaseOperation(); + + var refusal = Assert.ThrowsAsync(ReadAll); + + Assert.That(refusal.Message, Does.Contain(stored.Id).And.Contain("99"), "a status no category copies would otherwise be left behind with nothing to show for it"); + } + + [Test] + public async Task Hands_over_attempt_times_as_UTC() + { + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(numberOfAttempts: 2)); + await CompleteDatabaseOperation(); + + var attempts = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts; + + Assert.That(attempts.SelectMany(attempt => new[] { attempt.AttemptedAt.Kind, attempt.FailureDetails.TimeOfFailure.Kind }), Is.All.EqualTo(DateTimeKind.Utc)); + } + + [Test] + public async Task Resolves_StatusChangedAt_to_the_newest_attempt_when_it_is_neither_first_nor_last() + { + var failure = new IngestedFailure(); + var message = failure.ToFailedMessage(FailedMessageStatus.Unresolved, numberOfAttempts: 3); + var attempts = message.ProcessingAttempts; + message.ProcessingAttempts = [attempts[1], attempts[2], attempts[0]]; + await SeedFailedMessage(message); + await CompleteDatabaseOperation(); + + var row = (await ReadAll()).Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.Metadata[MigrationMetadataKeys.StatusChangedAt], Is.EqualTo(failure.AttemptedAt)); + Assert.That(((DateTime)row.Metadata[MigrationMetadataKeys.StatusChangedAt]).Kind, Is.EqualTo(DateTimeKind.Utc)); + } + } + + [Test] + public async Task Sends_only_the_migration_metadata_keys_across_the_seam() + { + await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + await CompleteDatabaseOperation(); + + var row = (await ReadAll()).Single(); + + Assert.That(row.Metadata.Keys, Is.EquivalentTo(new[] { MigrationMetadataKeys.StatusChangedAt, MigrationMetadataKeys.LastModified }), "a store's own metadata, such as @expires, never travels"); + } + + [Test] + public async Task Carries_the_documents_own_last_modified() + { + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + await CompleteDatabaseOperation(); + + DateTime lastModified; + + using (var session = await SessionProvider.OpenSession()) + { + lastModified = session.Advanced.GetLastModifiedFor(await session.LoadAsync(stored.Id)).Value; + } + + var row = (await ReadAll()).Single(); + + Assert.That(row.Metadata[MigrationMetadataKeys.LastModified], Is.EqualTo(lastModified.ToUniversalTime()), "the value RavenDB shows as the message's LastModified, not the copy time"); + } + + [Test] + public async Task A_run_of_dropped_documents_yields_an_empty_batch_only_once_the_walk_limit_is_reached() + { + var ids = await SeedInIdOrder(Enumerable.Repeat(FailedMessageStatus.Archived, 7).ToArray()); + + var batches = await StreamWithWalkLimit(batchSize: 2, maxDocumentsWalkedPerBatch: 3); + + using (Assert.EnterMultipleScope()) + { + Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { ids[2], ids[5], ids[6] }), "an empty stretch commits once per walk limit, plus the tail so the cursor past it is saved"); + Assert.That(batches.SelectMany(batch => batch.Rows), Is.Empty); + } + } + + [Test] + public async Task The_category_reads_no_archived_or_resolved_document() + { + await SeedInIdOrder(FailedMessageStatus.Archived, FailedMessageStatus.Resolved, FailedMessageStatus.Archived); + + await using var source = await OpenMigrationSource(); + var batches = await CollectBatches(source, Category, batchSize: 2); + + Assert.That(batches, Is.Empty, "the copy runs while ServiceControl is closed, so reading the archive only to drop it makes the outage grow with the archive"); + } + + [Test] + public async Task Resuming_after_a_cursor_reads_only_the_unresolved_messages_after_it() + { + var ids = await SeedInIdOrder( + FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.RetryIssued, + FailedMessageStatus.Unresolved, FailedMessageStatus.Resolved, FailedMessageStatus.Unresolved); + + await using var source = await OpenMigrationSource(); + var first = (await CollectBatches(source, Category, batchSize: 2))[0]; + var resumed = await CollectBatches(source, Category, resumeAfter: first.Cursor, batchSize: 2); + + using (Assert.EnterMultipleScope()) + { + Assert.That(first.Rows.Select(row => row.SourceId), Is.EqualTo(new[] { ids[0], ids[2] })); + Assert.That(first.Cursor, Is.EqualTo(ids[2])); + Assert.That(resumed.SelectMany(batch => batch.Rows).Select(row => row.SourceId), Is.EqualTo(new[] { ids[3], ids[5] }), "a resume neither repeats a row nor skips one"); + } + } + + [Test] + public async Task Resuming_after_a_cursor_this_source_never_issued_refuses_instead_of_starting_past_it() + { + await SeedInIdOrder(FailedMessageStatus.Unresolved, FailedMessageStatus.Unresolved); + const string cursor = "FailedMessages/zzz-from-another-database"; + + await using var source = await OpenMigrationSource(); + var refusal = Assert.ThrowsAsync(() => CollectBatches(source, Category, resumeAfter: cursor)); + + Assert.That(refusal.Message, Does.Contain(Category.Id).And.Contain(cursor), "a cursor from a restored backup or another database would otherwise resume part way through and skip rows"); + } + + [Test] + public async Task Waits_for_the_index_to_catch_up_so_no_unresolved_message_is_missed() + { + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived)); + await CompleteDatabaseOperation(); + await DocumentStore.Maintenance.SendAsync(new StopIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); + + // The stopped index still counts the message as archived, so its count matches the collection, and only the wait for the index keeps the read from finding nothing. + using (var session = DocumentStore.OpenAsyncSession()) + { + (await session.LoadAsync(stored.Id, TestContext.CurrentContext.CancellationToken)).Status = FailedMessageStatus.Unresolved; + await session.SaveChangesAsync(TestContext.CurrentContext.CancellationToken); + } + + var reading = ReadAll(); + var finishedWhileStale = await Task.WhenAny(reading, Task.Delay(TimeSpan.FromSeconds(3), TestContext.CurrentContext.CancellationToken)) == reading; + + await DocumentStore.Maintenance.SendAsync(new StartIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); + var rows = await reading.WaitAsync(TimeSpan.FromSeconds(60), TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(finishedWhileStale, Is.False, "a query on an index that has not caught up leaves out the messages it has not indexed yet"); + Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { stored.Id })); + } + } + + [Test] + public async Task Resumes_after_a_cursor_that_names_an_archived_message() + { + var ids = await SeedInIdOrder( + FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Unresolved); + + await using var source = await OpenMigrationSource(); + var resumed = await CollectBatches(source, Category, resumeAfter: ids[1]); + + Assert.That(resumed.SelectMany(batch => batch.Rows).Select(row => row.SourceId), Is.EqualTo(new[] { ids[2], ids[4] }), "a run that read the whole collection can save an archived message as its cursor, and the next run may query the index"); + } + + [Test] + public async Task Reads_through_a_Corax_index_as_through_a_Lucene_one() + { + await UseCorax(); + var ids = await SeedInIdOrder(FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.RetryIssued, FailedMessageStatus.Resolved); + + await using var source = await OpenMigrationSource(); + var batches = await CollectBatches(source, Category); + + Assert.That(batches.Select(batch => (batch.Cursor, batch.Rows.Select(row => row.SourceId))), Is.EqualTo(new[] { (ids[2], new[] { ids[0], ids[2] }) }), "a full read would walk on to the resolved message and save its cursor past it"); + } + + [Test] + public async Task Resumes_through_a_Corax_index_with_no_row_repeated_or_skipped() + { + await UseCorax(); + var statuses = Enumerable.Range(0, 60).Select(position => (FailedMessageStatus)((position % 4) + 1)).ToArray(); + var ids = await SeedInIdOrder(statuses); + var expected = ids.Where((_, position) => statuses[position] is FailedMessageStatus.Unresolved or FailedMessageStatus.RetryIssued).ToList(); + + await using var source = await OpenMigrationSource(); + var read = new List(); + string cursor = null; + + // Every batch is read by a fresh call, as a restart after each commit would. + while ((await CollectBatches(source, Category, resumeAfter: cursor, batchSize: 7)).FirstOrDefault() is { } batch) + { + Assert.That(batch.Cursor, Is.Not.EqualTo(cursor), "a resume that hands back the batch it resumed after never ends"); + read.AddRange(batch.Rows.Select(row => row.SourceId)); + cursor = batch.Cursor; + } + + var afterArchived = await CollectBatches(source, Category, resumeAfter: ids[Array.IndexOf(statuses, FailedMessageStatus.Archived)]); + + using (Assert.EnterMultipleScope()) + { + Assert.That(read, Is.EqualTo(expected), "Corax has to order and compare id() as the cursor expects"); + Assert.That(afterArchived.SelectMany(batch => batch.Rows).Select(row => row.SourceId), Is.EqualTo(expected.SkipWhile(id => string.CompareOrdinal(id, ids[Array.IndexOf(statuses, FailedMessageStatus.Archived)]) < 0))); + } + } + + [Test] + public async Task Reads_the_whole_collection_when_the_index_is_missing() + { + await DocumentStore.Maintenance.SendAsync(new DeleteIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + + var rows = await ReadAll().WaitAsync(TimeSpan.FromSeconds(30), TestContext.CurrentContext.CancellationToken); + + Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { stored.Id }), "the copy cannot build the index on a source it only reads, so reading everything is the only way through"); + } + + [Test] + public async Task Reads_the_whole_collection_instead_of_waiting_for_a_disabled_index() + { + await DocumentStore.Maintenance.SendAsync(new DisableIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + + var rows = await ReadAll().WaitAsync(TimeSpan.FromSeconds(30), TestContext.CurrentContext.CancellationToken); + + Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { stored.Id }), "a disabled index never catches up, so waiting for it would keep ServiceControl closed for good"); + } + + [Test] + public async Task Reads_the_whole_collection_instead_of_waiting_for_an_errored_index() + { + var unindexable = await SeedFailedMessage(FailedMessageWithATimeSentThatIsNotADate()); + await CompleteDatabaseOperation(); + Assert.That(await IndexStateNow(), Is.EqualTo(IndexState.Error), "one document that fails out of one attempt errors the index"); + + // An errored index stops part way, so this one stays unindexed and the index stays stale. + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + + var rows = await ReadAll().WaitAsync(TimeSpan.FromSeconds(30), TestContext.CurrentContext.CancellationToken); + + Assert.That(rows.Select(row => row.SourceId), Is.EquivalentTo(new[] { unindexable.Id, stored.Id }), "an errored index never catches up, so waiting for it would keep ServiceControl closed for good"); + } + + [Test] + public async Task Reads_the_whole_collection_when_the_index_leaves_out_a_document_it_could_not_index() + { + await using (var bulkInsert = DocumentStore.BulkInsert(token: TestContext.CurrentContext.CancellationToken)) + { + foreach (var message in Enumerable.Range(0, 200).Select(_ => new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived))) + { + await bulkInsert.StoreAsync(message, FailedMessageIdGenerator.MakeDocumentId(message.UniqueMessageId)); + } + } + + var unindexable = await SeedFailedMessage(FailedMessageWithATimeSentThatIsNotADate()); + await CompleteDatabaseOperation(); + Assert.That(await IndexStateNow(), Is.EqualTo(IndexState.Normal), "RavenDB errors the whole index only above a 15 percent failure rate, so one bad document among many leaves it working and silently short"); + + var rows = await ReadAll(); + + Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { unindexable.Id }), "a query would leave out the document the index could not map, where the target should record it as a skip"); + } + + [Test] + public async Task A_batch_yields_as_soon_as_it_holds_batch_size_rows_however_far_below_the_walk_limit() + { + var ids = await SeedInIdOrder(Enumerable.Repeat(FailedMessageStatus.Unresolved, 5).ToArray()); + + var batches = await StreamWithWalkLimit(batchSize: 2, maxDocumentsWalkedPerBatch: 3); + + using (Assert.EnterMultipleScope()) + { + Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { ids[1], ids[3], ids[4] })); + Assert.That(batches.Select(batch => batch.Rows.Count), Is.EqualTo(new[] { 2, 2, 1 }), "a batch never holds more than the target can write in one go"); + } + } + + [Test] + public async Task A_batch_short_of_batch_size_rows_yields_once_the_walk_limit_is_reached() + { + var ids = await SeedInIdOrder( + FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Archived, FailedMessageStatus.Archived, + FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Archived); + + var batches = await StreamWithWalkLimit(batchSize: 2, maxDocumentsWalkedPerBatch: 4); + + using (Assert.EnterMultipleScope()) + { + Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { ids[3], ids[6] }), "a sparse stretch still saves its cursor every walk limit, so a resume walks at most that far again"); + Assert.That(batches.Select(batch => batch.Rows.Select(row => row.SourceId)), Is.EqualTo(new[] { new[] { ids[0] }, new[] { ids[4] } })); + } + } + + [Test] + public async Task A_full_batch_yields_on_rows_held_not_on_documents_walked() + { + var ids = await SeedInIdOrder( + FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Unresolved, FailedMessageStatus.Archived, FailedMessageStatus.Unresolved); + + var batches = await StreamWithWalkLimit(batchSize: 2, maxDocumentsWalkedPerBatch: 10); + + using (Assert.EnterMultipleScope()) + { + Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { ids[2], ids[4] }), "a batch is full when it holds batch size rows, however many documents it walked to find them"); + Assert.That(batches.Select(batch => batch.Rows.Select(row => row.SourceId)), Is.EqualTo(new[] { new[] { ids[0], ids[2] }, new[] { ids[4] } })); + } + } + + [TestCase(0)] + [TestCase(-1)] + public void Refuses_a_walk_limit_that_is_not_positive(int maxDocumentsWalkedPerBatch) + { + var refusal = Assert.ThrowsAsync(() => StreamWithWalkLimit(batchSize: 2, maxDocumentsWalkedPerBatch)); + + Assert.That(refusal.ParamName, Is.EqualTo("maxDocumentsWalkedPerBatch"), "a limit of zero or less is never reached, so every batch would wait for batch size rows however far it walked"); + } + + [Test] + public async Task The_category_saves_its_cursor_at_least_every_ten_thousand_documents_read() + { + var messages = Enumerable.Range(0, 10_001) + .Select(_ => new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)) + .OrderBy(message => FailedMessageIdGenerator.MakeDocumentId(message.UniqueMessageId), StringComparer.Ordinal) + .ToList(); + + await using (var bulkInsert = DocumentStore.BulkInsert(token: TestContext.CurrentContext.CancellationToken)) + { + foreach (var message in messages) + { + await bulkInsert.StoreAsync(message, FailedMessageIdGenerator.MakeDocumentId(message.UniqueMessageId)); + } + } + + await CompleteDatabaseOperation(); + + await using var source = await OpenMigrationSource(); + var batches = await CollectBatches(source, Category, batchSize: 20_000); + + Assert.That( + batches.Select(batch => (batch.Cursor, batch.Rows.Count)), + Is.EqualTo(new[] { (FailedMessageIdGenerator.MakeDocumentId(messages[9_999].UniqueMessageId), 10_000), (FailedMessageIdGenerator.MakeDocumentId(messages[10_000].UniqueMessageId), 1) }), + "the reader commits every RavenDocumentStream.MaxDocumentsWalkedPerBatch documents, which is 10,000, however large the batch size"); + } + + [Test] + public async Task A_document_stored_without_FailureDetails_comes_back_with_an_empty_failing_address() + { + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(numberOfAttempts: 2)); + await DocumentStore.Operations.SendAsync(new PatchOperation(stored.Id, null, new PatchRequest { Script = "delete this.ProcessingAttempts[0].FailureDetails; delete this.ProcessingAttempts[1].FailureDetails;" })); + await CompleteDatabaseOperation(); + var readStartedAt = DateTime.UtcNow; + + var attempts = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts; + + using (Assert.EnterMultipleScope()) + { + Assert.That(attempts.Select(attempt => attempt.FailureDetails?.AddressOfFailingEndpoint), Is.All.Empty, "a missing key reads back as the default object, not null, so the mapper has to treat an empty address as missing"); + Assert.That(attempts.Select(attempt => attempt.FailureDetails.TimeOfFailure), Is.All.GreaterThanOrEqualTo(readStartedAt), "the default object stamps the time it was read, so its time of failure is invented"); + } + } + + [Test] + public async Task A_FailureDetails_stored_without_its_TimeOfFailure_comes_back_with_no_time_rather_than_the_time_it_was_read() + { + var failure = new IngestedFailure { FailingEndpointAddress = "Sales@MACHINE" }; + var stored = await SeedFailedMessage(failure.ToFailedMessage(numberOfAttempts: 2)); + await DocumentStore.Operations.SendAsync(new PatchOperation(stored.Id, null, new PatchRequest { Script = "delete this.ProcessingAttempts[0].FailureDetails.TimeOfFailure; delete this.ProcessingAttempts[1].FailureDetails.TimeOfFailure;" })); + await CompleteDatabaseOperation(); + + var attempts = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts; + + using (Assert.EnterMultipleScope()) + { + Assert.That(attempts[^1].FailureDetails.TimeOfFailure, Is.EqualTo(default(DateTime)), "the mapper skips a newest attempt with no time of failure, and the read time would copy it with a made-up one"); + Assert.That(attempts[0].FailureDetails.TimeOfFailure, Is.EqualTo(default(DateTime)), "the mapper leaves an older attempt with no time out of the first and last failure times"); + Assert.That(attempts.Select(attempt => attempt.FailureDetails.AddressOfFailingEndpoint), Is.All.EqualTo("Sales@MACHINE"), "the rest of FailureDetails is still read as stored"); + Assert.That(attempts.Select(attempt => attempt.FailureDetails.Exception?.Message), Is.All.EqualTo(failure.ExceptionMessage)); + } + } + + [Test] + public async Task A_FailureDetails_stored_as_null_comes_back_as_null() + { + var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage()); + await DocumentStore.Operations.SendAsync(new PatchOperation(stored.Id, null, new PatchRequest { Script = "this.ProcessingAttempts[0].FailureDetails = null;" })); + await CompleteDatabaseOperation(); + + var attempt = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts.Single(); + + Assert.That(attempt.FailureDetails, Is.Null, "the mapper skips an attempt with no FailureDetails, so one bad document must not halt the category"); + } + + [TestCase("SendingEndpoint", NServiceBus.Headers.OriginatingHostId)] + [TestCase("ReceivingEndpoint", NServiceBus.Headers.HostId)] + public void RestoreMetadataTypes_gives_an_endpoint_the_empty_host_id_when_its_host_id_header_is_null(string key, string hostIdHeader) + { + var document = SenderAndReceiverHeadersFailure(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers[hostIdHeader] = null; + attempt.MessageMetadata[key] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata[key], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(Guid.Empty), "a header present with no value reads as no host id, as ingestion stores it, so the message copies"); + } + + [Test] + public void RestoreMetadataTypes_leaves_a_document_whose_attempt_list_is_null_for_the_mapper_to_skip() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts = null; + + Assert.DoesNotThrow(() => FailedMessageStream.RestoreMetadataTypes(document)); + } + + [Test] + public async Task Reads_a_document_whose_attempt_list_is_null_with_no_StatusChangedAt() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts = null; + await SeedFailedMessage(document); + await CompleteDatabaseOperation(); + + var row = (await ReadAll()).Single(); + + Assert.That(row.Metadata[MigrationMetadataKeys.StatusChangedAt], Is.Null, "the mapper skips a message with no attempts, so the source passes it through rather than halting the category"); + } + + [Test] + public async Task Hands_over_the_metadata_ingestion_reads_as_the_types_it_reads() + { + var failure = new IngestedFailure { IsSystemMessage = true }; + await SeedFailedMessage(failure.ToFailedMessage()); + await CompleteDatabaseOperation(); + + var metadata = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts.Single().MessageMetadata; + + using (Assert.EnterMultipleScope()) + { + Assert.That(metadata["TimeSent"], Is.TypeOf().And.EqualTo(failure.TimeSent)); + Assert.That(((DateTime)metadata["TimeSent"]).Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(metadata["SendingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(failure.SendingEndpoint.HostId)); + Assert.That(metadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(failure.ReceivingEndpoint.HostId)); + Assert.That(metadata["MessageType"], Is.TypeOf().And.EqualTo(failure.MessageType)); + Assert.That(metadata["ConversationId"], Is.TypeOf().And.EqualTo(failure.ConversationId)); + Assert.That(metadata["IsSystemMessage"], Is.TypeOf().And.True, "the mapper reads it with 'is true', so any other type reads as false"); + } + } + + [Test] + public async Task Restores_a_TimeSent_stored_as_text_and_an_endpoint_stored_without_its_type() + { + var document = new IngestedFailure().ToFailedMessage(); + var stored = document.ProcessingAttempts.Single(); + stored.Headers["NServiceBus.ProcessingMachine"] = "MACHINE"; + stored.MessageMetadata["TimeSent"] = "2026-07-22T09:59:00.0000000Z"; + stored.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + await SeedFailedMessage(document); + await CompleteDatabaseOperation(); + + var attempt = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts.Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(attempt.MessageMetadata["TimeSent"], Is.TypeOf().And.EqualTo(new DateTime(2026, 7, 22, 9, 59, 0, DateTimeKind.Utc))); + Assert.That(((DateTime)attempt.MessageMetadata["TimeSent"]).Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.Name)).EqualTo("Sales"), "rebuilt from the headers, not converted from the stored dictionary"); + Assert.That(((EndpointDetails)attempt.MessageMetadata["ReceivingEndpoint"]).Host, Is.EqualTo("MACHINE")); + } + } + + [Test] + public async Task Passes_a_TimeSent_that_is_not_a_date_through_as_stored_and_gives_endpoints_whose_host_id_is_not_a_GUID_the_empty_host_id() + { + var document = SenderAndReceiverHeadersFailure(); + var stored = document.ProcessingAttempts.Single(); + stored.Headers[NServiceBus.Headers.HostId] = "not-a-guid"; + stored.Headers[NServiceBus.Headers.OriginatingHostId] = "not-a-guid-either"; + stored.MessageMetadata["TimeSent"] = "garbage"; + stored.MessageMetadata["SendingEndpoint"] = new Dictionary { ["Name"] = "Stored sender" }; + stored.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored receiver" }; + await SeedFailedMessage(document); + await CompleteDatabaseOperation(); + + var metadata = ((FailedMessage)(await ReadAll()).Single().Document).ProcessingAttempts.Single().MessageMetadata; + + using (Assert.EnterMultipleScope()) + { + Assert.That(metadata["TimeSent"], Is.EqualTo("garbage"), "the target skips the document as malformed, so one bad document does not halt the category"); + Assert.That(metadata["SendingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(Guid.Empty)); + Assert.That(metadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(Guid.Empty)); + } + } + + [Test] + public void RestoreMetadataTypes_parses_text_as_UTC_and_rebuilds_untyped_endpoints_from_the_headers() + { + var document = new IngestedFailure().ToFailedMessage(numberOfAttempts: 2); + var sendingHostId = Guid.NewGuid(); + + foreach (var attempt in document.ProcessingAttempts) + { + attempt.Headers["NServiceBus.ProcessingMachine"] = "MACHINE"; + attempt.MessageMetadata["TimeSent"] = "2026-07-22T09:59:00.1234567"; + attempt.Headers[NServiceBus.Headers.OriginatingEndpoint] = "Ordering"; + attempt.Headers["NServiceBus.OriginatingMachine"] = "SENDER"; + attempt.Headers[NServiceBus.Headers.OriginatingHostId] = sendingHostId.ToString(); + attempt.MessageMetadata["SendingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + attempt.MessageMetadata["ReceivingEndpoint"] = "not an endpoint"; + } + + FailedMessageStream.RestoreMetadataTypes(document); + + var newest = document.ProcessingAttempts[^1]; + var older = document.ProcessingAttempts[0]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(newest.MessageMetadata["TimeSent"], Is.TypeOf().And.EqualTo(new DateTime(2026, 7, 22, 9, 59, 0, DateTimeKind.Utc).AddTicks(1234567))); + Assert.That(((DateTime)newest.MessageMetadata["TimeSent"]).Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(newest.MessageMetadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.Name)).EqualTo("Sales")); + var sending = (EndpointDetails)newest.MessageMetadata["SendingEndpoint"]; + Assert.That(sending, Is.Not.Null); + Assert.That(sending.Name, Is.EqualTo("Ordering")); + Assert.That(sending.Host, Is.EqualTo("SENDER")); + Assert.That(sending.HostId, Is.EqualTo(sendingHostId)); + Assert.That(older.MessageMetadata["TimeSent"], Is.EqualTo("2026-07-22T09:59:00.1234567"), "the mapper reads only the newest attempt, so an older one is left as stored"); + } + } + + [Test] + public void RestoreMetadataTypes_leaves_a_TimeSent_text_that_is_not_a_date_as_stored() + { + var document = new IngestedFailure().ToFailedMessage(); + var metadata = document.ProcessingAttempts.Single().MessageMetadata; + metadata["TimeSent"] = "garbage"; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(metadata["TimeSent"], Is.EqualTo("garbage"), "the target skips the document as malformed, so one bad document does not halt the category"); + } + + [TestCase("SendingEndpoint", NServiceBus.Headers.OriginatingHostId)] + [TestCase("ReceivingEndpoint", NServiceBus.Headers.HostId)] + public void RestoreMetadataTypes_gives_an_endpoint_the_empty_host_id_when_its_host_id_header_is_not_a_GUID(string key, string hostIdHeader) + { + var document = SenderAndReceiverHeadersFailure(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers[hostIdHeader] = "not-a-guid"; + attempt.MessageMetadata[key] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + var endpoint = (EndpointDetails)attempt.MessageMetadata[key]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(endpoint.HostId, Is.EqualTo(Guid.Empty), "ingestion stores an unparseable host id as the empty GUID, so the message copies rather than being skipped"); + Assert.That(endpoint.Name, Is.EqualTo(key == "SendingEndpoint" ? "Ordering" : "Sales")); + Assert.That(endpoint.Host, Is.EqualTo(key == "SendingEndpoint" ? "SENDER" : "MACHINE")); + } + } + + [Test] + public void RestoreMetadataTypes_builds_a_receiving_endpoint_from_the_failed_queue_when_the_headers_name_none() + { + var document = new IngestedFailure { QueueAddress = "Sales@MACHINE" }.ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers.Remove(NServiceBus.Headers.ProcessingEndpoint); + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + var endpoint = (EndpointDetails)attempt.MessageMetadata["ReceivingEndpoint"]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(endpoint.Name, Is.EqualTo("Sales")); + Assert.That(endpoint.Host, Is.EqualTo("MACHINE")); + Assert.That(endpoint.HostId, Is.EqualTo(new Guid("f0c10bfe-aba7-6ceb-d333-ab938515451c")), "ingestion derives the host id from the name and host when it has none"); + } + } + + [Test] + public void RestoreMetadataTypes_gives_a_receiving_endpoint_built_from_a_failed_queue_with_no_machine_the_unknown_host() + { + var document = new IngestedFailure { QueueAddress = "Sales" }.ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers.Remove(NServiceBus.Headers.ProcessingEndpoint); + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + var endpoint = (EndpointDetails)attempt.MessageMetadata["ReceivingEndpoint"]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(endpoint.Name, Is.EqualTo("Sales")); + Assert.That(endpoint.Host, Is.EqualTo("unknown")); + Assert.That(endpoint.HostId, Is.EqualTo(new Guid("862257db-407e-ec11-92cc-05ab0c3f7325"))); + } + } + + [Test] + public void RestoreMetadataTypes_leaves_a_newest_attempt_with_no_metadata_for_the_mapper_to_skip() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.MessageMetadata = null; + + Assert.DoesNotThrow(() => FailedMessageStream.RestoreMetadataTypes(document)); + Assert.That(attempt.MessageMetadata, Is.Null, "the mapper skips an attempt with no metadata, so an empty stand-in would hide the fault"); + } + + [Test] + public void RestoreMetadataTypes_leaves_an_untyped_endpoint_on_an_attempt_with_no_headers_for_the_mapper_to_skip() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers = null; + attempt.MessageMetadata["TimeSent"] = "2026-07-22T09:59:00Z"; + var stored = new Dictionary { ["Name"] = "Stored" }; + attempt.MessageMetadata["ReceivingEndpoint"] = stored; + + FailedMessageStream.RestoreMetadataTypes(document); + + using (Assert.EnterMultipleScope()) + { + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.SameAs(stored)); + Assert.That(attempt.MessageMetadata["TimeSent"], Is.EqualTo(new DateTime(2026, 7, 22, 9, 59, 0, DateTimeKind.Utc)), "TimeSent does not come from the headers, so it is still restored"); + } + } + + [Test] + public void RestoreMetadataTypes_lets_an_unexpected_fault_through() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers = new HeadersThatCannotBeRead(); + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + Assert.Throws(() => FailedMessageStream.RestoreMetadataTypes(document), "a fault the data cannot explain is a defect that must halt the category, not be swallowed as one bad document"); + } + + [TestCase(0)] + [TestCase(1)] + public async Task Reads_a_document_with_a_null_attempt_and_resolves_StatusChangedAt_from_the_others(int nullAttempt) + { + var failure = new IngestedFailure(); + var document = failure.ToFailedMessage(numberOfAttempts: 2); + var expected = document.ProcessingAttempts[1 - nullAttempt].AttemptedAt; + document.ProcessingAttempts[nullAttempt] = null; + await SeedFailedMessage(document); + await CompleteDatabaseOperation(); + + var row = (await ReadAll()).Single(); + + Assert.That(row.Metadata[MigrationMetadataKeys.StatusChangedAt], Is.EqualTo(expected), "a null attempt is a data fault for the mapper to skip, not a reason to halt the category"); + } + + [Test] + public void RestoreMetadataTypes_ignores_a_TimeSent_that_is_not_a_date_on_an_attempt_the_mapper_never_reads() + { + var document = new IngestedFailure().ToFailedMessage(numberOfAttempts: 2); + document.ProcessingAttempts[0].MessageMetadata["TimeSent"] = "garbage"; + + Assert.DoesNotThrow(() => FailedMessageStream.RestoreMetadataTypes(document), "a fault in data that is never copied must not halt the category"); + } + + [Test] + public void RestoreMetadataTypes_leaves_a_document_with_no_attempts_for_the_mapper_to_skip() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts.Clear(); + + Assert.DoesNotThrow(() => FailedMessageStream.RestoreMetadataTypes(document)); + } + + [Test] + public void RestoreMetadataTypes_keeps_the_receiving_host_id_header_on_the_named_header_path() + { + var hostId = Guid.NewGuid(); + var document = SenderAndReceiverHeadersFailure(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers[NServiceBus.Headers.HostId] = hostId.ToString(); + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(hostId)); + } + + [Test] + public void RestoreMetadataTypes_keeps_the_receiving_host_id_header_on_the_failed_queue_path() + { + var hostId = Guid.NewGuid(); + var document = new IngestedFailure { QueueAddress = "Sales@MACHINE" }.ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers.Remove(NServiceBus.Headers.ProcessingEndpoint); + attempt.Headers[NServiceBus.Headers.HostId] = hostId.ToString(); + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.HostId)).EqualTo(hostId), "a host id the headers carry is not replaced by one derived from the name and host"); + } + + [Test] + public void RestoreMetadataTypes_prefers_the_host_display_name_to_the_processing_machine() + { + var document = SenderAndReceiverHeadersFailure(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers[NServiceBus.Headers.HostDisplayName] = "DISPLAY"; + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.TypeOf().And.Property(nameof(EndpointDetails.Host)).EqualTo("DISPLAY")); + } + + [Test] + public void RestoreMetadataTypes_does_not_throw_when_every_header_it_reads_is_null() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + + foreach (var header in new[] + { + NServiceBus.Headers.OriginatingEndpoint, "NServiceBus.OriginatingMachine", NServiceBus.Headers.OriginatingHostId, NServiceBus.Headers.OriginatingAddress, + NServiceBus.Headers.ProcessingEndpoint, NServiceBus.Headers.HostDisplayName, "NServiceBus.ProcessingMachine", NServiceBus.Headers.HostId, "NServiceBus.FailedQ" + }) + { + attempt.Headers[header] = null; + } + + attempt.MessageMetadata["SendingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + Assert.DoesNotThrow(() => FailedMessageStream.RestoreMetadataTypes(document), "with the catch gone, a throw on any header value would halt the whole category"); + } + + [Test] + public void RestoreMetadataTypes_restores_no_sending_endpoint_when_its_address_header_is_null() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers[NServiceBus.Headers.OriginatingAddress] = null; + attempt.MessageMetadata["SendingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata["SendingEndpoint"], Is.Null, "ingestion stores no endpoint when the address it falls back to has no value"); + } + + [Test] + public void RestoreMetadataTypes_restores_no_receiving_endpoint_when_its_failed_queue_header_is_null() + { + var document = new IngestedFailure().ToFailedMessage(); + var attempt = document.ProcessingAttempts.Single(); + attempt.Headers["NServiceBus.FailedQ"] = null; + attempt.MessageMetadata["ReceivingEndpoint"] = new Dictionary { ["Name"] = "Stored" }; + + FailedMessageStream.RestoreMetadataTypes(document); + + Assert.That(attempt.MessageMetadata["ReceivingEndpoint"], Is.Null, "ingestion stores no endpoint when the failed queue it falls back to has no value"); + } + + static FailedMessage SenderAndReceiverHeadersFailure() + { + var document = new IngestedFailure().ToFailedMessage(); + var headers = document.ProcessingAttempts.Single().Headers; + headers[NServiceBus.Headers.OriginatingEndpoint] = "Ordering"; + headers["NServiceBus.OriginatingMachine"] = "SENDER"; + headers["NServiceBus.ProcessingMachine"] = "MACHINE"; + return document; + } + + // The index cannot turn this TimeSent into a date, so RavenDB fails to index the document. + static FailedMessage FailedMessageWithATimeSentThatIsNotADate() + { + var document = new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved); + document.ProcessingAttempts.Single().MessageMetadata["TimeSent"] = "garbage"; + return document; + } + + async Task UseCorax() + { + await DocumentStore.Maintenance.SendAsync(new DeleteIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); + await new FailedMessageViewIndex { Configuration = { ["Indexing.Static.SearchEngineType"] = SearchEngineType.Corax.ToString() } }.ExecuteAsync(DocumentStore, token: TestContext.CurrentContext.CancellationToken); + + var engine = (await DocumentStore.Maintenance.SendAsync(new GetIndexStatisticsOperation(IndexName), TestContext.CurrentContext.CancellationToken)).SearchEngineType; + Assert.That(engine, Is.EqualTo(SearchEngineType.Corax), "the test proves nothing about Corax if the index still runs on Lucene"); + } + + async Task IndexStateNow() => + (await DocumentStore.Maintenance.SendAsync(new GetStatisticsOperation(), TestContext.CurrentContext.CancellationToken)).Indexes.Single(index => index.Name == IndexName).State; + + async Task> ReadAll() + { + await using var source = await OpenMigrationSource(); + + return (await CollectBatches(source, Category)).SelectMany(batch => batch.Rows).ToList(); + } + + // RavenDB streams in id order, so statuses are handed out by sorted id rather than by seeding order. + async Task> SeedInIdOrder(params FailedMessageStatus[] statuses) + { + var messages = statuses.Select(_ => new IngestedFailure().ToFailedMessage()) + .OrderBy(message => FailedMessageIdGenerator.MakeDocumentId(message.UniqueMessageId), StringComparer.Ordinal) + .ToList(); + var ids = new List(messages.Count); + + for (var position = 0; position < messages.Count; position++) + { + messages[position].Status = statuses[position]; + ids.Add((await SeedFailedMessage(messages[position])).Id); + } + + await CompleteDatabaseOperation(); + + return ids; + } + + // Takes the walk limit as a parameter so a test can reach it without seeding the 10,000 documents of RavenDocumentStream.MaxDocumentsWalkedPerBatch. + async Task> StreamWithWalkLimit(int batchSize, int maxDocumentsWalkedPerBatch) + { + var settings = (RavenPersisterSettings)PersistenceSettings; + await using var lifecycle = new RavenReadOnlySourceLifecycle(settings, new SettingsRootNamespace("ServiceControl")); + await lifecycle.Open(TestContext.CurrentContext.CancellationToken); + + var batches = new List(); + + await foreach (var batch in RavenDocumentStream.ByPrefix( + lifecycle, Category.Id, settings.DatabaseName, $"{FailedMessageIdGenerator.CollectionName}/", null, batchSize, maxDocumentsWalkedPerBatch, + new FailedMessageStream(lifecycle).Project(MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses), TestContext.CurrentContext.CancellationToken)) + { + batches.Add(batch); + } + + return batches; + } + + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + static readonly string IndexName = new FailedMessageViewIndex().IndexName; + + // The parsers read headers through IReadOnlyDictionary, so re-implementing its lookup makes every header read fail. + sealed class HeadersThatCannotBeRead : Dictionary, IReadOnlyDictionary + { + bool IReadOnlyDictionary.TryGetValue(string key, out string value) => + throw new InvalidOperationException("The headers cannot be read."); + } +} diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/MigrationGroupInsertSeekTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationGroupInsertSeekTests.cs new file mode 100644 index 0000000000..f1914f23b1 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationGroupInsertSeekTests.cs @@ -0,0 +1,69 @@ +// ReSharper disable once CheckNamespace +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class MigrationGroupInsertSeekTests : PersistenceTestBase +{ + static readonly MigrationCategory FailedMessagesCategory = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + IMigrationTarget Target => ServiceProvider.GetRequiredService(); + + [SetUp] + public Task OpenTarget() => Target.Open(); + + [Test] + public async Task The_group_insert_seeks_existing_rows_by_key_while_the_group_table_is_small() + { + List sharedGroups = + [ + .. new[] { "Exception Type and Stack Trace", "Message Type", "Endpoint Address", "Endpoint Instance", "Endpoint Name" } + .Select(type => new FailedMessage.FailureGroup { Id = Guid.NewGuid().ToString(), Title = $"{type} title", Type = type }) + ]; + var rows = Enumerable.Range(0, 74) + .Select(_ => new IngestedFailure { Groups = sharedGroups }.ToFailedMessage()) + .Select(document => new MigrationRow($"FailedMessages/{document.UniqueMessageId}", document, new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + [MigrationMetadataKeys.LastModified] = new DateTime(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc) + }, new MigrationBody("1"u8.ToArray(), "text/xml"))) + .ToList(); + + var result = await Target.Write(FailedMessagesCategory, new MigrationBatch(rows, rows[^1].SourceId), new MigrationCheckpoint(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null)); + + using var scope = ServiceProvider.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(74)); + Assert.That(await GroupTableReadsInCachedPlan(db), Is.Not.Empty.And.None.Contains("Scan"), "a scan reads every row of an index and HOLDLOCK locks each one, which FORCESEEK prevents"); + } + } + + // Seek and scan operators both hold an IndexScan element, so the RelOp above it says which one it is. + static Task> GroupTableReadsInCachedPlan(ServiceControlDbContext db) + { + var statement = $"MERGE {MigrationInsert.For(db).Table} "; + return db.Database.SqlQuery($""" + WITH XMLNAMESPACES (DEFAULT 'http://schemas.microsoft.com/sqlserver/2004/07/showplan') + SELECT o.n.value('../../@PhysicalOp', 'nvarchar(60)') + ' ' + o.n.value('@Index', 'nvarchar(260)') AS [Value] + FROM sys.dm_exec_cached_plans AS p + CROSS APPLY sys.dm_exec_sql_text(p.plan_handle) AS t + CROSS APPLY sys.dm_exec_query_plan(p.plan_handle) AS q + CROSS APPLY q.query_plan.nodes('//RelOp/IndexScan/Object[@Table="[FailedMessageGroups]"]') AS o(n) + WHERE t.dbid = DB_ID() AND CHARINDEX({statement}, t.text) > 0 + """).ToListAsync(); + } +} diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/MigrationPlanReuseTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationPlanReuseTests.cs new file mode 100644 index 0000000000..f6135cc924 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationPlanReuseTests.cs @@ -0,0 +1,78 @@ +// ReSharper disable once CheckNamespace +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class MigrationPlanReuseTests : PersistenceTestBase +{ + static readonly MigrationCategory FailedMessagesCategory = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + IMigrationTarget Target => ServiceProvider.GetRequiredService(); + + [SetUp] + public Task OpenTarget() => Target.Open(); + + [Test] + public async Task Batches_whose_unbounded_text_sits_either_side_of_4000_characters_share_one_cached_plan() + { + var shortHeaders = new IngestedFailure(); + shortHeaders.Headers["Padding"] = new string('h', 3_000); + var longHeaders = new IngestedFailure { Groups = [new() { Id = Guid.NewGuid().ToString(), Title = new string('t', 5_000), Type = "Message Type" }] }; + longHeaders.Headers["Padding"] = new string('h', 5_000); + + await Target.Write(FailedMessagesCategory, BatchOf(shortHeaders.ToFailedMessage()), Checkpoint()); + var saved = await ServiceProvider.GetRequiredService().Read(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + var result = await Target.Write(FailedMessagesCategory, BatchOf(longHeaders.ToFailedMessage()), saved); + + using var scope = ServiceProvider.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + var headersJsonLengths = await db.FailedMessages.AsNoTracking().Select(message => message.HeadersJson.Length).ToListAsync(); + var titleLengths = await db.FailedMessageGroups.AsNoTracking().Select(group => group.Title.Length).ToListAsync(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(headersJsonLengths, Has.Exactly(1).LessThan(4_000).And.Exactly(1).GreaterThan(4_000), "one HeadersJson must fit nvarchar(4000) and the other must not. If the long one is exactly 4000, the insert cut it short"); + Assert.That(titleLengths, Does.Contain(5_000), "the long group Title must be stored whole"); + Assert.That(await PlanUseCountsFor(db), Is.EqualTo(new[] { 2 }), "short and long text must reuse one plan, because each new parameter declaration list compiles its own"); + Assert.That(await PlanUseCountsFor(db), Is.EqualTo(new[] { 2 }), "the group Title column is unbounded too"); + } + } + + // Scoped to this test's schema, because the suite's other fixtures share the server and its plan cache. + static Task> PlanUseCountsFor(ServiceControlDbContext db) where TEntity : class + { + var statement = $"MERGE {MigrationInsert.For(db).Table} "; + return db.Database.SqlQuery($""" + SELECT p.usecounts AS [Value] + FROM sys.dm_exec_cached_plans AS p + CROSS APPLY sys.dm_exec_sql_text(p.plan_handle) AS t + WHERE t.dbid = DB_ID() AND CHARINDEX({statement}, t.text) > 0 + """).ToListAsync(); + } + + static MigrationBatch BatchOf(FailedMessage document) + { + var row = new MigrationRow($"FailedMessages/{document.UniqueMessageId}", document, new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + [MigrationMetadataKeys.LastModified] = new DateTime(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc) + }, new MigrationBody("1"u8.ToArray(), "text/xml")); + + return new MigrationBatch([row], row.SourceId); + } + + static MigrationCheckpoint Checkpoint() => + new(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null); +} diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/MigrationValueLengthTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationValueLengthTests.cs new file mode 100644 index 0000000000..7f024576bc --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.SqlServer/MigrationValueLengthTests.cs @@ -0,0 +1,70 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class MigrationValueLengthTests : PersistenceTestBase +{ + static readonly string OutsideTheBasicPlane = char.ConvertFromUtf32(0x1F600); + + static readonly MigrationCategory KnownEndpointsCategory = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.KnownEndpoints); + + IMigrationTarget Target => ServiceProvider.GetRequiredService(); + + [SetUp] + public Task OpenTarget() => Target.Open(); + + [Test] + public async Task A_value_is_measured_in_utf16_units_as_nvarchar_counts_them() + { + var fits = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = Repeat(225), HostId = Guid.NewGuid(), Host = "SALES01" } }; + var tooLong = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = Repeat(226), HostId = Guid.NewGuid(), Host = "SALES02" } }; + var batch = new MigrationBatch( + [ + new MigrationRow("KnownEndpoints/1", fits, new Dictionary()), + new MigrationRow("KnownEndpoints/2", tooLong, new Dictionary()) + ], + "KnownEndpoints/2"); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new KnownEndpointsWriter(ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(KnownEndpointsCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.KnownEndpoints, MigrationCategoryState.InProgress, "KnownEndpoints/2", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "225 code points are 450 UTF-16 units, which an nvarchar(450) column holds"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { "KnownEndpoints/2" }), "226 code points are 452 UTF-16 units, which would fail the insert"); + Assert.That((await MonitoringDataStore.GetAllKnownEndpoints()).Select(endpoint => endpoint.EndpointDetails.Name), Is.EqualTo(new[] { Repeat(225) })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(KnownEndpointEntity.Name))} takes at most 450 characters and the value holds 452, on document 'KnownEndpoints/2'" + }), "the length reported is the one SQL Server counts"); + Assert.That(ServiceProvider.GetRequiredService().TextLength(Repeat(226)), Is.LessThanOrEqualTo(Repeat(226).Length), "the check skips measuring a value whose UTF-16 length fits, which is safe only while no count exceeds it"); + } + } + + [Test] + public async Task A_value_holding_a_NUL_character_copies() + { + var withNul = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = "Sales", HostId = Guid.NewGuid(), Host = "SAL\u0000ES" } }; + var batch = new MigrationBatch([new MigrationRow("KnownEndpoints/1", withNul, new Dictionary())], "KnownEndpoints/1"); + + var result = await Target.Write(KnownEndpointsCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.KnownEndpoints, MigrationCategoryState.InProgress, "KnownEndpoints/1", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "nvarchar stores U+0000, so a skip here would give up a row SQL Server can hold"); + Assert.That((await MonitoringDataStore.GetAllKnownEndpoints()).Single().EndpointDetails.Host, Is.EqualTo("SAL\u0000ES")); + } + } + + static string Repeat(int codePoints) => string.Concat(Enumerable.Repeat(OutsideTheBasicPlane, codePoints)); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/EndpointSettingsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/EndpointSettingsMigrationTargetTests.cs index 2826d26011..a6e01f9fc4 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/EndpointSettingsMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/EndpointSettingsMigrationTargetTests.cs @@ -7,7 +7,10 @@ namespace ServiceControl.Persistence.Tests; using Microsoft.Extensions.DependencyInjection; using NUnit.Framework; using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; class EndpointSettingsMigrationTargetTests : PersistenceTestBase { @@ -78,6 +81,31 @@ public async Task A_setting_whose_endpoint_is_not_known_is_copied_and_left_to_th } } + [Test] + public async Task A_setting_whose_name_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy() + { + var batch = BatchOf( + ("EndpointSettings/1", new EndpointSettings { Name = new string('s', 450), TrackInstances = true }), + ("EndpointSettings/2", new EndpointSettings { Name = new string('n', 451), TrackInstances = true }), + ("EndpointSettings/3", new EndpointSettings { Name = "Billing", TrackInstances = false })); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new EndpointSettingsWriter(ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(EndpointSettingsCategory, batch, CheckpointAfter("EndpointSettings/3")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2), "a name exactly as long as its column fits"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { "EndpointSettings/2" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.ValueTooLong] = 1 }), "an over-long value would fail the whole insert and halt the category"); + Assert.That(await TrackInstancesFor(new string('s', 450)), Is.True); + Assert.That(await TrackInstancesFor("Billing"), Is.False); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(EndpointSettingsEntity.Name))} takes at most 450 characters and the value holds 451, on document 'EndpointSettings/2'" + })); + } + } + [Test] public async Task A_batch_that_copies_nothing_still_saves_the_cursor() { diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs new file mode 100644 index 0000000000..616031e0c2 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs @@ -0,0 +1,59 @@ +#nullable enable +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; + +class FailedMessageMigrationCoverageTests : PersistenceTestBase +{ + [Test] + public void Every_failed_message_column_is_set_from_a_fully_populated_document() + { + var source = FullyPopulatedRow(); + var row = ((FailedMessageRowMapping.Mapped)FailedMessageRowMapper.Map(source)).Row; + // Over the cap, so it sets all four body columns: an inline prefix, the external flag, the size and the content type. + var overCapText = Encoding.UTF8.GetBytes(new string('a', BodyStorageSettings.DefaultMaxBodySizeToStore * 2)); + + FailedMessageRowMapper.SetBody(row, ((FailedMessage)source.Document).ProcessingAttempts[^1].Headers, new MigrationBody(overCapText, "text/plain"), BodyStorageSettings.DefaultMaxBodySizeToStore); + + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(Model(), row); + } + + [Test] + public void Every_failure_group_column_is_set_from_a_fully_populated_document() + { + var mapping = (FailedMessageRowMapping.Mapped)FailedMessageRowMapper.Map(FullyPopulatedRow()); + + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(Model(), mapping.Groups.Single()); + } + + static MigrationRow FullyPopulatedRow() + { + var failure = new IngestedFailure { IsSystemMessage = true, FailingEndpointAddress = "Sales@MACHINE" }; + + return new MigrationRow( + $"FailedMessages/{failure.UniqueMessageIdString}", + failure.ToFailedMessage(), + new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + [MigrationMetadataKeys.LastModified] = new DateTime(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc) + }); + } + + IModel Model() + { + using var scope = ServiceProvider.CreateScope(); + return scope.ServiceProvider.GetRequiredService().Model; + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageRowMapperTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageRowMapperTests.cs new file mode 100644 index 0000000000..93f1ac1583 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageRowMapperTests.cs @@ -0,0 +1,551 @@ +#nullable enable +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.Entities; + +[TestFixture] +[FixtureLifeCycle(LifeCycle.InstancePerTestCase)] +class FailedMessageRowMapperTests +{ + static readonly DateTime StatusChangedAtFromTheSource = new(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc); + static readonly DateTime LastModifiedFromTheSource = new(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc); + + [Test] + public void Carries_the_unique_message_id_across_unchanged() + { + var uniqueMessageId = Guid.NewGuid(); + var failure = new IngestedFailure(); + var document = failure.ToFailedMessage(); + document.UniqueMessageId = uniqueMessageId.ToString(); + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(row.UniqueMessageId, Is.EqualTo(uniqueMessageId)); + } + + [Test] + public void Skips_a_unique_message_id_that_is_not_a_guid() + { + var document = new IngestedFailure().ToFailedMessage(); + document.UniqueMessageId = "not-a-guid"; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.UnparseableUniqueMessageId)); + Assert.That(skip.Detail, Does.Contain("not-a-guid")); + } + } + + [Test] + public void Skips_a_document_with_no_processing_attempts() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts.Clear(); + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.NoProcessingAttempts)); + Assert.That(skip.Detail, Does.Contain(document.UniqueMessageId)); + } + } + + [Test] + public void Skips_a_document_with_no_processing_attempts_that_the_source_sent_with_no_StatusChangedAt() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts.Clear(); + var complete = Row(document); + var row = complete with { Metadata = new Dictionary(complete.Metadata) { [MigrationMetadataKeys.StatusChangedAt] = null } }; + + var skip = SkipOf(FailedMessageRowMapper.Map(row)); + + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.NoProcessingAttempts), "the source resolves StatusChangedAt from the newest attempt, so an attemptless document always arrives without one"); + } + + [Test] + public void Skips_a_document_whose_newest_attempt_has_no_failing_endpoint_address() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts[^1].FailureDetails.AddressOfFailingEndpoint = null!; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing), "FailingEndpointAddress is NOT NULL, so the insert would fail and halt the category"); + Assert.That(skip.Detail, Does.Contain(nameof(FailedMessageEntity.FailingEndpointAddress)).And.Contain(document.UniqueMessageId), "the log has to name the column and the message an operator would look up"); + } + } + + [TestCase(nameof(FailedMessage.ProcessingAttempt.FailureDetails))] + [TestCase(nameof(FailedMessage.ProcessingAttempt.MessageMetadata))] + [TestCase(nameof(FailedMessage.ProcessingAttempt.Headers))] + public void Skips_a_newest_attempt_with_no_value_the_row_is_built_from(string missing) + { + var document = new IngestedFailure().ToFailedMessage(); + var newest = document.ProcessingAttempts[^1]; + switch (missing) + { + case nameof(newest.FailureDetails): + newest.FailureDetails = null!; + break; + case nameof(newest.MessageMetadata): + newest.MessageMetadata = null!; + break; + default: + newest.Headers = null!; + break; + } + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing)); + Assert.That(skip.Detail, Does.Contain(missing).And.Contain(document.UniqueMessageId), "the log has to name what is missing and the message an operator would look up"); + } + } + + [TestCase("TimeSent", "garbage", "garbage")] + [TestCase("MessageType", 43L, "43")] + [TestCase("ConversationId", 44L, "44")] + [TestCase("IsSystemMessage", "yes", "yes")] + public void Skips_a_value_present_with_the_wrong_type_and_names_the_key_the_type_found_and_the_value(string key, object malformed, string shown) + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts[^1].MessageMetadata[key] = malformed; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.MalformedValue), "copying it as null or false would lose a value the source holds without a skip to show for it"); + Assert.That(skip.Detail, Does.Contain(key).And.Contain(malformed.GetType().FullName).And.Contain(document.UniqueMessageId)); + Assert.That(skip.Detail, Does.Contain($"'{shown}'"), "the operator should not have to open RavenDB to see the value"); + } + } + + [Test] + public void Shows_only_the_first_100_characters_of_a_malformed_value() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts[^1].MessageMetadata["TimeSent"] = new string('a', 100) + new string('b', 400); + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(skip.Detail, Does.Contain($"'{new string('a', 100)}...'").And.Not.Contain(new string('a', 100) + "b"), "a stored value can be any length, and the detail is one log line"); + } + + [Test] + public void Skips_a_newest_attempt_whose_FailureDetails_key_was_missing_from_the_document() + { + var document = new IngestedFailure().ToFailedMessage(); + // What RavenDB hands back for an attempt stored without the key. + document.ProcessingAttempts[^1].FailureDetails = new FailedMessage.ProcessingAttempt().FailureDetails; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing), "copying it would store an empty failing address to retry to and a time of failure that is the time it was read"); + Assert.That(skip.Detail, Does.Contain(nameof(FailedMessageEntity.FailingEndpointAddress)).And.Contain(document.UniqueMessageId)); + } + } + + [TestCase("TimeOfFailure")] + [TestCase("AttemptedAt")] + public void Skips_a_newest_attempt_with_no_time_it_failed_or_was_attempted(string missing) + { + var document = new IngestedFailure().ToFailedMessage(); + var newest = document.ProcessingAttempts[^1]; + if (missing == "TimeOfFailure") + { + newest.FailureDetails.TimeOfFailure = default; + } + else + { + newest.AttemptedAt = default; + } + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing), "the year 1 is not a time the message failed"); + Assert.That(skip.Detail, Does.Contain(missing).And.Contain(document.UniqueMessageId)); + } + } + + [Test] + public void Leaves_an_older_attempt_whose_FailureDetails_key_was_missing_out_of_the_first_and_last_failure_times() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[0].FailureDetails = new FailedMessage.ProcessingAttempt().FailureDetails; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[1].FailureDetails.TimeOfFailure)); + Assert.That(row.LastTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[2].FailureDetails.TimeOfFailure), "the default object's time of failure is the time it was read, which would become the last failure"); + } + } + + [Test] + public void Leaves_an_older_attempt_with_no_time_of_failure_out_of_the_first_failure_time() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[0].FailureDetails.TimeOfFailure = default; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[1].FailureDetails.TimeOfFailure), "the year 1 is not a time the message failed"); + } + + [Test] + public void Skips_a_document_whose_processing_attempts_list_is_null() + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts = null!; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.NoProcessingAttempts)); + } + + [Test] + public void Maps_a_null_failure_groups_list_to_no_groups() + { + var document = new IngestedFailure().ToFailedMessage(); + document.FailureGroups = null!; + + var mapping = FailedMessageRowMapper.Map(Row(document)); + + Assert.That(((FailedMessageRowMapping.Mapped)mapping).Groups, Is.Empty); + } + + [Test] + public void Drops_a_failure_group_entry_that_is_null() + { + var document = new IngestedFailure().ToFailedMessage(); + document.FailureGroups = [null!, new() { Id = "group-1", Title = "Title 1", Type = "Type 1" }]; + + var mapping = FailedMessageRowMapper.Map(Row(document)); + + Assert.That(((FailedMessageRowMapping.Mapped)mapping).Groups.Select(group => group.GroupId), Is.EqualTo(new[] { "group-1" })); + } + + [TestCase("SendingEndpoint", "not an endpoint")] + [TestCase("ReceivingEndpoint", 42)] + public void Refuses_an_endpoint_that_is_not_an_EndpointDetails_rather_than_copying_it_as_null(string key, object wrongType) + { + var document = new IngestedFailure().ToFailedMessage(); + document.ProcessingAttempts[^1].MessageMetadata[key] = wrongType; + + Assert.Throws(() => FailedMessageRowMapper.Map(Row(document)), "the source restores every endpoint before the mapper sees it, so a wrong type is a defect that must halt the copy, not a row with empty endpoint columns"); + } + + [TestCase(true)] + [TestCase(false)] + public void Maps_a_TimeSent_or_endpoint_that_is_missing_or_null_to_null(bool keyPresent) + { + var document = new IngestedFailure().ToFailedMessage(); + var metadata = document.ProcessingAttempts[^1].MessageMetadata; + foreach (var key in new[] { "TimeSent", "SendingEndpoint", "ReceivingEndpoint" }) + { + if (keyPresent) + { + metadata[key] = null!; + } + else + { + metadata.Remove(key); + } + } + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.TimeSent, Is.Null); + Assert.That(new[] { row.SendingEndpointName, row.SendingEndpointHost, row.ReceivingEndpointName, row.ReceivingEndpointHost }, Is.All.Null, "RavenDB's own read path shows a missing endpoint as null"); + Assert.That(new[] { row.SendingEndpointHostId, row.ReceivingEndpointHostId }, Is.All.Null); + } + } + + [Test] + public void Skips_a_document_whose_newest_attempt_is_null() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[2] = null!; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing)); + Assert.That(skip.Detail, Does.Contain(document.UniqueMessageId)); + } + } + + [Test] + public void Takes_the_earliest_failure_from_the_attempts_that_have_failure_details() + { + var document = ThreeAttempts(); + var earliestWithDetails = document.ProcessingAttempts[1].FailureDetails.TimeOfFailure; + document.ProcessingAttempts[0].FailureDetails = null!; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(earliestWithDetails), "an older attempt with no failure details has no time to offer, and must not halt the category"); + Assert.That(row.LastTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[2].FailureDetails.TimeOfFailure)); + } + } + + [Test] + public void Ignores_an_earlier_attempt_that_is_null() + { + var document = ThreeAttempts(); + var earliestRemaining = document.ProcessingAttempts[1].FailureDetails.TimeOfFailure; + document.ProcessingAttempts[0] = null!; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.NumberOfProcessingAttempts, Is.EqualTo(3), "RavenDB's read path counts every stored attempt, null ones included"); + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(earliestRemaining)); + Assert.That(row.LastTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[2].FailureDetails.TimeOfFailure)); + } + } + + [Test] + public void Maps_when_only_an_earlier_attempt_has_a_value_of_the_wrong_type() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[0].MessageMetadata["TimeSent"] = "garbage"; + document.ProcessingAttempts[0].MessageMetadata["ReceivingEndpoint"] = "not an endpoint"; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(row.FailingEndpointAddress, Is.EqualTo("newest-address"), "only the newest attempt's metadata becomes the row"); + } + + [Test] + public void Keeps_the_last_450_characters_of_a_message_type_longer_than_the_column() + { + var document = new IngestedFailure().ToFailedMessage(); + var longType = new string('a', 100) + new string('b', 450); + document.ProcessingAttempts[^1].MessageMetadata["MessageType"] = longType; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(row.MessageType, Is.EqualTo(new string('b', 450))); + } + + [Test] + public void Takes_the_newest_attempt_for_the_row_and_the_earliest_and_latest_failure_across_all() + { + var document = ThreeAttempts(); + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.NumberOfProcessingAttempts, Is.EqualTo(3)); + Assert.That(row.FirstTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[0].FailureDetails.TimeOfFailure)); + Assert.That(row.LastTimeOfFailure, Is.EqualTo(document.ProcessingAttempts[2].FailureDetails.TimeOfFailure)); + Assert.That(row.LastAttemptedAt, Is.EqualTo(document.ProcessingAttempts[2].AttemptedAt)); + Assert.That(row.FailingEndpointAddress, Is.EqualTo("newest-address")); + Assert.That(row.ExceptionMessage, Is.EqualTo("newest-exception")); + } + } + + [Test] + public void Skips_when_only_the_newest_of_three_attempts_has_no_failing_endpoint_address() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[2].FailureDetails.AddressOfFailingEndpoint = null!; + + var skip = SkipOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(skip.Reason, Is.EqualTo(MigrationSkipReason.RequiredValueMissing)); + } + + [Test] + public void Maps_when_only_an_earlier_attempt_has_no_failing_endpoint_address() + { + var document = ThreeAttempts(); + document.ProcessingAttempts[0].FailureDetails.AddressOfFailingEndpoint = null!; + + var row = RowOf(FailedMessageRowMapper.Map(Row(document))); + + Assert.That(row.FailingEndpointAddress, Is.EqualTo("newest-address")); + } + + [Test] + public void Maps_each_group_with_an_id_to_a_row_keyed_by_the_message_and_drops_a_group_with_no_id() + { + var uniqueMessageId = Guid.NewGuid(); + var document = new IngestedFailure().ToFailedMessage(); + document.UniqueMessageId = uniqueMessageId.ToString(); + document.FailureGroups = + [ + new() { Id = "group-1", Title = "Title 1", Type = "Type 1" }, + new() { Id = "group-2", Title = null, Type = null }, + new() { Id = null!, Title = "No id", Type = "Type" } + ]; + + var mapping = (FailedMessageRowMapping.Mapped)FailedMessageRowMapper.Map(Row(document)); + + Assert.That( + mapping.Groups.Select(group => (group.FailedMessageUniqueId, group.GroupId, group.Title, group.Type)), + Is.EqualTo(new[] + { + (uniqueMessageId, "group-1", "Title 1", "Type 1"), + (uniqueMessageId, "group-2", string.Empty, string.Empty) + })); + } + + [TestCase(FailedMessageStatus.Unresolved)] + [TestCase(FailedMessageStatus.Resolved)] + [TestCase(FailedMessageStatus.RetryIssued)] + [TestCase(FailedMessageStatus.Archived)] + public void Preserves_the_stored_status(FailedMessageStatus status) + { + var row = RowOf(FailedMessageRowMapper.Map(Row(new IngestedFailure().ToFailedMessage(status)))); + + Assert.That(row.Status, Is.EqualTo(status)); + } + + [Test] + public void Stores_the_StatusChangedAt_the_source_resolved() + { + var row = RowOf(FailedMessageRowMapper.Map(Row(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived)))); + + Assert.That(row.StatusChangedAt, Is.EqualTo(StatusChangedAtFromTheSource)); + } + + [Test] + public void Stores_the_LastModified_the_source_carried() + { + var row = RowOf(FailedMessageRowMapper.Map(Row(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Archived)))); + + Assert.That(row.LastModified, Is.EqualTo(LastModifiedFromTheSource), "RavenDB's @last-modified, not the copy time"); + } + + [TestCase(MigrationMetadataKeys.StatusChangedAt)] + [TestCase(MigrationMetadataKeys.LastModified)] + public void Refuses_a_row_the_source_sent_without_a_resolved_value(string missingKey) + { + var complete = Row(new IngestedFailure().ToFailedMessage()); + var row = complete with { Metadata = complete.Metadata.Where(entry => entry.Key != missingKey).ToDictionary(entry => entry.Key, entry => entry.Value) }; + + var ex = Assert.Throws(() => FailedMessageRowMapper.Map(row)); + + Assert.That(ex.Message, Does.Contain(missingKey).And.Contain(row.SourceId)); + } + + [TestCase(MigrationMetadataKeys.StatusChangedAt)] + [TestCase(MigrationMetadataKeys.LastModified)] + public void Refuses_a_resolved_value_of_the_wrong_type_and_names_the_type_it_found(string key) + { + var complete = Row(new IngestedFailure().ToFailedMessage()); + var row = complete with { Metadata = new Dictionary(complete.Metadata) { [key] = "2026-09-01T08:00:00Z" } }; + + var ex = Assert.Throws(() => FailedMessageRowMapper.Map(row)); + + Assert.That(ex.Message, Does.Contain(typeof(string).FullName).And.Contain(key).And.Contain(row.SourceId), "a value that is present but mistyped must not read as missing"); + } + + [TestCase(DateTimeKind.Unspecified)] + [TestCase(DateTimeKind.Local)] + public void Writes_every_date_as_UTC_whatever_kind_it_arrived_with(DateTimeKind kind) + { + var document = ThreeAttempts(); + foreach (var attempt in document.ProcessingAttempts) + { + attempt.FailureDetails.TimeOfFailure = DateTime.SpecifyKind(attempt.FailureDetails.TimeOfFailure, kind); + attempt.AttemptedAt = DateTime.SpecifyKind(attempt.AttemptedAt, kind); + } + + var newest = document.ProcessingAttempts[^1]; + var timeSent = new DateTime(2026, 7, 22, 9, 59, 0, kind); + newest.MessageMetadata["TimeSent"] = timeSent; + var statusChangedAt = DateTime.SpecifyKind(StatusChangedAtFromTheSource, kind); + var lastModified = DateTime.SpecifyKind(LastModifiedFromTheSource, kind); + var complete = Row(document); + var source = complete with + { + Metadata = new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = statusChangedAt, + [MigrationMetadataKeys.LastModified] = lastModified + } + }; + + var row = RowOf(FailedMessageRowMapper.Map(source)); + + DateTime[] written = [row.StatusChangedAt, row.LastModified, row.FirstTimeOfFailure, row.LastTimeOfFailure, row.LastAttemptedAt, row.TimeSent!.Value]; + DateTime[] expected = [statusChangedAt, lastModified, document.ProcessingAttempts[0].FailureDetails.TimeOfFailure, newest.FailureDetails.TimeOfFailure, newest.AttemptedAt, timeSent]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(written.Select(value => value.Kind), Is.All.EqualTo(DateTimeKind.Utc), "Npgsql refuses to write a DateTime of any other kind to a timestamp with time zone column"); + Assert.That(written, Is.EqualTo(expected.Select(value => kind == DateTimeKind.Local ? value.ToUniversalTime() : DateTime.SpecifyKind(value, DateTimeKind.Utc))), "a value with no kind is already UTC and must not be shifted by the local offset"); + } + } + + static FailedMessageEntity RowOf(FailedMessageRowMapping mapping) + { + Assert.That(mapping, Is.TypeOf()); + return ((FailedMessageRowMapping.Mapped)mapping).Row; + } + + static FailedMessageRowMapping.Skipped SkipOf(FailedMessageRowMapping mapping) + { + Assert.That(mapping, Is.TypeOf()); + return (FailedMessageRowMapping.Skipped)mapping; + } + + static FailedMessage ThreeAttempts() + { + var failure = new IngestedFailure(); + var document = failure.ToFailedMessage(numberOfAttempts: 3); + for (var i = 0; i < 3; i++) + { + var attempt = document.ProcessingAttempts[i]; + attempt.FailureDetails.TimeOfFailure = new DateTime(2026, 7, 22, 10 + i, 0, 0, DateTimeKind.Utc); + attempt.AttemptedAt = attempt.FailureDetails.TimeOfFailure; + } + + var newest = document.ProcessingAttempts[2]; + newest.FailureDetails.AddressOfFailingEndpoint = "newest-address"; + newest.FailureDetails.Exception!.Message = "newest-exception"; + return document; + } + + // Every row carries the two dates the source resolves, as a real source's rows do. + static MigrationRow Row(FailedMessage document) => + new( + $"FailedMessages/{document.UniqueMessageId}", + document, + new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = StatusChangedAtFromTheSource, + [MigrationMetadataKeys.LastModified] = LastModifiedFromTheSource + }); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs new file mode 100644 index 0000000000..59f058963a --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs @@ -0,0 +1,402 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class FailedMessagesMigrationTargetTests : PersistenceTestBase +{ + [SetUp] + public Task OpenTarget() => Target.Open(); + + IMigrationTarget Target => ServiceProvider.GetRequiredService(); + + IMigrationCheckpointStore CheckpointStore => ServiceProvider.GetRequiredService(); + + IBodyStoragePersistence BodyStore => ServiceProvider.GetRequiredService(); + + static readonly MigrationBody SmallTextBody = new(Encoding.UTF8.GetBytes("1"), "text/xml"); + + static readonly MigrationCategory FailedMessagesCategory = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + [Test] + public async Task A_copied_message_arrives_with_its_failure_group_rows() + { + var failure = new IngestedFailure(); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(failure.ToFailedMessage()), Checkpoint()); + + var groups = await Query(db => db.FailedMessageGroups.AsNoTracking().Where(group => group.FailedMessageUniqueId == failure.UniqueMessageId).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(groups.Select(group => (group.GroupId, group.Title, group.Type)), Is.EqualTo(failure.Groups.Select(group => (group.Id, group.Title, group.Type)))); + } + } + + [Test] + public async Task A_message_already_in_the_target_keeps_its_row_and_gets_no_group_rows() + { + var failure = new IngestedFailure(); + await Query(db => + { + db.FailedMessages.Add(new FailedMessageEntity { UniqueMessageId = failure.UniqueMessageId, Status = FailedMessageStatus.Unresolved, ExceptionMessage = "Already in the target", HeadersJson = "{}", FailingEndpointAddress = "error" }); + return db.SaveChangesAsync(); + }); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(failure.ToFailedMessage()), Checkpoint()); + + var row = await Query(db => db.FailedMessages.AsNoTracking().SingleAsync(message => message.UniqueMessageId == failure.UniqueMessageId)); + var groups = await Query(db => db.FailedMessageGroups.AsNoTracking().CountAsync(group => group.FailedMessageUniqueId == failure.UniqueMessageId)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.Zero); + Assert.That(result.AlreadyPresent, Is.EqualTo(1)); + Assert.That(result.Skipped, Is.Zero, "nothing was lost, so a skip would push a healthy category toward the halt threshold"); + Assert.That(row.ExceptionMessage, Is.EqualTo("Already in the target"), "insert-if-absent never updates"); + Assert.That(groups, Is.Zero, "the copy wrote the source's groups onto a message it did not insert"); + } + } + + [Test] + public async Task Copied_counts_the_messages_inserted_not_the_messages_handed_in() + { + var present = new IngestedFailure(); + var absent = new IngestedFailure(); + await Target.Write(FailedMessagesCategory, BatchOf(present.ToFailedMessage()), Checkpoint()); + var saved = await CheckpointStore.Read(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(present.ToFailedMessage(), absent.ToFailedMessage()), saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.AlreadyPresent, Is.EqualTo(1)); + } + } + + [Test] + public async Task Two_documents_sharing_a_unique_message_id_store_only_the_first_documents_groups() + { + var first = new IngestedFailure { Groups = [new() { Id = "group-of-the-first", Title = "First", Type = "Message Type" }] }; + var second = new IngestedFailure { Groups = [new() { Id = "group-of-the-second", Title = "Second", Type = "Message Type" }] }; + var secondDocument = second.ToFailedMessage(); + secondDocument.UniqueMessageId = first.UniqueMessageIdString; + + var result = await Target.Write(FailedMessagesCategory, BatchOf(first.ToFailedMessage(), secondDocument), Checkpoint()); + + var groupIds = await Query(db => db.FailedMessageGroups.AsNoTracking().Where(group => group.FailedMessageUniqueId == first.UniqueMessageId).Select(group => group.GroupId).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.AlreadyPresent, Is.EqualTo(1)); + Assert.That(groupIds, Is.EqualTo(new[] { "group-of-the-first" }), "the declined document's groups landed on the message the insert kept"); + } + } + + [Test] + public async Task A_document_the_mapper_skips_is_counted_under_its_reason_and_never_written() + { + var good = new IngestedFailure(); + var unparseable = new IngestedFailure().ToFailedMessage(); + unparseable.UniqueMessageId = "not-a-guid"; + var attemptless = new IngestedFailure().ToFailedMessage(); + attemptless.ProcessingAttempts.Clear(); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(good.ToFailedMessage(), unparseable, attemptless), Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.Skipped, Is.EqualTo(2)); + Assert.That(result.AlreadyPresent, Is.Zero); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary + { + [MigrationSkipReason.UnparseableUniqueMessageId] = 1, + [MigrationSkipReason.NoProcessingAttempts] = 1 + })); + Assert.That(await Query(db => db.FailedMessages.CountAsync()), Is.EqualTo(1)); + } + } + + [Test] + public async Task A_document_with_a_value_of_the_wrong_type_or_no_failure_details_is_skipped_and_the_rest_copy() + { + var good = new IngestedFailure(); + var badTimeSent = new IngestedFailure().ToFailedMessage(); + badTimeSent.ProcessingAttempts[^1].MessageMetadata["TimeSent"] = "garbage"; + var noFailureDetails = new IngestedFailure().ToFailedMessage(); + noFailureDetails.ProcessingAttempts[^1].FailureDetails = null; + + var result = await Target.Write(FailedMessagesCategory, BatchOf(badTimeSent, good.ToFailedMessage(), noFailureDetails), Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkippedIds, Is.EquivalentTo(new[] { $"FailedMessages/{badTimeSent.UniqueMessageId}", $"FailedMessages/{noFailureDetails.UniqueMessageId}" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary + { + [MigrationSkipReason.MalformedValue] = 1, + [MigrationSkipReason.RequiredValueMissing] = 1 + }), "one malformed document no longer halts the category"); + Assert.That(await Query(db => db.FailedMessages.AsNoTracking().Select(message => message.UniqueMessageId).ToListAsync()), Is.EqualTo(new[] { good.UniqueMessageId })); + } + } + + [Test] + public async Task A_message_or_group_value_longer_than_its_column_skips_the_message_whole_and_the_rest_copy() + { + var fitsExactly = new IngestedFailure + { + ConversationId = new string('c', 450), + Groups = [new() { Id = new string('g', 64), Title = "OrderPlaced", Type = new string('t', 255) }] + }; + var longConversation = new IngestedFailure { ConversationId = new string('c', 451) }; + var longGroupType = new IngestedFailure { Groups = [new() { Id = Guid.NewGuid().ToString(), Title = "OrderPlaced", Type = new string('t', 256) }] }; + var longMessageType = new IngestedFailure { MessageType = new string('m', 600) }; + var batch = BatchOf(fitsExactly.ToFailedMessage(), longConversation.ToFailedMessage(), longGroupType.ToFailedMessage(), longMessageType.ToFailedMessage()); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new FailedMessagesWriter(ServiceProvider.GetRequiredService(), BodyStore, ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + + var copiedIds = await Query(db => db.FailedMessages.AsNoTracking().Select(message => message.UniqueMessageId).ToListAsync()); + var groupOwners = await Query(db => db.FailedMessageGroups.AsNoTracking().Select(group => group.FailedMessageUniqueId).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2), "values exactly as long as their columns fit, and MessageType keeps its last 450 characters as ingestion does, so neither is skipped for length"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { $"FailedMessages/{longConversation.UniqueMessageIdString}", $"FailedMessages/{longGroupType.UniqueMessageIdString}" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.ValueTooLong] = 2 }), "an over-long value would fail the whole insert and halt the category"); + Assert.That(copiedIds, Is.EquivalentTo(new[] { fitsExactly.UniqueMessageId, longMessageType.UniqueMessageId })); + Assert.That(groupOwners, Is.EquivalentTo(new[] { fitsExactly.UniqueMessageId, longMessageType.UniqueMessageId }), "a message whose group row cannot be stored is skipped whole, not copied without its group"); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(FailedMessageEntity.ConversationId))} takes at most 450 characters and the value holds 451, on document 'FailedMessages/{longConversation.UniqueMessageIdString}'", + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(FailedMessageGroupEntity.Type))} takes at most 255 characters and the value holds 256, on document 'FailedMessages/{longGroupType.UniqueMessageIdString}'" + })); + } + } + + [Test] + public async Task Count_takes_in_only_unresolved_and_retry_issued_messages() + { + await Target.Write(FailedMessagesCategory, BatchOf(new IngestedFailure().ToFailedMessage(), new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued)), Checkpoint()); + await Query(db => + { + db.FailedMessages.AddRange( + new FailedMessageEntity { UniqueMessageId = Guid.NewGuid(), Status = FailedMessageStatus.Archived, HeadersJson = "{}", FailingEndpointAddress = "error" }, + new FailedMessageEntity { UniqueMessageId = Guid.NewGuid(), Status = FailedMessageStatus.Resolved, HeadersJson = "{}", FailingEndpointAddress = "error" }); + return db.SaveChangesAsync(); + }); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await Target.Count(FailedMessagesCategory), Is.EqualTo(2), "the archived and resolved rows belong to another category"); + Assert.That(await Query(db => db.FailedMessages.CountAsync()), Is.EqualTo(4), "the filter is doing the work, not an empty table"); + } + } + + // PostgreSQL takes a fixed number of rows per statement whatever the entity, so only SQL Server can catch the wrong entity here. + [Test] + public async Task A_batch_is_one_statement_of_failed_message_rows() + { + var dialect = ServiceProvider.GetRequiredService(); + var expected = await Query(db => Task.FromResult(dialect.RowsPerStatement(MigrationInsert.For(db).Columns.Count))); + + Assert.That(await Target.BatchSizeFor(FailedMessagesCategory), Is.EqualTo(expected)); + } + + [Test] + public async Task A_write_larger_than_one_statement_copies_every_message_and_its_groups() + { + var rowsPerStatement = await Target.BatchSizeFor(FailedMessagesCategory); + var documents = Enumerable.Range(0, (rowsPerStatement * 2) + 1).Select(_ => new IngestedFailure().ToFailedMessage()).ToArray(); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(documents), Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(documents.Length)); + Assert.That(await Query(db => db.FailedMessages.CountAsync()), Is.EqualTo(documents.Length)); + Assert.That(await Query(db => db.FailedMessageGroups.CountAsync()), Is.EqualTo(documents.Length)); + } + } + + [Test] + public async Task A_message_already_in_the_target_has_no_body_written_and_one_it_lacks_does() + { + var present = new IngestedFailure { ContentType = "application/octet-stream" }; + var absent = new IngestedFailure { ContentType = "application/octet-stream" }; + var binaryBody = new MigrationBody(new byte[] { 0x08, 0x00, 0x96, 0x01 }, "application/octet-stream"); + await Query(db => + { + db.FailedMessages.Add(new FailedMessageEntity { UniqueMessageId = present.UniqueMessageId, Status = FailedMessageStatus.Unresolved, HeadersJson = "{}", FailingEndpointAddress = "error" }); + return db.SaveChangesAsync(); + }); + + var result = await Target.Write(FailedMessagesCategory, BatchOf(RowOf(present.ToFailedMessage(), binaryBody), RowOf(absent.ToFailedMessage(), binaryBody)), Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(await BodyStored(present.UniqueMessageIdString), Is.False, "the insert declines a message the target holds, so a body written for it is a wasted write"); + Assert.That(await BodyStored(absent.UniqueMessageIdString), Is.True, "the test proves nothing if no body is ever written"); + } + } + + [Test] + public async Task Only_the_first_document_sharing_a_unique_message_id_has_its_body_stored() + { + var first = new IngestedFailure(); + var second = new IngestedFailure { ContentType = "application/octet-stream" }; + var secondDocument = second.ToFailedMessage(); + secondDocument.UniqueMessageId = first.UniqueMessageIdString; + + var result = await Target.Write(FailedMessagesCategory, BatchOf( + RowOf(first.ToFailedMessage(), SmallTextBody), + RowOf(secondDocument, new MigrationBody(new byte[] { 0x08, 0x00, 0x96, 0x01 }, "application/octet-stream"))), Checkpoint()); + + var row = await Query(db => db.FailedMessages.AsNoTracking().SingleAsync(message => message.UniqueMessageId == first.UniqueMessageId)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(row.BodyStoredExternally, Is.False, "the insert keeps the first document, whose body is inline"); + Assert.That(await BodyStored(first.UniqueMessageIdString), Is.False, "the declined document's body was stored under the id of the row the insert kept"); + } + } + + [Test] + public async Task A_body_content_type_longer_than_its_column_skips_the_message_and_the_rest_copy() + { + var good = new IngestedFailure(); + var longContentType = new IngestedFailure().ToFailedMessage(); + + var result = await Target.Write(FailedMessagesCategory, BatchOf( + RowOf(good.ToFailedMessage(), SmallTextBody), + RowOf(longContentType, SmallTextBody with { ContentType = $"text/{new string('x', 446)}" })), Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { $"FailedMessages/{longContentType.UniqueMessageId}" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.ValueTooLong] = 1 }), "an over-long content type would fail the whole insert and halt the category"); + } + } + + [Test] + public async Task A_body_that_cannot_be_written_fails_the_write_before_any_row_or_checkpoint_commits() + { + var target = new EFCoreMigrationTarget( + ServiceProvider.GetRequiredService(), + ServiceProvider.GetRequiredService(), + new UnwritableBodyStore(), + ServiceProvider.GetRequiredService(), + ServiceProvider.GetRequiredService>()); + var inline = new IngestedFailure(); + var binary = new IngestedFailure { ContentType = "application/octet-stream" }; + + var exception = Assert.ThrowsAsync(() => target.Write(FailedMessagesCategory, BatchOf( + RowOf(inline.ToFailedMessage(), SmallTextBody), + RowOf(binary.ToFailedMessage(), new MigrationBody(new byte[] { 0x08, 0x00, 0x96, 0x01 }, "application/octet-stream"))), Checkpoint())); + + using (Assert.EnterMultipleScope()) + { + Assert.That(exception.Message, Does.Contain($"'FailedMessages/{binary.UniqueMessageId}'").And.Contain($"'{binary.UniqueMessageId}'").And.Contain(nameof(IOException)), + "a storage SDK's own error does not name the body, so without the row and body id the operator cannot tell which message halted the category"); + Assert.That(exception.InnerException, Is.TypeOf().And.Message.EqualTo(UnwritableBodyStore.Failure)); + Assert.That(await Query(db => db.FailedMessages.CountAsync()), Is.Zero, "a row committed without its body would name a body that was never stored"); + Assert.That(await CheckpointStore.Read(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages), Is.Null, "a checkpoint past the batch would stop a retry from reading it again"); + } + } + + [Test] + public void A_document_that_cannot_be_prepared_fails_the_write_naming_the_row_and_category() + { + // A content type header stored with a null value is found by the classifier and then throws. + var document = new IngestedFailure { ContentType = null }.ToFailedMessage(); + + var exception = Assert.ThrowsAsync(() => Target.Write(FailedMessagesCategory, BatchOf(document), Checkpoint())); + + using (Assert.EnterMultipleScope()) + { + Assert.That(exception.Message, Does.Contain($"'FailedMessages/{document.UniqueMessageId}'").And.Contain(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages).And.Contain(nameof(NullReferenceException)), + "the halt's LastError is built from this message, and without the row the operator cannot tell which document to look at"); + Assert.That(exception.InnerException, Is.TypeOf()); + } + } + + [Test] + public void A_row_that_reaches_the_writer_without_a_body_fails_the_write_naming_the_row_and_category() + { + var document = new IngestedFailure().ToFailedMessage(); + + var exception = Assert.ThrowsAsync(() => Target.Write(FailedMessagesCategory, BatchOf(RowOf(document, body: null)), Checkpoint())); + + Assert.That(exception.Message, Does.Contain($"'FailedMessages/{document.UniqueMessageId}'").And.Contain(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)); + } + + // Disposes the stream, because the file store holds the body file open until it is. + async Task BodyStored(string bodyId) + { + var stored = await BodyStore.ReadBody(bodyId); + if (stored is null) + { + return false; + } + + await stored.Stream.DisposeAsync(); + return true; + } + + async Task Query(Func> query) + { + using var scope = ServiceProvider.CreateScope(); + return await query(scope.ServiceProvider.GetRequiredService()); + } + + static MigrationBatch BatchOf(params FailedMessage[] documents) => BatchOf([.. documents.Select(document => RowOf(document, SmallTextBody))]); + + static MigrationBatch BatchOf(params MigrationRow[] rows) => new(rows, rows[^1].SourceId); + + // Every row carries the two dates the source resolves, as a real source's rows do. + static MigrationRow RowOf(FailedMessage document, MigrationBody body) => + new($"FailedMessages/{document.UniqueMessageId}", document, new Dictionary + { + [MigrationMetadataKeys.StatusChangedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + [MigrationMetadataKeys.LastModified] = new DateTime(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc) + }, body); + + sealed class UnwritableBodyStore : IBodyStoragePersistence + { + public const string Failure = "The body store is unreachable"; + + public Task WriteBody(string bodyId, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) => Task.FromException(new IOException(Failure)); + + public Task ReadBody(string bodyId, CancellationToken cancellationToken = default) => throw new NotSupportedException(); + + public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => throw new NotSupportedException(); + } + + static MigrationCheckpoint Checkpoint() => + new(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/KnownEndpointsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/KnownEndpointsMigrationTargetTests.cs index 41cf3f5bd5..f63dbdfdd0 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/KnownEndpointsMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/KnownEndpointsMigrationTargetTests.cs @@ -10,7 +10,10 @@ namespace ServiceControl.Persistence.Tests; using ServiceControl.Operations; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; class KnownEndpointsMigrationTargetTests : PersistenceTestBase { @@ -81,6 +84,31 @@ public async Task A_known_endpoint_with_no_name_or_no_host_is_skipped_as_require } } + [Test] + public async Task A_known_endpoint_with_a_value_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy() + { + var fitsExactly = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = new string('n', 450), HostId = Guid.NewGuid(), Host = new string('h', 450) } }; + var longName = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = new string('n', 451), HostId = Guid.NewGuid(), Host = "SALES02" } }; + var longHost = new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = "Billing", HostId = Guid.NewGuid(), Host = new string('h', 452) } }; + var batch = BatchOf(("KnownEndpoints/1", fitsExactly), ("KnownEndpoints/2", longName), ("KnownEndpoints/3", longHost)); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new KnownEndpointsWriter(ServiceProvider.GetRequiredService()), batch); + var result = await Target.Write(KnownEndpointsCategory, batch, CheckpointAfter("KnownEndpoints/3")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "a value exactly as long as its column fits"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { "KnownEndpoints/2", "KnownEndpoints/3" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary { [MigrationSkipReason.ValueTooLong] = 2 }), "an over-long value would fail the whole insert and halt the category"); + Assert.That((await MonitoringDataStore.GetAllKnownEndpoints()).Select(endpoint => endpoint.EndpointDetails.Name), Is.EqualTo(new[] { new string('n', 450) })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(KnownEndpointEntity.Name))} takes at most 450 characters and the value holds 451, on document 'KnownEndpoints/2'", + $"{PreparedSkips.TableAndColumnOf(ServiceProvider, nameof(KnownEndpointEntity.Host))} takes at most 450 characters and the value holds 452, on document 'KnownEndpoints/3'" + })); + } + } + [Test] public async Task A_batch_of_nothing_but_skips_saves_the_cursor_and_writes_no_rows() { diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs index da87ec7dcd..eb7caebdcc 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.Tests; +using System.Collections.Generic; using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.DependencyInjection; @@ -13,7 +14,7 @@ public async Task Every_category_the_target_supports_has_a_batch_size_and_a_coun { var target = ServiceProvider.GetRequiredService(); - Assert.That(target.SupportedCategoryIds, Is.SupersetOf(new[] { MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.EndpointSettings }), "the test proves nothing if the target supports no category"); + Assert.That(target.SupportedCategoryIds, Is.SupersetOf(new[] { MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.EndpointSettings, MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages }), "the test proves nothing if the target supports no category"); foreach (var id in target.SupportedCategoryIds) { @@ -23,4 +24,21 @@ public async Task Every_category_the_target_supports_has_a_batch_size_and_a_coun Assert.That(await target.Count(category, CancellationToken.None), Is.Zero, id); } } + + [Test] + public void Every_category_the_target_supports_has_a_test_for_a_value_longer_than_its_column() + { + var target = ServiceProvider.GetRequiredService(); + + Assert.That(target.SupportedCategoryIds, Is.EquivalentTo(ValueTooLongTests.Keys), + "a writer that does not run the length check sends an over-long value to the insert, which fails the whole statement and halts the category; give the new category a ValueTooLong test like those listed here, then add it to this list"); + } + + // Uses nameof, so renaming or deleting one of these tests fails the build instead of leaving this list out of date. + static readonly Dictionary ValueTooLongTests = new() + { + [MigrationCategoryIds.KnownEndpoints] = nameof(KnownEndpointsMigrationTargetTests.A_known_endpoint_with_a_value_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), + [MigrationCategoryIds.EndpointSettings] = nameof(EndpointSettingsMigrationTargetTests.A_setting_whose_name_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), + [MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages] = nameof(FailedMessagesMigrationTargetTests.A_message_or_group_value_longer_than_its_column_skips_the_message_whole_and_the_rest_copy) + }; } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs new file mode 100644 index 0000000000..e04ebaee93 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs @@ -0,0 +1,36 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; + +static class PreparedSkips +{ + /// + /// Prepares a batch through one writer without writing anything, so a test can read the detail line the target logs for each skip. + /// + public static async Task> DetailsFrom(IServiceProvider services, IMigrationCategoryWriter writer, MigrationBatch batch) + { + using var scope = services.CreateScope(); + var prepared = await writer.Prepare(scope.ServiceProvider.GetRequiredService(), batch); + + return [.. prepared.Skips.Select(skip => skip.Detail)]; + } + + /// + /// The table and column a property is stored in, as this provider names them. + /// + public static string TableAndColumnOf(IServiceProvider services, string property) + { + using var scope = services.CreateScope(); + var entityType = scope.ServiceProvider.GetRequiredService().Model.FindEntityType(typeof(TEntity))!; + + return $"{entityType.GetTableName()}.{entityType.FindProperty(property)!.GetColumnName()}"; + } +} diff --git a/src/ServiceControl.Persistence/DataMigration/IMigrationSource.cs b/src/ServiceControl.Persistence/DataMigration/IMigrationSource.cs index 7e969ffa5f..d6c40216b4 100644 --- a/src/ServiceControl.Persistence/DataMigration/IMigrationSource.cs +++ b/src/ServiceControl.Persistence/DataMigration/IMigrationSource.cs @@ -40,7 +40,6 @@ public interface IMigrationSource : IAsyncDisposable /// /// Reads a category in batches, after the checkpoint cursor, or from the start when it is null. Throws on a cursor it never issued. /// - /// A ceiling, not a target: returning fewer costs nothing, returning more fails the target's write. IAsyncEnumerable Read( MigrationCategory category, string? resumeAfter, @@ -48,7 +47,8 @@ IAsyncEnumerable Read( CancellationToken cancellationToken = default); /// - /// Reads one row's message body when Read did not attach it. Returns null when the row has no body. + /// Reads one row's message body when Read did not attach it. The engine calls it only for a category that + /// carries bodies, and only for a row whose body Read left null. Returns null when the source has no body for the row. /// Task ReadBody(MigrationCategory category, string sourceId, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs b/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs index 1e457a27a8..e2b93060ab 100644 --- a/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs +++ b/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs @@ -24,8 +24,10 @@ public interface IMigrationTarget /// /// Saves the batch's rows and the checkpoint in one transaction, so progress never gets ahead of the data. /// Add this batch's own outcome with and save that, so the stored counts are never a guess. + /// For a category that carries bodies, every row must have its body. + /// Bodies that go to external storage are written first, outside the transaction. So a failed write can leave a + /// stored body that no row points to, but never a row that points to a body that was not stored. /// - /// Prior totals, the cursor this batch reached, and any rows the engine itself skipped. Nothing the target does is counted yet. Task Write( MigrationCategory category, MigrationBatch batch, @@ -55,10 +57,4 @@ Task Write( /// /// What the target did with one batch, and the checkpoint it committed alongside the rows. Every skipped row must have a reason in SkipReasons. /// -/// The checkpoint as the target stored it, carrying the version that save landed on. The engine carries on from this one, never from the one it passed in. -/// Rows this batch wrote. The same number must show up as the rise in the saved copied count, because the halt threshold reads one and status and verify read the other. -/// Rows this batch could not write, benign ones included. -/// The source ids of those rows, so the engine can name each one in the log. -/// Rows the target already held. They were neither copied nor skipped, and they still count as accounted for. -/// How many rows each reason skipped. The counts have to add up to Skipped, or the checkpoint refuses the batch. public sealed record MigrationWriteResult(MigrationCheckpoint Saved, int Copied, int Skipped, IReadOnlyList SkippedIds, int AlreadyPresent = 0, IReadOnlyDictionary? SkipReasons = null); diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationBodyLoader.cs b/src/ServiceControl.Persistence/DataMigration/MigrationBodyLoader.cs new file mode 100644 index 0000000000..e8af7f8ab2 --- /dev/null +++ b/src/ServiceControl.Persistence/DataMigration/MigrationBodyLoader.cs @@ -0,0 +1,129 @@ +namespace ServiceControl.Persistence.DataMigration; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; + +/// +/// The rows returns for one batch, each with its body, and the rows it left out +/// with the reason for each. +/// +sealed record MigrationBodyLoadResult(IReadOnlyList Rows, IReadOnlyList<(string SourceId, MigrationSkipReason Reason)> Skipped); + +/// +/// Attaches a message body to every row of a batch that arrived without one, reading at most +/// bodies from the source at a time. +/// +sealed class MigrationBodyLoader(IMigrationSource source, TimeProvider timeProvider, TimeSpan retryBackoff, ILogger logger) +{ + /// + /// How many body reads are open against the source at once. Each body is its own round trip, so reading them one at + /// a time makes the copy as slow as the link, while opening a whole batch at once puts a load spike on a source that + /// may still be live. + /// + public const int MaxConcurrentFetches = 8; + + /// + /// Gives every row without a body the body the source holds for it. A row that already has one keeps it and + /// is never read. A read that fails is tried up to times, + /// and a row whose every attempt fails is left out as . A row + /// the source holds no body for is read once and left out as . + /// Each skip is logged as a warning. + /// + public async Task Load(MigrationCategory category, MigrationBatch batch, CancellationToken cancellationToken = default) + { + using var inFlight = new SemaphoreSlim(MaxConcurrentFetches); + + var fetched = await Task.WhenAll(batch.Rows.Select(async row => + { + if (row.Body is not null) + { + return (row.Body, LastAttemptError: null); + } + + // The slot is held for the whole fetch, retries included, so each message uses one slot rather than one per attempt. + await inFlight.WaitAsync(cancellationToken); + + try + { + return await Fetch(category, row.SourceId, cancellationToken); + } + finally + { + inFlight.Release(); + } + })); + + var rows = new List(batch.Rows.Count); + var skipped = new List<(string SourceId, MigrationSkipReason Reason)>(); + + // Rebuilt in the batch's order, because that is the cursor's order and the fetches finish in any order. + for (var index = 0; index < batch.Rows.Count; index++) + { + var row = batch.Rows[index]; + var (body, lastAttemptError) = fetched[index]; + + if (body is not null) + { + rows.Add(row with { Body = body }); + } + else if (lastAttemptError is null) + { + logger.LogWarning("Skipped {SourceId} in category {CategoryId} as {Reason}: the source has no body for it, either because it holds the message without one, which a --migration-retry meets again, or because the message expired during the copy", + row.SourceId, category.Id, MigrationSkipReason.BodyMissing); + skipped.Add((row.SourceId, MigrationSkipReason.BodyMissing)); + } + else + { + logger.LogWarning(lastAttemptError, "Skipped {SourceId} in category {CategoryId} as {Reason}: body unreadable after {MaxAttempts} attempts", row.SourceId, category.Id, MigrationSkipReason.BodyUnreadable, MigrationEngine.MaxBodyReadAttempts); + skipped.Add((row.SourceId, MigrationSkipReason.BodyUnreadable)); + } + } + + return new MigrationBodyLoadResult(rows, skipped); + } + + // A null body with no error means the source holds no body; with an error, every attempt failed. + async Task<(MigrationBody? Body, Exception? LastAttemptError)> Fetch(MigrationCategory category, string sourceId, CancellationToken cancellationToken) + { + Exception? lastAttemptError = null; + + for (var attempt = 1; attempt <= MigrationEngine.MaxBodyReadAttempts; attempt++) + { + try + { + return (await source.ReadBody(category, sourceId, cancellationToken), null); + } + // Only the run's own cancellation stops here; a client timeout raised as a cancellation is retried. + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + // Logged here because the exception that halts the category does not say which message it was reading. + catch (Exception ex) when (IsDefect(ex)) + { + logger.LogError(ex, "Reading the body of {SourceId} in category {CategoryId} failed in a way no retry fixes, so the batch fails", sourceId, category.Id); + throw; + } + catch (Exception ex) + { + lastAttemptError = ex; + logger.LogWarning(ex, "Reading the body of {SourceId} failed on attempt {Attempt} of {MaxAttempts}", sourceId, attempt, MigrationEngine.MaxBodyReadAttempts); + + if (attempt < MigrationEngine.MaxBodyReadAttempts && retryBackoff > TimeSpan.Zero) + { + await Task.Delay(retryBackoff, timeProvider, cancellationToken); + } + } + } + + return (null, lastAttemptError); + } + + // These fail the same way on every attempt, so retrying would only turn a code defect into skipped messages. + static bool IsDefect(Exception exception) => + exception is NotSupportedException or NotImplementedException or InvalidOperationException or ArgumentException or NullReferenceException or InvalidCastException; +} diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs index 0a6ae5bb2c..8462c1e454 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs @@ -33,7 +33,13 @@ reason is MigrationSkipReason.PastRetention or MigrationSkipReason.BlankGroupComment; /// - /// Whether no retry can fix a row skipped for this reason, because the row itself can never be stored. + /// Whether no retry can fix a row skipped for this reason. /// - public static bool IsPermanent(this MigrationSkipReason reason) => reason is MigrationSkipReason.RequiredValueMissing; + public static bool IsPermanent(this MigrationSkipReason reason) => + reason is MigrationSkipReason.RequiredValueMissing + or MigrationSkipReason.UnparseableUniqueMessageId + or MigrationSkipReason.NoProcessingAttempts + or MigrationSkipReason.ValueTooLong + or MigrationSkipReason.MalformedValue + or MigrationSkipReason.BodyMissing; } diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs b/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs index bf975e1e21..e13538cdf7 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs @@ -24,9 +24,13 @@ public sealed class MigrationEngine( MigrationEngineOptions options, ILogger logger) { - /// How many times a message body is read before the row is skipped as unreadable. + /// + /// How many times a message body is read before the row is skipped as unreadable. + /// public const int MaxBodyReadAttempts = 3; + readonly MigrationBodyLoader bodyLoader = new(source, timeProvider, options.BodyRetryBackoff, logger); + /// /// The categories of one kind to copy, in the order to copy them. Optional categories the operator did not /// ask for are left out. @@ -43,9 +47,6 @@ public IReadOnlyList SelectCategories(MigrationCategoryKind k /// that halts does not stop the ones after it. Before copying anything it saves a not-started checkpoint for /// every category that has none, so a copy stopped between two categories still lists the ones it never reached. /// - /// Another writer saved one of these checkpoints, which means a second instance is copying into the same database. - /// The host is shutting down. - /// Saving the checkpoint a category stopped or settled on failed. Its stored row is the last one saved, and the categories after it were not run. // Runs in the order given without re-sorting: required and optional orders both start at 1, so // sorting a mixed list would put an optional category in front of a required one. public async Task> RunCategories( @@ -75,10 +76,6 @@ public async Task> RunCategories( /// Copies one category, carrying on from its saved cursor, and returns the checkpoint it ended on. A category /// already finished or Failed is returned untouched without reading the source. /// - /// The stored checkpoint, whose state says how it ended and whose LastError says why it stopped. - /// Another writer saved this category's checkpoint, which means a second instance is copying into the same database. The state is left as that writer set it. - /// The host is shutting down. The last committed batch saved its own counts, so the stored checkpoint is already correct and a restart carries on from it. - /// Saving the checkpoint the category stopped or settled on failed, such as a halt the store could not write. The stored row is the last one saved, so it still reads as copying from its last committed batch. public async Task RunCategoryAsync(MigrationCategory category, CancellationToken cancellationToken = default) { var checkpoint = await checkpointStore.Read(category.Id, cancellationToken) ?? NotStarted(category); @@ -146,16 +143,13 @@ public async Task RunCategoryAsync(MigrationCategory catego // Kept out of checkpoint until the write commits: the catch below saves checkpoint, and a // restart re-reads the uncommitted batch and would count these skips twice. var bodySkips = 0; + Dictionary? bodySkipReasons = null; if (category.CarriesBodies) { - var (withBodies, failed) = await FetchBodiesWithRetry(category, batch, cancellationToken); - batchToWrite = withBodies; - bodySkips = failed.Count; - - foreach (var (id, lastAttemptError) in failed) - { - logger.LogWarning(lastAttemptError, "Skipped {SourceId} in category {CategoryId}: body unreadable after {MaxAttempts} attempts", id, category.Id, MaxBodyReadAttempts); - } + var loaded = await bodyLoader.Load(category, batch, cancellationToken); + batchToWrite = batch with { Rows = loaded.Rows }; + bodySkips = loaded.Skipped.Count; + bodySkipReasons = loaded.Skipped.CountBy(skip => skip.Reason).ToDictionary(count => count.Key, count => (long)count.Value); } // The target adds its own outcome inside the transaction that writes the rows, @@ -165,7 +159,7 @@ public async Task RunCategoryAsync(MigrationCategory catego Cursor = batch.Cursor, LastProgressAt = timeProvider.GetUtcNow().UtcDateTime, SkippedCount = checkpoint.SkippedCount + bodySkips, - SkipReasons = MigrationCheckpoint.AddSkipReasons(checkpoint.SkipReasons, bodySkips == 0 ? null : new Dictionary { [MigrationSkipReason.BodyUnreadable] = bodySkips }) + SkipReasons = MigrationCheckpoint.AddSkipReasons(checkpoint.SkipReasons, bodySkipReasons) }; var result = await target.Write(category, batchToWrite, checkpointToExtend, cancellationToken); @@ -253,64 +247,6 @@ static long FaultSkips(IReadOnlyDictionary? skipReaso Task Settle(MigrationCheckpoint settled, CancellationToken cancellationToken) => checkpointStore.Upsert(settled with { SettledAt = timeProvider.GetUtcNow().UtcDateTime }, cancellationToken); - async Task<(MigrationBatch Batch, IReadOnlyList<(string SourceId, Exception LastAttemptError)> Failed)> FetchBodiesWithRetry(MigrationCategory category, MigrationBatch batch, CancellationToken cancellationToken) - { - var survivors = new List(batch.Rows.Count); - var failed = new List<(string SourceId, Exception LastAttemptError)>(); - - foreach (var row in batch.Rows) - { - if (row.Body is not null) - { - survivors.Add(row); - continue; - } - - MigrationBody? body = null; - Exception? lastAttemptError = null; - var succeeded = false; - - for (var attempt = 1; attempt <= MaxBodyReadAttempts && !succeeded; attempt++) - { - try - { - body = await source.ReadBody(category, row.SourceId, cancellationToken); - succeeded = true; - } - catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) - { - // A shutdown is not a transient body failure. Retrying it to the attempt limit and then - // recording the message as permanently unreadable would lose a row to a restart. - throw; - } - catch (Exception ex) when (!IsDefect(ex)) - { - lastAttemptError = ex; - logger.LogWarning(ex, "Attempt {Attempt} to read the body for {SourceId} failed", attempt, row.SourceId); - if (attempt < MaxBodyReadAttempts) - { - await Pause(options.BodyRetryBackoff, cancellationToken); - } - } - } - - if (succeeded) - { - survivors.Add(row with { Body = body }); - } - else - { - failed.Add((row.SourceId, lastAttemptError!)); - } - } - - return (batch with { Rows = survivors }, failed); - } - - // These fail the same way on every attempt, so retrying would only turn a code defect into skipped messages. - static bool IsDefect(Exception exception) => - exception is NotSupportedException or NotImplementedException or InvalidOperationException or ArgumentException or NullReferenceException or InvalidCastException; - // Zero means no throttling, and it must return without waiting: against a fake clock // nobody advances, even a zero-length wait never finishes. Task Pause(TimeSpan duration, CancellationToken cancellationToken) => diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs b/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs new file mode 100644 index 0000000000..808d5090f6 --- /dev/null +++ b/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence.DataMigration; + +/// +/// Keys for the extra values a source attaches to a . The source works each value out +/// itself, so the target never needs to know how the source stores it. +/// +public static class MigrationMetadataKeys +{ + /// + /// The instant a failed message moved to its current status, as a UTC . The retention sweep + /// ages a row out from this instant. + /// + public const string StatusChangedAt = "StatusChangedAt"; + + /// + /// The time the source last changed the row's document, as a UTC . + /// + public const string LastModified = "LastModified"; +} diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs b/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs index 11d03ffa52..6c4d0fe806 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs @@ -7,13 +7,20 @@ namespace ServiceControl.Persistence.DataMigration; /// public enum MigrationSkipReason { - /// The message body could not be read from the source, after the engine had tried more than once. + /// + /// Every attempt to read the message body from the source failed, so the whole message was left out rather than + /// written without one. A retry reads it again once the cause is fixed. + /// BodyUnreadable, - /// The row is older than the retention period, so the instance would have deleted it soon anyway. + /// + /// The row is older than the retention period, so the instance would have deleted it soon anyway. + /// PastRetention, - /// The source row has no value for something the target column requires. + /// + /// The source row has no value for something the target column requires. + /// RequiredValueMissing, /// @@ -21,6 +28,32 @@ public enum MigrationSkipReason /// BlankGroupComment, + /// + /// The document's unique message id is not a GUID, and the row's key has to be one. + /// + UnparseableUniqueMessageId, + + /// + /// The document holds no processing attempt, and every column that describes the failure comes from one. + /// + NoProcessingAttempts, + + /// + /// A value is longer than its target column allows, so storing the row would fail the whole insert. + /// + ValueTooLong, + + /// + /// A value the target needs is present but holds the wrong type, so the row cannot be built from it. + /// + MalformedValue, + + /// + /// The source holds the message but no body for it, so the whole message was left out rather than written + /// without one. + /// + BodyMissing, + /// /// A reason a newer build wrote and this one cannot name, so an older host still starts instead of throwing. /// Nothing in this build writes it. It counts as a fault, because harmless would let this build settle a diff --git a/src/ServiceControl.UnitTests/Migration/AllowUnreleasedMigrationSwitchTests.cs b/src/ServiceControl.UnitTests/Migration/AllowUnreleasedMigrationSwitchTests.cs new file mode 100644 index 0000000000..464ba86b26 --- /dev/null +++ b/src/ServiceControl.UnitTests/Migration/AllowUnreleasedMigrationSwitchTests.cs @@ -0,0 +1,37 @@ +#if DEBUG +namespace ServiceControl.UnitTests.Migration; + +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Hosting.Commands; +using ServiceControl.Migration.Checks; + +[TestFixture] +class AllowUnreleasedMigrationSwitchTests +{ + [TestCase("true")] + [TestCase("TRUE")] + public void The_switch_registers_the_marker_when_the_variable_is_true(string value) + { + var services = new ServiceCollection(); + + RunCommand.AllowUnreleasedMigrationWhenAsked(services, value); + + using var provider = services.BuildServiceProvider(); + Assert.That(provider.GetService(), Is.Not.Null); + } + + [TestCase(null)] + [TestCase("")] + [TestCase("false")] + public void The_switch_registers_nothing_unless_the_variable_is_true(string value) + { + var services = new ServiceCollection(); + + RunCommand.AllowUnreleasedMigrationWhenAsked(services, value); + + using var provider = services.BuildServiceProvider(); + Assert.That(provider.GetService(), Is.Null); + } +} +#endif diff --git a/src/ServiceControl.UnitTests/Migration/CheckpointMigrationStateTests.cs b/src/ServiceControl.UnitTests/Migration/CheckpointMigrationStateTests.cs index 7a71eb1c2d..9947ca3ca3 100644 --- a/src/ServiceControl.UnitTests/Migration/CheckpointMigrationStateTests.cs +++ b/src/ServiceControl.UnitTests/Migration/CheckpointMigrationStateTests.cs @@ -104,7 +104,7 @@ public void Unknown_stays_the_last_skip_reason() => public void Exactly_the_reasons_no_retry_can_fix_are_permanent() => Assert.That( Enum.GetValues().Where(reason => reason.IsPermanent()), - Is.EquivalentTo(new[] { MigrationSkipReason.RequiredValueMissing })); + Is.EquivalentTo(new[] { MigrationSkipReason.RequiredValueMissing, MigrationSkipReason.UnparseableUniqueMessageId, MigrationSkipReason.NoProcessingAttempts, MigrationSkipReason.ValueTooLong, MigrationSkipReason.MalformedValue, MigrationSkipReason.BodyMissing })); static MigrationCheckpoint Checkpoint(string categoryId, MigrationCategoryState state) => new(categoryId, state, null, 0, 0, null, null, null, null, null, null); diff --git a/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationSource.cs b/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationSource.cs index ebc2dd8253..0d3bbfceb6 100644 --- a/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationSource.cs +++ b/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationSource.cs @@ -16,6 +16,10 @@ public sealed class InMemoryMigrationSource : IMigrationSource readonly Dictionary bodyFailuresInTurn = []; readonly Dictionary bodyReadAttempts = []; readonly Dictionary bodies = []; + // The loader reads several bodies at once, so the attempt counts and the in-flight count are shared between threads. + readonly Lock bodyReadGate = new(); + TaskCompletionSource? heldBodyReads; + int bodyReadsInFlight; public MigrationSourceDescription Description { get; set; } = new("in-memory", [new MigrationSourceFact("Store", "in memory")]); @@ -35,7 +39,39 @@ public sealed class InMemoryMigrationSource : IMigrationSource /// public (string SourceId, CancellationTokenSource Source)? StopOnBodyRead { get; set; } - public int BodyReadAttempts(string sourceId) => bodyReadAttempts.GetValueOrDefault(sourceId); + public int BodyReadAttempts(string sourceId) + { + lock (bodyReadGate) + { + return bodyReadAttempts.GetValueOrDefault(sourceId); + } + } + + /// + /// The most body reads that were open at the same moment since the source was created. + /// + public int PeakBodyReadsInFlight { get; private set; } + + /// + /// Makes every body read wait until is called. A waiting read already counts as an attempt and as open. + /// + public void HoldBodyReads() => heldBodyReads = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + + /// + /// Lets every held body read carry on, and every later one run without waiting. Call first. + /// + public void ReleaseBodyReads() => heldBodyReads!.SetResult(); + + /// + /// Completes once at least this many body reads are open at the same moment. + /// + public async Task BodyReadsInFlight(int count, CancellationToken cancellationToken = default) + { + while (Volatile.Read(ref bodyReadsInFlight) < count) + { + await Task.Delay(TimeSpan.FromMilliseconds(5), cancellationToken); + } + } public Task Open(CancellationToken cancellationToken = default) => Task.CompletedTask; @@ -82,25 +118,43 @@ public async IAsyncEnumerable Read( { await Task.Yield(); - var attempt = bodyReadAttempts[sourceId] = BodyReadAttempts(sourceId) + 1; + int attempt; - if (StopOnBodyRead is { } stop && stop.SourceId == sourceId) + lock (bodyReadGate) { - await stop.Source.CancelAsync(); - throw new OperationCanceledException(stop.Source.Token); + attempt = bodyReadAttempts[sourceId] = bodyReadAttempts.GetValueOrDefault(sourceId) + 1; + PeakBodyReadsInFlight = Math.Max(PeakBodyReadsInFlight, Interlocked.Increment(ref bodyReadsInFlight)); } - if (bodyFailures.TryGetValue(sourceId, out var failures) && attempt <= failures.Times) + try { - throw failures.Failure; - } + if (heldBodyReads is { } held) + { + await held.Task; + } + + if (StopOnBodyRead is { } stop && stop.SourceId == sourceId) + { + await stop.Source.CancelAsync(); + throw new OperationCanceledException(stop.Source.Token); + } - if (bodyFailuresInTurn.TryGetValue(sourceId, out var failuresInTurn) && attempt <= failuresInTurn.Length) + if (bodyFailures.TryGetValue(sourceId, out var failures) && attempt <= failures.Times) + { + throw failures.Failure; + } + + if (bodyFailuresInTurn.TryGetValue(sourceId, out var failuresInTurn) && attempt <= failuresInTurn.Length) + { + throw failuresInTurn[attempt - 1]; + } + + return bodies.GetValueOrDefault(sourceId); + } + finally { - throw failuresInTurn[attempt - 1]; + Interlocked.Decrement(ref bodyReadsInFlight); } - - return bodies.GetValueOrDefault(sourceId); } // Every registered category, not just the seeded ones: Seed writes rowsByCategory after construction, diff --git a/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationTarget.cs b/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationTarget.cs index 84c171e912..b6ef99b0a1 100644 --- a/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationTarget.cs +++ b/src/ServiceControl.UnitTests/Migration/Fakes/InMemoryMigrationTarget.cs @@ -14,6 +14,7 @@ public sealed class InMemoryMigrationTarget(IMigrationCheckpointStore checkpoint readonly Dictionary> rowsHandedToWriteByCategory = []; readonly HashSet preExistingKeys = []; readonly Dictionary rejectedKeys = []; + readonly TaskCompletionSource ignoredCallReleased = new(TaskCreationOptions.RunContinuationsAsynchronously); public int DefaultBatchSize { get; set; } = 3; public string NoBatchSizeFor { get; set; } @@ -46,8 +47,22 @@ public sealed class InMemoryMigrationTarget(IMigrationCheckpointStore checkpoint /// Waits on this call until the token is cancelled and then throws, as a write stuck on a database that stopped answering does. /// public int? HangOnCall { get; set; } + /// + /// Waits on this call until ReleaseIgnoredCall and never looks at the token, then writes without it, as a driver call stuck below the token does. + /// If the store then refuses the checkpoint save, this fake has already recorded the batch's rows, unlike the real target, which rolls both back. + /// + public int? IgnoreTokenOnCall { get; set; } + /// + /// The write that IgnoreTokenOnCall holds, or null until that call is made. It ends only after ReleaseIgnoredCall. + /// + public Task IgnoredCall { get; private set; } int callCount; + /// + /// Lets the write that IgnoreTokenOnCall holds carry on. Calling it again does nothing. + /// + public void ReleaseIgnoredCall() => ignoredCallReleased.TrySetResult(); + public void SeedExistingKey(string sourceId) => preExistingKeys.Add(sourceId); public void RejectKey(string sourceId, MigrationSkipReason reason) => rejectedKeys[sourceId] = reason; @@ -66,32 +81,51 @@ public IReadOnlyList RowsHandedToWrite(string categoryId) => public Task BatchSizeFor(MigrationCategory category, CancellationToken cancellationToken = default) => category.Id == NoBatchSizeFor ? throw new InvalidOperationException($"No batch size is mapped for category {category.Id}") : Task.FromResult(DefaultBatchSize); - public async Task Write( + public Task Write( MigrationCategory category, MigrationBatch batch, MigrationCheckpoint checkpointToExtend, CancellationToken cancellationToken = default) { - callCount++; + var callNumber = ++callCount; BeforeWrite?.Invoke(checkpointToExtend); - if (HangOnCall == callCount) + return IgnoreTokenOnCall == callNumber + ? IgnoredCall = WriteOnceReleased(category, batch, checkpointToExtend, callNumber) + : WriteNow(category, batch, checkpointToExtend, callNumber, cancellationToken); + } + + // The commit ignores the token too, so a late commit meets the store's version check instead of the token. + async Task WriteOnceReleased(MigrationCategory category, MigrationBatch batch, MigrationCheckpoint checkpointToExtend, int callNumber) + { + await ignoredCallReleased.Task; + return await WriteNow(category, batch, checkpointToExtend, callNumber, CancellationToken.None); + } + + async Task WriteNow( + MigrationCategory category, + MigrationBatch batch, + MigrationCheckpoint checkpointToExtend, + int callNumber, + CancellationToken cancellationToken) + { + if (HangOnCall == callNumber) { await Task.Delay(Timeout.Infinite, cancellationToken); } - if (StopOnCall is { } stop && stop.CallNumber == callCount) + if (StopOnCall is { } stop && stop.CallNumber == callNumber) { await stop.Source.CancelAsync(); throw new OperationCanceledException(stop.Source.Token); } - if (FailOnCallNumber == callCount) + if (FailOnCallNumber == callNumber) { - throw FailWith ?? new InvalidOperationException($"Simulated failure on write {callCount}"); + throw FailWith ?? new InvalidOperationException($"Simulated failure on write {callNumber}"); } - if (CancelOnCall is { } cancel && cancel.CallNumber == callCount) + if (CancelOnCall is { } cancel && cancel.CallNumber == callNumber) { await cancel.Source.CancelAsync(); } diff --git a/src/ServiceControl.UnitTests/Migration/MigrationBodyLoaderTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationBodyLoaderTests.cs new file mode 100644 index 0000000000..201bb885c8 --- /dev/null +++ b/src/ServiceControl.UnitTests/Migration/MigrationBodyLoaderTests.cs @@ -0,0 +1,212 @@ +#nullable enable +namespace ServiceControl.UnitTests.Migration; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Time.Testing; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.UnitTests.Migration.Fakes; + +[TestFixture] +class MigrationBodyLoaderTests +{ + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)!; + static readonly MigrationBody Body = new(Encoding.UTF8.GetBytes("1"), "text/xml"); + + [Test] + public async Task Attaches_the_body_to_every_row() + { + var source = SourceWithBodies("FailedMessages/one", "FailedMessages/two"); + + var result = await Load(source, "FailedMessages/one", "FailedMessages/two"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Rows.Select(row => row.SourceId), Is.EqualTo(new[] { "FailedMessages/one", "FailedMessages/two" })); + Assert.That(result.Rows.Select(row => row.Body), Is.All.SameAs(Body)); + Assert.That(result.Skipped, Is.Empty); + } + } + + [Test] + public async Task Retries_a_failing_read_three_times_then_skips_the_message_as_BodyUnreadable() + { + var source = SourceWithBodies("FailedMessages/one", "FailedMessages/two"); + source.FailBodyReads("FailedMessages/two", MigrationEngine.MaxBodyReadAttempts, new TimeoutException("body store unreachable")); + + var result = await Load(source, "FailedMessages/one", "FailedMessages/two"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(source.BodyReadAttempts("FailedMessages/two"), Is.EqualTo(MigrationEngine.MaxBodyReadAttempts)); + Assert.That(result.Rows.Select(row => row.SourceId), Is.EqualTo(new[] { "FailedMessages/one" }), "a message with no body is a retry hazard, so it is left out rather than written without one"); + Assert.That(result.Skipped, Is.EqualTo(new[] { ("FailedMessages/two", MigrationSkipReason.BodyUnreadable) })); + } + } + + [Test] + public async Task Keeps_a_message_whose_read_succeeds_on_the_last_attempt() + { + var source = SourceWithBodies("FailedMessages/one"); + source.FailBodyReads("FailedMessages/one", MigrationEngine.MaxBodyReadAttempts - 1, new TimeoutException("blip")); + + var result = await Load(source, "FailedMessages/one"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Rows.Single().Body, Is.SameAs(Body)); + Assert.That(result.Skipped, Is.Empty); + } + } + + [Test] + public async Task Skips_a_message_the_source_has_no_body_for_as_BodyMissing_after_one_read() + { + var source = new InMemoryMigrationSource(); + + var result = await Load(source, "FailedMessages/one"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(source.BodyReadAttempts("FailedMessages/one"), Is.EqualTo(1), "a read that found nothing learns nothing by repeating"); + Assert.That(result.Rows, Is.Empty); + Assert.That(result.Skipped, Is.EqualTo(new[] { ("FailedMessages/one", MigrationSkipReason.BodyMissing) }), + "a missing body is not an unreadable one, and only a separate reason lets the refusal say no retry fixes it"); + } + } + + [Test] + public void Lets_a_defect_out_on_the_first_attempt_rather_than_skipping_the_message() + { + var source = SourceWithBodies("FailedMessages/one"); + source.FailBodyReads("FailedMessages/one", MigrationEngine.MaxBodyReadAttempts, new NotSupportedException("this source cannot read bodies")); + + Assert.ThrowsAsync(() => Load(source, "FailedMessages/one"), + "a defect fails the same way on every attempt, so retrying it and skipping the message would blame the body store for a bug"); + Assert.That(source.BodyReadAttempts("FailedMessages/one"), Is.EqualTo(1)); + } + + [Test] + public void Logs_a_defect_with_the_message_it_was_reading_before_letting_it_out() + { + var source = SourceWithBodies("FailedMessages/one"); + var defect = new NotSupportedException("this source cannot read bodies"); + source.FailBodyReads("FailedMessages/one", 1, defect); + var logger = new CapturingLogger(); + var batch = new MigrationBatch([new MigrationRow("FailedMessages/one", new object(), new Dictionary())], "FailedMessages/one"); + + Assert.ThrowsAsync(() => new MigrationBodyLoader(source, new FakeTimeProvider(), TimeSpan.Zero, logger).Load(Category, batch, TestContext.CurrentContext.CancellationToken)); + + var error = logger.Entries.Single(entry => entry.Level == LogLevel.Error); + + using (Assert.EnterMultipleScope()) + { + Assert.That(error.Message, Does.Contain("FailedMessages/one").And.Contain(Category.Id), "the exception that halts the category does not say which message it was reading, so this line is the only record of it"); + Assert.That(error.Exception, Is.SameAs(defect)); + } + } + + [Test] + public async Task Retries_a_cancellation_the_run_did_not_ask_for_and_then_skips_the_message_as_BodyUnreadable() + { + var source = SourceWithBodies("FailedMessages/one"); + source.FailBodyReads("FailedMessages/one", MigrationEngine.MaxBodyReadAttempts, new OperationCanceledException("the client timed out")); + + var result = await Load(source, "FailedMessages/one"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(source.BodyReadAttempts("FailedMessages/one"), Is.EqualTo(MigrationEngine.MaxBodyReadAttempts), "a client timeout raised as a cancellation is a transient failure, not a shutdown"); + Assert.That(result.Skipped, Is.EqualTo(new[] { ("FailedMessages/one", MigrationSkipReason.BodyUnreadable) })); + } + } + + [Test] + public async Task Keeps_a_body_the_source_attached_and_never_reads_it_again() + { + var source = SourceWithBodies("FailedMessages/two"); + var attached = new MigrationBody(Encoding.UTF8.GetBytes("attached in Read"), "text/plain"); + + var batch = new MigrationBatch( + [ + new MigrationRow("FailedMessages/one", new object(), new Dictionary(), attached), + new MigrationRow("FailedMessages/two", new object(), new Dictionary()) + ], + "FailedMessages/two"); + + var result = await Loader(source).Load(Category, batch, TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(source.BodyReadAttempts("FailedMessages/one"), Is.Zero, "a source that attaches bodies in Read would otherwise pay a second read per message for data it already had"); + Assert.That(result.Rows[0].Body, Is.SameAs(attached)); + Assert.That(result.Rows[1].Body, Is.SameAs(Body)); + } + } + + [Test] + public async Task Fetches_at_most_eight_bodies_at_a_time_and_keeps_the_batch_order() + { + var sourceIds = Enumerable.Range(0, 40).Select(index => $"FailedMessages/{index}").ToArray(); + var source = SourceWithBodies(sourceIds); + source.HoldBodyReads(); + + var load = Load(source, sourceIds); + + try + { + using var waitLimit = new CancellationTokenSource(TimeSpan.FromSeconds(5)); + await source.BodyReadsInFlight(MigrationBodyLoader.MaxConcurrentFetches, waitLimit.Token); + // Gives an unbounded loader time to open the rest of the forty reads, which it would do all at once. + await Task.Delay(TimeSpan.FromMilliseconds(100)); + + Assert.That(source.PeakBodyReadsInFlight, Is.EqualTo(MigrationBodyLoader.MaxConcurrentFetches), + "an unbounded loader opens all forty reads at once, which turns the copier into a load spike on a live source"); + } + finally + { + source.ReleaseBodyReads(); + } + + var result = await load.WaitAsync(TimeSpan.FromSeconds(5)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Rows.Select(row => row.SourceId), Is.EqualTo(sourceIds), "the batch's order is the cursor's order, so it has to survive being fetched out of order"); + Assert.That(result.Skipped, Is.Empty); + Assert.That(source.PeakBodyReadsInFlight, Is.EqualTo(MigrationBodyLoader.MaxConcurrentFetches)); + } + } + + static InMemoryMigrationSource SourceWithBodies(params string[] sourceIds) + { + var source = new InMemoryMigrationSource(); + + foreach (var sourceId in sourceIds) + { + source.SetBody(sourceId, Body); + } + + return source; + } + + // Zero backoff because the FakeTimeProvider is never advanced, and a wait on it would never finish. + static MigrationBodyLoader Loader(InMemoryMigrationSource source) => + new(source, new FakeTimeProvider(), TimeSpan.Zero, NullLogger.Instance); + + static Task Load(InMemoryMigrationSource source, params string[] sourceIds) + { + var batch = new MigrationBatch( + [.. sourceIds.Select(sourceId => new MigrationRow(sourceId, new object(), new Dictionary()))], + sourceIds[^1]); + + return Loader(source).Load(Category, batch, TestContext.CurrentContext.CancellationToken); + } +} diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineBodyRetryTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineBodyRetryTests.cs index dbc94ad95a..103d4a2043 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineBodyRetryTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineBodyRetryTests.cs @@ -167,7 +167,13 @@ public async Task The_skip_warning_for_an_unreadable_body_carries_the_last_attem await engine.RunCategoryAsync(category); - Assert.That(logger.Entries.Single(e => e.Message.StartsWith("Skipped msg-1")).Exception, Is.SameAs(failures[^1])); + var skipped = logger.Entries.Single(e => e.Message.StartsWith("Skipped msg-1")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(skipped.Exception, Is.SameAs(failures[^1])); + Assert.That(skipped.Message, Does.Contain(nameof(MigrationSkipReason.BodyUnreadable)), "the refusal counts skips by reason name, so the log line has to carry the same name for the operator to find them"); + } } [Test] @@ -235,6 +241,13 @@ public async Task A_body_store_outage_across_many_messages_halts_the_category() var checkpoint = await engine.RunCategoryAsync(category); - Assert.That(checkpoint.State, Is.EqualTo(MigrationCategoryState.Halted)); + using (Assert.EnterMultipleScope()) + { + Assert.That(checkpoint.State, Is.EqualTo(MigrationCategoryState.Halted)); + Assert.That(checkpoint.LastError, Does.Contain("exceeds the configured threshold"), + "the end-of-run accounting check halts a category whose body skips were never counted, so the state alone cannot tell the threshold fired"); + Assert.That(checkpoint.SkipReasons, Is.EquivalentTo(new Dictionary { [MigrationSkipReason.BodyUnreadable] = 200 }), + "the first batch's 100 skips only reach the 100-row floor, so the second batch's are what pass it"); + } } } diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineCollisionTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineCollisionTests.cs index 59b53c6a07..01dd62c96f 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineCollisionTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineCollisionTests.cs @@ -14,7 +14,8 @@ namespace ServiceControl.UnitTests.Migration; [TestFixture] class MigrationEngineCollisionTests { - static MigrationRow Row(string id, string name) => new(id, new { Name = name }, new Dictionary()); + // A body already attached, so the engine reads none and every row reaches the target. + static MigrationRow Row(string id, string name) => new(id, new { Name = name }, new Dictionary(), new MigrationBody(new byte[] { 1 }, "application/octet-stream")); [Test] public async Task Rows_already_present_in_the_target_are_counted_as_already_present_across_batches_not_skipped_or_overwritten() diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineHaltTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineHaltTests.cs index 0ece00a601..9e3a292a46 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineHaltTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineHaltTests.cs @@ -14,7 +14,8 @@ namespace ServiceControl.UnitTests.Migration; [TestFixture] class MigrationEngineHaltTests { - static MigrationRow Row(string id) => new(id, new object(), new Dictionary()); + // A body already attached, so the engine reads none and a row is lost only where a test rejects it. + static MigrationRow Row(string id) => new(id, new object(), new Dictionary(), new MigrationBody(new byte[] { 1 }, "application/octet-stream")); [Test] public async Task A_systemic_failure_halts_the_category_partway_through() diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineSkipReasonTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineSkipReasonTests.cs index 374a06de68..4053d7d71b 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineSkipReasonTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineSkipReasonTests.cs @@ -47,7 +47,7 @@ public async Task A_message_whose_body_is_never_read_is_counted_under_BodyUnread { var category = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)!; var source = new InMemoryMigrationSource(); - source.Seed(category.Id, Row("msg-1"), Row("msg-2")); + source.Seed(category.Id, Row("msg-1"), Row("msg-2") with { Body = new MigrationBody(new byte[] { 1 }, "application/octet-stream") }); source.FailBodyReads("msg-1", MigrationEngine.MaxBodyReadAttempts, new TimeoutException("down")); var checkpointStore = new InMemoryMigrationCheckpointStore(); var target = new InMemoryMigrationTarget(checkpointStore); @@ -59,6 +59,32 @@ public async Task A_message_whose_body_is_never_read_is_counted_under_BodyUnread Assert.That(checkpoint.SkipReasons, Is.EquivalentTo(new Dictionary { [MigrationSkipReason.BodyUnreadable] = 1 })); } + [Test] + public async Task A_message_the_source_holds_no_body_for_is_counted_under_BodyMissing() + { + var category = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)!; + var source = new InMemoryMigrationSource(); + source.Seed(category.Id, Row("msg-1"), Row("msg-2")); + source.SetBody("msg-2", new MigrationBody(new byte[] { 2 }, "text/plain")); + var checkpointStore = new InMemoryMigrationCheckpointStore(); + var target = new InMemoryMigrationTarget(checkpointStore); + var options = new MigrationEngineOptions(TimeSpan.Zero, 5, 100, []) { BodyRetryBackoff = TimeSpan.Zero }; + var logger = new CapturingLogger(); + var engine = new MigrationEngine(source, target, checkpointStore, new FakeTimeProvider(), options, logger); + + var checkpoint = await engine.RunCategoryAsync(category); + + using (Assert.EnterMultipleScope()) + { + Assert.That(checkpoint.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors), "msg-1 is the only skip, so a missing body has to count as a fault on its own"); + Assert.That(checkpoint.SkippedCount, Is.EqualTo(1)); + Assert.That(checkpoint.SkipReasons, Is.EquivalentTo(new Dictionary { [MigrationSkipReason.BodyMissing] = 1 }), + "the refusal tells the operator which skips no retry can fix only from these reasons"); + Assert.That(logger.Entries.Where(entry => entry.Message.StartsWith("Skipped msg-1")).Select(entry => entry.Message), Has.Exactly(1).Contains("BodyMissing"), + "the checkpoint holds counts, not ids, so the log is the operator's only record of which message was left out"); + } + } + [Test] public async Task A_batch_that_loses_rows_two_different_ways_records_both_reasons() { @@ -66,7 +92,7 @@ public async Task A_batch_that_loses_rows_two_different_ways_records_both_reason // summing would leave the total right and the reasons wrong, and verification would still balance. var category = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)!; var source = new InMemoryMigrationSource(); - source.Seed(category.Id, Row("msg-1"), Row("msg-2"), Row("msg-3")); + source.Seed(category.Id, Row("msg-1"), Row("msg-2") with { Body = new MigrationBody(new byte[] { 1 }, "application/octet-stream") }, Row("msg-3") with { Body = new MigrationBody(new byte[] { 1 }, "application/octet-stream") }); // One the engine skips itself because the body will not read, one the target refuses. source.FailBodyReads("msg-1", MigrationEngine.MaxBodyReadAttempts, new TimeoutException("body store unreachable")); var checkpointStore = new InMemoryMigrationCheckpointStore(); diff --git a/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs b/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs index 2077493bff..91945ae6e9 100644 --- a/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs +++ b/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs @@ -125,11 +125,12 @@ public void Every_failed_required_category_is_named_with_its_skips_and_both_comm var withErrors = Checkpoint(MigrationCategoryState.CompleteWithErrors, MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages) with { CopiedCount = 40, - SkippedCount = 5, + SkippedCount = 6, SkipReasons = new Dictionary { [MigrationSkipReason.RequiredValueMissing] = 2, - [MigrationSkipReason.BodyUnreadable] = 3 + [MigrationSkipReason.BodyUnreadable] = 3, + [MigrationSkipReason.BodyMissing] = 1 } }; var halted = Checkpoint(MigrationCategoryState.Halted, MigrationCategoryIds.MessageRedirects) with { LastError = "TimeoutException at cursor r-7: the target did not answer" }; @@ -146,9 +147,10 @@ public void Every_failed_required_category_is_named_with_its_skips_and_both_comm } Assert.That(exception.Message, Does.Contain($"{MigrationCategoryIds.MessageRedirects} is Failed (Halted)").And.Contain("the target did not answer")); - Assert.That(exception.Message, Does.Contain($"{MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages} is Failed (CompleteWithErrors) after copying 40 and skipping 5")); + Assert.That(exception.Message, Does.Contain($"{MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages} is Failed (CompleteWithErrors) after copying 40 and skipping 6")); Assert.That(exception.Message, Does.Contain("RequiredValueMissing 2, no retry can fix"), "a row no retry can store is a reason to abandon, and the operator has to be told which"); Assert.That(exception.Message, Does.Contain("BodyUnreadable 3").And.Not.Contain("BodyUnreadable 3, no retry can fix"), "an unreadable body may read on the next try"); + Assert.That(exception.Message, Does.Contain("BodyMissing 1, no retry can fix"), "a body the source does not hold is missing again on every retry"); } } @@ -234,6 +236,49 @@ public void A_failed_category_whose_faults_no_retry_can_fix_is_pointed_at_abando }); } + [Test] + public void A_halted_category_whose_faults_no_retry_can_fix_is_told_abandon_gives_up_every_row_not_yet_read() + { + var logger = new CapturingLogger(); + + MigrationStartup.ReportWhatTheCopyLeftBehind( + [Settled(MigrationCategoryState.Halted, copied: 7, skipped: 120, new Dictionary { [MigrationSkipReason.MalformedValue] = 120 }) + with { LastError = "Halted: 120 of 127 rows skipped in this run exceeds the configured threshold of 5% and 100 rows." }], + logger, + RunStartedAt); + + var entry = logger.Entries.Single(); + + Assert.Multiple(() => + { + Assert.That(entry.Message, Does.Contain("stopped before the end").And.Contain("every row not yet read"), + "a halted category never read the rows after its cursor, so abandon gives up far more than the skipped ones"); + Assert.That(entry.Message, Does.Contain(MigrationSettings.HaltThresholdPercentKey).And.Contain(MigrationSettings.HaltThresholdMinimumKey) + .And.Contain($"--migration-retry {MigrationCategoryIds.EndpointSettings}"), "raising the threshold and retrying copies every row the faults do not stop"); + Assert.That(entry.Message, Does.Contain("No retry can fix the MalformedValue ones")); + Assert.That(entry.Message, Does.Not.Contain("gives up the rest")); + }); + } + + [Test] + public void A_halted_category_with_faults_a_retry_can_fix_is_also_told_what_abandon_gives_up() + { + var logger = new CapturingLogger(); + + MigrationStartup.ReportWhatTheCopyLeftBehind( + [Settled(MigrationCategoryState.Halted, copied: 7, skipped: 120, new Dictionary { [MigrationSkipReason.BodyUnreadable] = 120 })], + logger, + RunStartedAt); + + var entry = logger.Entries.Single(); + + Assert.Multiple(() => + { + Assert.That(entry.Message, Does.Contain("every row not yet read").And.Contain($"--migration-retry {MigrationCategoryIds.EndpointSettings}")); + Assert.That(entry.Message, Does.Not.Contain("No retry can fix"), "an unreadable body may read on the next try"); + }); + } + [Test] public void A_category_that_failed_with_only_harmless_skips_is_still_offered_a_retry() { @@ -359,7 +404,7 @@ public void A_category_an_earlier_run_left_failed_is_not_reported_as_finished() { var logger = new CapturingLogger(); - var failedEarlier = Settled(MigrationCategoryState.Halted, copied: 7, skipped: 2, new Dictionary { [MigrationSkipReason.RequiredValueMissing] = 2 }) + var failedEarlier = Settled(MigrationCategoryState.CompleteWithErrors, copied: 7, skipped: 2, new Dictionary { [MigrationSkipReason.RequiredValueMissing] = 2 }) with { SettledAt = RunStartedAt.AddMinutes(-5) }; diff --git a/src/ServiceControl.UnitTests/Migration/StoppedCopyExplanationTests.cs b/src/ServiceControl.UnitTests/Migration/StoppedCopyExplanationTests.cs index 2f3b033642..4f5ff54295 100644 --- a/src/ServiceControl.UnitTests/Migration/StoppedCopyExplanationTests.cs +++ b/src/ServiceControl.UnitTests/Migration/StoppedCopyExplanationTests.cs @@ -55,6 +55,127 @@ public async Task A_stall_settles_the_stalled_category_failed_and_the_next_categ } } + [Test] + public async Task A_stall_in_a_call_that_ignores_its_token_still_settles_the_category_failed_without_waiting_for_the_call() + { + var clock = new TimerRecordingTimeProvider(); + var store = new CancellationHonouringCheckpointStore(); + var writing = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var target = new InMemoryMigrationTarget(store) { IgnoreTokenOnCall = 1, BeforeWrite = _ => writing.TrySetResult() }; + var source = new InMemoryMigrationSource(); + source.Seed(MigrationCategoryIds.KnownEndpoints, Row("KnownEndpoints-1")); + source.Seed(MigrationCategoryIds.MessageRedirects, Row("MessageRedirects-1")); + + try + { + var run = Task.Run(() => MigrationStartup.RunRequiredCategories( + Engine(source, target, store, clock), Categories(MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.MessageRedirects), store, clock, new CapturingLogger())); + + Assert.That(await clock.TimerCreated.WaitAsync(TimeSpan.FromSeconds(10)), Is.True, "the watchdog never started its timer, so advancing the clock would tick nothing"); + await writing.Task.WaitAsync(TimeSpan.FromSeconds(10)); + clock.Advance(StallLimit + PollInterval); + var results = await run.WaitAsync(TimeSpan.FromSeconds(10)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(results.Select(checkpoint => checkpoint.CategoryId), Is.EqualTo(new[] { MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.MessageRedirects })); + Assert.That(results[0].State, Is.EqualTo(MigrationCategoryState.Halted)); + Assert.That(results[0].SettledAt, Is.EqualTo(clock.GetUtcNow().UtcDateTime)); + Assert.That(results[0].LastError, Does.Contain($"committed nothing for {StallLimit.TotalMinutes:0.#} minutes")); + Assert.That(results[1].State, Is.EqualTo(MigrationCategoryState.Complete), "one stuck call ended the whole start"); + } + + target.ReleaseIgnoredCall(); + + Assert.That(async () => await target.IgnoredCall, Throws.InstanceOf(), + "the stuck call saves from the version it read before the stall, so the store must refuse it"); + Assert.That(await store.Read(MigrationCategoryIds.KnownEndpoints), Is.EqualTo(results[0]), "a late commit from the stuck call reopened a settled category"); + } + finally + { + target.ReleaseIgnoredCall(); + } + } + + [Test] + public async Task A_stalled_call_that_commits_while_the_stall_is_being_settled_still_leaves_the_category_failed() + { + var clock = new TimerRecordingTimeProvider(); + var store = new CancellationHonouringCheckpointStore(); + var writing = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var target = new InMemoryMigrationTarget(store) { IgnoreTokenOnCall = 1, BeforeWrite = _ => writing.TrySetResult() }; + // The stuck call commits after the settle has read the row and before it saves, so the settle's save is from an old version. + store.BeforeUpsert = async checkpoint => + { + if (checkpoint.State == MigrationCategoryState.Halted) + { + target.ReleaseIgnoredCall(); + await target.IgnoredCall; + } + }; + var source = new InMemoryMigrationSource(); + source.Seed(MigrationCategoryIds.KnownEndpoints, Row("KnownEndpoints-1")); + source.Seed(MigrationCategoryIds.MessageRedirects, Row("MessageRedirects-1")); + + try + { + var run = Task.Run(() => MigrationStartup.RunRequiredCategories( + Engine(source, target, store, clock), Categories(MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.MessageRedirects), store, clock, new CapturingLogger())); + + Assert.That(await clock.TimerCreated.WaitAsync(TimeSpan.FromSeconds(10)), Is.True, "the watchdog never started its timer, so advancing the clock would tick nothing"); + await writing.Task.WaitAsync(TimeSpan.FromSeconds(10)); + clock.Advance(StallLimit + PollInterval); + var results = await run.WaitAsync(TimeSpan.FromSeconds(10)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(results[0].State, Is.EqualTo(MigrationCategoryState.Halted)); + Assert.That(results[0].CopiedCount, Is.EqualTo(1), "the settle must keep the batch the stuck call committed"); + Assert.That(results[0].LastError, Does.Contain($"committed nothing for {StallLimit.TotalMinutes:0.#} minutes")); + Assert.That(await store.Read(MigrationCategoryIds.KnownEndpoints), Is.EqualTo(results[0])); + Assert.That(results[1].State, Is.EqualTo(MigrationCategoryState.Complete), "a late commit from our own stuck call ended the whole start"); + } + } + finally + { + target.ReleaseIgnoredCall(); + } + } + + [Test] + public async Task A_host_stopping_during_a_call_that_ignores_its_token_propagates_without_waiting_for_the_call() + { + var store = new InMemoryMigrationCheckpointStore(); + using var host = new CancellationTokenSource(); + var writing = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var target = new InMemoryMigrationTarget(store) { IgnoreTokenOnCall = 1, BeforeWrite = _ => writing.TrySetResult() }; + var source = new InMemoryMigrationSource(); + source.Seed(MigrationCategoryIds.KnownEndpoints, Row("KnownEndpoints-1")); + source.Seed(MigrationCategoryIds.MessageRedirects, Row("MessageRedirects-1")); + var clock = new FakeTimeProvider(); + + try + { + var run = Task.Run(() => MigrationStartup.RunRequiredCategories( + Engine(source, target, store, clock), Categories(MigrationCategoryIds.KnownEndpoints, MigrationCategoryIds.MessageRedirects), store, clock, new CapturingLogger(), host.Token)); + + await writing.Task.WaitAsync(TimeSpan.FromSeconds(10)); + await host.CancelAsync(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(async () => await run.WaitAsync(TimeSpan.FromSeconds(10)), Throws.InstanceOf(), + "a shutdown waited for a call that ignores its token"); + Assert.That((await store.Read(MigrationCategoryIds.KnownEndpoints))!.State, Is.EqualTo(MigrationCategoryState.InProgress), "a shutdown settled the category"); + Assert.That((await store.Read(MigrationCategoryIds.MessageRedirects))!.State, Is.EqualTo(MigrationCategoryState.NotStarted), "a stopping host started the next category"); + } + } + finally + { + target.ReleaseIgnoredCall(); + } + } + [Test] public async Task A_host_stopping_mid_category_leaves_it_copying_and_the_stop_propagates() { @@ -173,6 +294,11 @@ sealed class CancellationHonouringCheckpointStore : IMigrationCheckpointStore { readonly InMemoryMigrationCheckpointStore inner = new(); + /// + /// Runs before each save that gets past the token check, with the checkpoint about to be saved. + /// + public Func? BeforeUpsert { get; set; } + public Task> ReadAll(CancellationToken cancellationToken = default) { cancellationToken.ThrowIfCancellationRequested(); @@ -185,10 +311,16 @@ public Task> ReadAll(CancellationToken cancel return inner.Read(categoryId, cancellationToken); } - public Task Upsert(MigrationCheckpoint checkpoint, CancellationToken cancellationToken = default) + public async Task Upsert(MigrationCheckpoint checkpoint, CancellationToken cancellationToken = default) { cancellationToken.ThrowIfCancellationRequested(); - return inner.Upsert(checkpoint, cancellationToken); + + if (BeforeUpsert is not null) + { + await BeforeUpsert(checkpoint); + } + + return await inner.Upsert(checkpoint, cancellationToken); } } } diff --git a/src/ServiceControl/Hosting/Commands/RunCommand.cs b/src/ServiceControl/Hosting/Commands/RunCommand.cs index 436431f6f8..45cf63cc1c 100644 --- a/src/ServiceControl/Hosting/Commands/RunCommand.cs +++ b/src/ServiceControl/Hosting/Commands/RunCommand.cs @@ -15,10 +15,32 @@ using ServiceControl.Hosting.Auth; using ServiceControl.Hosting.Https; using ServiceControl.Migration; +#if DEBUG + using Microsoft.Extensions.Logging; + using ServiceControl.Infrastructure; + using ServiceControl.Migration.Checks; +#endif using ServicePulse; class RunCommand : AbstractCommand { +#if DEBUG + internal const string AllowUnreleasedMigrationVariable = "SERVICECONTROL_MIGRATION_ALLOW_UNRELEASED"; + + /// + /// When is "true" in any letter case, registers + /// so a Debug build can copy past the release check for a developer's smoke test, and logs a warning so the bypass shows in the log. + /// + internal static void AllowUnreleasedMigrationWhenAsked(IServiceCollection services, string variableValue) + { + if (string.Equals(variableValue, "true", StringComparison.OrdinalIgnoreCase)) + { + services.AddSingleton(); + LoggerUtil.CreateStaticLogger().LogWarning("{Variable} is true, so this Debug build skips the check that the build carries the whole migration and will copy for real", AllowUnreleasedMigrationVariable); + } + } +#endif + public override Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) => Run(settings, customize: null, cancellationToken); @@ -44,6 +66,10 @@ internal static async Task Run(Settings settings, Action customize?.Invoke(hostBuilder); +#if DEBUG + AllowUnreleasedMigrationWhenAsked(hostBuilder.Services, Environment.GetEnvironmentVariable(AllowUnreleasedMigrationVariable)); +#endif + if (settings.MigrationEnabled) { hostBuilder.Services.AddHostedService(provider => new RequiredCopyBeforeTheHostOpens(provider, settings)); diff --git a/src/ServiceControl/Migration/Checks/MigrationIsReleasedCheck.cs b/src/ServiceControl/Migration/Checks/MigrationIsReleasedCheck.cs index 68ab286716..6156b0bfc5 100644 --- a/src/ServiceControl/Migration/Checks/MigrationIsReleasedCheck.cs +++ b/src/ServiceControl/Migration/Checks/MigrationIsReleasedCheck.cs @@ -6,7 +6,9 @@ namespace ServiceControl.Migration.Checks; using ServiceControl.Persistence.DataMigration; /// -/// Registered by a test host that needs a copy to run on a build that does not yet carry the whole migration. +/// Registered by a test host, or by a Debug build whose SERVICECONTROL_MIGRATION_ALLOW_UNRELEASED environment +/// variable is true, to let a copy run on a build that does not yet carry the whole migration. A Release build +/// has no way to register it. /// class AllowUnreleasedMigration; diff --git a/src/ServiceControl/Migration/MigrationStartup.cs b/src/ServiceControl/Migration/MigrationStartup.cs index dc17bab1d1..f906c8d5b9 100644 --- a/src/ServiceControl/Migration/MigrationStartup.cs +++ b/src/ServiceControl/Migration/MigrationStartup.cs @@ -32,9 +32,6 @@ internal static IReadOnlyCollection CopyableCategoryIds(IReadOnlyCollect /// the host. Call it after the host is built and before it starts, because the copy has to finish before /// anything else opens on the target. /// - /// The built host's services, which is where the target, the checkpoint store and the migration state come from. - /// The instance settings, read for the source and target persistence types. - /// Cancelled when the host is shutting down, which ends the copy without a refusal message. public static async Task RunRequiredCopy(IServiceProvider services, Settings settings, CancellationToken cancellationToken = default) { await MigrationStartupCheckRunner.Run( @@ -141,9 +138,6 @@ static async Task SeedMigrationState(IServiceProvider services, IReadOnlyCollect /// them. The order is what the operator sees: a check that costs nothing comes before one that connects to a /// database, and the source's own checks run last because they need it open. /// - /// The instance settings, whose retention periods are the optional category windows when none is set, and whose retry history depth the copy must not be thrown away by. - /// The options read from the settings, which the window check parsed on its way past. - /// A check refused. The message names the check and says what to do, and this start has copied nothing. public static async Task RunChecksAndOpen(IServiceProvider services, Settings settings, IMigrationSource source, CancellationToken cancellationToken = default) { var options = await RunChecksAndOpenTarget(services, settings, cancellationToken); @@ -160,12 +154,6 @@ public static async Task RunChecksAndOpen(IServiceProvid /// opening the target. A check that costs nothing comes before one that connects to a database. The target /// is open when this returns, and the source has not been touched. /// - /// The built host's services, which is where the target and its readiness checks come from. - /// The instance settings, whose retention periods are the optional category windows when none is set, and whose retry history depth the copy must not be thrown away by. - /// Cancelled when the host is shutting down, which stops the checks without a refusal message. - /// The options read from the settings, which the window check parsed on its way past. - /// A check refused or the target did not open. The message names the check and says what to do, and this start has copied nothing. - /// The host is shutting down. public static async Task RunChecksAndOpenTarget(IServiceProvider services, Settings settings, CancellationToken cancellationToken = default) { var target = services.GetRequiredService(); @@ -191,11 +179,6 @@ .. readiness.ContributedChecks(), /// a category with no row gets a new not-started row carrying the error, and a row that is Done, Failed or /// Abandoned is left alone. The next start that resumes a row clears the error. /// - /// Where the rows are read and saved. - /// The optional categories this instance selected. - /// Why the source could not be opened. Its type and message go into the error. - /// Cancelled when the host is shutting down. - /// Another writer saved one of these rows since it was read. internal static async Task RecordSourceOutage(IMigrationCheckpointStore checkpointStore, IReadOnlyList optionalCategories, Exception exception, CancellationToken cancellationToken = default) { var error = $"The {PersistenceFactory.MigrationSourcePersistenceType} migration source could not be opened: {exception.GetType().Name}: {exception.Message.TrimEnd('.', ' ')}. This category stays as it is, and the next start tries the source again."; @@ -221,15 +204,6 @@ internal static async Task RecordSourceOutage(IMigrationCheckpointStore checkpoi /// stopped and settled Halted, and the categories after it still get their go. Before copying anything it saves /// a not-started checkpoint for every category that has none, as does. /// - /// Copies each category. - /// The categories to copy, in the order to copy them. - /// Where each category's row is seeded, watched and, after a stall, settled. - /// The clock the watchdog measures a stall on. - /// Receives the watchdog's progress lines and the stall. - /// The host's token. Cancelling it stops the copy and leaves the running category as its last committed batch left it, so the next start resumes it. - /// The checkpoint each category ended on. - /// Another writer saved one of these checkpoints, which means a second instance is copying into the same database. - /// The host is shutting down. internal static async Task> RunRequiredCategories( MigrationEngine engine, IReadOnlyList categories, @@ -255,40 +229,60 @@ internal static async Task> RunRequiredCatego { using var categoryCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); progress.Watch(category.Id, timeProvider.GetUtcNow().UtcDateTime, categoryCancellation); + var run = engine.RunCategoryAsync(category, categoryCancellation.Token); try { - results.Add(await engine.RunCategoryAsync(category, categoryCancellation.Token)); + // Ends the wait on a stall or a host stop even when a call underneath ignores its token. That call keeps running, and the version check refuses its late commit or the settle below re-reads past it. + results.Add(await run.WaitAsync(categoryCancellation.Token)); } // Only the watchdog cancels this source without the host's token, so the filter tells a stall from a shutdown. #pragma warning disable PS0020 catch (OperationCanceledException) when (categoryCancellation.IsCancellationRequested && !cancellationToken.IsCancellationRequested) #pragma warning restore PS0020 { - // The host's token, because the stall has already cancelled the category's and a store call on that would fail. - var stalled = await checkpointStore.Read(category.Id, cancellationToken); + MigrationCheckpoint settled; - results.Add(await checkpointStore.Upsert(stalled with + try { - State = MigrationCategoryState.Halted, - SettledAt = timeProvider.GetUtcNow().UtcDateTime, - LastError = StallExplanation(category.Id) - }, cancellationToken)); + settled = await SettleStalled(checkpointStore, category.Id, timeProvider, cancellationToken); + } + // The stalled call can commit between the settle's read and its save, so one re-read settles from that commit. A second conflict is another instance, and it propagates. + catch (MigrationCheckpointConflictException) + { + settled = await SettleStalled(checkpointStore, category.Id, timeProvider, cancellationToken); + } + + results.Add(settled); + + _ = run.ContinueWith( + ended => logger.LogWarning(ended.Exception.GetBaseException(), + "{CategoryId}: the call the stall stopped waiting for ended after the category was settled Halted, with {ExceptionType}. It changed nothing after the settle, because the checkpoint store refuses a save made from before it.", + category.Id, ended.Exception.GetBaseException().GetType().Name), + cancellationToken, TaskContinuationOptions.OnlyOnFaulted, TaskScheduler.Default); } } return results; } + // Takes the host's token, because the stall has already cancelled the category's and a store call on that would fail. + static async Task SettleStalled(IMigrationCheckpointStore checkpointStore, string categoryId, TimeProvider timeProvider, CancellationToken cancellationToken) + { + var stalled = await checkpointStore.Read(categoryId, cancellationToken); + + return await checkpointStore.Upsert(stalled with + { + State = MigrationCategoryState.Halted, + SettledAt = timeProvider.GetUtcNow().UtcDateTime, + LastError = StallExplanation(categoryId) + }, cancellationToken); + } + /// /// Waits for the copy and turns a second instance writing checkpoints to the same database into a refusal the /// operator can act on. A shutdown and every other failure come out as they are. /// - /// The copy, already running. - /// Read for the persistence type the refusal names. - /// What the copy returned. - /// A checkpoint saved by another instance. The message says what to do and how to go back. - /// The host is shutting down. #pragma warning disable PS0018 // The copy it waits on already runs under the host's token, so a token here would have nothing to cancel. internal static async Task> CopyOrExplainWhyItStopped( Task> copy, @@ -313,7 +307,6 @@ internal static async Task> CopyOrExplainWhyI /// Throws unless every attempted category finished. A category that reported no checkpoint at all counts as /// outstanding too, because nothing says how much of it was copied. /// - /// A category did not finish. The message names each one with its state and counts, names the commands that move each Failed one on, and says how to go back. internal static void RefuseIfAnyCategoryDidNotComplete( IReadOnlyList attempted, IReadOnlyList finished, @@ -344,9 +337,6 @@ internal static void RefuseIfAnyCategoryDidNotComplete( /// Says whether the host may open without the source: every category this start copies is finished, and no row /// outside the copy is unfinished. /// - /// Every checkpoint row in the target. - /// The categories this start copies. - /// True when nothing required is still outstanding. internal static bool RequiredCopyIsSettled(IReadOnlyList checkpoints, IReadOnlyList toCopy) => UnfinishedRowsOutsideTheCopy(checkpoints, toCopy).Count == 0 && toCopy.All(category => checkpoints.Any(checkpoint => checkpoint.CategoryId == category.Id && checkpoint.State.IsFinished())); @@ -355,11 +345,6 @@ internal static bool RequiredCopyIsSettled(IReadOnlyList ch /// Throws unless every attempted category finished and no row outside the copy is unfinished, naming each /// outstanding row in the same words. /// - /// The categories this start copied. - /// Where each attempted category ended. - /// The rows found. - /// Read for the persistence type the refusal names. - /// Something is outstanding. The message names it and says how to go back. internal static void RefuseIfAnyCategoryDidNotComplete( IReadOnlyList attempted, IReadOnlyList finished, @@ -372,9 +357,6 @@ internal static void RefuseIfAnyCategoryDidNotComplete( /// required because a newer build may have written it for a category that must finish, and a required row for a /// category this build cannot copy yet. A finished row, and a row known to be optional, hold nothing. /// - /// Every checkpoint row in the target. - /// The categories this start copies, whose rows the copy judges itself. - /// The rows that keep the host closed, in the order they were read. internal static IReadOnlyList UnfinishedRowsOutsideTheCopy(IReadOnlyList checkpoints, IReadOnlyList toCopy) => [.. checkpoints .Where(checkpoint => MigrationCategoryRegistry.Find(checkpoint.CategoryId)?.Kind != MigrationCategoryKind.Optional) @@ -402,7 +384,6 @@ static string LastErrorClause(MigrationCheckpoint checkpoint) => /// Logs one line per category saying what it copied and what it left behind. Skipped rows are warned about /// one at a time while the copy runs, and nothing else states the total or says what becomes of them. /// - /// When this start began, which is what tells a category this run finished from one an earlier run did. internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList finished, ILogger logger, DateTime runStartedAt) { foreach (var checkpoint in finished) @@ -455,19 +436,26 @@ internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList !reason.IsBenign()).ToArray() ?? []; var permanent = faults.Where(reason => reason.IsPermanent()).ToArray(); + var cannotFix = permanent.Length == 0 ? "" : $" No retry can fix the {string.Join(" or ", permanent)} ones."; + + // A Halted category never read the rows after its cursor, so abandoning it gives up far more than the skipped rows. + if (faults.Length > 0 && checkpoint.State == MigrationCategoryState.Halted) + { + return $"The copy stopped before the end of the category, so these and every row not yet read stay only in the source database. " + + $"If it stopped because the skips passed the halt threshold, raise {MigrationSettings.HaltThresholdPercentKey} or {MigrationSettings.HaltThresholdMinimumKey}, or fix their cause, then run --migration-retry {checkpoint.CategoryId}, which copies the category again from the start.{cannotFix} " + + $"--migration-abandon {checkpoint.CategoryId} keeps what was copied and gives up every row not yet read, not only these."; + } if (faults.Length > 0 && permanent.Length == faults.Length) { return $"They stay only in the source database, and no retry can fix them, so --migration-abandon {checkpoint.CategoryId} keeps what was copied and gives up the rest."; } - var cannotFix = permanent.Length == 0 ? "" : $" No retry can fix the {string.Join(" or ", permanent)} ones."; - return $"They stay only in the source database until --migration-retry {checkpoint.CategoryId} re-reads them once the cause is fixed.{cannotFix}"; } @@ -521,9 +509,6 @@ public ClosedWindowProgress(IMigrationCheckpointStore checkpointStore, TimeProvi /// Points the watchdog at the category that is starting now, in place of the one before it. Only this /// category is judged from here on, from the later of its last committed batch and . /// - /// The category starting now. - /// When this start began running it. A row an earlier start left carries an older stamp, which would read as a stall at once. - /// Cancelled when the category stalls. The category must run under its token, and nothing else should. public void Watch(string categoryId, DateTime runStartedAt, CancellationTokenSource stop) => running = new RunningCategory(categoryId, runStartedAt, stop);