Count an audit retention sweep that leaves expired rows as a failure

[warwickschroeder]

What this fixes

A sweep that left expired rows because the audit tables were too busy was recorded as a success, so the Audit retention check stayed green while retention stopped. It now counts as a failed sweep, so three in a row turn the check red.

• A deferred drop fails the sweep, with a reason naming the rows left behind.

How to reproduce

On PostgreSQL, create an expired audit partition and hold LOCK TABLE <partition> IN ACCESS SHARE MODE in an open transaction. Each sweep logs that the tables were too busy, and the Audit retention check stays green.

Tests

• ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests: three deferred sweeps turn the check red, on SQL Server and PostgreSQL.

[rbev]

Approved, but I'm not convinced this is a real issue

• Audit partition for the data is based on truncate(DateTime.Now) in the ingestion pipeline
• Audit retention is measured in days

So this task should only ever be dropping partitions that are no longer being touched.