Skip to content

Reuse one query plan per SQL Server ingestion statement - #6011

Open
warwickschroeder wants to merge 2 commits into
masterfrom
warwick/sqlserver-ingestion-parameter-sizes
Open

warwickschroeder wants to merge 2 commits into
masterfrom
warwick/sqlserver-ingestion-parameter-sizes

Conversation

@warwickschroeder

@warwickschroeder warwickschroeder commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

SQL Server ingestion declared each parameter by its value, so nearly every failed message compiled a new query plan and filled the plan cache. Every parameter is now declared by its column, so each statement shape has one plan. It does not touch the deadlocks that concurrent multi-message batches can hit.

  • Parameters take their type and size from the EF model, so value lengths and nulls no longer change the declaration.
  • A string longer than its column is still refused, not silently cut to the column size.
  • A string sent to a non-text column throws, so a misaligned caller fails loudly.
  • Every raw SQL Server write uses it: failed messages, groups, known endpoints, retries and throughput.

Evidence

Measured with tools/testing-tool (Aspire, RabbitMQ transport, SQL Server persistence, batch size 25, 4 parallel writers). The bypass writer sent background-noise failures at 2,000 msg/s for 11 minutes, more than either build could ingest, so the error queue grew throughout. One run per build, each on an empty database. Figures cover the same 9.3-minute window after a 1-minute warm-up.

Before (master) After (this PR)
Messages ingested per second 384 518 (+35%)
SQL compilations per 1,000 messages 86.6 3.1
Cached MERGE FailedMessages plans at the end 1,415, each used about once 11, used 14,547 times
Plan cache (CACHESTORE_SQLCP) 3,055 MB, at its cap 144 MB

Audit was measured with a driver that calls the real BatchInsert.Rows directly: 10,000 audit messages, 32 a batch, 4 writers, three runs per build, each on an empty database and plan cache.

Bodies over 4,000 characters Messages per second, before After Cached plans, before After Plan cache, before After
None ~11,000 ~11,000 3 3 ~5.5 MB 6.2 MB
3% ~7,500 ~11,000 ~107 3 ~157 MB 6.2 MB
10% ~4,500 ~10,500 ~284 3 ~405 MB 6.2 MB

Saga snapshots behave the same way. With 10% of states over 4,000 characters, the plans go from 285 to 3 and the plan cache from 211 MB to 4.4 MB.

Tests

  • ServiceControl.Persistence.Tests.SqlServer/IngestionPlanReuseTests (new): one cached plan across value lengths and nulls, and an over-length value refused.
  • ServiceControl.Persistence.Tests/EFCore: the shared transaction helper moved to the base class, unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants