Skip to content

Add saga management satellite - #7996

Closed
SzymonPobiega wants to merge 3 commits into
saga-timeout-full-messagesfrom
saga-timeout-full-messages-satellite
Closed

SzymonPobiega wants to merge 3 commits into
saga-timeout-full-messagesfrom
saga-timeout-full-messages-satellite

Conversation

@SzymonPobiega

@SzymonPobiega SzymonPobiega commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Endpoints can report saga timeout changes to ServiceControl and apply timeout changes that an operator requests in ServicePulse. Reporting is opt-in with endpointConfiguration.ReportSagaTimeoutsToServiceControl().

This builds on saga-timeout-full-messages. The design across NServiceBus, ServiceControl, and ServicePulse is in docs/saga-timeout-management-design.md in Particular/ServiceControl#6025. The relevant sections are "Endpoint side" and "Rollout and compatibility".

Why opt-in

A ServiceControl version without timeout management does not know the report message types. Reports sent to it would fail on the primary instance's input queue and end up in the error queue as failed messages of ServiceControl itself. The feature also needs a persister that implements ISupportTimeoutManagement; today only the learning persister does.

Changes

  • ReportSagaTimeoutsToServiceControl() enables the ReportSagaTimeoutsToPlatform feature. Without it, the saga pipeline no longer reports timeout changes.
  • Every report now carries Host, HostId, and ManagementAddress, the transport address of the management satellite. SagaTimeoutRescheduled, SagaTimeoutCanceled, and SagaTimeoutChangeRejected carry OperationId when an operator requested the change. Reason can now be TimeoutNotFound, VersionMismatch, or SagaNotFound.
  • SagaCompleted is sent when a saga that had stored timeouts when it was loaded completes. Before, it depended on the timeouts left after the handler ran, so a handler that canceled the last timeout and then completed reported nothing.
  • A new satellite, "Saga timeout management", listens on {endpoint queue}.SagaTimeouts. It applies Reschedule and Cancel requests carried entirely in NServiceBus.SagaTimeoutManagement.* headers, so the body is never deserialized. A reschedule body, when present, replaces the timeout payload as is. "Trigger now" is a reschedule to the current time.

How the satellite applies a request

  1. Looks up the request's message id in the outbox and, for a duplicate, dispatches the stored operations without applying the request again.
  2. Opens a synchronized storage session on the outbox transaction, or on the receive transaction when the outbox is disabled.
  3. Loads the saga by type name and id, and compares ExpectedVersion with the stored timeout.
  4. Applies the change and saves the saga. A missing saga, missing timeout, or different version saves nothing and reports SagaTimeoutChangeRejected.
  5. Stores the rescheduled timeout message and the reports in the outbox record, commits, dispatches, and marks the record as dispatched.

The outbox handling mirrors TransportReceiveToPhysicalMessageConnector, whose outbox conversion helpers are now internal so the satellite can reuse them. A malformed request moves to the error queue without retries; other failures use the default recoverability policy.

Without the outbox, one gap remains. If the saga save commits and the dispatch fails, the retried request is rejected with VersionMismatch. The change is saved but not reported, and the rescheduled timeout message is lost. ServiceControl then shows the timeout as overdue.

Alternatives considered

The first design proposed a ChangeSagaTimeout command handled in the normal pipeline. A satellite was chosen instead. It does not benefit from the pipeline, and it does not need to: every field of the request travels in headers, and the body is stored as is.

Verification

  • SagaTimeoutManagementSatelliteTests covers the outbox paths with a fake outbox storage: a new request, a duplicate, and a rejection. No in-repo persister supports both timeout management and the outbox, so there is no end-to-end outbox test.
  • SagaTimeoutManagementRequestTests covers header parsing.
  • When_operator_changes_saga_timeout drives the satellite through the reported ManagementAddress: reschedule with a new payload, a stale cancel rejected with VersionMismatch, and a cancel. When_saga_timeouts_change covers the opt-in, the new fields, and completion after the last timeout is canceled.
  • Results:
    • Core unit tests for sagas, approvals, outbox, and pipeline: 256 passed.
    • Learning acceptance suite: 171 passed and 28 skipped, in three consecutive runs.
    • Saga, satellite, and outbox acceptance tests: 65 passed and 12 skipped.
  • Ran end to end with RabbitMQ, ServiceControl#6025, and ServicePulse#3166: saga reschedules, an operator reschedule with a changed payload, a stale-version rejection, trigger now, and saga completion.

🤖 Generated with Claude Code

@mauroservienti
mauroservienti force-pushed the saga-timeout-full-messages branch from 5c3bd73 to 46e9722 Compare October 19, 2026 16:02
@mauroservienti
mauroservienti force-pushed the saga-timeout-full-messages-satellite branch from e85a4e4 to 7e837ef Compare October 19, 2026 16:02

// Applies timeout changes requested by an operator in ServiceControl. Satellites bypass the message pipeline,
// so the saga is loaded and saved directly, and the outbox is used the same way TransportReceiveToPhysicalMessageConnector uses it.
sealed partial class SagaTimeoutManagementSatellite(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be worthwhile having a core concept for "satellites with pipelines"?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a spike one day I should be able to find. It is actually not that complicated. The problem is more whether you want to use the normal stages or whether you'd like to introduce specific stages so that user behavior don't automatically get loaded when they are placed into a specific stage but maybe that could also be a "desirable" trait

Updates the callers this branch adds to await FindTimeouts and CancelTimeout.
mauroservienti added a commit that referenced this pull request Oct 21, 2026
Updates the callers this branch adds to await the timeout management API.
@rbev rbev closed this Oct 23, 2026
@rbev

rbev commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Closing all intermediate spikes in favor of #8000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants