From 3b7bddaa7d686482d7487ed8b20f678a58def8cd Mon Sep 17 00:00:00 2001 From: wangbill Date: Tue, 22 Sep 2026 16:23:13 -0700 Subject: [PATCH 01/15] Expose blob auto-purge integration for alternate .NET hosts Reuse the existing purge tasks and bootstrap sequence with a task-hub-bound transport interface. Preserve standalone behavior and prove DTFx shim execution, failure handling, replay, and per-hub setup. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 34 +++ .../GetLargePayloadTombstonesActivity.cs | 35 ++- .../ReportLargePayloadPurgeResultsActivity.cs | 47 ++-- ...eTaskClientExtensions.AzureBlobPayloads.cs | 48 +++- .../Client/GrpcLargePayloadPurgeClient.cs | 73 +++++ .../Client/ILargePayloadPurgeClient.cs | 53 ++++ .../AutoPurge/Constants/BlobPurgeConstants.cs | 8 +- ...rkerBuilderExtensions.AzureBlobPayloads.cs | 11 +- .../AlternateHostPurgeClientTests.cs | 186 ++++++++++++ .../AutoPurge/AlternateHostPurgeTaskTests.cs | 264 ++++++++++++++++++ .../AutoPurge/EternalPurgeClientTests.cs | 59 +++- .../AzureBlobPayloads.Tests.csproj | 1 + 12 files changed, 768 insertions(+), 51 deletions(-) create mode 100644 src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs create mode 100644 src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs create mode 100644 test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs create mode 100644 test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs diff --git a/README.md b/README.md index 2fc37547..c04bfda2 100644 --- a/README.md +++ b/README.md @@ -200,6 +200,40 @@ For runnable DTS emulator examples that demonstrate versioning, see the [WorkerV The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) shows how to declare selected activities for Durable Task Scheduler (DTS)-managed on-demand sandbox execution and build the remote worker container image separately from the declarer app. +### Blob auto-purge integration for alternate .NET hosts + +`Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes infrastructure APIs for hosts that already +dispatch Durable Task Framework (`DurableTask.Core`) tasks. These APIs do not add another worker or require +application functions. Register `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, +`DeleteExternalBlobActivity`, and `ReportLargePayloadPurgeResultsActivity` under their exact class names +with an empty version, using `Microsoft.DurableTask.Worker.Shims.DurableTaskShimFactory.CreateOrchestration` +and `CreateActivity`. Keep these tasks registered even when auto-purge is disabled so existing work can finish. +This does not promise compatibility with arbitrary historical versioned purge runners. + +Construct the fetch and report activities with a host-owned `ILargePayloadPurgeClient` and their typed +loggers, and the delete activity with the task hub's `PayloadStore` and logger. The narrow purge client must +use that hub's existing authenticated transport, follow reconnection and credential refresh, honor the +activities' UTC deadlines, and preserve opaque tombstone tokens. Fetch/report gRPC failures are classified +by the activities, not swallowed by the adapter. The host retains ownership of its clients and store. + +For explicit setup, call `orchestrationClient.SetLargePayloadAutoPurgeAsync(purgeClient, enabled, batchSize, +cancellationToken)`. Both clients must target the **same authenticated task hub**. A host with an existing +`IOrchestrationServiceClient` can obtain the orchestration client through +`AddDurableTaskClient(name).UseOrchestrationService(options => { options.Client = serviceClient; +options.EnableEntitySupport = false; })` from `Microsoft.DurableTask.Client.OrchestrationServiceClientShim`. +Use the SDK's default data converter consistently for this client and the task shims. DTFx wraps activity +arguments in an outer JSON array; the shims handle this envelope. Preserve structured activity failure details +(`ErrorPropagationMode.UseFailureDetails`) and unprocessed external events across continue-as-new. +Cancellation remains subject to the supplied client's capabilities; an in-flight service call without +cancellation support cannot be canceled by this helper. + +Enabling writes the setting, starts the reserved per-task-hub instance with live-status deduplication, +verifies the runner's identity and Running status, then sends `SetBatchSize`. Disabling **only** writes the +setting and ignores batch size. The steps are not transactional; failures propagate without rollback. +Repeated desired-state setup after host takeover is supported. Owner election and deciding whether an omitted +setting requires any action belong to the host, not this SDK. `BlobPurgeConstants` provides the reserved +instance ID, event name and batch bounds; never use the reserved instance ID for application work. + ## Obtaining the Protobuf definitions This project utilizes protobuf definitions from [durabletask-protobuf](https://github.com/microsoft/durabletask-protobuf), which are copied (vendored) into this repository under the `src/Grpc` directory. See the corresponding [README.md](./src/Grpc/README.md) for more information about how to update the protobuf definitions. diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs index 2c314721..59767fa2 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs @@ -5,7 +5,6 @@ using Microsoft.DurableTask.Client; using Microsoft.Extensions.Logging; using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; -using LP = Microsoft.DurableTask.Protobuf.LargePayloads; namespace Microsoft.DurableTask.AzureBlobPayloads; @@ -15,15 +14,30 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// /// The large-payload purge service client used to query the backend for tombstones. /// The logger instance. +/// +/// Infrastructure integration API for alternate .NET hosts. The supplied client must be bound to this +/// worker's authenticated task hub. Its transport lifetime remains owned by the host. +/// [DurableTask] -internal sealed class GetLargePayloadTombstonesActivity( - LargePayloadPurgeClient client, +public sealed class GetLargePayloadTombstonesActivity( + ILargePayloadPurgeClient client, ILogger logger) : TaskActivity> { - readonly LargePayloadPurgeClient client = Check.NotNull(client); + readonly ILargePayloadPurgeClient client = Check.NotNull(client); readonly ILogger logger = Check.NotNull(logger); + /// + /// Initializes a new instance of the class using the worker's transport. + /// + /// The worker's purge client. + /// The activity logger. + internal GetLargePayloadTombstonesActivity( + LargePayloadPurgeClient client, ILogger logger) + : this(new GrpcLargePayloadPurgeClient(client), logger) + { + } + /// /// Gets or sets the timeout for one backend RPC attempt. /// @@ -41,13 +55,10 @@ public override async Task> RunAsync(TaskActivityCon nameof(input), input, $"Limit must be greater than 0 and less than or equal to {LargePayloadTombstone.MaxRequestLimit}."); } - LP.GetLargePayloadTombstonesResponse response; + List tombstones; try { - using var call = this.client.GetLargePayloadTombstonesAsync( - new LP.GetLargePayloadTombstonesRequest { Limit = input }, - deadline: DateTime.UtcNow.Add(this.RpcTimeout)); - response = await call; + tombstones = await this.client.GetLargePayloadTombstonesAsync(input, DateTime.UtcNow.Add(this.RpcTimeout)); } catch (RpcException e) when (e.StatusCode == StatusCode.Cancelled) { @@ -76,12 +87,6 @@ public override async Task> RunAsync(TaskActivityCon e); } - List tombstones = new(response.Tombstones.Count); - foreach (LP.LargePayloadTombstone tombstone in response.Tombstones) - { - tombstones.Add(new LargePayloadTombstone(tombstone.TombstoneToken, tombstone.PayloadToken)); - } - this.logger.BlobPurgeFetchedTombstones(tombstones.Count); return tombstones; } diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs index a2a7e05a..cf3501b6 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs @@ -5,7 +5,6 @@ using Microsoft.DurableTask.Client; using Microsoft.Extensions.Logging; using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; -using LP = Microsoft.DurableTask.Protobuf.LargePayloads; namespace Microsoft.DurableTask.AzureBlobPayloads; @@ -17,15 +16,30 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// /// The large-payload purge service client used to report purge results to the backend. /// The logger instance. +/// +/// Infrastructure integration API for alternate .NET hosts. The supplied client must be bound to this +/// worker's authenticated task hub. Its transport lifetime remains owned by the host. +/// [DurableTask] -internal sealed class ReportLargePayloadPurgeResultsActivity( - LargePayloadPurgeClient client, +public sealed class ReportLargePayloadPurgeResultsActivity( + ILargePayloadPurgeClient client, ILogger logger) : TaskActivity, object?> { - readonly LargePayloadPurgeClient client = Check.NotNull(client); + readonly ILargePayloadPurgeClient client = Check.NotNull(client); readonly ILogger logger = Check.NotNull(logger); + /// + /// Initializes a new instance of the class using the worker's transport. + /// + /// The worker's purge client. + /// The activity logger. + internal ReportLargePayloadPurgeResultsActivity( + LargePayloadPurgeClient client, ILogger logger) + : this(new GrpcLargePayloadPurgeClient(client), logger) + { + } + /// /// Gets or sets the timeout for one backend RPC attempt. /// @@ -43,32 +57,9 @@ internal sealed class ReportLargePayloadPurgeResultsActivity( return null; } - LP.ReportLargePayloadPurgeResultsRequest request = new(); - foreach (LargePayloadPurgeResult result in input) - { - request.Results.Add(new LP.LargePayloadPurgeResult - { - // Echoed back exactly as it was received. The SDK never parses or rebuilds this token, so a - // change to what the backend puts in it needs no change here. - TombstoneToken = result.TombstoneToken, - - // The managed disposition enum declares the same numeric values as its protobuf counterpart, - // so it maps across by value. This is the only enum on the message and it only travels - // outbound, so the SDK can never receive a value it does not know. - Disposition = (LP.LargePayloadPurgeDisposition)result.Disposition, - }); - } - - if (request.Results.Count == 0) - { - return null; - } - try { - using var call = this.client.ReportLargePayloadPurgeResultsAsync( - request, deadline: DateTime.UtcNow.Add(this.RpcTimeout)); - await call; + await this.client.ReportLargePayloadPurgeResultsAsync(input, DateTime.UtcNow.Add(this.RpcTimeout)); } catch (RpcException e) when (e.StatusCode == StatusCode.Cancelled) { diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs index 4f077aed..f583fb67 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs @@ -69,7 +69,53 @@ public static async Task SetLargePayloadAutoPurgeAsync( throw new NotSupportedException($"Large-payload auto-purge requires a gRPC Durable Task client, not '{client.GetType().FullName}'."); } - await autoPurgeClient.SetLargePayloadAutoPurgeAsync(enabled, cancellationToken); + await SetLargePayloadAutoPurgeCoreAsync( + client, autoPurgeClient.SetLargePayloadAutoPurgeAsync, enabled, batchSize, cancellationToken); + } + + /// + /// Applies explicit auto-purge configuration using an alternate host's task-hub-bound clients. + /// + /// The orchestration client targeting the same authenticated task hub as . + /// The host-owned transport for the task hub's auto-purge setting. + /// True to enable cleanup; false to pause new fetches through the backend setting. + /// The requested batch size, from 1 through 1000. Ignored when disabling. + /// Cancels any setting, start, wait or configuration operation. + /// A task that completes after the setting and, when enabling, start verification and event enqueue. + /// + /// Infrastructure integration API, not an application orchestration API. The caller must bind both clients + /// to the same task hub; neither client is disposed by this method. This uses the same nontransactional + /// setting, fixed-ID deduplicated start, Running identity verification and batch-size event sequence as + /// the standalone overload. Disabling only writes the setting. Repeating desired state after host takeover + /// is supported; this does not implement owner election or guarantee once-ever setup. + /// The host must register the SDK purge tasks before enabling and retain their canonical names and inputs. + /// + public static async Task SetLargePayloadAutoPurgeAsync( + this DurableTaskClient client, + ILargePayloadPurgeClient purgeClient, + bool enabled, + int batchSize = BlobPurgeConstants.DefaultBatchSize, + CancellationToken cancellationToken = default) + { + Check.NotNull(client); + Check.NotNull(purgeClient); + if (enabled && (batchSize < 1 || batchSize > BlobPurgeConstants.MaxBatchSize)) + { + throw new ArgumentOutOfRangeException(nameof(batchSize), batchSize, "Purge batch size is out of range."); + } + + await SetLargePayloadAutoPurgeCoreAsync( + client, purgeClient.SetLargePayloadAutoPurgeAsync, enabled, batchSize, cancellationToken); + } + + static async Task SetLargePayloadAutoPurgeCoreAsync( + DurableTaskClient client, + Func setEnabled, + bool enabled, + int batchSize, + CancellationToken cancellationToken) + { + await setEnabled(enabled, cancellationToken); if (!enabled) { return; diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs new file mode 100644 index 00000000..1cda3265 --- /dev/null +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs @@ -0,0 +1,73 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using Grpc.Core; +using Microsoft.DurableTask.Client; +using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; +using LP = Microsoft.DurableTask.Protobuf.LargePayloads; + +namespace Microsoft.DurableTask.AzureBlobPayloads; + +/// +/// Adapts the worker's existing, rebindable purge transport without owning its lifetime. +/// +sealed class GrpcLargePayloadPurgeClient(LargePayloadPurgeClient client) : ILargePayloadPurgeClient +{ + readonly LargePayloadPurgeClient client = Check.NotNull(client); + + /// + public async Task SetLargePayloadAutoPurgeAsync(bool enabled, CancellationToken cancellationToken = default) + { + try + { + using var call = this.client.SetLargePayloadAutoPurgeAsync( + new LP.SetLargePayloadAutoPurgeRequest { Enabled = enabled }, cancellationToken: cancellationToken); + await call; + } + catch (RpcException e) when (e.StatusCode == StatusCode.Cancelled) + { + throw new OperationCanceledException( + "The SetLargePayloadAutoPurge operation was canceled.", e, cancellationToken); + } + catch (RpcException e) when (e.StatusCode == StatusCode.Unimplemented) + { + throw new NotImplementedException(e.Status.Detail); + } + } + + /// + public async Task> GetLargePayloadTombstonesAsync( + int limit, DateTime deadline, CancellationToken cancellationToken = default) + { + using var call = this.client.GetLargePayloadTombstonesAsync( + new LP.GetLargePayloadTombstonesRequest { Limit = limit }, deadline: deadline, cancellationToken: cancellationToken); + LP.GetLargePayloadTombstonesResponse response = await call; + List tombstones = new(response.Tombstones.Count); + foreach (LP.LargePayloadTombstone tombstone in response.Tombstones) + { + tombstones.Add(new LargePayloadTombstone(tombstone.TombstoneToken, tombstone.PayloadToken)); + } + + return tombstones; + } + + /// + public async Task ReportLargePayloadPurgeResultsAsync( + IReadOnlyList results, DateTime deadline, CancellationToken cancellationToken = default) + { + LP.ReportLargePayloadPurgeResultsRequest request = new(); + foreach (LargePayloadPurgeResult result in results) + { + request.Results.Add(new LP.LargePayloadPurgeResult + { + // Echo the opaque correlation token unchanged. The managed and protobuf enums share values. + TombstoneToken = result.TombstoneToken, + Disposition = (LP.LargePayloadPurgeDisposition)result.Disposition, + }); + } + + using var call = this.client.ReportLargePayloadPurgeResultsAsync( + request, deadline: deadline, cancellationToken: cancellationToken); + await call; + } +} diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs new file mode 100644 index 00000000..b5f2ea66 --- /dev/null +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs @@ -0,0 +1,53 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using Microsoft.DurableTask.Client; + +namespace Microsoft.DurableTask.AzureBlobPayloads; + +/// +/// Provides task-hub-bound transport operations for integrating blob auto-purge with an alternate .NET host. +/// +/// +/// This is an infrastructure integration API, not an application orchestration API. Implementations must +/// use the same authenticated task hub as the associated orchestration client and preserve its authentication, +/// metadata, reconnection and transport lifetime. The SDK does not own or dispose the supplied client. +/// Fetch and report must propagate gRPC status exceptions unchanged: the activities own their cancellation, +/// unsupported-backend and fetch-precondition handling. They must honor the supplied UTC deadline. +/// Correlation tokens must be returned and reported exactly as received, without parsing or reconstruction. +/// +public interface ILargePayloadPurgeClient +{ + /// + /// Writes the auto-purge setting for this client's authenticated task hub. + /// + /// The setting to persist. + /// Cancels the setting operation. + /// A task that completes when the backend acknowledges the setting. + /// + /// Map backend cancellation to with the supplied token and an + /// unsupported operation to . Other failures must propagate. + /// This operation alone must not start, terminate or wait for an orchestration. + /// + Task SetLargePayloadAutoPurgeAsync(bool enabled, CancellationToken cancellationToken = default); + + /// + /// Fetches a bounded batch of due tombstones for this client's authenticated task hub. + /// + /// The requested maximum number of tombstones, from 1 through 1000. + /// The absolute UTC deadline for this backend attempt. + /// Cancels the fetch operation. + /// The fetched tombstones, with their opaque correlation and payload tokens unchanged. + Task> GetLargePayloadTombstonesAsync( + int limit, DateTime deadline, CancellationToken cancellationToken = default); + + /// + /// Reports deletion outcomes for this client's authenticated task hub. + /// + /// The outcomes, each carrying the exact correlation token received during fetch. + /// The absolute UTC deadline for this backend attempt. + /// Cancels the report operation. + /// A task that completes when the backend acknowledges the results. + Task ReportLargePayloadPurgeResultsAsync( + IReadOnlyList results, DateTime deadline, CancellationToken cancellationToken = default); +} diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs index 24e8c4f6..f39560a2 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs @@ -8,7 +8,11 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// /// Constants used throughout the blob payload auto-purge functionality. /// -static class BlobPurgeConstants +/// +/// Infrastructure integration constants. The orchestration ID is reserved per task hub and must not be used +/// for application orchestrations. +/// +public static class BlobPurgeConstants { /// /// The fixed orchestration instance ID for the auto-purge job in each task hub. @@ -36,5 +40,5 @@ static class BlobPurgeConstants /// /// The maximum duration of an individual fetch or report RPC attempt. /// - public const int RpcTimeoutSeconds = 60; + internal const int RpcTimeoutSeconds = 60; } diff --git a/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs b/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs index dafe5784..604d44b3 100644 --- a/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs +++ b/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs @@ -8,6 +8,7 @@ using Microsoft.DurableTask.Worker.Grpc.Internal; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; using P = Microsoft.DurableTask.Protobuf; @@ -131,12 +132,14 @@ static IDurableTaskWorkerBuilder UseExternalizedPayloadsCore(IDurableTaskWorkerB { r.AddOrchestrator(); r.AddActivity(nameof(GetLargePayloadTombstonesActivity), sp => - ActivatorUtilities.CreateInstance( - sp, sp.GetRequiredKeyedService(builder.Name))); + new GetLargePayloadTombstonesActivity( + sp.GetRequiredKeyedService(builder.Name), + sp.GetRequiredService>())); r.AddActivity(); r.AddActivity(nameof(ReportLargePayloadPurgeResultsActivity), sp => - ActivatorUtilities.CreateInstance( - sp, sp.GetRequiredKeyedService(builder.Name))); + new ReportLargePayloadPurgeResultsActivity( + sp.GetRequiredKeyedService(builder.Name), + sp.GetRequiredService>())); }); return builder; diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs new file mode 100644 index 00000000..d63772ef --- /dev/null +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs @@ -0,0 +1,186 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using DurableTask.Core; +using DurableTask.Core.Exceptions; +using DurableTask.Core.History; +using Microsoft.DurableTask.AzureBlobPayloads; +using Microsoft.DurableTask.Client; +using Microsoft.DurableTask.Converters; +using Microsoft.Extensions.DependencyInjection; + +namespace Microsoft.DurableTask.Extensions.AzureBlobPayloads.Tests.AutoPurge; + +public class AlternateHostPurgeClientTests +{ + [Fact] + public async Task Enable_UsesExistingServiceClient_AndRepeatedSetupIsIndependentPerHubAsync() + { + // Arrange + Hub first = new("first"); + Hub second = new("second"); + ServiceCollection services = new(); + first.Register(services); + second.Register(services); + await using ServiceProvider provider = services.BuildServiceProvider(); + IDurableTaskClientProvider clients = provider.GetRequiredService(); + using CancellationTokenSource cancellation = new(); + + // Act + await clients.GetClient("first").SetLargePayloadAutoPurgeAsync(first.Purge.Object, true, 123, cancellation.Token); + await clients.GetClient("second").SetLargePayloadAutoPurgeAsync(second.Purge.Object, true, 456, cancellation.Token); + first.AlreadyExists = true; + await clients.GetClient("first").SetLargePayloadAutoPurgeAsync(first.Purge.Object, true, 789, cancellation.Token); + + // Assert + Assert.Equal(new[] { "Set:True", "Start", "Wait", "Event", "Set:True", "Start", "Wait", "Event" }, first.Calls); + Assert.Equal(new[] { "Set:True", "Start", "Wait", "Event" }, second.Calls); + Assert.Equal(new[] { "123", "789" }, first.Events.Select(e => e.Input)); + Assert.Equal("456", Assert.Single(second.Events).Input); + Assert.All(first.Tokens.Concat(second.Tokens), token => Assert.Equal(cancellation.Token, token)); + Assert.All(first.Starts.Concat(second.Starts), start => + { + Assert.Equal(nameof(BlobPurgeJobOrchestrator), start.Name); + Assert.Equal(BlobPurgeConstants.OrchestratorInstanceId, start.OrchestrationInstance.InstanceId); + Assert.Equal(string.Empty, start.Version); + Assert.InRange(JsonDataConverter.Default.Deserialize(start.Input)!.PurgeBatchSize, 1, 1000); + }); + } + + [Fact] + public async Task Disable_OnlyWritesSetting_AndStandaloneOverloadStillRejectsShimAsync() + { + // Arrange + Hub hub = new("hub"); + ServiceCollection services = new(); + hub.Register(services); + await using ServiceProvider provider = services.BuildServiceProvider(); + DurableTaskClient client = provider.GetRequiredService().GetClient("hub"); + + // Act + await client.SetLargePayloadAutoPurgeAsync(hub.Purge.Object, false, -1); + + // Assert + Assert.Equal(new[] { "Set:False" }, hub.Calls); + await Assert.ThrowsAsync(() => client.SetLargePayloadAutoPurgeAsync(true)); + Assert.Equal(new[] { "Set:False" }, hub.Calls); + } + + [Theory] + [InlineData(OrchestrationStatus.Suspended, nameof(BlobPurgeJobOrchestrator))] + [InlineData(OrchestrationStatus.Pending, nameof(BlobPurgeJobOrchestrator))] + [InlineData(OrchestrationStatus.ContinuedAsNew, nameof(BlobPurgeJobOrchestrator))] + [InlineData(OrchestrationStatus.Running, "BusinessOrchestration")] + public async Task Enable_LiveCollisionOrNonRunningState_DoesNotSendEventAsync(OrchestrationStatus status, string name) + { + // Arrange + Hub hub = new("hub") { AlreadyExists = true, Status = status, Name = name }; + ServiceCollection services = new(); + hub.Register(services); + await using ServiceProvider provider = services.BuildServiceProvider(); + DurableTaskClient client = provider.GetRequiredService().GetClient("hub"); + + // Act / Assert + // The real shim waits on Pending, so cancellation bounds that case rather than changing its semantics. + using CancellationTokenSource cancellation = new(TimeSpan.FromMilliseconds(100)); + Exception? error = await Record.ExceptionAsync(() => + client.SetLargePayloadAutoPurgeAsync(hub.Purge.Object, true, cancellationToken: cancellation.Token)); + Assert.NotNull(error); + Assert.True(error is InvalidOperationException or OperationCanceledException, error.ToString()); + Assert.Empty(hub.Events); + } + + [Theory] + [InlineData(0)] + [InlineData(1001)] + public async Task InvalidBatch_DoesNotWriteSettingAsync(int batchSize) + { + // Arrange + Mock client = new(MockBehavior.Strict, "hub"); + Mock purge = new(MockBehavior.Strict); + + // Act / Assert + await Assert.ThrowsAsync(() => + client.Object.SetLargePayloadAutoPurgeAsync(purge.Object, true, batchSize)); + client.VerifyNoOtherCalls(); + purge.VerifyNoOtherCalls(); + } + + [Fact] + public async Task SettingFailure_PropagatesWithoutStartingAsync() + { + // Arrange + Mock client = new(MockBehavior.Strict, "hub"); + Mock purge = new(MockBehavior.Strict); + using CancellationTokenSource cancellation = new(); + OperationCanceledException failure = new(cancellation.Token); + purge.Setup(p => p.SetLargePayloadAutoPurgeAsync(true, cancellation.Token)).ThrowsAsync(failure); + + // Act / Assert + Assert.Same(failure, await Assert.ThrowsAsync(() => + client.Object.SetLargePayloadAutoPurgeAsync(purge.Object, true, cancellationToken: cancellation.Token))); + client.VerifyNoOtherCalls(); + } + + sealed class Hub + { + readonly string key; + readonly Mock service = new(MockBehavior.Strict); + + public Hub(string key) + { + this.key = key; + this.Purge.Setup(p => p.SetLargePayloadAutoPurgeAsync(It.IsAny(), It.IsAny())) + .Callback((enabled, token) => + { + this.Calls.Add($"Set:{enabled}"); + this.Tokens.Add(token); + }).Returns(Task.CompletedTask); + this.service.Setup(s => s.CreateTaskOrchestrationAsync(It.IsAny(), It.IsAny())) + .Returns((TaskMessage message, OrchestrationStatus[] statuses) => + { + this.Calls.Add("Start"); + Assert.Equal(new[] { OrchestrationStatus.Running, OrchestrationStatus.Pending, OrchestrationStatus.Suspended, OrchestrationStatus.ContinuedAsNew }, statuses); + this.Starts.Add(Assert.IsType(message.Event)); + return this.AlreadyExists ? Task.FromException(new OrchestrationAlreadyExistsException("existing")) : Task.CompletedTask; + }); + this.service.Setup(s => s.GetOrchestrationStateAsync(BlobPurgeConstants.OrchestratorInstanceId, false)) + .Returns(() => + { + this.Calls.Add("Wait"); + return Task.FromResult>([new() + { + Name = this.Name, + OrchestrationStatus = this.Status, + OrchestrationInstance = new() { InstanceId = BlobPurgeConstants.OrchestratorInstanceId }, + }]); + }); + this.service.Setup(s => s.SendTaskOrchestrationMessageAsync(It.IsAny())) + .Callback(message => + { + this.Calls.Add("Event"); + Assert.Equal(BlobPurgeConstants.OrchestratorInstanceId, message.OrchestrationInstance.InstanceId); + EventRaisedEvent raised = Assert.IsType(message.Event); + Assert.Equal(BlobPurgeConstants.SetBatchSizeEvent, raised.Name); + this.Events.Add(raised); + }).Returns(Task.CompletedTask); + } + + public Mock Purge { get; } = new(MockBehavior.Strict); + public List Calls { get; } = []; + public List Starts { get; } = []; + public List Events { get; } = []; + public List Tokens { get; } = []; + public bool AlreadyExists { get; set; } + public OrchestrationStatus Status { get; init; } = OrchestrationStatus.Running; + public string Name { get; init; } = nameof(BlobPurgeJobOrchestrator); + + public void Register(IServiceCollection services) => + services.AddDurableTaskClient(this.key, builder => builder.UseOrchestrationService(options => + { + options.Client = this.service.Object; + options.EnableEntitySupport = false; + options.DataConverter = JsonDataConverter.Default; + })); + } +} diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs new file mode 100644 index 00000000..571755b2 --- /dev/null +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs @@ -0,0 +1,264 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using DurableTask.Core; +using DurableTask.Core.Command; +using DurableTask.Core.History; +using Grpc.Core; +using Microsoft.DurableTask.AzureBlobPayloads; +using Microsoft.DurableTask.Client; +using Microsoft.DurableTask.Converters; +using Microsoft.DurableTask.Worker.Shims; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Microsoft.DurableTask.Extensions.AzureBlobPayloads.Tests.AutoPurge; + +public class AlternateHostPurgeTaskTests +{ + [Fact] + public async Task ActualTasks_ExecuteWithDtfXArguments_AndPreserveCorrelationAcrossReplayAsync() + { + // Arrange + const string FirstToken = "opaque:row/one+=="; + const string SecondToken = "opaque:\"row two\""; + List tombstones = + [ + new(FirstToken, "blob:v2:https://account.blob.core.windows.net/payloads/one"), + new(SecondToken, "blob:v2:https://account.blob.core.windows.net/payloads/two"), + ]; + Mock purge = new(MockBehavior.Strict); + DateTime? fetchDeadline = null; + DateTime? reportDeadline = null; + purge.Setup(p => p.GetLargePayloadTombstonesAsync(37, It.IsAny(), default)) + .Callback((_, deadline, _) => fetchDeadline = deadline) + .ReturnsAsync(tombstones); + List? reported = null; + purge.Setup(p => p.ReportLargePayloadPurgeResultsAsync(It.IsAny>(), It.IsAny(), default)) + .Callback, DateTime, CancellationToken>((results, deadline, _) => + { + reported = results.ToList(); + reportDeadline = deadline; + }).Returns(Task.CompletedTask); + Mock store = new(MockBehavior.Strict); + store.Setup(s => s.DeleteAsync(tombstones[0].PayloadToken, It.IsAny())).ReturnsAsync(PayloadDeleteOutcome.Deleted); + store.Setup(s => s.DeleteAsync(tombstones[1].PayloadToken, It.IsAny())).ThrowsAsync(new TimeoutException()); + Driver driver = new(new(37)); + DateTime earliestDeadline = DateTime.UtcNow.AddSeconds(60); + + // Act + ScheduleTaskOrchestratorAction fetch = driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)); + Assert.Equal("[37]", fetch.Input); + await driver.RunActivityAsync(fetch, new GetLargePayloadTombstonesActivity(purge.Object, NullLogger.Instance)); + ScheduleTaskOrchestratorAction delete = driver.SingleActivity(nameof(DeleteExternalBlobActivity)); + Assert.StartsWith("[[", delete.Input); + await driver.RunActivityAsync(delete, new DeleteExternalBlobActivity(store.Object, NullLogger.Instance)); + ScheduleTaskOrchestratorAction report = driver.SingleActivity(nameof(ReportLargePayloadPurgeResultsActivity)); + Assert.StartsWith("[[", report.Input); + await driver.RunActivityAsync(report, new ReportLargePayloadPurgeResultsActivity(purge.Object, NullLogger.Instance)); + + // Assert + Assert.Equal(new[] + { + new LargePayloadPurgeResult(FirstToken, LargePayloadPurgeDisposition.Deleted), + new LargePayloadPurgeResult(SecondToken, LargePayloadPurgeDisposition.Retry), + }, reported); + Assert.Contains("\"PurgedCount\":1", driver.Result.CustomStatus); + Assert.Equal("[37]", driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)).Input); + Assert.All(new[] { fetchDeadline, reportDeadline }, deadline => + { + Assert.NotNull(deadline); + Assert.Equal(DateTimeKind.Utc, deadline.Value.Kind); + Assert.InRange(deadline.Value, earliestDeadline, DateTime.UtcNow.AddSeconds(60)); + }); + Assert.Equal(driver.Snapshot(), driver.ReplaySnapshot()); + store.VerifyAll(); + purge.VerifyAll(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task UnsupportedActivity_FailureDetailsReachOrchestrator_AndEventResumesWithoutRetryAsync(bool reportFailure) + { + // Arrange + Mock purge = new(MockBehavior.Strict); + RpcException unsupported = new(new Status(StatusCode.Unimplemented, "unsupported")); + purge.Setup(p => p.GetLargePayloadTombstonesAsync(It.IsAny(), It.IsAny(), default)) + .ThrowsAsync(unsupported); + purge.Setup(p => p.ReportLargePayloadPurgeResultsAsync(It.IsAny>(), It.IsAny(), default)) + .ThrowsAsync(unsupported); + Driver driver = new(new(20)); + ScheduleTaskOrchestratorAction activity = driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)); + ITaskActivity implementation = new GetLargePayloadTombstonesActivity(purge.Object, NullLogger.Instance); + if (reportFailure) + { + driver.Complete(activity, new[] { new LargePayloadTombstone("correlation", "blob:v2:payload") }); + driver.Complete(driver.SingleActivity(nameof(DeleteExternalBlobActivity)), new[] { new BlobPurgeOutcome(LargePayloadPurgeDisposition.Deleted) }); + activity = driver.SingleActivity(nameof(ReportLargePayloadPurgeResultsActivity)); + implementation = new ReportLargePayloadPurgeResultsActivity(purge.Object, NullLogger.Instance); + } + + // Act + Exception? error = await Record.ExceptionAsync(() => driver.InvokeActivityAsync(activity, implementation)); + NotImplementedException failure = Assert.IsType(error); + driver.Fail(activity, failure); + + // Assert + Assert.Same(unsupported, failure.InnerException); + Assert.Empty(driver.Result.Actions); + Assert.Contains("\"Status\":\"BackendUnsupported\"", driver.Result.CustomStatus); + Assert.Equal(driver.Snapshot(), driver.ReplaySnapshot()); + driver.Turn(Driver.Configure(71)); + Assert.Equal("[71]", driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)).Input); + } + + [Fact] + public async Task TransientActivityFailure_KeepsDtfXRetryTimerAsync() + { + // Arrange + Mock purge = new(MockBehavior.Strict); + RpcException unavailable = new(new Status(StatusCode.Unavailable, "unavailable")); + purge.Setup(p => p.GetLargePayloadTombstonesAsync(It.IsAny(), It.IsAny(), default)).ThrowsAsync(unavailable); + Driver driver = new(new(20)); + ScheduleTaskOrchestratorAction fetch = driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)); + GetLargePayloadTombstonesActivity activity = new(purge.Object, NullLogger.Instance); + + // Act + Exception? error = await Record.ExceptionAsync(() => driver.InvokeActivityAsync(fetch, activity)); + driver.Fail(fetch, Assert.IsType(error)); + + // Assert + CreateTimerOrchestratorAction retry = Assert.IsType(Assert.Single(driver.Result.Actions)); + Assert.Equal(driver.Now.AddSeconds(15), retry.FireAt); + Assert.Equal(driver.Snapshot(), driver.ReplaySnapshot()); + } + + [Fact] + public void ConfigurationAndContinueAsNew_PreserveBufferedEventsAndStateThroughDtfXReplay() + { + // Arrange + Driver driver = new(new(100, 9)); + for (int i = 0; i < 5; i++) + { + driver.Complete(driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)), Array.Empty()); + CreateTimerOrchestratorAction timer = Assert.IsType(Assert.Single(driver.Result.Actions)); + if (i < 4) + { + driver.Turn(Driver.TimerFired(timer)); + } + } + + // Act + driver.Turn(Driver.Configure(700), Driver.Configure(800)); + OrchestrationCompleteOrchestratorAction completed = Assert.IsType(Assert.Single(driver.Result.Actions)); + + // Assert + Assert.Equal(OrchestrationStatus.ContinuedAsNew, completed.OrchestrationStatus); + BlobPurgeJobRunRequest next = JsonDataConverter.Default.Deserialize(completed.Result)!; + Assert.Equal(new BlobPurgeJobRunRequest(700, 9), next); + EventRaisedEvent carried = Assert.IsType(Assert.Single(completed.CarryoverEvents)); + Assert.Equal(BlobPurgeConstants.SetBatchSizeEvent, carried.Name); + Assert.Equal("800", carried.Input); + Assert.Equal(driver.Snapshot(), driver.ReplaySnapshot()); + Driver nextDriver = new(next, carried); + nextDriver.Complete(nextDriver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)), Array.Empty()); + Assert.Equal("[800]", nextDriver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)).Input); + } + + sealed class Driver + { + readonly DurableTaskShimFactory factory = new(); + readonly List history = []; + readonly OrchestrationInstance instance = new() + { + InstanceId = BlobPurgeConstants.OrchestratorInstanceId, + ExecutionId = "alternate-host", + }; + List lastPast = []; + List lastNew = []; + + public Driver(BlobPurgeJobRunRequest input, params HistoryEvent[] events) + { + this.Turn([new ExecutionStartedEvent(-1, JsonDataConverter.Default.Serialize(input)) + { + Name = nameof(BlobPurgeJobOrchestrator), + Version = string.Empty, + OrchestrationInstance = this.instance, + }, .. events]); + } + + public DateTime Now { get; private set; } = new(2026, 9, 1, 0, 0, 0, DateTimeKind.Utc); + public OrchestratorExecutionResult Result { get; private set; } = null!; + + public static EventRaisedEvent Configure(int size) => new(-1, JsonDataConverter.Default.Serialize(size)) { Name = BlobPurgeConstants.SetBatchSizeEvent }; + public static TimerFiredEvent TimerFired(CreateTimerOrchestratorAction timer) => new(-1, timer.FireAt) { TimerId = timer.Id }; + + public ScheduleTaskOrchestratorAction SingleActivity(string name) + { + ScheduleTaskOrchestratorAction task = Assert.IsType(Assert.Single(this.Result.Actions)); + Assert.Equal(name, task.Name); + Assert.Equal(string.Empty, task.Version); + return task; + } + + public Task InvokeActivityAsync(ScheduleTaskOrchestratorAction action, ITaskActivity implementation) => + this.factory.CreateActivity(Assert.IsType(action.Name), implementation).RunAsync( + new TaskContext(this.instance, action.Name, action.Version, action.Id), action.Input); + + public async Task RunActivityAsync(ScheduleTaskOrchestratorAction action, ITaskActivity implementation) => + this.Turn(new TaskCompletedEvent(-1, action.Id, await this.InvokeActivityAsync(action, implementation))); + + public void Complete(ScheduleTaskOrchestratorAction action, object result) => + this.Turn(new TaskCompletedEvent(-1, action.Id, JsonDataConverter.Default.Serialize(result))); + + public void Fail(ScheduleTaskOrchestratorAction action, Exception failure) => + this.Turn(new TaskFailedEvent(-1, action.Id, failure.Message, null, new FailureDetails(failure))); + + public string Snapshot() => Serialize(this.Result); + public string ReplaySnapshot() => Serialize(this.Replay()); + + public void Turn(params HistoryEvent[] events) + { + this.Now = events.OfType().Select(e => e.FireAt).Append(this.Now.AddSeconds(1)).Max(); + this.lastPast = [.. this.history]; + this.lastNew = [new OrchestratorStartedEvent(-1) { Timestamp = this.Now }]; + foreach (HistoryEvent item in events) + { + item.Timestamp = this.Now; + this.lastNew.Add(item); + } + + this.Result = this.Replay(); + this.history.AddRange(this.lastNew); + foreach (OrchestratorAction action in this.Result.Actions) + { + if (action is ScheduleTaskOrchestratorAction task) + { + this.history.Add(new TaskScheduledEvent(task.Id, Assert.IsType(task.Name), task.Version, task.Input) { Timestamp = this.Now }); + } + else if (action is CreateTimerOrchestratorAction timer) + { + this.history.Add(new TimerCreatedEvent(timer.Id, timer.FireAt) { Timestamp = this.Now }); + } + } + + this.history.Add(new OrchestratorCompletedEvent(-1) { Timestamp = this.Now }); + } + + static string Serialize(OrchestratorExecutionResult result) => + Newtonsoft.Json.JsonConvert.SerializeObject(new { result.CustomStatus, Actions = result.Actions.ToArray() }); + + OrchestratorExecutionResult Replay() + { + OrchestrationRuntimeState state = new(this.lastPast); + foreach (HistoryEvent item in this.lastNew) + { + state.AddEvent(item); + } + + TaskOrchestration task = this.factory.CreateOrchestration(nameof(BlobPurgeJobOrchestrator), new BlobPurgeJobOrchestrator()); + TaskOrchestrationExecutor executor = new(state, task, BehaviorOnContinueAsNew.Carryover, ErrorPropagationMode.UseFailureDetails); + return executor.Execute(); + } + } +} diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs index e89637d6..58a2841a 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs @@ -3,6 +3,7 @@ using Google.Protobuf.WellKnownTypes; using Grpc.Core; +using Microsoft.DurableTask.AzureBlobPayloads; using Microsoft.DurableTask.Client; using Microsoft.DurableTask.Client.Grpc; using Microsoft.Extensions.Logging.Abstractions; @@ -19,6 +20,60 @@ public class EternalPurgeClientTests const string RunnerId = "BlobPurgeJob-__dt_blob_payload_autopurge__"; const string RunnerName = "BlobPurgeJobOrchestrator"; + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ExplicitPurgeTransport_UsesSameBootstrapAndDisposesCallsAsync(bool enabled) + { + // Arrange + RecordingInvoker invoker = new(); + await using GrpcDurableTaskClient client = CreateClient(invoker); + GrpcLargePayloadPurgeClient purge = new(new LP.LargePayloadPurge.LargePayloadPurgeClient(invoker)); + + // Act + await client.SetLargePayloadAutoPurgeAsync(purge, enabled, enabled ? 42 : -1); + + // Assert + Assert.Equal(enabled, Assert.Single(invoker.Sets).Enabled); + Assert.Contains("SetLargePayloadAutoPurge", invoker.DisposedCalls); + Assert.Equal(enabled + ? new[] { "SetLargePayloadAutoPurge", "StartInstance", "WaitForInstanceStart", "RaiseEvent" } + : new[] { "SetLargePayloadAutoPurge" }, invoker.Methods); + } + + [Theory] + [InlineData(StatusCode.Cancelled)] + [InlineData(StatusCode.Unimplemented)] + [InlineData(StatusCode.Unavailable)] + public async Task ExplicitPurgeTransport_SettingFailurePreservesContractAsync(StatusCode status) + { + // Arrange + RecordingInvoker invoker = new() { ErrorAt = "SetLargePayloadAutoPurge", ErrorStatus = status }; + await using GrpcDurableTaskClient client = CreateClient(invoker); + GrpcLargePayloadPurgeClient purge = new(new LP.LargePayloadPurge.LargePayloadPurgeClient(invoker)); + using CancellationTokenSource cancellation = new(); + + // Act + Exception? failure = await Record.ExceptionAsync(() => + client.SetLargePayloadAutoPurgeAsync(purge, true, cancellationToken: cancellation.Token)); + + // Assert + if (status == StatusCode.Cancelled) + { + Assert.Equal(cancellation.Token, Assert.IsType(failure).CancellationToken); + } + else if (status == StatusCode.Unimplemented) + { + Assert.IsType(failure); + } + else + { + Assert.Equal(status, Assert.IsType(failure).StatusCode); + } + Assert.Equal(new[] { "SetLargePayloadAutoPurge" }, invoker.Methods); + Assert.Equal(new[] { "SetLargePayloadAutoPurge" }, invoker.DisposedCalls); + } + [Fact] public async Task Disable_OnlyWritesBackendSettingAsync() { @@ -354,6 +409,7 @@ static GrpcDurableTaskClient CreateClient(RecordingInvoker invoker) => sealed class RecordingInvoker : CallInvoker { public List Methods { get; } = []; + public List DisposedCalls { get; } = []; public List Sets { get; } = []; public List Starts { get; } = []; public List Events { get; } = []; @@ -401,7 +457,8 @@ public override AsyncUnaryCall AsyncUnaryCall( { result = Task.FromException(new RpcException(new Status(this.ErrorStatus, "synthetic failure"))); } - return new(result, Task.FromResult(new Metadata()), () => Status.DefaultSuccess, () => new Metadata(), () => { }); + return new(result, Task.FromResult(new Metadata()), () => Status.DefaultSuccess, () => new Metadata(), + () => this.DisposedCalls.Add(method.Name)); } static async Task ConvertAsync(Task task) => (T)(object)await task; diff --git a/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj b/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj index 6accf793..e6d3da2e 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj +++ b/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj @@ -13,6 +13,7 @@ + From e17066d92eae4f860f5c4a46ba14e926443d49ec Mon Sep 17 00:00:00 2001 From: wangbill Date: Tue, 22 Sep 2026 16:26:53 -0700 Subject: [PATCH 02/15] Limit purge integration surface to worker activity transport Reuse the existing native gRPC client for setup instead of adding an alternate bootstrap overload. Remove the unused setting facade and client shim tests while retaining actual DTFx purge task execution coverage. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 17 +- ...eTaskClientExtensions.AzureBlobPayloads.cs | 48 +---- .../Client/GrpcLargePayloadPurgeClient.cs | 21 -- .../Client/ILargePayloadPurgeClient.cs | 13 -- .../AlternateHostPurgeClientTests.cs | 186 ------------------ .../AutoPurge/EternalPurgeClientTests.cs | 59 +----- .../AzureBlobPayloads.Tests.csproj | 1 - 7 files changed, 10 insertions(+), 335 deletions(-) delete mode 100644 test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs diff --git a/README.md b/README.md index c04bfda2..a0f8aa6d 100644 --- a/README.md +++ b/README.md @@ -216,16 +216,15 @@ use that hub's existing authenticated transport, follow reconnection and credent activities' UTC deadlines, and preserve opaque tombstone tokens. Fetch/report gRPC failures are classified by the activities, not swallowed by the adapter. The host retains ownership of its clients and store. -For explicit setup, call `orchestrationClient.SetLargePayloadAutoPurgeAsync(purgeClient, enabled, batchSize, -cancellationToken)`. Both clients must target the **same authenticated task hub**. A host with an existing -`IOrchestrationServiceClient` can obtain the orchestration client through -`AddDurableTaskClient(name).UseOrchestrationService(options => { options.Client = serviceClient; -options.EnableEntitySupport = false; })` from `Microsoft.DurableTask.Client.OrchestrationServiceClientShim`. -Use the SDK's default data converter consistently for this client and the task shims. DTFx wraps activity -arguments in an outer JSON array; the shims handle this envelope. Preserve structured activity failure details +For explicit setup, reuse `GrpcDurableTaskClient.SetLargePayloadAutoPurgeAsync(enabled, batchSize, +cancellationToken)`. Construct the client with `GrpcDurableTaskClientOptions.CallInvoker` set to the host's +existing authenticated remote task-hub invoker, and `EnableEntitySupport = false`. Supplying this invoker +opens no additional channel and does not transfer its ownership to the SDK client. The host must keep the +invoker bound to the **same authenticated task hub** as the activities and follow any channel replacement. +Do not route these calls to a language worker's local RPC endpoint. +Use the SDK's default data converter consistently for the client and task shims. DTFx wraps activity arguments +in an outer JSON array; the shims handle this envelope. Preserve structured activity failure details (`ErrorPropagationMode.UseFailureDetails`) and unprocessed external events across continue-as-new. -Cancellation remains subject to the supplied client's capabilities; an in-flight service call without -cancellation support cannot be canceled by this helper. Enabling writes the setting, starts the reserved per-task-hub instance with live-status deduplication, verifies the runner's identity and Running status, then sends `SetBatchSize`. Disabling **only** writes the diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs index f583fb67..4f077aed 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/DurableTaskClientExtensions.AzureBlobPayloads.cs @@ -69,53 +69,7 @@ public static async Task SetLargePayloadAutoPurgeAsync( throw new NotSupportedException($"Large-payload auto-purge requires a gRPC Durable Task client, not '{client.GetType().FullName}'."); } - await SetLargePayloadAutoPurgeCoreAsync( - client, autoPurgeClient.SetLargePayloadAutoPurgeAsync, enabled, batchSize, cancellationToken); - } - - /// - /// Applies explicit auto-purge configuration using an alternate host's task-hub-bound clients. - /// - /// The orchestration client targeting the same authenticated task hub as . - /// The host-owned transport for the task hub's auto-purge setting. - /// True to enable cleanup; false to pause new fetches through the backend setting. - /// The requested batch size, from 1 through 1000. Ignored when disabling. - /// Cancels any setting, start, wait or configuration operation. - /// A task that completes after the setting and, when enabling, start verification and event enqueue. - /// - /// Infrastructure integration API, not an application orchestration API. The caller must bind both clients - /// to the same task hub; neither client is disposed by this method. This uses the same nontransactional - /// setting, fixed-ID deduplicated start, Running identity verification and batch-size event sequence as - /// the standalone overload. Disabling only writes the setting. Repeating desired state after host takeover - /// is supported; this does not implement owner election or guarantee once-ever setup. - /// The host must register the SDK purge tasks before enabling and retain their canonical names and inputs. - /// - public static async Task SetLargePayloadAutoPurgeAsync( - this DurableTaskClient client, - ILargePayloadPurgeClient purgeClient, - bool enabled, - int batchSize = BlobPurgeConstants.DefaultBatchSize, - CancellationToken cancellationToken = default) - { - Check.NotNull(client); - Check.NotNull(purgeClient); - if (enabled && (batchSize < 1 || batchSize > BlobPurgeConstants.MaxBatchSize)) - { - throw new ArgumentOutOfRangeException(nameof(batchSize), batchSize, "Purge batch size is out of range."); - } - - await SetLargePayloadAutoPurgeCoreAsync( - client, purgeClient.SetLargePayloadAutoPurgeAsync, enabled, batchSize, cancellationToken); - } - - static async Task SetLargePayloadAutoPurgeCoreAsync( - DurableTaskClient client, - Func setEnabled, - bool enabled, - int batchSize, - CancellationToken cancellationToken) - { - await setEnabled(enabled, cancellationToken); + await autoPurgeClient.SetLargePayloadAutoPurgeAsync(enabled, cancellationToken); if (!enabled) { return; diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs index 1cda3265..a4311e96 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs @@ -1,7 +1,6 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. -using Grpc.Core; using Microsoft.DurableTask.Client; using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; using LP = Microsoft.DurableTask.Protobuf.LargePayloads; @@ -15,26 +14,6 @@ sealed class GrpcLargePayloadPurgeClient(LargePayloadPurgeClient client) : ILarg { readonly LargePayloadPurgeClient client = Check.NotNull(client); - /// - public async Task SetLargePayloadAutoPurgeAsync(bool enabled, CancellationToken cancellationToken = default) - { - try - { - using var call = this.client.SetLargePayloadAutoPurgeAsync( - new LP.SetLargePayloadAutoPurgeRequest { Enabled = enabled }, cancellationToken: cancellationToken); - await call; - } - catch (RpcException e) when (e.StatusCode == StatusCode.Cancelled) - { - throw new OperationCanceledException( - "The SetLargePayloadAutoPurge operation was canceled.", e, cancellationToken); - } - catch (RpcException e) when (e.StatusCode == StatusCode.Unimplemented) - { - throw new NotImplementedException(e.Status.Detail); - } - } - /// public async Task> GetLargePayloadTombstonesAsync( int limit, DateTime deadline, CancellationToken cancellationToken = default) diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs index b5f2ea66..a545401c 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs @@ -18,19 +18,6 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// public interface ILargePayloadPurgeClient { - /// - /// Writes the auto-purge setting for this client's authenticated task hub. - /// - /// The setting to persist. - /// Cancels the setting operation. - /// A task that completes when the backend acknowledges the setting. - /// - /// Map backend cancellation to with the supplied token and an - /// unsupported operation to . Other failures must propagate. - /// This operation alone must not start, terminate or wait for an orchestration. - /// - Task SetLargePayloadAutoPurgeAsync(bool enabled, CancellationToken cancellationToken = default); - /// /// Fetches a bounded batch of due tombstones for this client's authenticated task hub. /// diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs deleted file mode 100644 index d63772ef..00000000 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeClientTests.cs +++ /dev/null @@ -1,186 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -using DurableTask.Core; -using DurableTask.Core.Exceptions; -using DurableTask.Core.History; -using Microsoft.DurableTask.AzureBlobPayloads; -using Microsoft.DurableTask.Client; -using Microsoft.DurableTask.Converters; -using Microsoft.Extensions.DependencyInjection; - -namespace Microsoft.DurableTask.Extensions.AzureBlobPayloads.Tests.AutoPurge; - -public class AlternateHostPurgeClientTests -{ - [Fact] - public async Task Enable_UsesExistingServiceClient_AndRepeatedSetupIsIndependentPerHubAsync() - { - // Arrange - Hub first = new("first"); - Hub second = new("second"); - ServiceCollection services = new(); - first.Register(services); - second.Register(services); - await using ServiceProvider provider = services.BuildServiceProvider(); - IDurableTaskClientProvider clients = provider.GetRequiredService(); - using CancellationTokenSource cancellation = new(); - - // Act - await clients.GetClient("first").SetLargePayloadAutoPurgeAsync(first.Purge.Object, true, 123, cancellation.Token); - await clients.GetClient("second").SetLargePayloadAutoPurgeAsync(second.Purge.Object, true, 456, cancellation.Token); - first.AlreadyExists = true; - await clients.GetClient("first").SetLargePayloadAutoPurgeAsync(first.Purge.Object, true, 789, cancellation.Token); - - // Assert - Assert.Equal(new[] { "Set:True", "Start", "Wait", "Event", "Set:True", "Start", "Wait", "Event" }, first.Calls); - Assert.Equal(new[] { "Set:True", "Start", "Wait", "Event" }, second.Calls); - Assert.Equal(new[] { "123", "789" }, first.Events.Select(e => e.Input)); - Assert.Equal("456", Assert.Single(second.Events).Input); - Assert.All(first.Tokens.Concat(second.Tokens), token => Assert.Equal(cancellation.Token, token)); - Assert.All(first.Starts.Concat(second.Starts), start => - { - Assert.Equal(nameof(BlobPurgeJobOrchestrator), start.Name); - Assert.Equal(BlobPurgeConstants.OrchestratorInstanceId, start.OrchestrationInstance.InstanceId); - Assert.Equal(string.Empty, start.Version); - Assert.InRange(JsonDataConverter.Default.Deserialize(start.Input)!.PurgeBatchSize, 1, 1000); - }); - } - - [Fact] - public async Task Disable_OnlyWritesSetting_AndStandaloneOverloadStillRejectsShimAsync() - { - // Arrange - Hub hub = new("hub"); - ServiceCollection services = new(); - hub.Register(services); - await using ServiceProvider provider = services.BuildServiceProvider(); - DurableTaskClient client = provider.GetRequiredService().GetClient("hub"); - - // Act - await client.SetLargePayloadAutoPurgeAsync(hub.Purge.Object, false, -1); - - // Assert - Assert.Equal(new[] { "Set:False" }, hub.Calls); - await Assert.ThrowsAsync(() => client.SetLargePayloadAutoPurgeAsync(true)); - Assert.Equal(new[] { "Set:False" }, hub.Calls); - } - - [Theory] - [InlineData(OrchestrationStatus.Suspended, nameof(BlobPurgeJobOrchestrator))] - [InlineData(OrchestrationStatus.Pending, nameof(BlobPurgeJobOrchestrator))] - [InlineData(OrchestrationStatus.ContinuedAsNew, nameof(BlobPurgeJobOrchestrator))] - [InlineData(OrchestrationStatus.Running, "BusinessOrchestration")] - public async Task Enable_LiveCollisionOrNonRunningState_DoesNotSendEventAsync(OrchestrationStatus status, string name) - { - // Arrange - Hub hub = new("hub") { AlreadyExists = true, Status = status, Name = name }; - ServiceCollection services = new(); - hub.Register(services); - await using ServiceProvider provider = services.BuildServiceProvider(); - DurableTaskClient client = provider.GetRequiredService().GetClient("hub"); - - // Act / Assert - // The real shim waits on Pending, so cancellation bounds that case rather than changing its semantics. - using CancellationTokenSource cancellation = new(TimeSpan.FromMilliseconds(100)); - Exception? error = await Record.ExceptionAsync(() => - client.SetLargePayloadAutoPurgeAsync(hub.Purge.Object, true, cancellationToken: cancellation.Token)); - Assert.NotNull(error); - Assert.True(error is InvalidOperationException or OperationCanceledException, error.ToString()); - Assert.Empty(hub.Events); - } - - [Theory] - [InlineData(0)] - [InlineData(1001)] - public async Task InvalidBatch_DoesNotWriteSettingAsync(int batchSize) - { - // Arrange - Mock client = new(MockBehavior.Strict, "hub"); - Mock purge = new(MockBehavior.Strict); - - // Act / Assert - await Assert.ThrowsAsync(() => - client.Object.SetLargePayloadAutoPurgeAsync(purge.Object, true, batchSize)); - client.VerifyNoOtherCalls(); - purge.VerifyNoOtherCalls(); - } - - [Fact] - public async Task SettingFailure_PropagatesWithoutStartingAsync() - { - // Arrange - Mock client = new(MockBehavior.Strict, "hub"); - Mock purge = new(MockBehavior.Strict); - using CancellationTokenSource cancellation = new(); - OperationCanceledException failure = new(cancellation.Token); - purge.Setup(p => p.SetLargePayloadAutoPurgeAsync(true, cancellation.Token)).ThrowsAsync(failure); - - // Act / Assert - Assert.Same(failure, await Assert.ThrowsAsync(() => - client.Object.SetLargePayloadAutoPurgeAsync(purge.Object, true, cancellationToken: cancellation.Token))); - client.VerifyNoOtherCalls(); - } - - sealed class Hub - { - readonly string key; - readonly Mock service = new(MockBehavior.Strict); - - public Hub(string key) - { - this.key = key; - this.Purge.Setup(p => p.SetLargePayloadAutoPurgeAsync(It.IsAny(), It.IsAny())) - .Callback((enabled, token) => - { - this.Calls.Add($"Set:{enabled}"); - this.Tokens.Add(token); - }).Returns(Task.CompletedTask); - this.service.Setup(s => s.CreateTaskOrchestrationAsync(It.IsAny(), It.IsAny())) - .Returns((TaskMessage message, OrchestrationStatus[] statuses) => - { - this.Calls.Add("Start"); - Assert.Equal(new[] { OrchestrationStatus.Running, OrchestrationStatus.Pending, OrchestrationStatus.Suspended, OrchestrationStatus.ContinuedAsNew }, statuses); - this.Starts.Add(Assert.IsType(message.Event)); - return this.AlreadyExists ? Task.FromException(new OrchestrationAlreadyExistsException("existing")) : Task.CompletedTask; - }); - this.service.Setup(s => s.GetOrchestrationStateAsync(BlobPurgeConstants.OrchestratorInstanceId, false)) - .Returns(() => - { - this.Calls.Add("Wait"); - return Task.FromResult>([new() - { - Name = this.Name, - OrchestrationStatus = this.Status, - OrchestrationInstance = new() { InstanceId = BlobPurgeConstants.OrchestratorInstanceId }, - }]); - }); - this.service.Setup(s => s.SendTaskOrchestrationMessageAsync(It.IsAny())) - .Callback(message => - { - this.Calls.Add("Event"); - Assert.Equal(BlobPurgeConstants.OrchestratorInstanceId, message.OrchestrationInstance.InstanceId); - EventRaisedEvent raised = Assert.IsType(message.Event); - Assert.Equal(BlobPurgeConstants.SetBatchSizeEvent, raised.Name); - this.Events.Add(raised); - }).Returns(Task.CompletedTask); - } - - public Mock Purge { get; } = new(MockBehavior.Strict); - public List Calls { get; } = []; - public List Starts { get; } = []; - public List Events { get; } = []; - public List Tokens { get; } = []; - public bool AlreadyExists { get; set; } - public OrchestrationStatus Status { get; init; } = OrchestrationStatus.Running; - public string Name { get; init; } = nameof(BlobPurgeJobOrchestrator); - - public void Register(IServiceCollection services) => - services.AddDurableTaskClient(this.key, builder => builder.UseOrchestrationService(options => - { - options.Client = this.service.Object; - options.EnableEntitySupport = false; - options.DataConverter = JsonDataConverter.Default; - })); - } -} diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs index 58a2841a..e89637d6 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/EternalPurgeClientTests.cs @@ -3,7 +3,6 @@ using Google.Protobuf.WellKnownTypes; using Grpc.Core; -using Microsoft.DurableTask.AzureBlobPayloads; using Microsoft.DurableTask.Client; using Microsoft.DurableTask.Client.Grpc; using Microsoft.Extensions.Logging.Abstractions; @@ -20,60 +19,6 @@ public class EternalPurgeClientTests const string RunnerId = "BlobPurgeJob-__dt_blob_payload_autopurge__"; const string RunnerName = "BlobPurgeJobOrchestrator"; - [Theory] - [InlineData(false)] - [InlineData(true)] - public async Task ExplicitPurgeTransport_UsesSameBootstrapAndDisposesCallsAsync(bool enabled) - { - // Arrange - RecordingInvoker invoker = new(); - await using GrpcDurableTaskClient client = CreateClient(invoker); - GrpcLargePayloadPurgeClient purge = new(new LP.LargePayloadPurge.LargePayloadPurgeClient(invoker)); - - // Act - await client.SetLargePayloadAutoPurgeAsync(purge, enabled, enabled ? 42 : -1); - - // Assert - Assert.Equal(enabled, Assert.Single(invoker.Sets).Enabled); - Assert.Contains("SetLargePayloadAutoPurge", invoker.DisposedCalls); - Assert.Equal(enabled - ? new[] { "SetLargePayloadAutoPurge", "StartInstance", "WaitForInstanceStart", "RaiseEvent" } - : new[] { "SetLargePayloadAutoPurge" }, invoker.Methods); - } - - [Theory] - [InlineData(StatusCode.Cancelled)] - [InlineData(StatusCode.Unimplemented)] - [InlineData(StatusCode.Unavailable)] - public async Task ExplicitPurgeTransport_SettingFailurePreservesContractAsync(StatusCode status) - { - // Arrange - RecordingInvoker invoker = new() { ErrorAt = "SetLargePayloadAutoPurge", ErrorStatus = status }; - await using GrpcDurableTaskClient client = CreateClient(invoker); - GrpcLargePayloadPurgeClient purge = new(new LP.LargePayloadPurge.LargePayloadPurgeClient(invoker)); - using CancellationTokenSource cancellation = new(); - - // Act - Exception? failure = await Record.ExceptionAsync(() => - client.SetLargePayloadAutoPurgeAsync(purge, true, cancellationToken: cancellation.Token)); - - // Assert - if (status == StatusCode.Cancelled) - { - Assert.Equal(cancellation.Token, Assert.IsType(failure).CancellationToken); - } - else if (status == StatusCode.Unimplemented) - { - Assert.IsType(failure); - } - else - { - Assert.Equal(status, Assert.IsType(failure).StatusCode); - } - Assert.Equal(new[] { "SetLargePayloadAutoPurge" }, invoker.Methods); - Assert.Equal(new[] { "SetLargePayloadAutoPurge" }, invoker.DisposedCalls); - } - [Fact] public async Task Disable_OnlyWritesBackendSettingAsync() { @@ -409,7 +354,6 @@ static GrpcDurableTaskClient CreateClient(RecordingInvoker invoker) => sealed class RecordingInvoker : CallInvoker { public List Methods { get; } = []; - public List DisposedCalls { get; } = []; public List Sets { get; } = []; public List Starts { get; } = []; public List Events { get; } = []; @@ -457,8 +401,7 @@ public override AsyncUnaryCall AsyncUnaryCall( { result = Task.FromException(new RpcException(new Status(this.ErrorStatus, "synthetic failure"))); } - return new(result, Task.FromResult(new Metadata()), () => Status.DefaultSuccess, () => new Metadata(), - () => this.DisposedCalls.Add(method.Name)); + return new(result, Task.FromResult(new Metadata()), () => Status.DefaultSuccess, () => new Metadata(), () => { }); } static async Task ConvertAsync(Task task) => (T)(object)await task; diff --git a/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj b/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj index e6d3da2e..6accf793 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj +++ b/test/Extensions/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj @@ -13,7 +13,6 @@ - From 8a6a3a99385964556eb837092bc20f4eb6ef12f8 Mon Sep 17 00:00:00 2001 From: wangbill Date: Wed, 23 Sep 2026 10:01:56 -0700 Subject: [PATCH 03/15] Clarify isolated Functions blob purge integration Document normal function indexing, language-worker task execution, and the bound local-host transport. Preserve the existing SDK implementation and explicit bootstrap without claiming Functions end-to-end validation. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 70 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index a0f8aa6d..d2090954 100644 --- a/README.md +++ b/README.md @@ -200,38 +200,44 @@ For runnable DTS emulator examples that demonstrate versioning, see the [WorkerV The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) shows how to declare selected activities for Durable Task Scheduler (DTS)-managed on-demand sandbox execution and build the remote worker container image separately from the declarer app. -### Blob auto-purge integration for alternate .NET hosts - -`Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes infrastructure APIs for hosts that already -dispatch Durable Task Framework (`DurableTask.Core`) tasks. These APIs do not add another worker or require -application functions. Register `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, -`DeleteExternalBlobActivity`, and `ReportLargePayloadPurgeResultsActivity` under their exact class names -with an empty version, using `Microsoft.DurableTask.Worker.Shims.DurableTaskShimFactory.CreateOrchestration` -and `CreateActivity`. Keep these tasks registered even when auto-purge is disabled so existing work can finish. -This does not promise compatibility with arbitrary historical versioned purge runners. - -Construct the fetch and report activities with a host-owned `ILargePayloadPurgeClient` and their typed -loggers, and the delete activity with the task hub's `PayloadStore` and logger. The narrow purge client must -use that hub's existing authenticated transport, follow reconnection and credential refresh, honor the -activities' UTC deadlines, and preserve opaque tombstone tokens. Fetch/report gRPC failures are classified -by the activities, not swallowed by the adapter. The host retains ownership of its clients and store. - -For explicit setup, reuse `GrpcDurableTaskClient.SetLargePayloadAutoPurgeAsync(enabled, batchSize, -cancellationToken)`. Construct the client with `GrpcDurableTaskClientOptions.CallInvoker` set to the host's -existing authenticated remote task-hub invoker, and `EnableEntitySupport = false`. Supplying this invoker -opens no additional channel and does not transfer its ownership to the SDK client. The host must keep the -invoker bound to the **same authenticated task hub** as the activities and follow any channel replacement. -Do not route these calls to a language worker's local RPC endpoint. -Use the SDK's default data converter consistently for the client and task shims. DTFx wraps activity arguments -in an outer JSON array; the shims handle this envelope. Preserve structured activity failure details -(`ErrorPropagationMode.UseFailureDetails`) and unprocessed external events across continue-as-new. - -Enabling writes the setting, starts the reserved per-task-hub instance with live-status deduplication, -verifies the runner's identity and Running status, then sends `SetBatchSize`. Disabling **only** writes the -setting and ignores batch size. The steps are not transactional; failures propagate without rollback. -Repeated desired-state setup after host takeover is supported. Owner election and deciding whether an omitted -setting requires any action belong to the host, not this SDK. `BlobPurgeConstants` provides the reserved -instance ID, event name and batch bounds; never use the reserved instance ID for application work. +### Blob auto-purge infrastructure integration + +`Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes reusable orchestration and activity +implementations: `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, `DeleteExternalBlobActivity`, +and `ReportLargePayloadPurgeResultsActivity`. Preserve their exact task names, empty version, input/output +types, and retry/event/continue-as-new behavior. Keep the tasks registered even when auto-purge is disabled +so existing work can finish. `BlobPurgeConstants` provides the reserved per-task-hub instance ID, +configuration event name, and batch bounds; never use that instance ID for application work. + +For a **.NET isolated Durable Functions integration**, the Functions worker extension must expose these tasks +as ordinary orchestration/activity functions before indexing and execute them through normal language-worker +bindings and invocation. Merely referencing their assembly does not generate Functions wrappers: the SDK +source generator discovers `[DurableTask]` classes in the current project's source, not referenced libraries. +Delegate orchestration execution to the existing task with the bound `TaskOrchestrationContext` and its input. +Activity execution must receive a real `TaskActivityContext` carrying the canonical activity name and the +invoking orchestration's bound instance ID, not a null context or the Functions invocation ID. Use the normal +worker serialization and failure propagation paths, preserving structured failure details and unprocessed +events across continue-as-new. Do not substitute host-side `TaskHubWorker` registration or start another worker. + +Construct the fetch/report activities with an `ILargePayloadPurgeClient` and their typed loggers, and the +delete activity with the worker's configured `PayloadStore` and logger. Reuse `BlobPayloadStore` with +`LargePayloadStorageOptions` for storage access rather than copying its ownership checks or deletion policy. +Deletion runs in the language worker; purge RPCs do not carry storage credentials. The narrow purge client must honor UTC deadlines and +preserve opaque tombstone tokens; the activities classify fetch/report gRPC failures. + +The Functions integration must route setting, fetch, and report operations through the bound client's local +host endpoint to the provider's authenticated transport for the **same task hub**. The existing +`LargePayloadPurge` gRPC service is separate from `TaskHubSidecarService`. The Functions client wrapper can +forward the setting through the existing infrastructure `ILargePayloadAutoPurgeClient` interface so the +original `client.SetLargePayloadAutoPurgeAsync(enabled, batchSize, cancellationToken)` extension retains +ownership of bootstrap behavior. The integration owns client/store lifetimes, hub binding, and reconnection; +this SDK surface alone does not supply Functions metadata or the local-host bridge. + +Enabling explicitly writes the setting, starts the reserved instance with live-status deduplication and an +empty version, verifies its identity and Running status, then sends `SetBatchSize`. Disabling **only** writes +the setting and ignores batch size. Calling neither leaves the setting untouched. No automatic host +configuration or primary-host monitor is required. These steps are not transactional; failures propagate +without rollback. Existing standalone gRPC client and worker behavior is unchanged. ## Obtaining the Protobuf definitions From 7db4b8af2ec3737c1dc96571b5d319c8d707ec0e Mon Sep 17 00:00:00 2001 From: wangbill Date: Wed, 23 Sep 2026 10:30:57 -0700 Subject: [PATCH 04/15] Document optional isolated blob purge function package Clarify normal compiled-library Function indexing and opt-in package registration without a metadata transformer or generator. Preserve the base extension behavior and unchanged shared SDK tasks. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index d2090954..67486f0b 100644 --- a/README.md +++ b/README.md @@ -209,10 +209,17 @@ types, and retry/event/continue-as-new behavior. Keep the tasks registered even so existing work can finish. `BlobPurgeConstants` provides the reserved per-task-hub instance ID, configuration event name, and batch bounds; never use that instance ID for application work. -For a **.NET isolated Durable Functions integration**, the Functions worker extension must expose these tasks -as ordinary orchestration/activity functions before indexing and execute them through normal language-worker -bindings and invocation. Merely referencing their assembly does not generate Functions wrappers: the SDK -source generator discovers `[DurableTask]` classes in the current project's source, not referenced libraries. +The **.NET isolated Durable Functions integration** belongs in the optional companion package +`Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureBlobPayloads`. That package carries four +ordinary `[Function]` methods delegating to these tasks, together with worker-side payload-store configuration. +The base Functions worker extension does not carry these functions, so existing applications do not acquire +purge functions merely by updating the base extension. + +The Functions Worker SDK indexes the optional package's compiled `[Function]` methods through its normal +library indexing path. No custom metadata add/remove transformer or generator is required. This is distinct +from generating wrappers for library task classes: the Durable Task source generator discovers +`[DurableTask]` classes in the current project's source, not referenced libraries. Invoke the packaged +functions through ordinary trigger and `DurableClient` bindings in the .NET isolated language worker. Delegate orchestration execution to the existing task with the bound `TaskOrchestrationContext` and its input. Activity execution must receive a real `TaskActivityContext` carrying the canonical activity name and the invoking orchestration's bound instance ID, not a null context or the Functions invocation ID. Use the normal @@ -222,8 +229,8 @@ events across continue-as-new. Do not substitute host-side `TaskHubWorker` regis Construct the fetch/report activities with an `ILargePayloadPurgeClient` and their typed loggers, and the delete activity with the worker's configured `PayloadStore` and logger. Reuse `BlobPayloadStore` with `LargePayloadStorageOptions` for storage access rather than copying its ownership checks or deletion policy. -Deletion runs in the language worker; purge RPCs do not carry storage credentials. The narrow purge client must honor UTC deadlines and -preserve opaque tombstone tokens; the activities classify fetch/report gRPC failures. +Deletion runs in the language worker; purge RPCs do not carry storage credentials. The narrow purge client +must honor UTC deadlines and preserve opaque tombstone tokens; the activities classify fetch/report gRPC failures. The Functions integration must route setting, fetch, and report operations through the bound client's local host endpoint to the provider's authenticated transport for the **same task hub**. The existing From cfe25f47aab323f5b5de7efa483d7feaedf653e4 Mon Sep 17 00:00:00 2001 From: wangbill Date: Wed, 23 Sep 2026 10:32:02 -0700 Subject: [PATCH 05/15] Keep Functions purge registration guidance implementation neutral Describe explicit registration, normal function metadata and bindings without naming an unreleased package or prescribing its metadata implementation. Leave shared task APIs and bootstrap unchanged. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 67486f0b..6fe98e39 100644 --- a/README.md +++ b/README.md @@ -209,17 +209,12 @@ types, and retry/event/continue-as-new behavior. Keep the tasks registered even so existing work can finish. `BlobPurgeConstants` provides the reserved per-task-hub instance ID, configuration event name, and batch bounds; never use that instance ID for application work. -The **.NET isolated Durable Functions integration** belongs in the optional companion package -`Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureBlobPayloads`. That package carries four -ordinary `[Function]` methods delegating to these tasks, together with worker-side payload-store configuration. -The base Functions worker extension does not carry these functions, so existing applications do not acquire -purge functions merely by updating the base extension. - -The Functions Worker SDK indexes the optional package's compiled `[Function]` methods through its normal -library indexing path. No custom metadata add/remove transformer or generator is required. This is distinct -from generating wrappers for library task classes: the Durable Task source generator discovers -`[DurableTask]` classes in the current project's source, not referenced libraries. Invoke the packaged -functions through ordinary trigger and `DurableClient` bindings in the .NET isolated language worker. +For **.NET isolated Durable Functions**, explicitly register the purge functions and configure worker-side +payload storage through the Functions integration. Registration must provide ordinary orchestration/activity +function metadata before indexing; applications that do not opt in must not acquire purge functions. +Execution uses normal trigger and `DurableClient` bindings in the .NET isolated language worker, with +function entry points delegating to the shared SDK tasks. No new Durable Task source generator is needed: +the existing generator discovers source-defined `[DurableTask]` classes, not referenced library task classes. Delegate orchestration execution to the existing task with the bound `TaskOrchestrationContext` and its input. Activity execution must receive a real `TaskActivityContext` carrying the canonical activity name and the invoking orchestration's bound instance ID, not a null context or the Functions invocation ID. Use the normal From eb37b20146a04ac9f02cb7d47ecd145ce253d906 Mon Sep 17 00:00:00 2001 From: wangbill Date: Wed, 23 Sep 2026 11:09:31 -0700 Subject: [PATCH 06/15] Document verified isolated Functions purge package integration Describe the optional feature package's normal build-time Function discovery and generated invocation. Preserve shared SDK task APIs, explicit bootstrap behavior, and standalone operation. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 6fe98e39..3d41f122 100644 --- a/README.md +++ b/README.md @@ -209,12 +209,20 @@ types, and retry/event/continue-as-new behavior. Keep the tasks registered even so existing work can finish. `BlobPurgeConstants` provides the reserved per-task-hub instance ID, configuration event name, and batch bounds; never use that instance ID for application work. -For **.NET isolated Durable Functions**, explicitly register the purge functions and configure worker-side -payload storage through the Functions integration. Registration must provide ordinary orchestration/activity -function metadata before indexing; applications that do not opt in must not acquire purge functions. -Execution uses normal trigger and `DurableClient` bindings in the .NET isolated language worker, with -function entry points delegating to the shared SDK tasks. No new Durable Task source generator is needed: -the existing generator discovers source-defined `[DurableTask]` classes, not referenced library task classes. +The companion **.NET isolated Durable Functions** integration uses the optional +`Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureBlobPayloads` package. It supplies four ordinary +`[Function]` methods that delegate to the shared tasks, plus worker-side payload-store configuration. +The base Functions worker extension does not carry these function definitions. Referencing only this shared +SDK package does not register Functions or enable auto-purge. + +The Functions Worker SDK discovers the optional package's compiled `[Function]` methods during the normal +build, produces their function metadata and generated invocation paths, and executes them with ordinary +trigger and `DurableClient` bindings in the isolated language worker. Runtime metadata alone is insufficient +when the application's generated executor has no corresponding method calls. No new Durable Task source +generator, custom executor branch, or host-side task registration is needed. The Durable Task source +generator's discovery of source-defined `[DurableTask]` classes is separate from this compiled-function path. + +Configure payload storage in the worker and explicitly enable or disable cleanup through the bound client. Delegate orchestration execution to the existing task with the bound `TaskOrchestrationContext` and its input. Activity execution must receive a real `TaskActivityContext` carrying the canonical activity name and the invoking orchestration's bound instance ID, not a null context or the Functions invocation ID. Use the normal From f5c724b1e6b6f08e7d6833abe6160b886a2260cd Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 24 Sep 2026 09:27:04 -0700 Subject: [PATCH 07/15] Keep unused blob purge constants internal Restore the constants file to its upstream scope after verifying final consumers use only the public activities and transport facade. Trim integration documentation to the current isolated Functions path. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 30 +++++++------------ .../AutoPurge/Constants/BlobPurgeConstants.cs | 8 ++--- 2 files changed, 13 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 3d41f122..cc25f274 100644 --- a/README.md +++ b/README.md @@ -206,8 +206,8 @@ The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) s implementations: `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, `DeleteExternalBlobActivity`, and `ReportLargePayloadPurgeResultsActivity`. Preserve their exact task names, empty version, input/output types, and retry/event/continue-as-new behavior. Keep the tasks registered even when auto-purge is disabled -so existing work can finish. `BlobPurgeConstants` provides the reserved per-task-hub instance ID, -configuration event name, and batch bounds; never use that instance ID for application work. +so existing work can finish. The existing client API manages the reserved per-task-hub orchestration instance +and its configuration. The companion **.NET isolated Durable Functions** integration uses the optional `Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureBlobPayloads` package. It supplies four ordinary @@ -215,19 +215,11 @@ The companion **.NET isolated Durable Functions** integration uses the optional The base Functions worker extension does not carry these function definitions. Referencing only this shared SDK package does not register Functions or enable auto-purge. -The Functions Worker SDK discovers the optional package's compiled `[Function]` methods during the normal -build, produces their function metadata and generated invocation paths, and executes them with ordinary -trigger and `DurableClient` bindings in the isolated language worker. Runtime metadata alone is insufficient -when the application's generated executor has no corresponding method calls. No new Durable Task source -generator, custom executor branch, or host-side task registration is needed. The Durable Task source -generator's discovery of source-defined `[DurableTask]` classes is separate from this compiled-function path. - -Configure payload storage in the worker and explicitly enable or disable cleanup through the bound client. -Delegate orchestration execution to the existing task with the bound `TaskOrchestrationContext` and its input. -Activity execution must receive a real `TaskActivityContext` carrying the canonical activity name and the -invoking orchestration's bound instance ID, not a null context or the Functions invocation ID. Use the normal -worker serialization and failure propagation paths, preserving structured failure details and unprocessed -events across continue-as-new. Do not substitute host-side `TaskHubWorker` registration or start another worker. +The Functions Worker SDK discovers the compiled methods during the normal build and generates their metadata +and invocation paths. The functions use ordinary trigger and `DurableClient` bindings in the isolated worker, +passing the bound `TaskOrchestrationContext` and a `TaskActivityContext` with the invoking orchestration's +instance ID to the shared tasks. Normal serialization and failure propagation preserve structured failure +details and unprocessed events across continue-as-new. Construct the fetch/report activities with an `ILargePayloadPurgeClient` and their typed loggers, and the delete activity with the worker's configured `PayloadStore` and logger. Reuse `BlobPayloadStore` with @@ -235,7 +227,7 @@ delete activity with the worker's configured `PayloadStore` and logger. Reuse `B Deletion runs in the language worker; purge RPCs do not carry storage credentials. The narrow purge client must honor UTC deadlines and preserve opaque tombstone tokens; the activities classify fetch/report gRPC failures. -The Functions integration must route setting, fetch, and report operations through the bound client's local +The Functions integration routes setting, fetch, and report operations through the bound client's local host endpoint to the provider's authenticated transport for the **same task hub**. The existing `LargePayloadPurge` gRPC service is separate from `TaskHubSidecarService`. The Functions client wrapper can forward the setting through the existing infrastructure `ILargePayloadAutoPurgeClient` interface so the @@ -245,9 +237,9 @@ this SDK surface alone does not supply Functions metadata or the local-host brid Enabling explicitly writes the setting, starts the reserved instance with live-status deduplication and an empty version, verifies its identity and Running status, then sends `SetBatchSize`. Disabling **only** writes -the setting and ignores batch size. Calling neither leaves the setting untouched. No automatic host -configuration or primary-host monitor is required. These steps are not transactional; failures propagate -without rollback. Existing standalone gRPC client and worker behavior is unchanged. +the setting and ignores batch size. Calling neither leaves the setting untouched. These steps are not +transactional; failures propagate without rollback. Existing standalone gRPC client and worker behavior +is unchanged. ## Obtaining the Protobuf definitions diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs index f39560a2..24e8c4f6 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs @@ -8,11 +8,7 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// /// Constants used throughout the blob payload auto-purge functionality. /// -/// -/// Infrastructure integration constants. The orchestration ID is reserved per task hub and must not be used -/// for application orchestrations. -/// -public static class BlobPurgeConstants +static class BlobPurgeConstants { /// /// The fixed orchestration instance ID for the auto-purge job in each task hub. @@ -40,5 +36,5 @@ public static class BlobPurgeConstants /// /// The maximum duration of an individual fetch or report RPC attempt. /// - internal const int RpcTimeoutSeconds = 60; + public const int RpcTimeoutSeconds = 60; } From 4c3ec26bb41da78689ee4f73c7d09679a08b39e9 Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 24 Sep 2026 14:29:30 -0700 Subject: [PATCH 08/15] Own the optional large payload purge service contract in the SDK Preserve the interface package identity and canonical SDK Client model references with an independent version. Add focused API and signing tests, Apache license attribution, solution discovery, and existing release signing coverage. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- Microsoft.DurableTask.sln | 30 +++ README.md | 6 + azure-pipelines-release.yml | 32 ++++ doc/release_process.md | 9 + eng/templates/build.yml | 7 + ...estrationServiceLargePayloadPurgeClient.cs | 56 ++++++ src/LargePayloadPurge.Abstractions/LICENSE | 174 ++++++++++++++++++ .../LargePayloadPurge.Abstractions.csproj | 22 +++ src/LargePayloadPurge.Abstractions/README.md | 43 +++++ .../RELEASENOTES.md | 1 + ...argePayloadPurge.Abstractions.Tests.csproj | 13 ++ .../LargePayloadPurgeContractTests.cs | 139 ++++++++++++++ 12 files changed, 532 insertions(+) create mode 100644 src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs create mode 100644 src/LargePayloadPurge.Abstractions/LICENSE create mode 100644 src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj create mode 100644 src/LargePayloadPurge.Abstractions/README.md create mode 100644 src/LargePayloadPurge.Abstractions/RELEASENOTES.md create mode 100644 test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj create mode 100644 test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs diff --git a/Microsoft.DurableTask.sln b/Microsoft.DurableTask.sln index 380ee0a0..40a4716b 100644 --- a/Microsoft.DurableTask.sln +++ b/Microsoft.DurableTask.sln @@ -149,6 +149,10 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Extensions", "Extensions", EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AzureBlobPayloads.Tests", "test\Extensions\AzureBlobPayloads.Tests\AzureBlobPayloads.Tests.csproj", "{3E509481-3CCC-4006-BCB2-9E8FA7C275F1}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "LargePayloadPurge.Abstractions", "src\LargePayloadPurge.Abstractions\LargePayloadPurge.Abstractions.csproj", "{CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "LargePayloadPurge.Abstractions.Tests", "test\LargePayloadPurge.Abstractions.Tests\LargePayloadPurge.Abstractions.Tests.csproj", "{BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -855,6 +859,30 @@ Global {3E509481-3CCC-4006-BCB2-9E8FA7C275F1}.Release|x64.Build.0 = Release|Any CPU {3E509481-3CCC-4006-BCB2-9E8FA7C275F1}.Release|x86.ActiveCfg = Release|Any CPU {3E509481-3CCC-4006-BCB2-9E8FA7C275F1}.Release|x86.Build.0 = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|Any CPU.Build.0 = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|x64.ActiveCfg = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|x64.Build.0 = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|x86.ActiveCfg = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Debug|x86.Build.0 = Debug|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|Any CPU.ActiveCfg = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|Any CPU.Build.0 = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|x64.ActiveCfg = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|x64.Build.0 = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|x86.ActiveCfg = Release|Any CPU + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28}.Release|x86.Build.0 = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|Any CPU.Build.0 = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|x64.ActiveCfg = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|x64.Build.0 = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|x86.ActiveCfg = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Debug|x86.Build.0 = Debug|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|Any CPU.ActiveCfg = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|Any CPU.Build.0 = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|x64.ActiveCfg = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|x64.Build.0 = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|x86.ActiveCfg = Release|Any CPU + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE}.Release|x86.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -929,6 +957,8 @@ Global {3B8F957E-7773-4C0C-ACD7-91A1591D9312} = {5B448FF6-EC42-491D-A22E-1DC8B618E6D5} {00205C88-F000-28F2-A910-C6FA00E065EE} = {E5637F81-2FB9-4CD7-900D-455363B142A7} {3E509481-3CCC-4006-BCB2-9E8FA7C275F1} = {00205C88-F000-28F2-A910-C6FA00E065EE} + {CF0A6A55-1DEC-4027-8B8B-4B0DEE3C7A28} = {8AFC9781-F6F1-4696-BB4A-9ED7CA9D612B} + {BE1D641F-1F96-483E-A1A9-0F8A64E9BAFE} = {E5637F81-2FB9-4CD7-900D-455363B142A7} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {AB41CB55-35EA-4986-A522-387AB3402E71} diff --git a/README.md b/README.md index cc25f274..d133f290 100644 --- a/README.md +++ b/README.md @@ -202,6 +202,12 @@ The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) s ### Blob auto-purge infrastructure integration +The optional service contract is maintained in +[`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions`](src/LargePayloadPurge.Abstractions/README.md). +It defines `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` and references the +canonical SDK Client models without duplicating them. This package is versioned independently and is not +BCL-only: its Client dependency transitively depends on SDK Abstractions and Durable Task Core. + `Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes reusable orchestration and activity implementations: `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, `DeleteExternalBlobActivity`, and `ReportLargePayloadPurgeResultsActivity`. Preserve their exact task names, empty version, input/output diff --git a/azure-pipelines-release.yml b/azure-pipelines-release.yml index 3bfb6e83..2b084345 100644 --- a/azure-pipelines-release.yml +++ b/azure-pipelines-release.yml @@ -95,6 +95,38 @@ steps: } ] +# The service contract preserves its assembly name outside the Microsoft.DurableTask prefix. +- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@2 + displayName: 'ESRP CodeSigning: Large payload purge abstraction' + inputs: + ConnectedServiceName: 'ESRP Service' + FolderPath: $(bin_dir) + Pattern: 'DurableTask.LargePayloadPurge.Abstractions.dll' + signConfigType: inlineSignParams + inlineOperation: | + [ + { + "KeyCode": "CP-230012", + "OperationCode": "SigntoolSign", + "Parameters": { + "OpusName": "Microsoft", + "OpusInfo": "http://www.microsoft.com", + "FileDigest": "/fd \"SHA256\"", + "PageHash": "/NPH", + "TimeStamp": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + }, + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-230012", + "OperationCode": "SigntoolVerify", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + # SBOM generator task for additional supply chain protection - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 displayName: 'SBOM Manifest Generator' diff --git a/doc/release_process.md b/doc/release_process.md index 319a2cd4..ec3eb273 100644 --- a/doc/release_process.md +++ b/doc/release_process.md @@ -5,11 +5,20 @@ | Package prefix | Registry | |---|---| | `Microsoft.DurableTask.*` | [NuGet](https://www.nuget.org/profiles/durabletask) | +| `Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` | [NuGet](https://www.nuget.org/profiles/durabletask) | This repo publishes multiple NuGet packages. Most share a single version defined in `eng/targets/Release.props`. Individual packages can version independently by adding `` and `` properties directly in their `.csproj`. We follow an approach of releasing everything together, even if a package has no changes — unless we intentionally hold a package back. +`LargePayloadPurge.Abstractions` versions independently in its project file, starting at the planned +`0.1.0` release. It uses source references to the SDK Client models. Publish the corresponding Client and +Abstractions packages containing those models before publishing this package; released Client `1.26.0` +does not contain them. The package retains its Apache-2.0 interface license and uses the SDK strong-name key. +Its assembly name is `DurableTask.LargePayloadPurge.Abstractions`, so both assembly-signing pipelines +explicitly include it in addition to the `Microsoft.DurableTask.*` assemblies. The existing source traversal, +SBOM inclusion and NuGet signing/packing steps apply unchanged. + ### Versioning Scheme We follow [semver](https://semver.org/) with optional pre-release tags: diff --git a/eng/templates/build.yml b/eng/templates/build.yml index bcd404f8..420514ca 100644 --- a/eng/templates/build.yml +++ b/eng/templates/build.yml @@ -76,6 +76,13 @@ jobs: pattern: Microsoft.DurableTask.*.dll signType: dll + - template: ci/sign-files.yml@eng + parameters: + displayName: Sign large payload purge abstraction + folderPath: $(bin_dir) + pattern: DurableTask.LargePayloadPurge.Abstractions.dll + signType: dll + # Packaging needs to be a separate step from build. # This will automatically pick up the signed DLLs. - task: DotNetCoreCLI@2 diff --git a/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs b/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs new file mode 100644 index 00000000..e42ad5f1 --- /dev/null +++ b/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs @@ -0,0 +1,56 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +// Adapted to the Durable Task .NET SDK repository's namespace and using conventions. + +using Microsoft.DurableTask.Client; + +namespace DurableTask.LargePayloadPurge; + +/// +/// Optional orchestration service client capability for purging tombstoned large payloads. +/// +public interface IOrchestrationServiceLargePayloadPurgeClient +{ + /// + /// Records whether large payload auto-purge is enabled for the client's task hub. + /// + /// This operation does not start, stop, or wait for a purge runner. + /// Whether large payload auto-purge is enabled. + /// The caller's operation deadline in UTC, or + /// when the caller has not specified a deadline. + /// The token used to cancel the operation. + /// A task that represents the operation. + Task SetLargePayloadAutoPurgeAsync(bool enabled, DateTime deadlineUtc, CancellationToken cancellationToken); + + /// + /// Gets tombstoned large payloads that are ready to be purged. + /// + /// The maximum number of tombstones to return. + /// The caller's operation deadline in UTC, or + /// when the caller has not specified a deadline. + /// The token used to cancel the operation. + /// The tombstones to process. + Task> GetLargePayloadsToPurgeAsync( + int limit, DateTime deadlineUtc, CancellationToken cancellationToken); + + /// + /// Reports the outcomes of attempts to purge tombstoned large payloads. + /// + /// The purge outcomes, including the unchanged tombstone correlation tokens. + /// The caller's operation deadline in UTC, or + /// when the caller has not specified a deadline. + /// The token used to cancel the operation. + /// A task that represents the operation. + Task ReportLargePayloadPurgeResultsAsync( + IReadOnlyList results, DateTime deadlineUtc, CancellationToken cancellationToken); +} diff --git a/src/LargePayloadPurge.Abstractions/LICENSE b/src/LargePayloadPurge.Abstractions/LICENSE new file mode 100644 index 00000000..dd5b3a58 --- /dev/null +++ b/src/LargePayloadPurge.Abstractions/LICENSE @@ -0,0 +1,174 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj new file mode 100644 index 00000000..896e4947 --- /dev/null +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -0,0 +1,22 @@ + + + + netstandard2.0 + DurableTask.LargePayloadPurge.Abstractions + DurableTask.LargePayloadPurge + Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions + Optional orchestration service capability for large payload purge using the Durable Task SDK client models. + Apache-2.0 + 0.1.0 + + true + + $(NoWarn);SA1636 + + + + + + + + diff --git a/src/LargePayloadPurge.Abstractions/README.md b/src/LargePayloadPurge.Abstractions/README.md new file mode 100644 index 00000000..44e6d604 --- /dev/null +++ b/src/LargePayloadPurge.Abstractions/README.md @@ -0,0 +1,43 @@ +# Large payload purge service abstraction + +`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` provides the optional +`DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` capability for orchestration +service adapters. The assembly is `DurableTask.LargePayloadPurge.Abstractions`; its only public type is +the interface. + +The three operations set explicit auto-purge state, fetch tombstones, and report outcomes. Each accepts a +UTC deadline and cancellation token. `DateTime.MaxValue` represents an unspecified deadline. Setting the +flag alone does not start or stop a purge runner. + +The contract uses the canonical `LargePayloadTombstone`, `LargePayloadPurgeResult`, and +`LargePayloadPurgeDisposition` types from `Microsoft.DurableTask.Client`. It does not copy, move, wrap or +forward those types. Backend-issued tombstone tokens must be echoed unchanged. + +## Dependencies and release + +This project references the SDK Client project directly. Its packaged dependency graph is: + +```text +Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions + -> Microsoft.DurableTask.Client + -> Microsoft.DurableTask.Abstractions + -> Microsoft.Azure.DurableTask.Core +``` + +It is not a BCL-only package. Core does not depend on this package or the SDK Client. The contract package +contains no blob storage, gRPC transport, worker, or orchestration implementation. + +The initial version is planned as `0.1.0`, independently of the repository-wide SDK version. Release the +matching SDK Client and Abstractions dependencies containing the purge models before this package. +Published Client `1.26.0` predates those models and is not sufficient. Repository builds use source project +references; no external Client-version bootstrap property is required. + +The assembly uses this repository's strong-name key. Consumers of an earlier local prototype signed with +a different key must rebuild against the SDK-owned package; there is no compatibility promise for those +unreleased prototype binaries. + +## License + +The interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains its +Apache-2.0 license and copyright notice. See [LICENSE](LICENSE). The referenced SDK model assemblies +retain their own licenses. diff --git a/src/LargePayloadPurge.Abstractions/RELEASENOTES.md b/src/LargePayloadPurge.Abstractions/RELEASENOTES.md new file mode 100644 index 00000000..fb216229 --- /dev/null +++ b/src/LargePayloadPurge.Abstractions/RELEASENOTES.md @@ -0,0 +1 @@ +- Initial optional service capability for explicit large payload auto-purge, using the canonical Durable Task SDK Client models. diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj new file mode 100644 index 00000000..fa163cf7 --- /dev/null +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj @@ -0,0 +1,13 @@ + + + + net10.0 + DurableTask.LargePayloadPurge.Abstractions.Tests + DurableTask.LargePayloadPurge.Tests + + + + + + + diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs new file mode 100644 index 00000000..7eedd296 --- /dev/null +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs @@ -0,0 +1,139 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +// Adapted from the original contract tests for SDK xUnit, signing and dependency checks. + +using System.Reflection; +using Microsoft.DurableTask.Client; +using Xunit; + +namespace DurableTask.LargePayloadPurge.Tests; + +public class LargePayloadPurgeContractTests +{ + [Fact] + public void PackageExportsOnlyTheStandaloneInterface() + { + // Arrange + Type contract = typeof(IOrchestrationServiceLargePayloadPurgeClient); + + // Act + Type[] exported = contract.Assembly.GetExportedTypes(); + + // Assert + Assert.True(contract.IsInterface); + Assert.Equal("DurableTask.LargePayloadPurge", contract.Namespace); + Assert.Equal("DurableTask.LargePayloadPurge.Abstractions", contract.Assembly.GetName().Name); + Assert.Equal([contract], exported); + Assert.Empty(contract.GetInterfaces()); + Assert.Equal(3, contract.GetMethods().Length); + } + + [Fact] + public void SetAcceptsExplicitChoiceAndCallerDeadlineAndCancellation() + { + // Arrange / Act / Assert + AssertSignature( + nameof(IOrchestrationServiceLargePayloadPurgeClient.SetLargePayloadAutoPurgeAsync), + typeof(Task), + [typeof(bool), typeof(DateTime), typeof(CancellationToken)], + ["enabled", "deadlineUtc", "cancellationToken"]); + } + + [Fact] + public void GetReturnsCanonicalSdkTombstones() + { + // Arrange / Act / Assert + AssertSignature( + nameof(IOrchestrationServiceLargePayloadPurgeClient.GetLargePayloadsToPurgeAsync), + typeof(Task>), + [typeof(int), typeof(DateTime), typeof(CancellationToken)], + ["limit", "deadlineUtc", "cancellationToken"]); + } + + [Fact] + public void ReportAcceptsCanonicalSdkResults() + { + // Arrange / Act / Assert + AssertSignature( + nameof(IOrchestrationServiceLargePayloadPurgeClient.ReportLargePayloadPurgeResultsAsync), + typeof(Task), + [typeof(IReadOnlyList), typeof(DateTime), typeof(CancellationToken)], + ["results", "deadlineUtc", "cancellationToken"]); + } + + [Fact] + public void ModelsComeFromSdkClientNotTheInterfacePackage() + { + // Arrange + Assembly client = typeof(DurableTaskClient).Assembly; + + // Act + Assembly[] modelAssemblies = + [ + typeof(LargePayloadTombstone).Assembly, + typeof(LargePayloadPurgeResult).Assembly, + typeof(LargePayloadPurgeDisposition).Assembly, + ]; + + // Assert + Assert.All(modelAssemblies, assembly => Assert.Same(client, assembly)); + Assert.NotSame(client, typeof(IOrchestrationServiceLargePayloadPurgeClient).Assembly); + } + + [Fact] + public void ContractUsesSdkSigningAndIndependentAssemblyVersion() + { + // Arrange + AssemblyName sdk = typeof(DurableTaskClient).Assembly.GetName(); + + // Act + AssemblyName contract = typeof(IOrchestrationServiceLargePayloadPurgeClient).Assembly.GetName(); + + // Assert + Assert.Equal(new Version(0, 1, 0, 0), contract.Version); + Assert.Equal("6A4C0315C2D1D937", Convert.ToHexString(contract.GetPublicKeyToken()!)); + Assert.Equal(sdk.GetPublicKeyToken(), contract.GetPublicKeyToken()); + } + + [Fact] + public void ContractDependsOnSdkModelsWithoutAddingACoreReverseDependency() + { + // Arrange + Assembly contract = typeof(IOrchestrationServiceLargePayloadPurgeClient).Assembly; + Assembly core = typeof(Core.TaskHubClient).Assembly; + + // Act + string?[] contractReferences = contract.GetReferencedAssemblies().Select(name => name.Name).ToArray(); + string?[] coreReferences = core.GetReferencedAssemblies().Select(name => name.Name).ToArray(); + + // Assert + Assert.Contains("Microsoft.DurableTask.Client", contractReferences); + Assert.DoesNotContain("DurableTask.Core", contractReferences); + Assert.DoesNotContain("Microsoft.DurableTask.Worker", contractReferences); + Assert.DoesNotContain("Microsoft.DurableTask.Grpc", contractReferences); + Assert.DoesNotContain("Microsoft.DurableTask.Extensions.AzureBlobPayloads", contractReferences); + Assert.DoesNotContain(contract.GetName().Name, coreReferences); + Assert.DoesNotContain("Microsoft.DurableTask.Client", coreReferences); + } + + static void AssertSignature(string methodName, Type returnType, Type[] parameterTypes, string[] parameterNames) + { + MethodInfo method = typeof(IOrchestrationServiceLargePayloadPurgeClient).GetMethod(methodName)!; + Assert.NotNull(method); + Assert.Equal(returnType, method.ReturnType); + ParameterInfo[] parameters = method.GetParameters(); + Assert.Equal(parameterTypes, parameters.Select(parameter => parameter.ParameterType)); + Assert.Equal(parameterNames, parameters.Select(parameter => parameter.Name)); + Assert.All(parameters, parameter => Assert.False(parameter.IsOptional)); + } +} From c816c32ca6b2d9ba0e9f12961c32092541374dc1 Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 24 Sep 2026 14:35:07 -0700 Subject: [PATCH 09/15] Include purge service contract in gated package publication Add the preserved package ID to the existing independent NuGet publication jobs, retaining the manual approval gate and symbol exclusion. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- eng/publish/publish.yml | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/eng/publish/publish.yml b/eng/publish/publish.yml index a3009469..f8ac44f5 100644 --- a/eng/publish/publish.yml +++ b/eng/publish/publish.yml @@ -463,4 +463,27 @@ extends: nuGetFeedType: external publishFeedCredentials: 'DurableTask org NuGet API Key' packagesToPush: '$(System.DefaultWorkingDirectory)/drop/Microsoft.DurableTask.Extensions.AzureBlobPayloads.*.nupkg;!$(System.DefaultWorkingDirectory)/**/*.symbols.nupkg' # Despite this being a custom command, we need to keep this for 1ES validation - packageParentPath: $(System.DefaultWorkingDirectory) # This needs to be set to some prefix of the `packagesToPush` parameter. Apparently it helps with SDL tooling \ No newline at end of file + packageParentPath: $(System.DefaultWorkingDirectory) # This needs to be set to some prefix of the `packagesToPush` parameter. Apparently it helps with SDL tooling + + # NuGet release (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions) + - job: nugetRelease_Microsoft_Azure_DurableTask_LargePayloadPurge_Abstractions + displayName: NuGet Release (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions) + dependsOn: nugetApproval + condition: succeeded('nugetApproval') + templateContext: + type: releaseJob + isProduction: true + inputs: + - input: pipelineArtifact + pipeline: officialPipeline + artifactName: drop + targetPath: $(System.DefaultWorkingDirectory)/drop + steps: + - task: 1ES.PublishNuget@1 + displayName: 'NuGet push (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions)' + inputs: + command: push + nuGetFeedType: external + publishFeedCredentials: 'DurableTask org NuGet API Key' + packagesToPush: '$(System.DefaultWorkingDirectory)/drop/Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions.*.nupkg;!$(System.DefaultWorkingDirectory)/**/*.symbols.nupkg' + packageParentPath: $(System.DefaultWorkingDirectory) \ No newline at end of file From ec53e25d200f696a909bbedd67876aa867ced466 Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 24 Sep 2026 15:40:51 -0700 Subject: [PATCH 10/15] Address purge contract target and deadline assertion feedback Explain the interface-only netstandard2.0 target alongside the matching Client model library. Use a typed xUnit assertion before validating activity deadlines. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../LargePayloadPurge.Abstractions.csproj | 1 + .../AutoPurge/AlternateHostPurgeTaskTests.cs | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj index 896e4947..fde33450 100644 --- a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -1,6 +1,7 @@ + netstandard2.0 DurableTask.LargePayloadPurge.Abstractions DurableTask.LargePayloadPurge diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs index 571755b2..46d270ea 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs @@ -66,9 +66,9 @@ public async Task ActualTasks_ExecuteWithDtfXArguments_AndPreserveCorrelationAcr Assert.Equal("[37]", driver.SingleActivity(nameof(GetLargePayloadTombstonesActivity)).Input); Assert.All(new[] { fetchDeadline, reportDeadline }, deadline => { - Assert.NotNull(deadline); - Assert.Equal(DateTimeKind.Utc, deadline.Value.Kind); - Assert.InRange(deadline.Value, earliestDeadline, DateTime.UtcNow.AddSeconds(60)); + DateTime actualDeadline = Assert.IsType(deadline); + Assert.Equal(DateTimeKind.Utc, actualDeadline.Kind); + Assert.InRange(actualDeadline, earliestDeadline, DateTime.UtcNow.AddSeconds(60)); }); Assert.Equal(driver.Snapshot(), driver.ReplaySnapshot()); store.VerifyAll(); From 2cd4cfdb6ecc68925b7d83e632e9b15c7bf2a907 Mon Sep 17 00:00:00 2001 From: wangbill Date: Fri, 25 Sep 2026 12:51:20 -0700 Subject: [PATCH 11/15] Move purge activity transport interface into the contracts package Preserve both interface APIs and SDK model identities while removing the Blob implementation dependency from transport consumers. Add ownership and dependency regression coverage and package the existing Apache and MIT notices together. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- README.md | 8 ++-- doc/release_process.md | 4 +- .../AzureBlobPayloads.csproj | 1 + .../ILargePayloadPurgeClient.cs | 0 .../LargePayloadPurge.Abstractions.csproj | 5 ++- src/LargePayloadPurge.Abstractions/README.md | 32 ++++++++------ .../RELEASENOTES.md | 2 +- ...argePayloadPurge.Abstractions.Tests.csproj | 1 + .../LargePayloadPurgeContractTests.cs | 43 ++++++++++++++++++- 9 files changed, 75 insertions(+), 21 deletions(-) rename src/{Extensions/AzureBlobPayloads/AutoPurge/Client => LargePayloadPurge.Abstractions}/ILargePayloadPurgeClient.cs (100%) diff --git a/README.md b/README.md index d133f290..d382aceb 100644 --- a/README.md +++ b/README.md @@ -204,9 +204,11 @@ The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) s The optional service contract is maintained in [`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions`](src/LargePayloadPurge.Abstractions/README.md). -It defines `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` and references the -canonical SDK Client models without duplicating them. This package is versioned independently and is not -BCL-only: its Client dependency transitively depends on SDK Abstractions and Durable Task Core. +It defines `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` and +`Microsoft.DurableTask.AzureBlobPayloads.ILargePayloadPurgeClient`, using the canonical SDK Client models +without duplicating them. This package is versioned independently and is not BCL-only: its Client dependency +transitively depends on SDK Abstractions and Durable Task Core. The Azure Blob implementation depends on these +contracts; the contracts do not depend on Blob storage, gRPC or worker implementations. `Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes reusable orchestration and activity implementations: `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, `DeleteExternalBlobActivity`, diff --git a/doc/release_process.md b/doc/release_process.md index ec3eb273..e420cad3 100644 --- a/doc/release_process.md +++ b/doc/release_process.md @@ -14,7 +14,9 @@ We follow an approach of releasing everything together, even if a package has no `LargePayloadPurge.Abstractions` versions independently in its project file, starting at the planned `0.1.0` release. It uses source references to the SDK Client models. Publish the corresponding Client and Abstractions packages containing those models before publishing this package; released Client `1.26.0` -does not contain them. The package retains its Apache-2.0 interface license and uses the SDK strong-name key. +does not contain them. The package retains the service interface's Apache-2.0 notice and the activity +transport interface's MIT notice, includes both license texts, and declares `Apache-2.0 AND MIT`. +It uses the SDK strong-name key. Its assembly name is `DurableTask.LargePayloadPurge.Abstractions`, so both assembly-signing pipelines explicitly include it in addition to the `Microsoft.DurableTask.*` assemblies. The existing source traversal, SBOM inclusion and NuGet signing/packing steps apply unchanged. diff --git a/src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj b/src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj index 78d80fd7..c793b77b 100644 --- a/src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj +++ b/src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj @@ -18,6 +18,7 @@ + diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs b/src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs similarity index 100% rename from src/Extensions/AzureBlobPayloads/AutoPurge/Client/ILargePayloadPurgeClient.cs rename to src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj index fde33450..87fa4eca 100644 --- a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -6,8 +6,8 @@ DurableTask.LargePayloadPurge.Abstractions DurableTask.LargePayloadPurge Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions - Optional orchestration service capability for large payload purge using the Durable Task SDK client models. - Apache-2.0 + Service and activity transport contracts for large payload purge using the Durable Task SDK client models. + Apache-2.0 AND MIT 0.1.0 true @@ -18,6 +18,7 @@ + diff --git a/src/LargePayloadPurge.Abstractions/README.md b/src/LargePayloadPurge.Abstractions/README.md index 44e6d604..97eec7c1 100644 --- a/src/LargePayloadPurge.Abstractions/README.md +++ b/src/LargePayloadPurge.Abstractions/README.md @@ -1,13 +1,16 @@ -# Large payload purge service abstraction +# Large payload purge contracts -`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` provides the optional -`DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` capability for orchestration -service adapters. The assembly is `DurableTask.LargePayloadPurge.Abstractions`; its only public type is -the interface. +`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` provides two interfaces in the +`DurableTask.LargePayloadPurge.Abstractions` assembly: -The three operations set explicit auto-purge state, fetch tombstones, and report outcomes. Each accepts a -UTC deadline and cancellation token. `DateTime.MaxValue` represents an unspecified deadline. Setting the -flag alone does not start or stop a purge runner. +- `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` is the optional service + capability for explicit auto-purge state, tombstone fetches, and outcome reports. Each operation requires + a UTC deadline and cancellation token. `DateTime.MaxValue` represents an unspecified deadline. + Setting the flag alone does not start or stop a purge runner. +- `Microsoft.DurableTask.AzureBlobPayloads.ILargePayloadPurgeClient` is the activity transport contract for + fetching tombstones and reporting outcomes. It accepts a UTC deadline and optional cancellation token. + Transport implementations bind it to the same authenticated task hub as the associated orchestration + client and preserve the documented gRPC status behavior without requiring this package to reference gRPC. The contract uses the canonical `LargePayloadTombstone`, `LargePayloadPurgeResult`, and `LargePayloadPurgeDisposition` types from `Microsoft.DurableTask.Client`. It does not copy, move, wrap or @@ -26,6 +29,7 @@ Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions It is not a BCL-only package. Core does not depend on this package or the SDK Client. The contract package contains no blob storage, gRPC transport, worker, or orchestration implementation. +The Azure Blob implementation references this package, not the other way around. The initial version is planned as `0.1.0`, independently of the repository-wide SDK version. Release the matching SDK Client and Abstractions dependencies containing the purge models before this package. @@ -34,10 +38,14 @@ references; no external Client-version bootstrap property is required. The assembly uses this repository's strong-name key. Consumers of an earlier local prototype signed with a different key must rebuild against the SDK-owned package; there is no compatibility promise for those -unreleased prototype binaries. +unreleased prototype binaries. The activity transport interface also moved here from the unreleased +Azure Blob implementation while retaining its full namespace, methods and optional-parameter defaults. +Rebuild consumers against this assembly; no type forwarder is provided. ## License -The interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains its -Apache-2.0 license and copyright notice. See [LICENSE](LICENSE). The referenced SDK model assemblies -retain their own licenses. +The service interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains +its Apache-2.0 notice; see [LICENSE](LICENSE). The activity transport interface retains its MIT notice; +see the [SDK MIT license](https://github.com/microsoft/durabletask-dotnet/blob/main/LICENSE). Both license texts +are included in the package as `LICENSE` and `LICENSE-MIT`; the package license expression +is `Apache-2.0 AND MIT`. The referenced SDK model assemblies retain their own licenses. diff --git a/src/LargePayloadPurge.Abstractions/RELEASENOTES.md b/src/LargePayloadPurge.Abstractions/RELEASENOTES.md index fb216229..51224bf7 100644 --- a/src/LargePayloadPurge.Abstractions/RELEASENOTES.md +++ b/src/LargePayloadPurge.Abstractions/RELEASENOTES.md @@ -1 +1 @@ -- Initial optional service capability for explicit large payload auto-purge, using the canonical Durable Task SDK Client models. +- Initial service and activity transport contracts for large payload auto-purge, using the canonical Durable Task SDK Client models. diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj index fa163cf7..2cb18117 100644 --- a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj @@ -8,6 +8,7 @@ + diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs index 7eedd296..57483127 100644 --- a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs @@ -13,6 +13,7 @@ // Adapted from the original contract tests for SDK xUnit, signing and dependency checks. using System.Reflection; +using Microsoft.DurableTask.AzureBlobPayloads; using Microsoft.DurableTask.Client; using Xunit; @@ -21,10 +22,12 @@ namespace DurableTask.LargePayloadPurge.Tests; public class LargePayloadPurgeContractTests { [Fact] - public void PackageExportsOnlyTheStandaloneInterface() + public void PackageOwnsBothInterfacesWithoutBlobDefinitionsOrForwarders() { // Arrange Type contract = typeof(IOrchestrationServiceLargePayloadPurgeClient); + Type transport = typeof(ILargePayloadPurgeClient); + Assembly blob = typeof(GetLargePayloadTombstonesActivity).Assembly; // Act Type[] exported = contract.Assembly.GetExportedTypes(); @@ -33,9 +36,16 @@ public void PackageExportsOnlyTheStandaloneInterface() Assert.True(contract.IsInterface); Assert.Equal("DurableTask.LargePayloadPurge", contract.Namespace); Assert.Equal("DurableTask.LargePayloadPurge.Abstractions", contract.Assembly.GetName().Name); - Assert.Equal([contract], exported); + Assert.Equal(new[] { contract, transport }.OrderBy(type => type.FullName), exported.OrderBy(type => type.FullName)); + Assert.Same(contract.Assembly, transport.Assembly); + Assert.True(transport.IsInterface); + Assert.Equal("Microsoft.DurableTask.AzureBlobPayloads", transport.Namespace); Assert.Empty(contract.GetInterfaces()); + Assert.Empty(transport.GetInterfaces()); Assert.Equal(3, contract.GetMethods().Length); + Assert.Equal(2, transport.GetMethods().Length); + Assert.DoesNotContain(blob.GetTypes(), type => type.FullName == transport.FullName); + Assert.DoesNotContain(blob.GetForwardedTypes(), type => type.FullName == transport.FullName); } [Fact] @@ -58,6 +68,11 @@ public void GetReturnsCanonicalSdkTombstones() typeof(Task>), [typeof(int), typeof(DateTime), typeof(CancellationToken)], ["limit", "deadlineUtc", "cancellationToken"]); + AssertTransportSignature( + nameof(ILargePayloadPurgeClient.GetLargePayloadTombstonesAsync), + typeof(Task>), + [typeof(int), typeof(DateTime), typeof(CancellationToken)], + ["limit", "deadline", "cancellationToken"]); } [Fact] @@ -69,6 +84,11 @@ public void ReportAcceptsCanonicalSdkResults() typeof(Task), [typeof(IReadOnlyList), typeof(DateTime), typeof(CancellationToken)], ["results", "deadlineUtc", "cancellationToken"]); + AssertTransportSignature( + nameof(ILargePayloadPurgeClient.ReportLargePayloadPurgeResultsAsync), + typeof(Task), + [typeof(IReadOnlyList), typeof(DateTime), typeof(CancellationToken)], + ["results", "deadline", "cancellationToken"]); } [Fact] @@ -111,6 +131,7 @@ public void ContractDependsOnSdkModelsWithoutAddingACoreReverseDependency() // Arrange Assembly contract = typeof(IOrchestrationServiceLargePayloadPurgeClient).Assembly; Assembly core = typeof(Core.TaskHubClient).Assembly; + Assembly blob = typeof(GetLargePayloadTombstonesActivity).Assembly; // Act string?[] contractReferences = contract.GetReferencedAssemblies().Select(name => name.Name).ToArray(); @@ -122,6 +143,10 @@ public void ContractDependsOnSdkModelsWithoutAddingACoreReverseDependency() Assert.DoesNotContain("Microsoft.DurableTask.Worker", contractReferences); Assert.DoesNotContain("Microsoft.DurableTask.Grpc", contractReferences); Assert.DoesNotContain("Microsoft.DurableTask.Extensions.AzureBlobPayloads", contractReferences); + Assert.DoesNotContain(contractReferences, name => name!.StartsWith("Azure.", StringComparison.Ordinal)); + Assert.DoesNotContain(contractReferences, name => name!.StartsWith("Grpc.", StringComparison.Ordinal)); + Assert.DoesNotContain(contractReferences, name => name!.StartsWith("Microsoft.DurableTask.Worker", StringComparison.Ordinal)); + Assert.Contains(blob.GetReferencedAssemblies(), name => name.Name == contract.GetName().Name); Assert.DoesNotContain(contract.GetName().Name, coreReferences); Assert.DoesNotContain("Microsoft.DurableTask.Client", coreReferences); } @@ -136,4 +161,18 @@ static void AssertSignature(string methodName, Type returnType, Type[] parameter Assert.Equal(parameterNames, parameters.Select(parameter => parameter.Name)); Assert.All(parameters, parameter => Assert.False(parameter.IsOptional)); } + + static void AssertTransportSignature(string methodName, Type returnType, Type[] parameterTypes, string[] parameterNames) + { + MethodInfo method = typeof(ILargePayloadPurgeClient).GetMethod(methodName)!; + Assert.NotNull(method); + Assert.Equal(returnType, method.ReturnType); + ParameterInfo[] parameters = method.GetParameters(); + Assert.Equal(parameterTypes, parameters.Select(parameter => parameter.ParameterType)); + Assert.Equal(parameterNames, parameters.Select(parameter => parameter.Name)); + Assert.All(parameters.Take(2), parameter => Assert.False(parameter.IsOptional)); + Assert.True(parameters[2].IsOptional); + Assert.True(parameters[2].HasDefaultValue); + Assert.Null(parameters[2].DefaultValue); + } } From 8e79371523084fe0584d7bce45a42aa17eb54cde Mon Sep 17 00:00:00 2001 From: wangbill Date: Fri, 25 Sep 2026 12:53:26 -0700 Subject: [PATCH 12/15] Package the MIT notice under an explicit filename Use an extension-bearing package path so NuGet places the linked MIT license at the documented package root rather than treating it as a directory. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../LargePayloadPurge.Abstractions.csproj | 2 +- src/LargePayloadPurge.Abstractions/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj index 87fa4eca..5419d882 100644 --- a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -18,7 +18,7 @@ - + diff --git a/src/LargePayloadPurge.Abstractions/README.md b/src/LargePayloadPurge.Abstractions/README.md index 97eec7c1..bfc6e17c 100644 --- a/src/LargePayloadPurge.Abstractions/README.md +++ b/src/LargePayloadPurge.Abstractions/README.md @@ -47,5 +47,5 @@ Rebuild consumers against this assembly; no type forwarder is provided. The service interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains its Apache-2.0 notice; see [LICENSE](LICENSE). The activity transport interface retains its MIT notice; see the [SDK MIT license](https://github.com/microsoft/durabletask-dotnet/blob/main/LICENSE). Both license texts -are included in the package as `LICENSE` and `LICENSE-MIT`; the package license expression +are included in the package as `LICENSE` and `LICENSE-MIT.txt`; the package license expression is `Apache-2.0 AND MIT`. The referenced SDK model assemblies retain their own licenses. From b8a17635224bf7cc7263d841939288f0bc10682e Mon Sep 17 00:00:00 2001 From: wangbill Date: Fri, 25 Sep 2026 12:54:47 -0700 Subject: [PATCH 13/15] Use a dedicated package directory for the original MIT license Keep the repository license linked without duplicating it and document the actual NuGet directory layout. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../LargePayloadPurge.Abstractions.csproj | 2 +- src/LargePayloadPurge.Abstractions/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj index 5419d882..48476c6e 100644 --- a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -18,7 +18,7 @@ - + diff --git a/src/LargePayloadPurge.Abstractions/README.md b/src/LargePayloadPurge.Abstractions/README.md index bfc6e17c..1c9e99de 100644 --- a/src/LargePayloadPurge.Abstractions/README.md +++ b/src/LargePayloadPurge.Abstractions/README.md @@ -47,5 +47,5 @@ Rebuild consumers against this assembly; no type forwarder is provided. The service interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains its Apache-2.0 notice; see [LICENSE](LICENSE). The activity transport interface retains its MIT notice; see the [SDK MIT license](https://github.com/microsoft/durabletask-dotnet/blob/main/LICENSE). Both license texts -are included in the package as `LICENSE` and `LICENSE-MIT.txt`; the package license expression +are included in the package as `LICENSE` and `licenses/MIT/LICENSE`; the package license expression is `Apache-2.0 AND MIT`. The referenced SDK model assemblies retain their own licenses. From 9f206371365b96491514b95c8ed2fe4e2ea4a870 Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 1 Oct 2026 12:22:39 -0400 Subject: [PATCH 14/15] Address purge contract naming, reuse, and package review feedback Normalize the package and assembly identity, inherit shared fetch/report operations, construct gRPC adapters in named worker factories, and retain normal SDK signing and release conventions. Align verified task-original interface and tests with the SDK MIT convention after scoped authorization. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 2 + README.md | 5 +- azure-pipelines-release.yml | 32 ---- doc/release_process.md | 16 +- eng/publish/publish.yml | 10 +- eng/templates/build.yml | 7 - list-nuget-packages-links.ps1 | 1 + .../GetLargePayloadTombstonesActivity.cs | 12 -- .../ReportLargePayloadPurgeResultsActivity.cs | 12 -- .../Client/GrpcLargePayloadPurgeClient.cs | 19 +- ...rkerBuilderExtensions.AzureBlobPayloads.cs | 4 +- .../ILargePayloadPurgeClient.cs | 2 +- ...estrationServiceLargePayloadPurgeClient.cs | 44 +---- src/LargePayloadPurge.Abstractions/LICENSE | 174 ------------------ .../LargePayloadPurge.Abstractions.csproj | 7 - src/LargePayloadPurge.Abstractions/README.md | 36 ++-- .../PurgeActivityBackendStatusTests.cs | 20 +- .../AutoPurge/PurgeActivityDeadlineTests.cs | 4 +- .../BlobPayloadStoreDeleteResponseTests.cs | 2 +- ...argePayloadPurge.Abstractions.Tests.csproj | 1 - .../LargePayloadPurgeContractTests.cs | 34 +--- 21 files changed, 75 insertions(+), 369 deletions(-) delete mode 100644 src/LargePayloadPurge.Abstractions/LICENSE diff --git a/CHANGELOG.md b/CHANGELOG.md index 51bcbe91..71bc90e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased +- Add `Microsoft.DurableTask.LargePayloadPurge.Abstractions` with shared fetch/report and service-setting interfaces, and expose the existing blob purge tasks for host integration ([#805](https://github.com/microsoft/durabletask-dotnet/pull/805)). + ## v1.26.0 - Adding rewind to the sidecar by sophiatev ([#802](https://github.com/microsoft/durabletask-dotnet/pull/802)) - Prevent external-event loss after canceled waits in isolated worker by wangbill ([#801](https://github.com/microsoft/durabletask-dotnet/pull/801)) diff --git a/README.md b/README.md index d382aceb..c705f66e 100644 --- a/README.md +++ b/README.md @@ -203,12 +203,13 @@ The [on-demand sandbox activities sample](samples/on-demand-sandbox/README.md) s ### Blob auto-purge infrastructure integration The optional service contract is maintained in -[`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions`](src/LargePayloadPurge.Abstractions/README.md). +[`Microsoft.DurableTask.LargePayloadPurge.Abstractions`](src/LargePayloadPurge.Abstractions/README.md). It defines `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` and `Microsoft.DurableTask.AzureBlobPayloads.ILargePayloadPurgeClient`, using the canonical SDK Client models without duplicating them. This package is versioned independently and is not BCL-only: its Client dependency transitively depends on SDK Abstractions and Durable Task Core. The Azure Blob implementation depends on these -contracts; the contracts do not depend on Blob storage, gRPC or worker implementations. +contracts; the contracts do not depend on Blob storage, gRPC or worker implementations. The service capability +inherits the shared fetch/report interface and adds only the setting operation. `Microsoft.DurableTask.Extensions.AzureBlobPayloads` exposes reusable orchestration and activity implementations: `BlobPurgeJobOrchestrator`, `GetLargePayloadTombstonesActivity`, `DeleteExternalBlobActivity`, diff --git a/azure-pipelines-release.yml b/azure-pipelines-release.yml index 2b084345..3bfb6e83 100644 --- a/azure-pipelines-release.yml +++ b/azure-pipelines-release.yml @@ -95,38 +95,6 @@ steps: } ] -# The service contract preserves its assembly name outside the Microsoft.DurableTask prefix. -- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@2 - displayName: 'ESRP CodeSigning: Large payload purge abstraction' - inputs: - ConnectedServiceName: 'ESRP Service' - FolderPath: $(bin_dir) - Pattern: 'DurableTask.LargePayloadPurge.Abstractions.dll' - signConfigType: inlineSignParams - inlineOperation: | - [ - { - "KeyCode": "CP-230012", - "OperationCode": "SigntoolSign", - "Parameters": { - "OpusName": "Microsoft", - "OpusInfo": "http://www.microsoft.com", - "FileDigest": "/fd \"SHA256\"", - "PageHash": "/NPH", - "TimeStamp": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" - }, - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "CP-230012", - "OperationCode": "SigntoolVerify", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - # SBOM generator task for additional supply chain protection - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 displayName: 'SBOM Manifest Generator' diff --git a/doc/release_process.md b/doc/release_process.md index e420cad3..bc83dcc2 100644 --- a/doc/release_process.md +++ b/doc/release_process.md @@ -5,7 +5,6 @@ | Package prefix | Registry | |---|---| | `Microsoft.DurableTask.*` | [NuGet](https://www.nuget.org/profiles/durabletask) | -| `Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` | [NuGet](https://www.nuget.org/profiles/durabletask) | This repo publishes multiple NuGet packages. Most share a single version defined in `eng/targets/Release.props`. Individual packages can version independently by adding `` and `` properties directly in their `.csproj`. @@ -14,12 +13,15 @@ We follow an approach of releasing everything together, even if a package has no `LargePayloadPurge.Abstractions` versions independently in its project file, starting at the planned `0.1.0` release. It uses source references to the SDK Client models. Publish the corresponding Client and Abstractions packages containing those models before publishing this package; released Client `1.26.0` -does not contain them. The package retains the service interface's Apache-2.0 notice and the activity -transport interface's MIT notice, includes both license texts, and declares `Apache-2.0 AND MIT`. -It uses the SDK strong-name key. -Its assembly name is `DurableTask.LargePayloadPurge.Abstractions`, so both assembly-signing pipelines -explicitly include it in addition to the `Microsoft.DurableTask.*` assemblies. The existing source traversal, -SBOM inclusion and NuGet signing/packing steps apply unchanged. +does not contain them. The package uses the repository's MIT license and SDK strong-name key. +Its package and assembly name is `Microsoft.DurableTask.LargePayloadPurge.Abstractions`, covered by the +standard `Microsoft.DurableTask.*.dll` signing pattern. The existing source traversal, SBOM inclusion, +NuGet signing and per-package approval-gated publication steps apply. + +Keep its `RELEASENOTES.md`: `eng/targets/Release.targets` reads it into NuGet package metadata, whereas the +root `CHANGELOG.md` retains repository release history. The shared target also appends a link using the +package's own version (`releases/tag/v0.1.0` for its initial release). Verify or create the corresponding +release tag before publication; independent package versioning does not create that tag automatically. ### Versioning Scheme diff --git a/eng/publish/publish.yml b/eng/publish/publish.yml index f8ac44f5..8c3ee77a 100644 --- a/eng/publish/publish.yml +++ b/eng/publish/publish.yml @@ -465,9 +465,9 @@ extends: packagesToPush: '$(System.DefaultWorkingDirectory)/drop/Microsoft.DurableTask.Extensions.AzureBlobPayloads.*.nupkg;!$(System.DefaultWorkingDirectory)/**/*.symbols.nupkg' # Despite this being a custom command, we need to keep this for 1ES validation packageParentPath: $(System.DefaultWorkingDirectory) # This needs to be set to some prefix of the `packagesToPush` parameter. Apparently it helps with SDL tooling - # NuGet release (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions) - - job: nugetRelease_Microsoft_Azure_DurableTask_LargePayloadPurge_Abstractions - displayName: NuGet Release (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions) + # NuGet release (Microsoft.DurableTask.LargePayloadPurge.Abstractions) + - job: nugetRelease_Microsoft_DurableTask_LargePayloadPurge_Abstractions + displayName: NuGet Release (Microsoft.DurableTask.LargePayloadPurge.Abstractions) dependsOn: nugetApproval condition: succeeded('nugetApproval') templateContext: @@ -480,10 +480,10 @@ extends: targetPath: $(System.DefaultWorkingDirectory)/drop steps: - task: 1ES.PublishNuget@1 - displayName: 'NuGet push (Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions)' + displayName: 'NuGet push (Microsoft.DurableTask.LargePayloadPurge.Abstractions)' inputs: command: push nuGetFeedType: external publishFeedCredentials: 'DurableTask org NuGet API Key' - packagesToPush: '$(System.DefaultWorkingDirectory)/drop/Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions.*.nupkg;!$(System.DefaultWorkingDirectory)/**/*.symbols.nupkg' + packagesToPush: '$(System.DefaultWorkingDirectory)/drop/Microsoft.DurableTask.LargePayloadPurge.Abstractions.*.nupkg;!$(System.DefaultWorkingDirectory)/**/*.symbols.nupkg' packageParentPath: $(System.DefaultWorkingDirectory) \ No newline at end of file diff --git a/eng/templates/build.yml b/eng/templates/build.yml index 420514ca..bcd404f8 100644 --- a/eng/templates/build.yml +++ b/eng/templates/build.yml @@ -76,13 +76,6 @@ jobs: pattern: Microsoft.DurableTask.*.dll signType: dll - - template: ci/sign-files.yml@eng - parameters: - displayName: Sign large payload purge abstraction - folderPath: $(bin_dir) - pattern: DurableTask.LargePayloadPurge.Abstractions.dll - signType: dll - # Packaging needs to be a separate step from build. # This will automatically pick up the signed DLLs. - task: DotNetCoreCLI@2 diff --git a/list-nuget-packages-links.ps1 b/list-nuget-packages-links.ps1 index 2697869e..bd6187bd 100644 --- a/list-nuget-packages-links.ps1 +++ b/list-nuget-packages-links.ps1 @@ -72,6 +72,7 @@ $packages = @( "Microsoft.DurableTask.Worker.Grpc", "Microsoft.DurableTask.Client.OrchestrationServiceClientShim", "Microsoft.DurableTask.Extensions.AzureBlobPayloads", + "Microsoft.DurableTask.LargePayloadPurge.Abstractions", "Microsoft.DurableTask.Client.AzureManaged", "Microsoft.DurableTask.Worker.AzureManaged", "Microsoft.DurableTask.ScheduledTasks", diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs index 59767fa2..90f3e74c 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs @@ -4,7 +4,6 @@ using Grpc.Core; using Microsoft.DurableTask.Client; using Microsoft.Extensions.Logging; -using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; namespace Microsoft.DurableTask.AzureBlobPayloads; @@ -27,17 +26,6 @@ public sealed class GetLargePayloadTombstonesActivity( readonly ILargePayloadPurgeClient client = Check.NotNull(client); readonly ILogger logger = Check.NotNull(logger); - /// - /// Initializes a new instance of the class using the worker's transport. - /// - /// The worker's purge client. - /// The activity logger. - internal GetLargePayloadTombstonesActivity( - LargePayloadPurgeClient client, ILogger logger) - : this(new GrpcLargePayloadPurgeClient(client), logger) - { - } - /// /// Gets or sets the timeout for one backend RPC attempt. /// diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs index cf3501b6..1a86a03a 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs @@ -4,7 +4,6 @@ using Grpc.Core; using Microsoft.DurableTask.Client; using Microsoft.Extensions.Logging; -using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; namespace Microsoft.DurableTask.AzureBlobPayloads; @@ -29,17 +28,6 @@ public sealed class ReportLargePayloadPurgeResultsActivity( readonly ILargePayloadPurgeClient client = Check.NotNull(client); readonly ILogger logger = Check.NotNull(logger); - /// - /// Initializes a new instance of the class using the worker's transport. - /// - /// The worker's purge client. - /// The activity logger. - internal ReportLargePayloadPurgeResultsActivity( - LargePayloadPurgeClient client, ILogger logger) - : this(new GrpcLargePayloadPurgeClient(client), logger) - { - } - /// /// Gets or sets the timeout for one backend RPC attempt. /// diff --git a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs index a4311e96..3741f136 100644 --- a/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs +++ b/src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs @@ -2,27 +2,26 @@ // Licensed under the MIT License. using Microsoft.DurableTask.Client; -using static Microsoft.DurableTask.Protobuf.LargePayloads.LargePayloadPurge; -using LP = Microsoft.DurableTask.Protobuf.LargePayloads; +using Proto = Microsoft.DurableTask.Protobuf.LargePayloads; namespace Microsoft.DurableTask.AzureBlobPayloads; /// /// Adapts the worker's existing, rebindable purge transport without owning its lifetime. /// -sealed class GrpcLargePayloadPurgeClient(LargePayloadPurgeClient client) : ILargePayloadPurgeClient +sealed class GrpcLargePayloadPurgeClient(Proto.LargePayloadPurge.LargePayloadPurgeClient client) : ILargePayloadPurgeClient { - readonly LargePayloadPurgeClient client = Check.NotNull(client); + readonly Proto.LargePayloadPurge.LargePayloadPurgeClient client = Check.NotNull(client); /// public async Task> GetLargePayloadTombstonesAsync( int limit, DateTime deadline, CancellationToken cancellationToken = default) { using var call = this.client.GetLargePayloadTombstonesAsync( - new LP.GetLargePayloadTombstonesRequest { Limit = limit }, deadline: deadline, cancellationToken: cancellationToken); - LP.GetLargePayloadTombstonesResponse response = await call; + new Proto.GetLargePayloadTombstonesRequest { Limit = limit }, deadline: deadline, cancellationToken: cancellationToken); + Proto.GetLargePayloadTombstonesResponse response = await call; List tombstones = new(response.Tombstones.Count); - foreach (LP.LargePayloadTombstone tombstone in response.Tombstones) + foreach (Proto.LargePayloadTombstone tombstone in response.Tombstones) { tombstones.Add(new LargePayloadTombstone(tombstone.TombstoneToken, tombstone.PayloadToken)); } @@ -34,14 +33,14 @@ public async Task> GetLargePayloadTombstonesAsync( public async Task ReportLargePayloadPurgeResultsAsync( IReadOnlyList results, DateTime deadline, CancellationToken cancellationToken = default) { - LP.ReportLargePayloadPurgeResultsRequest request = new(); + Proto.ReportLargePayloadPurgeResultsRequest request = new(); foreach (LargePayloadPurgeResult result in results) { - request.Results.Add(new LP.LargePayloadPurgeResult + request.Results.Add(new Proto.LargePayloadPurgeResult { // Echo the opaque correlation token unchanged. The managed and protobuf enums share values. TombstoneToken = result.TombstoneToken, - Disposition = (LP.LargePayloadPurgeDisposition)result.Disposition, + Disposition = (Proto.LargePayloadPurgeDisposition)result.Disposition, }); } diff --git a/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs b/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs index 604d44b3..a5564431 100644 --- a/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs +++ b/src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs @@ -133,12 +133,12 @@ static IDurableTaskWorkerBuilder UseExternalizedPayloadsCore(IDurableTaskWorkerB r.AddOrchestrator(); r.AddActivity(nameof(GetLargePayloadTombstonesActivity), sp => new GetLargePayloadTombstonesActivity( - sp.GetRequiredKeyedService(builder.Name), + new GrpcLargePayloadPurgeClient(sp.GetRequiredKeyedService(builder.Name)), sp.GetRequiredService>())); r.AddActivity(); r.AddActivity(nameof(ReportLargePayloadPurgeResultsActivity), sp => new ReportLargePayloadPurgeResultsActivity( - sp.GetRequiredKeyedService(builder.Name), + new GrpcLargePayloadPurgeClient(sp.GetRequiredKeyedService(builder.Name)), sp.GetRequiredService>())); }); diff --git a/src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs b/src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs index a545401c..54dab01b 100644 --- a/src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs +++ b/src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs @@ -6,7 +6,7 @@ namespace Microsoft.DurableTask.AzureBlobPayloads; /// -/// Provides task-hub-bound transport operations for integrating blob auto-purge with an alternate .NET host. +/// Provides transport operations bound to a task hub for integrating blob auto-purge with an alternate .NET host. /// /// /// This is an infrastructure integration API, not an application orchestration API. Implementations must diff --git a/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs b/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs index e42ad5f1..c3c49ca6 100644 --- a/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs +++ b/src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs @@ -1,25 +1,17 @@ -// ---------------------------------------------------------------------------------- -// Copyright Microsoft Corporation -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// ---------------------------------------------------------------------------------- -// Adapted to the Durable Task .NET SDK repository's namespace and using conventions. +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. -using Microsoft.DurableTask.Client; +using Microsoft.DurableTask.AzureBlobPayloads; namespace DurableTask.LargePayloadPurge; /// /// Optional orchestration service client capability for purging tombstoned large payloads. /// -public interface IOrchestrationServiceLargePayloadPurgeClient +/// +/// Extends the shared fetch and report transport contract with control of the task hub's auto-purge setting. +/// +public interface IOrchestrationServiceLargePayloadPurgeClient : ILargePayloadPurgeClient { /// /// Records whether large payload auto-purge is enabled for the client's task hub. @@ -31,26 +23,4 @@ public interface IOrchestrationServiceLargePayloadPurgeClient /// The token used to cancel the operation. /// A task that represents the operation. Task SetLargePayloadAutoPurgeAsync(bool enabled, DateTime deadlineUtc, CancellationToken cancellationToken); - - /// - /// Gets tombstoned large payloads that are ready to be purged. - /// - /// The maximum number of tombstones to return. - /// The caller's operation deadline in UTC, or - /// when the caller has not specified a deadline. - /// The token used to cancel the operation. - /// The tombstones to process. - Task> GetLargePayloadsToPurgeAsync( - int limit, DateTime deadlineUtc, CancellationToken cancellationToken); - - /// - /// Reports the outcomes of attempts to purge tombstoned large payloads. - /// - /// The purge outcomes, including the unchanged tombstone correlation tokens. - /// The caller's operation deadline in UTC, or - /// when the caller has not specified a deadline. - /// The token used to cancel the operation. - /// A task that represents the operation. - Task ReportLargePayloadPurgeResultsAsync( - IReadOnlyList results, DateTime deadlineUtc, CancellationToken cancellationToken); } diff --git a/src/LargePayloadPurge.Abstractions/LICENSE b/src/LargePayloadPurge.Abstractions/LICENSE deleted file mode 100644 index dd5b3a58..00000000 --- a/src/LargePayloadPurge.Abstractions/LICENSE +++ /dev/null @@ -1,174 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. diff --git a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj index 48476c6e..4fac45e9 100644 --- a/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj +++ b/src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj @@ -3,22 +3,15 @@ netstandard2.0 - DurableTask.LargePayloadPurge.Abstractions DurableTask.LargePayloadPurge - Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions Service and activity transport contracts for large payload purge using the Durable Task SDK client models. - Apache-2.0 AND MIT 0.1.0 true - - $(NoWarn);SA1636 - - diff --git a/src/LargePayloadPurge.Abstractions/README.md b/src/LargePayloadPurge.Abstractions/README.md index 1c9e99de..5ac73b71 100644 --- a/src/LargePayloadPurge.Abstractions/README.md +++ b/src/LargePayloadPurge.Abstractions/README.md @@ -1,16 +1,17 @@ # Large payload purge contracts -`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` provides two interfaces in the -`DurableTask.LargePayloadPurge.Abstractions` assembly: +`Microsoft.DurableTask.LargePayloadPurge.Abstractions` provides two interfaces in the +`Microsoft.DurableTask.LargePayloadPurge.Abstractions` assembly: -- `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` is the optional service - capability for explicit auto-purge state, tombstone fetches, and outcome reports. Each operation requires - a UTC deadline and cancellation token. `DateTime.MaxValue` represents an unspecified deadline. - Setting the flag alone does not start or stop a purge runner. - `Microsoft.DurableTask.AzureBlobPayloads.ILargePayloadPurgeClient` is the activity transport contract for - fetching tombstones and reporting outcomes. It accepts a UTC deadline and optional cancellation token. + fetching tombstones with `GetLargePayloadTombstonesAsync` and reporting outcomes. It accepts a UTC deadline + and optional cancellation token. Transport implementations bind it to the same authenticated task hub as the associated orchestration client and preserve the documented gRPC status behavior without requiring this package to reference gRPC. +- `DurableTask.LargePayloadPurge.IOrchestrationServiceLargePayloadPurgeClient` inherits that shared transport + contract and adds only `SetLargePayloadAutoPurgeAsync`. The setting operation requires a deadline and + cancellation token; `DateTime.MaxValue` represents an unspecified deadline. Setting the flag alone does + not start, stop or wait for a purge runner. The contract uses the canonical `LargePayloadTombstone`, `LargePayloadPurgeResult`, and `LargePayloadPurgeDisposition` types from `Microsoft.DurableTask.Client`. It does not copy, move, wrap or @@ -21,7 +22,7 @@ forward those types. Backend-issued tombstone tokens must be echoed unchanged. This project references the SDK Client project directly. Its packaged dependency graph is: ```text -Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions +Microsoft.DurableTask.LargePayloadPurge.Abstractions -> Microsoft.DurableTask.Client -> Microsoft.DurableTask.Abstractions -> Microsoft.Azure.DurableTask.Core @@ -36,16 +37,9 @@ matching SDK Client and Abstractions dependencies containing the purge models be Published Client `1.26.0` predates those models and is not sufficient. Repository builds use source project references; no external Client-version bootstrap property is required. -The assembly uses this repository's strong-name key. Consumers of an earlier local prototype signed with -a different key must rebuild against the SDK-owned package; there is no compatibility promise for those -unreleased prototype binaries. The activity transport interface also moved here from the unreleased -Azure Blob implementation while retaining its full namespace, methods and optional-parameter defaults. -Rebuild consumers against this assembly; no type forwarder is provided. - -## License - -The service interface originated in [Azure/durabletask](https://github.com/Azure/durabletask) and retains -its Apache-2.0 notice; see [LICENSE](LICENSE). The activity transport interface retains its MIT notice; -see the [SDK MIT license](https://github.com/microsoft/durabletask-dotnet/blob/main/LICENSE). Both license texts -are included in the package as `LICENSE` and `licenses/MIT/LICENSE`; the package license expression -is `Apache-2.0 AND MIT`. The referenced SDK model assemblies retain their own licenses. +The assembly uses this repository's strong-name key. Consumers of the unreleased prototype package +`Microsoft.Azure.DurableTask.LargePayloadPurge.Abstractions` must update the package reference and rebuild; +the assembly name is now `Microsoft.DurableTask.LargePayloadPurge.Abstractions`. Source namespaces stay the same. +Service implementations should rename `GetLargePayloadsToPurgeAsync` to the inherited +`GetLargePayloadTombstonesAsync`, returning `Task>`; Report uses the existing +shared transport signature. No type forwarder or duplicate DTO is provided. diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityBackendStatusTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityBackendStatusTests.cs index ddc3ea91..0a5ed6c8 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityBackendStatusTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityBackendStatusTests.cs @@ -24,7 +24,7 @@ public async Task GetLargePayloadTombstones_WhenBackendUnimplemented_ThrowsNotIm // Arrange - the backend rejects the fetch RPC because it does not implement it. LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.Unimplemented, "unknown method")))); - GetLargePayloadTombstonesActivity activity = new(client, new TestLogger()); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); // Act Func act = () => activity.RunAsync(null!, 100); @@ -42,7 +42,7 @@ public async Task ReportLargePayloadPurgeResults_WhenBackendUnimplemented_Throws LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.Unimplemented, "unknown method")))); ReportLargePayloadPurgeResultsActivity activity = - new(client, new TestLogger()); + new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); List results = new() { new LargePayloadPurgeResult("tombstone-token-1", LargePayloadPurgeDisposition.Deleted), @@ -64,7 +64,7 @@ public async Task GetLargePayloadTombstones_WhenAutoPurgeDisabled_ReturnsEmptyAn TestLogger logger = new(); LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.FailedPrecondition, Detail)))); - GetLargePayloadTombstonesActivity activity = new(client, logger); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), logger); // Act List tombstones = await activity.RunAsync(null!, 100); @@ -88,7 +88,7 @@ public async Task GetLargePayloadTombstones_WhenTaskHubBeingDeleted_IsNotMislabe TestLogger logger = new(); LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.FailedPrecondition, Detail)))); - GetLargePayloadTombstonesActivity activity = new(client, logger); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), logger); // Act List tombstones = await activity.RunAsync(null!, 100); @@ -109,7 +109,7 @@ public async Task GetLargePayloadTombstones_WhenBackendCancels_ThrowsOperationCa // Arrange - cancellation is unrelated to the new precondition path and must keep its own translation. LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.Cancelled, "canceled")))); - GetLargePayloadTombstonesActivity activity = new(client, new TestLogger()); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); // Act Func act = () => activity.RunAsync(null!, 100); @@ -126,7 +126,7 @@ public async Task GetLargePayloadTombstones_WhenBackendFailsOtherwise_Propagates TestLogger logger = new(); LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.Unavailable, "backend down")))); - GetLargePayloadTombstonesActivity activity = new(client, logger); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), logger); // Act Func act = () => activity.RunAsync(null!, 100); @@ -147,7 +147,7 @@ public async Task ReportLargePayloadPurgeResults_WhenBackendCancels_ThrowsOperat LargePayloadPurgeClient client = new( new ThrowingCallInvoker(new RpcException(new Status(StatusCode.Cancelled, "canceled")))); ReportLargePayloadPurgeResultsActivity activity = - new(client, new TestLogger()); + new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); List results = new() { new LargePayloadPurgeResult("tombstone-token-1", LargePayloadPurgeDisposition.Deleted), @@ -172,7 +172,7 @@ public async Task ReportLargePayloadPurgeResults_WhenBackendFailsOtherwise_Propa RpcException error = new(new Status(statusCode, "backend failure")); LargePayloadPurgeClient client = new(new ThrowingCallInvoker(error)); TestLogger logger = new(); - ReportLargePayloadPurgeResultsActivity activity = new(client, logger); + ReportLargePayloadPurgeResultsActivity activity = new(new GrpcLargePayloadPurgeClient(client), logger); List results = new() { new LargePayloadPurgeResult("tombstone-token-1", LargePayloadPurgeDisposition.Deleted), @@ -196,7 +196,7 @@ public async Task GetLargePayloadTombstones_SetsDefaultUtcDeadlineAsync() RpcException error = new(new Status(StatusCode.DeadlineExceeded, "deadline exceeded")); ThrowingCallInvoker invoker = new(error); LargePayloadPurgeClient client = new(invoker); - GetLargePayloadTombstonesActivity activity = new(client, new TestLogger()); + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); DateTime earliestDeadline = DateTime.UtcNow.AddSeconds(60); // Act @@ -223,7 +223,7 @@ public async Task ReportLargePayloadPurgeResults_SetsDefaultUtcDeadlineAsync() ThrowingCallInvoker invoker = new(error); LargePayloadPurgeClient client = new(invoker); ReportLargePayloadPurgeResultsActivity activity = - new(client, new TestLogger()); + new(new GrpcLargePayloadPurgeClient(client), new TestLogger()); List results = new() { new LargePayloadPurgeResult("tombstone-token-1", LargePayloadPurgeDisposition.Deleted), diff --git a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityDeadlineTests.cs b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityDeadlineTests.cs index d3bb8964..b48d1d64 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityDeadlineTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityDeadlineTests.cs @@ -24,7 +24,7 @@ public async Task GetLargePayloadTombstones_WhenDeadlineExpires_CancelsRequestAs using BlockingHttpMessageHandler handler = new(); using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost", new GrpcChannelOptions { HttpHandler = handler }); LargePayloadPurgeClient client = new(channel); - GetLargePayloadTombstonesActivity activity = new(client, new TestLogger()) + GetLargePayloadTombstonesActivity activity = new(new GrpcLargePayloadPurgeClient(client), new TestLogger()) { RpcTimeout = TimeSpan.FromMilliseconds(200), }; @@ -47,7 +47,7 @@ public async Task ReportLargePayloadPurgeResults_WhenDeadlineExpires_CancelsRequ using BlockingHttpMessageHandler handler = new(); using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost", new GrpcChannelOptions { HttpHandler = handler }); LargePayloadPurgeClient client = new(channel); - ReportLargePayloadPurgeResultsActivity activity = new(client, new TestLogger()) + ReportLargePayloadPurgeResultsActivity activity = new(new GrpcLargePayloadPurgeClient(client), new TestLogger()) { RpcTimeout = TimeSpan.FromMilliseconds(200), }; diff --git a/test/Extensions/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreDeleteResponseTests.cs b/test/Extensions/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreDeleteResponseTests.cs index fa9be0ea..a0f7dd78 100644 --- a/test/Extensions/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreDeleteResponseTests.cs +++ b/test/Extensions/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreDeleteResponseTests.cs @@ -51,7 +51,7 @@ public async Task DeleteResponse_ReportsOnlyConfirmedSuccessAsync( TestLogger logger = new(); using GrpcChannel reportChannel = GrpcChannel.ForAddress("http://report.invalid", new() { HttpHandler = handler }); ReportLargePayloadPurgeResultsActivity report = new( - new LP.LargePayloadPurge.LargePayloadPurgeClient(reportChannel), + new GrpcLargePayloadPurgeClient(new LP.LargePayloadPurge.LargePayloadPurgeClient(reportChannel)), NullLogger.Instance); // Act diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj index 2cb18117..7414509c 100644 --- a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj @@ -2,7 +2,6 @@ net10.0 - DurableTask.LargePayloadPurge.Abstractions.Tests DurableTask.LargePayloadPurge.Tests diff --git a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs index 57483127..f752e084 100644 --- a/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs +++ b/test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs @@ -1,16 +1,5 @@ -// ---------------------------------------------------------------------------------- -// Copyright Microsoft Corporation -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// ---------------------------------------------------------------------------------- -// Adapted from the original contract tests for SDK xUnit, signing and dependency checks. +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. using System.Reflection; using Microsoft.DurableTask.AzureBlobPayloads; @@ -35,14 +24,15 @@ public void PackageOwnsBothInterfacesWithoutBlobDefinitionsOrForwarders() // Assert Assert.True(contract.IsInterface); Assert.Equal("DurableTask.LargePayloadPurge", contract.Namespace); - Assert.Equal("DurableTask.LargePayloadPurge.Abstractions", contract.Assembly.GetName().Name); + Assert.Equal("Microsoft.DurableTask.LargePayloadPurge.Abstractions", contract.Assembly.GetName().Name); Assert.Equal(new[] { contract, transport }.OrderBy(type => type.FullName), exported.OrderBy(type => type.FullName)); Assert.Same(contract.Assembly, transport.Assembly); Assert.True(transport.IsInterface); Assert.Equal("Microsoft.DurableTask.AzureBlobPayloads", transport.Namespace); - Assert.Empty(contract.GetInterfaces()); + Assert.Equal([transport], contract.GetInterfaces()); Assert.Empty(transport.GetInterfaces()); - Assert.Equal(3, contract.GetMethods().Length); + Assert.Equal(nameof(IOrchestrationServiceLargePayloadPurgeClient.SetLargePayloadAutoPurgeAsync), + Assert.Single(contract.GetMethods()).Name); Assert.Equal(2, transport.GetMethods().Length); Assert.DoesNotContain(blob.GetTypes(), type => type.FullName == transport.FullName); Assert.DoesNotContain(blob.GetForwardedTypes(), type => type.FullName == transport.FullName); @@ -63,11 +53,6 @@ public void SetAcceptsExplicitChoiceAndCallerDeadlineAndCancellation() public void GetReturnsCanonicalSdkTombstones() { // Arrange / Act / Assert - AssertSignature( - nameof(IOrchestrationServiceLargePayloadPurgeClient.GetLargePayloadsToPurgeAsync), - typeof(Task>), - [typeof(int), typeof(DateTime), typeof(CancellationToken)], - ["limit", "deadlineUtc", "cancellationToken"]); AssertTransportSignature( nameof(ILargePayloadPurgeClient.GetLargePayloadTombstonesAsync), typeof(Task>), @@ -79,11 +64,6 @@ public void GetReturnsCanonicalSdkTombstones() public void ReportAcceptsCanonicalSdkResults() { // Arrange / Act / Assert - AssertSignature( - nameof(IOrchestrationServiceLargePayloadPurgeClient.ReportLargePayloadPurgeResultsAsync), - typeof(Task), - [typeof(IReadOnlyList), typeof(DateTime), typeof(CancellationToken)], - ["results", "deadlineUtc", "cancellationToken"]); AssertTransportSignature( nameof(ILargePayloadPurgeClient.ReportLargePayloadPurgeResultsAsync), typeof(Task), @@ -166,6 +146,8 @@ static void AssertTransportSignature(string methodName, Type returnType, Type[] { MethodInfo method = typeof(ILargePayloadPurgeClient).GetMethod(methodName)!; Assert.NotNull(method); + Type inheritedContract = Assert.Single(typeof(IOrchestrationServiceLargePayloadPurgeClient).GetInterfaces()); + Assert.Equal(method, inheritedContract.GetMethod(methodName)); Assert.Equal(returnType, method.ReturnType); ParameterInfo[] parameters = method.GetParameters(); Assert.Equal(parameterTypes, parameters.Select(parameter => parameter.ParameterType)); From 1342f6e8d61b10b41658766076f550c80304180f Mon Sep 17 00:00:00 2001 From: wangbill Date: Thu, 1 Oct 2026 13:01:12 -0400 Subject: [PATCH 15/15] Gate purge contract publication on SDK prerequisites Require successful approval, Abstractions publication and Client publication before publishing the contract. Document conservative skipping on prerequisite failure without changing independent package jobs. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- doc/release_process.md | 4 ++++ eng/publish/publish.yml | 7 +++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/doc/release_process.md b/doc/release_process.md index bc83dcc2..116d71f9 100644 --- a/doc/release_process.md +++ b/doc/release_process.md @@ -18,6 +18,10 @@ Its package and assembly name is `Microsoft.DurableTask.LargePayloadPurge.Abstra standard `Microsoft.DurableTask.*.dll` signing pattern. The existing source traversal, SBOM inclusion, NuGet signing and per-package approval-gated publication steps apply. +Contract publication also waits for successful Client and Abstractions publication. If either prerequisite +fails, including a duplicate-version upload failure, or is skipped or canceled, contract publication is +skipped rather than treating that result as success. Other packages retain their independent publication jobs. + Keep its `RELEASENOTES.md`: `eng/targets/Release.targets` reads it into NuGet package metadata, whereas the root `CHANGELOG.md` retains repository release history. The shared target also appends a link using the package's own version (`releases/tag/v0.1.0` for its initial release). Verify or create the corresponding diff --git a/eng/publish/publish.yml b/eng/publish/publish.yml index 8c3ee77a..5f2c9a60 100644 --- a/eng/publish/publish.yml +++ b/eng/publish/publish.yml @@ -468,8 +468,11 @@ extends: # NuGet release (Microsoft.DurableTask.LargePayloadPurge.Abstractions) - job: nugetRelease_Microsoft_DurableTask_LargePayloadPurge_Abstractions displayName: NuGet Release (Microsoft.DurableTask.LargePayloadPurge.Abstractions) - dependsOn: nugetApproval - condition: succeeded('nugetApproval') + dependsOn: + - nugetApproval + - nugetRelease_Microsoft_DurableTask_Abstractions + - nugetRelease_Microsoft_DurableTask_Client + condition: succeeded() templateContext: type: releaseJob isProduction: true