Lessons from building durable Workflows on Duroxide: sub-orchestration handles that survive select losers and continue-as-new, plus six related runtime changes #79

Summary

While building durable Workflows into PilotSwarm (microsoft/PilotSwarm#270) we ended up not using sub-orchestrations for the per-run controller, and instead built a "PostgreSQL is the truth, the queue is a hint" control plane with one detached helper orchestration per step. This issue explains why, with source citations at main = 36b789d (0.1.32 + #74; identical at v0.1.32 for every file cited), and proposes a set of Duroxide changes that would let an eternal controller orchestration keep children and state in the orchestration itself.

The controller is an event loop: poll commands, admit steps, elapse deadlines, collect results, route, and continue_as_new every N passes. It must therefore (a) drop futures regularly (every select loser is a drop) and (b) roll its history over regularly. At 0.1.32 both of those are fatal for a sub-orchestration:

1. a child's future is the only handle to the child;
2. dropping that future cancels the child (CancelInstance is enqueued);
3. the handle cannot cross continue-as-new: the child's completion is routed to the execution that scheduled it, and the successor drops it.

Seven further engine limits then forced the store-as-truth design. The result on the PilotSwarm side at the PR head: a 5.6k-line controller, a 7.8k-line store, 20 workflow_* tables, six helper orchestration kinds, and a 30 s backstop timer on every wait. Most of that is control plane an orchestration runtime should own.

What we hit (with evidence)

I1 + I2 together are decisive. A long-lived parent has exactly three ways to stop awaiting a child: let the future lose a select (I1 cancels the child), let it go out of scope (same), or continue-as-new (I2 orphans it). There is no fourth way. The "Durable Supervisor" sketch in docs/proposals/durable-actors.md — a supervisor that carries children across continue_as_new — is exactly the pattern we needed, and it cannot run on the current runtime for these two reasons.

Proposals

Ordered by how much application code each removes. P1 is the architectural one; P2–P6 are small and independently shippable.

P1 — Re-awaitable sub-orchestration handles (survive select losers and CAN)

Goal. A parent can start a child, stop waiting for it (select loser, loop iteration, CAN), and wait for it again later — in any execution — without the child being cancelled and without losing its result. Cancellation becomes explicit.

// Existing: unchanged; cancel-on-drop preserved for compatibility.
let fut: DurableFuture<Result<String, String>> = ctx.schedule_sub_orchestration("Step", input);

// New: turn the future into a durable handle. Dropping the handle does nothing.
// SubOrchestrationHandle is serde-serializable: { child_instance: String }.
let h: SubOrchestrationHandle = fut.detach();

// Or start linked-but-not-awaited directly.
let h = ctx.start_sub_orchestration_linked("Step", input);              // auto id
let h = ctx.start_sub_orchestration_linked_with_id("Step", id, input);  // explicit id

// Wait (again) for a child, in this or any later execution.
// Dropping THIS future cancels only the wait, never the child.
let fut2 = ctx.await_sub_orchestration(&h);

// Explicit lifecycle.
ctx.cancel_sub_orchestration(&h);                                       // CancelInstance; idempotent
let st: Option<SubOrchestrationTerminal> = ctx.sub_orchestration_result(&h); // replay-recorded; no turn

// Across CAN: handles are plain data in the CAN input, and the runtime also
// carries outstanding linked children automatically (below).
ctx.continue_as_new_typed(&State { children: vec![h], .. });

History and routing:

Versioning: execution-pinned exactly like the 0.1.31 queue-race fix (RACE_CANCELLATION_FIX_SINCE, replay_engine.rs:20,560). New event kinds and work-item fields are additive; providers store events and work items as JSON, so no schema migration is expected (to be verified against duroxide-pg's work-item columns).

Tests (repository-native):

P2 — N-way select, nesting, deadline timers

• Core: ctx.select(Vec<DurableFuture<T>>) -> (usize, T) plus a typed SelectAny over mixed outputs (enum-wrapped), polled in index order for determinism (same bias rule as Select2, src/combinators.rs:193-212). Allow select/join nesting (box the inner).
• Node: lift the "exactly 2 tasks" and "no nesting" rejections in handlers.rs:497-543; the JS signature is already variadic.
• schedule_timer_until(deadline_ms) (Add an absolute-deadline timer variant (e.g. schedule_timer_until(deadline)) #34), so a controller does not need utc_now to convert absolute deadlines.

P3 — Lossless continue-as-new

1. Fix Detached start (schedule_orchestration) is dropped when the same turn ends in continue_as_new #78 and its sub-orchestration twin: in TurnResult::ContinueAsNew (execution.rs:300-371) call enqueue_detached_from_pending and enqueue pending StartSubOrchestration actions (as linked children of the successor) before pushing the ContinueAsNew item.
2. Spill instead of truncate at execution.rs:329-339: arrivals beyond MAX_CARRY_FORWARD_EVENTS are re-enqueued as WorkItem::QueueMessage for the instance (the dispatcher already defers items it cannot yet see across the CAN window), preserving order. The constant becomes a per-item payload bound, not a loss point. If spill is unwanted, fail the CAN with an Infrastructure error — never drop silently.

P4 — Messages to unstarted instances; bounded batch dequeue

• Do not ack QueueMessage / ExternalRaised for an instance with no history (orchestration.rs:676) while no StartOrchestration has been seen, up to a provider-side TTL. Report the outcome to the sender: Delivered | Buffered | Expired.
• ctx.dequeue_events(name, max) -> DurableFuture<Vec<String>> (one subscription, up to max arrivals), plus try_dequeue_event (Add tryDequeueEvent(queueName) to orchestration context #8) and queue_depth(name) (Expose event queue depth as orchestration-visible state #9). Together these make hint draining one bounded call and keep the quiescence bound (I7) out of reach.

P5 — Cancel dropped timers at the provider

ScheduleKind::Timer drop → WorkItem::CancelTimer { instance, execution_id, id } → the provider deletes the delayed orchestrator-queue row. If the row is already claimed, fall back to today's execution-id/duplicate filtering. Advertise as a provider capability; keep the no-op path for providers without it.

P6 — Small ergonomics

What the controller looks like after P1–P3

loop {
  state  = CAN input: cursors, parallels, ordinals, Vec<SubOrchestrationHandle>
  waits  = children.map(await_sub_orchestration)
         + [ dequeue_events(commands, 32), schedule_timer_until(nearest_deadline) ]
  (i, out) = select(waits)                        // losers keep running
  match out { child result => route, command => apply, timer => elapse }
  if history is large { continue_as_new(state) }  // children and unread commands are carried
}

Store writes become projections for the UI/API. No helper orchestrations, no wake hints, no backstop timer, no compare-and-set as the decision record.

Design choices to settle

1. Routing policy for legacy children. Route all children to the parent's latest execution under the new policy, or only linked ones? Routing all is simpler and matches "a parent is one instance"; keeping the stamped execution for ordinary schedule_sub_orchestration futures preserves today's determinism guarantees exactly. Our preference: route all under the new execution-pinned policy, since a completion that can only ever be dropped is not a useful guarantee.
2. Default at parent terminal. Cancel outstanding linked children (proposed) or leave them running? Cancel matches held-future semantics and avoids leaks; _detached is the explicit opt-out.
3. Spill vs fail at the carry cap. Spill preserves work and order; fail makes the bound visible. Either is acceptable; silent truncation is not.
4. Where unconsumed child results live after CAN. Seeded into the successor's history (proposed, mirrors unread queue arrivals) vs. re-enqueued as work items. Seeding keeps them replay-visible on the first turn.
5. Work-item schema in duroxide-pg. Confirm that SubOrchCompleted/SubOrchFailed gaining child_instance needs no migration.

Suggested sequencing

Related: #78, #34, #8, #9, #38, #21. We are happy to draft P3 and the P1 proposal doc plus failing tests as PRs if the direction is agreed.