Epic: staged private per-step queues and queue identity

[jumski]

Outcome

Deliver private per-step queues while preserving plain one-flow/one-queue execution:

one concrete flow
  -> one private queue per step
  -> one independently configured worker pool per step

This preserves one typed DAG and one run while removing cross-step starvation. This epic coordinates scope, sequencing, application validation, and the combined release. It is not another implementation task.

Issue map

Issue	Job	Release relationship
#650	Persist queue identity; retain one queue per flow	Required foundation
#651	Add withStepQueues() and typed workers; one generated queue per step	Required feature
#653	Validate the combined candidate and coordinate release	This epic

Additional open issues are separate work:

Issue	Job	Relationship
#652	Explicit routing, including multiple steps or flows on one queue	Deferred until grouping/sharing has a concrete use case
#648	Stable aliases for concrete flow versions	Deferred until aliases have a concrete use case; not required by #652
#655	Handler-selected delay for one retry	Independent application parity improvement; attempt accounting stays unchanged
#661	Design spike for queue-less manual completion	Deferred design, not a queue-release requirement
#678	Targeted database enforcement of identity immutability	Independent enhancement, not a queue-release requirement

The current release does not include grouped/shared queues. #652 retains that capability; it is not restricted to one step per queue. Neither grouping nor multi-flow dispatch needs stable aliases first.

Start #648 or #652 only after Stage 2 validation records the relevant concrete need. Older application plans cannot add aliases, shared queues, manual tasks, or identity triggers to #650/#651.

Released prerequisites

The task lifecycle prerequisites shipped in 0.15.1 through #666:

• Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638 — skipped paths terminalize task rows, fixed by Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #649.
• Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645 — failed runs terminalize unfinished tasks; recovery ignores terminal runs/steps, fixed by fix(core): terminalize unfinished tasks when runs fail #663.
• requeue_stalled_tasks() uses flow timeout instead of effective step timeout #621 — stalled recovery uses the effective step timeout, fixed by fix(core): use effective step timeout for stalled tasks #664.
• Make start_tasks() apply task visibility before handlers run #656 — start_tasks() applies visibility before handlers run, fixed by fix(core): guarantee task visibility extension in start_tasks() #667.

Startup and rollout prerequisites:

• Remove legacy flow compilation before queue identity and routing #647 — startup-only compilation replaced ControlPlane, migration-based flow compilation, and optional worker compilation; implemented by refactor: make worker startup the only flow compilation path #672 and released in 0.16.0 through Version Packages #674.
• Document the worker-function enable fence for production deployments #654 — documented the current in-place worker replacement procedure, fixed by docs: fence worker_functions row before deprecating workers in update guide #670.

Preserve these behaviors. The queue work starts from the released 0.16.0 baseline and does not restore a legacy compiler.

Stage 1 — persisted queue identity

• Complete Persist physical queue identity for flow tasks #650 against its approved issue body.

#650 is the source of truth for Stage 1 implementation and acceptance criteria. It persists step routes and task snapshots, keeps nullable message IDs, and uses (queue_name, message_id) for task message identity. It preserves existing slug spelling/validation except normalized default-flow uniqueness and keeps practical existing call signatures.

Dispatch uses the step route; subsequent message operations use the task snapshot. Runtime code leaves snapshots unchanged. Database-enforced immutability is separate in #678.

Implement from main using the approved requirements, without the rejected implementation branch or its old plan as implementation references.

Stage 2 — private per-step queues

• Complete Add private per-step queues with typed step workers #651 against its current issue body.

#651 is the source of truth for the public API, naming, type contract, route-map checks, step selection, and rollout criteria:

plain Flow
  every step uses lower(flow.slug)
  EdgeWorker.start(flow, config)
  no selector

withStepQueues(flow)
  every step has its own generated private queue
  EdgeWorker.start(wrapped, { stepSlug, ...config })
  exact typed step selector

#651 owns the additional shared naming restrictions, their focused migration checks, and documentation for plain and step-queued flows. Only queue names normalize to lowercase; accepted flow/step spelling and exact references remain intact. Generated queues use the concrete slug and the documented 47-character readable-name/index-fallback algorithm.

Queue mode and routes are deployment metadata, separate from DAG shape. Actual production shape/mode/route mismatches fail without destructive recompilation. Existing local recompilation behavior remains supported. Do not introduce a general protocol version merely to reject released workers.

Simplicity and correctness boundary

These rules also apply to deferred linked work; follow-up issues must not restore the scope rejected from #650:

• Read once and use stored identity. Extend existing claims with the selected flow/step or support set. Only successfully claimed eligible tasks reach handlers. Unknown/unsupported messages stay intact, warn with identifiers, and do not stop valid work. Recurrence after normal visibility timeout is accepted.
• Trust PGMQ operations. Use public creation, listing, validation, and deletion APIs, including pgmq.list_queues() for original metadata spelling. Retain existing set_vt_batch() and direct archive pruning. No physical-object inspection, pgmq.meta lock, queue-body reread, or adoption scan is needed.
• Keep real safeguards. Retain pgflow locks, foreign keys, uniqueness, input validation, terminal guards, exact message IDs, attempt accounting, and established visibility/recovery timing. Keep transaction rollback and flow-isolated cleanup. Avoid duplicate prechecks when a constraint already rejects the same write atomically, and avoid unrelated lifecycle/lock rewrites.
• Limit migration work. Use schema-first transactional migrations, bounded lock waits, focused new-constraint checks, and no automatic renaming or repair. Preserve released migrations. No general audit system or forced mixed-version rejection is required.
• Keep optional mechanisms separate. No body classifier, automatic foreign-message archival, batch-wide fatal shutdown, forced unknown-message visibility reset, persistent HTTP restart pause, global coverage coordinator, or unconditional identity trigger. Enforce immutability of persisted pgflow identities #678 owns narrowly scoped identity enforcement when separately undertaken.

pgflow manages its queues exclusively. Applications must not send unrelated messages or independently create, replace, or alter these resources. Reject visible external-name collisions and conflicting pgflow routes, but do not promise protection against concurrent external PGMQ changes.

Keep unknown data intact. Dropping a private queue with its flow needs no preceding per-message archival. Future shared queues are never dropped with one flow; #652 owns their minimal metadata and flow-isolated cleanup, not #650.

Application validation

• Validate one concrete StepQueuedFlow in a downstream application or focused application spike before stable release.

Choose the application before this gate, not before #650. Begin after the first working private-queue slice. A two-step internal E2E test gives early feedback but does not replace application-level validation.

community-threads#27 is the linked application spike. It uses #650 + #651's generated private queues, without aliases or shared-queue prerequisites. It is a candidate, not the only permitted validation application.

Preserve application domain tables, transitions, claims, and terminal outcomes while replacing queue/worker mechanics. Record:

• orchestration code and direct PGMQ references removed and retained;
• independent capacity and starvation behavior;
• success, retry, crash, ambiguous input, and concurrency behavior, including recovery gaps;
• type/autocomplete quality and any missing application parity;
• an explicit adopt, revise, or stop recommendation for Stage 2.

If the application needs provider-selected Retry-After, implement #655's minimum override before claiming retry parity, or explicitly report the gap. Do not change attempt accounting or treat that gap as evidence that private queue placement failed.

An adopt result permits release. A revise result adds only the missing scoped capability and repeats affected validation. A stop result releases neither #650 nor #651; remove unreleased queue changes while retaining the released #647 baseline. New aliases or sharing proposals still need their own concrete justification.

Rollout and upgrade

• Finish Add private per-step queues with typed step workers #651's per-step production rollout documentation before stable release.

Use #654's existing replacement procedure across the affected function set. Disable and drain existing workers, deploy replacements, and restore only previously enabled functions. For a new concrete version, start all selected step workers and check queue coverage before switching callers; retain old workers until no old run can produce work and no executable/recoverable old task remains.

Missing workers leave work queued durably. #651 supplies a copyable operator coverage query; startup logs describe only the selected worker. Do not build a cross-worker readiness or activation system.

Document a coordinated maintenance upgrade from 0.16.0: stop and drain workers, pause producers and definition/maintenance/recovery writers, apply migrations through Supabase's runner, replace or adapt any manually installed pruning helper, deploy matching packages, and resume. No old/new rolling-worker promise is required. An edited pruning snippet does not update an installed or customized copy automatically.

Delivery and combined release gate

Follow the repository's current workflow: #650 implementation, review, and merge, then #651 implementation, review, and merge. Use one issue and one editing implementer at a time, with the repository's focused, batch, and final check cadence.

0.16.0 baseline
  -> #650 queue identity and focused upgrade checks
  -> #651 private per-step queues and typed workers
  -> application validation after the first working slice
  -> close scoped gaps and finish rollout docs
  -> finalize the queue migration and freeze the combined candidate
  -> validate that exact candidate downstream and run final checks
  -> adopt: release
     revise: make the scoped change and repeat affected validation
     stop: remove unreleased #650/#651 changes; retain released #647

#650 and #651 each add the appropriate Changeset. The generated Version Packages PR stays unmerged until the complete gate passes. #650 may remain operational if #651 is delayed, but it does not publish alone. Follow the temporary-migration workflow if intermediate migrations are needed; finalize/consolidate them before exact-candidate validation and never rewrite a released migration.

Epic acceptance criteria

• Persist physical queue identity for flow tasks #650 and Add private per-step queues with typed step workers #651 meet their own current descriptions and acceptance criteria without adding deferred follow-up features.
• The populated 0.16.0 upgrade, new-constraint rollback, and plain-flow startup/execution checks pass alongside per-step behavior and required affected checks.
• Downstream validation records its recommendation against the exact combined candidate, including the final migration artifact.
• Maintenance, pruning-helper replacement, per-step coverage, and old-version drain instructions are complete.
• The stable release remains blocked until implementation, review, validation, rollout docs, and final checks finish; deferred issues are not extra release blockers.