From 01147fec6840b8c2b0ed7e4152e01cf4c86e8876 Mon Sep 17 00:00:00 2001 From: Agent Date: Sun, 13 Sep 2026 06:51:32 +0000 Subject: [PATCH] feat(core): persist physical queue identity for flow tasks Persist canonical queue routes on steps and immutable-by-runtime task snapshots so queue-scoped message IDs cannot cross task identity. Route dispatch, claims, lifecycle cleanup and optional pruning through stored routes while preserving concurrency, terminal-state and visibility safeguards. Require the canonical queue_name argument in SQL and TypeScript claims and remove the released three-argument overload. PGMQ already normalizes message-operation names, so remove hot-path spelling resolution, session caches and worker lookup plumbing. Keep public queue listing for provisioning collisions and fresh destructive deletion only. Backfill populated 0.16.0 databases before enforcing constraints, with bounded lock waits and atomic rollback on conflicts. Document the breaking maintenance upgrade: pause producers, save worker states, drain active work, stop workers and other writers, migrate, deploy matching packages and restore prior states. Queued messages remain intact; no queue recreation or mixed-version rolling upgrade. Add identity, isolation, no-listing, required-signature and populated-upgrade regressions; retain the new integration test with awaited startup from the parent harness fix. Closes #650. --- .changeset/persist-queue-identity.md | 11 + ARCHITECTURE_GUIDE.md | 9 +- pkgs/client/__tests__/helpers/polling.ts | 5 +- pkgs/core/README.md | 3 +- .../__tests__/types/PgflowSqlClient.test-d.ts | 19 +- pkgs/core/assets/flow-lifecycle.mermaid | 2 +- pkgs/core/assets/flow-lifecycle.svg | 2 +- pkgs/core/schemas/0030_utilities.sql | 17 + .../0035_function_listed_queue_name.sql | 47 + pkgs/core/schemas/0050_tables_definitions.sql | 9 + pkgs/core/schemas/0060_tables_runtime.sql | 15 +- .../0062_function_requeue_stalled_tasks.sql | 14 +- ...100_function__cascade_force_skip_steps.sql | 9 +- pkgs/core/schemas/0100_function_add_step.sql | 10 +- .../0100_function_archive_task_message.sql | 9 +- ...00_function_cascade_resolve_conditions.sql | 29 +- .../schemas/0100_function_complete_task.sql | 77 +- .../schemas/0100_function_create_flow.sql | 76 +- .../0100_function_delete_flow_and_data.sql | 34 +- .../0100_function_ensure_flow_compiled.sql | 4 +- pkgs/core/schemas/0100_function_fail_task.sql | 85 +- .../0100_function_start_ready_steps.sql | 19 +- .../schemas/0120_function_start_tasks.sql | 18 +- .../benchmarks/step_output_storage.sql | 10 +- pkgs/core/scripts/run-upgrade-fixture | 151 +- pkgs/core/src/PgflowSqlClient.ts | 8 +- pkgs/core/src/database-types.ts | 18 +- pkgs/core/src/types.ts | 24 +- ...13093141_pgflow_persist_queue_identity.sql | 2009 +++++++++++++++++ pkgs/core/supabase/migrations/atlas.sum | 3 +- pkgs/core/supabase/seed.sql | 7 +- ...s_task_messages_for_skipped_steps.test.sql | 2 +- .../idempotent_second_call.test.sql | 2 +- .../_shared/prune_data_older_than.sql.raw | 50 +- .../add_step/step_index_uniqueness.test.sql | 4 +- .../no_cascade_on_failed_run.test.sql | 2 +- ..._skip_does_not_mutate_step_or_run.test.sql | 4 +- .../no_mutations_on_failed_run.test.sql | 2 +- .../skipped_deps_excluded_from_input.test.sql | 2 +- .../queue_name_collisions.test.sql | 108 + .../drops_queues_by_routes.test.sql | 70 + .../guards_missing_flow_queue.test.sql | 68 + ...ecreate_resolves_fresh_after_drop.test.sql | 66 + .../rejects_ambiguous_queue_match.test.sql | 59 + .../archive_sibling_map_tasks.test.sql | 4 +- ...ask_late_callbacks_are_idempotent.test.sql | 4 +- .../late_callbacks_post_lock_race.test.sql | 2 +- ..._double_decrement_remaining_steps.test.sql | 4 +- .../skip_archives_sibling_messages.test.sql | 4 +- .../prune_deletes_all_child_statuses.test.sql | 3 +- .../prune_uses_task_snapshots.test.sql | 106 + .../message_id_identity.test.sql | 86 + .../message_paths_skip_queue_listing.test.sql | 93 + .../mixed_case_queue_lifecycle.test.sql | 155 ++ .../multi_queue_lifecycle.test.sql | 202 ++ .../queue_identity/queue_isolation.test.sql | 135 ++ .../queue_identity/snapshot_creation.test.sql | 69 + ...apshot_unchanged_across_lifecycle.test.sql | 120 + .../start_tasks/basic_start_tasks.test.sql | 4 +- .../start_tasks/benign_duplicate.test.sql | 6 +- ...ds_proper_input_from_deps_outputs.test.sql | 4 +- .../conditional_flow_input.test.sql | 12 +- .../dependent_map_element_extraction.test.sql | 6 +- ..._not_start_tasks_for_skipped_step.test.sql | 4 +- .../start_tasks/map_large_array.test.sql | 8 +- .../start_tasks/map_mixed_types.test.sql | 10 +- .../start_tasks/map_nested_arrays.test.sql | 8 +- .../start_tasks/map_object_elements.test.sql | 8 +- ...task_creation_scaling_performance.test.sql | 8 +- .../start_tasks/map_to_map_chain.test.sql | 8 +- .../multiple_task_processing.test.sql | 2 +- .../start_tasks/returns_flow_input.test.sql | 8 +- .../returns_only_claimed_tasks.test.sql | 2 +- .../start_tasks/returns_task_index.test.sql | 10 +- .../root_map_element_extraction.test.sql | 6 +- ..._tasks_input_assembly_performance.test.sql | 2 +- .../started_at_timestamps.test.sql | 2 +- .../start_tasks/status_transitions.test.sql | 2 +- .../task_index_returned_correctly.test.sql | 2 +- .../visibility_failure_rollback.test.sql | 4 +- .../start_tasks/visibility_timeout.test.sql | 2 +- .../start_tasks/worker_tracking.test.sql | 6 +- .../cancels_unfinished_tasks.test.sql | 4 +- .../upgrade_fixture/assertions_0_16.sql | 194 ++ .../supabase/upgrade_fixture/seed_0_16.sql | 43 + .../upgrade_fixture/seed_0_16_conflict.sql | 6 + pkgs/dsl/README.md | 4 +- .../types/context-inference.test-d.ts | 2 +- .../supabase-context-inference.test-d.ts | 2 +- pkgs/dsl/src/dsl.ts | 6 +- pkgs/edge-worker/README.md | 4 +- pkgs/edge-worker/src/core/context.ts | 2 +- pkgs/edge-worker/src/core/types.ts | 4 +- .../src/flow/FlowWorkerLifecycle.ts | 5 +- pkgs/edge-worker/src/flow/StepTaskExecutor.ts | 4 +- pkgs/edge-worker/src/flow/StepTaskPoller.ts | 36 +- pkgs/edge-worker/src/flow/createFlowWorker.ts | 8 +- pkgs/edge-worker/src/queue/Queue.ts | 6 +- .../integration/flow/queueIdentity.test.ts | 112 + .../messageExecutorContext.test.ts | 10 +- ...messageExecutorContextWorkerConfig.test.ts | 2 +- .../stepTaskExecutorContext.test.ts | 30 +- .../tests/unit/Poller.batchSize.test.ts | 60 + .../tests/unit/contextUtils.test.ts | 8 +- .../tests/unit/workerConfigContext.test.ts | 8 +- pkgs/example-flows/src/example-flow.ts | 2 +- .../src/content/docs/concepts/data-model.mdx | 9 +- .../src/content/docs/deploy/prune-records.mdx | 6 +- .../src/content/docs/deploy/update-pgflow.mdx | 70 +- .../src/content/docs/get-started/faq.mdx | 6 +- ...gflow-0-17-0-persistent-queue-identity.mdx | 37 + .../docs/reference/configuration/worker.mdx | 2 +- .../src/content/docs/reference/context.mdx | 4 +- 113 files changed, 4629 insertions(+), 380 deletions(-) create mode 100644 .changeset/persist-queue-identity.md create mode 100644 pkgs/core/schemas/0035_function_listed_queue_name.sql create mode 100644 pkgs/core/supabase/migrations/20260913093141_pgflow_persist_queue_identity.sql create mode 100644 pkgs/core/supabase/tests/create_flow/queue_name_collisions.test.sql create mode 100644 pkgs/core/supabase/tests/delete_flow_and_data/drops_queues_by_routes.test.sql create mode 100644 pkgs/core/supabase/tests/delete_flow_and_data/guards_missing_flow_queue.test.sql create mode 100644 pkgs/core/supabase/tests/delete_flow_and_data/recreate_resolves_fresh_after_drop.test.sql create mode 100644 pkgs/core/supabase/tests/delete_flow_and_data/rejects_ambiguous_queue_match.test.sql create mode 100644 pkgs/core/supabase/tests/maintenance/prune_uses_task_snapshots.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/message_id_identity.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/message_paths_skip_queue_listing.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/mixed_case_queue_lifecycle.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/multi_queue_lifecycle.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/queue_isolation.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/snapshot_creation.test.sql create mode 100644 pkgs/core/supabase/tests/queue_identity/snapshot_unchanged_across_lifecycle.test.sql create mode 100644 pkgs/core/supabase/upgrade_fixture/assertions_0_16.sql create mode 100644 pkgs/core/supabase/upgrade_fixture/seed_0_16.sql create mode 100644 pkgs/core/supabase/upgrade_fixture/seed_0_16_conflict.sql create mode 100644 pkgs/edge-worker/tests/integration/flow/queueIdentity.test.ts create mode 100644 pkgs/website/src/content/docs/news/pgflow-0-17-0-persistent-queue-identity.mdx diff --git a/.changeset/persist-queue-identity.md b/.changeset/persist-queue-identity.md new file mode 100644 index 000000000..48be3ef40 --- /dev/null +++ b/.changeset/persist-queue-identity.md @@ -0,0 +1,11 @@ +--- +'@pgflow/core': minor +'@pgflow/dsl': minor +'@pgflow/client': minor +'@pgflow/edge-worker': minor +'pgflow': minor +--- + +Persist physical queue identity on steps and tasks. A queued task's message identity is now `(queue_name, message_id)`, not `message_id` alone, preparing pgflow for private per-step queues while keeping one-flow/one-queue behavior. + +`pgflow.steps` and `pgflow.step_tasks` gain a canonical lowercase `queue_name` (snapshot at task creation), `(queue_name, message_id)` is unique per queue, and two flows can no longer share a normalized default queue. **Breaking:** `pgflow.start_tasks()` now requires the `queue_name` argument - the queue's canonical identity, `lower(flow_slug)` today - and the released three-argument form and the NULL default are gone, and `startTasks()` on `IPgflowClient`/`PgflowSqlClient` requires the queue argument as well. pgflow's own workers poll and claim through that canonical name; custom callers must pass it explicitly. There is no mixed-version rolling upgrade: stop and drain workers, pause producers and definition/maintenance/recovery writers, apply the transactional migration through Supabase's migration runner against the production database (`--linked` or `--db-url`, not the local default), replace the optional `prune_data_older_than()` helper, then deploy matching packages and workers together, restoring the exact worker `enabled` states recorded before the window (see the 0.17.0 upgrade guide). Message ids are exact decimal strings at the JavaScript boundary. Existing mixed-case queue names keep working through their original pgmq spelling - PGMQ's public message API normalizes names, so no message or queue migration is needed. diff --git a/ARCHITECTURE_GUIDE.md b/ARCHITECTURE_GUIDE.md index 44c9f400b..3783a8848 100644 --- a/ARCHITECTURE_GUIDE.md +++ b/ARCHITECTURE_GUIDE.md @@ -172,7 +172,7 @@ export default CompleteExample; **Critical Cross-Cutting Concepts**: -1. **Two-Phase Polling** - Worker calls `read_with_poll()` then `start_tasks(workerId)` to prevent race conditions +1. **Two-Phase Polling** - Worker calls `read_with_poll()` then `start_tasks(workerId, queue_name)` to prevent race conditions 2. **Empty Array Cascade** - When `initial_tasks=0`, `cascade_complete_taskless_steps()` completes entire dependent chain in one transaction 3. **Map Step `initial_tasks` Lifecycle**: - Root maps: Set at flow start from input array length @@ -204,7 +204,7 @@ export default CompleteExample; 2. Main loop: - `sendHeartbeat()` - Update status, check deprecation - If deprecated → exit gracefully - - Two-phase polling: `readMessages()` then `startTasks(workerId)` + - Two-phase polling: `readMessages()` then `startTasks(workerId, queueName)` - Execute handlers (up to `maxConcurrent` parallel) - `complete_task()` or `fail_task()` 3. On shutdown: @@ -228,8 +228,7 @@ const supabase = createClient( // Create worker with all configuration options const worker = createFlowWorker(supabase, MyFlow, { - // Queue configuration - queueName: 'tasks', // Default: 'tasks' + // The worker polls the flow's canonical queue: lower(flow_slug) // Polling configuration maxPollSeconds: 2, // Default: 2 @@ -313,7 +312,7 @@ await worker.start(); **How**: - Phase 1: Worker calls `read_with_poll()` - reserves messages, returns `msg_id`s -- Phase 2: Worker calls `start_tasks(flow_slug, msg_ids, workerId)` - creates `step_tasks`, returns details +- Phase 2: Worker calls `start_tasks(flow_slug, msg_ids, worker_id, queue_name)` - creates `step_tasks`, returns details **See**: - Worker implementation: `/pkgs/edge-worker/src/worker/FlowWorkerLifecycle.ts` diff --git a/pkgs/client/__tests__/helpers/polling.ts b/pkgs/client/__tests__/helpers/polling.ts index f86a95eab..eeef77b4d 100644 --- a/pkgs/client/__tests__/helpers/polling.ts +++ b/pkgs/client/__tests__/helpers/polling.ts @@ -53,9 +53,10 @@ export async function readAndStart( return []; } - // 4. Start the tasks and return the resulting rows + // 4. Start the tasks and return the resulting rows. The claim receives the + // queue this helper read from (the canonical lowercase flow slug) const msgIds = messages.map(m => m.msg_id); - const tasks = await sqlClient.startTasks(flowSlug, msgIds, workerId); + const tasks = await sqlClient.startTasks(flowSlug, msgIds, workerId, flowSlug.toLowerCase()); return tasks; } \ No newline at end of file diff --git a/pkgs/core/README.md b/pkgs/core/README.md index 2b48363d1..291333316 100644 --- a/pkgs/core/README.md +++ b/pkgs/core/README.md @@ -288,7 +288,8 @@ SELECT * FROM pgmq.read_with_poll( SELECT * FROM pgflow.start_tasks( flow_slug => 'analyze_website', msg_ids => ARRAY[101, 102, 103], -- message IDs from phase 1 - worker_id => '550e8400-e29b-41d4-a716-446655440000'::uuid + worker_id => '550e8400-e29b-41d4-a716-446655440000'::uuid, + queue_name => 'analyze_website' -- the queue's canonical name: lower(flow_slug), the exact spelling tasks store (#650) ); ``` diff --git a/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts b/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts index 6c8028966..5c59c03ea 100644 --- a/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts +++ b/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts @@ -26,7 +26,7 @@ describe('PgflowSqlClient Type Compatibility with Flow', () => { // Check startTasks method types expectTypeOf(client.startTasks).toBeFunction(); expectTypeOf(client.startTasks).parameters.toMatchTypeOf< - [string, number[], string] + [string, string[], string, string] >(); expectTypeOf(client.startTasks).returns.toEqualTypeOf< Promise[]> @@ -66,19 +66,22 @@ describe('PgflowSqlClient Type Compatibility with Flow', () => { const client = new PgflowSqlClient(sql); // Valid calls should compile - client.startTasks('flow_slug', [1, 2, 3], 'worker-id'); - client.startTasks('flow_slug', [], 'worker-id'); + client.startTasks('flow_slug', ['1', '2', '3'], 'worker-id', 'flow_slug'); + client.startTasks('flow_slug', [], 'worker-id', 'flow_slug'); + + // @ts-expect-error - queueName is required (#650): no default queue fallback + client.startTasks('flow_slug', ['1'], 'worker-id'); // @ts-expect-error - flowSlug must be string - client.startTasks(123, [1, 2, 3], 'worker-id'); + client.startTasks(123, ['1', '2', '3'], 'worker-id', 'flow_slug'); - // @ts-expect-error - msgIds must be number array - client.startTasks('flow_slug', ['1', '2', '3'], 'worker-id'); + // @ts-expect-error - msgIds must be string array (exact decimal strings, #650) + client.startTasks('flow_slug', [1, 2, 3], 'worker-id', 'flow_slug'); // @ts-expect-error - msgIds must be array - client.startTasks('flow_slug', 123, 'worker-id'); + client.startTasks('flow_slug', 123, 'worker-id', 'flow_slug'); // @ts-expect-error - workerId must be string - client.startTasks('flow_slug', [1, 2, 3], 123); + client.startTasks('flow_slug', ['1', '2', '3'], 123, 'flow_slug'); }); }); diff --git a/pkgs/core/assets/flow-lifecycle.mermaid b/pkgs/core/assets/flow-lifecycle.mermaid index bd77de5ee..ddd174c15 100644 --- a/pkgs/core/assets/flow-lifecycle.mermaid +++ b/pkgs/core/assets/flow-lifecycle.mermaid @@ -24,7 +24,7 @@ sequenceDiagram PGMQ-->>Worker: Return messages deactivate PGMQ - Worker->>pgflow: start_tasks(flow_slug, msg_ids, worker_id) + Worker->>pgflow: start_tasks(flow_slug, msg_ids, worker_id, queue_name) activate pgflow pgflow->>pgflow: Find step_tasks with matching message_ids pgflow->>pgflow: Mark tasks as 'started' with worker_id diff --git a/pkgs/core/assets/flow-lifecycle.svg b/pkgs/core/assets/flow-lifecycle.svg index 39a0355bc..74d6936a9 100644 --- a/pkgs/core/assets/flow-lifecycle.svg +++ b/pkgs/core/assets/flow-lifecycle.svg @@ -1 +1 @@ -Task HandlerEdge WorkerPGMQ Queuepgflow SQL CoreClientTask HandlerEdge WorkerPGMQ Queuepgflow SQL CoreClientTwo-Phase PollingTask ExecutionTask succeedsHandler throws or exceeds timeoutRetries remainingWorker attempts execution againNo retries remainingPermanent failure of a runcreate_flow(...)add_step(...)start_flow(...)Create run recordInitialize step_statesCreate step_tasks for root stepsEnqueue message for root step taskReturn run detailsread_with_poll(queue_name, vt, qty)Return messagesstart_tasks(flow_slug, msg_ids, worker_id)Find step_tasks with matching message_idsMark tasks as 'started' with worker_idIncrement attempts counter on taskBuild step input by combining run input & dependency outputsReturn tasks with metadata and inputsFind handler function for a taskCall handler function with task inputReturn resultcomplete_task(results)Update task status to 'completed'Archive messageUpdate step_state to 'completed'Check & start dependent stepsEnqueue messages for ready dependent stepsDecrement remaining_steps counterIf all steps completed, mark run as 'completed'ConfirmationThrow exception or exceeds timeoutfail_task(error_message)Check remaining retry attemptsDelay message visibilityMessage becomes visibleread_with_poll(...)Return messagesstart_tasks(...)Mark task as 'failed'Mark step as 'failed'Mark run as 'failed'Archive message \ No newline at end of file +Task HandlerEdge WorkerPGMQ Queuepgflow SQL CoreClientTask HandlerEdge WorkerPGMQ Queuepgflow SQL CoreClientTwo-Phase PollingTask ExecutionTask succeedsHandler throws or exceeds timeoutRetries remainingWorker attempts execution againNo retries remainingPermanent failure of a runcreate_flow(...)add_step(...)start_flow(...)Create run recordInitialize step_statesCreate step_tasks for root stepsEnqueue message for root step taskReturn run detailsread_with_poll(queue_name, vt, qty)Return messagesstart_tasks(flow_slug, msg_ids, worker_id, queue_name)Find step_tasks with matching message_idsMark tasks as 'started' with worker_idIncrement attempts counter on taskBuild step input by combining run input & dependency outputsReturn tasks with metadata and inputsFind handler function for a taskCall handler function with task inputReturn resultcomplete_task(results)Update task status to 'completed'Archive messageUpdate step_state to 'completed'Check & start dependent stepsEnqueue messages for ready dependent stepsDecrement remaining_steps counterIf all steps completed, mark run as 'completed'ConfirmationThrow exception or exceeds timeoutfail_task(error_message)Check remaining retry attemptsDelay message visibilityMessage becomes visibleread_with_poll(...)Return messagesstart_tasks(...)Mark task as 'failed'Mark step as 'failed'Mark run as 'failed'Archive message \ No newline at end of file diff --git a/pkgs/core/schemas/0030_utilities.sql b/pkgs/core/schemas/0030_utilities.sql index 2a72c5dab..e33538764 100644 --- a/pkgs/core/schemas/0030_utilities.sql +++ b/pkgs/core/schemas/0030_utilities.sql @@ -35,6 +35,23 @@ begin end; $$; +create or replace function pgflow.is_valid_queue_name( + queue_name text +) +returns boolean +language sql +immutable +set search_path = '' +as $$ + -- Mirrors pgmq.validate_queue_name() (47-character limit) and additionally + -- requires the canonical lowercase spelling pgflow stores (#650). + select + queue_name is not null + and queue_name <> '' + and length(queue_name) <= 47 + and queue_name = lower(queue_name) +$$; + create or replace function pgflow.calculate_retry_delay( base_delay numeric, attempts_count int diff --git a/pkgs/core/schemas/0035_function_listed_queue_name.sql b/pkgs/core/schemas/0035_function_listed_queue_name.sql new file mode 100644 index 000000000..0168ad369 --- /dev/null +++ b/pkgs/core/schemas/0035_function_listed_queue_name.sql @@ -0,0 +1,47 @@ +-- Resolve a stored canonical queue name to the spelling listed in pgmq. +-- +-- pgflow stores canonical lowercase queue names (#650). A queue created by an +-- older pgflow release may be listed under its original mixed-case spelling. +-- PGMQ's public message operations (send_batch, read_with_poll, set_vt, +-- archive, delete) normalize names themselves, so message paths address the +-- queue by its stored canonical name directly and never resolve anything. +-- +-- Only operations that must address the queue's physical objects by their +-- original spelling need this helper today: delete_flow_and_data (drop_queue +-- drops metadata and tables under the listed spelling). The helper resolves +-- the listed spelling through pgmq.list_queues() and never creates a second +-- metadata entry. An ambiguous case-insensitive match (external damage) is +-- rejected before any destructive work; an unlisted name is passed through +-- unchanged so PGMQ reports the operation's own error. + +-- Fresh resolution against pgmq.list_queues(). +create or replace function pgflow._listed_queue_name(p_queue_name text) +returns text +language plpgsql +stable +set search_path = '' +as $$ +declare + v_matches text[]; +begin + -- Resolve every listed spelling of the normalized name before preferring + -- any single match: an ambiguous pair is rejected even when one spelling + -- is the exact requested name (#650). + select array_agg(listed.queue_name order by listed.queue_name) + into v_matches + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(p_queue_name); + + if v_matches is null then + -- Not listed: pass through; PGMQ reports its own error (or no-ops) + return p_queue_name; + elsif cardinality(v_matches) > 1 then + raise exception + 'queue name "%" is ambiguous: it matches listed queues %', + p_queue_name, v_matches + using errcode = 'ambiguous_alias'; + else + return v_matches[1]; + end if; +end; +$$; diff --git a/pkgs/core/schemas/0050_tables_definitions.sql b/pkgs/core/schemas/0050_tables_definitions.sql index 452540c99..a239d8155 100644 --- a/pkgs/core/schemas/0050_tables_definitions.sql +++ b/pkgs/core/schemas/0050_tables_definitions.sql @@ -17,6 +17,9 @@ create table pgflow.flows ( create table pgflow.steps ( flow_slug text not null references pgflow.flows(flow_slug), step_slug text not null, + -- Canonical queue this step's tasks are dispatched to (#650). + -- For this stage every step routes to the flow's default queue: lower(flow_slug). + queue_name text not null, step_type text not null default 'single', step_index int not null default 0, deps_count int not null default 0 check (deps_count >= 0), @@ -38,6 +41,7 @@ create table pgflow.steps ( primary key (flow_slug, step_slug), unique (flow_slug, step_index), -- Ensure step_index is unique within a flow check (pgflow.is_valid_slug(step_slug)), + constraint queue_name_is_valid check (pgflow.is_valid_queue_name(queue_name)), check (step_type in ('single', 'map')), constraint opt_max_attempts_is_nonnegative check (opt_max_attempts is null or opt_max_attempts >= 0), constraint opt_base_delay_is_nonnegative check (opt_base_delay is null or opt_base_delay >= 0), @@ -63,3 +67,8 @@ create table pgflow.deps ( create index if not exists idx_deps_by_flow_step on pgflow.deps (flow_slug, step_slug); create index if not exists idx_deps_by_flow_dep on pgflow.deps (flow_slug, dep_slug); + +-- Two concrete flows must not address the same normalized default queue (#650). +-- The expression index also rejects direct SQL creation of conflicting flows. +create unique index if not exists idx_flows_normalized_slug +on pgflow.flows (lower(flow_slug)); diff --git a/pkgs/core/schemas/0060_tables_runtime.sql b/pkgs/core/schemas/0060_tables_runtime.sql index c487d687f..40624ebdf 100644 --- a/pkgs/core/schemas/0060_tables_runtime.sql +++ b/pkgs/core/schemas/0060_tables_runtime.sql @@ -85,6 +85,10 @@ create table pgflow.step_tasks ( flow_slug text not null references pgflow.flows(flow_slug), run_id uuid not null references pgflow.runs(run_id), step_slug text not null, + -- Snapshot of steps.queue_name taken at task creation (#650). + -- Runtime code never changes this value; PGMQ message IDs are queue-scoped, + -- so a task's message identity is (queue_name, message_id). + queue_name text not null, message_id bigint, task_index int not null default 0, status text not null default 'queued', @@ -117,10 +121,17 @@ create table pgflow.step_tasks ( constraint completed_at_is_after_started_at check ( completed_at is null or started_at is null or completed_at >= started_at ), - constraint failed_at_is_after_started_at check (failed_at is null or started_at is null or failed_at >= started_at) + constraint failed_at_is_after_started_at check (failed_at is null or started_at is null or failed_at >= started_at), + constraint queue_name_is_valid check (pgflow.is_valid_queue_name(queue_name)) ); -create index if not exists idx_step_tasks_message_id on pgflow.step_tasks (message_id); +-- A message ID identifies at most one task per queue (#650). +-- NULL message_ids (pre-dispatch rows) are not part of the identity. +-- This index also serves queue-scoped message lookups for claims and pruning, +-- replacing the former message_id-only index. +create unique index if not exists idx_step_tasks_queue_message +on pgflow.step_tasks (queue_name, message_id) +where message_id is not null; create index if not exists idx_step_tasks_queued on pgflow.step_tasks (run_id, step_slug) where status = 'queued'; create index if not exists idx_step_tasks_completed on pgflow.step_tasks (run_id, step_slug) where status = 'completed'; create index if not exists idx_step_tasks_failed on pgflow.step_tasks (run_id, step_slug) where status = 'failed'; diff --git a/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql b/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql index 76b2a1b23..5c6357adf 100644 --- a/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql +++ b/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql @@ -23,6 +23,7 @@ begin st.step_slug, st.task_index, st.message_id, + st.queue_name, r.flow_slug, st.requeued_count from pgflow.step_tasks st @@ -59,9 +60,11 @@ begin where st.run_id = tr.run_id and st.step_slug = tr.step_slug and st.task_index = tr.task_index - returning tr.flow_slug as queue_name, tr.message_id + returning tr.queue_name as queue_name, tr.message_id ), - -- Make requeued messages visible immediately (batched per queue) + -- Make requeued messages visible immediately (batched per queue, through + -- the tasks' stored queue snapshots #650; PGMQ message operations + -- normalize names themselves) visibility_reset as ( select pgflow.set_vt_batch( r.queue_name, @@ -84,10 +87,13 @@ begin ), -- Archive messages for tasks that exceeded max requeues (batched per queue) archived as ( - select pgmq.archive(ta.flow_slug, array_agg(ta.message_id)) + select pgmq.archive( + ta.queue_name, + array_agg(ta.message_id) + ) from to_archive ta where ta.message_id is not null - group by ta.flow_slug + group by ta.queue_name ), -- Force execution of visibility_reset CTE _vr as (select count(*) from visibility_reset), diff --git a/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql b/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql index 5cc14c16c..8762c42ae 100644 --- a/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql +++ b/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql @@ -100,13 +100,18 @@ BEGIN FROM skipped AS skipped_step ) AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.message_id, task.queue_name ), -- ---------- Archive queued/started task messages for skipped steps ---------- + -- Batched per stored queue route (#650) archived_messages AS ( - SELECT pgmq.archive(v_flow_slug, ARRAY_AGG(task.message_id)) as result + SELECT pgmq.archive( + task.queue_name, + ARRAY_AGG(task.message_id) + ) as result FROM skipped_tasks AS task WHERE task.message_id IS NOT NULL + GROUP BY task.queue_name HAVING COUNT(task.message_id) > 0 ), -- ---------- Update run counters ---------- diff --git a/pkgs/core/schemas/0100_function_add_step.sql b/pkgs/core/schemas/0100_function_add_step.sql index bb3475979..dc44c92db 100644 --- a/pkgs/core/schemas/0100_function_add_step.sql +++ b/pkgs/core/schemas/0100_function_add_step.sql @@ -37,15 +37,17 @@ BEGIN FROM pgflow.steps s WHERE s.flow_slug = add_step.flow_slug; - -- Create the step + -- Create the step. queue_name records the step's resolved default route: + -- lower(flow_slug) for this stage (#650). INSERT INTO pgflow.steps ( - flow_slug, step_slug, step_type, step_index, deps_count, + flow_slug, step_slug, queue_name, step_type, step_index, deps_count, opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay, required_input_pattern, forbidden_input_pattern, when_unmet, when_exhausted ) VALUES ( add_step.flow_slug, add_step.step_slug, + lower(add_step.flow_slug), COALESCE(add_step.step_type, 'single'), next_idx, COALESCE(array_length(add_step.deps_slugs, 1), 0), @@ -59,7 +61,9 @@ BEGIN add_step.when_exhausted ) ON CONFLICT ON CONSTRAINT steps_pkey - DO UPDATE SET step_slug = EXCLUDED.step_slug + DO UPDATE SET + step_slug = EXCLUDED.step_slug, + queue_name = EXCLUDED.queue_name RETURNING * INTO result_step; -- Insert dependencies diff --git a/pkgs/core/schemas/0100_function_archive_task_message.sql b/pkgs/core/schemas/0100_function_archive_task_message.sql index 3ed804114..daa9b496b 100644 --- a/pkgs/core/schemas/0100_function_archive_task_message.sql +++ b/pkgs/core/schemas/0100_function_archive_task_message.sql @@ -6,18 +6,19 @@ create or replace function pgflow._archive_task_message( returns void language sql volatile -set search_path to '' +set search_path = '' as $$ + -- Archive through the task's stored queue snapshot (#650); PGMQ message + -- operations normalize names themselves. SELECT pgmq.archive( - r.flow_slug, + st.queue_name, ARRAY_AGG(st.message_id) ) FROM pgflow.step_tasks st - JOIN pgflow.runs r ON st.run_id = r.run_id WHERE st.run_id = p_run_id AND st.step_slug = p_step_slug AND st.task_index = p_task_index AND st.message_id IS NOT NULL - GROUP BY r.flow_slug + GROUP BY st.queue_name HAVING COUNT(st.message_id) > 0; $$; diff --git a/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql b/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql index 8fe445c8e..680ab0dd7 100644 --- a/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql +++ b/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql @@ -19,7 +19,7 @@ DECLARE v_processed_count int; v_run_transitioned boolean; v_flow_slug text; - v_cancelled_message_ids bigint[]; + v_archived_queues int; BEGIN -- ========================================== -- GUARD: Early return if run is already terminal @@ -158,24 +158,27 @@ BEGIN ); -- Terminalize every unfinished task across all branches as cancelled, - -- capturing their message ids for archival below. Lock-order invariant: - -- always lock/update step_tasks before PGMQ queue rows. + -- then archive the messages batched per stored queue route (#650). + -- Lock-order invariant: always lock/update step_tasks before PGMQ + -- queue rows; the archive reads the terminalized rows through the CTE. WITH cancelled_tasks AS ( UPDATE pgflow.step_tasks AS task SET status = 'cancelled' WHERE task.run_id = cascade_resolve_conditions.run_id AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + ct.queue_name, + ARRAY_AGG(ct.message_id) + ) + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name ) - SELECT ARRAY_AGG(ct.message_id) INTO v_cancelled_message_ids - FROM cancelled_tasks ct - WHERE ct.message_id IS NOT NULL; - - -- Archive the cancelled task messages captured above (only after their - -- task rows are terminalized) - IF v_cancelled_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_first_fail.flow_slug, v_cancelled_message_ids); - END IF; + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; END IF; RETURN false; diff --git a/pkgs/core/schemas/0100_function_complete_task.sql b/pkgs/core/schemas/0100_function_complete_task.sql index 6b593bebd..8af3b8ed8 100644 --- a/pkgs/core/schemas/0100_function_complete_task.sql +++ b/pkgs/core/schemas/0100_function_complete_task.sql @@ -14,7 +14,7 @@ declare v_dependent_map_slug text; v_run_record pgflow.runs%ROWTYPE; v_step_record pgflow.step_states%ROWTYPE; - v_violation_archived_ids bigint[]; + v_violation_archived_queues int; begin -- ========================================== @@ -68,16 +68,13 @@ END IF; -- If the step is not in 'started' state, this is a late callback. -- Do not mutate step_states or runs, archive message, return task row. IF v_step_record.status != 'started' THEN - -- Archive the task message if present (prevents stuck work) - PERFORM pgmq.archive( - v_run_record.flow_slug, - st.message_id - ) - FROM pgflow.step_tasks st - WHERE st.run_id = complete_task.run_id - AND st.step_slug = complete_task.step_slug - AND st.task_index = complete_task.task_index - AND st.message_id IS NOT NULL; + -- Archive the task message if present (prevents stuck work), through the + -- task's stored queue snapshot (#650) + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); -- Return the current task row without any mutations RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = complete_task.run_id @@ -171,37 +168,43 @@ IF v_dependent_map_slug IS NOT NULL THEN false ); - -- Terminalize every other unfinished task as cancelled, capturing their - -- message ids for archival below. Lock-order invariant: always lock/update - -- step_tasks before PGMQ queue rows. The culprit task is already terminal - -- (failed above), so it is excluded from the cancellation set. + -- Terminalize every other unfinished task as cancelled, then archive the + -- culprit and cancelled messages batched per stored queue route (#650). + -- Lock-order invariant: always lock/update step_tasks before PGMQ queue + -- rows; the archive reads the terminalized rows through the CTE. + -- The culprit task is already terminal (failed above), so it is excluded + -- from the cancellation set. WITH cancelled_tasks AS ( UPDATE pgflow.step_tasks AS task SET status = 'cancelled' WHERE task.run_id = complete_task.run_id AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.message_id, task.queue_name ), culprit_task AS ( -- Terminal culprit row: safe to read for its message id after terminalization - SELECT st.message_id + SELECT st.message_id, st.queue_name FROM pgflow.step_tasks st WHERE st.run_id = complete_task.run_id AND st.step_slug = complete_task.step_slug AND st.task_index = complete_task.task_index AND st.message_id IS NOT NULL - ) - SELECT ARRAY_AGG(ids.message_id) INTO v_violation_archived_ids - FROM ( - SELECT message_id FROM culprit_task + ), + terminal_messages AS ( + SELECT message_id, queue_name FROM culprit_task UNION ALL - SELECT message_id FROM cancelled_tasks WHERE message_id IS NOT NULL - ) ids; - - -- Archive the culprit and cancelled task messages (only after their task rows are terminalized) - IF v_violation_archived_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_run_record.flow_slug, v_violation_archived_ids); - END IF; + SELECT message_id, queue_name FROM cancelled_tasks WHERE message_id IS NOT NULL + ), + archived_messages AS ( + SELECT pgmq.archive( + tm.queue_name, + ARRAY_AGG(tm.message_id) + ) + FROM terminal_messages tm + GROUP BY tm.queue_name + ) + SELECT COUNT(*)::int INTO v_violation_archived_queues + FROM archived_messages; -- Return the failed task row (API contract: always return task row) RETURN QUERY @@ -379,12 +382,10 @@ IF v_step_state.status = 'completed' THEN IF NOT pgflow.cascade_resolve_conditions(complete_task.run_id) THEN -- Run was failed due to a condition with when_unmet='fail' -- Archive the current task's message before returning - PERFORM pgmq.archive( - (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id), - (SELECT st.message_id FROM pgflow.step_tasks st - WHERE st.run_id = complete_task.run_id - AND st.step_slug = complete_task.step_slug - AND st.task_index = complete_task.task_index) + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index ); RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = complete_task.run_id @@ -399,18 +400,18 @@ IF v_step_state.status = 'completed' THEN END IF; -- ---------- Archive completed task message ---------- --- Move message from active queue to archive table +-- Move message from active queue to archive table, through the task's +-- stored queue snapshot (#650) PERFORM ( WITH completed_tasks AS ( - SELECT r.flow_slug, st.message_id + SELECT st.queue_name, st.message_id FROM pgflow.step_tasks st - JOIN pgflow.runs r ON st.run_id = r.run_id WHERE st.run_id = complete_task.run_id AND st.step_slug = complete_task.step_slug AND st.task_index = complete_task.task_index AND st.status = 'completed' ) - SELECT pgmq.archive(ct.flow_slug, ct.message_id) + SELECT pgmq.archive(ct.queue_name, ct.message_id) FROM completed_tasks ct WHERE EXISTS (SELECT 1 FROM completed_tasks) ); diff --git a/pkgs/core/schemas/0100_function_create_flow.sql b/pkgs/core/schemas/0100_function_create_flow.sql index 5c56312a3..e2cc61b0f 100644 --- a/pkgs/core/schemas/0100_function_create_flow.sql +++ b/pkgs/core/schemas/0100_function_create_flow.sql @@ -1,6 +1,13 @@ -- Create a new flow with optional configuration. -- NULL parameters use defaults defined in the 'defaults' CTE below. -- This allows callers to pass NULL to explicitly use the default value. +-- +-- Queue provisioning (#650): the flow's default queue is the normalized slug +-- lower(flow_slug). Before creating a new flow, a listed PGMQ queue with the +-- same normalized name is rejected unless an existing flow with the same +-- normalized slug owns it (then the queue is reused as listed, keeping its +-- original spelling). pgflow never creates a second metadata entry just to +-- lowercase an existing queue name. create or replace function pgflow.create_flow( flow_slug text, max_attempts int default null, @@ -8,33 +15,50 @@ create or replace function pgflow.create_flow( timeout int default null ) returns pgflow.flows -language sql -set search_path to '' +language plpgsql volatile +set search_path = '' as $$ -WITH - defaults AS ( - SELECT 3 AS def_max_attempts, 5 AS def_base_delay, 60 AS def_timeout - ), - flow_upsert AS ( - INSERT INTO pgflow.flows (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) - SELECT - flow_slug, - COALESCE(max_attempts, defaults.def_max_attempts), - COALESCE(base_delay, defaults.def_base_delay), - COALESCE(timeout, defaults.def_timeout) - FROM defaults - ON CONFLICT (flow_slug) DO UPDATE - SET flow_slug = pgflow.flows.flow_slug -- Dummy update - RETURNING * - ), - ensure_queue AS ( - SELECT pgmq.create(flow_slug) - WHERE NOT EXISTS ( - SELECT 1 FROM pgmq.list_queues() WHERE queue_name = flow_slug - ) +declare + v_flow pgflow.flows; +begin + if not exists ( + select 1 + from pgflow.flows as flow + where lower(flow.flow_slug) = lower(create_flow.flow_slug) + ) and exists ( + select 1 + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(create_flow.flow_slug) + ) then + raise exception + 'cannot create flow "%": queue "%" is already in use by another owner', + create_flow.flow_slug, lower(create_flow.flow_slug) + using errcode = 'unique_violation'; + end if; + + insert into pgflow.flows (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) + values ( + create_flow.flow_slug, + coalesce(max_attempts, 3), + coalesce(base_delay, 5), + coalesce(timeout, 60) ) -SELECT f.* -FROM flow_upsert f -LEFT JOIN (SELECT 1 FROM ensure_queue) _dummy ON true; -- Left join ensures flow is returned + on conflict on constraint flows_pkey + do update + set flow_slug = pgflow.flows.flow_slug -- Dummy update + returning * into v_flow; + + -- Ensure the default queue exists, including for an empty flow. Reuse the + -- listed queue when it exists under any spelling of the normalized name. + if not exists ( + select 1 + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(create_flow.flow_slug) + ) then + perform pgmq.create(lower(create_flow.flow_slug)); + end if; + + return v_flow; +end; $$; diff --git a/pkgs/core/schemas/0100_function_delete_flow_and_data.sql b/pkgs/core/schemas/0100_function_delete_flow_and_data.sql index 7cd14e110..d58b3c8cd 100644 --- a/pkgs/core/schemas/0100_function_delete_flow_and_data.sql +++ b/pkgs/core/schemas/0100_function_delete_flow_and_data.sql @@ -1,15 +1,43 @@ -- Deletes a flow and all its associated data -- WARNING: This is destructive - deletes flow definition AND all runtime data -- Used by ensure_flow_compiled for development mode recompilation +-- +-- The flow's queues are dropped through their persisted definition routes +-- (steps.queue_name), plus the default queue for an empty flow (#650). Queues +-- created by older releases keep their original listed spelling: drop_queue +-- addresses physical objects, so the drop resolves the spelling fresh +-- through pgmq.list_queues() (_listed_queue_name) and an ambiguous match is +-- rejected before any destructive work. Everything runs in one transaction: +-- a failed PGMQ operation rolls the whole deletion back. create or replace function pgflow.delete_flow_and_data(p_flow_slug text) returns void language plpgsql volatile -set search_path to '' +set search_path = '' as $$ BEGIN - -- Drop queue and archive table (pgmq) - PERFORM pgmq.drop_queue(p_flow_slug); + -- Only an exact pgflow.flows row authorizes destructive queue work: a + -- nonexistent or wrong-case slug must not drop any queue (#650). The + -- data deletes below stay exact-match and no-op without the row. + IF EXISTS ( + SELECT 1 FROM pgflow.flows AS flow WHERE flow.flow_slug = p_flow_slug + ) THEN + -- Drop queues and archive tables (pgmq) using persisted routes. The + -- listed spelling is resolved fresh on every call; message operations + -- never need it because PGMQ normalizes names itself. + PERFORM pgmq.drop_queue(pgflow._listed_queue_name(route.queue_name)) + FROM ( + SELECT DISTINCT queue_name + FROM pgflow.steps + WHERE flow_slug = p_flow_slug + UNION + -- Empty flow: no persisted routes, fall back to the default queue + SELECT lower(p_flow_slug) + WHERE NOT EXISTS ( + SELECT 1 FROM pgflow.steps WHERE flow_slug = p_flow_slug + ) + ) AS route; + END IF; -- Delete all associated data in the correct order (respecting FK constraints) DELETE FROM pgflow.step_tasks AS task WHERE task.flow_slug = p_flow_slug; diff --git a/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql b/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql index a2bfa7d07..9e562d3e4 100644 --- a/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql +++ b/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql @@ -17,8 +17,8 @@ DECLARE v_differences text[]; v_is_local boolean; BEGIN - -- Generate lock key from flow_slug (deterministic hash) - v_lock_key := hashtext(ensure_flow_compiled.flow_slug); + -- Generate lock key from the normalized flow identity (deterministic hash) + v_lock_key := hashtext(lower(ensure_flow_compiled.flow_slug)); -- Acquire transaction-level advisory lock -- Serializes concurrent compilation attempts for same flow diff --git a/pkgs/core/schemas/0100_function_fail_task.sql b/pkgs/core/schemas/0100_function_fail_task.sql index 6da065e4f..72e73da2e 100644 --- a/pkgs/core/schemas/0100_function_fail_task.sql +++ b/pkgs/core/schemas/0100_function_fail_task.sql @@ -19,8 +19,7 @@ DECLARE v_prev_step_status text; v_run_status text; v_flow_slug text; - v_skipped_message_ids bigint[]; - v_cancelled_message_ids bigint[]; + v_archived_queues int; begin -- If run is already failed, no retries allowed. @@ -59,14 +58,9 @@ IF v_run_status = 'failed' THEN END IF; IF v_prev_step_status IS NOT NULL AND v_prev_step_status != 'started' THEN - -- Archive the task message if present - PERFORM pgmq.archive(v_flow_slug, ARRAY_AGG(st.message_id)) - FROM pgflow.step_tasks st - WHERE st.run_id = fail_task.run_id - AND st.step_slug = fail_task.step_slug - AND st.task_index = fail_task.task_index - AND st.message_id IS NOT NULL - HAVING COUNT(st.message_id) > 0; + -- Archive the task message if present, through the task's stored queue + -- snapshot (#650) + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = fail_task.run_id @@ -217,24 +211,28 @@ END IF; -- Lock-order invariant: always lock/update step_tasks before PGMQ queue rows. -- requeue_stalled_tasks() uses the same order; archiving queue rows first -- deadlocks the two transactions against each other. - -- Terminalize all still-active sibling task rows for the skipped step, - -- capturing their message ids for archival below. + -- Terminalize all still-active sibling task rows for the skipped step, then + -- archive their messages batched per stored queue route (#650); the + -- archive reads the terminalized rows through the CTE. WITH skipped_tasks AS ( UPDATE pgflow.step_tasks AS task SET status = 'skipped' WHERE task.run_id = fail_task.run_id AND task.step_slug = fail_task.step_slug AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + st.queue_name, + ARRAY_AGG(st.message_id) + ) + FROM skipped_tasks st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name ) - SELECT ARRAY_AGG(st.message_id) INTO v_skipped_message_ids - FROM skipped_tasks st - WHERE st.message_id IS NOT NULL; - - -- Archive the sibling task messages captured above (only after their task rows are terminalized) - IF v_skipped_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_flow_slug, v_skipped_message_ids); - END IF; + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; -- Send broadcast event for step skipped PERFORM realtime.send( @@ -327,25 +325,30 @@ IF v_run_failed THEN END IF; -- Terminalize unfinished tasks as cancelled when the run fails, then archive --- their messages. Lock-order invariant: always lock/update step_tasks before --- PGMQ queue rows. The culprit task is already terminal (failed or requeued by --- fail_or_retry_task), so only unfinished queued/started siblings are cancelled. +-- their messages batched per stored queue route (#650). Lock-order invariant: +-- always lock/update step_tasks before PGMQ queue rows; the archive reads the +-- terminalized rows through the CTE. The culprit task is already terminal +-- (failed or requeued by fail_or_retry_task), so only unfinished queued/started +-- siblings are cancelled. IF v_run_failed THEN WITH cancelled_tasks AS ( UPDATE pgflow.step_tasks AS task SET status = 'cancelled' WHERE task.run_id = fail_task.run_id AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + ct.queue_name, + ARRAY_AGG(ct.message_id) + ) + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name ) - SELECT ARRAY_AGG(ct.message_id) INTO v_cancelled_message_ids - FROM cancelled_tasks ct - WHERE ct.message_id IS NOT NULL; - - -- Archive the cancelled task messages captured above (only after their task rows are terminalized) - IF v_cancelled_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_flow_slug, v_cancelled_message_ids); - END IF; + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; END IF; -- For queued tasks: delay the message for retry with exponential backoff @@ -361,7 +364,7 @@ PERFORM ( ), queued_tasks AS ( SELECT - r.flow_slug, + st.queue_name, st.message_id, pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay FROM pgflow.step_tasks st @@ -371,21 +374,25 @@ PERFORM ( AND st.task_index = fail_task.task_index AND st.status = 'queued' ) - SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + SELECT pgmq.set_vt( + qt.queue_name, + qt.message_id, + qt.calculated_delay + ) FROM queued_tasks qt WHERE EXISTS (SELECT 1 FROM queued_tasks) ); --- For failed tasks: archive the message -PERFORM pgmq.archive(r.flow_slug, ARRAY_AGG(st.message_id)) +-- For failed tasks: archive the message, through the task's stored queue +-- snapshot (#650) +PERFORM pgmq.archive(st.queue_name, ARRAY_AGG(st.message_id)) FROM pgflow.step_tasks st -JOIN pgflow.runs r ON st.run_id = r.run_id WHERE st.run_id = fail_task.run_id AND st.step_slug = fail_task.step_slug AND st.task_index = fail_task.task_index AND st.status = 'failed' AND st.message_id IS NOT NULL -GROUP BY r.flow_slug +GROUP BY st.queue_name HAVING COUNT(st.message_id) > 0; return query select * diff --git a/pkgs/core/schemas/0100_function_start_ready_steps.sql b/pkgs/core/schemas/0100_function_start_ready_steps.sql index a70ca9f26..f87f32e13 100644 --- a/pkgs/core/schemas/0100_function_start_ready_steps.sql +++ b/pkgs/core/schemas/0100_function_start_ready_steps.sql @@ -75,6 +75,7 @@ message_batches AS ( started_step.flow_slug, started_step.run_id, started_step.step_slug, + step.queue_name, COALESCE(step.opt_start_delay, 0) as delay, array_agg( jsonb_build_object( @@ -91,31 +92,41 @@ message_batches AS ( AND step.step_slug = started_step.step_slug -- Generate task indices from 0 to initial_tasks-1 CROSS JOIN LATERAL generate_series(0, started_step.initial_tasks - 1) AS task_idx(task_index) - GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.opt_start_delay + GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.queue_name, step.opt_start_delay ), -- ---------- Send messages to queue ---------- --- Uses batch sending for performance with large arrays +-- Uses batch sending for performance with large arrays. +-- Messages go through the step's persisted queue route (#650); PGMQ +-- message operations normalize names themselves. sent_messages AS ( SELECT mb.flow_slug, mb.run_id, mb.step_slug, + mb.queue_name, task_indices.task_index, msg_ids.msg_id FROM message_batches mb CROSS JOIN LATERAL unnest(mb.task_indices) WITH ORDINALITY AS task_indices(task_index, idx_ord) - CROSS JOIN LATERAL pgmq.send_batch(mb.flow_slug, mb.messages, mb.delay) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) + CROSS JOIN LATERAL pgmq.send_batch( + mb.queue_name, + mb.messages, + mb.delay + ) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) WHERE task_indices.idx_ord = msg_ids.msg_ord ) -- ========================================== -- PHASE 3: RECORD TASKS IN DATABASE -- ========================================== -INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, message_id) +-- The task stores the step's queue_name snapshot; runtime message operations +-- use that snapshot, never a queue reconstructed from flow_slug (#650). +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) SELECT sent_messages.flow_slug, sent_messages.run_id, sent_messages.step_slug, + sent_messages.queue_name, sent_messages.task_index, sent_messages.msg_id FROM sent_messages; diff --git a/pkgs/core/schemas/0120_function_start_tasks.sql b/pkgs/core/schemas/0120_function_start_tasks.sql index 5905b06c0..9dac83835 100644 --- a/pkgs/core/schemas/0120_function_start_tasks.sql +++ b/pkgs/core/schemas/0120_function_start_tasks.sql @@ -1,7 +1,19 @@ +-- Claim queued tasks for the given flow by persisted (queue_name, message_id) +-- identity (#650). +-- +-- queue_name is the canonical queue identity of the polled queue and is +-- required: there is no default and no flow_slug fallback, so a claim can +-- never target a queue the caller did not read from. Today every caller +-- passes lower(flow_slug) — the spelling tasks store — including when PGMQ +-- still lists the queue under an older mixed-case spelling: PGMQ's message +-- API normalizes names itself, but this match is exact against the stored +-- canonical snapshot, so the polled mixed-case spelling would match nothing. +-- An explicit NULL matches nothing (no silent fallback). create or replace function pgflow.start_tasks( flow_slug text, msg_ids bigint [], - worker_id uuid + worker_id uuid, + queue_name text ) returns setof pgflow.step_task_record volatile @@ -18,6 +30,7 @@ as $$ from pgflow.step_tasks as task join pgflow.runs r on r.run_id = task.run_id where task.flow_slug = start_tasks.flow_slug + and task.queue_name = start_tasks.queue_name and task.message_id = any(msg_ids) and task.status = 'queued' and r.status = 'started' @@ -43,6 +56,7 @@ as $$ from task_candidates as candidate where step_tasks.message_id = candidate.message_id and step_tasks.flow_slug = candidate.flow_slug + and step_tasks.queue_name = start_tasks.queue_name and step_tasks.status = 'queued' returning step_tasks.flow_slug, @@ -96,7 +110,7 @@ as $$ -- only the shorter initial PGMQ read visibility (#656). visibility_reset as ( select pgflow.set_vt_batch( - start_tasks.flow_slug, + start_tasks.queue_name, array_agg(t.message_id order by t.message_id), array_agg(t.vt_delay order by t.message_id) ) diff --git a/pkgs/core/scripts/benchmarks/step_output_storage.sql b/pkgs/core/scripts/benchmarks/step_output_storage.sql index d4ca1ff3c..ce6e5100d 100644 --- a/pkgs/core/scripts/benchmarks/step_output_storage.sql +++ b/pkgs/core/scripts/benchmarks/step_output_storage.sql @@ -91,7 +91,7 @@ BEGIN FOR i IN 1..(v_array_size - 1) LOOP SELECT * INTO v_msg FROM pgmq.read('bench_map_single', 1, 1) LIMIT 1; - PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid); + PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid, 'bench_map_single'); SELECT task_index INTO v_task_index FROM pgflow.step_tasks WHERE message_id = v_msg.msg_id; @@ -102,7 +102,7 @@ BEGIN -- Time the FINAL complete_task (triggers aggregation in OLD code, stores in NEW code) SELECT * INTO v_msg FROM pgmq.read('bench_map_single', 1, 1) LIMIT 1; - PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid); + PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid, 'bench_map_single'); SELECT task_index INTO v_task_index FROM pgflow.step_tasks WHERE message_id = v_msg.msg_id; v_start_time := clock_timestamp(); @@ -140,7 +140,7 @@ BEGIN v_start_time := clock_timestamp(); - PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid); + PERFORM pgflow.start_tasks('bench_map_single', ARRAY[v_msg.msg_id], '11111111-1111-1111-1111-111111111111'::uuid, 'bench_map_single'); v_end_time := clock_timestamp(); v_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; @@ -185,7 +185,7 @@ BEGIN FOR i IN 1..v_array_size LOOP SELECT * INTO v_msg FROM pgmq.read('bench_map_map', 1, 1) LIMIT 1; - PERFORM pgflow.start_tasks('bench_map_map', ARRAY[v_msg.msg_id], '22222222-2222-2222-2222-222222222222'::uuid); + PERFORM pgflow.start_tasks('bench_map_map', ARRAY[v_msg.msg_id], '22222222-2222-2222-2222-222222222222'::uuid, 'bench_map_map'); SELECT task_index INTO v_task_index FROM pgflow.step_tasks WHERE message_id = v_msg.msg_id; PERFORM pgflow.complete_task(v_run_id, 'producer', v_task_index, jsonb_build_object('idx', v_task_index, 'value', i * 10)); @@ -217,7 +217,7 @@ BEGIN v_start_time := clock_timestamp(); - PERFORM pgflow.start_tasks('bench_map_map', v_msg_ids, '22222222-2222-2222-2222-222222222222'::uuid); + PERFORM pgflow.start_tasks('bench_map_map', v_msg_ids, '22222222-2222-2222-2222-222222222222'::uuid, 'bench_map_map'); v_end_time := clock_timestamp(); v_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; diff --git a/pkgs/core/scripts/run-upgrade-fixture b/pkgs/core/scripts/run-upgrade-fixture index 78efa55f9..bd3979c23 100755 --- a/pkgs/core/scripts/run-upgrade-fixture +++ b/pkgs/core/scripts/run-upgrade-fixture @@ -1,61 +1,88 @@ #!/bin/bash set -euo pipefail -# 0.15.0 upgrade fixture for the consolidated task_lifecycle_hardening migration. +# Upgrade fixtures for released migration boundaries. # -# Proves on a fresh database that: +# Fixture 1 (0.15.0, consolidated task_lifecycle_hardening migration) proves: # 1. a database at 0.15.0 (migrations up to 20260607175525) upgrades cleanly, # 2. both migration-only data repairs run, in the required order, # 3. final runtime behavior works (start_tasks extends PGMQ visibility). # +# Fixtures 2/3 (0.16.0, persist_queue_identity migration) prove: +# 1. a populated database at 0.16.0 (migrations up to 20260907082520) +# backfills queue identity snapshots, including tasks with NULL +# message_id and ids beyond the JavaScript safe integer range, +# 2. post-upgrade runtime works through persisted queue identity, +# including a legacy mixed-case physical queue and deletion by route, +# 3. conflicting normalized flow definitions block the migration and the +# failed migration leaves the database unchanged (atomic rollback). +# # Uses the same postgres image and baseline schema as the atlas dev database # (pg_cron / pg_net cannot be created in a non-postgres database of the -# Supabase dev instance, so the fixture gets its own container). +# Supabase dev instance, so each fixture gets its own container). # Runs as part of `pnpm nx test:pgtap core`. cd "$(dirname "$0")/.." IMAGE="jumski/atlas-postgres-pgflow:17.6.1.054" -CONTAINER=pgflow-upgrade-fixture + # Last migration released in 0.15.0; everything after it is replaced by the # consolidated migration. -BASELINE_MIGRATION="20260607175525_pgflow_worker_start_mode.sql" +BASELINE_0_15="20260607175525_pgflow_worker_start_mode.sql" +# Last migration released in 0.16.0. +BASELINE_0_16="20260907082520_pgflow_remove_legacy_flow_compilation.sql" -cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } -trap cleanup EXIT -cleanup +CONTAINER="" -echo "upgrade fixture: starting postgres container" -docker run -d --name "$CONTAINER" "$IMAGE" >/dev/null +cleanup() { [[ -n "$CONTAINER" ]] && docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } +trap cleanup EXIT -for _ in $(seq 1 30); do - if docker exec "$CONTAINER" pg_isready -U postgres >/dev/null 2>&1; then - break - fi - sleep 1 -done +start_container() { + cleanup + CONTAINER="$1" + echo "upgrade fixture: starting postgres container $CONTAINER" + docker run -d --name "$CONTAINER" "$IMAGE" >/dev/null + local waited=0 + until docker exec "$CONTAINER" pg_isready -U postgres >/dev/null 2>&1; do + ((waited+=1)) + if ((waited > 30)); then + echo "upgrade fixture: $CONTAINER never became ready" >&2 + exit 1 + fi + sleep 1 + done +} psql_in() { docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -U postgres -d postgres } +# apply_migrations_up_to +apply_migrations_up_to() { + local baseline="$1" reached=false f + for f in supabase/migrations/*.sql; do + psql_in < "$f" + if [[ "$(basename "$f")" == "$baseline" ]]; then + reached=true + break + fi + done + if [[ "$reached" != true ]]; then + echo "upgrade fixture: baseline migration $baseline not found" >&2 + exit 1 + fi +} + +# ===================================================================== +# Fixture 1: 0.15.0 -> consolidated task_lifecycle_hardening +# ===================================================================== +start_container pgflow-upgrade-fixture + echo "upgrade fixture: applying supabase baseline schema" psql_in < atlas/supabase-baseline-schema.sql -echo "upgrade fixture: applying pgflow migrations up to 0.15.0 ($BASELINE_MIGRATION)" -reached_baseline=false -for f in supabase/migrations/*.sql; do - echo " $(basename "$f")" - psql_in < "$f" - if [[ "$(basename "$f")" == "$BASELINE_MIGRATION" ]]; then - reached_baseline=true - break - fi -done -if [[ "$reached_baseline" != true ]]; then - echo "upgrade fixture: baseline migration $BASELINE_MIGRATION not found" >&2 - exit 1 -fi +echo "upgrade fixture: applying pgflow migrations up to 0.15.0 ($BASELINE_0_15)" +apply_migrations_up_to "$BASELINE_0_15" echo "upgrade fixture: seeding stale 0.15.0 data" psql_in < supabase/upgrade_fixture/seed.sql @@ -65,4 +92,66 @@ echo "upgrade fixture: applying consolidated migration $(basename "$consolidated psql_in < "$consolidated" psql_in < supabase/upgrade_fixture/assertions.sql -echo "upgrade fixture: PASS" +echo "upgrade fixture: PASS (0.15.0)" + +# ===================================================================== +# Fixture 2: populated 0.16.0 -> persist_queue_identity +# ===================================================================== +start_container pgflow-upgrade-fixture-016 + +echo "upgrade fixture 0.16.0: applying supabase baseline schema" +psql_in < atlas/supabase-baseline-schema.sql + +echo "upgrade fixture 0.16.0: applying pgflow migrations up to 0.16.0 ($BASELINE_0_16)" +apply_migrations_up_to "$BASELINE_0_16" + +echo "upgrade fixture 0.16.0: seeding populated 0.16.0 data" +psql_in < supabase/upgrade_fixture/seed_0_16.sql + +persist_queue=$(ls supabase/migrations/*_pgflow_persist_queue_identity.sql) +echo "upgrade fixture 0.16.0: applying migration $(basename "$persist_queue") (single transaction)" +docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -1 -U postgres -d postgres < "$persist_queue" + +psql_in < supabase/upgrade_fixture/assertions_0_16.sql +echo "upgrade fixture: PASS (0.16.0 backfill + runtime)" + +# ===================================================================== +# Fixture 3: conflicting 0.16.0 definitions -> migration rolls back atomically +# ===================================================================== +start_container pgflow-upgrade-fixture-016-conflict + +echo "upgrade fixture conflict: applying supabase baseline schema" +psql_in < atlas/supabase-baseline-schema.sql + +echo "upgrade fixture conflict: applying pgflow migrations up to 0.16.0 ($BASELINE_0_16)" +apply_migrations_up_to "$BASELINE_0_16" + +echo "upgrade fixture conflict: seeding conflicting flow definitions" +psql_in < supabase/upgrade_fixture/seed_0_16_conflict.sql + +echo "upgrade fixture conflict: applying migration $(basename "$persist_queue") (must fail and roll back)" +if docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -1 -U postgres -d postgres < "$persist_queue" 2>/dev/null; then + echo "upgrade fixture conflict: migration unexpectedly succeeded on conflicting definitions" >&2 + exit 1 +fi + +# The failed single-transaction migration must leave the database unchanged: +# no queue_name column, conflicting flows intact. +psql_in <<'SQL' +do $$ +begin + if exists ( + select 1 from information_schema.columns + where table_schema = 'pgflow' and table_name = 'steps' and column_name = 'queue_name' + ) then + raise exception 'upgrade fixture conflict: steps.queue_name exists after a failed migration'; + end if; + + if (select count(*) from pgflow.flows where flow_slug in ('MyConflict', 'myconflict')) <> 2 then + raise exception 'upgrade fixture conflict: seeded flows were modified by the failed migration'; + end if; +end $$; +select 'PASS: conflict migration rolled back atomically' as result; +SQL + +echo "upgrade fixture: PASS (0.16.0 conflict rollback)" diff --git a/pkgs/core/src/PgflowSqlClient.ts b/pkgs/core/src/PgflowSqlClient.ts index e32e701b1..1d201bb5b 100644 --- a/pkgs/core/src/PgflowSqlClient.ts +++ b/pkgs/core/src/PgflowSqlClient.ts @@ -38,15 +38,17 @@ export class PgflowSqlClient async startTasks( flowSlug: string, - msgIds: number[], - workerId: string + msgIds: string[], + workerId: string, + queueName: string ): Promise[]> { return await this.sql[]>` SELECT * FROM pgflow.start_tasks( flow_slug => ${flowSlug}, msg_ids => ${msgIds}::bigint[], - worker_id => ${workerId}::uuid + worker_id => ${workerId}::uuid, + queue_name => ${queueName}::text ); `; } diff --git a/pkgs/core/src/database-types.ts b/pkgs/core/src/database-types.ts index 393ef0b2c..4a9376f06 100644 --- a/pkgs/core/src/database-types.ts +++ b/pkgs/core/src/database-types.ts @@ -208,6 +208,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string @@ -227,6 +228,7 @@ export type Database = { message_id?: number | null output?: Json | null permanently_stalled_at?: string | null + queue_name: string queued_at?: string requeued_count?: number run_id: string @@ -246,6 +248,7 @@ export type Database = { message_id?: number | null output?: Json | null permanently_stalled_at?: string | null + queue_name?: string queued_at?: string requeued_count?: number run_id?: string @@ -295,6 +298,7 @@ export type Database = { opt_max_attempts: number | null opt_start_delay: number | null opt_timeout: number | null + queue_name: string required_input_pattern: Json | null step_index: number step_slug: string @@ -311,6 +315,7 @@ export type Database = { opt_max_attempts?: number | null opt_start_delay?: number | null opt_timeout?: number | null + queue_name: string required_input_pattern?: Json | null step_index?: number step_slug: string @@ -327,6 +332,7 @@ export type Database = { opt_max_attempts?: number | null opt_start_delay?: number | null opt_timeout?: number | null + queue_name?: string required_input_pattern?: Json | null step_index?: number step_slug?: string @@ -426,6 +432,7 @@ export type Database = { Returns: undefined } _get_flow_shape: { Args: { p_flow_slug: string }; Returns: Json } + _listed_queue_name: { Args: { p_queue_name: string }; Returns: string } add_step: { Args: { base_delay?: number @@ -450,6 +457,7 @@ export type Database = { opt_max_attempts: number | null opt_start_delay: number | null opt_timeout: number | null + queue_name: string required_input_pattern: Json | null step_index: number step_slug: string @@ -497,6 +505,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string @@ -567,6 +576,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string @@ -584,6 +594,7 @@ export type Database = { } get_run_with_states: { Args: { run_id: string }; Returns: Json } is_local: { Args: never; Returns: boolean } + is_valid_queue_name: { Args: { queue_name: string }; Returns: boolean } is_valid_slug: { Args: { slug: string }; Returns: boolean } mark_worker_stopped: { Args: { worker_id: string }; Returns: undefined } maybe_complete_run: { Args: { run_id: string }; Returns: undefined } @@ -649,7 +660,12 @@ export type Database = { } start_ready_steps: { Args: { run_id: string }; Returns: undefined } start_tasks: { - Args: { flow_slug: string; msg_ids: number[]; worker_id: string } + Args: { + flow_slug: string + msg_ids: number[] + queue_name: string + worker_id: string + } Returns: Database["pgflow"]["CompositeTypes"]["step_task_record"][] SetofOptions: { from: "*" diff --git a/pkgs/core/src/types.ts b/pkgs/core/src/types.ts index 5789dbd8c..9f219ad1d 100644 --- a/pkgs/core/src/types.ts +++ b/pkgs/core/src/types.ts @@ -20,6 +20,9 @@ export type { Json }; * Note: flow_input is nullable because start_tasks only includes it for root non-map steps. * For dependent and map steps, flow_input is NULL to avoid data duplication. * Workers can access the original flow input via ctx.flowInput (lazy loaded). + * + * msg_id is an exact decimal string: PGMQ message ids are queue-scoped bigints + * that can exceed the JavaScript safe integer range (#650). */ export type StepTaskRecord = { [StepSlug in Extract, string>]: { @@ -28,7 +31,7 @@ export type StepTaskRecord = { step_slug: StepSlug; task_index: number; input: Simplify>; - msg_id: number; + msg_id: string; flow_input: ExtractFlowInput | null; }; }[Extract, string>]; @@ -43,9 +46,12 @@ export type StepTaskKey = Pick, 'run_id' | 'step_slug' | /** * Record representing a message from queue polling + * + * msg_id is an exact decimal string: PGMQ message ids are queue-scoped + * bigints that can exceed the JavaScript safe integer range (#650). */ export type MessageRecord = { - msg_id: number; + msg_id: string; read_ct: number; enqueued_at: string; vt: string; @@ -84,13 +90,21 @@ export interface IPgflowClient { /** * Starts tasks for given message IDs (phase 2 of two-phase approach) * @param flowSlug - The flow slug to start tasks from - * @param msgIds - Array of message IDs from readMessages + * @param msgIds - Array of message IDs (exact decimal strings) from readMessages * @param workerId - ID of the worker starting the tasks + * @param queueName - The canonical queue identity of the polled queue: + * `lower(flowSlug)` today, the exact spelling tasks store. pgflow workers + * poll the canonical lowercased queue, so they pass the queue they polled + * even when PGMQ lists an older queue with mixed-case spelling. Claims + * match the persisted (queue_name, message_id) identity exactly. Required + * (#650): there is no default and no fallback to a queue derived from the + * flow slug. */ startTasks( flowSlug: string, - msgIds: number[], - workerId: string + msgIds: string[], + workerId: string, + queueName: string ): Promise[]>; /** diff --git a/pkgs/core/supabase/migrations/20260913093141_pgflow_persist_queue_identity.sql b/pkgs/core/supabase/migrations/20260913093141_pgflow_persist_queue_identity.sql new file mode 100644 index 000000000..550919be4 --- /dev/null +++ b/pkgs/core/supabase/migrations/20260913093141_pgflow_persist_queue_identity.sql @@ -0,0 +1,2009 @@ +-- Bounded lock waits: fail fast instead of queueing indefinitely behind +-- long-running transactions when the migration takes table locks (#650). +SET lock_timeout = '10s'; +-- Create "is_valid_queue_name" function +CREATE FUNCTION "pgflow"."is_valid_queue_name" ("queue_name" text) RETURNS boolean LANGUAGE sql IMMUTABLE SET "search_path" = '' AS $$ +-- Mirrors pgmq.validate_queue_name() (47-character limit) and additionally + -- requires the canonical lowercase spelling pgflow stores (#650). + select + queue_name is not null + and queue_name <> '' + and length(queue_name) <= 47 + and queue_name = lower(queue_name) +$$; +-- Modify "step_tasks" table (staged: nullable column, backfill, then constraints) +ALTER TABLE "pgflow"."step_tasks" ADD COLUMN "queue_name" text NULL; +-- Modify "steps" table (staged: nullable column, backfill, then constraints) +ALTER TABLE "pgflow"."steps" ADD COLUMN "queue_name" text NULL; + +-- ========================================== +-- DATA BACKFILL: queue identity snapshots (#650) +-- Every step and task routes to its flow's default queue: lower(flow_slug). +-- Includes tasks whose message_id is NULL. Duplicate identities, queue names +-- beyond PGMQ's limit, or conflicting normalized flow slugs raise in the +-- constraint and index statements below and leave the database unchanged; +-- they are not repaired automatically. +-- ========================================== +UPDATE pgflow.steps SET queue_name = lower(flow_slug) WHERE queue_name IS NULL; +UPDATE pgflow.step_tasks SET queue_name = lower(flow_slug) WHERE queue_name IS NULL; + +ALTER TABLE "pgflow"."step_tasks" ALTER COLUMN "queue_name" SET NOT NULL, ADD CONSTRAINT "queue_name_is_valid" CHECK (pgflow.is_valid_queue_name(queue_name)); +ALTER TABLE "pgflow"."steps" ALTER COLUMN "queue_name" SET NOT NULL, ADD CONSTRAINT "queue_name_is_valid" CHECK (pgflow.is_valid_queue_name(queue_name)); +-- Drop index "idx_step_tasks_message_id" from table: "step_tasks" +-- Replaced by the queue-scoped unique index below (#650) +DROP INDEX "pgflow"."idx_step_tasks_message_id"; +-- Create index "idx_step_tasks_queue_message" to table: "step_tasks" +-- Created after the backfill (approved staged order): duplicate +-- (queue_name, message_id) pairs fail this statement and roll the whole +-- migration back atomically. +CREATE UNIQUE INDEX "idx_step_tasks_queue_message" ON "pgflow"."step_tasks" ("queue_name", "message_id") WHERE (message_id IS NOT NULL); +-- Create index "idx_flows_normalized_slug" to table: "flows" +-- Created after the backfill (approved staged order): conflicting normalized +-- flow slugs fail this statement and roll the whole migration back atomically. +CREATE UNIQUE INDEX "idx_flows_normalized_slug" ON "pgflow"."flows" ((lower(flow_slug))); +-- Modify "_archive_task_message" function +CREATE OR REPLACE FUNCTION "pgflow"."_archive_task_message" ("p_run_id" uuid, "p_step_slug" text, "p_task_index" integer) RETURNS void LANGUAGE sql SET "search_path" = '' AS $$ +-- Archive through the task's stored queue snapshot (#650); PGMQ message + -- operations normalize names themselves. + SELECT pgmq.archive( + st.queue_name, + ARRAY_AGG(st.message_id) + ) + FROM pgflow.step_tasks st + WHERE st.run_id = p_run_id + AND st.step_slug = p_step_slug + AND st.task_index = p_task_index + AND st.message_id IS NOT NULL + GROUP BY st.queue_name + HAVING COUNT(st.message_id) > 0; +$$; +-- Modify "_cascade_force_skip_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."_cascade_force_skip_steps" ("run_id" uuid, "step_slug" text, "skip_reason" text) RETURNS integer LANGUAGE plpgsql AS $$ +DECLARE + v_flow_slug text; + v_total_skipped int := 0; +BEGIN + -- Get flow_slug for this run + SELECT r.flow_slug INTO v_flow_slug + FROM pgflow.runs r + WHERE r.run_id = _cascade_force_skip_steps.run_id; + + IF v_flow_slug IS NULL THEN + RAISE EXCEPTION 'Run not found: %', _cascade_force_skip_steps.run_id; + END IF; + + -- ========================================== + -- SKIP STEPS IN TOPOLOGICAL ORDER + -- ========================================== + -- Use recursive CTE to find all downstream dependents, + -- then skip them in topological order (by step_index) + WITH RECURSIVE + -- ---------- Find all downstream steps ---------- + downstream_steps AS ( + -- Base case: the trigger step + SELECT + s.flow_slug, + s.step_slug, + s.step_index, + _cascade_force_skip_steps.skip_reason AS reason -- Original reason for trigger step + FROM pgflow.steps s + WHERE s.flow_slug = v_flow_slug + AND s.step_slug = _cascade_force_skip_steps.step_slug + + UNION ALL + + -- Recursive case: steps that depend on already-found steps + SELECT + s.flow_slug, + s.step_slug, + s.step_index, + 'dependency_skipped'::text AS reason -- Downstream steps get this reason + FROM pgflow.steps s + JOIN pgflow.deps d ON d.flow_slug = s.flow_slug AND d.step_slug = s.step_slug + JOIN downstream_steps ds ON ds.flow_slug = d.flow_slug AND ds.step_slug = d.dep_slug + ), + -- ---------- Deduplicate and order by step_index ---------- + steps_to_skip AS ( + SELECT DISTINCT ON (ds.step_slug) + ds.flow_slug, + ds.step_slug, + ds.step_index, + ds.reason + FROM downstream_steps ds + ORDER BY ds.step_slug, ds.step_index -- Keep first occurrence (trigger step has original reason) + ), + -- ---------- Skip the steps ---------- + skipped AS ( + UPDATE pgflow.step_states ss + SET status = 'skipped', + skip_reason = sts.reason, + skipped_at = now(), + remaining_tasks = NULL -- Clear remaining_tasks for skipped steps + FROM steps_to_skip sts + WHERE ss.run_id = _cascade_force_skip_steps.run_id + AND ss.step_slug = sts.step_slug + AND ss.status IN ('created', 'started') -- Only skip non-terminal steps + RETURNING + ss.*, + -- Broadcast step:skipped event + realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', ss.run_id, + 'flow_slug', ss.flow_slug, + 'step_slug', ss.step_slug, + 'status', 'skipped', + 'skip_reason', ss.skip_reason, + 'skipped_at', ss.skipped_at + ), + concat('step:', ss.step_slug, ':skipped'), + concat('pgflow:run:', ss.run_id), + false + ) as _broadcast_result + ), + -- ---------- Terminalize active tasks of newly skipped steps ---------- + skipped_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'skipped' + WHERE task.run_id = _cascade_force_skip_steps.run_id + AND task.step_slug IN ( + SELECT skipped_step.step_slug + FROM skipped AS skipped_step + ) + AND task.status IN ('queued', 'started') + RETURNING task.message_id, task.queue_name + ), + -- ---------- Archive queued/started task messages for skipped steps ---------- + -- Batched per stored queue route (#650) + archived_messages AS ( + SELECT pgmq.archive( + task.queue_name, + ARRAY_AGG(task.message_id) + ) as result + FROM skipped_tasks AS task + WHERE task.message_id IS NOT NULL + GROUP BY task.queue_name + HAVING COUNT(task.message_id) > 0 + ), + -- ---------- Update run counters ---------- + run_updates AS ( + UPDATE pgflow.runs r + SET remaining_steps = r.remaining_steps - skipped_count.count + FROM (SELECT COUNT(*) AS count FROM skipped) skipped_count + WHERE r.run_id = _cascade_force_skip_steps.run_id + AND skipped_count.count > 0 + ) + SELECT skipped_count.count + INTO v_total_skipped + FROM (SELECT COUNT(*) AS count FROM skipped) skipped_count + LEFT JOIN archived_messages ON true; + + RETURN v_total_skipped; +END; +$$; +-- Modify "add_step" function +CREATE OR REPLACE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "deps_slugs" text[] DEFAULT '{}', "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer, "start_delay" integer DEFAULT NULL::integer, "step_type" text DEFAULT 'single', "required_input_pattern" jsonb DEFAULT NULL::jsonb, "forbidden_input_pattern" jsonb DEFAULT NULL::jsonb, "when_unmet" text DEFAULT 'skip', "when_exhausted" text DEFAULT 'fail') RETURNS "pgflow"."steps" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + result_step pgflow.steps; + next_idx int; +BEGIN + -- Validate map step constraints + -- Map steps can have either: + -- 0 dependencies (root map - maps over flow input array) + -- 1 dependency (dependent map - maps over dependency output array) + IF COALESCE(add_step.step_type, 'single') = 'map' AND COALESCE(array_length(add_step.deps_slugs, 1), 0) > 1 THEN + RAISE EXCEPTION 'Map step "%" can have at most one dependency, but % were provided: %', + add_step.step_slug, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + array_to_string(add_step.deps_slugs, ', '); + END IF; + + -- Get next step index + SELECT COALESCE(MAX(s.step_index) + 1, 0) INTO next_idx + FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug; + + -- Create the step. queue_name records the step's resolved default route: + -- lower(flow_slug) for this stage (#650). + INSERT INTO pgflow.steps ( + flow_slug, step_slug, queue_name, step_type, step_index, deps_count, + opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay, + required_input_pattern, forbidden_input_pattern, when_unmet, when_exhausted + ) + VALUES ( + add_step.flow_slug, + add_step.step_slug, + lower(add_step.flow_slug), + COALESCE(add_step.step_type, 'single'), + next_idx, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + add_step.max_attempts, + add_step.base_delay, + add_step.timeout, + add_step.start_delay, + add_step.required_input_pattern, + add_step.forbidden_input_pattern, + add_step.when_unmet, + add_step.when_exhausted + ) + ON CONFLICT ON CONSTRAINT steps_pkey + DO UPDATE SET + step_slug = EXCLUDED.step_slug, + queue_name = EXCLUDED.queue_name + RETURNING * INTO result_step; + + -- Insert dependencies + INSERT INTO pgflow.deps (flow_slug, dep_slug, step_slug) + SELECT add_step.flow_slug, d.dep_slug, add_step.step_slug + FROM unnest(COALESCE(add_step.deps_slugs, '{}')) AS d(dep_slug) + WHERE add_step.deps_slugs IS NOT NULL AND array_length(add_step.deps_slugs, 1) > 0 + ON CONFLICT ON CONSTRAINT deps_pkey DO NOTHING; + + RETURN result_step; +END; +$$; +-- Modify "cascade_resolve_conditions" function +CREATE OR REPLACE FUNCTION "pgflow"."cascade_resolve_conditions" ("run_id" uuid) RETURNS boolean LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_input jsonb; + v_run_status text; + v_first_fail record; + v_iteration_count int := 0; + v_max_iterations int := 50; + v_processed_count int; + v_run_transitioned boolean; + v_flow_slug text; + v_archived_queues int; +BEGIN + -- ========================================== + -- GUARD: Early return if run is already terminal + -- ========================================== + SELECT r.status, r.input INTO v_run_status, v_run_input + FROM pgflow.runs r + WHERE r.run_id = cascade_resolve_conditions.run_id; + + IF v_run_status IN ('failed', 'completed') THEN + RETURN v_run_status != 'failed'; + END IF; + + -- ========================================== + -- ITERATE UNTIL CONVERGENCE + -- ========================================== + -- After skipping steps, dependents may become ready and need evaluation. + -- Loop until no more steps are processed. + LOOP + v_iteration_count := v_iteration_count + 1; + IF v_iteration_count > v_max_iterations THEN + RAISE EXCEPTION 'cascade_resolve_conditions exceeded safety limit of % iterations', v_max_iterations; + END IF; + + v_processed_count := 0; + + -- ========================================== + -- PHASE 1a: CHECK FOR FAIL CONDITIONS + -- ========================================== + -- Find first step (by topological order) with unmet condition and 'fail' mode. + -- Condition is unmet when: + -- (required_input_pattern is set AND input does NOT contain it) OR + -- (forbidden_input_pattern is set AND input DOES contain it) + WITH steps_with_conditions AS ( + SELECT + step_state.flow_slug, + step_state.step_slug, + step.required_input_pattern, + step.forbidden_input_pattern, + step.when_unmet, + step.deps_count, + step.step_index + FROM pgflow.step_states AS step_state + JOIN pgflow.steps AS step + ON step.flow_slug = step_state.flow_slug + AND step.step_slug = step_state.step_slug + WHERE step_state.run_id = cascade_resolve_conditions.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + ), + step_deps_output AS ( + SELECT + swc.step_slug, + jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM steps_with_conditions swc + JOIN pgflow.deps dep ON dep.flow_slug = swc.flow_slug AND dep.step_slug = swc.step_slug + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE swc.deps_count > 0 + GROUP BY swc.step_slug + ), + condition_evaluations AS ( + SELECT + swc.*, + -- condition_met = (if IS NULL OR input @> if) AND (ifNot IS NULL OR NOT(input @> ifNot)) + (swc.required_input_pattern IS NULL OR + CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.required_input_pattern) + AND + (swc.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.forbidden_input_pattern)) + AS condition_met + FROM steps_with_conditions swc + LEFT JOIN step_deps_output sdo ON sdo.step_slug = swc.step_slug + ) + SELECT + flow_slug, + step_slug, + required_input_pattern, + forbidden_input_pattern + INTO v_first_fail + FROM condition_evaluations + WHERE NOT condition_met AND when_unmet = 'fail' + ORDER BY step_index + LIMIT 1; + + -- Handle fail mode: fail step and run, return false + -- Note: Cannot use "v_first_fail IS NOT NULL" because records with NULL fields + -- evaluate to NULL in IS NOT NULL checks. Use FOUND instead. + IF FOUND THEN + -- Fail the run only if it is still started. The conditional UPDATE takes + -- the run row lock and rechecks status atomically, so replayed or + -- concurrent calls cannot duplicate the terminal transition or its events. + UPDATE pgflow.runs + SET status = 'failed', + failed_at = now() + WHERE pgflow.runs.run_id = cascade_resolve_conditions.run_id + AND pgflow.runs.status = 'started' + RETURNING true INTO v_run_transitioned; + + IF v_run_transitioned THEN + UPDATE pgflow.step_states + SET status = 'failed', + failed_at = now(), + error_message = 'Condition not met' + WHERE pgflow.step_states.run_id = cascade_resolve_conditions.run_id + AND pgflow.step_states.step_slug = v_first_fail.step_slug; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', cascade_resolve_conditions.run_id, + 'step_slug', v_first_fail.step_slug, + 'status', 'failed', + 'error_message', 'Condition not met', + 'failed_at', now() + ), + concat('step:', v_first_fail.step_slug, ':failed'), + concat('pgflow:run:', cascade_resolve_conditions.run_id), + false + ); + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', cascade_resolve_conditions.run_id, + 'flow_slug', v_first_fail.flow_slug, + 'status', 'failed', + 'error_message', 'Condition not met', + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', cascade_resolve_conditions.run_id), + false + ); + + -- Terminalize every unfinished task across all branches as cancelled, + -- then archive the messages batched per stored queue route (#650). + -- Lock-order invariant: always lock/update step_tasks before PGMQ + -- queue rows; the archive reads the terminalized rows through the CTE. + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = cascade_resolve_conditions.run_id + AND task.status IN ('queued', 'started') + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + ct.queue_name, + ARRAY_AGG(ct.message_id) + ) + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + ) + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; + END IF; + + RETURN false; + END IF; + + -- ========================================== + -- PHASE 1b: HANDLE SKIP CONDITIONS (with propagation) + -- ========================================== + -- Skip steps with unmet conditions and whenUnmet='skip'. + -- Also decrement remaining_deps on dependents and set initial_tasks=0 for map dependents. + WITH steps_with_conditions AS ( + SELECT + step_state.flow_slug, + step_state.step_slug, + step.required_input_pattern, + step.forbidden_input_pattern, + step.when_unmet, + step.deps_count, + step.step_index + FROM pgflow.step_states AS step_state + JOIN pgflow.steps AS step + ON step.flow_slug = step_state.flow_slug + AND step.step_slug = step_state.step_slug + WHERE step_state.run_id = cascade_resolve_conditions.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + ), + step_deps_output AS ( + SELECT + swc.step_slug, + jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM steps_with_conditions swc + JOIN pgflow.deps dep ON dep.flow_slug = swc.flow_slug AND dep.step_slug = swc.step_slug + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE swc.deps_count > 0 + GROUP BY swc.step_slug + ), + condition_evaluations AS ( + SELECT + swc.*, + -- condition_met = (if IS NULL OR input @> if) AND (ifNot IS NULL OR NOT(input @> ifNot)) + (swc.required_input_pattern IS NULL OR + CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.required_input_pattern) + AND + (swc.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.forbidden_input_pattern)) + AS condition_met + FROM steps_with_conditions swc + LEFT JOIN step_deps_output sdo ON sdo.step_slug = swc.step_slug + ), + unmet_skip_steps AS ( + SELECT * FROM condition_evaluations + WHERE NOT condition_met AND when_unmet = 'skip' + ), + skipped_steps AS ( + UPDATE pgflow.step_states ss + SET status = 'skipped', + skip_reason = 'condition_unmet', + skipped_at = now() + FROM unmet_skip_steps uss + WHERE ss.run_id = cascade_resolve_conditions.run_id + AND ss.step_slug = uss.step_slug + AND ss.status = 'created' + RETURNING + ss.*, + realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', ss.run_id, + 'flow_slug', ss.flow_slug, + 'step_slug', ss.step_slug, + 'status', 'skipped', + 'skip_reason', 'condition_unmet', + 'skipped_at', ss.skipped_at + ), + concat('step:', ss.step_slug, ':skipped'), + concat('pgflow:run:', ss.run_id), + false + ) AS _broadcast_result + ), + -- NEW: Update dependent steps (decrement remaining_deps by count of skipped parents, set initial_tasks=0 for maps) + skipped_parent_counts AS ( + -- Count how many skipped parents each child has + SELECT + dep.step_slug AS child_step_slug, + dep.flow_slug AS child_flow_slug, + COUNT(*) AS skipped_parent_count + FROM skipped_steps parent + JOIN pgflow.deps dep ON dep.flow_slug = parent.flow_slug AND dep.dep_slug = parent.step_slug + GROUP BY dep.step_slug, dep.flow_slug + ), + dependent_updates AS ( + UPDATE pgflow.step_states child_state + SET remaining_deps = child_state.remaining_deps - spc.skipped_parent_count, + -- If child is a map step and this skipped step is its only dependency, + -- set initial_tasks = 0 (skipped dep = empty array) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_step.deps_count = 1 THEN 0 + ELSE child_state.initial_tasks + END + FROM skipped_parent_counts spc + JOIN pgflow.steps child_step ON child_step.flow_slug = spc.child_flow_slug AND child_step.step_slug = spc.child_step_slug + WHERE child_state.run_id = cascade_resolve_conditions.run_id + AND child_state.step_slug = spc.child_step_slug + ), + run_update AS ( + UPDATE pgflow.runs r + SET remaining_steps = r.remaining_steps - (SELECT COUNT(*) FROM skipped_steps) + WHERE r.run_id = cascade_resolve_conditions.run_id + AND (SELECT COUNT(*) FROM skipped_steps) > 0 + ) + SELECT COUNT(*)::int INTO v_processed_count FROM skipped_steps; + + -- ========================================== + -- PHASE 1c: HANDLE SKIP-CASCADE CONDITIONS + -- ========================================== + -- Call _cascade_force_skip_steps for each step with unmet condition and whenUnmet='skip-cascade'. + -- Process in topological order; _cascade_force_skip_steps is idempotent. + PERFORM pgflow._cascade_force_skip_steps(cascade_resolve_conditions.run_id, ready_step.step_slug, 'condition_unmet') + FROM pgflow.step_states AS ready_step + JOIN pgflow.steps AS step + ON step.flow_slug = ready_step.flow_slug + AND step.step_slug = ready_step.step_slug + LEFT JOIN LATERAL ( + SELECT jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM pgflow.deps dep + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE dep.flow_slug = ready_step.flow_slug + AND dep.step_slug = ready_step.step_slug + ) AS agg_deps ON step.deps_count > 0 + WHERE ready_step.run_id = cascade_resolve_conditions.run_id + AND ready_step.status = 'created' + AND ready_step.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + AND step.when_unmet = 'skip-cascade' + -- Condition is NOT met when: (if fails) OR (ifNot fails) + AND NOT ( + (step.required_input_pattern IS NULL OR + CASE WHEN step.deps_count = 0 THEN v_run_input ELSE COALESCE(agg_deps.deps_output, '{}'::jsonb) END @> step.required_input_pattern) + AND + (step.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN step.deps_count = 0 THEN v_run_input ELSE COALESCE(agg_deps.deps_output, '{}'::jsonb) END @> step.forbidden_input_pattern)) + ) + ORDER BY step.step_index; + + -- Check if run was failed during cascade (e.g., if _cascade_force_skip_steps triggers fail) + SELECT r.status INTO v_run_status + FROM pgflow.runs r + WHERE r.run_id = cascade_resolve_conditions.run_id; + + IF v_run_status IN ('failed', 'completed') THEN + RETURN v_run_status != 'failed'; + END IF; + + -- Exit loop if no steps were processed in this iteration + EXIT WHEN v_processed_count = 0; + END LOOP; + + RETURN true; +END; +$$; +-- Modify "start_ready_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +BEGIN +-- ========================================== +-- GUARD: No mutations on terminal runs +-- ========================================== +IF EXISTS ( + SELECT 1 FROM pgflow.runs + WHERE pgflow.runs.run_id = start_ready_steps.run_id + AND pgflow.runs.status IN ('failed', 'completed') +) THEN + RETURN; +END IF; + +-- ========================================== +-- PHASE 1: START READY STEPS +-- ========================================== +-- NOTE: Condition evaluation and empty map handling are done by +-- cascade_resolve_conditions() and cascade_complete_taskless_steps() +-- which are called before this function. +WITH +-- ---------- Find ready steps ---------- +-- Steps with no remaining deps and known task count +ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND step_state.initial_tasks IS NOT NULL -- Cannot start with unknown count + AND step_state.initial_tasks > 0 -- Don't start taskless steps (handled by cascade_complete_taskless_steps) + ORDER BY step_state.step_slug + FOR UPDATE +), +-- ---------- Mark steps as started ---------- +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now(), + remaining_tasks = ready_steps.initial_tasks -- Copy initial_tasks to remaining_tasks when starting + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.*, + -- Broadcast step:started event atomically with the UPDATE + -- Using RETURNING ensures this executes during row processing + -- and cannot be optimized away by the query planner + realtime.send( + jsonb_build_object( + 'event_type', 'step:started', + 'run_id', pgflow.step_states.run_id, + 'step_slug', pgflow.step_states.step_slug, + 'status', 'started', + 'started_at', pgflow.step_states.started_at, + 'remaining_tasks', pgflow.step_states.remaining_tasks, + 'remaining_deps', pgflow.step_states.remaining_deps + ), + concat('step:', pgflow.step_states.step_slug, ':started'), + concat('pgflow:run:', pgflow.step_states.run_id), + false + ) as _broadcast_result -- Prefix with _ to indicate internal use only +), + +-- ========================================== +-- PHASE 2: TASK GENERATION AND QUEUE MESSAGES +-- ========================================== +-- ---------- Generate tasks and batch messages ---------- +-- Single steps: 1 task (index 0) +-- Map steps: N tasks (indices 0..N-1) +message_batches AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + step.queue_name, + COALESCE(step.opt_start_delay, 0) as delay, + array_agg( + jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', task_idx.task_index + ) ORDER BY task_idx.task_index + ) AS messages, + array_agg(task_idx.task_index ORDER BY task_idx.task_index) AS task_indices + FROM started_step_states AS started_step + JOIN pgflow.steps AS step + ON step.flow_slug = started_step.flow_slug + AND step.step_slug = started_step.step_slug + -- Generate task indices from 0 to initial_tasks-1 + CROSS JOIN LATERAL generate_series(0, started_step.initial_tasks - 1) AS task_idx(task_index) + GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.queue_name, step.opt_start_delay +), +-- ---------- Send messages to queue ---------- +-- Uses batch sending for performance with large arrays. +-- Messages go through the step's persisted queue route (#650); PGMQ +-- message operations normalize names themselves. +sent_messages AS ( + SELECT + mb.flow_slug, + mb.run_id, + mb.step_slug, + mb.queue_name, + task_indices.task_index, + msg_ids.msg_id + FROM message_batches mb + CROSS JOIN LATERAL unnest(mb.task_indices) WITH ORDINALITY AS task_indices(task_index, idx_ord) + CROSS JOIN LATERAL pgmq.send_batch( + mb.queue_name, + mb.messages, + mb.delay + ) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) + WHERE task_indices.idx_ord = msg_ids.msg_ord +) + +-- ========================================== +-- PHASE 3: RECORD TASKS IN DATABASE +-- ========================================== +-- The task stores the step's queue_name snapshot; runtime message operations +-- use that snapshot, never a queue reconstructed from flow_slug (#650). +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.queue_name, + sent_messages.task_index, + sent_messages.msg_id +FROM sent_messages; + +END; +$$; +-- Modify "complete_task" function +CREATE OR REPLACE FUNCTION "pgflow"."complete_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_step_state pgflow.step_states%ROWTYPE; + v_dependent_map_slug text; + v_run_record pgflow.runs%ROWTYPE; + v_step_record pgflow.step_states%ROWTYPE; + v_violation_archived_queues int; +begin + +-- ========================================== +-- GUARD: No mutations on failed runs +-- ========================================== +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = complete_task.run_id AND pgflow.runs.status = 'failed') THEN + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- LOCK ACQUISITION AND TYPE VALIDATION +-- ========================================== +-- Acquire locks first to prevent race conditions +SELECT * INTO v_run_record FROM pgflow.runs +WHERE pgflow.runs.run_id = complete_task.run_id +FOR UPDATE; + +SELECT * INTO v_step_record FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug +FOR UPDATE; + +-- ========================================== +-- GUARD: Run failed while this callback waited for the lock +-- ========================================== +-- The failed-run guard above ran before the failure committed. Recheck under +-- lock so cancellation wins: archived message stays archived, task row keeps +-- its terminal status, and no events or counters are emitted. +IF v_run_record.status = 'failed' THEN + -- Archive the task message if present (no-op when already archived) + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + -- Return the current task row without any mutations + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- GUARD: Late callback - step not started +-- ========================================== +-- If the step is not in 'started' state, this is a late callback. +-- Do not mutate step_states or runs, archive message, return task row. +IF v_step_record.status != 'started' THEN + -- Archive the task message if present (prevents stuck work), through the + -- task's stored queue snapshot (#650) + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + -- Return the current task row without any mutations + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- Check for type violations AFTER acquiring locks +SELECT child_step.step_slug INTO v_dependent_map_slug +FROM pgflow.deps dependency +JOIN pgflow.steps child_step ON child_step.flow_slug = dependency.flow_slug + AND child_step.step_slug = dependency.step_slug +JOIN pgflow.steps parent_step ON parent_step.flow_slug = dependency.flow_slug + AND parent_step.step_slug = dependency.dep_slug +JOIN pgflow.step_states child_state ON child_state.flow_slug = child_step.flow_slug + AND child_state.step_slug = child_step.step_slug +WHERE dependency.dep_slug = complete_task.step_slug -- parent is the completing step + AND dependency.flow_slug = v_run_record.flow_slug + AND parent_step.step_type = 'single' -- Only validate single steps + AND child_step.step_type = 'map' + AND child_state.run_id = complete_task.run_id + AND child_state.initial_tasks IS NULL + AND (complete_task.output IS NULL OR jsonb_typeof(complete_task.output) != 'array') +LIMIT 1; + +-- Handle type violation if detected +IF v_dependent_map_slug IS NOT NULL THEN + -- Mark current task as failed FIRST and store the output that caused the + -- violation, so the task row is terminal before any queue row is touched. + UPDATE pgflow.step_tasks + SET status = 'failed', + failed_at = now(), + output = complete_task.output, -- Store the output that caused the violation + error_message = '[TYPE_VIOLATION] Produced ' || + CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END || + ' instead of array' + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + + -- Mark run as failed immediately + UPDATE pgflow.runs + SET status = 'failed', + failed_at = now() + WHERE pgflow.runs.run_id = complete_task.run_id; + + -- Broadcast run:failed event + -- Uses PERFORM pattern to ensure execution (proven reliable pattern in this function) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', complete_task.run_id, + 'flow_slug', v_run_record.flow_slug, + 'status', 'failed', + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- Mark step state as failed + UPDATE pgflow.step_states + SET status = 'failed', + failed_at = now(), + error_message = '[TYPE_VIOLATION] Map step ' || v_dependent_map_slug || + ' expects array input but dependency ' || complete_task.step_slug || + ' produced ' || CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug; + + -- Broadcast step:failed event + -- Uses PERFORM pattern to ensure execution (proven reliable pattern in this function) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'failed', + 'error_message', '[TYPE_VIOLATION] Map step ' || v_dependent_map_slug || + ' expects array input but dependency ' || complete_task.step_slug || + ' produced ' || CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END, + 'failed_at', now() + ), + concat('step:', complete_task.step_slug, ':failed'), + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- Terminalize every other unfinished task as cancelled, then archive the + -- culprit and cancelled messages batched per stored queue route (#650). + -- Lock-order invariant: always lock/update step_tasks before PGMQ queue + -- rows; the archive reads the terminalized rows through the CTE. + -- The culprit task is already terminal (failed above), so it is excluded + -- from the cancellation set. + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = complete_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.message_id, task.queue_name + ), + culprit_task AS ( + -- Terminal culprit row: safe to read for its message id after terminalization + SELECT st.message_id, st.queue_name + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.message_id IS NOT NULL + ), + terminal_messages AS ( + SELECT message_id, queue_name FROM culprit_task + UNION ALL + SELECT message_id, queue_name FROM cancelled_tasks WHERE message_id IS NOT NULL + ), + archived_messages AS ( + SELECT pgmq.archive( + tm.queue_name, + ARRAY_AGG(tm.message_id) + ) + FROM terminal_messages tm + GROUP BY tm.queue_name + ) + SELECT COUNT(*)::int INTO v_violation_archived_queues + FROM archived_messages; + + -- Return the failed task row (API contract: always return task row) + RETURN QUERY + SELECT * FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- MAIN CTE CHAIN: Update task and propagate changes +-- ========================================== +WITH +-- ---------- Task completion ---------- +-- Update the task record with completion status and output +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + AND pgflow.step_tasks.status = 'started' + RETURNING * +), +-- ---------- Get step type for output handling ---------- +step_def AS ( + SELECT step.step_type + FROM pgflow.steps step + JOIN pgflow.runs run ON run.flow_slug = step.flow_slug + WHERE run.run_id = complete_task.run_id + AND step.step_slug = complete_task.step_slug +), +-- ---------- Step state update ---------- +-- Decrement remaining_tasks and potentially mark step as completed +-- Also store output atomically with status transition to completed +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1, + -- Store output atomically with completion (only when remaining_tasks = 1, meaning step completes) + output = CASE + -- Single step: store task output directly when completing + WHEN (SELECT step_type FROM step_def) = 'single' AND pgflow.step_states.remaining_tasks = 1 THEN + complete_task.output + -- Map step: aggregate on completion (ordered by task_index) + WHEN (SELECT step_type FROM step_def) = 'map' AND pgflow.step_states.remaining_tasks = 1 THEN + (SELECT COALESCE(jsonb_agg(all_outputs.output ORDER BY all_outputs.task_index), '[]'::jsonb) + FROM ( + -- All previously completed tasks + SELECT st.output, st.task_index + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.status = 'completed' + UNION ALL + -- Current task being completed (not yet visible as completed in snapshot) + SELECT complete_task.output, complete_task.task_index + ) all_outputs) + ELSE pgflow.step_states.output + END + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- ---------- Dependency resolution ---------- +-- Find all child steps that depend on the completed parent step (only if parent completed) +child_steps AS ( + SELECT deps.step_slug AS child_step_slug + FROM pgflow.deps deps + JOIN step_state parent_state ON parent_state.status = 'completed' AND deps.flow_slug = parent_state.flow_slug + WHERE deps.dep_slug = complete_task.step_slug -- dep_slug is the parent, step_slug is the child + ORDER BY deps.step_slug -- Ensure consistent ordering +), +-- ---------- Lock child steps ---------- +-- Acquire locks on all child steps before updating them +child_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT child_step_slug FROM child_steps) + FOR UPDATE +), +-- ---------- Update child steps ---------- +-- Decrement remaining_deps and resolve NULL initial_tasks for map steps +child_steps_update AS ( + UPDATE pgflow.step_states child_state + SET remaining_deps = child_state.remaining_deps - 1, + -- Resolve NULL initial_tasks for child map steps + -- This is where child maps learn their array size from the parent + -- This CTE only runs when the parent step is complete (see child_steps JOIN) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_state.initial_tasks IS NULL THEN + CASE + WHEN parent_step.step_type = 'map' THEN + -- Map->map: Count all completed tasks from parent map + -- We add 1 because the current task is being completed in this transaction + -- but isn't yet visible as 'completed' in the step_tasks table + -- TODO: Refactor to use future column step_states.total_tasks + -- Would eliminate the COUNT query and just use parent_state.total_tasks + (SELECT COUNT(*)::int + 1 + FROM pgflow.step_tasks parent_tasks + WHERE parent_tasks.run_id = complete_task.run_id + AND parent_tasks.step_slug = complete_task.step_slug + AND parent_tasks.status = 'completed' + AND parent_tasks.task_index != complete_task.task_index) + ELSE + -- Single->map: Use output array length (single steps complete immediately) + CASE + WHEN complete_task.output IS NOT NULL + AND jsonb_typeof(complete_task.output) = 'array' THEN + jsonb_array_length(complete_task.output) + ELSE NULL -- Keep NULL if not an array + END + END + ELSE child_state.initial_tasks -- Keep existing value (including NULL) + END + FROM child_steps children + JOIN pgflow.steps child_step ON child_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND child_step.step_slug = children.child_step_slug + JOIN pgflow.steps parent_step ON parent_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND parent_step.step_slug = complete_task.step_slug + WHERE child_state.run_id = complete_task.run_id + AND child_state.step_slug = children.child_step_slug +) +-- ---------- Update run remaining_steps ---------- +-- Decrement the run's remaining_steps counter if step completed +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +-- ========================================== +-- POST-COMPLETION ACTIONS +-- ========================================== + +-- ---------- Get updated state for broadcasting ---------- +SELECT * INTO v_step_state FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id AND pgflow.step_states.step_slug = complete_task.step_slug; + +-- ---------- Handle step completion ---------- +IF v_step_state.status = 'completed' THEN + -- Broadcast step:completed event FIRST (before cascade) + -- This ensures parent broadcasts before its dependent children + -- Use stored output from step_states (set atomically during status transition) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:completed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'completed', + 'output', v_step_state.output, -- Use stored output instead of re-aggregating + 'completed_at', v_step_state.completed_at + ), + concat('step:', complete_task.step_slug, ':completed'), + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- THEN evaluate conditions on newly-ready dependent steps + -- This must happen before cascade_complete_taskless_steps so that + -- skipped steps can set initial_tasks=0 for their map dependents + IF NOT pgflow.cascade_resolve_conditions(complete_task.run_id) THEN + -- Run was failed due to a condition with when_unmet='fail' + -- Archive the current task's message before returning + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; + END IF; + + -- THEN cascade complete any taskless steps that are now ready + -- This ensures dependent children broadcast AFTER their parent + PERFORM pgflow.cascade_complete_taskless_steps(complete_task.run_id); +END IF; + +-- ---------- Archive completed task message ---------- +-- Move message from active queue to archive table, through the task's +-- stored queue snapshot (#650) +PERFORM ( + WITH completed_tasks AS ( + SELECT st.queue_name, st.message_id + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.status = 'completed' + ) + SELECT pgmq.archive(ct.queue_name, ct.message_id) + FROM completed_tasks ct + WHERE EXISTS (SELECT 1 FROM completed_tasks) +); + +-- ---------- Trigger next steps ---------- +-- Start any steps that are now ready (deps satisfied) +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +-- Check if the entire run is complete +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +-- ---------- Return completed task ---------- +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Modify "create_flow" function +CREATE OR REPLACE FUNCTION "pgflow"."create_flow" ("flow_slug" text, "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer) RETURNS "pgflow"."flows" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_flow pgflow.flows; +begin + if not exists ( + select 1 + from pgflow.flows as flow + where lower(flow.flow_slug) = lower(create_flow.flow_slug) + ) and exists ( + select 1 + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(create_flow.flow_slug) + ) then + raise exception + 'cannot create flow "%": queue "%" is already in use by another owner', + create_flow.flow_slug, lower(create_flow.flow_slug) + using errcode = 'unique_violation'; + end if; + + insert into pgflow.flows (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) + values ( + create_flow.flow_slug, + coalesce(max_attempts, 3), + coalesce(base_delay, 5), + coalesce(timeout, 60) + ) + on conflict on constraint flows_pkey + do update + set flow_slug = pgflow.flows.flow_slug -- Dummy update + returning * into v_flow; + + -- Ensure the default queue exists, including for an empty flow. Reuse the + -- listed queue when it exists under any spelling of the normalized name. + if not exists ( + select 1 + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(create_flow.flow_slug) + ) then + perform pgmq.create(lower(create_flow.flow_slug)); + end if; + + return v_flow; +end; +$$; +-- Create "_listed_queue_name" function +CREATE FUNCTION "pgflow"."_listed_queue_name" ("p_queue_name" text) RETURNS text LANGUAGE plpgsql STABLE SET "search_path" = '' AS $$ +declare + v_matches text[]; +begin + -- Resolve every listed spelling of the normalized name before preferring + -- any single match: an ambiguous pair is rejected even when one spelling + -- is the exact requested name (#650). + select array_agg(listed.queue_name order by listed.queue_name) + into v_matches + from pgmq.list_queues() as listed + where lower(listed.queue_name) = lower(p_queue_name); + + if v_matches is null then + -- Not listed: pass through; PGMQ reports its own error (or no-ops) + return p_queue_name; + elsif cardinality(v_matches) > 1 then + raise exception + 'queue name "%" is ambiguous: it matches listed queues %', + p_queue_name, v_matches + using errcode = 'ambiguous_alias'; + else + return v_matches[1]; + end if; +end; +$$; +-- Modify "delete_flow_and_data" function +CREATE OR REPLACE FUNCTION "pgflow"."delete_flow_and_data" ("p_flow_slug" text) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +BEGIN + -- Only an exact pgflow.flows row authorizes destructive queue work: a + -- nonexistent or wrong-case slug must not drop any queue (#650). The + -- data deletes below stay exact-match and no-op without the row. + IF EXISTS ( + SELECT 1 FROM pgflow.flows AS flow WHERE flow.flow_slug = p_flow_slug + ) THEN + -- Drop queues and archive tables (pgmq) using persisted routes. The + -- listed spelling is resolved fresh on every call; message operations + -- never need it because PGMQ normalizes names itself. + PERFORM pgmq.drop_queue(pgflow._listed_queue_name(route.queue_name)) + FROM ( + SELECT DISTINCT queue_name + FROM pgflow.steps + WHERE flow_slug = p_flow_slug + UNION + -- Empty flow: no persisted routes, fall back to the default queue + SELECT lower(p_flow_slug) + WHERE NOT EXISTS ( + SELECT 1 FROM pgflow.steps WHERE flow_slug = p_flow_slug + ) + ) AS route; + END IF; + + -- Delete all associated data in the correct order (respecting FK constraints) + DELETE FROM pgflow.step_tasks AS task WHERE task.flow_slug = p_flow_slug; + DELETE FROM pgflow.step_states AS state WHERE state.flow_slug = p_flow_slug; + DELETE FROM pgflow.runs AS run WHERE run.flow_slug = p_flow_slug; + DELETE FROM pgflow.deps AS dep WHERE dep.flow_slug = p_flow_slug; + DELETE FROM pgflow.steps AS step WHERE step.flow_slug = p_flow_slug; + DELETE FROM pgflow.flows AS flow WHERE flow.flow_slug = p_flow_slug; +END; +$$; +-- Modify "ensure_flow_compiled" function +CREATE OR REPLACE FUNCTION "pgflow"."ensure_flow_compiled" ("flow_slug" text, "shape" jsonb) RETURNS jsonb LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_lock_key int; + v_flow_exists boolean; + v_db_shape jsonb; + v_differences text[]; + v_is_local boolean; +BEGIN + -- Generate lock key from the normalized flow identity (deterministic hash) + v_lock_key := hashtext(lower(ensure_flow_compiled.flow_slug)); + + -- Acquire transaction-level advisory lock + -- Serializes concurrent compilation attempts for same flow + PERFORM pg_advisory_xact_lock(1, v_lock_key); + + -- 1. Check if flow exists + SELECT EXISTS(SELECT 1 FROM pgflow.flows AS flow WHERE flow.flow_slug = ensure_flow_compiled.flow_slug) + INTO v_flow_exists; + + -- 2. If flow missing: compile (both environments) + IF NOT v_flow_exists THEN + PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + RETURN jsonb_build_object('status', 'compiled', 'differences', '[]'::jsonb); + END IF; + + -- 3. Get current shape from DB + v_db_shape := pgflow._get_flow_shape(ensure_flow_compiled.flow_slug); + + -- 4. Compare shapes + v_differences := pgflow._compare_flow_shapes(ensure_flow_compiled.shape, v_db_shape); + + -- 5. If shapes match: return verified + IF array_length(v_differences, 1) IS NULL THEN + RETURN jsonb_build_object('status', 'verified', 'differences', '[]'::jsonb); + END IF; + + -- 6. Shapes differ - auto-detect environment via is_local() + v_is_local := pgflow.is_local(); + + -- Local mode is the only destructive branch; production mismatches never + -- delete data and return mismatch so worker startup fails. + IF v_is_local THEN + -- Recompile in local/dev: full deletion + fresh compile + PERFORM pgflow.delete_flow_and_data(ensure_flow_compiled.flow_slug); + PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + RETURN jsonb_build_object('status', 'recompiled', 'differences', to_jsonb(v_differences)); + ELSE + -- Fail in production + RETURN jsonb_build_object('status', 'mismatch', 'differences', to_jsonb(v_differences)); + END IF; +END; +$$; +-- Modify "requeue_stalled_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."requeue_stalled_tasks" () RETURNS integer LANGUAGE plpgsql SECURITY DEFINER SET "search_path" = '' AS $$ +declare + result_count int := 0; + max_requeues constant int := 3; +begin + -- Find and requeue stalled tasks (where started_at > effective timeout + 30s buffer) + -- Tasks with requeued_count >= max_requeues will have their message archived + -- but status left as 'started' for easy identification via requeued_count column + -- Eligibility requires the parent run AND parent step to still be 'started': + -- stale rows on failed runs or terminal steps must not be revived (#645). + with stalled_tasks as ( + select + st.run_id, + st.step_slug, + st.task_index, + st.message_id, + st.queue_name, + r.flow_slug, + st.requeued_count + from pgflow.step_tasks st + join pgflow.runs r on r.run_id = st.run_id + join pgflow.step_states ss on ss.run_id = st.run_id and ss.step_slug = st.step_slug + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = st.step_slug + where st.status = 'started' + and r.status = 'started' + and ss.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + for update of st skip locked + ), + -- Separate tasks that can be requeued from those that exceeded max requeues + to_requeue as ( + select * from stalled_tasks where requeued_count < max_requeues + ), + to_archive as ( + select * from stalled_tasks where requeued_count >= max_requeues + ), + -- Update tasks that will be requeued + requeued as ( + update pgflow.step_tasks st + set + status = 'queued', + started_at = null, + last_worker_id = null, + requeued_count = st.requeued_count + 1, + last_requeued_at = now() + from to_requeue tr + where st.run_id = tr.run_id + and st.step_slug = tr.step_slug + and st.task_index = tr.task_index + returning tr.queue_name as queue_name, tr.message_id + ), + -- Make requeued messages visible immediately (batched per queue, through + -- the tasks' stored queue snapshots #650; PGMQ message operations + -- normalize names themselves) + visibility_reset as ( + select pgflow.set_vt_batch( + r.queue_name, + array_agg(r.message_id), + array_agg(0) -- all offsets are 0 (immediate visibility) + ) + from requeued r + where r.message_id is not null + group by r.queue_name + ), + -- Mark tasks as permanently stalled before archiving + mark_permanently_stalled as ( + update pgflow.step_tasks st + set permanently_stalled_at = now() + from to_archive ta + where st.run_id = ta.run_id + and st.step_slug = ta.step_slug + and st.task_index = ta.task_index + returning st.run_id + ), + -- Archive messages for tasks that exceeded max requeues (batched per queue) + archived as ( + select pgmq.archive( + ta.queue_name, + array_agg(ta.message_id) + ) + from to_archive ta + where ta.message_id is not null + group by ta.queue_name + ), + -- Force execution of visibility_reset CTE + _vr as (select count(*) from visibility_reset), + -- Force execution of mark_permanently_stalled CTE + _mps as (select count(*) from mark_permanently_stalled), + -- Force execution of archived CTE + _ar as (select count(*) from archived) + select count(*) into result_count + from requeued, _vr, _mps, _ar; + + return result_count; +end; +$$; +-- Modify "fail_task" function +CREATE OR REPLACE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_failed boolean; + v_step_failed boolean; + v_step_skipped boolean; + v_when_exhausted text; + v_task_exhausted boolean; + v_flow_slug_for_deps text; + v_prev_step_status text; + v_run_status text; + v_flow_slug text; + v_archived_queues int; +begin + +-- If run is already failed, no retries allowed. +-- Cancellation wins: tasks terminalized by the run failure (failed culprit or +-- cancelled siblings) keep their terminal status. This late callback only +-- archives any still-active message and returns the current row unchanged. +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id AND pgflow.runs.status = 'failed') THEN + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +-- Late callback guard: lock run + step rows and use current statuses +-- under lock so concurrent fail_task calls cannot read stale status. +SELECT ss.status, r.status, r.flow_slug INTO v_prev_step_status, v_run_status, v_flow_slug +FROM pgflow.runs r +JOIN pgflow.step_states ss ON ss.run_id = r.run_id +WHERE ss.run_id = fail_task.run_id + AND ss.step_slug = fail_task.step_slug +FOR UPDATE OF r, ss; + +-- Recheck under lock: the run may have failed while this callback waited +-- for the lock (the EXISTS guard above ran before the failure committed). +IF v_run_status = 'failed' THEN + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +IF v_prev_step_status IS NOT NULL AND v_prev_step_status != 'started' THEN + -- Archive the task message if present, through the task's stored queue + -- snapshot (#650) + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +WITH flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), + config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay, + s.when_exhausted + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), + -- Determine if task exhausted retries and get when_exhausted mode + task_status AS ( + SELECT + (select status from fail_or_retry_task) AS new_task_status, + (select when_exhausted from config) AS when_exhausted_mode, + -- Task is exhausted when it's failed (no more retries) + ((select status from fail_or_retry_task) = 'failed') AS is_exhausted +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + -- Status logic: + -- - If task not exhausted (retrying): keep current status + -- - If exhausted AND when_exhausted='fail': set to 'failed' + -- - If exhausted AND when_exhausted IN ('skip', 'skip-cascade'): set to 'skipped' + status = CASE + WHEN NOT (select is_exhausted from task_status) THEN pgflow.step_states.status + WHEN (select when_exhausted_mode from task_status) = 'fail' THEN 'failed' + ELSE 'skipped' -- skip or skip-cascade + END, + failed_at = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) = 'fail' THEN now() + ELSE NULL + END, + error_message = CASE + WHEN (select is_exhausted from task_status) THEN fail_task.error_message + ELSE NULL + END, + skip_reason = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN 'handler_failed' + ELSE pgflow.step_states.skip_reason + END, + skipped_at = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN now() + ELSE pgflow.step_states.skipped_at + END, + -- Clear remaining_tasks when skipping (required by remaining_tasks_state_consistency constraint) + remaining_tasks = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN NULL + ELSE pgflow.step_states.remaining_tasks + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +), +run_update AS ( + -- Update run status: only fail when when_exhausted='fail' and step was failed + UPDATE pgflow.runs + SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END, + -- Decrement remaining_steps only on FIRST transition to skipped + -- (not when step was already skipped and a second task fails) + -- Uses PL/pgSQL variable captured before CTE chain + remaining_steps = CASE + WHEN (select status from maybe_fail_step) = 'skipped' + AND v_prev_step_status != 'skipped' + THEN pgflow.runs.remaining_steps - 1 + ELSE pgflow.runs.remaining_steps + END + WHERE pgflow.runs.run_id = fail_task.run_id + RETURNING pgflow.runs.status +) +SELECT + COALESCE((SELECT status = 'failed' FROM run_update), false), + COALESCE((SELECT status = 'failed' FROM maybe_fail_step), false), + COALESCE((SELECT status = 'skipped' FROM maybe_fail_step), false), + COALESCE((SELECT is_exhausted FROM task_status), false) +INTO v_run_failed, v_step_failed, v_step_skipped, v_task_exhausted; + + -- Capture when_exhausted mode for later skip handling + SELECT s.when_exhausted INTO v_when_exhausted + FROM pgflow.steps s +JOIN pgflow.runs r ON r.flow_slug = s.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug; + +-- Send broadcast event for step failure if the step was failed +IF v_task_exhausted AND v_step_failed THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + concat('step:', fail_task.step_slug, ':failed'), + concat('pgflow:run:', fail_task.run_id), + false + ); +END IF; + +-- Handle step skipping (when_exhausted = 'skip' or 'skip-cascade') + IF v_task_exhausted AND v_step_skipped THEN + -- Lock-order invariant: always lock/update step_tasks before PGMQ queue rows. + -- requeue_stalled_tasks() uses the same order; archiving queue rows first + -- deadlocks the two transactions against each other. + -- Terminalize all still-active sibling task rows for the skipped step, then + -- archive their messages batched per stored queue route (#650); the + -- archive reads the terminalized rows through the CTE. + WITH skipped_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'skipped' + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.status IN ('queued', 'started') + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + st.queue_name, + ARRAY_AGG(st.message_id) + ) + FROM skipped_tasks st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name + ) + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; + + -- Send broadcast event for step skipped + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'skipped', + 'skip_reason', 'handler_failed', + 'error_message', fail_task.error_message, + 'skipped_at', now() + ), + concat('step:', fail_task.step_slug, ':skipped'), + concat('pgflow:run:', fail_task.run_id), + false + ); + + -- For skip-cascade: cascade skip to all downstream dependents + IF v_when_exhausted = 'skip-cascade' THEN + PERFORM pgflow._cascade_force_skip_steps(fail_task.run_id, fail_task.step_slug, 'handler_failed'); + ELSE + -- For plain 'skip': decrement remaining_deps on dependent steps + -- (This mirrors the pattern in cascade_resolve_conditions.sql for when_unmet='skip') + SELECT flow_slug INTO v_flow_slug_for_deps + FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id; + + UPDATE pgflow.step_states AS child_state + SET remaining_deps = child_state.remaining_deps - 1, + -- If child is a map step and this skipped step is its only dependency, + -- set initial_tasks = 0 (skipped dep = empty array) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_step.deps_count = 1 THEN 0 + ELSE child_state.initial_tasks + END + FROM pgflow.deps AS dep + JOIN pgflow.steps AS child_step ON child_step.flow_slug = dep.flow_slug AND child_step.step_slug = dep.step_slug + WHERE child_state.run_id = fail_task.run_id + AND dep.flow_slug = v_flow_slug_for_deps + AND dep.dep_slug = fail_task.step_slug + AND child_state.step_slug = dep.step_slug; + + -- Evaluate conditions on newly-ready dependent steps + -- This must happen before cascade_complete_taskless_steps so that + -- skipped steps can set initial_tasks=0 for their map dependents + IF NOT pgflow.cascade_resolve_conditions(fail_task.run_id) THEN + -- Run was failed due to a condition with when_unmet='fail' + -- Archive the failed task's message before returning + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + -- Return the task row (API contract) + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; + END IF; + + -- Auto-complete taskless steps (e.g., map steps with initial_tasks=0 from skipped dep) + PERFORM pgflow.cascade_complete_taskless_steps(fail_task.run_id); + + -- Start steps that became ready after condition resolution and taskless completion + PERFORM pgflow.start_ready_steps(fail_task.run_id); + END IF; + + -- Try to complete the run (remaining_steps may now be 0) + PERFORM pgflow.maybe_complete_run(fail_task.run_id); +END IF; + +-- Send broadcast event for run failure if the run was failed +IF v_run_failed THEN + DECLARE + v_flow_slug text; + BEGIN + SELECT flow_slug INTO v_flow_slug FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', fail_task.run_id, + 'flow_slug', v_flow_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', fail_task.run_id), + false + ); + END; +END IF; + +-- Terminalize unfinished tasks as cancelled when the run fails, then archive +-- their messages batched per stored queue route (#650). Lock-order invariant: +-- always lock/update step_tasks before PGMQ queue rows; the archive reads the +-- terminalized rows through the CTE. The culprit task is already terminal +-- (failed or requeued by fail_or_retry_task), so only unfinished queued/started +-- siblings are cancelled. +IF v_run_failed THEN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = fail_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.message_id, task.queue_name + ), + archived_messages AS ( + SELECT pgmq.archive( + ct.queue_name, + ARRAY_AGG(ct.message_id) + ) + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + ) + SELECT COUNT(*)::int INTO v_archived_queues + FROM archived_messages; +END IF; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + st.queue_name, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt( + qt.queue_name, + qt.message_id, + qt.calculated_delay + ) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message, through the task's stored queue +-- snapshot (#650) +PERFORM pgmq.archive(st.queue_name, ARRAY_AGG(st.message_id)) +FROM pgflow.step_tasks st +WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + AND st.message_id IS NOT NULL +GROUP BY st.queue_name +HAVING COUNT(st.message_id) > 0; + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Create "start_tasks" function +CREATE FUNCTION "pgflow"."start_tasks" ("flow_slug" text, "msg_ids" bigint[], "worker_id" uuid, "queue_name" text) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE sql SET "search_path" = '' AS $$ +with task_candidates as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + join pgflow.runs r on r.run_id = task.run_id + where task.flow_slug = start_tasks.flow_slug + and task.queue_name = start_tasks.queue_name + and task.message_id = any(msg_ids) + and task.status = 'queued' + and r.status = 'started' + and exists ( + select 1 + from pgflow.step_states ss + where ss.run_id = task.run_id + and ss.step_slug = task.step_slug + and ss.status = 'started' + ) + ), + -- Claim rows with a guarded update and return only what was actually + -- claimed. A concurrent skip can win the row lock between the candidate + -- select and this update; the status = 'queued' recheck then claims nothing, + -- so no stale candidate row must escape to the worker (#638). + tasks as ( + update pgflow.step_tasks + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = worker_id + from task_candidates as candidate + where step_tasks.message_id = candidate.message_id + and step_tasks.flow_slug = candidate.flow_slug + and step_tasks.queue_name = start_tasks.queue_name + and step_tasks.status = 'queued' + returning + step_tasks.flow_slug, + step_tasks.run_id, + step_tasks.step_slug, + step_tasks.task_index, + step_tasks.message_id + ), + runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + -- Read output directly from step_states (already aggregated by writers) + dep_state.output as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_states dep_state on + dep_state.run_id = st.run_id and + dep_state.step_slug = dep.dep_slug and + dep_state.status = 'completed' -- Only include completed deps (not skipped) + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output, + count(*) as dep_count + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + task.message_id, + task.flow_slug, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug + ), + -- Batch update visibility timeouts for all messages. + -- The final statement must force this CTE to run: an unreferenced SELECT + -- CTE is not guaranteed to execute, which would leave a claimed task with + -- only the shorter initial PGMQ read visibility (#656). + visibility_reset as ( + select pgflow.set_vt_batch( + start_tasks.queue_name, + array_agg(t.message_id order by t.message_id), + array_agg(t.vt_delay order by t.message_id) + ) + from timeouts t + ), + -- Force execution of the visibility_reset CTE (same pattern as + -- requeue_stalled_tasks) and guard completeness: set_vt_batch updates + -- only queue rows it finds, so fewer returned rows than claimed tasks + -- means a visibility extension did not run (#656). SQL functions cannot + -- RAISE, so the mismatch branch casts a descriptive message to int4: + -- the cast error fails the whole statement, rolling back the task + -- transition and attempt increment, and returns nothing. + _vr as ( + select case + when updated.updated_count = claimed.claimed_count then updated.updated_count + else format( + 'start_tasks(): visibility updated %s of %s claimed messages', + updated.updated_count, + claimed.claimed_count + )::int4 + end as visibility_updates + from (select count(*) as updated_count from visibility_reset) as updated + cross join (select count(*) as claimed_count from tasks) as claimed + ) + select + st.flow_slug, + st.run_id, + st.step_slug, + -- ========================================== + -- INPUT CONSTRUCTION LOGIC + -- ========================================== + -- This nested CASE statement determines how to construct the input + -- for each task based on the step type (map vs non-map). + -- + -- The fundamental difference: + -- - Map steps: Receive RAW array elements (e.g., just 42 or "hello") + -- - Non-map steps: Receive structured objects with named keys + -- (e.g., {"run": {...}, "dependency1": {...}}) + -- ========================================== + CASE + -- -------------------- MAP STEPS -------------------- + -- Map steps process arrays element-by-element. + -- Each task receives ONE element from the array at its task_index position. + WHEN step.step_type = 'map' THEN + -- Map steps get raw array elements without any wrapper object + CASE + -- ROOT MAP: Gets array from run input + -- Example: run input = [1, 2, 3] + -- task 0 gets: 1 + -- task 1 gets: 2 + -- task 2 gets: 3 + WHEN step.deps_count = 0 THEN + -- Root map (deps_count = 0): no dependencies, reads from run input. + -- Extract the element at task_index from the run's input array. + -- Note: If run input is not an array, this will return NULL + -- and the flow will fail (validated in start_flow). + jsonb_array_element(r.input, st.task_index) + + -- DEPENDENT MAP: Gets array from its single dependency + -- Example: dependency output = ["a", "b", "c"] + -- task 0 gets: "a" + -- task 1 gets: "b" + -- task 2 gets: "c" + ELSE + -- Has dependencies (should be exactly 1 for map steps). + -- Extract the element at task_index from the dependency's output array. + -- + -- Why the subquery with jsonb_each? + -- - The dependency outputs a raw array: [1, 2, 3] + -- - deps_outputs aggregates it into: {"dep_name": [1, 2, 3]} + -- - We need to unwrap and get just the array value + -- - Map steps have exactly 1 dependency (enforced by add_step) + -- - So jsonb_each will return exactly 1 row + -- - We extract the 'value' which is the raw array [1, 2, 3] + -- - Then get the element at task_index from that array + (SELECT jsonb_array_element(value, st.task_index) + FROM jsonb_each(dep_out.deps_output) + LIMIT 1) + END + + -- -------------------- NON-MAP STEPS -------------------- + -- Regular (non-map) steps receive dependency outputs as a structured object. + -- Root steps (no dependencies) get empty object - they access flowInput via context. + -- Dependent steps get only their dependency outputs. + ELSE + -- Non-map steps get structured input with dependency keys only + -- Example for dependent step: { + -- "step1": {"output": "from_step1"}, + -- "step2": {"output": "from_step2"} + -- } + -- Example for root step: {} + -- + -- Note: flow_input is available separately in the returned record + -- for workers to access via context.flowInput + coalesce(dep_out.deps_output, '{}'::jsonb) + END as input, + st.message_id as msg_id, + st.task_index as task_index, + -- flow_input: Original run input for worker context + -- Only included for root non-map steps to avoid data duplication. + -- Root map steps: flowInput IS the array, useless to include + -- Dependent steps: lazy load via ctx.flowInput when needed + CASE + WHEN step.step_type != 'map' AND step.deps_count = 0 + THEN r.input + ELSE NULL + END as flow_input + from tasks st + join runs r on st.run_id = r.run_id + join pgflow.steps step on + step.flow_slug = st.flow_slug and + step.step_slug = st.step_slug + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug + cross join _vr + where _vr.visibility_updates >= 0 +$$; +-- Drop "start_tasks" function +DROP FUNCTION "pgflow"."start_tasks" (text, bigint[], uuid); diff --git a/pkgs/core/supabase/migrations/atlas.sum b/pkgs/core/supabase/migrations/atlas.sum index 34ed859b5..5e4082199 100644 --- a/pkgs/core/supabase/migrations/atlas.sum +++ b/pkgs/core/supabase/migrations/atlas.sum @@ -1,4 +1,4 @@ -h1:9ItIcsLHe6xhjbaHTXXGIUkY4L65yjgul6QLoVfawo8= +h1:FYuGvhoGN8MbE2ZNnATvcOnjXsppuMtlQyKP1lJ9acE= 20250429164909_pgflow_initial.sql h1:I3n/tQIg5Q5nLg7RDoU3BzqHvFVjmumQxVNbXTPG15s= 20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql h1:wTuXuwMxVniCr3ONCpodpVWJcHktoQZIbqMZ3sUHKMY= 20250609105135_pgflow_add_start_tasks_and_started_status.sql h1:ggGanW4Wyt8Kv6TWjnZ00/qVb3sm+/eFVDjGfT8qyPg= @@ -22,3 +22,4 @@ h1:9ItIcsLHe6xhjbaHTXXGIUkY4L65yjgul6QLoVfawo8= 20260607175525_pgflow_worker_start_mode.sql h1:PFAfoGaHe5stKF7YAFg6AqBxmRisqDvV60vVpnnVdBE= 20260904095427_pgflow_task_lifecycle_hardening.sql h1:27b0BfBcQxeu5XSqVtQYvDCRzTsvLqzS/5hx14/2VyM= 20260907082520_pgflow_remove_legacy_flow_compilation.sql h1:LNFDz+ZZlWb19FmWNPK57eiD+FXySMbStVij8MTSvDw= +20260913093141_pgflow_persist_queue_identity.sql h1:awtPRUJe3PS+vX85+GCjkHtv+mZwmUzawSTcrNsTXjU= diff --git a/pkgs/core/supabase/seed.sql b/pkgs/core/supabase/seed.sql index 6b3f1ecad..9e4ca9fad 100644 --- a/pkgs/core/supabase/seed.sql +++ b/pkgs/core/supabase/seed.sql @@ -111,12 +111,15 @@ as $$ ids AS ( SELECT array_agg(msg_id) AS msg_ids FROM msgs ) - -- 4. start the tasks and return the resulting rows + -- 4. start the tasks and return the resulting rows. The claim receives + -- the queue this helper read from (canonical lower(flow_slug), matching + -- the tasks' stored queue snapshots #650). SELECT * FROM pgflow.start_tasks( flow_slug, (SELECT msg_ids FROM ids), - (SELECT wid FROM w) + (SELECT wid FROM w), + lower(flow_slug) ); $$; diff --git a/pkgs/core/supabase/tests/_cascade_force_skip_steps/archives_task_messages_for_skipped_steps.test.sql b/pkgs/core/supabase/tests/_cascade_force_skip_steps/archives_task_messages_for_skipped_steps.test.sql index 116333826..e93f9d261 100644 --- a/pkgs/core/supabase/tests/_cascade_force_skip_steps/archives_task_messages_for_skipped_steps.test.sql +++ b/pkgs/core/supabase/tests/_cascade_force_skip_steps/archives_task_messages_for_skipped_steps.test.sql @@ -18,7 +18,7 @@ with tasks as ( where flow_slug = 'cascade_skip_archive' and step_slug = 'map_a' order by task_index ) -select pgflow.start_tasks('cascade_skip_archive', array[(select message_id from tasks where task_index = 0)::bigint], pgflow_tests.ensure_worker('cascade_skip_archive')); +select pgflow.start_tasks('cascade_skip_archive', array[(select message_id from tasks where task_index = 0)::bigint], pgflow_tests.ensure_worker('cascade_skip_archive'), 'cascade_skip_archive'); select ok( (select count(*) = 3 from pgflow.step_tasks diff --git a/pkgs/core/supabase/tests/_cascade_force_skip_steps/idempotent_second_call.test.sql b/pkgs/core/supabase/tests/_cascade_force_skip_steps/idempotent_second_call.test.sql index ed87e27c4..a0d714da2 100644 --- a/pkgs/core/supabase/tests/_cascade_force_skip_steps/idempotent_second_call.test.sql +++ b/pkgs/core/supabase/tests/_cascade_force_skip_steps/idempotent_second_call.test.sql @@ -18,7 +18,7 @@ with tasks as ( where flow_slug = 'idempotent_test' and step_slug = 'map_step' order by task_index ) -select pgflow.start_tasks('idempotent_test', array[(select message_id from tasks where task_index = 0)::bigint], pgflow_tests.ensure_worker('idempotent_test')); +select pgflow.start_tasks('idempotent_test', array[(select message_id from tasks where task_index = 0)::bigint], pgflow_tests.ensure_worker('idempotent_test'), 'idempotent_test'); create temporary table test_run as select run_id from pgflow.runs where flow_slug = 'idempotent_test'; diff --git a/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw b/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw index b68b823ab..3336bd23e 100644 --- a/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw +++ b/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw @@ -11,13 +11,19 @@ * * WARNING: Ensure retention_interval is longer than your longest start_delay to avoid * deleting tasks before they have a chance to execute. + * + * pgflow 0.17 note: message cleanup uses each task's stored queue snapshot + * (step_tasks.queue_name) and archive cleanup uses persisted definition routes + * (pgflow.steps.queue_name). If you installed an earlier copy of this snippet, + * replace it with this version; adapt any customized copies yourself. */ create or replace function pgflow.prune_data_older_than( retention_interval INTERVAL ) returns void language plpgsql as $$ DECLARE cutoff_timestamp TIMESTAMPTZ := now() - retention_interval; - flow_record RECORD; + task_record RECORD; + route_record RECORD; archive_table TEXT; dynamic_sql TEXT; BEGIN @@ -27,9 +33,10 @@ BEGIN -- Delete PGMQ messages from active queues BEFORE deleting step_tasks -- This prevents orphaned messages that would appear after tasks are deleted - FOR flow_record IN + -- Messages are addressed through each task's stored queue snapshot (#650) + FOR task_record IN SELECT - r.flow_slug, + st.queue_name, ARRAY_AGG(st.message_id) FILTER (WHERE st.message_id IS NOT NULL) as message_ids FROM pgflow.runs r JOIN pgflow.step_tasks st ON st.run_id = r.run_id @@ -37,11 +44,15 @@ BEGIN (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) ) - GROUP BY r.flow_slug + GROUP BY st.queue_name LOOP - -- Delete messages in batch (pgmq.delete ignores non-existent messages) - IF flow_record.message_ids IS NOT NULL AND array_length(flow_record.message_ids, 1) > 0 THEN - PERFORM pgmq.delete(flow_record.flow_slug, flow_record.message_ids); + -- Delete messages in batch (pgmq.delete ignores non-existent messages; + -- PGMQ message operations normalize names themselves) + IF task_record.message_ids IS NOT NULL AND array_length(task_record.message_ids, 1) > 0 THEN + PERFORM pgmq.delete( + task_record.queue_name, + task_record.message_ids + ); END IF; END LOOP; @@ -73,12 +84,25 @@ BEGIN (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) ); - -- Prune archived messages from PGMQ archive tables (pgmq.a_{flow_slug}) - -- For each flow, delete old archived messages - FOR flow_record IN SELECT DISTINCT flow_slug FROM pgflow.flows + -- Prune archived messages from PGMQ archive tables. + -- Walk the persisted definition routes (plus each flow's default queue) so + -- queues that keep an original mixed-case spelling are found (#650). + -- pgmq.format_table_name() lowercases its input itself, so the canonical + -- route addresses the physical archive table directly: no queue listing + -- is consulted and no ambiguous-match failure can abort pruning. + FOR route_record IN + SELECT DISTINCT queue_name FROM ( + SELECT queue_name FROM pgflow.steps + UNION + SELECT lower(flow_slug) FROM pgflow.flows + ) routes LOOP - -- Build the archive table name - archive_table := pgmq.format_table_name(flow_record.flow_slug, 'a'); + -- Build the archive table name from the canonical route + -- (format_table_name lowercases the input itself) + archive_table := pgmq.format_table_name( + route_record.queue_name, + 'a' + ); -- Check if the archive table exists IF EXISTS ( @@ -95,4 +119,4 @@ BEGIN END IF; END LOOP; END -$$; \ No newline at end of file +$$; diff --git a/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql b/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql index 236f6cf7b..199a9ef5a 100644 --- a/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql +++ b/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql @@ -19,8 +19,8 @@ select is( -- Test: Cannot have two steps with the same index in the same flow select throws_ok( $$ - INSERT INTO pgflow.steps (flow_slug, step_slug, step_index) - VALUES ('test_flow', 'duplicate_index_step', 0) + INSERT INTO pgflow.steps (flow_slug, step_slug, queue_name, step_index) + VALUES ('test_flow', 'duplicate_index_step', 'test_flow', 0) $$, '23505', -- Unique violation error code 'duplicate key value violates unique constraint "steps_flow_slug_step_index_key"', diff --git a/pkgs/core/supabase/tests/cascade_complete_taskless_steps/no_cascade_on_failed_run.test.sql b/pkgs/core/supabase/tests/cascade_complete_taskless_steps/no_cascade_on_failed_run.test.sql index 683f0f0d6..ed962e62a 100644 --- a/pkgs/core/supabase/tests/cascade_complete_taskless_steps/no_cascade_on_failed_run.test.sql +++ b/pkgs/core/supabase/tests/cascade_complete_taskless_steps/no_cascade_on_failed_run.test.sql @@ -22,7 +22,7 @@ where run_id = :'run_id' and step_slug = 'step1' limit 1 \gset select pgflow_tests.ensure_worker('test_flow'); -- Start and fail step1 which will fail the entire run -select pgflow.start_tasks('test_flow', ARRAY[:msg1]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid); +select pgflow.start_tasks('test_flow', ARRAY[:msg1]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid, 'test_flow'); select pgflow.fail_task(:'run_id', 'step1', 0, 'Simulated failure'); -- Call cascade_complete_taskless_steps directly on the failed run diff --git a/pkgs/core/supabase/tests/complete_task/late_complete_after_skip_does_not_mutate_step_or_run.test.sql b/pkgs/core/supabase/tests/complete_task/late_complete_after_skip_does_not_mutate_step_or_run.test.sql index c562fa9fc..1411bf8b6 100644 --- a/pkgs/core/supabase/tests/complete_task/late_complete_after_skip_does_not_mutate_step_or_run.test.sql +++ b/pkgs/core/supabase/tests/complete_task/late_complete_after_skip_does_not_mutate_step_or_run.test.sql @@ -28,12 +28,12 @@ select pgflow_tests.ensure_worker('late_complete_test') as test_worker_id \gset select message_id as msg_0 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_a' and task_index = 0 \gset -select pgflow.start_tasks('late_complete_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('late_complete_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid, 'late_complete_test'); select message_id as msg_1 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_a' and task_index = 1 \gset -select pgflow.start_tasks('late_complete_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('late_complete_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid, 'late_complete_test'); -- Fail map_a[0] to trigger skip (max_attempts=0, when_exhausted='skip') -- This makes the step transition to 'skipped' diff --git a/pkgs/core/supabase/tests/complete_task/no_mutations_on_failed_run.test.sql b/pkgs/core/supabase/tests/complete_task/no_mutations_on_failed_run.test.sql index 19456ca5a..2c5e68738 100644 --- a/pkgs/core/supabase/tests/complete_task/no_mutations_on_failed_run.test.sql +++ b/pkgs/core/supabase/tests/complete_task/no_mutations_on_failed_run.test.sql @@ -26,7 +26,7 @@ where run_id = :'run_id' and step_slug = 'step2' limit 1 \gset select pgflow_tests.ensure_worker('test_flow'); -- Start both tasks (simulating workers picking them up) -select pgflow.start_tasks('test_flow', ARRAY[:msg1, :msg2]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid); +select pgflow.start_tasks('test_flow', ARRAY[:msg1, :msg2]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid, 'test_flow'); -- Fail step2 which will fail the entire run (max_attempts=1) select pgflow.fail_task(:'run_id', 'step2', 0, 'Simulated failure'); diff --git a/pkgs/core/supabase/tests/condition_evaluation/skipped_deps_excluded_from_input.test.sql b/pkgs/core/supabase/tests/condition_evaluation/skipped_deps_excluded_from_input.test.sql index 52b342d4c..f89109157 100644 --- a/pkgs/core/supabase/tests/condition_evaluation/skipped_deps_excluded_from_input.test.sql +++ b/pkgs/core/supabase/tests/condition_evaluation/skipped_deps_excluded_from_input.test.sql @@ -103,7 +103,7 @@ start_result as ( 'skip_diamond', (select ids from msg_ids), pgflow_tests.ensure_worker('skip_diamond') - ) st + , 'skip_diamond') st ) -- Store the input for later testing select input, step_slug, run_id into temporary step_c_inputs diff --git a/pkgs/core/supabase/tests/create_flow/queue_name_collisions.test.sql b/pkgs/core/supabase/tests/create_flow/queue_name_collisions.test.sql new file mode 100644 index 000000000..80380120e --- /dev/null +++ b/pkgs/core/supabase/tests/create_flow/queue_name_collisions.test.sql @@ -0,0 +1,108 @@ +-- Provisioning rules (#650): reject a listed queue that already uses a flow's +-- normalized default queue name; existing matching flows reuse their queue; +-- normalized flow collisions are rejected atomically, including direct SQL. +begin; +select plan(13); + +select pgflow_tests.reset_db(); + +-- External queue with the same normalized name blocks a new flow +select pgmq.create('TakenName'); + +select throws_ok( + $$select pgflow.create_flow('TakenName')$$, + 'cannot create flow "TakenName": queue "takenname" is already in use by another owner', + 'flow with an externally taken normalized queue name is rejected' +); + +select throws_ok( + $$select pgflow.create_flow('takenname')$$, + 'cannot create flow "takenname": queue "takenname" is already in use by another owner', + 'normalized collision is rejected regardless of case' +); + +select is( + (select count(*) from pgflow.flows where lower(flow_slug) = 'takenname'), + 0::bigint, + 'rejected flow creation left no flow row' +); + +-- Normalized flow collision is rejected atomically, including direct SQL +insert into pgflow.flows (flow_slug) values ('MyFlow'); + +select throws_ok( + $$insert into pgflow.flows (flow_slug) values ('myflow')$$, + 'duplicate key value violates unique constraint "idx_flows_normalized_slug"', + 'direct SQL cannot create a second flow with the same normalized name' +); + +select throws_ok( + $$select pgflow.create_flow('MYFLOW')$$, + 'duplicate key value violates unique constraint "idx_flows_normalized_slug"', + 'create_flow cannot register a case variant of an existing flow' +); + +-- An existing matching flow reuses its queue (idempotent re-registration) +select pgmq.create('MyFlow'); + +select lives_ok( + $$select pgflow.create_flow('MyFlow')$$, + 'existing flow re-registers and reuses its listed queue' +); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'myflow'), + 1::bigint, + 'reuse does not create a second metadata entry' +); + +-- Empty flows still get their default queue +select pgflow.create_flow('emptyflow'); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'emptyflow'), + 1::bigint, + 'create_flow provisions the default queue for an empty flow' +); + +-- Queue-name validity on the persisted columns +select throws_ok( + $$select pgflow.create_flow(repeat('a', 48))$$, + 'queue name is too long, maximum length is 48 characters', + 'flow slugs beyond the queue-name limit are rejected by PGMQ validation' +); + +select throws_ok( + $$ + insert into pgflow.steps (flow_slug, step_slug, queue_name) + values ('emptyflow', 's', 'NotLowercase') + $$, + 'new row for relation "steps" violates check constraint "queue_name_is_valid"', + 'steps reject a non-canonical queue name' +); + +select throws_ok( + $$ + insert into pgflow.steps (flow_slug, step_slug, queue_name) + values ('emptyflow', 's', repeat('q', 48)) + $$, + 'new row for relation "steps" violates check constraint "queue_name_is_valid"', + 'steps reject a queue name beyond 47 characters' +); + +select is( + (select pgflow.is_valid_queue_name('ok')), + true, + 'is_valid_queue_name accepts canonical lowercase names' +); + +select ok( + not pgflow.is_valid_queue_name('Mixed') + and pgflow.is_valid_queue_name(repeat('q', 47)) + and not pgflow.is_valid_queue_name(repeat('q', 48)) + and not pgflow.is_valid_queue_name(''), + 'is_valid_queue_name enforces lowercase and the 47-character limit' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/delete_flow_and_data/drops_queues_by_routes.test.sql b/pkgs/core/supabase/tests/delete_flow_and_data/drops_queues_by_routes.test.sql new file mode 100644 index 000000000..70759afc2 --- /dev/null +++ b/pkgs/core/supabase/tests/delete_flow_and_data/drops_queues_by_routes.test.sql @@ -0,0 +1,70 @@ +-- Deletion (#650): a flow's queues are dropped through the persisted +-- definition routes, resolved to their original listed spelling, including +-- the default queue of an empty flow. Deletion stays transactional. +begin; +select plan(7); + +select pgflow_tests.reset_db(); + +-- Flow with a legacy mixed-case physical queue +select pgflow.create_flow('DropCase', null, null, 5); +select pgflow.add_step('DropCase', 'a'); +select pgmq.drop_queue('dropcase'); +select pgmq.create('DropCase'); +select pgflow.start_flow('DropCase', '"x"'::jsonb); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'DropCase'), + 1::bigint, + 'physical mixed-case queue in place before deletion' +); + +select pgflow.delete_flow_and_data('DropCase'); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'dropcase'), + 0::bigint, + 'deletion drops the queue through its persisted route and original spelling' +); + +select is( + (select count(*) from pgflow.flows where flow_slug = 'DropCase'), + 0::bigint, + 'flow definition deleted' +); + +-- Empty flow: default queue dropped without persisted routes +select pgflow.create_flow('DropEmpty'); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'dropempty'), + 1::bigint, + 'empty flow has its default queue' +); + +select pgflow.delete_flow_and_data('DropEmpty'); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'dropempty'), + 0::bigint, + 'deletion drops the default queue of an empty flow' +); + +-- Deleting a flow whose queue is already gone still removes the data +select pgflow.create_flow('DropGone'); +select pgflow.add_step('DropGone', 'a'); +select pgmq.drop_queue('dropgone'); + +select lives_ok( + $$select pgflow.delete_flow_and_data('DropGone')$$, + 'deletion tolerates an already-dropped queue' +); + +select is( + (select count(*) from pgflow.steps where flow_slug = 'DropGone'), + 0::bigint, + 'definition removed when the queue was missing' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/delete_flow_and_data/guards_missing_flow_queue.test.sql b/pkgs/core/supabase/tests/delete_flow_and_data/guards_missing_flow_queue.test.sql new file mode 100644 index 000000000..6449e2b8d --- /dev/null +++ b/pkgs/core/supabase/tests/delete_flow_and_data/guards_missing_flow_queue.test.sql @@ -0,0 +1,68 @@ +-- Deletion (#650): destructive queue work must be authorized by an exact +-- pgflow.flows row. A nonexistent flow slug must not drop a same-named +-- queue owned by someone else, and a wrong-case slug must not drop another +-- flow's queue or data (review regressions). +begin; +select plan(8); + +select pgflow_tests.reset_db(); + +-- Nonexistent flow, existing same-named queue owned outside pgflow +select pgmq.create('guard650_external'); + +select lives_ok( + $$select pgflow.delete_flow_and_data('guard650_external')$$, + 'deleting a nonexistent flow is a quiet no-op' +); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'guard650_external'), + 1::bigint, + 'nonexistent flow does not drop the same-named external queue' +); + +select is( + (select count(*) from pgflow.flows where flow_slug = 'guard650_external'), + 0::bigint, + 'no flow row appeared' +); + +-- Existing flow reached through a wrong-case slug: everything survives +select pgflow.create_flow('GuardCase'); +select pgflow.add_step('GuardCase', 'a'); +select pgflow.start_flow('GuardCase', '"x"'::jsonb); + +select lives_ok( + $$select pgflow.delete_flow_and_data('guardcase')$$, + 'wrong-case slug is a quiet no-op' +); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'guardcase'), + 1::bigint, + 'wrong-case slug does not drop the flow queue' +); + +select is( + (select count(*) from pgflow.flows where flow_slug = 'GuardCase'), + 1::bigint, + 'wrong-case slug leaves the flow definition intact' +); + +select is( + (select count(*) from pgflow.steps where flow_slug = 'GuardCase'), + 1::bigint, + 'wrong-case slug leaves the flow data intact' +); + +-- The exact slug still deletes flow, data, and queue +select pgflow.delete_flow_and_data('GuardCase'); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'guardcase'), + 0::bigint, + 'exact slug still drops the flow queue' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/delete_flow_and_data/recreate_resolves_fresh_after_drop.test.sql b/pkgs/core/supabase/tests/delete_flow_and_data/recreate_resolves_fresh_after_drop.test.sql new file mode 100644 index 000000000..55a03df73 --- /dev/null +++ b/pkgs/core/supabase/tests/delete_flow_and_data/recreate_resolves_fresh_after_drop.test.sql @@ -0,0 +1,66 @@ +-- Deletion (#650 review correction): drop_queue addresses physical objects, +-- so every deletion resolves the listed spelling fresh through +-- pgmq.list_queues(). After drop -> recreate, a second deletion in the same +-- session must drop the recreated queue: no caching may authorize (or skip) +-- destructive work. +begin; +select plan(6); + +select pgflow_tests.reset_db(); + +-- Legacy flow: canonical name stored, physical queue mixed-case +select pgflow.create_flow('AmbCase', null, null, 5); +select pgflow.add_step('AmbCase', 'a', max_attempts => 2); +select pgmq.drop_queue('ambcase'); +select pgmq.create('AmbCase'); + +-- Dispatch a message to the mixed-case queue through the canonical name +select pgflow.start_flow('AmbCase', '"x"'::jsonb); + +select is( + (select count(*) from pgmq.q_AmbCase), + 1::bigint, + 'dispatch used the listed mixed-case spelling' +); + +-- First deletion drops the listed spelling and the flow data +select pgflow.delete_flow_and_data('AmbCase'); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'ambcase'), + 0::bigint, + 'first deletion drops the mixed-case queue' +); + +select is( + (select count(*) from pgflow.flows where lower(flow_slug) = 'ambcase'), + 0::bigint, + 'first deletion removes the flow' +); + +-- Recreate the flow: the queue now exists under the canonical spelling +select pgflow.create_flow('AmbCase', null, null, 5); +select pgflow.add_step('AmbCase', 'a'); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'ambcase'), + 1::bigint, + 'recreation provisions the canonical lowercase queue' +); + +-- Second deletion in the same session: resolves the listed spelling fresh +select pgflow.delete_flow_and_data('AmbCase'); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'ambcase'), + 0::bigint, + 'second deletion drops the recreated queue through a fresh resolution' +); +select is( + (select count(*) from pgflow.flows where lower(flow_slug) = 'ambcase'), + 0::bigint, + 'second deletion removes the recreated flow' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/delete_flow_and_data/rejects_ambiguous_queue_match.test.sql b/pkgs/core/supabase/tests/delete_flow_and_data/rejects_ambiguous_queue_match.test.sql new file mode 100644 index 000000000..bb617f96b --- /dev/null +++ b/pkgs/core/supabase/tests/delete_flow_and_data/rejects_ambiguous_queue_match.test.sql @@ -0,0 +1,59 @@ +-- Deletion must resolve every listed spelling of a normalized queue name +-- before destructive work: an ambiguous case-insensitive match is rejected +-- and the flow is left fully intact (#650 review regression). +begin; +select plan(6); + +select pgflow_tests.reset_db(); + +-- Flow whose physical queue exists under two spellings of one name +-- (external damage), one of them the exact canonical spelling +select pgflow.create_flow('ambcase', null, null, 5); +select pgflow.add_step('ambcase', 'a'); +select pgflow.start_flow('ambcase', '"x"'::jsonb); +select pgmq.drop_queue('ambcase'); +select pgmq.create('ambcase'); +select pgmq.create('AmbCase'); + +select throws_ok( + $$select pgflow.delete_flow_and_data('ambcase')$$, + 'queue name "ambcase" is ambiguous: it matches listed queues {ambcase,AmbCase}', + 'deletion rejects the ambiguous match before destructive work' +); + +select is( + (select count(*) from pgflow.flows where flow_slug = 'ambcase'), + 1::bigint, + 'flow definition intact after rejection' +); + +select is( + (select count(*) from pgflow.step_tasks where flow_slug = 'ambcase'), + 1::bigint, + 'runtime data intact after rejection' +); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'ambcase'), + 2::bigint, + 'both queue spellings still listed after rejection' +); + +-- Fresh resolution rejects the ambiguous pair even when one spelling is the +-- exact canonical name: destructive resolution (drop_queue, archive-table +-- walks) goes through this function, so the exact match must not +-- short-circuit the ambiguity rejection +select throws_ok( + $$select pgflow._listed_queue_name('ambcase')$$, + 'queue name "ambcase" is ambiguous: it matches listed queues {ambcase,AmbCase}', + 'fresh resolution rejects the ambiguous match before any message operation' +); + +select is( + (select status from pgflow.step_tasks where flow_slug = 'ambcase'), + 'queued', + 'task untouched after the rejected archive' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/fail_task/archive_sibling_map_tasks.test.sql b/pkgs/core/supabase/tests/fail_task/archive_sibling_map_tasks.test.sql index 6876c82f2..003b62d2f 100644 --- a/pkgs/core/supabase/tests/fail_task/archive_sibling_map_tasks.test.sql +++ b/pkgs/core/supabase/tests/fail_task/archive_sibling_map_tasks.test.sql @@ -30,12 +30,12 @@ select pgflow_tests.ensure_worker('test_map_fail') as test_worker_id \gset -- Start task 0 (will be the failing task) select message_id as msg_0 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_step' and task_index = 0 \gset -select pgflow.start_tasks('test_map_fail', array[:'msg_0'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('test_map_fail', array[:'msg_0'::bigint], :'test_worker_id'::uuid, 'test_map_fail'); -- Start task 1 (unfinished started sibling) select message_id as msg_1 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_step' and task_index = 1 \gset -select pgflow.start_tasks('test_map_fail', array[:'msg_1'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('test_map_fail', array[:'msg_1'::bigint], :'test_worker_id'::uuid, 'test_map_fail'); -- Task 2 stays queued diff --git a/pkgs/core/supabase/tests/fail_task_when_exhausted/cancelled_task_late_callbacks_are_idempotent.test.sql b/pkgs/core/supabase/tests/fail_task_when_exhausted/cancelled_task_late_callbacks_are_idempotent.test.sql index e80bed616..63a762d3d 100644 --- a/pkgs/core/supabase/tests/fail_task_when_exhausted/cancelled_task_late_callbacks_are_idempotent.test.sql +++ b/pkgs/core/supabase/tests/fail_task_when_exhausted/cancelled_task_late_callbacks_are_idempotent.test.sql @@ -20,11 +20,11 @@ select pgflow_tests.ensure_worker('late_callback_test') as test_worker_id \gset select message_id as msg_0 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_step' and task_index = 0 \gset -select pgflow.start_tasks('late_callback_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('late_callback_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid, 'late_callback_test'); select message_id as msg_1 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_step' and task_index = 1 \gset -select pgflow.start_tasks('late_callback_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('late_callback_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid, 'late_callback_test'); -- Fail task 0: run fails, task 1 becomes cancelled select pgflow.fail_task(:'test_run_id'::uuid, 'map_step', 0, 'Task 0 failed'); diff --git a/pkgs/core/supabase/tests/fail_task_when_exhausted/late_callbacks_post_lock_race.test.sql b/pkgs/core/supabase/tests/fail_task_when_exhausted/late_callbacks_post_lock_race.test.sql index 1676b9c21..6ec514672 100644 --- a/pkgs/core/supabase/tests/fail_task_when_exhausted/late_callbacks_post_lock_race.test.sql +++ b/pkgs/core/supabase/tests/fail_task_when_exhausted/late_callbacks_post_lock_race.test.sql @@ -72,7 +72,7 @@ select msg_boom from dblink('ctrl', $$select message_id from pgflow.step_tasks w select dblink_exec( 'ctrl', format( - $$do $do$ begin perform pgflow.start_tasks('race_flow', ARRAY[%s, %s]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid); end $do$;$$, + $$do $do$ begin perform pgflow.start_tasks('race_flow', ARRAY[%s, %s]::bigint[], '11111111-1111-1111-1111-111111111111'::uuid, 'race_flow'); end $do$;$$, :'msg_single', :'msg_boom' ) ); diff --git a/pkgs/core/supabase/tests/fail_task_when_exhausted/late_fail_after_skip_does_not_double_decrement_remaining_steps.test.sql b/pkgs/core/supabase/tests/fail_task_when_exhausted/late_fail_after_skip_does_not_double_decrement_remaining_steps.test.sql index a85ed4ba7..94a91d030 100644 --- a/pkgs/core/supabase/tests/fail_task_when_exhausted/late_fail_after_skip_does_not_double_decrement_remaining_steps.test.sql +++ b/pkgs/core/supabase/tests/fail_task_when_exhausted/late_fail_after_skip_does_not_double_decrement_remaining_steps.test.sql @@ -19,7 +19,7 @@ select pgflow.start_tasks( 'double_decrement_test', (select array_agg(message_id) from pgflow.step_tasks st join pgflow.runs r on st.run_id = r.run_id where r.flow_slug = 'double_decrement_test' and st.step_slug = 'map_a'), '00000000-0000-0000-0000-000000000001'::uuid -); +, 'double_decrement_test'); select is(count(*), 2::bigint, 'Both map_a tasks are started') from pgflow.step_tasks st join pgflow.runs r on st.run_id = r.run_id where r.flow_slug = 'double_decrement_test' and st.step_slug = 'map_a' and st.status = 'started'; -- Start 'other' task (to keep run alive) @@ -27,7 +27,7 @@ select pgflow.start_tasks( 'double_decrement_test', (select array_agg(message_id) from pgflow.step_tasks st join pgflow.runs r on st.run_id = r.run_id where r.flow_slug = 'double_decrement_test' and st.step_slug = 'other'), '00000000-0000-0000-0000-000000000001'::uuid -); +, 'double_decrement_test'); -- Capture remaining_steps BEFORE first fail create temp table baseline as diff --git a/pkgs/core/supabase/tests/fail_task_when_exhausted/skip_archives_sibling_messages.test.sql b/pkgs/core/supabase/tests/fail_task_when_exhausted/skip_archives_sibling_messages.test.sql index a7860bb8d..883171105 100644 --- a/pkgs/core/supabase/tests/fail_task_when_exhausted/skip_archives_sibling_messages.test.sql +++ b/pkgs/core/supabase/tests/fail_task_when_exhausted/skip_archives_sibling_messages.test.sql @@ -33,13 +33,13 @@ select pgflow_tests.ensure_worker('skip_archive_test') as test_worker_id \gset select message_id as msg_0 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_a' and task_index = 0 \gset -select pgflow.start_tasks('skip_archive_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('skip_archive_test', array[:'msg_0'::bigint], :'test_worker_id'::uuid, 'skip_archive_test'); -- Get message_id for task 1 select message_id as msg_1 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'map_a' and task_index = 1 \gset -select pgflow.start_tasks('skip_archive_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid); +select pgflow.start_tasks('skip_archive_test', array[:'msg_1'::bigint], :'test_worker_id'::uuid, 'skip_archive_test'); -- Verify: 2 started, 1 queued select is( diff --git a/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql b/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql index 979c5bfaa..262a1da58 100644 --- a/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql +++ b/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql @@ -58,11 +58,12 @@ set failed_at = NULL where flow_slug = 'status_test_flow' and step_slug = 'step2'; -insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, status, queued_at, started_at) +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, status, queued_at, started_at) select 'status_test_flow', run_id, 'step2', + 'status_test_flow', 0, 'started', now() - interval '36 days', diff --git a/pkgs/core/supabase/tests/maintenance/prune_uses_task_snapshots.test.sql b/pkgs/core/supabase/tests/maintenance/prune_uses_task_snapshots.test.sql new file mode 100644 index 000000000..a299039f0 --- /dev/null +++ b/pkgs/core/supabase/tests/maintenance/prune_uses_task_snapshots.test.sql @@ -0,0 +1,106 @@ +-- Pruning (#650): the optional helper cleans PGMQ messages through task +-- snapshots and archives through persisted definition routes, so a queue +-- that keeps an original mixed-case spelling is still cleaned. The helper +-- never consults the pgmq queue listing: pgmq.list_queues() is shadowed +-- with a raising function before the prune runs, and setup (which does use +-- the listing for provisioning collision checks) runs before the trap. +begin; +select plan(7); + +select pgflow_tests.reset_db(); + +-- Load the prune_data_older_than function +\i _shared/prune_data_older_than.sql.raw + +-- Flow with a legacy mixed-case physical queue, completed and aged out +select pgflow.create_flow('PruneCase', null, null, 5); +select pgflow.add_step('PruneCase', 'a'); +select pgmq.drop_queue('prunecase'); +select pgmq.create('PruneCase'); +select pgflow.start_flow('PruneCase', '"x"'::jsonb); + +select pgflow_tests.ensure_worker('PruneCase'); +select pgflow_tests.read_and_start('PruneCase'); +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'PruneCase'), + 'a', + 0, + null +); + +select is( + (select count(*) from pgmq.a_PruneCase), + 1::bigint, + 'message archived on the physical mixed-case archive table' +); + +-- Age the run past retention +select pgflow_tests.set_completed_flow_timestamps('PruneCase', 40); +update pgmq.a_PruneCase set archived_at = now() - interval '40 days'; + +-- Active-queue message cleanup through task snapshots: a queued message of +-- an old failed run is deleted from the physical queue +select pgflow.create_flow('PruneAct', null, null, 5); +select pgflow.add_step('PruneAct', 'a'); +select pgmq.drop_queue('pruneact'); +select pgmq.create('PruneAct'); +select pgflow.start_flow('PruneAct', '"x"'::jsonb); +-- leave the task queued (message invisible via a long read window) +select pgmq.read_with_poll('PruneAct', 3600, 5, 1, 10); + +select is( + (select count(*) from pgmq.q_PruneAct), + 1::bigint, + 'queued message present on the physical queue before pruning' +); + +update pgflow.runs +set started_at = now() - interval '41 days', + failed_at = now() - interval '40 days', + status = 'failed' +where flow_slug = 'PruneAct'; + +-- Arm the trap: the archive walk must derive table names without the listing +create or replace function pgmq.list_queues() +returns setof pgmq.queue_record +language plpgsql +as $$ +begin + raise exception 'pruning called pgmq.list_queues()'; +end; +$$; + +select lives_ok( + $$ select pgflow.prune_data_older_than(make_interval(days => 30)) $$, + 'prune completed without the queue listing' +); + +select is( + (select count(*) from pgflow.runs where flow_slug = 'PruneCase'), + 0::bigint, + 'old run pruned' +); + +select is( + (select count(*) from pgflow.step_tasks where flow_slug = 'PruneCase'), + 0::bigint, + 'old tasks pruned' +); + +-- Archive cleanup walks persisted routes and pgmq.format_table_name() +-- lowercases the route itself, so the mixed-case archive table is found +-- without resolving the listed spelling +select is( + (select count(*) from pgmq.a_PruneCase), + 0::bigint, + 'archive cleaned through the persisted route without the queue listing' +); + +select is( + (select count(*) from pgmq.q_PruneAct), + 0::bigint, + 'active-queue message deleted through the task snapshot' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/message_id_identity.test.sql b/pkgs/core/supabase/tests/queue_identity/message_id_identity.test.sql new file mode 100644 index 000000000..2d9a67d94 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/message_id_identity.test.sql @@ -0,0 +1,86 @@ +-- Queue identity snapshots (#650): the stored (queue_name, message_id) +-- identity admits NULL message ids and rejects duplicate non-null pairs, +-- including message ids beyond the JavaScript safe integer range. +begin; +select plan(6); + +select pgflow_tests.reset_db(); + +select pgflow.create_flow('identq', null, null, 5); +select pgflow.add_step('identq', 'a'); +select pgflow.start_flow('identq', '"x"'::jsonb); + +-- NULL message_id rows are allowed and not part of the identity +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) +values + ('identq', (select run_id from pgflow.runs where flow_slug = 'identq'), 'a', 'identq', 97, null), + ('identq', (select run_id from pgflow.runs where flow_slug = 'identq'), 'a', 'identq', 98, null); + +select is( + (select count(*) from pgflow.step_tasks where step_slug = 'a' and message_id is null), + 2::bigint, + 'multiple NULL message_id task rows are allowed' +); + +-- Duplicate non-null (queue_name, message_id) is rejected +select throws_ok( + $$ + insert into pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) + select flow_slug, run_id, 'a', queue_name, 99, message_id + from pgflow.step_tasks + where step_slug = 'a' and message_id is not null + $$, + 'duplicate key value violates unique constraint "idx_step_tasks_queue_message"', + 'duplicate (queue_name, message_id) pair is rejected' +); + +-- Message ids beyond the safe integer range are stored and compared exactly +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) +values + ('identq', (select run_id from pgflow.runs where flow_slug = 'identq'), 'a', 'identq', 100, 9007199254740993), + ('identq', (select run_id from pgflow.runs where flow_slug = 'identq'), 'a', 'identq', 101, 9223372036854775807); + +select is( + (select count(*) from pgflow.step_tasks where message_id = 9007199254740993::bigint), + 1::bigint, + 'message id 2^53+1 is stored exactly' +); + +select is( + (select count(*) from pgflow.step_tasks where message_id = 9007199254740992::bigint), + 0::bigint, + 'exact bigint comparison does not round to the nearest double' +); + +select throws_ok( + $$ + insert into pgflow.step_tasks (flow_slug, run_id, step_slug, queue_name, task_index, message_id) + values ( + 'identq', + (select run_id from pgflow.runs where flow_slug = 'identq'), + 'a', + 'identq', + 102, + 9007199254740993 + ) + $$, + 'duplicate key value violates unique constraint "idx_step_tasks_queue_message"', + 'duplicate pair rejected with ids beyond the safe integer range' +); + +-- The same message id under a different queue is a different identity +select pgflow.create_flow('identq2', null, null, 5); +select pgflow.add_step('identq2', 'a'); +select pgflow.start_flow('identq2', '"x"'::jsonb); + +select lives_ok( + $$ + update pgflow.step_tasks + set message_id = (select message_id from pgflow.step_tasks where flow_slug = 'identq' and task_index = 0) + where flow_slug = 'identq2' and task_index = 0 + $$, + 'the same message id in another queue does not collide' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/message_paths_skip_queue_listing.test.sql b/pkgs/core/supabase/tests/queue_identity/message_paths_skip_queue_listing.test.sql new file mode 100644 index 000000000..4178d7e65 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/message_paths_skip_queue_listing.test.sql @@ -0,0 +1,93 @@ +-- Message hot paths (#650 review correction): dispatch, claim, visibility, +-- completion, retry, and stalled recovery address queues by the stored +-- canonical name and never consult the pgmq queue listing. PGMQ's public +-- message API normalizes names itself, so no hot-path resolution is needed. +-- pgmq.list_queues() is shadowed with a function that raises: any hot-path +-- call to the listing fails the test. Provisioning and deletion (which do +-- need the listing) run before the trap is armed. +begin; +select plan(6); + +select pgflow_tests.reset_db(); +select pgflow_tests.setup_flow('sequential'); +select pgflow.start_flow('sequential', '"x"'::jsonb); + +-- Arm the trap: any message-path call to the queue listing raises +create or replace function pgmq.list_queues() +returns setof pgmq.queue_record +language plpgsql +as $$ +begin + raise exception 'message hot path called pgmq.list_queues()'; +end; +$$; + +-- Claim: read_with_poll + start_tasks visibility extension (set_vt_batch) +select pgflow_tests.read_and_start('sequential'); + +select is( + (select status from pgflow.step_tasks where step_slug = 'first'), + 'started', + 'claim and visibility extension ran without the queue listing' +); + +-- Retry: fail_task with retries left applies the delay through pgmq.set_vt +select pgflow.fail_task( + (select run_id from pgflow.runs where flow_slug = 'sequential'), + 'first', + 0, + 'boom' +); + +select is( + (select status from pgflow.step_tasks where step_slug = 'first'), + 'queued', + 'retry visibility delay ran without the queue listing' +); + +-- Recovery: requeue_stalled_tasks resets visibility through set_vt_batch +select pgflow_tests.reset_message_visibility('sequential'); +select pgflow_tests.read_and_start('sequential'); + +update pgflow.step_tasks +set queued_at = now() - interval '120 seconds', + started_at = now() - interval '119 seconds' +where step_slug = 'first'; + +select is( + pgflow.requeue_stalled_tasks()::int, + 1, + 'stalled recovery requeued without the queue listing' +); + +-- Completion archives the message and dispatches the next step (send_batch) +select pgflow_tests.reset_message_visibility('sequential'); +select pgflow_tests.read_and_start('sequential'); + +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'sequential'), + 'first', + 0, + null +); + +select is( + (select status from pgflow.step_tasks where step_slug = 'first'), + 'completed', + 'completion archived the message without the queue listing' +); + +select is( + (select count(*) from pgflow.step_tasks where step_slug = 'second' and status = 'queued')::int, + 1, + 'next-step dispatch ran without the queue listing' +); + +select is( + (select count(*) from pgmq.a_sequential)::int, + 1, + 'message reached the physical archive table without the queue listing' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/mixed_case_queue_lifecycle.test.sql b/pkgs/core/supabase/tests/queue_identity/mixed_case_queue_lifecycle.test.sql new file mode 100644 index 000000000..05bb76311 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/mixed_case_queue_lifecycle.test.sql @@ -0,0 +1,155 @@ +-- Queues created by older pgflow releases keep their original mixed-case +-- spelling in pgmq (#650). PGMQ's public message API normalizes names, so +-- every pgflow message operation addresses the physical queue by the stored +-- canonical name directly: dispatch, claim, visibility, completion, retry, +-- recovery. No queue listing is consulted and no second metadata entry is +-- created. +begin; +select plan(13); + +select pgflow_tests.reset_db(); + +-- Simulate an upgraded 0.16.0 flow: canonical name stored, physical queue +-- still listed under the original spelling +select pgflow.create_flow('LegacyCase', null, null, 5); +select pgflow.add_step('LegacyCase', 'a', max_attempts => 2); +select pgmq.drop_queue('legacycase'); +select pgmq.create('LegacyCase'); + +select pgflow.start_flow('LegacyCase', '"x"'::jsonb); + +select is( + (select queue_name from pgflow.step_tasks where flow_slug = 'LegacyCase'), + 'legacycase', + 'task stores the canonical lowercase name' +); + +select is( + (select count(*) from pgmq.q_LegacyCase), + 1::bigint, + 'dispatch resolved the original spelling for the send' +); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'legacycase'), + 1::bigint, + 'no second metadata entry was created' +); + +-- The claim requires the queue's canonical identity, not the polled +-- spelling: messages are read from the physical 'LegacyCase' queue, but +-- tasks store 'legacycase', and the match is exact. Passing the polled +-- mixed-case spelling claims nothing (#650). +select pgflow_tests.ensure_worker('legacycase'); +select pgflow_tests.ensure_worker('LegacyCase'); + +select array_agg(msg_id) as ids into temporary legacy_msgs +from pgmq.read_with_poll('LegacyCase', 30, 5, 1, 50); + +select is( + (select count(*) from pgflow.start_tasks( + 'LegacyCase', + (select ids from legacy_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + 'LegacyCase' + ))::int, + 0, + 'claim with the polled mixed-case spelling matches nothing' +); + +select is( + (select count(*) from pgflow.start_tasks( + 'LegacyCase', + (select ids from legacy_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + 'legacycase' + ))::int, + 1, + 'claim with the canonical stored name claims the task' +); + +-- Visibility was extended on the physical queue table +select ok( + (select extract(epoch from (q.vt - clock_timestamp()))::int >= 5 + from pgmq.q_LegacyCase q + join pgflow.step_tasks st on st.message_id = q.msg_id), + 'claim visibility extension reached the physical queue' +); + +-- Failure with retries left: retry delay set on the physical queue +select pgflow.fail_task( + (select run_id from pgflow.runs where flow_slug = 'LegacyCase'), + 'a', + 0, + 'boom' +); + +select is( + (select status from pgflow.step_tasks where flow_slug = 'LegacyCase'), + 'queued', + 'task requeued after failure' +); + +select ok( + (select extract(epoch from (q.vt - clock_timestamp()))::int > 0 + from pgmq.q_LegacyCase q + join pgflow.step_tasks st on st.message_id = q.msg_id), + 'retry visibility delay applied on the physical queue' +); + +-- Stalled recovery requeues through the stored queue +-- Re-claim the requeued task so recovery sees a started task +select pgflow_tests.reset_message_visibility('LegacyCase'); +select pgflow_tests.read_and_start('LegacyCase'); + +update pgflow.step_tasks +set queued_at = now() - interval '120 seconds', + started_at = now() - interval '119 seconds' +where flow_slug = 'LegacyCase'; + +select pgflow.requeue_stalled_tasks(); + +select is( + (select status from pgflow.step_tasks where flow_slug = 'LegacyCase'), + 'queued', + 'recovery requeued the stalled task' +); + +select ok( + (select q.vt <= clock_timestamp() + from pgmq.q_LegacyCase q + join pgflow.step_tasks st on st.message_id = q.msg_id), + 'recovery made the message immediately visible on the physical queue' +); + +-- Re-claim and complete: message archived into the physical archive table +select pgflow_tests.ensure_worker('LegacyCase'); +select pgflow_tests.read_and_start('LegacyCase'); + +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'LegacyCase'), + 'a', + 0, + null +); + +select is( + (select count(*) from pgflow.step_tasks where flow_slug = 'LegacyCase' and status = 'completed')::int, + 1, + 'task completed through the stored queue' +); + +select is( + (select count(*) from pgmq.a_LegacyCase), + 1::bigint, + 'completion archived the message on the physical archive table' +); + +select is( + (select count(*) from pgmq.q_LegacyCase), + 0::bigint, + 'physical queue is empty after archival' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/multi_queue_lifecycle.test.sql b/pkgs/core/supabase/tests/queue_identity/multi_queue_lifecycle.test.sql new file mode 100644 index 000000000..ece590ecb --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/multi_queue_lifecycle.test.sql @@ -0,0 +1,202 @@ +-- Multi-queue lifecycle (#650 review regression): when a run's tasks are +-- routed to more than one queue, terminal cleanup archives every queue's +-- messages. pgflow routes every step to the flow's default queue today, but +-- steps already carry a per-step queue route; this fixture re-routes one step +-- to a second queue before dispatch to prove no message stays active after +-- its task reaches a terminal state. +begin; +select plan(16); + +select pgflow_tests.reset_db(); + +-- ---------------------------------------------------------------- +-- Scenario 1: type violation in complete_task cancels across queues +-- ---------------------------------------------------------------- +select pgflow.create_flow('mqviol'); +select pgflow.add_step('mqviol', 'producer'); +select pgflow.add_step('mqviol', 'sibling'); +select pgflow.add_step('mqviol', 'consumer', array['producer'], step_type => 'map'); +select pgmq.create('mqviol_side'); +update pgflow.steps +set queue_name = 'mqviol_side' +where flow_slug = 'mqviol' and step_slug = 'sibling'; + +select run_id as viol_run_id from pgflow.start_flow('mqviol', '{}') \gset +select pgflow_tests.ensure_worker('mqviol'); +select pgflow_tests.read_and_start('mqviol'); + +select pgflow.complete_task( + :'viol_run_id'::uuid, + 'producer', + 0, + '{"not": "an array"}'::jsonb +); + +select is( + (select count(*) from pgflow.step_tasks + where run_id = :'viol_run_id'::uuid and status in ('queued', 'started')), + 0::bigint, + 'S1: no task left non-terminal after the type violation' +); + +select is( + (select count(*) from pgmq.q_mqviol), + 0::bigint, + 'S1: culprit message left the default queue' +); + +select is( + (select count(*) from pgmq.q_mqviol_side), + 0::bigint, + 'S1: cancelled sibling message left the side queue' +); + +select is( + (select count(*) from pgmq.a_mqviol), + 1::bigint, + 'S1: culprit message archived on the default queue' +); + +select is( + (select count(*) from pgmq.a_mqviol_side), + 1::bigint, + 'S1: cancelled sibling message archived on the side queue' +); + +-- ---------------------------------------------------------------- +-- Scenario 2: run failure in fail_task cancels across queues +-- ---------------------------------------------------------------- +select pgflow.create_flow('mqfail'); +select pgflow.add_step('mqfail', 'doomed', max_attempts => 1); +select pgflow.add_step('mqfail', 'sib_default'); +select pgflow.add_step('mqfail', 'sib_side'); +select pgmq.create('mqfail_side'); +update pgflow.steps +set queue_name = 'mqfail_side' +where flow_slug = 'mqfail' and step_slug = 'sib_side'; + +select run_id as fail_run_id from pgflow.start_flow('mqfail', '"x"'::jsonb) \gset +select pgflow_tests.ensure_worker('mqfail'); + +-- Claim the doomed task and the side-queue sibling; sib_default stays queued +select array_agg(msg_id) as fail_doomed_ids +from pgmq.read_with_poll('mqfail', 30, 1, 1, 50) \gset +select pgflow.start_tasks( + 'mqfail', + :'fail_doomed_ids'::bigint[], + '11111111-1111-1111-1111-111111111111'::uuid +, 'mqfail'); + +select array_agg(msg_id) as fail_sibling_ids +from pgmq.read_with_poll('mqfail_side', 30, 5, 1, 50) \gset +select pgflow.start_tasks( + 'mqfail', + :'fail_sibling_ids'::bigint[], + '11111111-1111-1111-1111-111111111111'::uuid, + 'mqfail_side' +); + +select is( + (select count(*) from pgflow.step_tasks + where run_id = :'fail_run_id'::uuid and status = 'started'), + 2::bigint, + 'S2: claimed tasks started across two queues' +); + +select pgflow.fail_task(:'fail_run_id'::uuid, 'doomed', 0, 'boom'); + +select is( + (select count(*) from pgflow.step_tasks + where run_id = :'fail_run_id'::uuid and status in ('queued', 'started')), + 0::bigint, + 'S2: no task left non-terminal after the run failed' +); + +select is( + (select count(*) from pgmq.q_mqfail), + 0::bigint, + 'S2: failed culprit and cancelled default-queue messages left the default queue' +); + +select is( + (select count(*) from pgmq.q_mqfail_side), + 0::bigint, + 'S2: cancelled sibling message left the side queue' +); + +select is( + (select count(*) from pgmq.a_mqfail), + 2::bigint, + 'S2: culprit and default-queue sibling archived on the default queue' +); + +select is( + (select count(*) from pgmq.a_mqfail_side), + 1::bigint, + 'S2: cancelled sibling message archived on the side queue' +); + +-- ---------------------------------------------------------------- +-- Scenario 3: condition failure in cascade_resolve_conditions cancels +-- across queues +-- ---------------------------------------------------------------- +select pgflow.create_flow('mqcond'); +select pgflow.add_step('mqcond', 'gate'); +select pgflow.add_step('mqcond', 'sib_default'); +select pgflow.add_step('mqcond', 'sib_side'); +select pgflow.add_step( + 'mqcond', + 'guarded', + array['gate'], + required_input_pattern => '{"go": true}', + when_unmet => 'fail' +); +select pgmq.create('mqcond_side'); +update pgflow.steps +set queue_name = 'mqcond_side' +where flow_slug = 'mqcond' and step_slug = 'sib_side'; + +select run_id as cond_run_id from pgflow.start_flow('mqcond', '{}') \gset +select pgflow_tests.ensure_worker('mqcond'); +select pgflow_tests.read_and_start('mqcond'); + +select pgflow.complete_task( + :'cond_run_id'::uuid, + 'gate', + 0, + '{"no": "go"}'::jsonb +); + +select is( + (select count(*) from pgflow.step_tasks + where run_id = :'cond_run_id'::uuid and status in ('queued', 'started')), + 0::bigint, + 'S3: no task left non-terminal after the condition failed the run' +); + +select is( + (select count(*) from pgmq.q_mqcond), + 0::bigint, + 'S3: completed gate and cancelled default-queue messages left the default queue' +); + +select is( + (select count(*) from pgmq.q_mqcond_side), + 0::bigint, + 'S3: cancelled sibling message left the side queue' +); + +select is( + (select count(*) from pgmq.a_mqcond), + 2::bigint, + 'S3: gate and default-queue sibling archived on the default queue' +); + +select is( + (select count(*) from pgmq.a_mqcond_side), + 1::bigint, + 'S3: cancelled sibling message archived on the side queue' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/queue_isolation.test.sql b/pkgs/core/supabase/tests/queue_identity/queue_isolation.test.sql new file mode 100644 index 000000000..aeb19458d --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/queue_isolation.test.sql @@ -0,0 +1,135 @@ +-- Queue isolation (#650): message ids are queue-scoped. A claim carrying the +-- wrong queue never crosses task identity; unmatched and wrong-flow messages +-- stay untouched while valid work from the same batch proceeds. +begin; +select plan(11); + +select pgflow_tests.reset_db(); + +-- Two flows; both queues number their first message 1 +select pgflow.create_flow('iso_a', null, null, 5); +select pgflow.add_step('iso_a', 'a', step_type => 'map'); +select pgflow.start_flow('iso_a', '[10, 20]'::jsonb); + +select pgflow.create_flow('iso_b', null, null, 5); +select pgflow.add_step('iso_b', 'b'); +select pgflow.start_flow('iso_b', '"x"'::jsonb); + +select is( + (select count(*) from pgflow.step_tasks st + join pgflow.step_tasks st2 on st2.message_id = st.message_id and st2.queue_name <> st.queue_name), + 2::bigint, + 'the same message id exists in both queues without crossing identity' +); + +select pgflow_tests.ensure_worker('iso_a'); +select pgflow_tests.ensure_worker('iso_b'); + +-- Read queue iso_a once: messages [1, 2] +select array_agg(msg_id) as ids_a into temporary iso_a_msgs +from pgmq.read_with_poll('iso_a', 30, 5, 1, 50); + +select is( + (select cardinality(ids_a) from iso_a_msgs)::text, + '2', + 'both messages read from queue iso_a' +); + +-- Claim with an explicitly mismatched queue name claims nothing +select is( + (select count(*) from pgflow.start_tasks( + 'iso_a', + (select ids_a from iso_a_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + 'elsewhere' + ))::int, + 0, + 'claim with a mismatched queue name claims nothing' +); + +-- A worker polling queue iso_a but asking for flow iso_b claims nothing: +-- the (queue_name, message_id) identity does not authorize a cross-flow claim +select is( + (select count(*) from pgflow.start_tasks( + 'iso_b', + (select ids_a from iso_a_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + 'iso_a' + ))::int, + 0, + 'claim for another flow through a foreign queue claims nothing' +); + +-- Defaulted queue name (released signature) was removed: the omitted +-- argument is rejected outright, and an explicit NULL claims nothing +-- (no silent fallback to a reconstructed queue) +select throws_ok( + $$ select pgflow.start_tasks( + 'iso_a', + (select ids_a from iso_a_msgs), + '11111111-1111-1111-1111-111111111111'::uuid + ) $$, + 'function pgflow.start_tasks(unknown, bigint[], uuid) does not exist', + 'omitted queue_name argument is rejected' +); + +select is( + (select count(*) from pgflow.start_tasks( + 'iso_a', + (select ids_a from iso_a_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + null + ))::int, + 0, + 'explicit NULL queue name claims nothing instead of falling back' +); + +select is( + (select count(*) from pgflow.start_tasks( + 'iso_a', + (select ids_a from iso_a_msgs), + '11111111-1111-1111-1111-111111111111'::uuid, + 'iso_a' + ))::int, + 2, + 'explicit queue name claims both iso_a tasks' +); + +select is( + (select status from pgflow.step_tasks where flow_slug = 'iso_b'), + 'queued', + 'identical message id in queue iso_b is untouched' +); + +-- Unmatched message: no task row for it. It is preserved, not archived, not +-- deleted, and valid work continues in the same batch. +select pgmq.send('iso_b', '{"foreign": true}'::jsonb); + +select array_agg(msg_id) as ids_b into temporary iso_b_msgs +from pgmq.read_with_poll('iso_b', 30, 5, 1, 50); + +select is( + (select count(*) from pgflow.start_tasks( + 'iso_b', + (select ids_b from iso_b_msgs), + '11111111-1111-1111-1111-111111111111'::uuid + , 'iso_b'))::int, + 1, + 'the valid task in the batch is claimed through its queue' +); + +select is( + (select count(*) from pgmq.q_iso_b q + where q.message->>'foreign' = 'true'), + 1::bigint, + 'unmatched message is preserved in the queue' +); + +select is( + (select count(*) from pgflow.step_tasks where flow_slug = 'iso_b' and status = 'started')::int, + 1, + 'valid work continued in the same batch' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/snapshot_creation.test.sql b/pkgs/core/supabase/tests/queue_identity/snapshot_creation.test.sql new file mode 100644 index 000000000..f384c066f --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/snapshot_creation.test.sql @@ -0,0 +1,69 @@ +-- Queue identity snapshots (#650): task creation copies the step's resolved +-- default route (lower(flow_slug)) and stores it on every task row. +begin; +select plan(8); + +select pgflow_tests.reset_db(); + +select pgflow.create_flow('SnapFlow', null, null, 5); +select pgflow.add_step('SnapFlow', 'first'); +select pgflow.add_step('SnapFlow', 'second', ARRAY['first']); + +select is( + (select array_agg(queue_name order by step_slug) from pgflow.steps where flow_slug = 'SnapFlow'), + array['snapflow', 'snapflow'], + 'add_step records the canonical lowercase default queue' +); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'snapflow'), + 1::bigint, + 'create_flow provisions the lowercase default queue' +); + +select pgflow.start_flow('SnapFlow', '"x"'::jsonb); + +select is( + (select count(*) from pgflow.step_tasks where queue_name = 'snapflow'), + 1::bigint, + 'task creation copies the step queue snapshot' +); + +select is( + (select queue_name from pgflow.step_tasks where step_slug = 'first'), + 'snapflow', + 'task row carries the stored queue name' +); + +-- Mixed-case flow slug: canonical queue is still lower(flow_slug) +select pgflow.create_flow('MyFlow', null, null, 5); +select pgflow.add_step('MyFlow', 'a'); + +select is( + (select queue_name from pgflow.steps where flow_slug = 'MyFlow'), + 'myflow', + 'mixed-case slug records the lowercase canonical queue name' +); + +select is( + (select count(*) from pgmq.list_queues() where queue_name = 'myflow'), + 1::bigint, + 'new mixed-case flow creates the lowercase queue' +); + +select is( + (select count(*) from pgmq.list_queues() where lower(queue_name) = 'myflow'), + 1::bigint, + 'no second metadata entry is created for the mixed-case flow' +); + +select pgflow.start_flow('MyFlow', '"x"'::jsonb); + +select is( + (select queue_name from pgflow.step_tasks where flow_slug = 'MyFlow'), + 'myflow', + 'task under a mixed-case flow stores the canonical queue name' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/snapshot_unchanged_across_lifecycle.test.sql b/pkgs/core/supabase/tests/queue_identity/snapshot_unchanged_across_lifecycle.test.sql new file mode 100644 index 000000000..ad8e434c7 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/snapshot_unchanged_across_lifecycle.test.sql @@ -0,0 +1,120 @@ +-- Queue identity snapshots (#650): lifecycle transitions (claim, retry, +-- recovery, completion, downstream dispatch, late callback) never rewrite the +-- stored queue name, and every message operation runs through it. +begin; +select plan(9); + +select pgflow_tests.reset_db(); + +select pgflow.create_flow('stableq', null, null, 5); +select pgflow.add_step('stableq', 'a', max_attempts => 2); +select pgflow.add_step('stableq', 'b', ARRAY['a']); +select pgflow.start_flow('stableq', '"x"'::jsonb); +select pgflow_tests.ensure_worker('stableq'); + +-- 1) Claim through the stored queue identity +select pgflow_tests.read_and_start('stableq'); + +select is( + (select queue_name from pgflow.step_tasks where step_slug = 'a'), + 'stableq', + 'claim keeps the stored queue snapshot' +); + +-- 2) Failure with retries left: message visibility set through stored queue +select pgflow.fail_task( + (select run_id from pgflow.runs where flow_slug = 'stableq'), + 'a', + 0, + 'boom' +); + +select is( + (select status from pgflow.step_tasks where step_slug = 'a'), + 'queued', + 'task requeued after failure' +); + +select is( + (select count(*) from pgmq.q_stableq q + join pgflow.step_tasks st on st.queue_name = 'stableq' and st.message_id = q.msg_id), + 1::bigint, + 'retry delay applied through the stored queue' +); + +-- 3) Stalled recovery requeues through the stored queue +select pgflow_tests.reset_message_visibility('stableq'); +select pgflow_tests.read_and_start('stableq'); +update pgflow.step_tasks +set queued_at = now() - interval '120 seconds', + started_at = now() - interval '119 seconds' +where step_slug = 'a'; + +select pgflow.requeue_stalled_tasks(); + +select is( + (select status from pgflow.step_tasks where step_slug = 'a'), + 'queued', + 'stalled task requeued by recovery' +); + +select is( + (select queue_name from pgflow.step_tasks where step_slug = 'a'), + 'stableq', + 'recovery keeps the stored queue snapshot' +); + +-- 4) Complete the recovered task: dependent step dispatches through the route +select pgflow_tests.reset_message_visibility('stableq'); +select pgflow_tests.read_and_start('stableq'); +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'stableq'), + 'a', + 0, + null +); + +select is( + (select count(*) from pgflow.step_tasks where queue_name = 'stableq'), + 2::bigint, + 'completion and downstream dispatch keep the stored queue snapshot' +); + +-- 5) Finish the run +select pgflow_tests.reset_message_visibility('stableq'); +select pgflow_tests.read_and_start('stableq'); +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'stableq'), + 'b', + 0, + null +); + +select is( + (select status from pgflow.runs where flow_slug = 'stableq'), + 'completed', + 'run completed' +); + +-- 6) Late callback on the terminal task: guarded, snapshot untouched +select pgflow.complete_task( + (select run_id from pgflow.runs where flow_slug = 'stableq'), + 'a', + 0, + null +); + +select is( + (select count(*) from pgflow.step_tasks where flow_slug = 'stableq' and queue_name <> 'stableq'), + 0::bigint, + 'no lifecycle path ever rewrote a task queue snapshot' +); + +select is( + (select count(*) from pgmq.a_stableq), + 2::bigint, + 'both messages archived through the stored queue' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/start_tasks/basic_start_tasks.test.sql b/pkgs/core/supabase/tests/start_tasks/basic_start_tasks.test.sql index c097e18f2..122a8198e 100644 --- a/pkgs/core/supabase/tests/start_tasks/basic_start_tasks.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/basic_start_tasks.test.sql @@ -21,7 +21,7 @@ started_tasks as ( 'simple', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'simple') ) -- TEST: start_tasks returns tasks for valid message IDs select is( @@ -75,7 +75,7 @@ select is( 'simple', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'simple')), 0, 'start_tasks should return no tasks when queue is empty' ); diff --git a/pkgs/core/supabase/tests/start_tasks/benign_duplicate.test.sql b/pkgs/core/supabase/tests/start_tasks/benign_duplicate.test.sql index 37b7ea51e..8097bdc2f 100644 --- a/pkgs/core/supabase/tests/start_tasks/benign_duplicate.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/benign_duplicate.test.sql @@ -47,7 +47,7 @@ with started as ( 'mixeddup', (select ids from mixeddup_msgs), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'mixeddup') ) select is( (select count(*)::int from started), @@ -70,7 +70,7 @@ with started as ( 'mixeddup', (select ids from mixeddup_msgs), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'mixeddup') ) select is( (select count(*)::int from started @@ -104,7 +104,7 @@ select is( (select count(*)::int from pgflow.start_tasks( 'mixeddup', (select array_agg(msg_id) from mixeddup_dup_msg where msg_id is not null), - '11111111-1111-1111-1111-111111111111'::uuid)), + '11111111-1111-1111-1111-111111111111'::uuid, 'mixeddup')), 0, 'repeatedly visible started message returns no task' ); diff --git a/pkgs/core/supabase/tests/start_tasks/builds_proper_input_from_deps_outputs.test.sql b/pkgs/core/supabase/tests/start_tasks/builds_proper_input_from_deps_outputs.test.sql index 675288bec..96685663e 100644 --- a/pkgs/core/supabase/tests/start_tasks/builds_proper_input_from_deps_outputs.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/builds_proper_input_from_deps_outputs.test.sql @@ -20,7 +20,7 @@ select pgflow.start_tasks( 'dep_flow', (select ids from first_msg_ids), '11111111-1111-1111-1111-111111111111'::uuid -); +, 'dep_flow'); select pgflow.complete_task( run_id => (select run_id from pgflow.runs where flow_slug = 'dep_flow'), @@ -39,7 +39,7 @@ select is( 'dep_flow', (select ids from second_msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) where step_slug = 'second'), + , 'dep_flow') where step_slug = 'second'), 1, 'start_tasks should return one task for dependent step' ); diff --git a/pkgs/core/supabase/tests/start_tasks/conditional_flow_input.test.sql b/pkgs/core/supabase/tests/start_tasks/conditional_flow_input.test.sql index bffff444b..a8a8f633c 100644 --- a/pkgs/core/supabase/tests/start_tasks/conditional_flow_input.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/conditional_flow_input.test.sql @@ -28,7 +28,7 @@ started_tasks as ( 'root_step_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'root_step_flow') ) select is( (select flow_input from started_tasks), @@ -69,7 +69,7 @@ started_tasks as ( 'dep_flow', (select ids from msg_ids), '22222222-2222-2222-2222-222222222222'::uuid - ) + , 'dep_flow') ) select is( (select flow_input from started_tasks), @@ -97,7 +97,7 @@ started_tasks as ( 'root_map_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'root_map_flow') ) select is( (select flow_input from started_tasks limit 1), @@ -138,7 +138,7 @@ started_tasks as ( 'dep_map_flow', (select ids from msg_ids), '33333333-3333-3333-3333-333333333333'::uuid - ) + , 'dep_map_flow') ) select is( (select flow_input from started_tasks limit 1), @@ -166,7 +166,7 @@ started_tasks as ( 'parallel_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'parallel_flow') ) select ok( (select bool_and(flow_input = '{"batch": "test"}'::jsonb) from started_tasks), @@ -195,7 +195,7 @@ started_tasks as ( 'mixed_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'mixed_flow') ) select is( (select count(*)::int from started_tasks where flow_input is not null), diff --git a/pkgs/core/supabase/tests/start_tasks/dependent_map_element_extraction.test.sql b/pkgs/core/supabase/tests/start_tasks/dependent_map_element_extraction.test.sql index 70ef7653a..614b9435c 100644 --- a/pkgs/core/supabase/tests/start_tasks/dependent_map_element_extraction.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/dependent_map_element_extraction.test.sql @@ -89,7 +89,7 @@ select is( 'dep_map_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'dep_map_flow')), '10'::jsonb, 'Task 0 should receive first element (10) from producer_step' ); @@ -99,7 +99,7 @@ select is( 'dep_map_flow', ARRAY[:'msg_id_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'dep_map_flow')), '20'::jsonb, 'Task 1 should receive second element (20) from producer_step' ); @@ -109,7 +109,7 @@ select is( 'dep_map_flow', ARRAY[:'msg_id_3'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'dep_map_flow')), '40'::jsonb, 'Task 3 should receive fourth element (40) from producer_step' ); diff --git a/pkgs/core/supabase/tests/start_tasks/does_not_start_tasks_for_skipped_step.test.sql b/pkgs/core/supabase/tests/start_tasks/does_not_start_tasks_for_skipped_step.test.sql index cc764736d..e9d1c516e 100644 --- a/pkgs/core/supabase/tests/start_tasks/does_not_start_tasks_for_skipped_step.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/does_not_start_tasks_for_skipped_step.test.sql @@ -25,7 +25,7 @@ with task0 as ( where flow_slug = 'skip_start_guard' and step_slug = 'map_a' and task_index = 0 ) select is( - (select count(*) from pgflow.start_tasks('skip_start_guard', array[(select message_id from task0)::bigint], pgflow_tests.ensure_worker('skip_start_guard'))), + (select count(*) from pgflow.start_tasks('skip_start_guard', array[(select message_id from task0)::bigint], pgflow_tests.ensure_worker('skip_start_guard'), 'skip_start_guard')), 1::bigint, 'Should start 1 task for map_a[0]' ); @@ -67,7 +67,7 @@ with task1 as ( where flow_slug = 'skip_start_guard' and step_slug = 'map_a' and task_index = 1 ) select is( - (select count(*) from pgflow.start_tasks('skip_start_guard', array[(select message_id from task1)::bigint], pgflow_tests.ensure_worker('skip_start_guard'))), + (select count(*) from pgflow.start_tasks('skip_start_guard', array[(select message_id from task1)::bigint], pgflow_tests.ensure_worker('skip_start_guard'), 'skip_start_guard')), 0::bigint, 'Should NOT start task for map_a[1] when step is skipped' ); diff --git a/pkgs/core/supabase/tests/start_tasks/map_large_array.test.sql b/pkgs/core/supabase/tests/start_tasks/map_large_array.test.sql index 4e59af75c..c5f704eb2 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_large_array.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_large_array.test.sql @@ -41,7 +41,7 @@ select is( 'large_array_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'large_array_flow')), row('1'::jsonb, 0), 'Task at index 0 should receive element 1 with task_index = 0' ); @@ -55,7 +55,7 @@ select is( 'large_array_flow', ARRAY[:'msg_id_49'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'large_array_flow')), '50'::jsonb, 'Task at index 49 should receive element 50' ); @@ -69,7 +69,7 @@ select is( 'large_array_flow', ARRAY[:'msg_id_99'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'large_array_flow')), '100'::jsonb, 'Task at index 99 should receive element 100' ); @@ -83,7 +83,7 @@ select is( 'large_array_flow', ARRAY[:'msg_id_149'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'large_array_flow')), row('150'::jsonb, 149), 'Task at index 149 should receive element 150 with task_index = 149' ); diff --git a/pkgs/core/supabase/tests/start_tasks/map_mixed_types.test.sql b/pkgs/core/supabase/tests/start_tasks/map_mixed_types.test.sql index 27fa0b42b..790b4bf46 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_mixed_types.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_mixed_types.test.sql @@ -54,7 +54,7 @@ select is( 'mixed_types_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'mixed_types_flow')), '"text value"'::jsonb, 'Task 0 should receive string element' ); @@ -64,7 +64,7 @@ select is( 'mixed_types_flow', ARRAY[:'msg_id_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'mixed_types_flow')), '42'::jsonb, 'Task 1 should receive number element' ); @@ -74,7 +74,7 @@ select is( 'mixed_types_flow', ARRAY[:'msg_id_2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'mixed_types_flow')), '{"key": "value", "nested": {"id": 1}}'::jsonb, 'Task 2 should receive object element with nested structure' ); @@ -84,7 +84,7 @@ select is( 'mixed_types_flow', ARRAY[:'msg_id_3'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'mixed_types_flow')), 'true'::jsonb, 'Task 3 should receive boolean element' ); @@ -94,7 +94,7 @@ select is( 'mixed_types_flow', ARRAY[:'msg_id_4'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'mixed_types_flow')), 'null'::jsonb, 'Task 4 should receive null element' ); diff --git a/pkgs/core/supabase/tests/start_tasks/map_nested_arrays.test.sql b/pkgs/core/supabase/tests/start_tasks/map_nested_arrays.test.sql index 809b53015..adff0006e 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_nested_arrays.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_nested_arrays.test.sql @@ -53,7 +53,7 @@ select is( 'nested_arrays_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'nested_arrays_flow')), '[1, 2]'::jsonb, 'Task 0 should receive first sub-array [1, 2]' ); @@ -63,7 +63,7 @@ select is( 'nested_arrays_flow', ARRAY[:'msg_id_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'nested_arrays_flow')), '[3, 4, 5]'::jsonb, 'Task 1 should receive second sub-array [3, 4, 5]' ); @@ -73,7 +73,7 @@ select is( 'nested_arrays_flow', ARRAY[:'msg_id_2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'nested_arrays_flow')), '[]'::jsonb, 'Task 2 should receive empty sub-array []' ); @@ -83,7 +83,7 @@ select is( 'nested_arrays_flow', ARRAY[:'msg_id_4'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'nested_arrays_flow')), '[{"id": 1}, {"id": 2}]'::jsonb, 'Task 4 should receive array of objects' ); diff --git a/pkgs/core/supabase/tests/start_tasks/map_object_elements.test.sql b/pkgs/core/supabase/tests/start_tasks/map_object_elements.test.sql index fb49ef835..143a0e550 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_object_elements.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_object_elements.test.sql @@ -81,7 +81,7 @@ select is( 'object_elements_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'object_elements_flow')), 'Alice', 'Task 0 should receive complete Alice object' ); @@ -91,7 +91,7 @@ select is( 'object_elements_flow', ARRAY[:'msg_id_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'object_elements_flow')), 'light', 'Task 1 should receive complete Bob object with nested preferences' ); @@ -101,7 +101,7 @@ select is( 'object_elements_flow', ARRAY[:'msg_id_2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'object_elements_flow')), 'null'::jsonb, 'Task 2 should receive Charlie object with null metadata' ); @@ -117,7 +117,7 @@ select is( 'object_elements_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'object_elements_flow')), 2, 'Task 0 object should have tags array with 2 elements' ); diff --git a/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql b/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql index edc547fb7..963f265ba 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql @@ -61,7 +61,7 @@ BEGIN 'map_perf_flow', v_msg_ids, '11111111-1111-1111-1111-111111111111'::uuid - ); + , 'map_perf_flow'); v_end_time := clock_timestamp(); v_start_tasks_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; @@ -117,7 +117,7 @@ BEGIN 'map_perf_flow', v_msg_ids, '11111111-1111-1111-1111-111111111111'::uuid - ); + , 'map_perf_flow'); v_end_time := clock_timestamp(); v_start_tasks_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; @@ -173,7 +173,7 @@ BEGIN 'map_perf_flow', v_msg_ids, '11111111-1111-1111-1111-111111111111'::uuid - ); + , 'map_perf_flow'); v_end_time := clock_timestamp(); v_start_tasks_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; @@ -229,7 +229,7 @@ BEGIN 'map_perf_flow', v_msg_ids, '11111111-1111-1111-1111-111111111111'::uuid - ); + , 'map_perf_flow'); v_end_time := clock_timestamp(); v_start_tasks_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; diff --git a/pkgs/core/supabase/tests/start_tasks/map_to_map_chain.test.sql b/pkgs/core/supabase/tests/start_tasks/map_to_map_chain.test.sql index 4b397dad5..42c797dcd 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_to_map_chain.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_to_map_chain.test.sql @@ -52,7 +52,7 @@ select is( 'map_chain_flow', ARRAY[:'first_map_msg_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'map_chain_flow')), '1'::jsonb, 'First map task 0 should receive element 1' ); @@ -63,7 +63,7 @@ select is( 'map_chain_flow', ARRAY[:'first_map_msg_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'map_chain_flow')), '2'::jsonb, 'First map task 1 should receive element 2' ); @@ -134,7 +134,7 @@ select is( 'map_chain_test2', ARRAY[:'consumer_msg_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'map_chain_test2')), '{"value": 10}'::jsonb, 'Consumer map task 0 should receive first element from producer array' ); @@ -144,7 +144,7 @@ select is( 'map_chain_test2', ARRAY[:'consumer_msg_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'map_chain_test2')), '{"value": 20}'::jsonb, 'Consumer map task 1 should receive second element from producer array' ); diff --git a/pkgs/core/supabase/tests/start_tasks/multiple_task_processing.test.sql b/pkgs/core/supabase/tests/start_tasks/multiple_task_processing.test.sql index a75b23e4a..fab1eb460 100644 --- a/pkgs/core/supabase/tests/start_tasks/multiple_task_processing.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/multiple_task_processing.test.sql @@ -26,7 +26,7 @@ select is( 'multi_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'multi_flow')), 4, 'start_tasks should return multiple tasks when multiple messages available' ); diff --git a/pkgs/core/supabase/tests/start_tasks/returns_flow_input.test.sql b/pkgs/core/supabase/tests/start_tasks/returns_flow_input.test.sql index cea8c936d..73a73e7f3 100644 --- a/pkgs/core/supabase/tests/start_tasks/returns_flow_input.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/returns_flow_input.test.sql @@ -28,7 +28,7 @@ started_tasks as ( 'simple_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'simple_flow') ) select is( (select flow_input from started_tasks), @@ -69,7 +69,7 @@ started_tasks as ( 'dep_flow', (select ids from msg_ids), '22222222-2222-2222-2222-222222222222'::uuid - ) + , 'dep_flow') ) select is( (select flow_input from started_tasks), @@ -98,7 +98,7 @@ select * from pgflow.start_tasks( 'map_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid -); +, 'map_flow'); -- Test 3: All map tasks should have NULL flow_input (consistent) select is( @@ -137,7 +137,7 @@ started_tasks as ( 'multi_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'multi_flow') ) select ok( (select bool_and(flow_input = '{"batch": "test"}'::jsonb) from started_tasks), diff --git a/pkgs/core/supabase/tests/start_tasks/returns_only_claimed_tasks.test.sql b/pkgs/core/supabase/tests/start_tasks/returns_only_claimed_tasks.test.sql index 590f73618..affc823af 100644 --- a/pkgs/core/supabase/tests/start_tasks/returns_only_claimed_tasks.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/returns_only_claimed_tasks.test.sql @@ -42,7 +42,7 @@ select is( 'start_claim_guard', array[:'test_message_id'::bigint], pgflow_tests.ensure_worker('start_claim_guard') - )), + , 'start_claim_guard')), 0, 'start_tasks should return only rows claimed by the guarded update' ); diff --git a/pkgs/core/supabase/tests/start_tasks/returns_task_index.test.sql b/pkgs/core/supabase/tests/start_tasks/returns_task_index.test.sql index 0bf2d8ae1..4020711a7 100644 --- a/pkgs/core/supabase/tests/start_tasks/returns_task_index.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/returns_task_index.test.sql @@ -21,7 +21,7 @@ started_tasks as ( 'single_task', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'single_task') ) select is( (select task_index from started_tasks), @@ -50,7 +50,7 @@ started_tasks as ( 'map_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) order by task_index + , 'map_flow') order by task_index ) select is( array_agg(task_index order by task_index), @@ -79,7 +79,7 @@ started_tasks as ( 'map_five', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) order by task_index + , 'map_five') order by task_index ) select is( array_agg(task_index order by task_index), @@ -130,7 +130,7 @@ started_tasks as ( 'map_chain', (select ids from msg_ids), '22222222-2222-2222-2222-222222222222'::uuid - ) order by task_index + , 'map_chain') order by task_index ) select is( array_agg(task_index order by task_index), @@ -171,7 +171,7 @@ started_tasks as ( 'sequential', (select ids from msg_ids), '33333333-3333-3333-3333-333333333333'::uuid - ) + , 'sequential') ) select is( (select task_index from started_tasks), diff --git a/pkgs/core/supabase/tests/start_tasks/root_map_element_extraction.test.sql b/pkgs/core/supabase/tests/start_tasks/root_map_element_extraction.test.sql index 31461a9e7..bd7de0e8f 100644 --- a/pkgs/core/supabase/tests/start_tasks/root_map_element_extraction.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/root_map_element_extraction.test.sql @@ -49,7 +49,7 @@ select is( 'root_map_flow', ARRAY[:'msg_id_0'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'root_map_flow')), row('"apple"'::jsonb, 0), 'Task 0 should receive first array element (apple) with task_index = 0' ); @@ -60,7 +60,7 @@ select is( 'root_map_flow', ARRAY[:'msg_id_1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'root_map_flow')), row('"banana"'::jsonb, 1), 'Task 1 should receive second array element (banana) with task_index = 1' ); @@ -71,7 +71,7 @@ select is( 'root_map_flow', ARRAY[:'msg_id_2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid - )), + , 'root_map_flow')), row('"cherry"'::jsonb, 2), 'Task 2 should receive third array element (cherry) with task_index = 2' ); diff --git a/pkgs/core/supabase/tests/start_tasks/start_tasks_input_assembly_performance.test.sql b/pkgs/core/supabase/tests/start_tasks/start_tasks_input_assembly_performance.test.sql index bceafe205..94d996edf 100644 --- a/pkgs/core/supabase/tests/start_tasks/start_tasks_input_assembly_performance.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/start_tasks_input_assembly_performance.test.sql @@ -78,7 +78,7 @@ BEGIN 'input_perf_flow', v_msg_ids, '11111111-1111-1111-1111-111111111111'::uuid - ); + , 'input_perf_flow'); v_end_time := clock_timestamp(); v_current_ms := EXTRACT(EPOCH FROM (v_end_time - v_start_time)) * 1000; diff --git a/pkgs/core/supabase/tests/start_tasks/started_at_timestamps.test.sql b/pkgs/core/supabase/tests/start_tasks/started_at_timestamps.test.sql index 7adb9243e..9daa92050 100644 --- a/pkgs/core/supabase/tests/start_tasks/started_at_timestamps.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/started_at_timestamps.test.sql @@ -23,7 +23,7 @@ select pgflow.start_tasks( 'timestamp_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid -); +, 'timestamp_flow'); -- TEST: started_at should be set after start_tasks select isnt( diff --git a/pkgs/core/supabase/tests/start_tasks/status_transitions.test.sql b/pkgs/core/supabase/tests/start_tasks/status_transitions.test.sql index 3b583491c..f9a284338 100644 --- a/pkgs/core/supabase/tests/start_tasks/status_transitions.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/status_transitions.test.sql @@ -23,7 +23,7 @@ select pgflow.start_tasks( 'status_flow', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid -); +, 'status_flow'); -- TEST: Task status should be 'started' after start_tasks select is( diff --git a/pkgs/core/supabase/tests/start_tasks/task_index_returned_correctly.test.sql b/pkgs/core/supabase/tests/start_tasks/task_index_returned_correctly.test.sql index f935bebeb..8b939243f 100644 --- a/pkgs/core/supabase/tests/start_tasks/task_index_returned_correctly.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/task_index_returned_correctly.test.sql @@ -62,7 +62,7 @@ started_tasks as ( 'test_task_index', (select ids from msg_ids), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'test_task_index') ) -- TEST: All returned task_index values match the expected indices select is( diff --git a/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql b/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql index 03a99e0e6..baee58d1b 100644 --- a/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql @@ -36,7 +36,7 @@ select throws_ok( 'visfail', (select ids from visfail_msgs), '11111111-1111-1111-1111-111111111111'::uuid - ) $$, + , 'visfail') $$, 'relation "pgmq.q_visfail" does not exist', 'missing queue table makes start_tasks fail' ); @@ -95,7 +95,7 @@ select throws_ok( 'vispartial', (select ids from vispartial_msgs), '11111111-1111-1111-1111-111111111111'::uuid - ) $$, + , 'vispartial') $$, 'invalid input syntax for type integer: "start_tasks(): visibility updated 1 of 2 claimed messages"', 'partial visibility mismatch fails the whole statement and returns nothing' ); diff --git a/pkgs/core/supabase/tests/start_tasks/visibility_timeout.test.sql b/pkgs/core/supabase/tests/start_tasks/visibility_timeout.test.sql index 17b439192..9b9a3e561 100644 --- a/pkgs/core/supabase/tests/start_tasks/visibility_timeout.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/visibility_timeout.test.sql @@ -80,7 +80,7 @@ with started as ( 'vt_formula', (select array_agg(msg_id) from vt_formula_msg), '11111111-1111-1111-1111-111111111111'::uuid - ) + , 'vt_formula') ) select is( (select count(*)::int from started), diff --git a/pkgs/core/supabase/tests/start_tasks/worker_tracking.test.sql b/pkgs/core/supabase/tests/start_tasks/worker_tracking.test.sql index e06d377a0..6b845408d 100644 --- a/pkgs/core/supabase/tests/start_tasks/worker_tracking.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/worker_tracking.test.sql @@ -19,7 +19,7 @@ select pgflow.start_tasks( 'simple', (select ids from msg_ids), '00000000-0000-0000-0000-000000000001'::uuid -); +, 'simple'); -- TEST: Task should be assigned to the worker select is( @@ -38,7 +38,7 @@ select pgflow.start_tasks( 'simple', (select ids from msg_ids), '00000000-0000-0000-0000-000000000002'::uuid -); +, 'simple'); -- TEST: Second task should be assigned to different worker select is( @@ -53,7 +53,7 @@ select is( 'simple', array[]::bigint[], '00000000-0000-0000-0000-000000000001'::uuid - )), + , 'simple')), 0, 'start_tasks with empty array should return no tasks' ); diff --git a/pkgs/core/supabase/tests/type_violations/cancels_unfinished_tasks.test.sql b/pkgs/core/supabase/tests/type_violations/cancels_unfinished_tasks.test.sql index b9a6d8d2e..4c293c2de 100644 --- a/pkgs/core/supabase/tests/type_violations/cancels_unfinished_tasks.test.sql +++ b/pkgs/core/supabase/tests/type_violations/cancels_unfinished_tasks.test.sql @@ -78,11 +78,11 @@ select :'test_run_id'::uuid as run_id into temporary test_run_ids; -- Start branch1 and branch2 (started siblings); branch3 stays queued select message_id as msg_b1 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'branch1' \gset -select pgflow.start_tasks('type_violation_cancel', array[:'msg_b1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid); +select pgflow.start_tasks('type_violation_cancel', array[:'msg_b1'::bigint], '11111111-1111-1111-1111-111111111111'::uuid, 'type_violation_cancel'); select message_id as msg_b2 from pgflow.step_tasks where run_id = :'test_run_id'::uuid and step_slug = 'branch2' \gset -select pgflow.start_tasks('type_violation_cancel', array[:'msg_b2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid); +select pgflow.start_tasks('type_violation_cancel', array[:'msg_b2'::bigint], '11111111-1111-1111-1111-111111111111'::uuid, 'type_violation_cancel'); -- Trigger type violation by completing branch1 with a non-array (consumer_map expects array) select lives_ok( diff --git a/pkgs/core/supabase/upgrade_fixture/assertions_0_16.sql b/pkgs/core/supabase/upgrade_fixture/assertions_0_16.sql new file mode 100644 index 000000000..79b3aa588 --- /dev/null +++ b/pkgs/core/supabase/upgrade_fixture/assertions_0_16.sql @@ -0,0 +1,194 @@ +-- 0.16.0 upgrade fixture assertions: run AFTER the persist_queue_identity +-- migration is applied to the seeded 0.16.0 database (#650). +-- Plain DO-block asserts (the fixture container has no pgTAP); any failure +-- raises, psql runs with ON_ERROR_STOP=1, the script exits non-zero. + +do $$ +declare + v_count int; + v_queue text; +begin + -- ========================================== + -- Backfill: every step routes to lower(flow_slug) + -- ========================================== + select count(*) into v_count + from pgflow.steps + where queue_name is distinct from lower(flow_slug); + if v_count <> 0 then + raise exception 'steps backfill: % rows without the canonical queue name', v_count; + end if; + + -- Every task (including NULL message_id rows) carries the snapshot + if (select count(*) from pgflow.step_tasks where queue_name is null) <> 0 then + raise exception 'step_tasks backfill: rows without a queue name'; + end if; + + select count(*) into v_count + from pgflow.step_tasks st + join pgflow.runs r on r.run_id = st.run_id + where st.queue_name is distinct from lower(r.flow_slug); + if v_count <> 0 then + raise exception 'step_tasks backfill: % rows with a wrong queue name', v_count; + end if; + + -- Exact bigint identity preserved + if (select count(*) from pgflow.step_tasks where message_id = 9223372036854775807) <> 1 then + raise exception 'message id beyond the safe integer range was not preserved exactly'; + end if; + + -- Constraints are installed + if not exists ( + select 1 from pg_constraint + where conname = 'queue_name_is_valid' and conrelid = 'pgflow.steps'::regclass + ) then + raise exception 'steps queue_name_is_valid constraint missing'; + end if; + + if not exists ( + select 1 from pg_constraint + where conname = 'queue_name_is_valid' and conrelid = 'pgflow.step_tasks'::regclass + ) then + raise exception 'step_tasks queue_name_is_valid constraint missing'; + end if; + + if not exists ( + select 1 from pg_class c join pg_namespace n on n.oid = c.relnamespace + where c.relname = 'idx_step_tasks_queue_message' and n.nspname = 'pgflow' + ) then + raise exception 'queue/message unique index missing'; + end if; +end $$; + +-- ========================================== +-- Runtime after upgrade: legacy mixed-case queue stays usable end to end +-- ========================================== +select queue_name into temporary fixture_mixed_queue +from pgflow.step_tasks st +join pgflow.runs r on r.run_id = st.run_id +where r.flow_slug = 'MixedCaseFlow' +limit 1; + +do $$ +declare + v_queue text; +begin + select queue_name into v_queue from fixture_mixed_queue; + if v_queue is distinct from 'mixedcaseflow' then + raise exception 'mixed-case flow stores canonical name, got %', v_queue; + end if; +end $$; + +-- Dispatch through the resolved original spelling +select pgflow.start_flow('MixedCaseFlow', '[30]'::jsonb); + +do $$ +declare + v_count int; +begin + -- The new run's messages landed on the physical mixed-case queue + select count(*) into v_count + from pgmq.q_MixedCaseFlow q + join pgflow.step_tasks st on st.queue_name = 'mixedcaseflow' and st.message_id = q.msg_id + join pgflow.runs r on r.run_id = st.run_id + where r.status = 'started'; + if v_count < 1 then + raise exception 'post-upgrade dispatch did not reach the physical mixed-case queue'; + end if; +end $$; + +-- Read the physical mixed-case queue and claim through the canonical name. +-- queue_name is required (#650): the claim passes the canonical stored name. +create temp table fixture_claim as +select msg_id from pgmq.read('MixedCaseFlow', 30, 10); + +select pgflow.start_tasks( + 'MixedCaseFlow', + (select array_agg(msg_id) from fixture_claim), + '11111111-1111-1111-1111-111111111111'::uuid, + 'mixedcaseflow' +); + +-- The obsolete three-argument overload is gone after the upgrade: an +-- omitted queue_name argument is rejected outright +do $fix$ +begin + begin + perform pgflow.start_tasks( + 'MixedCaseFlow', + (select array_agg(msg_id)::bigint[] from fixture_claim), + '11111111-1111-1111-1111-111111111111'::uuid + ); + raise exception 'obsolete 3-argument start_tasks call still works after upgrade'; + exception + when undefined_function then null; + end; +end +$fix$; + +do $$ +declare + r record; +begin + for r in + select run_id, task_index + from pgflow.step_tasks + where flow_slug = 'MixedCaseFlow' and status = 'started' + order by run_id, task_index + loop + perform pgflow.complete_task(r.run_id, 'a', r.task_index, null); + end loop; + + if (select count(*) from pgflow.step_tasks + where flow_slug = 'MixedCaseFlow' and status = 'started') <> 0 then + raise exception 'post-upgrade completion left started tasks'; + end if; + + if (select count(*) from pgmq.a_MixedCaseFlow) <> 3 then + raise exception 'post-upgrade archive missed the physical mixed-case queue'; + end if; +end $$; + +-- Deletion drops the queue through its persisted route and original spelling +select pgflow.delete_flow_and_data('MixedCaseFlow'); + +do $$ +begin + if exists (select 1 from pgmq.list_queues() where lower(queue_name) = 'mixedcaseflow') then + raise exception 'delete_flow_and_data did not drop the mixed-case queue'; + end if; +end $$; + +-- Plain flow startup and execution after upgrade: verification and a claim +do $$ +declare + v_result jsonb; + v_ids bigint[]; +begin + select pgflow.ensure_flow_compiled('plain_flow', jsonb_build_object('steps', jsonb_build_array( + jsonb_build_object( + 'slug', 'a', + 'stepType', 'map', + 'dependencies', '[]'::jsonb, + 'requiredInputPattern', jsonb_build_object('defined', false), + 'forbiddenInputPattern', jsonb_build_object('defined', false) + ) + ))) into v_result; + if v_result->>'status' is distinct from 'verified' then + raise exception 'plain flow startup verification failed: %', v_result; + end if; + + select array_agg(msg_id) into v_ids from pgmq.read('plain_flow', 30, 1); + perform pgflow.start_tasks( + 'plain_flow', + v_ids, + '11111111-1111-1111-1111-111111111111'::uuid, + 'plain_flow' + ); + + if (select count(*) from pgflow.step_tasks + where flow_slug = 'plain_flow' and status = 'started') <> 2 then + raise exception 'plain flow execution after upgrade did not claim the queued task'; + end if; +end $$; + +select 'PASS: 0.16.0 upgrade fixture (backfill + runtime)' as result; diff --git a/pkgs/core/supabase/upgrade_fixture/seed_0_16.sql b/pkgs/core/supabase/upgrade_fixture/seed_0_16.sql new file mode 100644 index 000000000..48676f999 --- /dev/null +++ b/pkgs/core/supabase/upgrade_fixture/seed_0_16.sql @@ -0,0 +1,43 @@ +-- 0.16.0 upgrade fixture seed: populated database state immediately before +-- the persist_queue_identity migration (#650). Runs on a database at 0.16.0 +-- (migrations up to 20260907082520 only), using the 0.16.0 functions. + +-- Plain lowercase flow with real dispatched messages +select pgflow.create_flow('plain_flow', null, null, 5); +select pgflow.add_step('plain_flow', 'a', max_attempts => 2, step_type => 'map'); +select pgflow.start_flow('plain_flow', '[1,2]'::jsonb); + +-- Mixed-case flow: 0.16.0 create_flow provisions the queue under the exact +-- slug spelling, so the physical queue is 'MixedCaseFlow' +select pgflow.create_flow('MixedCaseFlow', null, null, 5); +select pgflow.add_step('MixedCaseFlow', 'a', max_attempts => 2, step_type => 'map'); +select pgflow.start_flow('MixedCaseFlow', '[10,20]'::jsonb); + +-- Claim one plain_flow task so the upgraded database has a started task +insert into pgflow.workers (worker_id, queue_name, function_name, last_heartbeat_at) +values ('11111111-1111-1111-1111-111111111111', 'plain_flow', 'fixture_worker', now()); + +create temp table fixture_msgs as +select msg_id from pgmq.read('plain_flow', 30, 1); + +select pgflow.start_tasks( + 'plain_flow', + (select array_agg(msg_id) from fixture_msgs), + '11111111-1111-1111-1111-111111111111'::uuid +); + +-- Task with NULL message_id (pre-dispatch row): must backfill too +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, message_id) +select 'plain_flow', run_id, 'a', 97, null +from pgflow.runs +where flow_slug = 'plain_flow'; + +-- Task with a message id beyond the JavaScript safe integer range: must be +-- stored and compared exactly after the upgrade +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, message_id) +select 'plain_flow', run_id, 'a', 98, 9223372036854775807 +from pgflow.runs +where flow_slug = 'plain_flow'; + +-- Empty flow: definition only, queue provisioned, no steps +select pgflow.create_flow('empty_flow'); diff --git a/pkgs/core/supabase/upgrade_fixture/seed_0_16_conflict.sql b/pkgs/core/supabase/upgrade_fixture/seed_0_16_conflict.sql new file mode 100644 index 000000000..7c30271ac --- /dev/null +++ b/pkgs/core/supabase/upgrade_fixture/seed_0_16_conflict.sql @@ -0,0 +1,6 @@ +-- 0.16.0 conflict fixture seed: two existing flow definitions whose normalized +-- default queue collides ('MyConflict' and 'myconflict'). The 0.16.0 schema +-- allows this; the persist_queue_identity migration must reject it and leave +-- the database unchanged (#650). +insert into pgflow.flows (flow_slug) values ('MyConflict'); +insert into pgflow.flows (flow_slug) values ('myconflict'); diff --git a/pkgs/dsl/README.md b/pkgs/dsl/README.md index d0a9eee6e..c7e072d19 100644 --- a/pkgs/dsl/README.md +++ b/pkgs/dsl/README.md @@ -243,7 +243,7 @@ All platforms provide these core resources: - **`ctx.rawMessage`** - Original pgmq message with metadata ```typescript interface PgmqMessageRecord { - msg_id: number; + msg_id: string; // queue-scoped exact decimal string read_ct: number; enqueued_at: Date; vt: Date; @@ -256,7 +256,7 @@ All platforms provide these core resources: flow_slug: string; run_id: string; step_slug: string; - msg_id: number; + msg_id: string; // queue-scoped exact decimal string } ``` - **`ctx.workerConfig`** - Resolved worker configuration with all defaults applied diff --git a/pkgs/dsl/__tests__/types/context-inference.test-d.ts b/pkgs/dsl/__tests__/types/context-inference.test-d.ts index 0d2ddf6e6..fc1c7134e 100644 --- a/pkgs/dsl/__tests__/types/context-inference.test-d.ts +++ b/pkgs/dsl/__tests__/types/context-inference.test-d.ts @@ -21,7 +21,7 @@ describe('Context Type Inference Tests', () => { expectTypeOf(context.env).toEqualTypeOf>(); expectTypeOf(context.shutdownSignal).toEqualTypeOf(); expectTypeOf(context.stepTask.run_id).toEqualTypeOf(); - expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); + expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); return { processed: true }; }); diff --git a/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts b/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts index 76df19397..7b717ee26 100644 --- a/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts +++ b/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts @@ -13,7 +13,7 @@ describe('Supabase Flow Context Inference', () => { // FlowContext properties expectTypeOf(context.stepTask.run_id).toEqualTypeOf(); - expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); + expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); expectTypeOf(context.workerConfig.maxConcurrent).toEqualTypeOf(); expectTypeOf(context.env).toMatchTypeOf>(); expectTypeOf(context.shutdownSignal).toEqualTypeOf(); diff --git a/pkgs/dsl/src/dsl.ts b/pkgs/dsl/src/dsl.ts index 4cc611000..e27b3d8ed 100644 --- a/pkgs/dsl/src/dsl.ts +++ b/pkgs/dsl/src/dsl.ts @@ -393,8 +393,10 @@ export interface WorkerConfig { } // Message record interface (minimal contract - actual type defined in @pgflow/core) +// msg_id is an exact decimal string: PGMQ message ids are queue-scoped +// bigints that can exceed the JavaScript safe integer range (#650). export interface MessageRecord { - msg_id: number; + msg_id: string; read_ct: number; enqueued_at: string; vt: string; @@ -407,7 +409,7 @@ export interface StepTaskRecord { run_id: string; step_slug: string; input: Json; // JSON-serializable input from database (JSONB column) - msg_id: number; + msg_id: string; } // Base context for queue workers (no stepTask) diff --git a/pkgs/edge-worker/README.md b/pkgs/edge-worker/README.md index 28e761dec..e217453cb 100644 --- a/pkgs/edge-worker/README.md +++ b/pkgs/edge-worker/README.md @@ -105,7 +105,7 @@ These resources are provided regardless of platform: - **`rawMessage`** - Original pgmq message with metadata ```typescript interface PgmqMessageRecord { - msg_id: number; + msg_id: string; // queue-scoped exact decimal string read_ct: number; enqueued_at: Date; vt: Date; @@ -119,7 +119,7 @@ These resources are provided regardless of platform: run_id: string; step_slug: string; input: StepInput; - msg_id: number; + msg_id: string; // queue-scoped exact decimal string } ``` diff --git a/pkgs/edge-worker/src/core/context.ts b/pkgs/edge-worker/src/core/context.ts index 38490d762..7e9b6f53c 100644 --- a/pkgs/edge-worker/src/core/context.ts +++ b/pkgs/edge-worker/src/core/context.ts @@ -63,7 +63,7 @@ export type StepTaskContext< * immediately (if provided) or lazy-loads from the runs table. */ export interface StepTaskWithMessage { - msg_id : number; + msg_id : string; message: PgmqMessageRecord>; task : StepTaskRecord; flowInput: ExtractFlowInput | null; diff --git a/pkgs/edge-worker/src/core/types.ts b/pkgs/edge-worker/src/core/types.ts index e74b1e1dc..8bf895ce3 100644 --- a/pkgs/edge-worker/src/core/types.ts +++ b/pkgs/edge-worker/src/core/types.ts @@ -11,12 +11,12 @@ export interface IPoller { } export interface IExecutor { - get msgId(): number; + get msgId(): string; execute(): Promise; } export interface IMessage { - msg_id: number; + msg_id: string; } export interface ILifecycle { diff --git a/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts b/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts index 1ca8b6dad..4c12fc259 100644 --- a/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts +++ b/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts @@ -112,7 +112,10 @@ export class FlowWorkerLifecycle implements InternalLifec } get queueName() { - return this.flow.slug; + // Canonical queue identity: lower(flow slug) (#650). PGMQ message + // operations normalize names themselves, so polling and claiming address + // mixed-case physical queues through the canonical name directly. + return this.flow.slug.toLowerCase(); } // TODO: Temporary getter for supplier pattern until we refactor initialization diff --git a/pkgs/edge-worker/src/flow/StepTaskExecutor.ts b/pkgs/edge-worker/src/flow/StepTaskExecutor.ts index 6f932cffd..18e113f66 100644 --- a/pkgs/edge-worker/src/flow/StepTaskExecutor.ts +++ b/pkgs/edge-worker/src/flow/StepTaskExecutor.ts @@ -68,8 +68,8 @@ export class StepTaskExecutor } const workerId = this.getWorkerId(); + const queueName = this.config.queueName; const batchSize = limit === undefined ? this.config.batchSize : Math.min(this.config.batchSize, limit); @@ -53,7 +57,7 @@ export class StepTaskPoller try { // Phase 1: Read messages from queue const messages = await this.adapter.readMessages( - this.config.queueName, + queueName, this.config.visibilityTimeout ?? 2, batchSize, this.config.maxPollSeconds, @@ -67,30 +71,40 @@ export class StepTaskPoller this.logger.debug(`Found ${messages.length} messages, starting tasks`); - // Phase 2: Start tasks for the retrieved messages + // Phase 2: Start tasks for the retrieved messages. The claim receives + // this poller's queue name — the canonical lowercased flow slug, the + // exact spelling tasks store — and matches the persisted + // (queue_name, message_id) identity (#650). const msgIds = messages.map((msg) => msg.msg_id); const tasks = await this.adapter.startTasks( - this.config.queueName, + this.config.flowSlug, msgIds, - workerId + workerId, + queueName ); this.logger.debug( `Started ${tasks.length} tasks from ${messages.length} messages` ); - // Log if we got fewer tasks than messages (indicates some messages had no matching queued tasks) + // Messages without a claimable task are preserved: they recur after + // their visibility timeout until an operator handles them. Warn with + // identifiers only, never message bodies (#650). if (tasks.length < messages.length) { - this.logger.debug( - `Note: Started ${tasks.length} tasks from ${messages.length} messages. ` + - `${ - messages.length - tasks.length - } messages had no queued tasks (may retry later).` + const claimedIds = new Set(tasks.map((task) => task.msg_id)); + const unmatchedIds = messages + .filter((msg) => !claimedIds.has(msg.msg_id)) + .map((msg) => msg.msg_id); + this.logger.warn( + `Queue '${queueName}': ${unmatchedIds.length} of ${messages.length} message(s) ` + + `matched no claimable task for flow '${this.config.flowSlug}' ` + + `(msg_ids: ${unmatchedIds.join(', ')}). ` + + 'Messages are left for their visibility timeout; recurring ids need operator attention.' ); } // Create a map of message ID to message for quick lookup - const messageMap = new Map>>(); + const messageMap = new Map>>(); for (const msg of messages) { messageMap.set(msg.msg_id, msg as PgmqMessageRecord>); } diff --git a/pkgs/edge-worker/src/flow/createFlowWorker.ts b/pkgs/edge-worker/src/flow/createFlowWorker.ts index ceb68a135..860c94aec 100644 --- a/pkgs/edge-worker/src/flow/createFlowWorker.ts +++ b/pkgs/edge-worker/src/flow/createFlowWorker.ts @@ -95,8 +95,9 @@ export function createFlowWorker< // Create the pgflow adapter const pgflowAdapter = new PgflowSqlClient(sql); - // Use flow slug as queue name, or fallback to 'tasks' - const queueName = flow.slug || 'tasks'; + // Canonical queue identity is the normalized slug; the worker polls it + // directly (PGMQ message operations normalize names themselves) (#650) + const queueName = (flow.slug || 'tasks').toLowerCase(); logger.debug(`Using queue name: ${queueName}`); // Create specialized FlowWorkerLifecycle with the proxied queue and flow @@ -116,7 +117,8 @@ export function createFlowWorker< // Create StepTaskPoller with two-phase approach const pollerConfig: StepTaskPollerConfig = { batchSize: resolvedConfig.batchSize, - queueName: flow.slug, + flowSlug: flow.slug, + queueName, visibilityTimeout: resolvedConfig.visibilityTimeout, maxPollSeconds: resolvedConfig.maxPollSeconds, pollIntervalMs: resolvedConfig.pollIntervalMs, diff --git a/pkgs/edge-worker/src/queue/Queue.ts b/pkgs/edge-worker/src/queue/Queue.ts index 081e857c4..11afb2e73 100644 --- a/pkgs/edge-worker/src/queue/Queue.ts +++ b/pkgs/edge-worker/src/queue/Queue.ts @@ -42,7 +42,7 @@ export class Queue { `; } - async archive(msgId: number): Promise { + async archive(msgId: string): Promise { this.logger.debug( `Archiving message ${msgId} from queue '${this.queueName}'` ); @@ -51,7 +51,7 @@ export class Queue { `; } - async archiveBatch(msgIds: number[]): Promise { + async archiveBatch(msgIds: string[]): Promise { this.logger.debug( `Archiving ${msgIds.length} messages from queue '${this.queueName}'` ); @@ -99,7 +99,7 @@ export class Queue { * The only change made is now() replaced with clock_timestamp(). */ async setVt( - msgId: number, + msgId: string, vtOffsetSeconds: number ): Promise> { this.logger.debug( diff --git a/pkgs/edge-worker/tests/integration/flow/queueIdentity.test.ts b/pkgs/edge-worker/tests/integration/flow/queueIdentity.test.ts new file mode 100644 index 000000000..5f8dd172c --- /dev/null +++ b/pkgs/edge-worker/tests/integration/flow/queueIdentity.test.ts @@ -0,0 +1,112 @@ +import { assert, assertEquals } from '@std/assert'; +import { withPgNoTransaction } from '../../db.ts'; +import { Flow } from '@pgflow/dsl'; +import { startFlow, startWorker } from '../_helpers.ts'; +import { + waitForRunCompletion, + getStepTasks, + assertAllTasksCompleted, +} from './_testHelpers.ts'; + +// Queue identity (#650): a plain flow runs end-to-end through its stored +// queue identity, foreign messages in the same queue stay untouched while +// valid work continues, and a legacy mixed-case physical queue keeps working. +const QueueIdentityFlow = new Flow({ slug: 'test_queue_identity' }) + .step({ slug: 'doubleIt' }, (flowInput) => flowInput * 2); + +const LegacyQueueFlow = new Flow({ slug: 'TestLegacyQueue' }) + .step({ slug: 'doubleIt' }, (flowInput) => flowInput * 2); + +Deno.test( + 'flow executes through the stored queue identity; unmatched messages are preserved', + withPgNoTransaction(async (sql) => { + await sql`select pgflow_tests.reset_db();`; + + // Create the flow definition (worker startup will verify it) + await sql`select pgflow.create_flow('test_queue_identity');`; + await sql`select pgflow.add_step('test_queue_identity', 'doubleIt');`; + + // A foreign message with no matching task, sent directly into the queue + await sql` + select pgmq.send(queue_name => 'test_queue_identity', msg => '{"foreign": true}'::jsonb) + `; + + const worker = await startWorker(sql, QueueIdentityFlow, { + maxConcurrent: 1, + batchSize: 10, + maxPollSeconds: 1, + pollIntervalMs: 200, + }); + + try { + const flowRun = await startFlow(sql, QueueIdentityFlow, 21); + const polledRun = await waitForRunCompletion(sql, flowRun.run_id); + + assert(polledRun.status === 'completed', 'Run should be completed'); + + const stepTasks = await getStepTasks(sql, flowRun.run_id); + assertEquals(stepTasks.length, 1, 'Should have 1 step task'); + assertAllTasksCompleted(stepTasks); + + // The foreign message was never claimed, archived, or deleted + const [foreign] = await sql<{ count: string }[]>` + select count(*)::text as count from pgmq.q_test_queue_identity + where message->>'foreign' = 'true' + `; + assertEquals(foreign.count, '1', 'foreign message must remain in the queue'); + + // Tasks store the canonical queue snapshot + const [task] = await sql<{ queue_name: string }[]>` + select queue_name from pgflow.step_tasks where run_id = ${flowRun.run_id} + `; + assertEquals(task.queue_name, 'test_queue_identity'); + } finally { + await worker.stop(); + } + }) +); + +Deno.test( + 'flow worker runs a legacy mixed-case physical queue through the stored identity', + withPgNoTransaction(async (sql) => { + await sql`select pgflow_tests.reset_db();`; + + const flowSlug = 'TestLegacyQueue'; + + // Simulate a flow upgraded from 0.16.0: definition and queue exist, the + // queue keeps its original mixed-case spelling + await sql`select pgflow.create_flow(${flowSlug});`; + await sql`select pgflow.add_step(${flowSlug}, 'doubleIt');`; + await sql`select pgmq.drop_queue('testlegacyqueue');`; + await sql`select pgmq.create('TestLegacyQueue');`; + + const worker = await startWorker(sql, LegacyQueueFlow, { + maxConcurrent: 1, + batchSize: 10, + maxPollSeconds: 1, + pollIntervalMs: 200, + }); + + try { + const flowRun = await startFlow(sql, LegacyQueueFlow, 5); + const polledRun = await waitForRunCompletion(sql, flowRun.run_id); + + assert(polledRun.status === 'completed', 'Run should be completed'); + + // The message was dispatched, claimed, and archived on the physical queue + const [archiveCount] = await sql<{ count: string }[]>` + select count(*)::text as count from pgmq.a_TestLegacyQueue + `; + assertEquals(archiveCount.count, '1', 'message archived on the physical queue'); + + // No second metadata entry was created for the canonical name + const [listed] = await sql<{ count: string }[]>` + select count(*)::text as count from pgmq.list_queues() + where lower(queue_name) = 'testlegacyqueue' + `; + assertEquals(listed.count, '1', 'exactly one queue for the flow'); + } finally { + await worker.stop(); + } + }) +); diff --git a/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts b/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts index e42821887..7849bf312 100644 --- a/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts +++ b/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts @@ -22,7 +22,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ data: string }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -79,7 +79,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ data: string }> = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -113,7 +113,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ id: number; name: string }> = { - msg_id: 789, + msg_id: '789', read_ct: 2, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -146,7 +146,7 @@ Deno.test( // Verify rawMessage in context matches the original message assertExists(receivedRawMessage); - assertEquals(receivedRawMessage.msg_id, 789); + assertEquals(receivedRawMessage.msg_id, '789'); assertEquals(receivedRawMessage.read_ct, 2); assertEquals(receivedRawMessage.message, { id: 42, name: 'test item' }); }) @@ -156,7 +156,7 @@ Deno.test( 'MessageExecutor - Supabase clients are available when env vars exist', withTransaction(async (sql) => { const mockMessage: PgmqMessageRecord<{ test: string }> = { - msg_id: 999, + msg_id: '999', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', diff --git a/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts b/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts index 8f42da49a..64f8166c3 100644 --- a/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts +++ b/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts @@ -29,7 +29,7 @@ Deno.test( }; const mockMessage: PgmqMessageRecord<{test: string}> = { - msg_id: 123, + msg_id: '123', read_ct: 2, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', diff --git a/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts b/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts index 534c7b322..20f399f43 100644 --- a/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts +++ b/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts @@ -53,7 +53,7 @@ Deno.test( // Mock step task record - root steps get flow input directly const mockTask: StepTaskRecord = { flow_slug: 'context_test_flow', - msg_id: 123, + msg_id: '123', run_id: 'test-run-id', step_slug: 'test_step', task_index: 0, @@ -63,7 +63,7 @@ Deno.test( // Create context with mock task and message using proper flow worker context creation const mockMessage = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -75,7 +75,7 @@ Deno.test( sql: _sql, abortSignal: abortController.signal, taskWithMessage: { - msg_id: 123, + msg_id: '123', message: mockMessage, task: mockTask, flowInput: { data: 'test data' }, @@ -117,7 +117,7 @@ Deno.test( // Mock step task record - input is the unwrapped flowInput for root steps const mockTask: StepTaskRecord = { flow_slug: 'legacy_flow', - msg_id: 456, + msg_id: '456', run_id: 'legacy_run_id', step_slug: 'legacy_step', task_index: 0, @@ -130,7 +130,7 @@ Deno.test( // Create proper context for legacy handler test const mockMessage = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -142,7 +142,7 @@ Deno.test( sql: _sql, abortSignal: new AbortController().signal, taskWithMessage: { - msg_id: 456, + msg_id: '456', message: mockMessage, task: mockTask, flowInput: { value: 42 }, @@ -176,7 +176,7 @@ Deno.test( // Mock message - root steps get flow input directly (empty object for this flow) const mockMessage = { - msg_id: 789, + msg_id: '789', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -186,7 +186,7 @@ Deno.test( // Mock step task record - root steps get flow input directly const mockTask: StepTaskRecord = { flow_slug: 'rawmessage_flow', - msg_id: 789, + msg_id: '789', run_id: 'raw_run_id', step_slug: 'check_raw', task_index: 0, @@ -196,7 +196,7 @@ Deno.test( // Create context - for this test we need a mock taskWithMessage const mockTaskWithMessage = { - msg_id: 789, + msg_id: '789', message: mockMessage, task: mockTask, flowInput: {}, @@ -238,7 +238,7 @@ Deno.test( // Mock message - root steps get flow input directly (empty object for this flow) const mockMessage = { - msg_id: 999, + msg_id: '999', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -248,7 +248,7 @@ Deno.test( // Mock step task record const mockTask: StepTaskRecord = { flow_slug: 'supabase_flow', - msg_id: 999, + msg_id: '999', run_id: 'supabase_run_id', step_slug: 'check_clients', task_index: 0, @@ -258,7 +258,7 @@ Deno.test( // Create context with Supabase env vars const mockTaskWithMessage = { - msg_id: 999, + msg_id: '999', message: mockMessage, task: mockTask, flowInput: {}, @@ -322,7 +322,7 @@ Deno.test( // Create context - root steps get flow input directly const mockMessageForComplex = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -331,7 +331,7 @@ Deno.test( const mockTaskForComplex: StepTaskRecord = { flow_slug: 'complex_context_flow', - msg_id: 456, + msg_id: '456', run_id: 'complex_run', step_slug: 'fetch_data', task_index: 0, @@ -344,7 +344,7 @@ Deno.test( sql: _sql, abortSignal: abortController.signal, taskWithMessage: { - msg_id: 456, + msg_id: '456', message: mockMessageForComplex, task: mockTaskForComplex, flowInput: { id: 123 }, diff --git a/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts b/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts index 44cd94f91..b4d6a907f 100644 --- a/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts +++ b/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts @@ -103,6 +103,7 @@ Deno.test('StepTaskPoller caps limit at configured batchSize', async () => { new AbortController().signal, { batchSize: 5, + flowSlug: 'test_flow', queueName: 'test_flow', visibilityTimeout: 10, maxPollSeconds: 1, @@ -136,6 +137,7 @@ Deno.test('StepTaskPoller uses smaller available slot limit', async () => { new AbortController().signal, { batchSize: 5, + flowSlug: 'test_flow', queueName: 'test_flow', visibilityTimeout: 10, maxPollSeconds: 1, @@ -169,6 +171,7 @@ Deno.test('StepTaskPoller uses configured batchSize without limit', async () => new AbortController().signal, { batchSize: 5, + flowSlug: 'test_flow', queueName: 'test_flow', visibilityTimeout: 10, maxPollSeconds: 1, @@ -194,6 +197,7 @@ Deno.test('StepTaskPoller rethrows readMessages failures instead of returning an new AbortController().signal, { batchSize: 5, + flowSlug: 'test_flow', queueName: 'test_flow', visibilityTimeout: 10, maxPollSeconds: 1, @@ -205,3 +209,59 @@ Deno.test('StepTaskPoller rethrows readMessages failures instead of returning an await assertRejects(() => poller.poll(), Error, 'connection refused'); }); + +Deno.test('StepTaskPoller claims through the polled queue name and warns for unmatched messages', async () => { + const started: unknown[][] = []; + const warnings: string[] = []; + const messages = [ + { msg_id: '7', read_ct: 1, enqueued_at: '', vt: '', message: {} }, + { msg_id: '9223372036854775807', read_ct: 1, enqueued_at: '', vt: '', message: {} }, + ]; + const adapter = { + readMessages: (_queueName: string, _vt: number, _qty: number) => + Promise.resolve(messages), + startTasks: (...args: unknown[]) => { + started.push(args); + // Only the first message has a claimable task + return Promise.resolve([ + { flow_slug: 'f', run_id: 'r', step_slug: 's', task_index: 0, input: {}, msg_id: '7', flow_input: null }, + ]); + }, + }; + const logger = { + ...fakeLogger, + warn: (msg: string) => warnings.push(msg), + }; + + const poller = new StepTaskPoller( + adapter as never, + new AbortController().signal, + { + batchSize: 5, + flowSlug: 'TestFlow', + queueName: 'testflow', + visibilityTimeout: 2, + maxPollSeconds: 1, + pollIntervalMs: 100, + }, + () => 'worker-id', + logger + ); + + const tasks = await poller.poll(); + + // The claim received the expected flow, exact msg ids, and the polled queue + assertEquals(started, [[ + 'TestFlow', + ['7', '9223372036854775807'], + 'worker-id', + 'testflow', + ]]); + assertEquals(tasks.length, 1); + assertEquals(tasks[0].msg_id, '7'); + + // The unmatched message is reported with identifiers only + assertEquals(warnings.length, 1); + assertEquals(warnings[0].includes('9223372036854775807'), true); + assertEquals(warnings[0].includes("Queue 'testflow'"), true); +}); diff --git a/pkgs/edge-worker/tests/unit/contextUtils.test.ts b/pkgs/edge-worker/tests/unit/contextUtils.test.ts index 5ab5001fa..c66c7de94 100644 --- a/pkgs/edge-worker/tests/unit/contextUtils.test.ts +++ b/pkgs/edge-worker/tests/unit/contextUtils.test.ts @@ -29,7 +29,7 @@ const minimalEnv = { // Mock pgmq message record const mockMessage: PgmqMessageRecord<{ test: string }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -39,7 +39,7 @@ const mockMessage: PgmqMessageRecord<{ test: string }> = { // Mock pgmq message record with step input structure const mockStepMessage: PgmqMessageRecord<{ run: { test: string } }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -56,7 +56,7 @@ const mockStepTask = { run_id: 'run-456', step_slug: 'test-step', input: { run: { test: 'input' } }, - msg_id: 123, + msg_id: '123', flow_input: mockFlowInput, // Can be actual value or null - test helper wraps in Promise task_index: 0 } as unknown as StepTaskRecord; @@ -134,7 +134,7 @@ Deno.test('context - rawMessage is accessible', () => { sql: mockSql }); - assertEquals(context.rawMessage.msg_id, 123); + assertEquals(context.rawMessage.msg_id, '123'); assertEquals(context.rawMessage.message, { test: 'data' }); }); diff --git a/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts b/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts index 6dcc7f4bb..cedbe0e98 100644 --- a/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts +++ b/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts @@ -31,7 +31,7 @@ Deno.test('createContextSafeConfig excludes sql field and freezes result', () => Deno.test('Queue worker context includes workerConfig for GitHub issue use case', async () => { const mockMessage: PgmqMessageRecord<{test: string}> = { - msg_id: 123, + msg_id: '123', read_ct: 2, // Current retry attempt enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -91,7 +91,7 @@ Deno.test('Queue worker config immutability prevents handler modifications', asy const context = { env: {}, shutdownSignal: new AbortController().signal, - rawMessage: { msg_id: 1, read_ct: 1, message: {} }, + rawMessage: { msg_id: '1', read_ct: 1, message: {} }, workerConfig: createContextSafeConfig(mockConfig), }; @@ -120,8 +120,8 @@ Deno.test('Flow worker context includes workerConfig', async () => { const context = { env: {}, shutdownSignal: new AbortController().signal, - rawMessage: { msg_id: 456, read_ct: 1, message: {} }, - stepTask: { flow_slug: 'test', step_slug: 'step', msg_id: 456, run_id: 'run', input: {} }, + rawMessage: { msg_id: '456', read_ct: 1, message: {} }, + stepTask: { flow_slug: 'test', step_slug: 'step', msg_id: '456', run_id: 'run', input: {} }, workerConfig: createContextSafeConfig(mockConfig), }; diff --git a/pkgs/example-flows/src/example-flow.ts b/pkgs/example-flows/src/example-flow.ts index a524d34d1..eb7bed5ee 100644 --- a/pkgs/example-flows/src/example-flow.ts +++ b/pkgs/example-flows/src/example-flow.ts @@ -39,7 +39,7 @@ export const stepTaskRecord: StepTaskRecord = { // thirdStep: { finalValue: 23 }, --- this should be an error // normalStep: { doubledValueArray: [1, 2, 3] }, --- this should be an error }, - msg_id: 1, + msg_id: '1', flow_input: { value: 23 }, }; diff --git a/pkgs/website/src/content/docs/concepts/data-model.mdx b/pkgs/website/src/content/docs/concepts/data-model.mdx index 44393eb17..e79a12718 100644 --- a/pkgs/website/src/content/docs/concepts/data-model.mdx +++ b/pkgs/website/src/content/docs/concepts/data-model.mdx @@ -22,13 +22,13 @@ The schema uses composite primary and foreign keys that include denormalized slu ```sql -- Definition tables flows: (flow_slug) -steps: (flow_slug, step_slug) +steps: (flow_slug, step_slug) + denormalized queue_name deps: (flow_slug, dep_slug, step_slug) -- Runtime tables runs: (run_id) + denormalized flow_slug step_states: (run_id, step_slug) + denormalized flow_slug -step_tasks: (run_id, step_slug, task_index) + denormalized flow_slug +step_tasks: (run_id, step_slug, task_index) + denormalized flow_slug, queue_name ``` Denormalized slugs simplify queries - no joins needed: @@ -39,6 +39,10 @@ SELECT * FROM pgflow.step_tasks WHERE flow_slug = 'my_flow' AND step_slug = 'my_step'; ``` +### 📮 Queue identity + +PGMQ message ids are queue-scoped, so a task's message identity is `(queue_name, message_id)`, not `message_id` alone. Every step stores its canonical queue name (`steps.queue_name`, always the lowercase `lower(flow_slug)` today), and every task snapshots that value at creation (`step_tasks.queue_name`). Runtime code never rewrites the snapshot: dispatch, claims, visibility, completion, retries, skip/cancel cascades, late callbacks, stalled recovery, and pruning all address the queue through it. `(queue_name, message_id)` is unique per queue, while `message_id` alone stays nullable. pgflow manages its queues exclusively - do not send to them or create, replace, or alter them from application code. + ## Two Categories of Tables The schema organizes tables into two distinct categories that serve different purposes in the flow lifecycle. All tables live in the `pgflow` schema. @@ -81,6 +85,7 @@ These tables track the execution state of flow instances: - Single steps create 1 task, map steps create N tasks - Each task has retry counter and attempts tracking - Contains `task_index` for map task array elements +- Stores `queue_name`, the queue snapshot the task's message was dispatched to; the `(queue_name, message_id)` pair identifies the message and is unique per queue - Tracks task status (`queued`, `started`, `completed`, `failed`, `skipped`, `cancelled`) - `skipped` marks the logical orchestration state: the parent step was skipped, so the task will never run; an already-running handler is not forcibly terminated - `cancelled` marks unfinished tasks whose run failed: the task that caused the failure stays `failed` (with its error), completed work stays `completed`, and every remaining queued or started task becomes `cancelled`. `runs.failed_at` is the cancellation time; there is no separate `cancelled_at` column diff --git a/pkgs/website/src/content/docs/deploy/prune-records.mdx b/pkgs/website/src/content/docs/deploy/prune-records.mdx index 37c16a860..8e07d3b16 100644 --- a/pkgs/website/src/content/docs/deploy/prune-records.mdx +++ b/pkgs/website/src/content/docs/deploy/prune-records.mdx @@ -31,7 +31,7 @@ When a run (completed or failed) exceeds the retention period, **ALL** associate **Deleted:** - Run records (`pgflow.runs`) - Step states and tasks - all statuses (`pgflow.step_states`, `pgflow.step_tasks`) -- PGMQ messages - active queue (`pgmq.q_{flow_slug}`) and archived (`pgmq.a_{flow_slug}`) +- PGMQ messages - active queues addressed through each task's stored queue name (`step_tasks.queue_name`) and archive tables through persisted queue routes (`pgflow.steps.queue_name`) - Inactive workers (based on `last_heartbeat_at`) **Preserved:** @@ -72,6 +72,10 @@ Use pg_cron to schedule automated pruning. **Run during low-traffic periods** as This function is not yet included in default pgflow migrations. Install it manually by running the [pruning function SQL](#the-pruning-function) using psql or Supabase Studio. + + ### 2. Setup pg_cron schedule Run it in Supabase Studio or include in a migration file: diff --git a/pkgs/website/src/content/docs/deploy/update-pgflow.mdx b/pkgs/website/src/content/docs/deploy/update-pgflow.mdx index 9ab211264..bcb4c9507 100644 --- a/pkgs/website/src/content/docs/deploy/update-pgflow.mdx +++ b/pkgs/website/src/content/docs/deploy/update-pgflow.mdx @@ -60,7 +60,7 @@ import { EdgeWorker } from "jsr:@pgflow/edge-worker@^0.6.0"; ### 3. Run pgflow install to update migrations -**Always run the install command after updating packages** to check for and apply new database migrations: +**Always run the install command after updating packages** to check for new database migrations and copy them into your project (the Supabase migration runner, not the installer, applies them): ```bash frame="none" npx pgflow@latest install @@ -99,12 +99,18 @@ pgflow is tested on PostgreSQL 17 for compatibility. ### 5. Apply new migrations + + Apply the new migrations to your database: ```bash frame="none" npx supabase migrations up ``` +By default this applies to your **local** development database. For a linked Supabase project, use `npx supabase migrations up --linked`; for any other database, pass an explicit percent-encoded connection string with `--db-url`. + ## Understanding Migration Timestamps pgflow uses a simple but effective migration system to ensure migrations are applied correctly: @@ -130,6 +136,66 @@ Where: The original timestamp allows pgflow to search for and detect which migrations have already been installed, while the new timestamp prefix ensures Supabase can apply the migration without timestamp ordering errors. +## Persist queue identity (0.17.0) + +pgflow 0.17.0 stores each step's and task's physical queue identity in the database: `pgflow.steps.queue_name` and `pgflow.step_tasks.queue_name`, always the canonical lowercase `lower(flow_slug)` for now. A task's message identity is `(queue_name, message_id)` instead of `message_id` alone, so the same message id in two queues can never collide. Queues created by older releases keep their original mixed-case spelling in pgmq and continue to work through it: PGMQ's public message API normalizes names, and pgflow resolves the original listed spelling only where PGMQ deletes physical objects by exact name (queue deletion); the pruning helper's archive walk derives table names from the canonical route directly. No queues or messages are migrated or recreated, flow slug casing is untouched, and the existing queue metadata spelling is preserved. + + + +This is a maintenance upgrade from 0.16.0. Update the pgflow packages and run `npx pgflow@latest install` **before** the maintenance window below, so the new migration files are already in your project - install only copies migration files, it writes no flow definitions. Then follow the steps in order: + + + +1. ### Pause new producers + + Stop new `pgflow.start_flow()` calls (application code, webhooks, cron) so no new runs and messages appear during the upgrade. Runs already in the database are kept as they are; you do not need to empty any queue. + +2. ### Record the pre-upgrade worker snapshot + + Save the state you restore later, before anything is disabled, deprecated, or stopped: + + - Edge Function workers: record each row's current value with `select function_name, enabled from pgflow.worker_functions;`. You restore exactly these values in step 9, including rows that were already `false`. + - Process workers on Node or Bun: note which service-manager units are actually running. You restart only those units in step 9. + +3. ### Drain in-flight work while the old schema is present + + Gracefully stop each worker so handlers finish under the old schema: disable automatic restarts first, then deprecate the workers, and wait until every claimed task leaves `started` - the exact sequence and drain queries are in [Update Deployed Flows](/deploy/supabase/update-deployed-flows/) (its first step records each function's `enabled` value: the same pre-disable values you saved in step 2). What you are waiting for is in-flight handlers and their transactions to finish, **not** for queues to become empty: queued tasks and their messages are retained by the migration and are claimed by the new workers afterwards. `worker.stop()` is a graceful-shutdown signal that finishes in-flight tasks; it does not empty any queue. + +4. ### Stop every worker, including re-invocation + + After the drain, make sure nothing can start a worker again before the migration runs: + + - Edge Function workers: keep every `pgflow.worker_functions` row `enabled = false` (the drain in the previous step disabled them; their pre-upgrade values are saved in your step 2 snapshot) so the `ensure_workers()` cron does not re-invoke the function, and stop any external pings or direct HTTP requests to the function (a direct request starts a worker even while the row is disabled). + - Process workers on Node or Bun: stop the processes (the units you noted in step 2 are the ones that were running), and make sure your service manager does not restart them. + +5. ### Quiesce maintenance, recovery, and definition writers + + Stop the writers that touch pgflow state besides workers: the stalled-task recovery (`requeue_stalled_tasks` cron), the optional pruning helper's pg_cron job, and flow definition writes (worker startups, `create_flow`/`add_step`). No pgflow code may run against the database between here and the matching deployment. + +6. ### Apply the migration transactionally + + Apply the new migrations through Supabase's migration runner while everything is stopped: `npx supabase migrations up --linked` for a linked production project, `npx supabase migrations up --db-url ""` for an explicit database, or `npx supabase migrations up --local` for a local project - the CLI applies to the **local** database by default, so a bare `migrations up` leaves a linked production database untouched. Your project's migration pipeline is equally fine. The migration runs in a single transaction with bounded lock waits: if your database contains conflicting data (two flows whose slugs differ only by case, duplicate `(queue_name, message_id)` pairs, or queue names longer than PGMQ's limit), it **fails and leaves the database unchanged** - fix the conflict manually (pick one spelling, deduplicate, or rename before migrating; pgflow never renames or deletes definitions for you) and re-run it. A failed migration rolls back completely; after a **successful** migration, old workers cannot simply restart - the three-argument claim signature no longer exists. + +7. ### Replace the pruning helper if installed + + If you installed the optional `pgflow.prune_data_older_than()` helper, replace it with the current version from [Prune Old Data](/deploy/prune-records/). If you customized your copy, adapt it yourself: message cleanup now goes through each task's stored `queue_name` snapshot, and archive-table cleanup derives each table name from the persisted route through `pgmq.format_table_name()` (which lowercases the name itself - no queue listing is consulted). + +8. ### Deploy matching packages and workers + + Update the whole pgflow package set to the same version - npm packages, the JSR edge-worker import, and any custom code that calls `start_tasks()`/`startTasks()` explicitly (it must now pass the queue's canonical name: `lower(flow_slug)`) - and deploy the new worker code. + +9. ### Resume in the safe order + + Re-enable maintenance jobs and the pruning cron, then restore each `pgflow.worker_functions` row to the exact value you recorded in step 2 - set `enabled = true` only for rows that were `true` before the upgrade; rows you had intentionally disabled stay `false`. Restart only the process-worker units you noted as running in step 2. Then resume your producers of new runs. Queued tasks from before the upgrade are claimed and processed by the new workers. + + + + + ## Remove manual flow compilation pgflow 0.16.0 removed `compileFlow()`, ControlPlane, `pgflow compile`, and the `FlowWorkerConfig.compilation` option, and replaced `pgflow.ensure_flow_compiled(text, jsonb, boolean)` with a two-argument startup-only signature. Old workers cannot start after the database migration. @@ -204,6 +270,8 @@ npx pgflow@latest install -y # Auto-confirm for faster development npx supabase migrations up ``` +If this update crosses 0.16.0 to 0.17.0, do not apply the migration here: follow the [Persist queue identity (0.17.0)](#persist-queue-identity-0170) maintenance sequence, even for a local database, with workers stopped. + ### Production Environment For production updates, follow a more careful approach: diff --git a/pkgs/website/src/content/docs/get-started/faq.mdx b/pkgs/website/src/content/docs/get-started/faq.mdx index 5b5a52953..2c93390b2 100644 --- a/pkgs/website/src/content/docs/get-started/faq.mdx +++ b/pkgs/website/src/content/docs/get-started/faq.mdx @@ -28,7 +28,7 @@ import { MyFlow } from '../../flows/my_flow.ts'; EdgeWorker.start(MyFlow); ``` -In this mode, Edge Worker processes tasks for a specific workflow, handling step dependencies, data flow between steps, and automatic coordination. The queue name matches your flow slug. +In this mode, Edge Worker processes tasks for a specific workflow, handling step dependencies, data flow between steps, and automatic coordination. The queue name matches your flow slug, lowercased.
Read more about choosing the right mode @@ -37,7 +37,7 @@ In this mode, Edge Worker processes tasks for a specific workflow, handling step **Flows:** Multi-step processes with dependencies, data flow between steps, parallel execution (AI pipelines, data transformations, business workflows) -**Configuration:** Both modes share the same Edge Worker options. Background jobs use `queueName` option (default: `tasks`), flows use the flow slug as queue name. +**Configuration:** Both modes share the same Edge Worker options. Background jobs use `queueName` option (default: `tasks`), flows use the lowercased flow slug as queue name. **🤔 Hard to decide?** Consider flows if your "single step" has multiple distinct operations that could fail independently, or you might add steps later. Handler functions are reusable between modes - start with background jobs and migrate to flows when needed. @@ -61,7 +61,7 @@ Actual costs depend on message volume, message size, and worker stability. Monit Each Edge Worker instance is bound to a single queue: - **Background jobs mode:** Queue name set via `queueName` configuration (default: `tasks`) -- **Flow mode:** Queue name automatically matches the flow slug +- **Flow mode:** Queue name automatically matches the lowercased flow slug **Scaling options:** - **Multiple workers on same queue:** Deploy multiple Edge Workers pointing to the same queue for increased throughput diff --git a/pkgs/website/src/content/docs/news/pgflow-0-17-0-persistent-queue-identity.mdx b/pkgs/website/src/content/docs/news/pgflow-0-17-0-persistent-queue-identity.mdx new file mode 100644 index 000000000..7dccea6b7 --- /dev/null +++ b/pkgs/website/src/content/docs/news/pgflow-0-17-0-persistent-queue-identity.mdx @@ -0,0 +1,37 @@ +--- +title: 'pgflow 0.17.0: Persistent Queue Identity' +description: 'Tasks now store the physical queue they were dispatched to, and message identity is (queue_name, message_id). start_tasks() requires the canonical queue argument - a breaking low-level change with a stop-the-world maintenance upgrade.' +date: 2026-09-13 +authors: + - jumski +tags: + - release + - breaking +featured: false +--- + +import { Aside } from '@astrojs/starlight/components'; + +pgflow now persists the physical queue identity of every step and task. `pgflow.steps` and `pgflow.step_tasks` gain a `queue_name` column that stores the canonical lowercase queue (`lower(flow_slug)` today), snapshotted when the task is created and never rewritten. A queued task's message identity is `(queue_name, message_id)` instead of `message_id` alone - PGMQ message ids are queue-scoped, so the same id in two different queues can no longer be mistaken for the same message. This is the foundation for private per-step queues; one-flow/one-queue behavior is unchanged. + +Read the [data model](/concepts/data-model/) for what is stored, and the [update guide](/deploy/update-pgflow/) for the full 0.16.0 → 0.17.0 procedure. + + + +## What changes + +- **Queue identity snapshots.** Every step records its default route in `steps.queue_name`; every task copies it at creation (`step_tasks.queue_name`). Dispatch, claims, visibility, completion, retries, skip/cancel cascades, late callbacks, stalled recovery, and pruning all address the queue through the stored snapshot. +- **Queue-scoped message identity.** `(queue_name, message_id)` is unique per queue where `message_id` is set; `message_id` alone stays nullable. Message ids cross the JavaScript boundary as exact decimal strings (they can exceed the safe integer range). +- **Provisioning safeguards.** Two flows can no longer address the same normalized default queue: a listed PGMQ queue with a colliding normalized name is rejected before a new flow is created, and `pgflow.flows` is unique on `lower(flow_slug)` - including direct SQL creation. +- **Legacy mixed-case queues keep working.** Queues created by earlier releases keep their original pgmq spelling. PGMQ's public message API normalizes names, so pgflow addresses them by the stored canonical name directly - no per-message queue listing. Only queue deletion resolves the original spelling through `pgmq.list_queues()` (PGMQ drops queue metadata by exact listed name), rejecting an ambiguous match before any destructive work; the pruning helper's archive walk derives table names from the canonical route (`pgmq.format_table_name()` lowercases it). +- **Optional pruning helper updated.** The manually installed `pgflow.prune_data_older_than()` snippet cleans messages through task snapshots and archive tables through persisted routes. Replace an installed copy; adapt customized copies yourself. + +## Maintenance upgrade from 0.16.0 + +The ordered procedure with the exact SQL lives in the [update guide](/deploy/update-pgflow/#persist-queue-identity-0170). In short: update packages and copy migrations in advance, pause new producers, gracefully drain in-flight work while the old schema is present (waiting for running handlers to finish - not for queues to empty; queued tasks and messages are retained), stop every worker including HTTP re-invocation and long-running processes, quiesce maintenance/recovery/definition writers, apply the transactional migration through Supabase's migration runner (against the production database - `--linked` or `--db-url`, not the local default), replace the pruning helper if installed, deploy the matching package set, then resume maintenance, workers, and producers in that order - restoring the exact worker `enabled` states recorded before the window. + +The migration backfills existing steps and tasks with `lower(flow_slug)` - including tasks whose `message_id` is NULL - and fails atomically on conflicting data (flows differing only by slug case, duplicate `(queue_name, message_id)` pairs, queue names beyond PGMQ's 47-character limit). A failed migration leaves the database unchanged; resolve conflicts manually and re-run it. After a successful migration, old workers cannot simply restart. No queues or messages are migrated or recreated, and flow slug casing is untouched. diff --git a/pkgs/website/src/content/docs/reference/configuration/worker.mdx b/pkgs/website/src/content/docs/reference/configuration/worker.mdx index f5de34087..41149c046 100644 --- a/pkgs/website/src/content/docs/reference/configuration/worker.mdx +++ b/pkgs/website/src/content/docs/reference/configuration/worker.mdx @@ -181,7 +181,7 @@ my-worker: ↻ retry 1/3 in 5s When using Edge Worker in [Background Jobs Mode](/get-started/faq/#what-are-the-two-edge-worker-modes) (without pgflow orchestration), there are a few key differences: - **No flow/step configuration**: Queue workers don't have `maxAttempts`, `baseDelay`, `timeout`, or `startDelay` options in the flow definition. Instead, these are configured directly in the worker options. -- **`queueName` option**: Queue workers can specify a custom queue name (default is `tasks`), while flow workers automatically use the flow slug as the queue name. +- **`queueName` option**: Queue workers can specify a custom queue name (default is `tasks`), while flow workers automatically use the lowercased flow slug as the queue name. pgflow manages its flow-worker queues: do not send to them or alter them from application code. Background-job queues are yours; enqueue jobs with `pgmq.send()` as shown in [Create a Worker](/get-started/background-jobs/create-worker/). - **Handler signature**: Queue workers receive a simple payload, while flow workers receive a context object with `input`, `run`, and previous step outputs. For complete queue worker configuration, see [Queue Worker Configuration](/reference/queue-worker/configuration/). diff --git a/pkgs/website/src/content/docs/reference/context.mdx b/pkgs/website/src/content/docs/reference/context.mdx index cb36d0249..a98791a79 100644 --- a/pkgs/website/src/content/docs/reference/context.mdx +++ b/pkgs/website/src/content/docs/reference/context.mdx @@ -57,7 +57,7 @@ The original message from the pgmq queue, containing metadata like message ID, r ```typescript interface PgmqMessageRecord { - msg_id: number; // Unique message ID from pgmq + msg_id: string; // Queue-scoped exact decimal message ID from pgmq read_ct: number; // How many times this message has been read enqueued_at: string; // ISO timestamp when message was enqueued vt: string; // ISO timestamp for visibility timeout @@ -86,7 +86,7 @@ interface StepTaskRecord { step_slug: string; // Slug identifier of the current step task_index: number; // Task index (0 for single steps, 0..N-1 for map steps) input: StepInput; // Typed input for this specific step (inferred from flow) - msg_id: number; // pgmq message ID + msg_id: string; // Queue-scoped exact decimal pgmq message ID } ```