Skip to content

Preserve transient database failures during readiness schema inspection - #165

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/operator-schema-readiness
Sep 21, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
fix/operator-schema-readiness

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 21, 2026

Copy link
Copy Markdown
Member

Problem

Addresses #164 and durable-workflow/sdk-php#70. A database disconnect during Workflow operator-table inspection became a missing-table diagnosis, producing workflow_v2_blocked instead of the existing retryable backend_unavailable response.

Change

  • Use strict schema inspection from published Workflow 2.0.15, preserving the original database exception for the existing Server classifier.
  • Keep genuinely missing tables and permanent database/configuration errors blocked; do not make every HTTP 503 retryable.
  • Cover heartbeat plus workflow/activity/query/update-validation polling with transient PDO 2002/2006/2013 and permanent 1045/1049 failures. Verify retry identity, no false heartbeat acknowledgement, no private diagnostics, and success after recovery.
  • Pin the published dependency and prepare Server 2.3.13 / Helm 0.1.92 through the existing source-release generator. No wire-format or database migration change.

Verification

The first commit reproduces the failure on Workflow 2.0.14. With published Workflow 2.0.15 from Packagist and no source overlay: WorkerDatabaseUnavailableTest, WorkflowBootstrapRouteGatingTest and HealthControllerTest pass (72 tests / 3,486 assertions, PHP 8.4.25). Pint and source-release synchronization checks pass; Composer reports no security advisories and only its existing exact-version warning.

Published PHP SDK 2.0.11 separately passes its recovery contract tests (14 tests / 42 assertions), recorded on sdk-php#70. No SDK version bump is needed for this Server classification fix.

Release Follow-Through

Workflow 2.0.15 is published. After Server CI and merge, publish the immutable Server image and Helm chart, then run a bounded disposable published-image worker recovery check. Keep #164 and sdk-php#70 open until that verification is complete. Default embedded Waterline behavior is unchanged; no Waterline or SDK release is required merely to align versions. Sample App and other image consumers must resolve the new published Server artifact rather than assume a library release updated their image.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Root fix is durable-workflow/workflow#516: opt-in strict operator-schema inspection propagates the original exception while preserving embedded Waterline diagnostics by default. A one-line ServerReadiness adoption plus25 heartbeat/poll error-classification cases is implemented locally. With that exact candidate source mounted over published Workflow2.0.14, the focused readiness/bootstrap/worker suite passes72 tests /3,486 assertions. The original unpatched regression failed as recorded.

The25 cases distinguish PDO2002/2006/2013 from permanent1045/1049 across heartbeat and all four poll endpoints, check that private diagnostics do not leak, preserve poll identity, and verify recovery after removing the fault. Existing genuinely-missing-table/configuration tests also pass.

Next on this PR: adopt the published Workflow patch in composer.json/lock, push the Server change and expanded tests together, run normal CI and publish Server. The source-mounted result is not released-artifact proof. This is active maintainer work, not a request for human review or new approval.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 21, 2026 19:39
@rmcdaniel rmcdaniel changed the title Preserve transient schema failures in worker readiness responses Preserve transient database failures during readiness schema inspection Sep 21, 2026
@rmcdaniel
rmcdaniel merged commit 6bbeaaf into main Sep 21, 2026
14 checks passed
@rmcdaniel
rmcdaniel deleted the fix/operator-schema-readiness branch September 21, 2026 19:53
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged at 6bbeaaf after every PR check passed. The merge tree is identical to the checked PR head. Immutable Server 2.3.13 publication is running: https://github.com/durable-workflow/server/actions/runs/35647725858. Next: verify the published image with an uninterrupted PHP worker through database stop/recovery, then verify the corresponding published Python/Rust retry behavior. Issues remain open until that follow-through is complete.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Follow-through: all post-merge source workflows passed, including feature, replay/query, rolling database upgrade, Helm and perf qualification. Release run 35647725858 remains on its original multi-architecture image build; no restart or replacement run was started. The bounded published-image recovery check is prepared for PHP SDK 2.0.11, Python 2.0.4 and Rust 2.0.6. Rust fresh-install failure discovered during setup was independently fixed and published (sdk-rust#42/#43).

Sample App currently pins the previous Server/Workflow tuple. Its two-file downstream update is pushed on fix/published-schema-recovery (commit 86e7b45), with Composer artifact-graph validation passing and no unrelated dependency changes. Open its PR once Server 2.3.13 is pullable so normal sample CI uses the real published image. Keep #164 and sdk-php#70 open until live disposable worker interruption/recovery outcomes are recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants