Deploy each service independently in static environments

[vmelikyan]

Problem

Lifecycle reconciles an environment with one job for the whole environment. One generation counter and one run token control all services. A merge to service B raises the counter. The run for service A reads the new counter and loses its authority: its build, its after-build step and its rollout are discarded, and the next pass must do that work again. In a static environment with hundreds of services and a steady flow of merges, the pass restarts before the slowest services land. Those services do not deploy for a long time.

The same design marks work as complete when the work did not run. When the configuration of one service is invalid, the job records the error for the environment and marks the whole generation complete. Every merge that the pass had collected is consumed. Service A stays at its old version and no job returns to it.

Solution

Each deploys row carries its own desired generation and observed generation. Each accepted change is one intent, and each intent is one job. The job claims only the rows that belong to its intent. A run owns a row only when three conditions are true: the row carries the run token, the row is at the run generation, and the environment is not blocked. The rule does not refer to other services.

• A merge to one repository stamps only the rows of that repository. Runs for other services continue and finish.
• A new merge to the same repository replaces the older intent for that repository. Its rows drop their run token and read as queued until the new run claims them. The older run can no longer write to them, stops at its next check, and does not publish its result.
• A change to the configuration repository, or a manual redeploy, stamps all rows. It replaces older environment-wide intents and marks them finished. The accepted SHA of each merge stays as a floor: the full pass deploys each service at that SHA, or at a newer head that GitHub confirms as a descendant. The root configuration is read at its floor too.
• Prerequisites order services inside one run only. A service launches as soon as the prerequisites of its own run are ready, so a slow service never holds back unrelated work. A service never waits for, and never fails because of, a run of another service. The running version of a prerequisite in another run serves until that run replaces it.
• A service with an invalid lifecycle.yaml fails alone, with the parse error on its row. Its dependents are not deployed. All other services continue. A failed service that has no row yet is kept on the intent and shown in the environment status as "Not imported" until a later import clears it.
• A root configuration error is recorded on the environment as a configuration error by any intent, also by a push to one service. The intent takes its rows so a superseded run cannot write to them. The error stays until a later import of the root configuration succeeds.
• The environment status is derived from the rows: building or deploying while any row is pending, error when a row failed or a service could not be imported, else deployed.
• An intent marks itself finished and recomputes the environment watermark in one transaction that holds the Build row lock, so intents that finish at the same time cannot leave the watermark behind. The sweep repairs a stale watermark instead of signalling a finished generation. A claim adopts rows only for its own intent.
• Every renewable deployment lock, for the generation, the configuration step and each service's rollout, expires after 60 seconds and is renewed every 20 seconds. A worker that stops during a run frees its locks within one minute, and the queue retry resumes the rows the run already owns.
• Before a service is applied, the run waits until each earlier deploy job of that service has stopped. A job has stopped when it is terminal and the kubelet reports each of its pods as stopped. This covers apply jobs and native Helm jobs, which outlive the worker that created them. The wait is limited to 45 minutes; then the service fails with a message that asks for a redeploy.
• When a launch fails outright, for example because a promotion lease was lost, the run lets every launched service finish and launches nothing more. Only then does it raise the error for the queue retry, so the retry never overlaps work still in flight.
• A service that did not deploy is never marked done. When the ownership check around a launch fails, the service is recorded as deploy-failed. When the ownership check after a failed rollout fails, the job goes back to the queue.
• A teardown closes the work accepted before it started, in one transaction that holds the Build row lock: those intents are marked finished and their pending rows observed. A deploy request accepted during the teardown, such as a re-added deploy label, keeps its generation and runs after the teardown. The teardown also uninstalls the Helm release of a row that was still queued.
• An Aurora restore keeps a pinned endpoint. For an endpoint Lifecycle created, it checks AWS first: a database that is confirmed gone, as after a teardown that kept the row, is restored again; a check that fails keeps the endpoint the row has instead of restoring over a live database.
• Each Codefresh submission gets an ownership check and its own YAML file. The ingress is applied once per intent after its rollout.

Schema: migration 037 adds desiredGeneration and observedGeneration to deploys and a partial index on pending rows. It marks each accepted intent at or below the old environment watermark as finished, so the new worker does not run intents the old worker already completed. There is no new table. The migration is safe to run again.

Upgrade

Scale the worker deployment to zero before the rollout and up again after the new version starts. An old worker takes a promotion lock for the full environment and writes the watermark with the old rules, so it must not run on the new schema. Old web pods are safe during their own rollout: their accepts do not stamp rows, and the new worker adopts the rows of a claimed intent at claim time. The worst case is one extra full pass for an environment-wide intent accepted in that window.

Verification

Automated

• pnpm test:coverage, the CI gate: 502 suites, 9811 tests pass (11 skipped). Coverage is 98.10% statements, 91.78% branches and 98.12% functions. The thresholds are 98, 91 and 98. Every statement this change adds is covered, except three log lines.
• serviceReconciliation.test.ts runs the real reconciliation path on a simulated clock. Only the image build, the apply, the YAML import and the queue are simulated. The queue has worker slots and keeps one job per id, as BullMQ does. Transactions run one at a time, as the Build row lock makes them. The scenarios include: a merge to B while A builds, a newer merge to A, a config-repo push during a run, a service config error, a root config error on a service push, a transient failure with retry, a lost signal recovered by the sweep, a retry after a successful rollout, a service that moves to another repository, a config error that lands during a rollout, and dependents that build while their prerequisite runs elsewhere with only two worker slots.
• deploymentManager.test.ts covers per-service ownership checks, the wait for an earlier deploy job and its 45-minute limit, a failed ownership check before a launch, scheduling by prerequisite inside a run, the drain after a lost lease, supersession at admission and the Codefresh submission gate.
• Direct tests cover the authority predicates at row and environment level, the mailbox, the apply-job check, the teardown close and its cutoff, the Aurora endpoint check, a broken service YAML in the real import, the derived environment status, and the ingress job clearing a stale failure note only after a clean apply. The queue's failure policy is tested case by case: a configuration error that escapes the configuration step finishes the intent, an error after supersession is ignored without a retry, a transient error retries until the final attempt, a lost lock rethrows, and an environment-wide claim that fails leaves its generation pending.
• Eleven targeted mutations of the invariants were run against the full suite: the generation in the row fence, the observed check in ownership, the row stamping on accept, the finishing of older environment-wide entries, unfinished entries in the watermark, the no-wait rule for prerequisites outside the run, the running-pod check for earlier jobs, the teardown cutoff, the generation filter on claim, the block-reason check in the environment predicate, and the pending-before-failed status order. Each one fails at least one test.
• tsc --project tsconfig.server.json --noEmit, ESLint and Prettier: clean.
• The migration SQL was run against a copy of the local database state and on the populated local database with a rollback and a re-run.

Live test on the local stack

Stack: mise prd (kind, Tilt, buildkit, local registry), webhooks through ngrok. Static environment ancient-glitter-149521 from vmelikyan/lc-test-2 branch static. It tracks lc-test-gh-type (lc-test), lc-test-3 (lc-test-3), lc-test-4 (lc-test-4) and a Helm service on main. Two Dockerfiles have a 120-second step, so builds overlap.

Scenario	Result
Push to one service	Only that row moves to a new generation. Three seconds after the push the row reads queued with no run token; its job claims it two seconds later.
Push to B while A builds	B is deployed first. The run of A continues and lands on its own SHA.
Two pushes to A in quick succession	Both older builds complete, but their results are not published. Only the newest SHA is deployed.
Push to the configuration repository	One environment-wide generation claims all rows at the accepted SHAs. Live image tags match.
Invalid lifecycle.yaml in B, then a fix	Only the row of B fails, with the parse error. A and C are not touched. The fix push recovers B.
Invalid root configuration, then a push to one service, then a fix	The environment is CONFIG_ERROR after the broken push. The service push keeps CONFIG_ERROR and does not mark its own row as failed. The fix push runs a full pass and the environment is DEPLOYED 108 seconds later.
Manual redeploy from the UI while A builds	The full pass takes the row of A and deploys A at the pushed SHA through the floor.
Configuration push that adds a Helm service with a dependency	The import creates the row. The Helm service deploys through native Helm after its prerequisite.
Push to Y, then to its dependent C while Y still builds	C does not wait for the run of Y. C deploys and finishes first; the run of Y continues and lands Y.
An earlier apply job of Y still running when the new apply of Y starts	The row reads "Waiting for the previous deploy to finish". The new apply job is created one second after the earlier job completes.
Worker pod killed during a build	The queue retries the job. The retry resumes under the same run token after the generation lock of the dead worker expires, and rolls out once.
Worker pod killed while the apply job of Y is being created	The retry resumes under the same run token 75 seconds after the kill. The environment settles 103 seconds after it.
Pull-request environment: open, push, invalid root config, fix, failed image build, fix, close during a run	Each step gives the expected status. The close tears the environment down and rejects the late write of the active run.
Pull-request environment: push to the branch, then close the pull request while the run builds	The environment is TORN_DOWN 42 seconds after the close, with no pending row and no open intent, and stays so. The namespace is removed.
Migration 037 on the populated local database	After a rollback and a re-run, every row reads generation 0, an intent planted as open at the old watermark reads finished, and no run is replayed. A push after the migration is accepted as the next generation and deploys; the environment settles at that generation.

After every scenario the environment reported the expected final status with all generations observed, and the sweep sent no duplicate or stale signals.

Not tested live

• After-build pipelines. Codefresh is not configured on the local stack. Unit tests cover the fenced result publication.
• A re-added deploy label while a teardown waits for promotion locks. The teardown test reproduces the cutoff.
• An Aurora restore. The local stack has no AWS. Unit tests cover the three outcomes of the endpoint check.
• The drain after a lost promotion lease. The manager test reproduces it with a held sibling apply.

Follow-ups

• Cancel the native build job of a superseded run instead of letting it finish.
• Fence the Codefresh provider itself. The submission check closes the window before the handoff only.
• Read the YAML files outside the configuration lock.
• Apply the ingress per service. Each intent now applies the ingress of the full environment.
• A full pass keeps the last received SHA of a branch as its floor. After a force push whose webhook Lifecycle never received, a full pass deploys that SHA until the next received push.
• A cleanup of one service does not wait for an earlier deploy job of that service.