Repository navigation
Release dev → main: 0% default rollout, staged releases for new SKUs, compiled image, e2e tests - #90
Merged
Merged
Conversation
…78) A prefix with no release at 100% made the default lookup throw a 500 before eligibility was checked, so no device got the staged build. That is every device for a new prefix until its first release is fully rolled out, and every JetKVM device the day no app or system row sits at 100%. The default lookup now returns null when nothing is at 100%. A device inside the rollout bucket gets the staged release as before; a device outside it gets a 404 saying no release is rolled out yet for its SKU, instead of a 500. Found by Bugbot on the release PR (#77).
* feat(releases): sync new R2 releases from the API every 30 minutes The release sync only ran when an operator invoked scripts/sync-releases.ts by hand, so a firmware upload sat in R2 until someone remembered to run it. The API process now runs the same sync on start and every 30 minutes, registering each new stable version at the default 10% rollout. The non-interactive core (bucket listing, artifact collection, DB insert) moves from the script into src/release-sync.ts so the API can import it; the script keeps the GPG check and the confirmation prompt and passes them in as the release decider. Both entry points share one R2 client via src/s3.ts. Scheduled runs skip a tick while the previous run is still going, log and survive a failed run, and treat a unique-constraint hit on insert as "already synced" so several API instances can race without an error. The existence check now precedes the R2 artifact walk, so a tick over an already-synced bucket costs one list call per prefix plus one DB lookup per version. * fix(release-sync): defer versions whose upload is still settling A scheduled tick can land while a version is still being uploaded and register it with a partial SKU set or a hash that changes afterwards. Sync never rewrites a row, so that snapshot would be permanent. Before collecting artifacts, list every object under the version folder and skip the version when the newest object changed in the last 10 minutes. The window is shorter than the schedule interval, so a real upload costs at most one extra tick. * refactor(release-sync): share R2 config, paginate via SDK, probe SKUs concurrently * src/s3.ts now exports bucketName and baseUrl next to the client, so the request handlers, the scheduler and the script read R2 config from one place instead of three. * The upload settle window moves into SyncConfig.uploadSettleMs. The scheduler sets it; the one-shot operator script leaves it unset because an operator sees the artifact list before confirming and has no next run. * newestUploadTime uses the SDK's paginateListObjectsV2 instead of a hand-rolled continuation loop. * collectReleaseArtifacts probes SKUs concurrently and folds the results in SKU order, so artifact order and the primary artifact are unchanged. * Tests drop empty per-prefix listing stubs that the file-level default already covers. * fix(release-sync): take the settle listing after the artifact scan Listing the version folder before the scan left a gap: an upload that started between the listing and the scan passed the check and was captured partially. Listing after the scan closes it, because any upload active during the scan leaves an object newer than the window and the snapshot is discarded instead of registered. * fix(release-sync): paginate the version listing ListObjectsV2 returns at most 1,000 common prefixes per page. Once a release prefix grows past that, versions on later pages were never seen on any tick. listStableVersions now walks every page with the SDK paginator, as newestUploadTime already does, and the mock in the sync tests can serve a truncated listing.
) * fix(release-sync): only treat a unique violation as a race when the row exists Sync caught every P2002 from the release insert as "created concurrently elsewhere". On staging the id sequences were behind the rows after a data import, so each insert failed on the primary key, was logged as a race, and left nothing in the table. After a unique violation, createRelease now looks the (version, type) row up. Present means another instance registered it first; absent means the insert really failed, and the error is rethrown with the type and version in its message so the scheduled run log names the release. * fix(release-sync): name the release in every per-version failure Wrapping only the insert error left the row lookup, and the S3 scan before it, free to escape without the type and version. syncReleases now wraps whatever createRelease throws for a version.
The scheduled tick found a new version at most 30 minutes plus the settle window after upload. The upload script knows when its last object is written, so it can trigger the sync itself. One runner now owns the in-progress flag and serves both the timer and the endpoint. The endpoint is guarded by RELEASE_SYNC_TOKEN as a bearer token, compared in constant time, and is not registered without it. It runs with the settle window off and answers with the per-outcome counts, or 409 while a run is in progress.
Scheme names are case-insensitive (RFC 9110). The token is still compared exactly.
…#84) The endpoint turned the settle window off for every version in the bucket. A call made while a different upload was still running registered that upload half done, and sync never revisits a row. The body now names the version the caller finished, and only that version skips the window. Without a body the call is a normal tick.
getDefaultRelease threw when 100% releases existed for the prefix but none shipped a binary for the SKU. Retrieve awaits that fallback before it checks whether the device is in the staged release's bucket, so the first staged release for a new SKU on a shared prefix answered 404 to every device of that SKU. It now returns null, like a prefix with no 100% release at all, and the caller's bucket check decides.
* feat(releases): register new releases at 0% rollout A newly registered release went to 10% of devices at once. It now starts at 0%: registered, but served to no device until an operator raises it. Existing rows keep their percentage. * fix(releases): a 0% release does not displace a rollout in progress Selection took the single newest release as the rollout candidate, so registering the next version at 0% pushed every device off an older release still mid-rollout and onto the last 100% one. The latest lookup now ignores rows at 0%. Version-constrained requests, which skip rollout, still see them.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Run the TypeScript source with ts-node, as the App Platform deploy does. Migrations are not part of the image build; run them separately.
Compile with tsc at build time, so a type error fails the build instead of crash-looping the container. The runtime stage has production dependencies only, runs as the node user under tini, and has a HEALTHCHECK on /healthz. Move ts-node, typescript, @tsconfig/node22 and @types/* to devDependencies. Keep .env.* files out of the build context. Run CI on Node.js 22.23.4 to match engines.
npm run test:e2e builds the API and runs dist/index.js against Postgres and SeaweedFS (S3): release registration and update checks against real files and hashes, rollouts from upload to 100%, a new SKU's first staged release, auth, CORS, headers, device registration and removal, and WebSocket signaling. CI runs them after the unit tests. npm run compare sends the same safe requests to two deployments, groups the differences by cause and, in a terminal, asks whether each group is expected. It replaces scripts/compare-releases.sh. DELETE /devices/:id without credentials answers 401. The WebSocket upgrade handler and device disconnect cleanup catch their errors. With appVersion or systemVersion, only that part is pinned; the other follows the rollout. Prerelease update checks fall back to the newest of the five newest versions with a build for the SKU. The latest-file redirects answer 404 when the newest version has no file for the SKU.
main has the Node.js 22.23.4 and Dockerfile changes as squash commits; dev has the same changes and the multi-stage Dockerfile from #92, so dev's files are kept.
adamshiervani
changed the title
adamshiervani
added a commit
that referenced
this pull request
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes from
devthatmaindoes not have yet (#78–#85 and Node.js 22.23.4 are already onmain):dist/index.jsasnodeundertini, with aHEALTHCHECK.npm run compare.DELETE /devices/:idwithout credentials answers 401; WebSocket upgrade and device disconnect errors are caught. A request that pins one part gets the other part from the rollout. Prerelease checks fall back to the newest of the 5 newest versions with a build for the SKU; the latest-file redirects answer 404 when the newest version has no file for the SKU.Operator change
A new release is invisible to stable update checks until its rollout is raised: register (tick or upload workflow), check the row, set the percentage. Prerelease checks and the latest-file redirects read the bucket and do not wait for the rollout.
Checked
npm run compare, this tree on a copy of production's release tables againstapi.jetkvm.com: 488 of 489 requests identical. The difference is which part a 404 message names when both pinned versions are missing.No migration.
Note
Medium Risk
Touches production rollout selection, S3 “latest” semantics, session-auth device deletion, and the container entrypoint; behavior changes are substantial but covered by new e2e and expanded unit tests.
Overview
Core product behavior
New stable releases register at 0% rollout until an operator raises them. "Latest" for update checks is the newest release with rollout > 0; 0% rows no longer displace an in‑progress rollout. Default fallback is the newest 100% release that actually ships a binary for the SKU (or null), so a new SKU’s first staged build can reach in‑bucket devices without older 100% rows masking it. Pinned
appVersion/systemVersionstill resolve from the DB range; the unpinned part follows rollout as before.S3 / prerelease paths
newestVersionWithArtifactwalks recent semver versions until one has the artifact for the SKU (with a short cap), so missing SKU builds don’t hide older versions. Latest redirect handlers return 404 if the newest version has no file for that SKU—no silent fallback to an older build. Legacy (pre‑SKU) paths are used only when the object exists.Auth & realtime
DELETE /devices/:idwith a session usesverifySession(proper 401 for bad/missing Google‑signed ID tokens). WebSocket upgrade and lastSeen updates are wrapped to avoid unhandled rejections; duplicate device connections still terminate the old socket before the new one completes.Tooling & ops
Multi‑stage Dockerfile: build+
npm prune,tini,USER node, healthcheck on/healthz,node dist/index.js.npm startandmaintarget compiled output; TypeScript/tooling moved to devDependencies. SeaweedFS S3 in dev compose; PR CI bumps Node 22.23.4, starts S3, runs Vitest e2e after unit tests.Tests & scripts
Large
e2e/Vitest suite (global setup: temp DB, bucket, built API) covers devices, HTTP/CORS, release sync, rollouts, signaling.npm run test:e2eandvitest.e2e.config.tsadded.scripts/compare-api.mtsreplacescompare-releases.shwith a broader, interactive diff against reference/candidate APIs. Unit tests andrelease-syncexpectations updated for 0% default..dockerignoretightens env file handling; README documents e2e and compare workflows.Reviewed by Cursor Bugbot for commit 66614c2. Bugbot is set up for automated code reviews on this repo. Configure here.