Skip to content

Add end-to-end tests and a deployment compare script - #93

Merged
adamshiervani merged 8 commits into
devfrom
e2e-tests
Nov 6, 2026
Merged

adamshiervani merged 8 commits into
devfrom
e2e-tests

Conversation

@adamshiervani

@adamshiervani adamshiervani commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor
  • npm run test:e2e: builds the API and runs dist/index.js against Postgres and SeaweedFS (S3). 87 tests:
    • Releases: registration via /releases/sync, update checks (URLs exist and match their hashes), upload-to-100% rollout, a new SKU's first staged release, rollout buckets, SKU fallback, prerelease, redirects.
    • HTTP: auth on every protected route, CORS, security headers, error bodies, logout, OIDC callback.
    • Devices: token exchange, removal.
    • WebSocket signaling: device and client auth, offer/answer/ICE relay, REAL_IP_HEADER, reconnects.
  • CI runs the e2e tests after the unit tests. compose.development.yaml adds the S3 service.
  • npm run compare -- --candidate <url|local|staging>: sends the same safe requests to two deployments (reference defaults to prod) and groups differences by cause. In a terminal it asks whether each group is expected. Replaces scripts/compare-releases.sh.
  • DELETE /devices/:id without credentials answers 401. The WebSocket upgrade handler and device disconnect cleanup catch their errors.
  • Prerelease /releases falls back to the newest of the 5 newest versions that has a build for the SKU, as the stable path does. /releases/app/latest and /releases/system_recovery/latest answer 404 instead of an older file.
  • With appVersion or systemVersion, only that part is pinned; the other follows the rollout like a background check.

Note

Medium Risk
Touches core release selection (rollouts, S3 fallbacks, latest redirects) and device delete auth responses; wrong logic could change what devices receive or break tooling. E2E adds CI complexity (Postgres, SeaweedFS, longer runs).

Overview
Adds a Vitest e2e layer (e2e/, vitest.e2e.config.ts) that builds dist/index.js, provisions a dedicated Postgres DB and SeaweedFS bucket with fixture artifacts, and exercises releases (sync, rollouts, pins, prerelease, redirects), HTTP (auth, CORS, headers), devices, and WebSocket signaling. CI starts the S3 service from compose.development.yaml, waits on :8333, then runs npx vitest run --config vitest.e2e.config.ts after unit tests.

Introduces npm run compare (scripts/compare-api.mts) to diff safe GET/preflight responses between two API bases with grouped output and optional TTY confirmation; removes scripts/compare-releases.sh. README documents e2e and compare.

Release behavior changes: stable /releases now mixes pinned and rollout—only unconstrained (*) app/system use staged rollout; the other part follows getReleaseByRange or rollout as described. Prerelease S3 picks the newest semver match that actually has an artifact for the SKU (up to five candidates), so a missing build on the latest upload does not block older builds. /releases/app/latest and /releases/system_recovery/latest require the newest version to ship that SKU—otherwise 404 (no silent fallback to older objects). Legacy layout paths require the object to exist in S3.

Auth/devices: verifySession is extracted; DELETE /devices/:id without a device bearer uses it so invalid sessions return 401 consistently. WebRTC upgrade handling and lastSeen updates are wrapped to avoid unhandled rejections; device reconnect flow is adjusted so the new connection proceeds after the old socket closes.

Unit tests in test/releases.test.ts cover the new release and redirect rules.

Reviewed by Cursor Bugbot for commit b0c6e7d. Bugbot is set up for automated code reviews on this repo. Configure here.

npm run test:e2e builds the API and runs it as the container does,
against Postgres and an S3-compatible bucket (SeaweedFS). The tests cover
release registration and update checks against real files and hashes,
the build-tooling redirects, auth on every protected route, CORS and
security headers, device registration and removal, and WebSocket
signaling. CI runs them after the unit tests.

npm run compare sends the same safe requests to two deployments and
reports the differences grouped by cause. It replaces
scripts/compare-releases.sh.

DELETE /devices/:id without credentials now answers 401. The upgrade
handler and device disconnect cleanup catch their errors.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4b0f576. Configure here.

Comment thread scripts/compare-api.mts Resolved
The prerelease /releases path, /releases/app/latest and
/releases/system_recovery/latest took the newest version folder and
answered 404 when it had no build for the SKU. They now share one
lookup that takes the newest version with a build for the SKU, as the
stable path does. A legacy-layout version counts only if its file
exists.

npm run compare asks, in a terminal, whether each group of differences
is expected, and exits with 0 only when every group is marked expected.
@adamshiervani
adamshiervani marked this pull request as ready for review November 6, 2026 10:54
A pinned ">=0.0.0" normalizes to "*", which the API treats as not
pinned, so discovery followed the rollout table. Start at a range that
stays pinned. A chain stops at a version with no build for its SKU, so
every SKU of a release type now gets the versions any of its SKUs found.
A request with appVersion or systemVersion took every part from the
version range, so the unpinned part was the newest registered release:
possibly at 0%, possibly without a build for the SKU (404). The
unpinned part now follows the rollout, as a background check does.
…ged release

Upload and sync register the release at 0%, raising it to 30% reaches
the devices in the bucket, 100% reaches every device. The first staged
release with a build for a new SKU reaches that SKU's devices in the
bucket, and the rest get 404.
Each version costs one or two S3 calls and a miss is not cached, so a
SKU with no build in a long range walked every version on every
request.
… for the SKU

Build tooling (rv1106-system update_app.sh) packages whatever this
redirect points to, so a missing SKU build must fail the build instead
of silently packaging an older app. The recovery image redirect and the
prerelease channel keep the fallback.
…ion has no image for the SKU

Both latest-file redirects now serve only the newest version. The
fallback to an older build remains for prerelease update checks, where
the device ignores a version that is not newer than its own.
@adamshiervani
adamshiervani merged commit 0f95ac2 into dev Nov 6, 2026
3 checks passed
adamshiervani added a commit that referenced this pull request Nov 6, 2026
… compiled image, e2e tests (#90)

- #88: a new SKU's first staged release reaches devices in its bucket; no compatible default is null, not 404.
- #89: new releases register at 0%, and a 0% row never displaces a rollout in progress.
- #92: multi-stage Docker image; the container runs compiled dist/index.js as node under tini, with a HEALTHCHECK.
- #93: e2e tests and npm run compare. DELETE /devices/:id without credentials answers 401; WebSocket upgrade and device disconnect errors are caught. A request that pins one part gets the other part from the rollout. Prerelease checks fall back to the newest of the 5 newest versions with a build for the SKU; the latest-file redirects answer 404 when the newest version has no file for the SKU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant