Skip to content

fix(deploy): reach the changelog identity service over cluster DNS [CPONETOPS-1176] - #374

Merged
morten-andersen merged 1 commit into
mainfrom
fix/changelog-identity-over-cluster-dns
Oct 7, 2026
Merged

morten-andersen merged 1 commit into
mainfrom
fix/changelog-identity-over-cluster-dns

Conversation

@cweinberger

Copy link
Copy Markdown
Member

Why

The Tailscale steps added in #362 never worked. create-changelog runs on arc-arm64-1cpu-2gb-nodocker, and those pods get neither /dev/net/tun nor NET_ADMIN, so tailscaled cannot start. From service-ecotap-gateway run 37486823410 (job 112350595118, runner arc-arm64-1cpu-2gb-nodocker-qbwgz-runner-p4gq6):

Starting tailscaled daemon...
Waiting for tailscaled daemon to become ready...
failed to connect to local tailscaled; it doesn't appear to be running   × 8

continue-on-error: true reported that step as green, and the wait step correctly skipped on outcome == 'failure'. Per kube-manifests infra/arc/README.md only the legacy self-hosted-arm64-2xl pool has Tailscale access. So every release notification since the feature landed has fallen back to GitHub profile emails — the thing the identity service exists to replace.

There was never a perimeter to cross: ARC runs on the internal EKS cluster, which is also where the identity service lives.

What

  • Drop the Tailscale and Wait for the identity service… steps, and the job-level TAILSCALE_AUTHKEY env, from create-changelog in both deploy-kotlin.yml and deploy-kotlin-v2.yml. vars.CHANGELOG_IDENTITY_RESOLVE_URL is still passed to changelog-cli-action exactly as before.
  • Update docs/workflow-guide.md (both the v1 and v2 sections) and CLAUDE.md.

Pairs with changing the org variable to the cluster-local URL:

http://project-tracker-internal.project-tracker-internal.svc.cluster.local:3001/api/identity/resolve

Note the service name and the port — project-tracker-internal-internal is the Linkerd mirror name, which exists on main-production/main-staging but not on internal itself, and the service listens on 3001, not 80.

Measured, not assumed

A one-step probe job on arc-arm64-1cpu-2gb-nodocker:

project-tracker-internal.project-tracker-internal.svc.cluster.local            172.20.38.219
project-tracker-internal-internal.project-tracker-internal.svc.cluster.local   NXDOMAIN

…-internal.project-tracker-internal.svc.cluster.local:3001/api/identity/resolve?github=…   HTTP 200, ~80 ms
…-internal.project-tracker-internal.svc.cluster.local:3001/api/identity/resolve            HTTP 400  (the route's own validation)

Blast radius

IdentityResolver is only constructed inside ReleaseNotificationService, which ChangeLogService only builds when both releaseNotifyChannel and releaseNotifySlackToken are set. Two repos set changelog-release-notify-channel today, both private, so both land on ARC:

Repo Workflow Channel
service-control deploy-ocpp-proxy-production.yml #ocpp-proxy
service-ecotap-gateway deploy-staging.yml #service-ecotap-gateway

Public-repo callers run on ubuntu-24.04-arm, where the internal name does not resolve; the CLI degrades to email lookups and the job still succeeds, same as today.

TAILSCALE_AUTHKEY stays declared

Nothing reads it any more, but passing a secret a reusable workflow does not declare is a startup error, and service-ecotap-gateway passes it explicitly to deploy-kotlin-v2.yml (deploy-staging.yml:61) and deploy-kotlin.yml (deploy-production.yml:39). Removing the input here would break both deploys. Its description now says it is unused; the input can go once those callers drop it. sonar-cloud.yml declares and uses its own, untouched.

🤖 Generated with Claude Code

…PONETOPS-1176]

The tailnet steps could never have worked where they ran. `create-changelog`
is on `arc-arm64-1cpu-2gb-nodocker`, whose pods get neither `/dev/net/tun` nor
`NET_ADMIN`, so `tailscaled` never came up — service-ecotap-gateway run
37486823410 shows `failed to connect to local tailscaled` eight times, the step
green only because of `continue-on-error`, and the wait step skipped on
`outcome == 'failure'`. Of the ARC pools only the legacy `self-hosted-arm64-2xl`
has Tailscale. So every release notification since the feature landed has been
on the GitHub-profile-email fallback it was meant to replace.

The runners are in the same EKS cluster (`internal`) as the identity service, so
there was never a perimeter to cross: with a `*.svc.cluster.local` value in
`CHANGELOG_IDENTITY_RESOLVE_URL` the lookup resolves directly. Measured from an
`arc-arm64-1cpu-2gb-nodocker` runner: 200 in ~80 ms, and ~27 s of dead Tailscale
install and retry gone from the job.

`TAILSCALE_AUTHKEY` stays declared on both workflows even though nothing reads
it. Passing a secret a reusable workflow does not declare is a startup error,
and service-ecotap-gateway passes it explicitly in both of its deploy workflows;
the input goes once those callers drop it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cweinberger
cweinberger requested a review from a team as a code owner October 7, 2026 07:33
@cweinberger
cweinberger requested review from sudheer-monta and removed request for a team October 7, 2026 07:33
@morten-andersen
morten-andersen merged commit 3956212 into main Oct 7, 2026
1 check passed
@morten-andersen
morten-andersen deleted the fix/changelog-identity-over-cluster-dns branch October 7, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants