diff --git a/src/assets/blog/generated/eks-upgrades.jpg b/src/assets/blog/generated/eks-upgrades.jpg
new file mode 100644
index 0000000..0ee2886
Binary files /dev/null and b/src/assets/blog/generated/eks-upgrades.jpg differ
diff --git a/src/assets/blog/generated/eks-upgrades.svg b/src/assets/blog/generated/eks-upgrades.svg
new file mode 100644
index 0000000..fce2028
--- /dev/null
+++ b/src/assets/blog/generated/eks-upgrades.svg
@@ -0,0 +1,43 @@
+
diff --git a/src/assets/blog/generated/platform-audit.jpg b/src/assets/blog/generated/platform-audit.jpg
new file mode 100644
index 0000000..32dd092
Binary files /dev/null and b/src/assets/blog/generated/platform-audit.jpg differ
diff --git a/src/assets/blog/generated/platform-audit.svg b/src/assets/blog/generated/platform-audit.svg
new file mode 100644
index 0000000..d3c14df
--- /dev/null
+++ b/src/assets/blog/generated/platform-audit.svg
@@ -0,0 +1,112 @@
+
diff --git a/src/assets/blog/generated/private-eks-day-2.jpg b/src/assets/blog/generated/private-eks-day-2.jpg
new file mode 100644
index 0000000..d2bdf7b
Binary files /dev/null and b/src/assets/blog/generated/private-eks-day-2.jpg differ
diff --git a/src/assets/blog/generated/private-eks-day-2.svg b/src/assets/blog/generated/private-eks-day-2.svg
new file mode 100644
index 0000000..bfa0dc7
--- /dev/null
+++ b/src/assets/blog/generated/private-eks-day-2.svg
@@ -0,0 +1,59 @@
+
diff --git a/src/assets/blog/generated/private-eks-tailscale.jpg b/src/assets/blog/generated/private-eks-tailscale.jpg
new file mode 100644
index 0000000..0a42580
Binary files /dev/null and b/src/assets/blog/generated/private-eks-tailscale.jpg differ
diff --git a/src/assets/blog/generated/private-eks-tailscale.svg b/src/assets/blog/generated/private-eks-tailscale.svg
new file mode 100644
index 0000000..fdf6ebd
--- /dev/null
+++ b/src/assets/blog/generated/private-eks-tailscale.svg
@@ -0,0 +1,71 @@
+
diff --git a/src/content/blog/eks-upgrades-without-drama.md b/src/content/blog/eks-upgrades-without-drama.md
new file mode 100644
index 0000000..8a8bd67
--- /dev/null
+++ b/src/content/blog/eks-upgrades-without-drama.md
@@ -0,0 +1,227 @@
+---
+title: "EKS upgrades without the drama"
+description: "Why EKS upgrades keep sneaking up on teams, the order to do them in, finding deprecated APIs from CI on a private cluster, and what rolling back really means."
+pubDate: 2026-10-31
+heroImage: ../../assets/blog/generated/eks-upgrades.jpg
+heroAlt: "A staircase of version blocks climbing from left to right, with a cluster stepping up onto the next one"
+draft: true
+---
+
+Kubernetes ships three minor versions a year. EKS supports each one for 14 months, and then for another 12 at six
+times the price. Upgrades never stop coming, and teams that treat each one as a project tend to fall behind, then pay
+for it.
+
+This post covers how to make EKS upgrades routine. It builds on the private cluster from
+[the first post in this series](/blog/bootstrapping-private-eks-argocd/), which adds a twist: CI can't reach the
+Kubernetes API, so the usual "run a checker against the cluster" advice needs rethinking.
+
+## Why falling behind is expensive
+
+Every EKS version gets **14 months of standard support**. After that it moves to **extended support** for another 12
+months, and the control plane price goes from $0.10 to $0.60 an hour. That's roughly $73 a month per cluster going up
+to $438. Across a dev, staging and production cluster, that's over $13,000 a year spent on not upgrading.
+
+At the end of extended support, AWS upgrades the cluster for you, whether your workloads are ready or not.
+
+You can also choose to skip extended support. Set the cluster's upgrade policy to `STANDARD`, and AWS upgrades it
+automatically when standard support ends:
+
+```hcl
+module "eks" {
+ # ...
+ upgrade_policy = {
+ support_type = "STANDARD"
+ }
+}
+```
+
+That sounds alarming, but it's a useful forcing function. It turns "we should upgrade at some point" into a date, and
+the date is always 14 months after the release. In practice, if you upgrade on a schedule you'll never get close to it.
+
+The other catch is that **the control plane can only move one minor version at a time.** A cluster that's three versions
+behind needs three upgrades in a row, each with its own checks. Falling behind doesn't just cost more. It makes the
+eventual catch-up much bigger.
+
+## The order matters
+
+An EKS upgrade isn't one thing. It's at least four, and they have to happen in order:
+
+1. **Things in the cluster that need to be ready for the new version.** Workloads using APIs that the new version
+ removes, and add-ons whose current version doesn't support it.
+2. **The control plane.** This is the bit EKS does for you, and it can't be undone.
+3. **EKS add-ons**: the VPC CNI, CoreDNS, kube-proxy and the Pod Identity agent. Each has versions built for each
+ Kubernetes version.
+4. **Nodes.** Kubelets can run up to three minor versions behind the control plane, but never ahead of it, so nodes
+ always go last.
+
+Most upgrade incidents come from doing step 2 before step 1 is finished.
+
+## Step 1: find what will break
+
+### Deprecated and removed APIs
+
+When Kubernetes removes an API version, anything still using it stops working: manifests fail to apply, and controllers
+that call the old API get errors. Finding these is the most important prep work.
+
+The usual advice is to run [pluto](https://github.com/FairwindsOps/pluto) or kubent against the live cluster. On a
+private cluster, CI can't do that. But there are two other angles that don't need cluster access at all.
+
+**EKS upgrade insights** come through the AWS API. EKS reads your cluster's audit logs and flags any deprecated API
+that something has called in the last 30 days, along with which client called it. It also checks things like kubelet
+version skew and add-on compatibility. GitHub Actions can query it like any other AWS API:
+
+```yaml
+ upgrade-readiness:
+ if: github.event_name == 'pull_request'
+ runs-on: ubuntu-latest
+ steps:
+ - uses: aws-actions/configure-aws-credentials@v6
+ with:
+ role-to-assume: ${{ vars.AWS_ROLE_ARN }}
+ aws-region: eu-west-2
+
+ - name: Check EKS upgrade insights
+ run: |
+ aws eks list-insights --cluster-name platform \
+ --filter categories=UPGRADE_READINESS \
+ --query 'insights[].[insightStatus.status, kubernetesVersion, name]' \
+ --output table
+
+ failing=$(aws eks list-insights --cluster-name platform \
+ --filter categories=UPGRADE_READINESS \
+ --query "length(insights[?insightStatus.status=='ERROR'])" --output text)
+
+ if [ "$failing" != "0" ]; then
+ echo "::error::$failing upgrade insight(s) failing. Fix them before upgrading."
+ exit 1
+ fi
+```
+
+Run it on pull requests that change `kubernetes_version`, and the PR can't go green until the insights pass. Insights
+refresh once a day, so after a fix it can take a while to clear.
+
+**Your manifests are in Git.** With Argo CD, everything in the cluster comes from a repo. You can render your own charts
+in CI and check them without going near the cluster:
+
+```bash
+helm template ./my-app | pluto detect - --target-versions k8s=v1.36.0
+```
+
+Between the two, you cover both what's in Git and what's actually calling the API, including things you didn't deploy
+through Argo CD.
+
+### Add-on compatibility
+
+Every controller you run has a range of Kubernetes versions it supports: the load balancer controller, cert-manager,
+External Secrets, Argo CD itself, and any operators your teams have installed. Check each one's compatibility matrix
+before upgrading. Often the fix is to bump the chart version first, in its own pull request, while still on the old
+Kubernetes version.
+
+This is where keeping chart versions in one `values.yaml` (as in the
+[Day 2 post](/blog/private-eks-day-2/)) pays off. The list of things to check is one file.
+
+### Read the release notes
+
+Read the EKS documentation page for the new version, and the Kubernetes changelog's "Urgent upgrade notes" section.
+Most releases have nothing that affects you. Every so often there's one that does, like the end of Amazon Linux 2
+AMIs. EKS doesn't publish AL2 images for Kubernetes 1.33 or later, so any node group still on AL2 had to move to AL2023
+or Bottlerocket before it could upgrade.
+
+## Step 2: the control plane
+
+With the EKS Terraform module, the upgrade is a one-line change:
+
+```hcl
+module "eks" {
+ # ...
+ kubernetes_version = "1.36"
+}
+```
+
+The control plane upgrade usually takes somewhere between 10 and 30 minutes. EKS replaces its control plane instances
+gradually, so the API stays available, though long-running watches may reconnect.
+
+There's a catch with managed node groups. By default they follow the cluster's version, so the same apply that upgrades
+the control plane also starts rolling every node. For a small cluster that can be fine. For anything bigger, pin the
+node groups so the control plane and nodes go in separate pull requests:
+
+```hcl
+ eks_managed_node_groups = {
+ default = {
+ kubernetes_version = "1.35" # bump in a follow-up PR once the control plane is done
+ # ...
+ }
+ }
+```
+
+That gives you a point between the two where you can check everything still works before nodes start moving.
+
+## Step 3: EKS add-ons
+
+Each EKS add-on has versions built for each Kubernetes version, and `aws eks describe-addon-versions` lists them. The
+EKS module's add-ons default to `most_recent = true`, so the `coredns = {}` entries from part 1 already move to the
+latest compatible version when the cluster version changes.
+
+That's convenient, but it also means a plan can change add-on versions on a day you weren't expecting to. If you'd
+rather every change be deliberate, set `addon_version` on each one and bump them in the upgrade PR. This lists the
+versions that work with the version you're moving to, newest first:
+
+```bash
+aws eks describe-addon-versions --addon-name coredns --kubernetes-version 1.36 \
+ --query 'addons[0].addonVersions[].addonVersion' --output text
+```
+
+kube-proxy is the one to keep in step with the control plane. It shouldn't be newer than the control plane, and it
+shouldn't be more than three minor versions older than it.
+
+## Step 4: nodes
+
+Managed node groups replace nodes in a rolling update. A new node joins, then an old one is cordoned and drained. That
+drain is where most problems turn up:
+
+- **PodDisruptionBudgets that allow no disruption.** A PDB with `maxUnavailable: 0`, or `minAvailable` equal to the
+ replica count, blocks the drain forever. Managed node groups give up after 15 minutes and fail the update. Find these
+ before you start: `kubectl get pdb -A` and look for `ALLOWED DISRUPTIONS` of 0.
+- **Single-replica workloads.** They'll have a short outage when their node drains. If that matters, they need a second
+ replica, not a careful upgrade.
+- **Admission webhooks with `failurePolicy: Fail`.** If a webhook's own pods are being rescheduled, every request the
+ webhook covers fails in the meantime, which can include the pods needed to bring it back. Run webhook pods with at
+ least two replicas and a PDB.
+
+If you use Karpenter instead of managed node groups, nodes pick up the new AMI through drift detection and are replaced
+within the disruption budgets you've set on the NodePool. The same PDB problems apply.
+
+## Rolling back
+
+**You can't roll back the EKS control plane.** Once it's on 1.36, it stays on 1.36. That's worth saying clearly to
+whoever approves the change, because it changes what a rollback plan means:
+
+- **Add-ons can be rolled back.** Revert the version and apply.
+- **Nodes can be rolled back.** Point the node group back at the old version, or create a new node group on it.
+ Kubelets can be behind the control plane, so old nodes will happily join.
+- **Workloads can be rolled back** through Git, like any other change.
+- **The control plane can't**, so your rollback plan for it is everything in step 1. Test the upgrade on a non-production
+ cluster first, and make sure the insights pass before you touch production.
+
+There's one more option for clusters where an upgrade really can't go wrong: **don't upgrade in place.** Build a new
+cluster on the new version and move traffic across. If your clusters are built like the one in this series, that's a
+lot less work than it sounds. Terraform creates the cluster, the bootstrap installs Argo CD, and Argo CD installs
+everything else from Git. A new cluster is a new Terraform workspace and a DNS change. It's more effort than an in-place
+upgrade, but it's the only approach with a real rollback.
+
+## Making it routine
+
+The teams that find upgrades painless aren't doing anything clever. They upgrade often enough that each one is small:
+
+- **Pick a cadence and stick to it.** With three Kubernetes releases a year, aim to upgrade each cluster roughly every
+ four months. You'll always be one or two versions behind the newest, which is a comfortable place to be.
+- **Upgrade non-production first**, and leave a week or two before production. Most problems show up in that gap.
+- **Keep the checks in CI.** Upgrade insights on every PR that touches `kubernetes_version`, and pluto on every chart.
+- **Write down what happened.** A short note per upgrade (what broke, what you'd do differently) makes the next one
+ faster, especially when it's a different person doing it.
+
+An upgrade that happens every few months, follows the same steps, and gets checked by CI isn't a project. It's just
+maintenance.
+
+If your clusters have fallen a few versions behind and you'd like a hand catching up, that's something we do a lot of.
+See [platform engineering](/services/platform-engineering/) or [get in touch](/contact/).
diff --git a/src/content/blog/platform-audit-findings.md b/src/content/blog/platform-audit-findings.md
new file mode 100644
index 0000000..63d7ef2
--- /dev/null
+++ b/src/content/blog/platform-audit-findings.md
@@ -0,0 +1,182 @@
+---
+title: "What we find when we audit a Kubernetes platform"
+description: "The ten things that turn up again and again when we first look at a team's Kubernetes, Terraform and pipelines, why they matter, and the smallest useful fix for each."
+pubDate: 2026-11-07
+heroImage: ../../assets/blog/generated/platform-audit.jpg
+heroAlt: "A magnifying glass over a grid of cluster nodes, with a few highlighted in amber"
+draft: true
+---
+
+A lot of our work starts the same way. A team has a Kubernetes platform that works, mostly, but it's slower, more
+expensive or more fragile than it should be, and they'd like someone to look at it with fresh eyes.
+
+Every platform is different, but the findings are remarkably consistent. These are the ten that turn up most often,
+roughly in order of how often we see them. None of them are signs of a bad team. They're what happens when a platform
+grows faster than the time anyone has to look after it.
+
+For each one there's a quick way to check your own platform, and the smallest fix that makes a real difference.
+
+
+
+## 1. Terraform that doesn't match reality
+
+**What we see:** `terraform plan` on a supposedly up-to-date environment shows a page of changes nobody can explain.
+Someone fixed something in the console during an incident, a security group got edited by hand, or an apply failed
+halfway months ago. Now nobody wants to run `apply`, because nobody knows what it'll do.
+
+**Why it matters:** Infrastructure as code only helps if the code is true. Once people stop trusting the plan, changes
+go back to the console, and the drift gets worse.
+
+**Check yours:** Run a plan against every environment. If any of them isn't clean, you have drift.
+
+**Smallest fix:** Run a scheduled plan in CI every night and alert on any change. Then work through the existing drift
+one resource at a time, deciding whether the code or the console is right. Once plans are clean, keep them that way.
+
+
+
+## 2. Pipelines with admin keys
+
+**What we see:** Long-lived AWS access keys stored as CI secrets, attached to a user with `AdministratorAccess`. Often
+they were created by someone who has since left, and nobody knows what else uses them.
+
+**Why it matters:** Those keys are the most valuable secret you have. Anyone who can change a workflow file, or any
+compromised dependency in your build, can use them.
+
+**Check yours:** In IAM, look for users with access keys, and when each key was last used and rotated. The IAM
+credential report has all of this in one CSV.
+
+**Smallest fix:** Switch CI to OIDC, so pipelines get short-lived credentials and there's nothing to leak. GitHub
+Actions, GitLab and most other CI systems support it. Then scope the role down, and give pull request plans a separate
+read-only role. We covered the setup in
+[Bootstrapping a private EKS cluster](/blog/bootstrapping-private-eks-argocd/#step-1-let-github-actions-into-aws-without-keys).
+
+## 3. Clusters several versions behind
+
+**What we see:** Clusters two or three minor versions behind, often in extended support. The last upgrade was painful,
+so the next one keeps getting pushed back.
+
+**Why it matters:** Extended support costs six times as much for the control plane, and the catch-up gets harder the
+longer you leave it, since you can only move one version at a time. Add-ons and Helm charts fall behind too, and
+eventually you can't upgrade one without the other.
+
+**Check yours:** Compare each cluster's version with the
+[EKS release calendar](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
+
+**Smallest fix:** Upgrade non-production now, on its own, to learn what breaks. Then put upgrades on a schedule. We
+wrote up [how we approach EKS upgrades](/blog/eks-upgrades-without-drama/).
+
+## 4. Resource requests that don't match usage
+
+**What we see:** Either no resource requests at all, or requests copied from a template and never revisited. Most
+often it's the second: pods requesting a full CPU and 2 GB of memory while using a tenth of that.
+
+**Why it matters:** The scheduler and the autoscaler both work from requests, not usage. Requests that are too high
+mean you pay for nodes that sit idle. Requests that are missing mean noisy neighbours and pods getting evicted under
+memory pressure.
+
+**Check yours:** Compare requests with actual usage across the cluster. `kubectl top pods -A` gives a rough snapshot,
+and your metrics stack can give you the trend. A cluster where usage is under 30% of requests has room to shrink.
+
+**Smallest fix:** Set requests from observed usage, with headroom. Run the Vertical Pod Autoscaler in recommendation
+mode to get suggested values without it changing anything. Set memory limits equal to memory requests, and think hard
+before setting CPU limits at all, since they cause throttling even when the node has CPU to spare.
+
+
+
+## 5. Surprising network costs
+
+**What we see:** NAT gateway charges that rival the cost of the clusters. Every image pull, every S3 request and every
+call to an AWS API goes out through the NAT gateway and back, charged per gigabyte.
+
+**Why it matters:** NAT data processing is charged on everything that passes through it, even traffic to AWS services
+in the same region. A cluster that pulls large images often, or reads a lot from S3, can rack up thousands a month in
+NAT charges alone.
+
+**Check yours:** In Cost Explorer, filter on the `NatGateway-Bytes` usage type. If it's a big number, find out what's
+sending the traffic with VPC flow logs.
+
+**Smallest fix:** Add a gateway endpoint for S3, which is free and takes S3 traffic off the NAT gateway entirely.
+Then add interface endpoints for whatever else is heavy, usually ECR. Pulling images through a cache in the same region
+helps too.
+
+## 6. Secrets in the wrong places
+
+**What we see:** Secrets in Helm values files, in Git, or in Kubernetes Secrets that someone created by hand and nobody
+can recreate. Sometimes all three for the same secret.
+
+**Why it matters:** A Kubernetes Secret is base64-encoded, not encrypted. Secrets in Git are in every clone, forever.
+And secrets created by hand are lost when the cluster is.
+
+**Check yours:** Search your repos for anything that looks like a credential (gitleaks or trufflehog will do it), and
+list the Secrets in each cluster that aren't managed by anything.
+
+**Smallest fix:** Pick one source of truth, such as AWS Secrets Manager, and sync from it with External Secrets. Move
+the most sensitive secrets first. Anything that has ever been in Git should be rotated, not just moved. There's an
+example setup in our [Day 2 post](/blog/private-eks-day-2/#step-4-one-secret-store-and-a-first-real-secret).
+
+## 7. Everyone is cluster-admin
+
+**What we see:** One shared kubeconfig, or every engineer mapped to `system:masters`. Access was set up in a hurry when
+the cluster was new and never revisited.
+
+**Why it matters:** It makes every mistake a potential outage, and it makes it impossible to tell from the audit log
+who did what.
+
+**Check yours:** On EKS, list the access entries (or the `aws-auth` ConfigMap on older clusters) and see who gets which
+policy.
+
+**Smallest fix:** Give people access through your identity provider, with their own identity. Make read-only the
+default, and keep admin for a small group or for break-glass. On EKS, move from `aws-auth` to access entries, which are
+managed through the AWS API and so can live in Terraform.
+
+## 8. Deploys that cause downtime
+
+**What we see:** Single-replica deployments, no readiness probes, and no PodDisruptionBudgets. Every deploy, node
+upgrade or autoscaler scale-down causes a short outage, which has become normal enough that nobody mentions it.
+
+**Why it matters:** These outages are small, but they add up, and they make every change feel risky. Teams start
+deploying less often, which makes each deploy bigger and riskier still.
+
+**Check yours:** Look for Deployments with one replica, containers without a readiness probe, and workloads without a
+PDB. Then watch what happens to error rates during a deploy.
+
+**Smallest fix:** For anything user-facing: at least two replicas, a readiness probe that checks the app can actually
+serve, and a PDB with `maxUnavailable: 1`. Add a `preStop` sleep of a few seconds, so the load balancer stops sending
+traffic before the pod shuts down. A policy engine like Kyverno can then warn when new workloads are missing them.
+
+## 9. Monitoring nobody looks at
+
+**What we see:** Plenty of dashboards and plenty of alerts, all going to a Slack channel that everyone has muted.
+Meanwhile, the first sign of a real outage is a customer emailing support.
+
+**Why it matters:** Alerts that fire all the time teach people to ignore alerts. And the ones that matter get lost in
+the noise.
+
+**Check yours:** Count how many alerts fired last week, and how many of them someone acted on.
+
+**Smallest fix:** Delete or silence every alert that nobody acts on. Then add a few that page on symptoms users would
+notice, like error rates and latency for the main user journeys, rather than causes like CPU usage. Fewer, better
+alerts beat more of them every time.
+
+## 10. Backups nobody has restored
+
+**What we see:** Backups configured for databases and clusters, sometimes with Velero, but never tested. Sometimes the
+Terraform state bucket doesn't have versioning turned on, so a bad apply or an accidental delete is unrecoverable.
+
+**Why it matters:** An untested backup is a guess. The time you find out it doesn't work is the time you need it.
+
+**Check yours:** When was the last time someone restored something from backup, end to end, and timed it? Is
+versioning on for your state buckets?
+
+**Smallest fix:** Turn on versioning for state buckets today. Then schedule a restore test, even just once a quarter,
+of your most important database to a scratch environment. Write down how long it took. If your clusters are built from
+Git, test rebuilding one too. That's often the fastest disaster recovery plan there is.
+
+## The common thread
+
+Most of these come from the same place. The platform was built quickly, it worked, and the team moved on to the next
+thing. Nobody had time to go back. None of the fixes above need a rewrite, and most of them are a few days' work each.
+The hard part is making the time.
+
+If you'd like someone to go through your platform and tell you which of these apply, and which to fix first, that's
+exactly what our [platform reviews](/services/reviews-and-training/) cover. [Get in touch](/contact/).
diff --git a/src/content/blog/private-eks-day-2.md b/src/content/blog/private-eks-day-2.md
new file mode 100644
index 0000000..87ac0e8
--- /dev/null
+++ b/src/content/blog/private-eks-day-2.md
@@ -0,0 +1,381 @@
+---
+title: "Day 2 on a private EKS cluster"
+description: "Load balancers, DNS and secrets with Pod Identity and Argo CD, plus the options for giving people kubectl access when the API server has no public endpoint."
+pubDate: 2026-10-24
+heroImage: ../../assets/blog/generated/private-eks-day-2.jpg
+heroAlt: "A padlocked VPC boundary with a cluster inside, and a narrow tunnel reaching in to a laptop outside"
+draft: true
+---
+
+In [the last post](/blog/bootstrapping-private-eks-argocd/) we built an EKS cluster with no public API endpoint and got
+Argo CD onto it from a CodeBuild build inside the VPC. Argo CD then took over managing itself from Git.
+
+That leaves a cluster that can run things but can't do much useful with them yet. Nothing can be reached from outside,
+nothing has a DNS name, apps have nowhere to get secrets from, and the only thing with `kubectl` access is a CodeBuild
+role. This post fills those gaps:
+
+- **AWS Load Balancer Controller** to turn Ingresses into ALBs
+- **external-dns** to give them names in Route 53
+- **External Secrets** to pull secrets from AWS Secrets Manager
+- **A way in for people**, for the times someone really does need `kubectl`
+
+All of it is in the same example repo,
+[HeyalSolutions/eks-private-argocd-bootstrap](https://github.com/HeyalSolutions/eks-private-argocd-bootstrap).
+
+## The split: Terraform does AWS, Argo CD does Kubernetes
+
+Every add-on here needs two things. It needs **AWS permissions** (an IAM role) and it needs **installing into the
+cluster** (a Helm chart). The first is a job for Terraform, which can reach the AWS API from GitHub Actions. The second
+is a job for Argo CD, which is inside the cluster. Neither has to reach across to do the other's job.
+
+The glue between them used to be IRSA: an OIDC provider on the cluster, a role trust policy per service account, and an
+annotation on the service account with the role ARN. **EKS Pod Identity** replaces all of that. You tell EKS "this
+service account in this namespace gets this role", and the Pod Identity agent (which we installed as an add-on in part
+1) hands out credentials. There's nothing to annotate on the Kubernetes side, so the Helm values don't need to know any
+role ARNs.
+
+## Step 1: IAM roles with Pod Identity
+
+The [eks-pod-identity module](https://registry.terraform.io/modules/terraform-aws-modules/eks-pod-identity/aws) has
+ready-made policies for the common add-ons. Here's the load balancer controller:
+
+```hcl
+module "aws_lb_controller_pod_identity" {
+ source = "terraform-aws-modules/eks-pod-identity/aws"
+ version = "~> 2.10"
+
+ name = "${local.name}-aws-lb-controller"
+ attach_aws_lb_controller_policy = true
+
+ associations = {
+ this = {
+ cluster_name = module.eks.cluster_name
+ namespace = "kube-system"
+ service_account = "aws-load-balancer-controller"
+ }
+ }
+}
+```
+
+external-dns and External Secrets follow the same pattern, but they're worth scoping down. external-dns only gets the
+one hosted zone it manages, and External Secrets gets read-only access to secrets under the `platform/` prefix:
+
+```hcl
+module "external_dns_pod_identity" {
+ # ...
+ attach_external_dns_policy = true
+ external_dns_hosted_zone_arns = [data.aws_route53_zone.internal.arn]
+}
+
+module "external_secrets_pod_identity" {
+ # ...
+ attach_external_secrets_policy = true
+ external_secrets_secrets_manager_arns = [
+ "arn:aws:secretsmanager:${local.region}:${data.aws_caller_identity.current.account_id}:secret:${local.name}/*",
+ ]
+ external_secrets_create_permission = false
+}
+```
+
+The same file creates a wildcard ACM certificate for `*.internal.example.com`, validated through Route 53. The load
+balancer controller finds certificates by hostname, so no Ingress needs a certificate ARN.
+
+## Step 2: getting values from Terraform to Argo CD
+
+Some of the Helm values come from Terraform: the cluster name, the region and the VPC ID. There are a few ways to get
+them across:
+
+- **Commit them to the `cluster-resources` repo.** Simple and visible. They rarely change, and when they do it's a pull
+ request like everything else. This is what the example does.
+- **Annotate the Argo CD cluster Secret** and read the annotations with an ApplicationSet cluster generator. This is
+ the "GitOps bridge" pattern, and it pays off once you have more than a handful of clusters.
+- **Have Terraform write a ConfigMap.** Not an option here, since Terraform can't reach the cluster.
+
+For one cluster, committing them wins. A Terraform output prints exactly what to paste:
+
+```hcl
+output "cluster_values" {
+ value = {
+ name = module.eks.cluster_name
+ region = local.region
+ vpcId = module.vpc.vpc_id
+ domain = local.domain
+ }
+}
+```
+
+```yaml
+# cluster-resources/values.yaml
+cluster:
+ name: platform
+ region: eu-west-2
+ vpcId: vpc-0123456789abcdef0
+ domain: internal.example.com
+```
+
+## Step 3: the add-ons
+
+Each add-on is one more Application template in the `cluster-resources` chart, the same as cert-manager in part 1. The
+load balancer controller:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: aws-load-balancer-controller
+ namespace: argocd
+spec:
+ project: default
+ source:
+ repoURL: https://aws.github.io/eks-charts
+ chart: aws-load-balancer-controller
+ targetRevision: {{ .Values.awsLoadBalancerController.chartVersion | quote }}
+ helm:
+ valuesObject:
+ clusterName: {{ .Values.cluster.name }}
+ # Set explicitly so the controller doesn't depend on instance metadata
+ region: {{ .Values.cluster.region }}
+ vpcId: {{ .Values.cluster.vpcId }}
+ serviceAccount:
+ name: aws-load-balancer-controller
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: kube-system
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+```
+
+The service account name has to match the Pod Identity association exactly. That's the only link between the two, so
+it's worth setting explicitly rather than relying on the chart's default.
+
+external-dns watches Ingresses and Services and writes records into the internal zone. `txtOwnerId` marks the records
+as belonging to this cluster, so two clusters sharing a zone don't delete each other's records:
+
+```yaml
+ valuesObject:
+ provider:
+ name: aws
+ policy: sync
+ txtOwnerId: {{ .Values.cluster.name }}
+ domainFilters:
+ - {{ .Values.cluster.domain }}
+ sources: [ingress, service]
+ serviceAccount:
+ name: external-dns
+ env:
+ - name: AWS_DEFAULT_REGION
+ value: {{ .Values.cluster.region }}
+```
+
+External Secrets is a plain chart install. The interesting part is what we do with it next.
+
+## Step 4: one secret store, and a first real secret
+
+A `ClusterSecretStore` with no `auth` block tells External Secrets to use its own Pod Identity role, which can only read
+`platform/*`:
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+ name: aws-secrets-manager
+ annotations:
+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+ provider:
+ aws:
+ service: SecretsManager
+ region: {{ .Values.cluster.region }}
+```
+
+A good first secret to manage is one we already have. In part 1 the bootstrap build created Argo CD's repo credentials
+with `kubectl create secret`. That's the one secret in the cluster that came from a script rather than from Git. An
+`ExternalSecret` with the same name takes it over:
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+ name: cluster-resources-repo
+ namespace: argocd
+ annotations:
+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+ refreshInterval: 1h
+ secretStoreRef:
+ kind: ClusterSecretStore
+ name: aws-secrets-manager
+ target:
+ name: cluster-resources-repo
+ # No owner reference, so deleting this ExternalSecret doesn't delete the
+ # credentials Argo CD needs to put it back
+ creationPolicy: Orphan
+ template:
+ metadata:
+ labels:
+ argocd.argoproj.io/secret-type: repository
+ data:
+ type: git
+ url: {{ .Values.clusterResourcesRepo }}
+ sshPrivateKey: "{{ `{{ .deployKey }}` }}"
+ data:
+ - secretKey: deployKey
+ remoteRef:
+ key: platform/cluster-resources-deploy-key
+```
+
+Rotating the deploy key is now a change in Secrets Manager, picked up within the hour. The doubled-up braces are Helm
+escaping, because both Helm and External Secrets use Go templates.
+
+Note `creationPolicy: Orphan`. With the default (`Owner`), the Secret gets an owner reference to the ExternalSecret,
+and Kubernetes deletes it along with its owner. If someone ever prunes that ExternalSecret by accident, Argo CD would
+lose access to the very repo it needs to restore it. It's an unlikely failure, but a horrible one to recover from on a
+cluster you can't easily reach.
+
+## Step 5: an internal Ingress for Argo CD
+
+With all three running, exposing something is a handful of annotations. The Argo CD chart can create its own Ingress:
+
+```yaml
+argocd:
+ values:
+ global:
+ domain: argocd.internal.example.com
+ configs:
+ params:
+ server.insecure: true # TLS terminates at the load balancer
+ server:
+ ingress:
+ enabled: true
+ ingressClassName: alb
+ annotations:
+ alb.ingress.kubernetes.io/scheme: internal
+ alb.ingress.kubernetes.io/target-type: ip
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
+ alb.ingress.kubernetes.io/group.name: internal
+```
+
+The controller creates an internal ALB in the private subnets (tagged for it back in part 1), finds the wildcard
+certificate by hostname, and external-dns points `argocd.internal.example.com` at it. `group.name` puts every internal
+Ingress behind one shared ALB, so you pay for one load balancer, not one per app.
+
+The zone is public, so anyone can look up the name, but it resolves to private IPs that only work from inside the VPC.
+If you'd rather not publish internal names at all, use a private hosted zone associated with the VPC instead. That
+means your certificates need validating somewhere else, though.
+
+The load balancer controller also supports the Gateway API now, behind the `ALBGatewayAPI` and `NLBGatewayAPI` feature
+gates. Ingress is still the simpler starting point, but it's worth a look if you're starting fresh.
+
+## Step 6: getting people in
+
+CI never needs to reach the cluster again. People still do, sometimes: to debug a failed sync, read logs Argo CD doesn't
+show, or recover from a bad change. And now there's an internal ALB they'll want to open in a browser.
+
+There are two separate needs here, and they point at different answers. **Break-glass `kubectl`** is occasional and
+should be audited. **Daily access to internal tools** like the Argo CD UI is frequent and needs to be painless.
+
+| Option | How it works | Good for | Watch out for | Rough cost |
+|---|---|---|---|---|
+| **SSM port forwarding** | A small instance with no inbound ports; Session Manager tunnels a local port to the API endpoint | Break-glass `kubectl`; every session is IAM-controlled and logged | One port per tunnel, so browsing internal UIs is clunky; needs the Session Manager plugin | ~$3/month |
+| **Tailscale subnet router** | A small instance advertises the VPC's routes to your tailnet | Daily access to everything internal; very little setup | A third party in your access path; another identity system to keep in sync | ~$3/month plus Tailscale's per-user pricing |
+| **AWS Client VPN** | Managed OpenVPN endpoint in the VPC, with SSO through IAM Identity Center | Teams that need full network access and want it all in AWS | The cost; certificate management; clients on every laptop | ~$73/month per subnet association, plus ~$0.05 per connection hour |
+| **Bastion with SSH** | Public instance with port 22 open | Nothing you can't do better with SSM | Public attack surface, SSH keys to manage | ~$7/month |
+
+**Start with SSM.** It costs almost nothing, adds no new identity system, and every session lands in CloudTrail with the
+person's name on it. That's exactly the trail you want for break-glass access. If a lot of people end up in internal UIs
+every day, add Tailscale or Client VPN for that, and keep SSM for `kubectl`. We cover that next step in
+[Cluster access for a growing team with Tailscale](/blog/private-eks-tailscale/).
+
+The instance needs no inbound rules at all, just the `AmazonSSMManagedInstanceCore` policy and outbound HTTPS:
+
+```hcl
+resource "aws_instance" "access" {
+ ami = data.aws_ssm_parameter.al2023_arm64.value
+ instance_type = "t4g.nano"
+ subnet_id = module.vpc.private_subnets[0]
+ iam_instance_profile = aws_iam_instance_profile.access.name
+ vpc_security_group_ids = [aws_security_group.access.id]
+
+ metadata_options {
+ http_tokens = "required"
+ }
+
+ tags = {
+ Name = "${local.name}-access"
+ }
+}
+```
+
+Two more changes go into the EKS module: a security group rule letting that instance reach the API server, and an
+access entry for the people using it. Here that's an IAM Identity Center permission set role, so access follows group
+membership in your identity provider:
+
+```hcl
+ access_entries = {
+ # ...bootstrap, from part 1
+
+ platform_admins = {
+ principal_arn = local.platform_admin_role_arn
+ policy_associations = {
+ admin = {
+ policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
+ access_scope = { type = "cluster" }
+ }
+ }
+ }
+ }
+```
+
+Note that the instance itself has **no Kubernetes permissions**. It only forwards TCP. Authentication still happens with
+the person's own IAM role, through `aws eks get-token`, so the audit trail in the EKS logs shows who did what, not
+"the bastion".
+
+The tunnel has one wrinkle. kubectl connects to `localhost`, but the API server's certificate is for its real hostname.
+`tls-server-name` keeps TLS checking the right name. The
+[full script](https://github.com/HeyalSolutions/eks-private-argocd-bootstrap/blob/main/scripts/kubectl-tunnel.sh)
+boils down to:
+
+```bash
+host=$(aws eks describe-cluster --name platform --query cluster.endpoint --output text)
+host=${host#https://}
+
+aws eks update-kubeconfig --name platform --kubeconfig ~/.kube/platform-tunnel
+kubectl --kubeconfig ~/.kube/platform-tunnel config set-cluster "$cluster_arn" \
+ --server https://localhost:8443 \
+ --tls-server-name "$host"
+
+aws ssm start-session \
+ --target "$instance_id" \
+ --document-name AWS-StartPortForwardingSessionToRemoteHost \
+ --parameters "host=${host},portNumber=443,localPortNumber=8443"
+```
+
+Leave that running, and in another terminal `kubectl` works as normal. For the Argo CD UI, `kubectl port-forward
+svc/argocd-server -n argocd 8080:80` over the same tunnel is enough for occasional use.
+
+## Things to watch out for
+
+- **Order isn't guaranteed.** The `ClusterSecretStore` needs a CRD that the External Secrets Application installs, but
+ everything syncs at once. `SkipDryRunOnMissingResource` stops Argo CD failing the whole sync up front, and a `retry`
+ block on the root Application tries again until the CRD exists. Sync waves don't help here, because Argo CD doesn't
+ wait for child Applications to become healthy unless you add a custom health check.
+- **Match service account names exactly.** If the Pod Identity association says `external-dns` and the chart creates
+ `external-dns-sa`, the pod quietly falls back to the node's role and gets access denied.
+- **Look after the access instance.** It has no open ports, but it's still a server. Turn on SSM Patch Manager, or
+ replace it on a schedule.
+- **Log sessions.** Session Manager can log every session to S3 or CloudWatch. Turn on EKS audit logging too, so you
+ can see what people did once they were in.
+
+## Wrapping up
+
+The cluster can now expose services privately, give them DNS names and certificates, and pull secrets from Secrets
+Manager. None of that needed CI to talk to the cluster. Terraform handled the AWS side, Argo CD handled the Kubernetes
+side, and Pod Identity joined them up without either knowing the other's details. People get in through an audited
+tunnel with their own identity, rather than a shared key.
+
+Next up: [keeping it upgraded](/blog/eks-upgrades-without-drama/), which on a private cluster needs a bit more thought.
+
+If you'd like help building or reviewing a platform like this, that's what our
+[platform engineering](/services/platform-engineering/) work covers. [Get in touch](/contact/).
diff --git a/src/content/blog/private-eks-tailscale.md b/src/content/blog/private-eks-tailscale.md
new file mode 100644
index 0000000..07c07aa
--- /dev/null
+++ b/src/content/blog/private-eks-tailscale.md
@@ -0,0 +1,372 @@
+---
+title: "Cluster access for a growing team with Tailscale"
+description: "When SSM tunnels and AWS roles for everyone stop scaling: kubectl and internal tools on a private EKS cluster through the Tailscale operator, with Kubernetes RBAC from your identity provider and no stored secrets."
+pubDate: 2026-11-14
+heroImage: ../../assets/blog/generated/private-eks-tailscale.jpg
+heroAlt: "Several laptops outside a padlocked VPC boundary, each joined to the cluster inside by its own line through a single gateway"
+draft: true
+---
+
+In [Day 2 on a private EKS cluster](/blog/private-eks-day-2/) we gave people a way into a cluster with no public API
+endpoint: a small instance with no open ports, and a Session Manager tunnel through it. Every session is tied to an IAM
+role and logged in CloudTrail. For a platform team of three, all of whom already have AWS access, it's ideal.
+
+Then the team grows. Thirty developers want to check their pods' logs and open the Argo CD UI. Giving each of them an
+AWS role just to run `kubectl get pods` means more IAM to manage and more people who can do things in your AWS account.
+And asking them all to install the Session Manager plugin and keep a tunnel running in a spare terminal gets old fast.
+
+This post replaces that day-to-day access with [Tailscale](https://tailscale.com). People sign in with your identity
+provider, and get `kubectl` access with the Kubernetes permissions their group allows, plus a route to internal load
+balancers. They don't need an AWS account, a VPN client config or a tunnel. Everything is installed by Argo CD and
+configured with Terraform, and there isn't a single stored secret involved.
+
+The code is in the same example repo,
+[HeyalSolutions/eks-private-argocd-bootstrap](https://github.com/HeyalSolutions/eks-private-argocd-bootstrap).
+
+## How it fits together
+
+The [Tailscale Kubernetes operator](https://tailscale.com/kb/1236/kubernetes-operator) runs inside the cluster and does
+two jobs for us:
+
+- **An API server proxy.** It joins the tailnet as a device. When someone sends it a `kubectl` request, it checks who
+ they are on the tailnet and forwards the request to the API server, *impersonating* that person. Kubernetes RBAC then
+ decides what they can do, as if they'd authenticated directly.
+- **A subnet router.** It advertises the VPC's address range to the tailnet, so people can reach internal load
+ balancers like the one in front of Argo CD.
+
+
+
+
+
+Both run as pods in the cluster, installed by Argo CD like every other add-on, so there are no extra instances to
+patch. Who can use them is set in the **tailnet policy**, which we'll manage with Terraform next to everything else.
+
+## Step 1: no secrets, anywhere
+
+Two things need to talk to the Tailscale API: Terraform in GitHub Actions, which manages the policy, and the operator,
+which registers devices on the tailnet. The usual way to authenticate them is an OAuth client secret. Tailscale now
+supports **workload identity federation**, which works like the GitHub-to-AWS OIDC setup from
+[part 1](/blog/bootstrapping-private-eks-argocd/#step-1-let-github-actions-into-aws-without-keys). The workload shows
+Tailscale a signed token from its own platform, and gets a short-lived API token back.
+
+For GitHub Actions, the federated identities go in the one-time `bootstrap/` stack, applied from your machine. As in
+part 1, there are two: one that can make changes, trusted only on `main`, and a read-only one for pull request plans:
+
+```hcl
+resource "tailscale_federated_identity" "github_apply" {
+ description = "GitHub Actions apply on main"
+ issuer = "https://token.actions.githubusercontent.com"
+ subject = "repo:${local.github_repo}:ref:refs/heads/main"
+ scopes = [
+ "policy_file",
+ "devices:core:read",
+ "devices:posture_attributes",
+ "dns",
+ "federated_keys", # to create the operator's identity
+ ]
+}
+
+resource "tailscale_federated_identity" "github_plan" {
+ description = "GitHub Actions plan on pull requests"
+ issuer = "https://token.actions.githubusercontent.com"
+ subject = "repo:${local.github_repo}:pull_request"
+ scopes = ["policy_file:read", "devices:core:read", "devices:posture_attributes:read", "dns:read", "federated_keys:read"]
+}
+```
+
+Applying that needs a Tailscale API key once, in `TAILSCALE_API_KEY`. Revoke it afterwards. The workflow then picks
+the right identity for the event. Neither value is sensitive, so they're repository variables, not secrets:
+
+```yaml
+ env:
+ TAILSCALE_OAUTH_CLIENT_ID: ${{ github.event_name == 'pull_request' && vars.TAILSCALE_PLAN_CLIENT_ID || vars.TAILSCALE_APPLY_CLIENT_ID }}
+ TAILSCALE_AUDIENCE: ${{ github.event_name == 'pull_request' && vars.TAILSCALE_PLAN_AUDIENCE || vars.TAILSCALE_APPLY_AUDIENCE }}
+```
+
+With those set, the Tailscale Terraform provider finds the job's GitHub OIDC token on its own. The `id-token: write`
+permission from part 1 already covers it.
+
+The operator is neater still. Every EKS cluster has an OIDC issuer for its service account tokens, and its signing keys
+are published on a public AWS URL, even when the API server itself is private. So Tailscale can verify a token signed by
+the cluster without being able to reach it. The operator's federated identity trusts exactly one service account:
+
+```hcl
+resource "tailscale_federated_identity" "operator" {
+ description = "Tailscale operator on ${local.name}"
+ issuer = module.eks.cluster_oidc_issuer_url
+ subject = "system:serviceaccount:tailscale:operator"
+ scopes = ["devices:core", "auth_keys", "services"]
+ tags = ["tag:k8s-operator"]
+}
+```
+
+## Step 2: the tailnet policy
+
+The tailnet policy decides who can reach what. Managing it with the `tailscale_acl` resource means access changes go
+through pull requests and show up in a plan, just like IAM.
+
+```hcl
+resource "tailscale_acl" "this" {
+ overwrite_existing_content = true
+
+ acl = jsonencode({
+ groups = {
+ "group:platform" = ["alice@example.com"]
+ "group:developers" = ["bob@example.com", "carol@example.com"]
+ }
+
+ tagOwners = {
+ "tag:k8s-operator" = []
+ "tag:k8s" = ["tag:k8s-operator"]
+ }
+
+ autoApprovers = {
+ routes = {
+ (local.vpc_cidr) = ["tag:k8s"]
+ }
+ }
+
+ grants = [
+ # kubectl through the API server proxy, as a Kubernetes group
+ {
+ src = ["group:developers"]
+ dst = ["tag:k8s-operator"]
+ ip = ["tcp:443"]
+ app = {
+ "tailscale.com/cap/kubernetes" = [{ impersonate = { groups = ["developers"] } }]
+ }
+ },
+ # ...the same for group:platform, impersonating platform-admins
+
+ # Internal load balancers and DNS through the subnet router
+ {
+ src = ["group:platform", "group:developers"]
+ dst = [local.vpc_cidr]
+ ip = ["tcp:443", "udp:53", "tcp:53"]
+ },
+ ]
+
+ tests = [
+ { src = "bob@example.com", accept = ["10.0.0.10:443"], deny = ["10.0.0.10:22"] },
+ ]
+ })
+}
+```
+
+There's a lot packed into that, so in order:
+
+- **Groups** are lists of people. In a real setup, sync them from your identity provider with SCIM, so leavers lose
+ access when they leave the directory, not when someone remembers to edit a list.
+- **Tag owners** let the operator tag the devices it creates. The operator itself is `tag:k8s-operator`, and the
+ devices it creates (like the subnet router) are `tag:k8s`.
+- **Auto approvers** let the subnet router advertise the VPC range without someone clicking approve in the admin
+ console.
+- **Grants** do the real work. The first one is the interesting part: it lets developers reach the API server proxy, and
+ the `tailscale.com/cap/kubernetes` capability tells the proxy to add the `developers` Kubernetes group when it
+ impersonates them. The tailnet decides which group you're in, and Kubernetes RBAC decides what that group can do.
+- **Tests** run every time the policy changes. If an edit would let a developer SSH into the VPC, Tailscale rejects it.
+
+The resource owns the whole policy file. Anything someone changes in the admin console gets put back on the next
+apply, which is the point.
+
+## Step 3: the operator, installed by Argo CD
+
+The operator is one more Application in the `cluster-resources` chart. The client ID and audience come from a Terraform
+output and get committed to `values.yaml`, the same way as the VPC ID in the Day 2 post:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+ name: tailscale-operator
+ namespace: argocd
+spec:
+ project: default
+ source:
+ repoURL: https://pkgs.tailscale.com/helmcharts
+ chart: tailscale-operator
+ targetRevision: {{ .Values.tailscale.chartVersion | quote }}
+ helm:
+ valuesObject:
+ # Workload identity federation: no client secret anywhere
+ oauth:
+ clientId: {{ .Values.tailscale.operatorClientId | quote }}
+ audience: {{ .Values.tailscale.operatorAudience | quote }}
+ operatorConfig:
+ hostname: {{ .Values.cluster.name }}-k8s
+ apiServerProxyConfig:
+ mode: "true" # impersonate the caller's tailnet identity
+ destination:
+ server: https://kubernetes.default.svc
+ namespace: tailscale
+ syncPolicy:
+ automated:
+ prune: true
+ selfHeal: true
+ syncOptions:
+ - CreateNamespace=true
+ - ServerSideApply=true
+```
+
+With `audience` set, the chart mounts a service account token with that audience into the operator's pod, and the
+operator exchanges it for a Tailscale API token. There's nothing in Secrets Manager and nothing to rotate.
+
+`apiServerProxyConfig.mode: "true"` turns on the API server proxy in **auth mode**, and gives the operator permission to
+impersonate users and groups. (There's also a `noauth` mode, which forwards requests without any identity. Don't use it
+for this.)
+
+## Step 4: Kubernetes RBAC for tailnet groups
+
+The proxy impersonates each person as their Tailscale login name, plus whatever groups their grants add. Those groups
+mean nothing to Kubernetes until something binds them to a role:
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: tailscale-developers
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: view
+subjects:
+ - apiGroup: rbac.authorization.k8s.io
+ kind: Group
+ name: developers
+```
+
+`platform-admins` gets `cluster-admin` the same way. `view` is a sensible default for developers: they can see pods,
+logs and events, but not Secrets, and they can't change anything. Changes still go through Git and Argo CD. If teams
+need more in their own namespaces, add RoleBindings there, in the same chart.
+
+Because every request is impersonated, the EKS audit log records the real person (`bob@example.com`), not the operator.
+You get the same per-person trail you had with SSM.
+
+## Step 5: a route to internal services
+
+The API server proxy covers `kubectl`. For the Argo CD UI and anything else behind the internal ALB, a `Connector`
+tells the operator to run a subnet router:
+
+```yaml
+apiVersion: tailscale.com/v1alpha1
+kind: Connector
+metadata:
+ name: vpc
+ annotations:
+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+ hostnamePrefix: {{ .Values.cluster.name }}-vpc
+ replicas: 2
+ subnetRouter:
+ advertiseRoutes:
+ - {{ .Values.tailscale.vpcCidr }}
+```
+
+Two replicas give you high availability. If one pod is rescheduled, clients fail over to the other. (Use
+`hostnamePrefix` rather than `hostname` with more than one replica, since each replica needs its own name.)
+
+The internal zone from the Day 2 post is a public Route 53 zone, so `argocd.internal.example.com` already resolves
+anywhere. It's just that the private IP it resolves to is only reachable through the subnet router. If you use a
+private hosted zone instead, add **split DNS** so laptops ask the VPC's resolver about that one domain:
+
+```hcl
+resource "tailscale_dns_split_nameservers" "internal" {
+ domain = local.domain
+ nameservers = [cidrhost(local.vpc_cidr, 2)] # the VPC's built-in resolver
+}
+```
+
+There's another way to do this. The operator can also expose individual Services on the tailnet with its own Ingress
+class, each with its own name and its own grants. That's tighter, since people get one service rather than port 443
+across the whole VPC. But it means changing every Ingress. The subnet router reuses everything from the Day 2 post as it
+is, which makes it the easier place to start.
+
+## What it looks like for a developer
+
+Install Tailscale, sign in with your company SSO, and then:
+
+```bash
+tailscale configure kubeconfig platform-k8s
+kubectl get pods -n my-team
+```
+
+That's it. The kubeconfig points at the operator's tailnet name, and Tailscale handles authentication, so there's no
+`aws eks get-token` and no AWS profile. `https://argocd.internal.example.com` opens in a browser. And when someone
+leaves, removing them from the group in your identity provider removes their access to all of it.
+
+## Keep SSM as the break-glass
+
+Tailscale is now in the access path for day-to-day work, and so is the operator running inside the cluster. If the
+cluster is unhealthy enough that the operator isn't running, or Tailscale itself is having a bad day, nobody can get in
+through it.
+
+That's exactly when you need to get in. So keep the SSM instance and the IAM access entry from the Day 2 post, for the
+platform team only. It costs a few dollars a month and doesn't depend on anything inside the cluster. Day-to-day access
+goes through Tailscale, and break-glass goes through AWS.
+
+## Things to watch out for
+
+- **One proxy is a single point of failure.** The proxy in this post runs inside the operator pod. For anything
+ important, run it as a `ProxyGroup` of type `kube-apiserver` with several replicas instead. When you do, set
+ `apiServerProxyConfig.mode` back to `"false"` and `allowImpersonation` to `"true"`, so the ProxyGroup still gets
+ permission to impersonate.
+- **Network access isn't app access.** Being able to reach Argo CD doesn't log you in to it. Set up SSO for Argo CD too,
+ and map the same groups to Argo CD roles.
+- **The policy resource owns the whole file.** `overwrite_existing_content = true` replaces whatever's there on the
+ first apply. Copy any rules you already have into Terraform before you turn it on.
+- **Consider recording `kubectl exec`.** The operator can run a `Recorder` that records `exec` and `attach` sessions to
+ S3. It's useful for regulated environments, where "who did what in a shell" matters as much as "who called the API".
+- **Pin the operator version.** The chart and the operator move quickly. Bump the version in a pull request, read the
+ changelog, and let Argo CD roll it out, like every other add-on.
+
+## Wrapping up
+
+The cluster still has no public endpoint, and CI still never talks to it. But anyone in the right group in your
+identity provider can now run `kubectl` with sensible permissions and open internal tools in a browser, without an AWS
+account, a tunnel or a client config. The access rules live in Terraform, the operator is installed by Argo CD, and
+neither needs a stored secret.
+
+If you're working out how to give a growing team safe access to private infrastructure, we'd be happy to help. See our
+[platform engineering](/services/platform-engineering/) work or [get in touch](/contact/).