Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/manifests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,18 @@ jobs:
echo "$preset: $(grep -c '^kind:' "/tmp/$preset.yaml") resources"
done

# The CloudNativePG example's Cluster and Database are custom resources, not in the default schema
# set. They are validated against the CRD schemas in the datreeio CRDs-catalog (the second
# location: postgresql.cnpg.io/cluster_v1.json and database_v1.json), NOT skipped with
# -ignore-missing-schemas: a wrong field name in the example must fail here. -strict rejects an
# unknown field.
- name: Schemas
run: |
kubeconform -strict -summary \
-schema-location default \
-schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
deploy/k8s/simple/spire.yaml deploy/k8s/production/spire.yaml
deploy/k8s/simple/spire.yaml deploy/k8s/production/spire.yaml \
deploy/examples/cloudnative-pg/cluster.yaml

kind-install:
name: kind install (simple preset)
Expand Down
92 changes: 91 additions & 1 deletion deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,13 +126,103 @@ Point `SPIRE_OIDC_AUTH_SERVER_URL` at it and drop the `keycloak` service. The re
- **Two realm roles**, `spire-viewer` and `spire-admin`, readable from the **access** token. Reading
them from the ID token yields an operator with no roles, and every endpoint then denies.

`deploy/keycloak/realm-spire.json` is a working example of all of that.
`deploy/keycloak/realm-spire.json` is a working example of all of that, for development only: it has
users, client secrets, the password grant and `localhost` URIs. For a real identity provider, start
from `deploy/keycloak/realm-spire-production.json` (see "Install on any cluster", step 4).

**The issuer must be reachable under one name from both the browser and the containers.** An
unpinned Keycloak derives its issuer from the `Host` it was called on, so the two would disagree and
every token would fail validation. The bundled instance pins `KC_HOSTNAME` and sets
`KC_HOSTNAME_BACKCHANNEL_DYNAMIC` so front- and backchannel can differ while the issuer stays fixed.

## Install on any cluster

The Helm chart with the `production` preset, Postgres you run (or CloudNativePG), the chart's bundled
Redpanda, and an identity provider you already run. This installs the **reviewer only**: gateway,
orchestrator, review worker and dashboard. Agent work (harness reviews, `/fix`, the factory) needs the
run worker, which comes to Kubernetes with M5 (below).

**Not yet run on a real cluster.** The chart renders and its invariants are checked in CI, but no
install with external Postgres and an external identity provider has been done yet
(`docs/UNVERIFIED.md`). Do the steps in this order.

1. **Namespace.** `kubectl create namespace spire`. Everything below goes in it: a pod reads a Secret
only from its own namespace.

2. **Postgres.** Version 15 or later (the migrations use `NULLS NOT DISTINCT`). Pick one:
- **Bundled**: the `simple` preset's single Postgres. For evaluation only (no backup).
- **External**: set `postgres.host`, make the gateway role and schema with the two statements in
"The gateway's database role" above, and set `postgres.provisionGatewayRole: false` if the chart
must not make them.
- **CloudNativePG** (operator 1.26 or later): first make the gateway role's basic-auth Secret, then
apply the example:

```bash
kubectl -n spire create secret generic spire-db-gateway --type=kubernetes.io/basic-auth \
--from-literal=username=gateway --from-literal=password="$(openssl rand -hex 24)"
kubectl -n spire apply -f deploy/examples/cloudnative-pg/cluster.yaml
```

CloudNativePG makes the owner `spire` and its Secret `spire-db-app`; a `Database` object makes the
`gateway` schema, owned by `gateway`. The chart reads both passwords from those two Secrets
(`postgres.passwordSecret`, `postgres.gatewayPasswordSecret`), so each password is stored once.
Connect to the `-rw` Service, never through a Pooler in transaction mode: the orchestrator holds
a `LISTEN` session.

3. **The two Secrets.** The chart takes Secret names only ("The four secrets" above says why).

| Secret | Keys |
|---|---|
| `secrets.existingSecret` | `SPIRE_ENCRYPTION_KEYSET`, `SPIRE_OIDC_ORCHESTRATOR_SECRET`, `SPIRE_OIDC_GATEWAY_SECRET`, `SPIRE_OIDC_WORKER_SECRET`, and `POSTGRES_PASSWORD` unless `postgres.passwordSecret` is set |
| `secrets.gatewayExistingSecret` | `SPIRE_ENCRYPTION_WEBHOOK_KEYSET`, and `GATEWAY_POSTGRES_PASSWORD` unless `postgres.gatewayPasswordSecret` is set |

They must be two different Secrets. The three OIDC secrets come from step 4, so make the main Secret
after it, or update it then.

4. **The realm.** Copy `deploy/keycloak/realm-spire-production.json` and replace the two
placeholders: `SPIRE_REALM` with the realm name, `SPIRE_HOST` with the host the browser uses (no
scheme, no path). Create a **new** realm from the file (a partial import into an existing realm is
a different path). Then, for each of the three clients, open its Credentials tab, press
**Regenerate**, and copy the secret into the main Secret's matching key. Whether Keycloak makes a
secret on import when the file has none is not proven, so do not skip Regenerate. Give your
operators the realm roles `spire-viewer` and `spire-admin`.

5. **Install.** Copy `deploy/examples/cloudnative-pg/values.yaml`, replace each placeholder
(`example.invalid`, the pod CIDR in `trustedProxies`, the database host), then:

```bash
helm install spire deploy/helm/spire -n spire \
-f deploy/helm/spire/values-production.yaml -f my-values.yaml
```

With an empty `image.tag` the chart pulls its own `appVersion`: a chart from a release pulls that
release's images; this repository's chart pulls `edge`. For settings the chart does not model,
each service takes `extraEnv` and `extraEnvFrom`. The render refuses a name the chart already sets,
and on the gateway the master keyset or any reference to the main Secret. It cannot see a
different Secret that also holds the master keyset: keep that key out of anything the gateway
loads.

6. **Point the TLS terminator at the dashboard.** The `spire-ui` Service is `ClusterIP`. A terminator
in the cluster forwards to `spire-ui.spire.svc.cluster.local:8080`; for one outside it,
`ui.service.type` makes the Service a `NodePort` or `LoadBalancer` (`docs/TLS.md`, "Reaching the
dashboard from outside the cluster").

7. **Check it worked.** `deploy/e2e.sh` cannot check this install: it signs in with the password grant
and the dev realm's users, assumes the realm name `spire`, and runs `docker compose exec`. Check
these by hand:
- every pod is Ready: `kubectl -n spire get pods -l app.kubernetes.io/name=spire`;
- each service answers `/q/health/ready` with `200`, for example
`kubectl -n spire port-forward svc/spire-orchestrator 8080:8080` then
`curl -fsS http://localhost:8080/q/health/ready` (the same for `spire-gateway` and `spire-worker`);
- `https://<host>` signs you in through the identity provider and the dashboard shows your roles;
- the gateway answers a forge's webhook ping (`https://<host>/webhooks/...`);
- one review on a test pull request.

**No NetworkPolicy is shipped.** The services believe forwarded headers from `trustedProxies`, the pod
network, so on a shared cluster any pod can reach a service port and forge its apparent client
address. Add a NetworkPolicy that lets only the `spire-ui` pod reach the three services if other
workloads share the cluster (`techdebt/global/3-2-no-networkpolicy-isolates-the-services.md`).

## The factory, and why it is not on by default

The run worker — the service that executes agent runs — is behind a compose **profile**, so
Expand Down
67 changes: 67 additions & 0 deletions deploy/examples/cloudnative-pg/cluster.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# A CloudNativePG Postgres for Code Spire. An example: read it, change the names and the size, then apply
# it in the SAME namespace as Code Spire (a pod can read a Secret only from its own namespace, and the
# services read the passwords from the Secrets below):
#
# kubectl -n spire apply -f deploy/examples/cloudnative-pg/cluster.yaml
#
# NOT YET RUN on a real cluster (docs/UNVERIFIED.md). The field names are checked against the
# CloudNativePG 1.28 schemas by kubeconform in CI (.github/workflows/manifests.yml).
#
# Needs: the CloudNativePG operator 1.26 or later (the Database object's `schemas` field), and
# Postgres 15 or later (the migrations use NULLS NOT DISTINCT). The image is the operator's default
# unless you set spec.imageName.
#
# What this makes:
# - the database `spire`, owned by the role `spire`. CloudNativePG makes the role and its password and
# writes them to the basic-auth Secret `spire-db-app` (keys `username`, `password`). The chart reads
# that Secret (postgres.passwordSecret), so the password is stored once.
# - the role `gateway`, login, password from the basic-auth Secret `spire-db-gateway`, which YOU make
# first (deploy/README.md, "Install on any cluster"). The chart reads the same Secret
# (postgres.gatewayPasswordSecret).
# - the schema `gateway` in the database `spire`, owned by `gateway`. That is the only thing the
# gateway role gets: no grant on any other schema, no membership in `spire` (deploy/README.md, "The
# gateway's database role").
#
# Why a Database object and not bootstrap.initdb.postInitApplicationSQL: post-init SQL runs once, at
# bootstrap, BEFORE the managed role exists, so `CREATE SCHEMA gateway AUTHORIZATION gateway` would fail
# and the cluster would not come up. The Database object is reconciled after the role exists, and it
# works on a cluster that already exists too.
#
# Connect the services to `spire-db-rw` (the read-write Service). Do not put a transaction-mode Pooler
# between them: the orchestrator holds a LISTEN session.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: spire-db
spec:
instances: 1
storage:
size: 10Gi
# storageClass: your-storage-class # empty means the cluster's default
bootstrap:
initdb:
database: spire
owner: spire
managed:
roles:
- name: gateway
ensure: present
login: true
passwordSecret:
name: spire-db-gateway
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: spire-db-spire
spec:
cluster:
name: spire-db
name: spire
owner: spire
# Keep the database when this object is deleted.
databaseReclaimPolicy: retain
schemas:
- name: gateway
owner: gateway
ensure: present
65 changes: 65 additions & 0 deletions deploy/examples/cloudnative-pg/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Code Spire on any cluster: Postgres on CloudNativePG (cluster.yaml beside this file), the chart's
# bundled Redpanda, and an identity provider you already run. Use it on top of the production preset:
#
# helm install spire deploy/helm/spire -n spire \
# -f deploy/helm/spire/values-production.yaml \
# -f deploy/examples/cloudnative-pg/values.yaml
#
# Every value here is a placeholder (example.invalid, a sample pod CIDR). Change each one. The steps,
# in order, are in deploy/README.md, "Install on any cluster". NOT YET RUN on a real cluster
# (docs/UNVERIFIED.md).
#
# tests/render.sh renders this file with NO --set, so it must stay complete: a value the chart
# requires that is missing here fails CI.

# The two Secrets you make before the install (deploy/README.md lists their keys). With the passwords
# below read from the CloudNativePG Secrets, neither needs a database password key.
secrets:
existingSecret: spire-secrets
gatewayExistingSecret: spire-gateway-secrets

# Your cluster's pod CIDR (k3s uses 10.42.0.0/16 by default). The services believe forwarded headers only
# from this network (docs/TLS.md requirement 4).
trustedProxies: 10.244.0.0/16

# The host the browser uses. Set it whenever the Ingress is off: nothing else pins the Host that the
# login redirect is built from.
publicHost: spire.example.invalid

postgres:
bundled: false
# The CloudNativePG read-write Service: <cluster>-rw.<namespace>.svc.cluster.local. Not a Pooler in
# transaction mode: the orchestrator holds a LISTEN session.
host: spire-db-rw.spire.svc.cluster.local
database: spire
user: spire
gatewayUser: gateway
# CloudNativePG's application owner cannot CREATE ROLE, and cluster.yaml makes the role and the schema
# already, so the chart's role Job must not run.
provisionGatewayRole: false
# The basic-auth Secret CloudNativePG made for the owner `spire` (<cluster>-app).
passwordSecret:
name: spire-db-app
key: password
# The basic-auth Secret you made for the managed role `gateway` (cluster.yaml).
gatewayPasswordSecret:
name: spire-db-gateway
key: password

# The chart's single-replica Redpanda. Enough while every Kafka client is in the cluster.
kafka:
bundled: true
# storageClass: your-storage-class # empty means the cluster's default

oidc:
bundled: false
# Must be reachable from the pods AND be the issuer the browser sees. The realm:
# deploy/keycloak/realm-spire-production.json.
authServerUrl: https://auth.example.invalid/realms/spire

# No ingress controller: a TLS terminator in the cluster forwards to the spire-ui Service
# (spire-ui.<namespace>.svc.cluster.local:8080). See docs/TLS.md.
ingress:
enabled: false

# image.tag is left empty: the chart's appVersion. A chart from a release pulls that release's images.
15 changes: 13 additions & 2 deletions deploy/helm/spire/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ Dashboard:
{{- if .Values.ingress.enabled }}
http{{ if .Values.ingress.tls.enabled }}s{{ end }}://{{ .Values.ingress.host }}
{{- else }}
No Ingress. Point your TLS terminator at the spire-ui Service
(spire-ui.{{ .Release.Namespace }}.svc.cluster.local:8080, type {{ .Values.ui.service.type }}).
See docs/TLS.md. For a quick look without TLS:
kubectl port-forward svc/spire-ui 34700:8080 then http://localhost:34700
{{- end }}

Expand All @@ -33,15 +36,23 @@ is what puts all four services on one origin. Two things follow:
Check it:

kubectl get pods -l app.kubernetes.io/name=spire
{{- if .Values.oidc.bundled }}
kubectl port-forward svc/spire-ui 34700:8080
./deploy/e2e.sh http://localhost:34700 <your-idp-url>
{{- else }}

deploy/e2e.sh cannot check this install: it signs in with the bundled realm's
dev users. Use the checks in deploy/README.md, "Install on any cluster":
every pod Ready, /q/health/ready of each service, a sign-in at the dashboard
through your identity provider, a webhook ping, and one review.
{{- end }}

{{ if not .Values.oidc.bundled }}
Using your own identity provider. It must offer three confidential clients
(spire-orchestrator, spire-gateway, spire-review-worker), each with an audience
mapper, plus the roles spire-viewer and spire-admin readable from the ACCESS
token. See deploy/README.md; deploy/keycloak/realm-spire.json is a worked
example.
token. See deploy/README.md. deploy/keycloak/realm-spire-production.json is a
template for a production realm.
{{ end }}
{{- if .Values.gateway.giveMasterKeyset }}

Expand Down
Loading
Loading