diff --git a/.github/workflows/helm-template-check.yml b/.github/workflows/helm-template-check.yml index df5e251..e8cc1ca 100644 --- a/.github/workflows/helm-template-check.yml +++ b/.github/workflows/helm-template-check.yml @@ -40,6 +40,17 @@ jobs: --set service.fqdn=https://example.codetogether.io \ --set imageCredentials.username=test \ --set imageCredentials.password=test + helm template test ./charts/ai \ + --set-file ctaiPropertiesFile=./charts/ai/ctai-configuration.properties.template \ + --set service.fqdn=https://example.codetogether.io \ + --set imageCredentials.username=test \ + --set imageCredentials.password=test \ + --set ingress.enabled=false \ + --set gateway.enabled=true \ + --set 'gateway.parentRefs[0].name=shared-gateway' \ + --set 'gateway.parentRefs[0].namespace=gateway-system' \ + --api-versions gateway.networking.k8s.io/v1/HTTPRoute \ + --api-versions gateway.nginx.org/v1alpha1/ClientSettingsPolicy - name: Validate Live Chart run: | diff --git a/charts/ai/Chart.yaml b/charts/ai/Chart.yaml index 71962b1..9817f81 100644 --- a/charts/ai/Chart.yaml +++ b/charts/ai/Chart.yaml @@ -3,7 +3,7 @@ name: codetogether-ai description: CodeTogether AI on-premises Kubernetes deployment type: application -version: 0.1.0 +version: 0.2.0 appVersion: "0.1.1204" icon: https://www.codetogether.com/wp-content/uploads/2020/02/codetogether-circle-128.png diff --git a/charts/ai/README.md b/charts/ai/README.md index 565cd4f..7acccaa 100644 --- a/charts/ai/README.md +++ b/charts/ai/README.md @@ -13,6 +13,8 @@ storage master key is stored in a separate Kubernetes Secret. - PostgreSQL reachable from the cluster. This chart does not install Postgres. - A DNS-routable HTTPS host for CodeTogether AI, configured as `service_fqdn`. - A TLS Secret for Ingress when `ingress.enabled=true`. +- Or, instead of an Ingress controller, Gateway API CRDs and a `Gateway` when + `gateway.enabled=true` (see "Gateway API" below). - CodeTogether-provided credentials for `hub.edge.codetogether.com`. ## PostgreSQL Prerequisites @@ -161,6 +163,54 @@ explicitly: If you set `ingress.className=codetogether-nginx`, the chart also creates a matching `IngressClass`. +### Gateway API (NGINX Gateway Fabric) + +Clusters that route with Gateway API instead of an Ingress controller (for +example NGINX Gateway Fabric, which replaces the retired ingress-nginx) can have +the chart attach an `HTTPRoute` to an existing `Gateway`: + +```yaml +ingress: + enabled: false +gateway: + enabled: true + parentRefs: + - name: shared-gateway # your Gateway + namespace: gateway-system # omit if it is in the release namespace + sectionName: https # the HTTPS listener with the certificate +``` + +The chart renders: + +- an `HTTPRoute` for the `service_fqdn` host, `/` to the chart Service; +- with NGINX Gateway Fabric, a `ClientSettingsPolicy` that raises the request + body limit to `16m` (`gateway.nginxGatewayFabric.clientSettingsPolicy`). It + replaces the Ingress annotation `nginx.ingress.kubernetes.io/proxy-body-size`; + without it, tracker uploads over 1MB fail with 413. With `enabled: auto` it is + rendered only when the cluster serves that API. For another Gateway API + implementation, set the equivalent limit on your side. + +The chart doesn't create the `Gateway`, its listeners or certificates. On the +Gateway side, check that: + +- the listener allows routes from the release namespace + (`allowedRoutes.namespaces`); +- the listener can read its certificate Secret. A Secret in another namespace + needs a `ReferenceGrant` there. +- the route is accepted: + `kubectl get httproute -o jsonpath='{.status.parents[*].conditions}'` + (the install notes print the exact command). + +### Client IP addresses + +`clientIp.trustedProxyHops` (default `2`: Ingress controller or Gateway, plus the +nginx in the pod) tells the server how many proxies in front of the pod append +to `X-Forwarded-For`. Add one for each extra proxy in front (load balancer, WAF, +corporate proxy). Set it to exactly the proxies you control: too low records a +proxy's address, and too high can record an address the client put in its own +`X-Forwarded-For` when your outermost proxy keeps that header. A +`CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS` entry in `extraEnvVars` takes precedence. + ## Configuration The following table lists the primary configurable parameters and defaults. @@ -193,6 +243,12 @@ The following table lists the primary configurable parameters and defaults. | `ingress.enabled` | Create an Ingress resource | `true` | | `ingress.annotations` | Ingress annotations | `nginx.ingress.kubernetes.io/proxy-body-size: "16m"` | | `ingress.tls.secretName` | TLS Secret for Ingress | `codetogether-ai-tls` | +| `gateway.enabled` | Create a Gateway API `HTTPRoute` | `false` | +| `gateway.parentRefs` | Gateway(s) the `HTTPRoute` attaches to; required when enabled | `[]` | +| `gateway.annotations` / `gateway.labels` | Extra `HTTPRoute` metadata | `{}` | +| `gateway.nginxGatewayFabric.clientSettingsPolicy.enabled` | NGINX Gateway Fabric body-size policy: `auto`, `true` or `false` | `auto` | +| `gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize` | Request body limit on the Gateway | `16m` | +| `clientIp.trustedProxyHops` | Proxies in front of the pod that append to `X-Forwarded-For` | `2` | | `serviceAccount.create` | Create a ServiceAccount | `true` | | `serviceAccount.name` | ServiceAccount name | `codetogether-ai` | | `replicaCount` | Deployment replica count | `1` | diff --git a/charts/ai/templates/NOTES.txt b/charts/ai/templates/NOTES.txt index 07dbbff..03371c9 100644 --- a/charts/ai/templates/NOTES.txt +++ b/charts/ai/templates/NOTES.txt @@ -1,7 +1,6 @@ Application URL: -{{- if .Values.ingress.enabled }} - {{- $host := (urlParse (include "ctai.serviceFqdn" .)).host }} - https://{{ $host }}/ +{{- if or .Values.ingress.enabled .Values.gateway.enabled }} + https://{{ include "ctai.publicHost" . }}/ {{- else }} Use: kubectl port-forward svc/{{ include "ctai.fullname" . }} 1080:{{ .Values.service.port }} {{- end }} @@ -34,3 +33,28 @@ Protected storage master key: Production: run ./generate-master-key.sh in this chart directory before install. {{- end }} {{- end }} + +{{- if .Values.gateway.enabled }} + +Gateway API: + {{- $parents := list }} + {{- range .Values.gateway.parentRefs }} + {{- $p := .name }} + {{- with .namespace }}{{ $p = printf "%s (namespace %s)" $p . }}{{ end }} + {{- with .sectionName }}{{ $p = printf "%s listener %s" $p . }}{{ end }} + {{- $parents = append $parents $p }} + {{- end }} + HTTPRoute {{ include "ctai.fullname" . }} attaches to: {{ join ", " $parents }} + Check it was accepted: kubectl get httproute {{ include "ctai.fullname" . }} -n {{ .Release.Namespace }} -o jsonpath='{.status.parents[*].conditions}' +{{- if eq (include "ctai.renderClientSettingsPolicy" .) "true" }} + Request body limit: ClientSettingsPolicy {{ include "ctai.fullname" . }} ({{ .Values.gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize }}). +{{- else }} + WARNING: no request body limit was rendered for the Gateway. OTLP uploads over + the gateway's default (1MB on NGINX Gateway Fabric) will fail with 413. Set the + equivalent limit for your Gateway implementation. +{{- end }} +{{- if .Values.ingress.enabled }} + WARNING: ingress.enabled is also true, so both an Ingress and an HTTPRoute + serve this host. Set ingress.enabled=false once the Gateway path works. +{{- end }} +{{- end }} diff --git a/charts/ai/templates/_helpers.tpl b/charts/ai/templates/_helpers.tpl index ccff0b2..7422a00 100644 --- a/charts/ai/templates/_helpers.tpl +++ b/charts/ai/templates/_helpers.tpl @@ -229,3 +229,63 @@ Host:port for CTAI_SERVICE_EXTERNAL_IDENTIFIER from the resolved service URL. {{- $parsed.host -}} {{- end -}} {{- end }} + +{{/* +Proxies in front of the pod (Ingress controller or Gateway + in-pod nginx), +passed to the server as CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS. +*/}} +{{- define "ctai.trustedProxyHops" -}} +{{- $clientIp := .Values.clientIp | default dict -}} +{{- $hops := 2 -}} +{{- if hasKey $clientIp "trustedProxyHops" -}} +{{- $hops = $clientIp.trustedProxyHops -}} +{{- end -}} +{{- /* Validate the value as given: int would turn 2.9 into 2 and true into 1. */ -}} +{{- if or (kindIs "bool" $hops) (not (regexMatch "^[0-9]+$" (toString $hops))) -}} +{{- fail (printf "clientIp.trustedProxyHops must be a whole number of at least 1 (the in-pod nginx is always a hop), got %v" $hops) -}} +{{- end -}} +{{- $n := int $hops -}} +{{- if lt $n 1 -}} +{{- fail (printf "clientIp.trustedProxyHops must be a whole number of at least 1 (the in-pod nginx is always a hop), got %v" $hops) -}} +{{- end -}} +{{- $n -}} +{{- end }} + +{{/* +Public host (with any port) from service_fqdn, as shown in the application URL. +*/}} +{{- define "ctai.publicHost" -}} +{{- (urlParse (include "ctai.serviceFqdn" .)).host -}} +{{- end }} + +{{/* +HTTPRoute hostname: the public host without a port (Gateway API hostnames +can't carry one). +*/}} +{{- define "ctai.routeHostname" -}} +{{- regexReplaceAll ":[0-9]+$" (include "ctai.publicHost" .) "" -}} +{{- end }} + +{{/* +"true" when extraEnvVars already sets CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS, so +the chart doesn't render a duplicate. +*/}} +{{- define "ctai.trustedProxyHopsInExtraEnv" -}} +{{- range .Values.extraEnvVars -}} +{{- if eq (toString .name) "CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS" -}}true{{- end -}} +{{- end -}} +{{- end }} + +{{/* +Whether to render the NGINX Gateway Fabric ClientSettingsPolicy: "true" or "". +*/}} +{{- define "ctai.renderClientSettingsPolicy" -}} +{{- if .Values.gateway.enabled -}} +{{- $mode := toString ((.Values.gateway.nginxGatewayFabric | default dict).clientSettingsPolicy | default dict).enabled -}} +{{- if eq $mode "true" -}} +true +{{- else if and (eq $mode "auto") (.Capabilities.APIVersions.Has "gateway.nginx.org/v1alpha1/ClientSettingsPolicy") -}} +true +{{- end -}} +{{- end -}} +{{- end }} diff --git a/charts/ai/templates/clientsettingspolicy.yaml b/charts/ai/templates/clientsettingspolicy.yaml new file mode 100644 index 0000000..08f9e54 --- /dev/null +++ b/charts/ai/templates/clientsettingspolicy.yaml @@ -0,0 +1,18 @@ +{{- if eq (include "ctai.renderClientSettingsPolicy" .) "true" }} +{{- $fullName := include "ctai.fullname" . }} +# NGINX Gateway Fabric: request body limit for this route (replaces the Ingress +# annotation nginx.ingress.kubernetes.io/proxy-body-size). +apiVersion: gateway.nginx.org/v1alpha1 +kind: ClientSettingsPolicy +metadata: + name: {{ $fullName }} + labels: + {{- include "ctai.labels" . | nindent 4 }} +spec: + targetRef: + group: gateway.networking.k8s.io + kind: HTTPRoute + name: {{ $fullName }} + body: + maxSize: {{ .Values.gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize | quote }} +{{- end }} diff --git a/charts/ai/templates/deployment.yaml b/charts/ai/templates/deployment.yaml index 6e34fa3..dd87f27 100644 --- a/charts/ai/templates/deployment.yaml +++ b/charts/ai/templates/deployment.yaml @@ -71,6 +71,11 @@ spec: key: {{ .Values.masterKey.existingSecretVersionKey | default "CTAI_PROTECTED_STORAGE_MASTER_KEY_VERSION" }} optional: true {{- end }} + {{- if not (include "ctai.trustedProxyHopsInExtraEnv" .) }} + # X-Forwarded-For trust: see clientIp in values.yaml. + - name: CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS + value: {{ include "ctai.trustedProxyHops" . | quote }} + {{- end }} {{- with .Values.extraEnvVars }} {{- toYaml . | nindent 12 }} {{- end }} diff --git a/charts/ai/templates/httproute.yaml b/charts/ai/templates/httproute.yaml new file mode 100644 index 0000000..af005aa --- /dev/null +++ b/charts/ai/templates/httproute.yaml @@ -0,0 +1,35 @@ +{{- if .Values.gateway.enabled }} +{{- if not .Values.gateway.parentRefs }} +{{- fail "gateway.enabled=true requires gateway.parentRefs: the Gateway (name, and namespace/sectionName if needed) the HTTPRoute attaches to." }} +{{- end }} +{{- if not (.Capabilities.APIVersions.Has "gateway.networking.k8s.io/v1/HTTPRoute") }} +{{- fail "gateway.enabled=true but the cluster does not serve gateway.networking.k8s.io/v1 HTTPRoute. Install the Gateway API CRDs (or pass --api-versions gateway.networking.k8s.io/v1/HTTPRoute to helm template)." }} +{{- end }} +{{- $fullName := include "ctai.fullname" . }} +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: {{ $fullName }} + labels: + {{- include "ctai.labels" . | nindent 4 }} + {{- with .Values.gateway.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.gateway.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + parentRefs: + {{- toYaml .Values.gateway.parentRefs | nindent 4 }} + hostnames: + - {{ include "ctai.routeHostname" . | quote }} + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - name: {{ $fullName }} + port: {{ .Values.service.port }} +{{- end }} diff --git a/charts/ai/values.yaml b/charts/ai/values.yaml index 5ae934f..4f9f431 100644 --- a/charts/ai/values.yaml +++ b/charts/ai/values.yaml @@ -78,6 +78,60 @@ ingress: tls: secretName: codetogether-ai-tls +# Gateway API (gateway.networking.k8s.io/v1), for clusters that route with a +# Gateway instead of an Ingress controller (for example NGINX Gateway Fabric, +# replacing the retired ingress-nginx). Off by default. +# +# The chart attaches an HTTPRoute to a Gateway you already run; it does not +# create the Gateway, its listeners or its certificate. Set ingress.enabled=false +# when you switch, unless you deliberately run both during a migration. +gateway: + enabled: false + # Gateway(s) the HTTPRoute attaches to. Required when enabled. Example: + # parentRefs: + # - name: shared-gateway + # namespace: gateway-system + # sectionName: https # the HTTPS listener that holds the certificate + parentRefs: [] + annotations: {} + labels: {} + # NGINX Gateway Fabric only. Replaces the Ingress annotation + # nginx.ingress.kubernetes.io/proxy-body-size: OTLP telemetry batches exceed + # the 1MB default and would be rejected with 413 before reaching the backend. + nginxGatewayFabric: + clientSettingsPolicy: + # auto: render only when the cluster serves + # gateway.nginx.org/v1alpha1 ClientSettingsPolicy + # true: always render (e.g. `helm template` without cluster access) + # false: never render (another Gateway API implementation) + enabled: auto + bodyMaxSize: "16m" + +# Client address handling (X-Forwarded-For trust). +# +# The server counts proxies from the right-hand end of +# `X-Forwarded-For ++ [socket peer]` and records the entry just past the proxies +# it trusts. Set the count to exactly the proxies in front of the pod that you +# control and that append to X-Forwarded-For: +# +# 2 your Ingress controller or Gateway, plus the nginx inside the pod. +# Correct for a stock Kubernetes install (the default). +# 3+ add one for each proxy in front that appends to X-Forwarded-For +# (external load balancer, WAF, corporate reverse proxy). +# 1 nothing proxies in front of the Service (for example a LoadBalancer +# Service reached directly, with no Ingress or Gateway). +# +# Match the real chain. Too low records one of your proxies' addresses. Too high +# reads past the client: if your outermost proxy keeps an incoming +# X-Forwarded-For (many Gateways append to it), that entry is whatever the +# client sent, so it can be spoofed. Only when the outermost proxy replaces the +# header does too high resolve to no address (recorded as degraded). +# Without this setting the server assumes 1 (Docker), and every address +# resolves to the Ingress controller or Gateway. +# A CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS entry in extraEnvVars takes precedence. +clientIp: + trustedProxyHops: 2 + serviceAccount: create: true annotations: {}