AITRAX-2721: codetogether-ai chart 0.2.0 - Gateway API and client IP hops (release of #216)

.github/workflows/helm-template-check.yml

@@ -40,6 +40,17 @@ jobs:
             --set service.fqdn=https://example.codetogether.io \
             --set imageCredentials.username=test \
             --set imageCredentials.password=test
+          helm template test ./charts/ai \
+            --set-file ctaiPropertiesFile=./charts/ai/ctai-configuration.properties.template \
+            --set service.fqdn=https://example.codetogether.io \
+            --set imageCredentials.username=test \
+            --set imageCredentials.password=test \
+            --set ingress.enabled=false \
+            --set gateway.enabled=true \
+            --set 'gateway.parentRefs[0].name=shared-gateway' \
+            --set 'gateway.parentRefs[0].namespace=gateway-system' \
+            --api-versions gateway.networking.k8s.io/v1/HTTPRoute \
+            --api-versions gateway.nginx.org/v1alpha1/ClientSettingsPolicy
 
       - name: Validate Live Chart
         run: |

charts/ai/Chart.yaml

@@ -3,7 +3,7 @@ name: codetogether-ai
 description: CodeTogether AI on-premises Kubernetes deployment
 
 type: application
-version: 0.1.0
+version: 0.2.0
 appVersion: "0.1.1204"
 
 icon: https://www.codetogether.com/wp-content/uploads/2020/02/codetogether-circle-128.png

charts/ai/README.md

@@ -13,6 +13,8 @@ storage master key is stored in a separate Kubernetes Secret.
 - PostgreSQL reachable from the cluster. This chart does not install Postgres.
 - A DNS-routable HTTPS host for CodeTogether AI, configured as `service_fqdn`.
 - A TLS Secret for Ingress when `ingress.enabled=true`.
+- Or, instead of an Ingress controller, Gateway API CRDs and a `Gateway` when
+  `gateway.enabled=true` (see "Gateway API" below).
 - CodeTogether-provided credentials for `hub.edge.codetogether.com`.
 
 ## PostgreSQL Prerequisites
@@ -161,6 +163,54 @@ explicitly:
 If you set `ingress.className=codetogether-nginx`, the chart also creates a
 matching `IngressClass`.
 
+### Gateway API (NGINX Gateway Fabric)
+
+Clusters that route with Gateway API instead of an Ingress controller (for
+example NGINX Gateway Fabric, which replaces the retired ingress-nginx) can have
+the chart attach an `HTTPRoute` to an existing `Gateway`:
+
+```yaml
+ingress:
+  enabled: false
+gateway:
+  enabled: true
+  parentRefs:
+    - name: shared-gateway        # your Gateway
+      namespace: gateway-system   # omit if it is in the release namespace
+      sectionName: https          # the HTTPS listener with the certificate
+```
+
+The chart renders:
+
+- an `HTTPRoute` for the `service_fqdn` host, `/` to the chart Service;
+- with NGINX Gateway Fabric, a `ClientSettingsPolicy` that raises the request
+  body limit to `16m` (`gateway.nginxGatewayFabric.clientSettingsPolicy`). It
+  replaces the Ingress annotation `nginx.ingress.kubernetes.io/proxy-body-size`;
+  without it, tracker uploads over 1MB fail with 413. With `enabled: auto` it is
+  rendered only when the cluster serves that API. For another Gateway API
+  implementation, set the equivalent limit on your side.
+
+The chart doesn't create the `Gateway`, its listeners or certificates. On the
+Gateway side, check that:
+
+- the listener allows routes from the release namespace
+  (`allowedRoutes.namespaces`);
+- the listener can read its certificate Secret. A Secret in another namespace
+  needs a `ReferenceGrant` there.
+- the route is accepted:
+  `kubectl get httproute <release-fullname> -o jsonpath='{.status.parents[*].conditions}'`
+  (the install notes print the exact command).
+
+### Client IP addresses
+
+`clientIp.trustedProxyHops` (default `2`: Ingress controller or Gateway, plus the
+nginx in the pod) tells the server how many proxies in front of the pod append
+to `X-Forwarded-For`. Add one for each extra proxy in front (load balancer, WAF,
+corporate proxy). Set it to exactly the proxies you control: too low records a
+proxy's address, and too high can record an address the client put in its own
+`X-Forwarded-For` when your outermost proxy keeps that header. A
+`CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS` entry in `extraEnvVars` takes precedence.
+
 ## Configuration
 
 The following table lists the primary configurable parameters and defaults.
@@ -193,6 +243,12 @@ The following table lists the primary configurable parameters and defaults.
 | `ingress.enabled` | Create an Ingress resource | `true` |
 | `ingress.annotations` | Ingress annotations | `nginx.ingress.kubernetes.io/proxy-body-size: "16m"` |
 | `ingress.tls.secretName` | TLS Secret for Ingress | `codetogether-ai-tls` |
+| `gateway.enabled` | Create a Gateway API `HTTPRoute` | `false` |
+| `gateway.parentRefs` | Gateway(s) the `HTTPRoute` attaches to; required when enabled | `[]` |
+| `gateway.annotations` / `gateway.labels` | Extra `HTTPRoute` metadata | `{}` |
+| `gateway.nginxGatewayFabric.clientSettingsPolicy.enabled` | NGINX Gateway Fabric body-size policy: `auto`, `true` or `false` | `auto` |
+| `gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize` | Request body limit on the Gateway | `16m` |
+| `clientIp.trustedProxyHops` | Proxies in front of the pod that append to `X-Forwarded-For` | `2` |
 | `serviceAccount.create` | Create a ServiceAccount | `true` |
 | `serviceAccount.name` | ServiceAccount name | `codetogether-ai` |
 | `replicaCount` | Deployment replica count | `1` |

charts/ai/templates/NOTES.txt

@@ -1,7 +1,6 @@
 Application URL:
-{{- if .Values.ingress.enabled }}
-  {{- $host := (urlParse (include "ctai.serviceFqdn" .)).host }}
-  https://{{ $host }}/
+{{- if or .Values.ingress.enabled .Values.gateway.enabled }}
+  https://{{ include "ctai.publicHost" . }}/
 {{- else }}
   Use: kubectl port-forward svc/{{ include "ctai.fullname" . }} 1080:{{ .Values.service.port }}
 {{- end }}
@@ -34,3 +33,28 @@ Protected storage master key:
   Production: run ./generate-master-key.sh in this chart directory before install.
 {{- end }}
 {{- end }}
+
+{{- if .Values.gateway.enabled }}
+
+Gateway API:
+  {{- $parents := list }}
+  {{- range .Values.gateway.parentRefs }}
+  {{- $p := .name }}
+  {{- with .namespace }}{{ $p = printf "%s (namespace %s)" $p . }}{{ end }}
+  {{- with .sectionName }}{{ $p = printf "%s listener %s" $p . }}{{ end }}
+  {{- $parents = append $parents $p }}
+  {{- end }}
+  HTTPRoute {{ include "ctai.fullname" . }} attaches to: {{ join ", " $parents }}
+  Check it was accepted: kubectl get httproute {{ include "ctai.fullname" . }} -n {{ .Release.Namespace }} -o jsonpath='{.status.parents[*].conditions}'
+{{- if eq (include "ctai.renderClientSettingsPolicy" .) "true" }}
+  Request body limit: ClientSettingsPolicy {{ include "ctai.fullname" . }} ({{ .Values.gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize }}).
+{{- else }}
+  WARNING: no request body limit was rendered for the Gateway. OTLP uploads over
+  the gateway's default (1MB on NGINX Gateway Fabric) will fail with 413. Set the
+  equivalent limit for your Gateway implementation.
+{{- end }}
+{{- if .Values.ingress.enabled }}
+  WARNING: ingress.enabled is also true, so both an Ingress and an HTTPRoute
+  serve this host. Set ingress.enabled=false once the Gateway path works.
+{{- end }}
+{{- end }}

charts/ai/templates/_helpers.tpl

@@ -229,3 +229,63 @@ Host:port for CTAI_SERVICE_EXTERNAL_IDENTIFIER from the resolved service URL.
 {{- $parsed.host -}}
 {{- end -}}
 {{- end }}
+
+{{/*
+Proxies in front of the pod (Ingress controller or Gateway + in-pod nginx),
+passed to the server as CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS.
+*/}}
+{{- define "ctai.trustedProxyHops" -}}
+{{- $clientIp := .Values.clientIp | default dict -}}
+{{- $hops := 2 -}}
+{{- if hasKey $clientIp "trustedProxyHops" -}}
+{{- $hops = $clientIp.trustedProxyHops -}}
+{{- end -}}
+{{- /* Validate the value as given: int would turn 2.9 into 2 and true into 1. */ -}}
+{{- if or (kindIs "bool" $hops) (not (regexMatch "^[0-9]+$" (toString $hops))) -}}
+{{- fail (printf "clientIp.trustedProxyHops must be a whole number of at least 1 (the in-pod nginx is always a hop), got %v" $hops) -}}
+{{- end -}}
+{{- $n := int $hops -}}
+{{- if lt $n 1 -}}
+{{- fail (printf "clientIp.trustedProxyHops must be a whole number of at least 1 (the in-pod nginx is always a hop), got %v" $hops) -}}
+{{- end -}}
+{{- $n -}}
+{{- end }}
+
+{{/*
+Public host (with any port) from service_fqdn, as shown in the application URL.
+*/}}
+{{- define "ctai.publicHost" -}}
+{{- (urlParse (include "ctai.serviceFqdn" .)).host -}}
+{{- end }}
+
+{{/*
+HTTPRoute hostname: the public host without a port (Gateway API hostnames
+can't carry one).
+*/}}
+{{- define "ctai.routeHostname" -}}
+{{- regexReplaceAll ":[0-9]+$" (include "ctai.publicHost" .) "" -}}
+{{- end }}
+
+{{/*
+"true" when extraEnvVars already sets CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS, so
+the chart doesn't render a duplicate.
+*/}}
+{{- define "ctai.trustedProxyHopsInExtraEnv" -}}
+{{- range .Values.extraEnvVars -}}
+{{- if eq (toString .name) "CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS" -}}true{{- end -}}
+{{- end -}}
+{{- end }}
+
+{{/*
+Whether to render the NGINX Gateway Fabric ClientSettingsPolicy: "true" or "".
+*/}}
+{{- define "ctai.renderClientSettingsPolicy" -}}
+{{- if .Values.gateway.enabled -}}
+{{- $mode := toString ((.Values.gateway.nginxGatewayFabric | default dict).clientSettingsPolicy | default dict).enabled -}}
+{{- if eq $mode "true" -}}
+true
+{{- else if and (eq $mode "auto") (.Capabilities.APIVersions.Has "gateway.nginx.org/v1alpha1/ClientSettingsPolicy") -}}
+true
+{{- end -}}
+{{- end -}}
+{{- end }}

charts/ai/templates/clientsettingspolicy.yaml

@@ -0,0 +1,18 @@
+{{- if eq (include "ctai.renderClientSettingsPolicy" .) "true" }}
+{{- $fullName := include "ctai.fullname" . }}
+# NGINX Gateway Fabric: request body limit for this route (replaces the Ingress
+# annotation nginx.ingress.kubernetes.io/proxy-body-size).
+apiVersion: gateway.nginx.org/v1alpha1
+kind: ClientSettingsPolicy
+metadata:
+  name: {{ $fullName }}
+  labels:
+    {{- include "ctai.labels" . | nindent 4 }}
+spec:
+  targetRef:
+    group: gateway.networking.k8s.io
+    kind: HTTPRoute
+    name: {{ $fullName }}
+  body:
+    maxSize: {{ .Values.gateway.nginxGatewayFabric.clientSettingsPolicy.bodyMaxSize | quote }}
+{{- end }}

charts/ai/templates/deployment.yaml

@@ -71,6 +71,11 @@ spec:
                   key: {{ .Values.masterKey.existingSecretVersionKey | default "CTAI_PROTECTED_STORAGE_MASTER_KEY_VERSION" }}
                   optional: true
             {{- end }}
+            {{- if not (include "ctai.trustedProxyHopsInExtraEnv" .) }}
+            # X-Forwarded-For trust: see clientIp in values.yaml.
+            - name: CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS
+              value: {{ include "ctai.trustedProxyHops" . | quote }}
+            {{- end }}
             {{- with .Values.extraEnvVars }}
             {{- toYaml . | nindent 12 }}
             {{- end }}

charts/ai/templates/httproute.yaml

@@ -0,0 +1,35 @@
+{{- if .Values.gateway.enabled }}
+{{- if not .Values.gateway.parentRefs }}
+{{- fail "gateway.enabled=true requires gateway.parentRefs: the Gateway (name, and namespace/sectionName if needed) the HTTPRoute attaches to." }}
+{{- end }}
+{{- if not (.Capabilities.APIVersions.Has "gateway.networking.k8s.io/v1/HTTPRoute") }}
+{{- fail "gateway.enabled=true but the cluster does not serve gateway.networking.k8s.io/v1 HTTPRoute. Install the Gateway API CRDs (or pass --api-versions gateway.networking.k8s.io/v1/HTTPRoute to helm template)." }}
+{{- end }}
+{{- $fullName := include "ctai.fullname" . }}
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: {{ $fullName }}
+  labels:
+    {{- include "ctai.labels" . | nindent 4 }}
+    {{- with .Values.gateway.labels }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+  {{- with .Values.gateway.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  parentRefs:
+    {{- toYaml .Values.gateway.parentRefs | nindent 4 }}
+  hostnames:
+    - {{ include "ctai.routeHostname" . | quote }}
+  rules:
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /
+      backendRefs:
+        - name: {{ $fullName }}
+          port: {{ .Values.service.port }}
+{{- end }}

charts/ai/values.yaml

@@ -78,6 +78,60 @@ ingress:
   tls:
     secretName: codetogether-ai-tls
 
+# Gateway API (gateway.networking.k8s.io/v1), for clusters that route with a
+# Gateway instead of an Ingress controller (for example NGINX Gateway Fabric,
+# replacing the retired ingress-nginx). Off by default.
+#
+# The chart attaches an HTTPRoute to a Gateway you already run; it does not
+# create the Gateway, its listeners or its certificate. Set ingress.enabled=false
+# when you switch, unless you deliberately run both during a migration.
+gateway:
+  enabled: false
+  # Gateway(s) the HTTPRoute attaches to. Required when enabled. Example:
+  #   parentRefs:
+  #     - name: shared-gateway
+  #       namespace: gateway-system
+  #       sectionName: https   # the HTTPS listener that holds the certificate
+  parentRefs: []
+  annotations: {}
+  labels: {}
+  # NGINX Gateway Fabric only. Replaces the Ingress annotation
+  # nginx.ingress.kubernetes.io/proxy-body-size: OTLP telemetry batches exceed
+  # the 1MB default and would be rejected with 413 before reaching the backend.
+  nginxGatewayFabric:
+    clientSettingsPolicy:
+      # auto: render only when the cluster serves
+      #       gateway.nginx.org/v1alpha1 ClientSettingsPolicy
+      # true: always render (e.g. `helm template` without cluster access)
+      # false: never render (another Gateway API implementation)
+      enabled: auto
+      bodyMaxSize: "16m"
+
+# Client address handling (X-Forwarded-For trust).
+#
+# The server counts proxies from the right-hand end of
+# `X-Forwarded-For ++ [socket peer]` and records the entry just past the proxies
+# it trusts. Set the count to exactly the proxies in front of the pod that you
+# control and that append to X-Forwarded-For:
+#
+#   2  your Ingress controller or Gateway, plus the nginx inside the pod.
+#      Correct for a stock Kubernetes install (the default).
+#   3+ add one for each proxy in front that appends to X-Forwarded-For
+#      (external load balancer, WAF, corporate reverse proxy).
+#   1  nothing proxies in front of the Service (for example a LoadBalancer
+#      Service reached directly, with no Ingress or Gateway).
+#
+# Match the real chain. Too low records one of your proxies' addresses. Too high
+# reads past the client: if your outermost proxy keeps an incoming
+# X-Forwarded-For (many Gateways append to it), that entry is whatever the
+# client sent, so it can be spoofed. Only when the outermost proxy replaces the
+# header does too high resolve to no address (recorded as degraded).
+# Without this setting the server assumes 1 (Docker), and every address
+# resolves to the Ingress controller or Gateway.
+# A CTAI_INGEST_OTEL_TRUSTED_PROXY_HOPS entry in extraEnvVars takes precedence.
+clientIp:
+  trustedProxyHops: 2
+
 serviceAccount:
   create: true
   annotations: {}