diff --git a/docs/api-operator-guide.md b/docs/api-operator-guide.md index 6bd84407..ef8bf45e 100644 --- a/docs/api-operator-guide.md +++ b/docs/api-operator-guide.md @@ -26,6 +26,8 @@ A practical guide for deploying, configuring, and operating the HyperFleet API c - [Server Binding](#34-server-binding) - [Logging Configuration](#35-logging-configuration) - [Schema Validation](#36-schema-validation) + - [Tenant Enforcement](#37-tenant-enforcement) + - [Gateway and In-App JWT Modes](#38-gateway-and-in-app-jwt-modes) 4. [Deployment Checklist](#4-deployment-checklist) - [Phase 1: Database Preparation](#phase-1-database-preparation) - [Phase 2: Configuration Planning](#phase-2-configuration-planning) @@ -731,6 +733,55 @@ The API uses a two-step process to validate specs: For details on how schemas are imported for code generation and which schema components map to each resource type, see [openapi/README.md](../openapi/README.md) in this repository. +### 3.7 Tenant Enforcement + +Tenant enforcement scopes each resource read, list, update, and delete to the caller's tenant. It is **disabled by default** and is a separate concern from JWT authentication. + +**Only enable it when the API runs behind the Envoy + Authorino gateway** ([ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md)): tenant identity comes from trusted gateway-injected headers, never JWT claims, and the gateway must strip client-supplied tenant headers and block direct routes to the API pod. Without it, clients could forge tenancy. + +Enable it under `server.tenant.*` (`HYPERFLEET_SERVER_TENANT_*` for the scalar fields; dimensions are YAML/Helm-values only): + +```yaml +server: + tenant: + enabled: true + system_header: X-HyperFleet-System # value "true" marks system callers (Sentinel, adapters) + dimensions: + - header: X-HyperFleet-Org # trusted gateway-injected header + key: org # tenancy map key + required: true + - header: X-HyperFleet-Project + key: project + required: false +``` + +At runtime, once enabled: + +- **System callers** (system header value `true`, e.g. Sentinel and adapters) bypass scoping, but may only write `status`/`conditions` (reported through a separate status path). Any other resource mutation — create, update, or delete — from a system identity returns `403 Forbidden`. +- **Tenant-scoped callers** must present the configured dimension headers. A missing required dimension, an invalid value, or zero resolved dimensions is rejected with `403 Forbidden` before any database access. +- **Cross-tenant access** returns `404 Not Found` (not `403`) for reads, updates, and deletes, so a resource's existence is never leaked across tenants. + +For the field reference, environment variables, and validation rules see [Configuration Guide - Tenant Enforcement](config.md#tenant-enforcement); for the full trust model see [Tenant isolation](authentication.md#tenant-isolation). Common tenant errors are in [Appendix B: Troubleshooting](#appendix-b-troubleshooting). + +### 3.8 Gateway and In-App JWT Modes + +The gateway ([ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md)) authenticates callers and injects trusted identity/tenant headers; the API's in-app JWT middleware (`server.jwt.enabled`) validates `Bearer` tokens directly. These are independent switches, but not every combination is currently valid. + +Two `Authorization` schemes reach the gateway: human operators use `Bearer `, while machine callers (Sentinel, adapters) use `ServiceAccount `, validated by Kubernetes TokenReview at the gateway. The in-app JWT middleware accepts the `Bearer` scheme **only** — a `ServiceAccount` token presented to the API is rejected with `401 Unauthorized`. This does not block machine callers outright: a Kubernetes service-account JWT presented directly as `Bearer ` (bypassing gateway TokenReview) validates like any other issuer's token — see [Creating a service account token](authentication.md#creating-a-service-account-token). Only the gateway's `ServiceAccount ` scheme itself has no in-app equivalent. + +| `server.jwt.enabled` | `server.tenant.enabled` | Behavior | +|---|---|---| +| `false` | `false` | No auth, no scoping. Local development only (`make run-no-auth`). | +| `true` | `false` | In-app JWT validation, no tenant scoping. `Bearer` callers only — human OIDC or a Kubernetes service-account JWT presented as `Bearer`; the gateway's `ServiceAccount` scheme is not supported. | +| `true` | `true` | Full production posture behind the gateway: JWT validated, requests scoped to gateway-injected tenant headers. `Bearer` callers only; the gateway's `ServiceAccount` scheme is not supported until [HYPERFLEET-1484](https://redhat.atlassian.net/browse/HYPERFLEET-1484). | +| `false` | `true` | Gateway performs all authentication (including machine `ServiceAccount` callers); the API trusts injected headers and scopes on them. | + +A machine caller authenticating with the gateway's `ServiceAccount ` scheme cannot yet traverse the in-app JWT middleware, because it is `Bearer`-only. Until [HYPERFLEET-1484](https://redhat.atlassian.net/browse/HYPERFLEET-1484) adds `ServiceAccount`-scheme support to the in-app middleware, deployments that must authenticate such callers should keep `server.jwt.enabled: false` and let the gateway authenticate every caller (bottom row above). + +#### Mixed dimension cardinality + +Within a single deployment, all callers are expected to resolve the same set of tenant dimensions. Mixing callers that resolve different dimension sets (for example, some presenting only `org` and others presenting `org` + `project`) against the same resources is not yet supported — JSONB containment scoping (`tenancy @> caller`) would let a coarser-scoped caller match finer-scoped resources. Support for heterogeneous dimension cardinality is tracked by [HYPERFLEET-1634](https://redhat.atlassian.net/browse/HYPERFLEET-1634). + --- ## 4. Deployment Checklist @@ -1048,7 +1099,8 @@ This section provides a **quick reference** for common API-specific issues and t | **High API latency, slow responses** | Resource limits, database slow queries, or connection pool exhausted | Check metrics: `curl http://:9090/metrics \| grep hyperfleet_api_request_duration_seconds`. Check resources: `kubectl top pods -n hyperfleet-system`. Check slow queries: `kubectl logs -n hyperfleet-system deployment/hyperfleet-api \| grep "slow query"`. Resolution: Increase resource limits/replicas, add database indexes, or increase `--db-max-open-connections` (default: 50). | | **400 Bad Request** | Resource spec doesn't match OpenAPI schema | Check the loaded schema path: `kubectl logs -n hyperfleet-system deployment/hyperfleet-api \| grep "schema_path"`. Retrieve and inspect the schema: `kubectl exec -n hyperfleet-system deployment/hyperfleet-api -- cat $HYPERFLEET_SERVER_OPENAPI_SCHEMA_PATH`. Validate and fix spec. | | **401 Unauthorized** | Missing or invalid JWT token | Verify authentication is enabled (`server.jwt.enabled=true`). If production, ensure valid JWT token is provided. Reference: [Authentication Guide](authentication.md). | -| **404 Not Found** | Resource doesn't exist | Verify resource ID is correct. Check if resource was deleted: `curl http://:8000/api/hyperfleet/v1/clusters/$CLUSTER_ID`. | +| **403 Forbidden** | Tenant enforcement rejected the caller: missing/empty required dimension header, invalid dimension value, zero resolved dimensions, or a system identity attempting a resource create/update | Confirm the gateway (Envoy + Authorino) is injecting the configured dimension headers and the system header. Check config: `kubectl get configmap -config -o yaml \| grep -A6 tenant`. Verify each `required: true` dimension header is present and its value matches `^[A-Za-z0-9._-]+$` (max 63 chars). A system caller's request must carry the configured system header (`server.tenant.system_header`, e.g. `X-HyperFleet-System`) with the value `true`; system callers may only write status/conditions. See [Tenant isolation](authentication.md#tenant-isolation). | +| **404 Not Found** | Resource doesn't exist, or (with tenant enforcement enabled) the resource belongs to a different tenant | Verify resource ID is correct. Check if resource was deleted: `curl http://:8000/api/hyperfleet/v1/clusters/$CLUSTER_ID`. If tenant enforcement is enabled, confirm the caller's dimension headers scope to the resource's tenancy — cross-tenant resources return 404 by design. | | **409 Conflict** | Concurrent update or generation mismatch | Retry with exponential backoff. Ensure only one controller updates the same resource. | | **500 Internal Server Error** | Database error or unexpected panic | Check API logs: `kubectl logs -n hyperfleet-system -l app=hyperfleet-api --tail=100`. Verify database connectivity with `/readyz` endpoint. | | **503 Service Unavailable** | Readiness probe failing | Check readiness: `curl http://:8080/readyz`. Verify database connectivity and API initialization. Check logs for startup errors. | diff --git a/docs/api-resources.md b/docs/api-resources.md index 9c8ddb82..090b41e0 100644 --- a/docs/api-resources.md +++ b/docs/api-resources.md @@ -10,6 +10,17 @@ Mutating requests (POST, PATCH, PUT, DELETE) additionally require a resolvable c > **Note**: The API does not enforce role-based access control (RBAC). Any authenticated caller can invoke any endpoint, including destructive operations like force-delete. Access control should be enforced at the infrastructure layer (e.g., ingress policies, gateway authorization). +### Tenant scoping + +When tenant enforcement is enabled (`server.tenant.enabled=true`), reads, lists, updates, and deletes are scoped to the caller's tenant, which is resolved from trusted gateway-injected headers (never from JWT claims). This affects API behavior: + +- A resource that belongs to a different tenant returns `404 Not Found`, not `403`, on `GET`, `PATCH`/update, and `DELETE` — cross-tenant existence is never leaked. +- List endpoints return only resources within the caller's tenancy; `total` reflects the scoped result set. +- A non-system caller missing a required tenant dimension header (or presenting an invalid value) is rejected with `403 Forbidden` before the request reaches any resource. +- The `tenancy` field is server-populated on create from the caller's resolved dimensions; it is read-only and any `tenancy` supplied in a create or patch body is ignored. + +System callers (e.g. Sentinel, adapters) bypass scoping but may only write `status`/`conditions` — any other resource mutation (create, update, or delete) from a system identity is rejected with `403 Forbidden`. See [Tenant isolation](authentication.md#tenant-isolation) for details. + ## Cluster Management ### Endpoints @@ -649,6 +660,7 @@ See **[search.md](search.md)** for complete documentation. - `updated_time` - When resource was last updated (API-managed) - `created_by` - User who created the resource (email) - `updated_by` - User who last updated the resource (email) +- `tenancy` - Tenant dimensions the resource belongs to (server-populated, read-only). Present as `{}` when tenant enforcement is disabled or the caller resolved no dimensions. Ignored if supplied in a create/patch body. See [Tenant scoping](#tenant-scoping) ### Status Fields diff --git a/docs/authentication.md b/docs/authentication.md index fd864d79..e176dfa2 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -11,6 +11,8 @@ This document describes authentication mechanisms for the HyperFleet API. - [Issuer configuration reference](#issuer-configuration-reference) - [Creating a service account token](#creating-a-service-account-token) - [Caller identity for audit](#caller-identity-for-audit) +- [Gateway authentication (Envoy and Authorino)](#gateway-authentication-envoy-and-authorino) + - [Trusted header contract](#trusted-header-contract) - [Tenant isolation](#tenant-isolation) - [Configuration](#configuration-1) - [Troubleshooting](#troubleshooting) @@ -282,13 +284,56 @@ server: Identity values from both sources are validated: trimmed of whitespace, limited to 256 characters, and rejected if they contain control characters. +## Gateway authentication (Envoy and Authorino) + +In production, HyperFleet API runs behind the Envoy + Authorino gateway ([ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md)), which authenticates every caller and injects trusted identity and tenant headers. Two caller types use different `Authorization` schemes: + +| Caller | Scheme | Validated by | +|--------|--------|--------------| +| Human operators | `Bearer ` | OIDC issuer | +| Sentinel and adapters (machine) | `ServiceAccount ` | Kubernetes TokenReview at the gateway, against a subject allowlist | + +The `ServiceAccount` (TokenReview) validation for machine callers happens at the gateway; the API itself does not perform it. The API's in-app JWT middleware accepts the `Bearer` scheme only and returns `401 Unauthorized` ("authorization header does not use Bearer scheme") for anything else. + +> **Note:** Because the in-app JWT middleware is `Bearer`-only, in-app JWT validation and gateway machine authentication cannot both be enabled for machine callers yet — a `ServiceAccount` token would be rejected by the in-app middleware. Running both is tracked by [HYPERFLEET-1484](https://redhat.atlassian.net/browse/HYPERFLEET-1484). + +### Trusted header contract + +After authenticating a caller, the gateway injects identity and tenant headers derived from validated claims, and the API treats these as authoritative. This is safe only because Envoy strips any client-supplied copy of these headers **before** the authorization filter runs, so a client cannot forge them ([ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md)). It applies to: + +- the tenant system header (`server.tenant.system_header`, gateway convention `X-HyperFleet-System`) — value `true` bypasses tenant scoping, and only the gateway may set it +- the caller-identity header (per-issuer `identity_header`, gateway convention `X-HyperFleet-Identity`) — only consumed when `server.jwt.enabled` is `true`; the resolver that reads it is not mounted otherwise, so with JWT disabled the API ignores this header even if tenant scoping is on +- the tenant dimension headers (`server.tenant.dimensions[].header`, gateway convention `X-HyperFleet-Org`, `X-HyperFleet-Project`, …) — resolved independently of JWT, so these still apply when `server.jwt.enabled` is `false` and `server.tenant.enabled` is `true` + +The API reads whatever header names are configured; the `X-HyperFleet-*` names are the gateway's convention, not hardcoded. See [Tenant isolation](#tenant-isolation) for how these drive scoping. + ## Tenant isolation Authentication (JWT validation) and tenant isolation are separate concerns. Tenant identity arrives as trusted headers injected by a gateway (e.g. Envoy + Authorino) — the API does not extract tenant dimensions from JWT claims. This is safe only because the gateway strips client-supplied system/dimension headers, injects validated values, and blocks any direct route to the API pod — see [ADR-0020 — Envoy and Authorino as the API Authentication Gateway](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md). -When `server.tenant.enabled` is `true`, a resolver middleware reads a system header (grants an unscoped context, e.g. for internal services like Sentinel and adapters) and configured dimension headers (collected into the caller's tenancy map). A non-system caller missing a required dimension, or resolving zero dimensions, is rejected with `403 Forbidden`. +When `server.tenant.enabled` is `true`, a resolver middleware reads a system header (grants an unscoped context, e.g. for internal services like Sentinel and adapters) and configured dimension headers (collected into the caller's tenancy map). A caller is treated as a system caller when the `system_header` value equals `true` (case-insensitive). A non-system caller missing a required dimension, presenting an invalid dimension value (values must match `^[A-Za-z0-9._-]+$` and be at most 63 characters), or resolving zero dimensions, is rejected with `403 Forbidden` before any database access. + +Once a tenant context is resolved, resource reads, lists, updates, and deletes are scoped to it (see [database.md](database.md#tenant-scoping)). A resource outside the caller's tenancy returns `404 Not Found`, not `403`, on read, update, and delete — this avoids leaking a cross-tenant resource's existence. System callers bypass this scoping entirely, but may only write `status`/`conditions`: any other resource mutation (create, update, or delete) from a system identity is rejected with `403 Forbidden`. + +### Configuration + +The tenant middleware runs after JWT validation and caller-identity resolution when those are mounted (`server.jwt.enabled: true`), and is only mounted itself when `server.tenant.enabled` is `true`. In gateway-only mode (`server.jwt.enabled: false`, `server.tenant.enabled: true`), JWT validation and caller-identity resolution are not mounted at all, and the tenant middleware runs directly off the gateway-injected headers: + +```yaml +server: + tenant: + enabled: true + system_header: X-HyperFleet-System # value "true" marks system callers that bypass scoping + dimensions: + - header: X-HyperFleet-Org # trusted gateway-injected header + key: org # tenancy map key + required: true + - header: X-HyperFleet-Project + key: project + required: false +``` -Once a tenant context is resolved, resource reads, lists, and deletes are scoped to it (see [database.md](database.md#jsonb-fields)). A resource outside the caller's tenancy returns `404 Not Found`, not `403` — this avoids leaking a cross-tenant resource's existence. System callers bypass this scoping entirely. +See [Configuration Guide - Tenant Enforcement](config.md#tenant-enforcement) for the full field reference, environment variables, and validation rules. ## Configuration diff --git a/docs/config.md b/docs/config.md index 7a26292c..5978d59b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -266,6 +266,42 @@ See [Issuer configuration reference](authentication.md#issuer-configuration-refe See [Caller identity for audit](authentication.md#caller-identity-for-audit) for full details on identity resolution, precedence rules, and per-issuer configuration. +### Tenant Enforcement + +Optional per-request tenant scoping. Tenant identity is **not** taken from JWT claims — it arrives as trusted HTTP headers injected by a gateway (Envoy + Authorino). See [Tenant isolation](authentication.md#tenant-isolation) for the conceptual model and trust boundary. + +| Property | Type | Default | Description | +|----------|------|---------|-------------| +| `server.tenant.enabled` | bool | `false` | Enable the tenant enforcement middleware | +| `server.tenant.system_header` | string | `""` | Trusted header marking system callers (e.g. Sentinel, adapters) that bypass scoping. A caller is treated as system when this header's value equals `true` (case-insensitive). Required when `enabled` is `true`. | +| `server.tenant.dimensions` | list | `[]` | YAML only. Tenant dimension mappings. Required (non-empty) when `enabled` is `true`, and at least one entry must have `required: true`. | + +Each entry in `server.tenant.dimensions` has the following fields: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `header` | string | Yes | Trusted gateway-injected HTTP header carrying this dimension's value | +| `key` | string | Yes | Tenancy map key the header value is stored under (drives the `tenancy @> ?` DB match) | +| `required` | bool | No (default `false`) | Whether a non-system caller must present this dimension | + +**Example:** + +```yaml +server: + tenant: + enabled: true + system_header: X-HyperFleet-System + dimensions: + - header: X-HyperFleet-Org + key: org + required: true + - header: X-HyperFleet-Project + key: project + required: false +``` + +Header values for dimensions must be at most 63 characters and match `^[A-Za-z0-9._-]+$`. A non-system caller missing a required dimension, presenting an invalid dimension value, or resolving zero dimensions is rejected with `403 Forbidden` before any database access. +
@@ -412,6 +448,9 @@ Complete table of all configuration properties, their environment variables, and | `server.tls.key_file` | `HYPERFLEET_SERVER_TLS_KEY_FILE` | string | `""` | | `server.jwt.enabled` | `HYPERFLEET_SERVER_JWT_ENABLED` | bool | `true` | | `server.jwt.configs` | (YAML only) | list | `[]` | +| `server.tenant.enabled` | `HYPERFLEET_SERVER_TENANT_ENABLED` | bool | `false` | +| `server.tenant.system_header` | `HYPERFLEET_SERVER_TENANT_SYSTEM_HEADER` | string | `""` | +| `server.tenant.dimensions` | (YAML only) | list | `[]` | | **Database** | | | | | `database.dialect` | `HYPERFLEET_DATABASE_DIALECT` | string | `postgres` | | `database.host` | `HYPERFLEET_DATABASE_HOST` | string | `localhost` | @@ -472,6 +511,8 @@ All CLI flags and their corresponding configuration paths. | `--server-https-cert-file` | `server.tls.cert_file` | string | | `--server-https-key-file` | `server.tls.key_file` | string | | `--server-jwt-enabled` | `server.jwt.enabled` | bool | +| `--server-tenant-enabled` | `server.tenant.enabled` | bool | +| `--server-tenant-system-header` | `server.tenant.system_header` | string | | **Database** | | | | `--db-dialect` | `database.dialect` | string | | `--db-host` | `database.host` | string | @@ -549,6 +590,10 @@ The application performs comprehensive validation at startup. - `server.timeouts.write`: ≥ 1s - `server.jwt.configs`: required non-empty when `server.jwt.enabled=true`; see [Issuer configuration reference](authentication.md#issuer-configuration-reference) for per-field validation rules - `server.jwt.configs[].issuer_url` / `jwk_cert_url`: must use `https` (`http` allowed only for loopback: `localhost`, `127.0.0.1`, `::1`) +- `server.tenant` (validated only when `server.tenant.enabled=true`): + - `system_header`: required; must be a valid HTTP header name and must not be an authentication header (`Authorization`, `Cookie`, `Set-Cookie`, `X-Api-Key`, `X-Auth-Token`, `X-Forwarded-Authorization`, `Proxy-Authorization`) + - `dimensions`: at least one entry required, and at least one entry must have `required: true` + - `dimensions[].header` / `dimensions[].key`: both required; `header` must be a valid HTTP header name, must not be an authentication header (same denylist as `system_header`), must differ from `system_header`, and must be unique (case-insensitive) across dimensions; `key` must be unique across dimensions **Database**: diff --git a/docs/database.md b/docs/database.md index edae1ad9..dbea6920 100644 --- a/docs/database.md +++ b/docs/database.md @@ -50,6 +50,41 @@ Flexible schema storage for: Adapter statuses do not use soft delete — they are hard-deleted when their parent resource is hard-deleted. +### Tenant Scoping + +When tenant enforcement is enabled, the caller's resolved tenancy is applied as a JSONB containment predicate (`tenancy @> ?`) on reads, lists, updates, and deletes (updates and deletes go through the same scoped `GetForUpdate` lookup, so a cross-tenant target is not found). Matching is by **containment, not equality**: a caller is authorized for a resource when the caller's tenancy map is a subset of the resource's. An org-scoped caller (`{org: acme}`) therefore sees every resource under that org, including those with extra dimensions (`{org: acme, project: platform}`), while a caller scoped to `{org: acme, project: p1}` does not see `{org: acme, project: p2}`. + +#### Where `tenancy` comes from + +On create by a non-system caller, the API fills the resource's `tenancy` column from the caller's resolved dimensions. Clients cannot set it: any `tenancy` sent in the request body is ignored, and the column can never change after creation. So a resource permanently carries the tenancy of whoever created it. + +System identities cannot create resources at all — they may write only `status`/`conditions`, so a create (or any other spec mutation) from a system identity is rejected with `403 Forbidden` before any `tenancy` is assigned. The creator-dimension rule therefore applies only to non-system creates. + +#### Fail-closed on zero dimensions + +A non-system caller that resolves to *no* dimensions is rejected with `403` in the middleware, before any query runs. The DAO has a second safeguard in case such a request ever slips through: instead of running an unscoped query, it applies a `1 = 0` predicate that matches nothing. + +This backstop exists because matching is by containment and the empty map `{}` is a subset of *every* row. Without it, a zero-dimension caller's predicate (`tenancy @> '{}'`) would match every resource — an accidental "see everything." Failing closed turns that into "see nothing." + +#### System callers and pre-existing rows + +System callers (e.g. Sentinel, adapters) skip scoping entirely — no `tenancy` predicate is added, so they see all resources. + +> **Security:** because `System=true` removes the tenancy predicate, the system header is only trustworthy when it originates from the Envoy + Authorino gateway. The gateway must authenticate and allowlist system clients, strip any client-supplied system header, and block direct routes to the API pod (see [ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md)). Without that boundary, a spoofed system header — or direct pod access — would expose cross-tenant resources. Only enable tenant enforcement behind such a gateway. + +Rows created *before* enforcement was enabled carry `tenancy = '{}'`. Since a scoped caller is only authorized when its (non-empty) tenancy is a subset of the row's, no scoped caller can be a subset of `{}` — so these legacy rows are visible only to system/unscoped callers. Because `tenancy` is immutable after creation and there is no backfill or re-stamp operation, this is permanent: a legacy `{}` resource can never be brought into a tenant's scope. Enable tenant enforcement before creating tenant-owned resources. + +#### Uniqueness vs. visibility + +These use `tenancy` in two different ways, and it matters: + +- **Uniqueness** — the root-resource index `(kind, name, tenancy)` compares `tenancy` by **exact equality**. +- **Visibility** — access scoping compares `tenancy` by **containment** (subset). + +Because uniqueness is by exact equality, two callers scoped to different tenancy documents can each create a resource with the same `kind` and `name` without colliding — their rows differ in the `tenancy` column. + +See [Tenant isolation](authentication.md#tenant-isolation) for the request-path behavior and trust model. + ### Delete Policies Resources use delete policies to control child behavior when a parent is deleted. Each resource type declares its policy in its entity descriptor: diff --git a/docs/deployment.md b/docs/deployment.md index 012bd354..b7dd3dfc 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -185,6 +185,41 @@ See [Issuer configuration reference](authentication.md#issuer-configuration-refe --- +## Configuring Tenant Enforcement + +Tenant enforcement scopes resource reads, lists, updates, and deletes to the caller's tenant. It is **disabled by default** and is **only safe behind the Envoy + Authorino gateway**, which injects the trusted tenant headers the API relies on. That trust holds only when a NetworkPolicy restricts API pod ingress to the Envoy pod, so no in-cluster workload can reach the API directly and forge tenant headers — see [ADR-0020](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/adrs/0020-envoy-authorino-api-gateway.md). See [Tenant isolation](authentication.md#tenant-isolation) for the trust model. + +Enable it by setting the `config.server.tenant.*` values: + +```yaml +config: + server: + tenant: + enabled: true + system_header: X-HyperFleet-System # value "true" marks system callers that bypass scoping + dimensions: + - header: X-HyperFleet-Org + key: org + required: true + - header: X-HyperFleet-Project + key: project + required: false +``` + +| Value | Required when tenant enabled | Description | +|-------|------------------------------|-------------| +| `config.server.tenant.enabled` | Yes | Set to `true` | +| `config.server.tenant.system_header` | Yes | Trusted header that marks system callers (Sentinel, adapters). A caller is treated as system when this header's value equals `true`. | +| `config.server.tenant.dimensions` | Yes | List of dimension mappings (`header`, `key`, `required`). At least one entry is required, and at least one must have `required: true`. | + +For tenant values supplied inline through Helm, the chart's `values.schema.json` enforces these invariants at install time, so a misconfigured tenant block (e.g. enabled without `system_header` or without a required dimension) fails `helm install`/`helm upgrade` before reaching the cluster. This check does not cover a tenant block loaded via `config.existingConfigMap` (see the note below) — that ConfigMap is consumed as-is and is not validated against the schema. + +> **Note:** When `config.existingConfigMap` is set, these `config.server.tenant.*` values are ignored — tenant settings must come from the referenced ConfigMap. + +See [Configuration Guide - Tenant Enforcement](config.md#tenant-enforcement) for the full field reference and validation rules. + +--- + ## Configuring Required Adapters Adapters are external components (validation, DNS, pull-secret, HyperShift) that report status back to HyperFleet API. Each entity type declares its required adapters via the `required_adapters` field in the entity descriptor. These define which adapters must report "ready" before a resource is considered **Reconciled**. @@ -359,6 +394,9 @@ helm install hyperfleet-api oci://quay.io/redhat-services-prod/hyperfleet-tenant | `image.tag` | Image tag | `""` (must be set) | | `image.pullPolicy` | Image pull policy | `Always` | | `config.server.jwt.enabled` | Enable JWT authentication | `false` | +| `config.server.tenant.enabled` | Enable tenant enforcement middleware | `false` | +| `config.server.tenant.system_header` | Trusted header marking system callers that bypass tenant scoping | `""` | +| `config.server.tenant.dimensions` | Tenant dimension mappings (`header`, `key`, `required`) | `[]` | | `config.entities` | Entity descriptors (kinds, required adapters, schemas) | (see values.yaml) | | `database.postgresql.enabled` | Enable built-in PostgreSQL | `true` | | `database.external.enabled` | Use external database | `false` | diff --git a/docs/runbook.md b/docs/runbook.md index 6abb3e63..18711c0f 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -296,6 +296,35 @@ kubectl get secret hyperfleet-db -n hyperfleet-system -o go-template='{{range $k 4. Check database is running and accepting connections 5. Verify SSL settings match database requirements +### Tenant Enforcement Issues + +Applies only when tenant enforcement is enabled (`server.tenant.enabled=true`). See [Tenant isolation](authentication.md#tenant-isolation) for the model. + +**Symptoms:** Callers get unexpected `403 Forbidden` on all requests, or `404 Not Found` for resources they expect to see. + +**Diagnosis:** + +```bash +# Inspect the rendered tenant config +kubectl get configmap -config -n hyperfleet-system -o yaml | grep -A8 'tenant:' + +# Look for rejection logs (middleware logs a warning on every rejected request) +kubectl logs deployment/hyperfleet-api -n hyperfleet-system --since=15m | grep -i "Tenant identity rejected" +``` + +**Common causes:** + +- **All requests get 403** — the gateway (Envoy + Authorino) is not injecting the configured dimension headers, or a `required: true` dimension header is missing/empty. Verify the gateway `AuthConfig` injects the headers named in `server.tenant.dimensions[].header`. +- **Invalid dimension value** — dimension header values must match `^[A-Za-z0-9._-]+$` and be ≤ 63 characters; other values are rejected with 403. +- **System caller can't write resources** — system identities (system header value `true`) may only write `status`/`conditions`; any other resource mutation (create, update, or delete) returns 403. This is expected — route resource writes through a tenant-scoped identity. +- **Resources "disappear" (404)** — the resource's tenancy does not contain the caller's resolved tenancy (the caller's dimensions are not a subset of the resource's). Confirm the caller's dimension headers match the tenant the resource was created under. Cross-tenant reads return 404 by design. + +**Resolution:** + +1. Confirm the API sits behind the gateway and direct pod access is blocked (NetworkPolicy) — tenant headers are only trustworthy in that topology. +2. Compare the gateway-injected headers against `server.tenant.system_header` and `server.tenant.dimensions[].header`. +3. If tenant enforcement was enabled after resources already existed, those rows carry `tenancy = {}` and are only visible to unscoped/system callers — see [database.md](database.md#tenant-scoping). + ### Memory Issues **Symptoms:** OOMKilled, high memory usage