Repository navigation
feat: Global Load Balancer controller + real API-schema codegen (draft) - #418
Draft
Mitch Connors (therealmitchconnors) wants to merge 14 commits into
Draft
Mitch Connors (therealmitchconnors) wants to merge 14 commits into
Mitch Connors (therealmitchconnors) wants to merge 14 commits into
Conversation
…-gen/openapi-gen Introduce hand-written codegen infrastructure so the generated clientset, apply-configurations, and OpenAPI schema are produced from the real API types instead of a hand-rolled analogue: - Makefile: add client-gen, applyconfiguration-gen, and openapi-gen tool targets alongside the existing controller-gen target. - hack/tools/openapi-schema-gen: new generator entrypoint that wires the API types into a real OpenAPI schema. - api/v1alpha1, api/v1beta1: add/adjust codegen markers and hand-written type definitions (doc.go, groupversion_info.go, *_types.go), including the new MultiClusterLoadBalancer type, that drive the generators above. - go.mod/go.sum: pull in k8s.io/code-generator and k8s.io/kube-openapi. - .devcontainer/devcontainer.json: tooling support for the above. - .gitattributes: mark the mechanically-generated output (zz_generated deepcopy, applyconfigurations, generated clientset/openapi, CRD YAML) as linguist-generated so it collapses by default in the GitHub PR diff view and is excluded from review/line-count stats. This removes the need for the project's own analogue of fake.ClientSet (see the follow-up controller commit), since the real generators now produce a fully-featured fake clientset with Apply support. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ated, no manual review needed) Mechanically produced output from 'make generate' using the client-gen/applyconfiguration-gen/openapi-gen tooling added in the previous commit. Includes: - api/*/zz_generated.deepcopy.go (updated deepcopy for new/changed types) - config/crd/bases/*.yaml (CRD manifests, including the new MultiClusterLoadBalancer CRD) - pkg/applyconfigurations/** (apply-configuration types, including Apply/ApplyStatus support on the generated fake clientset) - pkg/generated/** (typed clientset, fake clientset, OpenAPI schema) These paths are marked linguist-generated in .gitattributes and are collapsed by default in the GitHub diff view. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The actual feature work: a new hub controller that deploys/manages
Azure Global Load Balancers for MultiClusterLoadBalancer resources,
plus the supporting kube-client infrastructure that makes this
possible.
- pkg/apiclient: a kube.Client wrapper (istio.io/istio/pkg/kube) that
exposes both the generic Core() client and the generated
Networking() clientset, needed so the controller's krt-based
(istio.io/istio/pkg/kube/krt) collections can watch/apply these
resources alongside existing fleet-networking controller-runtime
controllers.
- pkg/controllers/hub/globalserviceexport: the GLB controller itself,
including:
- krt-based transformation pipeline from MultiClusterLoadBalancer
inputs to Azure deployment stacks.
- Option C non-blocking deployment: a work queue populated by the
krt registration batch, with writeDeployment moved off the
transformation hot path.
- Finalizer/status-apply field-manager separation
(fieldManagerName vs statusFieldManagerName) so finalizer and
status updates don't stomp each other's managed fields.
- A Kubernetes Warning event emitted on deployment failure.
- Updated/expanded unit tests, including coverage for the event
emission and the new fake clientset's Apply/ApplyStatus behavior.
- cmd/hub-net-controller-manager: wire up the new reconciler and its
event recorder.
- docs/concepts/GlobalLoadBalancer, docs/concepts/KrtApiClient:
architecture and usage documentation for the above.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve conflicts from main's dependency/CVE bump commits that landed after this branch's fork point: - Makefile: keep the client-gen/applyconfiguration-gen/openapi-gen targets added here, take main's updated STATICCHECK_VER and the improved multi-arch docker-buildx-builder emulation logic. - go.mod: take main's go 1.26.6 toolchain bump and dependency version bumps (golang.org/x/*, protobuf, jsonpatch, etc.). - go.sum: regenerated via 'go mod tidy' after resolving go.mod so the lock file reflects both main's bumped versions and this branch's added dependencies (istio.io/istio, Azure SDK resource manager packages, structured-merge-diff/v6). Verified go build/vet and 'go test -race ./pkg/controllers/hub/globalserviceexport/...' still pass after the merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Those controllers (EndpointsliceExport, InternalServiceExport, InternalServiceImport, ServiceImport, MemberCluster, TrafficManagerProfile, TrafficManagerBackend) were commented out earlier purely to simplify demoing the new GLB controller against a live cluster. Restore them so main.go starts the full set of hub controllers again, alongside the new GlobalServiceExport controller. Restores the corresponding imports (endpointsliceexport, internalserviceexport, internalserviceimport, membercluster, serviceimport, trafficmanagerbackend, trafficmanagerprofile, k8s.io/client-go/discovery, go.goms.io/fleet/pkg/utils) and runs 'go mod tidy' to add the resulting transitive dependencies (k8s.io/metrics, sigs.k8s.io/work-api) and reconcile a few indirect version bumps. Left '// kube.EnableCrdWatcher(client)' commented out: it references an unimported, unused helper and isn't one of the controller-startup blocks that were disabled for the demo. Verified go build/vet pass and 'go test ./pkg/controllers/hub/globalserviceexport/... ./pkg/apiclient/...' still pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
.devcontainer/devcontainer.json, test/scripts/aks-shared-vnet.sh, test/scripts/bootstrap.sh, and docs/demos/TrafficManagerProfile/testfiles/gateway.yaml were only needed to demo the GLB controller against a live cluster and aren't part of this feature. Drop the new devcontainer/demo-doc file and restore the two test scripts to their origin/main content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- globalserviceexport/controller.go: check the (previously ignored) errors returned by RemoveFinalizer/ApplyFinalizer instead of checking a stale, always-nil 'err' variable (errcheck); extract the repeated "mclb" finalizer-name literal into a mclbFinalizer constant (goconst); use errors.As instead of a direct type assertion to unwrap the Azure SDK ResponseError (errorlint). - controller_test.go: use the new mclbFinalizer constant in place of the literal. - hub-net-controller-manager/main.go: check mgr.Add's error (errcheck); remove the unused structureWatcher helper (unused) along with its now-unused 'k8s.io/apimachinery/pkg/watch' import, which also resolves the errcheck finding on the FromUnstructured call inside it. - hack/tools/openapi-schema-gen/main.go: tighten the generated OpenAPI schema file's permissions from 0o644 to 0o600 (gosec G306). Verified go build/vet, goimports -l (no diff), and 'go test -race ./pkg/controllers/hub/globalserviceexport/... ./pkg/apiclient/... ./cmd/...' (only pre-existing, unrelated cmd/net-crd-installer/utils failures remain: missing envtest etcd binary and a CRD-list test not yet updated for the new MultiClusterLoadBalancer CRD). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- pkg/generated/openapi/doc.go (new): add a hand-written, untagged file to the generated openapi package. zz_generated.openapi.go carries '//go:build !ignore_autogenerated', and controller-gen v0.20.0+ loads packages with the 'ignore_autogenerated' build tag set by default; without another file in the package, every file is excluded under that tag set and controller-gen fails with 'build constraints exclude all Go files' the moment it scans './...', breaking 'make manifests'/'make generate'/'make test' in CI. - config/crd/bases/networking.fleet.azure.com_multiclusterloadbalancers.yaml: regenerate with the pinned controller-gen v0.20.0 (previously generated with a stale local v0.16.0), matching what CI produces. - docs/concepts/GlobalLoadBalancer/README.md: replace the placeholder '(TODO)' ClusterResourcePlacement link (a dead relative link in this repo) with a link to the type's definition in Azure/fleet, fixing markdown-link-check. Verified 'make manifests'/'make generate' now produce only the CRD annotation/schema diff above (no other unexpected changes), plus go build/vet and the globalserviceexport/apiclient/generated test suites still pass. Also retitled PR #418 to 'feat: ...' to satisfy the repo's pr-title-checker, which requires a conventional-commit-style prefix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The previously pinned k8s.io/kube-openapi/cmd/openapi-gen version (v0.0.0-20250814151709-d7b6acb124c3) depends on golang.org/x/tools@v0.24.0, which fails to compile under the Go toolchain CI now resolves to: golang.org/x/tools/internal/tokeninternal/tokeninternal.go:64:9: invalid array length -delta * delta (constant -256 of type int64) Bump OPENAPI_GEN_VER to v0.0.0-20250910181357-589584f1c912, which pulls in golang.org/x/tools@v0.36.0 and builds cleanly. Verified locally with 'go install' using the exact pinned version, and regenerated pkg/generated/openapi/zz_generated.openapi.go with it (a couple of now-inlined/removed schema entries from the newer k8s.io/apimachinery version openapi-gen resolves against; no behavioral changes). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The controller-runtime v0.19->v0.22 and client-go v0.31->v0.34 bump (needed for the real codegen/fake-clientset work) changed the fake client's behavior to no longer populate TypeMeta on objects returned by Get/List, matching real client-go/API server behavior for typed clients. This exposed two issues: - pkg/controllers/multiclusterservice/controller.go: isServiceImportOwnedByOthers compared owner.APIVersion/owner.Kind against mcs.APIVersion/mcs.Kind (i.e. mcs.TypeMeta), which is empty once the fake client stops artificially populating it. Fixed by comparing against the well-known MultiClusterService GroupVersionKind instead of relying on the possibly-empty TypeMeta of the in-memory object. This is also a latent production bug: real client-go consumers of this controller's informer cache can likewise have empty TypeMeta on MultiClusterService objects. - pkg/controllers/multiclusterservice/controller_test.go: updated TestHandleUpdate's comparison options to ignore TypeMeta on objects fetched via fakeClient.Get(), since those are no longer expected to carry Kind/APIVersion. - cmd/net-crd-installer/utils/util_test.go: the 'hub mode' CRD list fixture predates the multiclusterloadbalancers CRD added by this PR; hub mode installs every networking.fleet.azure.com CRD, so the fixture (and its stale/inaccurate name) needed updating to include it. Verified via 'go test' for pkg/controllers/multiclusterservice and cmd/net-crd-installer/utils (envtest-dependent suites in both packages still fail locally due to a pre-existing sandbox limitation: missing /usr/local/kubebuilder/bin/etcd, unrelated to this change). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The test's deleteCalled flag was a plain bool written from the krt worker goroutine (via the fake deployment-stacks server callback) and read from the test goroutine without synchronization, tripping the race detector under -race in CI. Use atomic.Bool instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Mirrors the linguist-generated paths in .gitattributes so codecov ignores the same mechanically-generated files that GitHub collapses in the PR diff view. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
/kind api-change
What this PR does / why we need it:
Adds a new hub controller that deploys and manages Azure Global Load
Balancers (GLBs) for
MultiClusterLoadBalancerresources, built on anistio.io/istio/pkg/kube/krt-based reconciliation pipeline. Alongsidethe controller, this PR introduces real
client-gen/applyconfiguration-gen/openapi-gencodegen tooling, reducing theamount of code required to build and test the controller substantially.
🚧 This PR is a draft. It is not ready to merge — opening it now to
get early feedback on direction (especially the krt-based pipeline and
the codegen approach) before polishing further.
Which issue(s) this PR fixes:
Fixes #
How to review this PR
This PR is organized as 3 commits, each a self-contained concern.
Please review commit-by-commit (GitHub's "Commits" tab, or
git log -p --first-parent) rather than the full squashed diff — thefull diff is large (~90 files) but almost all of it is generated code
that doesn't need line-by-line review.
943277dAdd real API-schema codegen tooling...openapi-schema-genentrypoint, hand-written API types/markers (incl. the newMultiClusterLoadBalancertype), go.mod/go.sumfd818f9Regenerate clientset, apply-configurations, and OpenAPI schema...make generateusing the tooling from commit 1:zz_generated.deepcopy.go, CRD YAML,pkg/applyconfigurations/**,pkg/generated/**linguist-generatedin.gitattributesso GitHub collapses it by default in "Files changed"5cba71aAdd Global Load Balancer controller with krt-based kube clientpkg/controllers/hub/globalserviceexport(the GLB controller),pkg/apiclient(the krt-compatible kube client wrapper), tests, and docsIf your GitHub diff view isn't collapsing the generated files automatically, you can also view just the reviewable surface with:
Suggested reading order
docs/concepts/GlobalLoadBalancer/architecture.mdanddocs/concepts/KrtApiClient/README.mdfor context before diving into code.pkg/apiclient/(small, ~2 files) — the kube client wrapper the controller depends on.pkg/controllers/hub/globalserviceexport/controller.go— the controller itself. Notable things to look at:fieldManagerNamevsstatusFieldManagerName— separate field managers for main-resource vs. status-subresource Apply calls (needed to avoid one clobbering the other's managed fields).Recorder-based Kubernetes event emitted on deployment failure.pkg/controllers/hub/globalserviceexport/controller_test.gofor coverage of the above.Requirements:
make reviewablefor basic local test (note:make lint/make staticcheckcould not be run in the dev sandbox due to a Go toolchain version mismatch with the pinned tool versions — please run these in CI/locally before merge)How has this code been tested
go build ./...andgo vet ./...pass across the repo.go test -v -race ./pkg/controllers/hub/globalserviceexport/...passes, including the new async-deployment-queue and event-emission tests.Special notes for your reviewer