Skip to content

feat: Global Load Balancer controller + real API-schema codegen (draft) - #418

Draft
Mitch Connors (therealmitchconnors) wants to merge 14 commits into
mainfrom
global-ingress-complete
Draft

Mitch Connors (therealmitchconnors) wants to merge 14 commits into
mainfrom
global-ingress-complete

Conversation

@therealmitchconnors

@therealmitchconnors Mitch Connors (therealmitchconnors) commented Oct 5, 2026 •

Copy link
Copy Markdown

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

Adds a new hub controller that deploys and manages Azure Global Load
Balancers (GLBs) for MultiClusterLoadBalancer resources, built on an
istio.io/istio/pkg/kube/krt-based reconciliation pipeline. Alongside
the controller, this PR introduces real client-gen/
applyconfiguration-gen/openapi-gen codegen tooling, reducing the
amount of code required to build and test the controller substantially.

🚧 This PR is a draft. It is not ready to merge — opening it now to
get early feedback on direction (especially the krt-based pipeline and
the codegen approach) before polishing further.

Which issue(s) this PR fixes:

Fixes #


How to review this PR

This PR is organized as 3 commits, each a self-contained concern.
Please review commit-by-commit (GitHub's "Commits" tab, or
git log -p --first-parent) rather than the full squashed diff — the
full diff is large (~90 files) but almost all of it is generated code
that doesn't need line-by-line review.

# Commit What it is Review priority
1 943277d Add real API-schema codegen tooling... Makefile codegen targets, the openapi-schema-gen entrypoint, hand-written API types/markers (incl. the new MultiClusterLoadBalancer type), go.mod/go.sum Please review — this is hand-written infra
2 fd818f9 Regenerate clientset, apply-configurations, and OpenAPI schema... Mechanical output of make generate using the tooling from commit 1: zz_generated.deepcopy.go, CRD YAML, pkg/applyconfigurations/**, pkg/generated/** Skip — generated, marked linguist-generated in .gitattributes so GitHub collapses it by default in "Files changed"
3 5cba71a Add Global Load Balancer controller with krt-based kube client The actual feature: pkg/controllers/hub/globalserviceexport (the GLB controller), pkg/apiclient (the krt-compatible kube client wrapper), tests, and docs Please review — this is the point of the PR

If your GitHub diff view isn't collapsing the generated files automatically, you can also view just the reviewable surface with:

git diff 943277d~1..5cba71a -- . ':!pkg/generated' ':!pkg/applyconfigurations' ':!*/zz_generated.*' ':!config/crd/bases'

Suggested reading order

  1. docs/concepts/GlobalLoadBalancer/architecture.md and docs/concepts/KrtApiClient/README.md for context before diving into code.
  2. pkg/apiclient/ (small, ~2 files) — the kube client wrapper the controller depends on.
  3. pkg/controllers/hub/globalserviceexport/controller.go — the controller itself. Notable things to look at:
    • The krt-based transformation pipeline and the async deployment work queue (non-blocking writes off the transform hot path).
    • fieldManagerName vs statusFieldManagerName — separate field managers for main-resource vs. status-subresource Apply calls (needed to avoid one clobbering the other's managed fields).
    • The Recorder-based Kubernetes event emitted on deployment failure.
  4. pkg/controllers/hub/globalserviceexport/controller_test.go for coverage of the above.

Requirements:

  • run make reviewable for basic local test (note: make lint/make staticcheck could not be run in the dev sandbox due to a Go toolchain version mismatch with the pinned tool versions — please run these in CI/locally before merge)
  • Read and followed fleet-networking's Code of conduct.
  • uses conventional commit messages
  • includes documentation
  • adds unit tests

How has this code been tested

  • go build ./... and go vet ./... pass across the repo.
  • go test -v -race ./pkg/controllers/hub/globalserviceexport/... passes, including the new async-deployment-queue and event-emission tests.

Special notes for your reviewer

  • This is a draft PR. Expect rough edges — in particular the krt/controller-runtime coexistence pattern and the hardcoded ARM-related values in the controller are likely to need discussion before this is merge-ready.
  • Generated-file commit (fd818f9) is large by line count but mechanical; please focus review time on commits 1 and 3.

…-gen/openapi-gen

Introduce hand-written codegen infrastructure so the generated
clientset, apply-configurations, and OpenAPI schema are produced
from the real API types instead of a hand-rolled analogue:

- Makefile: add client-gen, applyconfiguration-gen, and openapi-gen
  tool targets alongside the existing controller-gen target.
- hack/tools/openapi-schema-gen: new generator entrypoint that wires
  the API types into a real OpenAPI schema.
- api/v1alpha1, api/v1beta1: add/adjust codegen markers and hand-written
  type definitions (doc.go, groupversion_info.go, *_types.go), including
  the new MultiClusterLoadBalancer type, that drive the generators above.
- go.mod/go.sum: pull in k8s.io/code-generator and k8s.io/kube-openapi.
- .devcontainer/devcontainer.json: tooling support for the above.
- .gitattributes: mark the mechanically-generated output (zz_generated
  deepcopy, applyconfigurations, generated clientset/openapi, CRD YAML)
  as linguist-generated so it collapses by default in the GitHub PR
  diff view and is excluded from review/line-count stats.

This removes the need for the project's own analogue of
fake.ClientSet (see the follow-up controller commit), since the real
generators now produce a fully-featured fake clientset with Apply
support.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ated, no manual review needed)

Mechanically produced output from 'make generate' using the
client-gen/applyconfiguration-gen/openapi-gen tooling added in the
previous commit. Includes:

- api/*/zz_generated.deepcopy.go (updated deepcopy for new/changed types)
- config/crd/bases/*.yaml (CRD manifests, including the new
  MultiClusterLoadBalancer CRD)
- pkg/applyconfigurations/** (apply-configuration types, including
  Apply/ApplyStatus support on the generated fake clientset)
- pkg/generated/** (typed clientset, fake clientset, OpenAPI schema)

These paths are marked linguist-generated in .gitattributes and are
collapsed by default in the GitHub diff view.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The actual feature work: a new hub controller that deploys/manages
Azure Global Load Balancers for MultiClusterLoadBalancer resources,
plus the supporting kube-client infrastructure that makes this
possible.

- pkg/apiclient: a kube.Client wrapper (istio.io/istio/pkg/kube) that
  exposes both the generic Core() client and the generated
  Networking() clientset, needed so the controller's krt-based
  (istio.io/istio/pkg/kube/krt) collections can watch/apply these
  resources alongside existing fleet-networking controller-runtime
  controllers.
- pkg/controllers/hub/globalserviceexport: the GLB controller itself,
  including:
  - krt-based transformation pipeline from MultiClusterLoadBalancer
    inputs to Azure deployment stacks.
  - Option C non-blocking deployment: a work queue populated by the
    krt registration batch, with writeDeployment moved off the
    transformation hot path.
  - Finalizer/status-apply field-manager separation
    (fieldManagerName vs statusFieldManagerName) so finalizer and
    status updates don't stomp each other's managed fields.
  - A Kubernetes Warning event emitted on deployment failure.
  - Updated/expanded unit tests, including coverage for the event
    emission and the new fake clientset's Apply/ApplyStatus behavior.
- cmd/hub-net-controller-manager: wire up the new reconciler and its
  event recorder.
- docs/concepts/GlobalLoadBalancer, docs/concepts/KrtApiClient:
  architecture and usage documentation for the above.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve conflicts from main's dependency/CVE bump commits that landed
after this branch's fork point:
- Makefile: keep the client-gen/applyconfiguration-gen/openapi-gen
  targets added here, take main's updated STATICCHECK_VER and the
  improved multi-arch docker-buildx-builder emulation logic.
- go.mod: take main's go 1.26.6 toolchain bump and dependency version
  bumps (golang.org/x/*, protobuf, jsonpatch, etc.).
- go.sum: regenerated via 'go mod tidy' after resolving go.mod so the
  lock file reflects both main's bumped versions and this branch's
  added dependencies (istio.io/istio, Azure SDK resource manager
  packages, structured-merge-diff/v6).

Verified go build/vet and
'go test -race ./pkg/controllers/hub/globalserviceexport/...' still
pass after the merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Those controllers (EndpointsliceExport, InternalServiceExport,
InternalServiceImport, ServiceImport, MemberCluster,
TrafficManagerProfile, TrafficManagerBackend) were commented out
earlier purely to simplify demoing the new GLB controller against a
live cluster. Restore them so main.go starts the full set of hub
controllers again, alongside the new GlobalServiceExport controller.

Restores the corresponding imports (endpointsliceexport,
internalserviceexport, internalserviceimport, membercluster,
serviceimport, trafficmanagerbackend, trafficmanagerprofile,
k8s.io/client-go/discovery, go.goms.io/fleet/pkg/utils) and runs
'go mod tidy' to add the resulting transitive dependencies
(k8s.io/metrics, sigs.k8s.io/work-api) and reconcile a few indirect
version bumps.

Left '// kube.EnableCrdWatcher(client)' commented out: it references
an unimported, unused helper and isn't one of the controller-startup
blocks that were disabled for the demo.

Verified go build/vet pass and
'go test ./pkg/controllers/hub/globalserviceexport/... ./pkg/apiclient/...'
still pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
.devcontainer/devcontainer.json, test/scripts/aks-shared-vnet.sh,
test/scripts/bootstrap.sh, and
docs/demos/TrafficManagerProfile/testfiles/gateway.yaml were only
needed to demo the GLB controller against a live cluster and aren't
part of this feature. Drop the new devcontainer/demo-doc file and
restore the two test scripts to their origin/main content.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- globalserviceexport/controller.go: check the (previously ignored)
  errors returned by RemoveFinalizer/ApplyFinalizer instead of
  checking a stale, always-nil 'err' variable (errcheck); extract the
  repeated "mclb" finalizer-name literal into a mclbFinalizer
  constant (goconst); use errors.As instead of a direct type
  assertion to unwrap the Azure SDK ResponseError (errorlint).
- controller_test.go: use the new mclbFinalizer constant in place of
  the literal.
- hub-net-controller-manager/main.go: check mgr.Add's error (errcheck);
  remove the unused structureWatcher helper (unused) along with its
  now-unused 'k8s.io/apimachinery/pkg/watch' import, which also
  resolves the errcheck finding on the FromUnstructured call inside
  it.
- hack/tools/openapi-schema-gen/main.go: tighten the generated OpenAPI
  schema file's permissions from 0o644 to 0o600 (gosec G306).

Verified go build/vet, goimports -l (no diff), and
'go test -race ./pkg/controllers/hub/globalserviceexport/...
./pkg/apiclient/... ./cmd/...' (only pre-existing, unrelated
cmd/net-crd-installer/utils failures remain: missing envtest
etcd binary and a CRD-list test not yet updated for the new
MultiClusterLoadBalancer CRD).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@therealmitchconnors Mitch Connors (therealmitchconnors) changed the title Global Load Balancer controller + real API-schema codegen feat: Global Load Balancer controller + real API-schema codegen (draft) Oct 6, 2026
- pkg/generated/openapi/doc.go (new): add a hand-written, untagged file
  to the generated openapi package. zz_generated.openapi.go carries
  '//go:build !ignore_autogenerated', and controller-gen v0.20.0+
  loads packages with the 'ignore_autogenerated' build tag set by
  default; without another file in the package, every file is
  excluded under that tag set and controller-gen fails with
  'build constraints exclude all Go files' the moment it scans
  './...', breaking 'make manifests'/'make generate'/'make test' in CI.
- config/crd/bases/networking.fleet.azure.com_multiclusterloadbalancers.yaml:
  regenerate with the pinned controller-gen v0.20.0 (previously
  generated with a stale local v0.16.0), matching what CI produces.
- docs/concepts/GlobalLoadBalancer/README.md: replace the placeholder
  '(TODO)' ClusterResourcePlacement link (a dead relative link in this
  repo) with a link to the type's definition in Azure/fleet, fixing
  markdown-link-check.

Verified 'make manifests'/'make generate' now produce only the CRD
annotation/schema diff above (no other unexpected changes), plus
go build/vet and the globalserviceexport/apiclient/generated test
suites still pass.

Also retitled PR #418 to 'feat: ...' to satisfy the repo's
pr-title-checker, which requires a conventional-commit-style prefix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The previously pinned k8s.io/kube-openapi/cmd/openapi-gen version
(v0.0.0-20250814151709-d7b6acb124c3) depends on
golang.org/x/tools@v0.24.0, which fails to compile under the Go
toolchain CI now resolves to:

  golang.org/x/tools/internal/tokeninternal/tokeninternal.go:64:9:
  invalid array length -delta * delta (constant -256 of type int64)

Bump OPENAPI_GEN_VER to v0.0.0-20250910181357-589584f1c912, which
pulls in golang.org/x/tools@v0.36.0 and builds cleanly. Verified
locally with 'go install' using the exact pinned version, and
regenerated pkg/generated/openapi/zz_generated.openapi.go with it
(a couple of now-inlined/removed schema entries from the newer
k8s.io/apimachinery version openapi-gen resolves against; no
behavioral changes).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The controller-runtime v0.19->v0.22 and client-go v0.31->v0.34 bump
(needed for the real codegen/fake-clientset work) changed the fake
client's behavior to no longer populate TypeMeta on objects returned
by Get/List, matching real client-go/API server behavior for typed
clients. This exposed two issues:

- pkg/controllers/multiclusterservice/controller.go:
  isServiceImportOwnedByOthers compared owner.APIVersion/owner.Kind
  against mcs.APIVersion/mcs.Kind (i.e. mcs.TypeMeta), which is empty
  once the fake client stops artificially populating it. Fixed by
  comparing against the well-known MultiClusterService
  GroupVersionKind instead of relying on the possibly-empty TypeMeta
  of the in-memory object. This is also a latent production bug:
  real client-go consumers of this controller's informer cache can
  likewise have empty TypeMeta on MultiClusterService objects.
- pkg/controllers/multiclusterservice/controller_test.go: updated
  TestHandleUpdate's comparison options to ignore TypeMeta on objects
  fetched via fakeClient.Get(), since those are no longer expected to
  carry Kind/APIVersion.
- cmd/net-crd-installer/utils/util_test.go: the 'hub mode' CRD list
  fixture predates the multiclusterloadbalancers CRD added by this
  PR; hub mode installs every networking.fleet.azure.com CRD, so the
  fixture (and its stale/inaccurate name) needed updating to include
  it.

Verified via 'go test' for pkg/controllers/multiclusterservice and
cmd/net-crd-installer/utils (envtest-dependent suites in both
packages still fail locally due to a pre-existing sandbox limitation:
missing /usr/local/kubebuilder/bin/etcd, unrelated to this change).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The test's deleteCalled flag was a plain bool written from the krt
worker goroutine (via the fake deployment-stacks server callback) and
read from the test goroutine without synchronization, tripping the
race detector under -race in CI. Use atomic.Bool instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Oct 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 16.90402% with 1342 lines in your changes missing coverage. Please review.
✅ Project coverage is 11.64%. Comparing base (76f455e) to head (ae9fa8e).

Files with missing lines Patch % Lines
...nfigurations/api/v1alpha1/internalserviceexport.go 0.00% 109 Missing ⚠️
...nfigurations/api/v1alpha1/internalserviceimport.go 0.00% 109 Missing ⚠️
...gurations/api/v1alpha1/multiclusterloadbalancer.go 0.00% 106 Missing ⚠️
.../applyconfigurations/api/v1alpha1/serviceexport.go 0.00% 106 Missing ⚠️
.../applyconfigurations/api/v1alpha1/serviceimport.go 0.00% 106 Missing ⚠️
...g/applyconfigurations/api/v1beta1/serviceexport.go 0.00% 106 Missing ⚠️
.../controllers/hub/globalserviceexport/controller.go 80.59% 43 Missing and 22 partials ⚠️
...tset/internalclientset/fake/clientset_generated.go 0.00% 50 Missing ⚠️
...generated/clientset/internalclientset/clientset.go 0.00% 45 Missing ⚠️
...internalclientset/typed/api/v1alpha1/api_client.go 0.00% 42 Missing ⚠️
... and 33 more
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #418       +/-   ##
===========================================
- Coverage   67.06%   11.64%   -55.43%     
===========================================
  Files          34       77       +43     
  Lines        3859    24480    +20621     
===========================================
+ Hits         2588     2850      +262     
- Misses       1090    21426    +20336     
- Partials      181      204       +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Mirrors the linguist-generated paths in .gitattributes so codecov
ignores the same mechanically-generated files that GitHub collapses
in the PR diff view.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant