You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Run the agent as a privileged container on Azure Container Linux - #370
Experimental. Azure Container Linux (ACL) support, in containers or on the host, is experimental and not for production nodes. The operator guide, docs/usage/azure-container-linux.md, opens with a warning that says so, and lists what isn't supported yet. The agent's containers needed ACL to ship one SELinux rule, allow spc_t kernel_t:dbus send_msg;, and to label containerd's files, so the agent's containers run as spc_t, where the rule applies (SELinux in DESIGN.md). ACL 3.0.20261026 has both, in a development gallery build, and the harness runs on that build as it ships. The commit "Add an opt-in SELinux workaround for the agent container" and its revert stay in the history as the record. Merging waits until the rule and the labels reach the image AKS will use; see Known gaps.
Summary
This adds a second way to run the AKS Flex Node agent, for Azure Container Linux: in privileged containerd containers of the agent's own image, provisioned by Ignition, rather than installed on the host. main installs the agent on an ACL host, under /opt/unbounded/agent, as an interim. Here, nothing of the agent is installed on the host, and the agent doesn't install host packages, which ACL's read-only /usr doesn't allow.
On ACL, the agent keeps main's config, commands, unit name, MachineOperations, and reset. The new behavior is in how the agent is installed, upgraded, and reaches the host, and it applies only on ACL. Ubuntu hosts behave as on main, apart from the changes in bold under "Changes that apply everywhere" below.
The agent image for ACL (Dockerfile.aks-flex-node-agent) sets AKS_FLEX_NODE_CONTAINERIZED=true, and daemon.Containerized() (pkg/daemon/container.go) reads it. Nothing else sets it: on Ubuntu, bootstrap.sh installs the release binary, and every switch below takes main's path.
Takes an archive's HTTP or HTTPS downloadURL, as on main
Takes an image's oci://downloadURL
CredentialPlugin, for the agent's own Kubernetes clients in start and the daemon, with a managed identity or a service principal and no bootstrap token (pkg/daemon/container.go)
The host's agent, config.HostBinaryPath()
The image's binary, which is running. The host has no agent.
bootstrap.sh downloads a release archive, and the agent installs itself under /opt/unbounded/agent
Ignition writes a settings file naming the image, bootstrap's inputs, a bootstrap unit, and a reset unit (scripts/aks-flex-node-agent-container.bu). Nothing of the agent is on the host.
Bootstrap
bootstrap.sh renders the config, runs preflight, and runs start
The bootstrap unit runs the same bootstrap.sh in a container of the image, with --installed-agent, naming the image's agent. Its inputs are AKS_FLEX_NODE_* lines in /etc/aks-flex-node/bootstrap.env, which the Butane config takes, as host mode's does. It retries a failed bootstrap, but not one bootstrap.sh exits with 78 for, as main's first-boot unit.
The agent's unit
aks-flex-node-agent.service runs the agent's current binary under /opt/unbounded/agent
Same name; it runs the daemon in a container of the image, through ctr run
Machine hooks
At every start of a machine, its hooks start the agent's nspawn lifecycle units, which run its binary
The same hooks start units that run the phase in a short-lived container of the image the agent's unit runs, which is on the host, so a machine's start pulls nothing
Host access
Direct
Each container gets the host's root at /host. Its entrypoint mounts the host paths the agent uses at their own paths. The host's systemctl, machinectl, ctr, and other tools run through host-exec: under chroot /host, or, for mounts, in the host's mount namespace.
AgentUpgrade
Takes a release archive's downloadURL and sha256, and switches blue/green binaries
Takes the same parameters for an image: downloadURL is oci:// and the image's fully qualified repository, with or without a tag, and sha256 is the image's digest, which ctr verifies on pull, as main requires sha256 (#369). A request without the digest, with it in the URL, or naming an archive fails as InvalidParameters. The agent pulls the image, checks that its agent runs, selects it in /etc/aks-flex-node/agent-image.env, and restarts on it. main's handoff judges the result.
Upgrade recovery
The recovery unit's script restores the last good binary
The recovery unit runs the last good image, which selects itself again and restarts the agent's unit
Daemon start
Converges its binary layout and units
Rewrites its units from its image, copies its binary into the active machine for kubelet, and removes the images its upgrades pulled that it won't run again.
Reset
AgentReset, the deletion signal, or aks-flex-node reset
The same code; aks-flex-node reset runs through the reset unit. Afterwards nothing of the agent runs, and the install stays.
Joining again
Run bootstrap.sh again, which downloads the agent its inputs name
Write new inputs and start the bootstrap unit. agent-image.env may name the image; without it, the install's image runs.
Uninstall
scripts/uninstall.sh
Reprovision without the agent's Butane config
Repave and NodeReboot run the same code on both.
Changes that apply everywhere
These are in shared code. Only the ones in bold change anything on Ubuntu:
A reset empties a runtime directory that's a mount point.RemoveRuntimeDirs, which reset uses, empties a directory it can't remove because it's a mount point, as the agent's directories are in its container. On a host, that changes only a reset where /etc/aks-flex-node or /var/log/aks-flex-node is a mount of its own, which failed before.
An AgentUpgrade in progress says it's staging the upgraded agent, not its binary, which in the container is an image.
The handoff between an upgrading daemon and its successor now goes through a small interface, with the host's executor as one implementation and the container's as the other. Parsing an AgentUpgrade's parameters checks what every agent requires, a download URL and a SHA-256, and the executor checks the URL: the host's an archive's, as main does, so an oci:// URL fails on a host as on main, and the container's an image's. The host's results are main's, except that a request with both a malformed URL and a malformed digest reports the digest first.
aksmachine.MachineClientOptions.CredentialPlugin names the credential plugin start's Machine client runs. Empty, it's the host's agent, as on main.
utilexec.Ctr(), and one new direct dependency, github.com/distribution/reference, already in the module graph.
Prompt restart after an upgrade. After switching to a new agent, the daemon restarts itself from a transient timer set for 1 s. systemd timers default to AccuracySec=1min, so the restart came up to a minute late; it measured 26–58 s on ACL. The timer now has AccuracySec=100ms, so every AgentUpgrade, on Ubuntu too, finishes up to a minute sooner.
The agent's image
Each release publishes the agent's image, ghcr.io/azure/aks-flex-node-agent, for linux/amd64 and linux/arm64, as it publishes the controller's: release.yml builds it in its dry run, through agent-image-build.yml, and publishes it once the release is approved.
Tags: the release's version and sha-<commit>. The image is pushed by digest and its build provenance attested first, so a tag only ever names an attested image.
Release notes: the image pinned by digest, for the install's settings, and the AgentUpgrade parameters that name it.
Checks: ACL Container Checks builds it for both platforms, without pushing, when its files change.
Testing
hack/acl-container/harness.py boots ACL under QEMU/KVM, creates a local kind cluster running main's in-cluster controller, and bootstraps the VM into it:
hack/acl-container/harness.py up # exports and boots ACL 3.0.20261026
hack/acl-container/harness.py check
check runs its checks in 7 to 10 minutes, mostly downloads, and covers:
a pod on the node
repave from kube1 to kube2
the agent's binary, which kubelet runs as its credential plugin, running in the machine
no binary of the agent on the host, and the machines' hooks starting units that run the agent's phases in containers of the image
NodeReboot, whose restart of the machine runs both phases in containers
stopping the agent's unit while its daemon ignores SIGTERM: the container still ends, within the unit's stop, and the agent starts again
AgentUpgrade: success, a broken candidate that rolls back, requests without the image's digest, with it in the URL, or naming an archive, which are rejected, back again, and removal of the broken image
a VM reboot
AgentReset, reset, and the deletion signal, each leaving no unit or link behind, and each followed by a re-join, once with a given image and once with the agent's registry unreachable
a bootstrap with a setting that can't work, which fails with 78 and isn't retried
It passes on ACL 3.0.20261026 as that build ships, with the agent's containers in spc_t, where the rule applies, and no SELinux denials.
Go tests, also with the race detector, golangci-lint, scripts/bootstrap_test.sh, scripts/install_test.sh, scripts/uninstall_test.sh, ShellCheck, actionlint on the release and image workflows, and the docs checks pass. The arm64 image was built and run under QEMU, but not on ACL.
Known gaps
Before merging: ACL's SELinux rule and its labels on containerd's files have to reach the image AKS uses. Today they're in a development gallery build.
Swap in the host's /etc/fstab: in the container, start turns it off and masks its units on the host, but comments it out of the image's fstab, not the host's, so it would come back after a reboot. ACL has no swap and no /etc/fstab, so it only matters if an operator's Ignition config adds one (Comment out swap in the host's /etc/fstab, not the image's, with the agent in its container #417).
Not supported yet, as the operator guide says: offline install, and pulling the agent's image from a registry that needs credentials. Pull flags appear on ctr's command lines, so they mustn't carry credentials.
The reason will be displayed to describe this comment to others. Learn more.
Initial review of the draft. I read TODO.md and DESIGN.md first and left out anything they already track.
I found no critical bugs in the Go changes. I traced the image-based upgrade through success, a broken candidate, abort, recovery, and the kernel_t fallback where reset-failed is denied, and the handoff holds up. The Containerized() switches leave Ubuntu on main's path. At df70b9c, go build ./..., go vet, go test ./pkg/daemon/ ./pkg/config/ ./pkg/cmd/..., and golangci-lint run ./pkg/... all pass.
Suggested order: the Stage timeouts, the ctr cleanup race in the units, the bootstrap unit's retry, the selinux-repro.sh policy restore, and the unit write order. Details are inline.
Not inline
The PR description's Known gaps still says ctr places the agent's container outside its unit's cgroup, but 6c1095d moved it into a scope in system-aksflex.slice. DESIGN.md has the same leftover; see inline.
release.yml builds the agent with upstream Go through setup-go. The image builds it with Microsoft Go and ms_nocgo_opensslcrypto. The image's binary is also copied into the machines, where kubelet runs it as its credential plugin, and onto the host as the nspawn lifecycle helper. It's worth confirming that check exercises the credential plugin; with bootstrap-token auth it may not.
Harness (dev-only, low)
harness.py:1594-1600: config.ign holds the bootstrap token. It's written with the default umask in a 0755 state directory and only changed to 0600 afterwards. It's also served over plain HTTP on the bridge, and the local registry accepts pushes from anyone while the VM pulls the agent by tag. Create it with os.open(..., 0o600) and make the state directory 0700.
harness.py:813-838: vm_running and stop_vm trust the PID file, so after PID reuse they can signal an unrelated process. Check /proc/<pid>/cmdline for qemu-system first.
harness.py:1733: down runs shutil.rmtree(STATE) on whatever ACLC_STATE_DIR names. Require a marker file first.
cmd_up (harness.py:1564): if up fails partway, the bridge, iptables rules, ip_forward=1, the kind cluster, and the registry stay until down.
Machine names read from the VM go unquoted into remote shell strings, such as harness.py:1033, :1037, and :1189. Use shlex.quote. At :1189, an empty before makes machinectl show-image succeed, so that check can never pass.
selinux-verify.sh:32 and :116-123: root runs a script from a fixed /tmp/acl-selinux-verify on the host, which another local user could create first. Use mktemp -d on the host.
selinux-repro.sh:244: swapoff -a is never undone, though the header says the host is put back.
The Containerfile's base images, and the harness's registry, Butane, Fedora, and kind images, are pulled by tag. That matters most for the published preview image.
The reason will be displayed to describe this comment to others. Learn more.
Nit: pkg/cmd/preflight now imports all of pkg/daemon for one filter. Moving Containerized() and PreflightChecks into a small package that both import would keep the dependency graph shallow and avoid future import cycles.
The reason will be displayed to describe this comment to others. Learn more.
Won't fix. It's the pattern the command packages already follow: pkg/cmd/start, daemon, reset, and nspawnlifecycle import pkg/daemon too, and pkg/daemon doesn't import pkg/cmd, so there's no cycle to avoid. If Containerized() moves to a package of its own later, this can follow it.
DESIGN.md now describes the image's upstream Go build and pinned base
images, the bootstrap unit's condition on the agent's unit being
enabled and its backoff, the entrypoint resolving host paths within
the host's root, the bootstrap wrapper ignoring what the image sets,
the time limits on an upgrade's pull and check, the recovery unit's
condition on a last good image, removing only the images upgrades
pulled, and the install's settings being 0600. Its open questions no
longer list the cgroup, which the systemd scopes settled, or whether
ctr stays the launcher, which it does.
check now has 68 checks, with the agent's binary running in its
machine, and takes 7 to 10 minutes: 9m42s and 9m30s today, as each
rejoin pulls the machine's image and binaries again, and downloads
were slower.
TODO.md: the upgraded daemon rewrites its recovery unit before it
judges the upgrade, as main's does on a host; raise it with main.
DESIGN.md now describes the image's upstream Go build and pinned base
images, the bootstrap unit's condition on the agent's unit being
enabled and its backoff, the entrypoint resolving host paths within
the host's root, the bootstrap wrapper ignoring what the image sets,
the time limits on an upgrade's pull and check, the recovery unit's
condition on a last good image, removing only the images upgrades
pulled, and the install's settings being 0600. Its open questions no
longer list the cgroup, which the systemd scopes settled, or whether
ctr stays the launcher, which it does.
check now has 68 checks, with the agent's binary running in its
machine, and takes 7 to 10 minutes: 9m42s and 9m30s today, as each
rejoin pulls the machine's image and binaries again, and downloads
were slower.
TODO.md: the upgraded daemon rewrites its recovery unit before it
judges the upgrade, as main's does on a host; raise it with main.
DESIGN.md now describes the image's upstream Go build and pinned base
images, the bootstrap unit's condition on the agent's unit being
enabled and its backoff, the entrypoint resolving host paths within
the host's root, the bootstrap wrapper ignoring what the image sets,
the time limits on an upgrade's pull and check, the recovery unit's
condition on a last good image, removing only the images upgrades
pulled, and the install's settings being 0600. Its open questions no
longer list the cgroup, which the systemd scopes settled, or whether
ctr stays the launcher, which it does.
check now has 68 checks, with the agent's binary running in its
machine, and takes 7 to 10 minutes: 9m42s and 9m30s today, as each
rejoin pulls the machine's image and binaries again, and downloads
were slower.
TODO.md: the upgraded daemon rewrites its recovery unit before it
judges the upgrade, as main's does on a host; raise it with main.
DESIGN.md now describes the image's upstream Go build and pinned base
images, the bootstrap unit's condition on the agent's unit being
enabled and its backoff, the entrypoint resolving host paths within
the host's root, the bootstrap wrapper ignoring what the image sets,
the time limits on an upgrade's pull and check, the recovery unit's
condition on a last good image, removing only the images upgrades
pulled, and the install's settings being 0600. Its open questions no
longer list the cgroup, which the systemd scopes settled, or whether
ctr stays the launcher, which it does.
check now has 68 checks, with the agent's binary running in its
machine, and takes 7 to 10 minutes: 9m42s and 9m30s today, as each
rejoin pulls the machine's image and binaries again, and downloads
were slower.
TODO.md: the upgraded daemon rewrites its recovery unit before it
judges the upgrade, as main's does on a host; raise it with main.
Releases publish the agent's image instead, from main, with the
release's version; see "Publish the agent's image for Azure Container
Linux with each release". The preview package,
ghcr.io/azure/aks-flex-node-agent-acl-preview, keeps its images until
it's deleted.
This reverts commit 5e91041.
A release publishes the agent's image, ghcr.io/azure/aks-flex-node-agent,
tagged with the release's version and its commit, for linux/amd64 and
linux/arm64, as it publishes the controller's: a reusable workflow,
agent-image-build.yml, builds it in the release's dry run and publishes
it once the release is approved. It's pushed by digest, its build
provenance attested, and only then tagged, so a tag only ever names an
attested image, as the preview's. The release notes give it pinned by
digest, for the install's settings, and the AgentUpgrade parameters that
name it. ACL Container Checks builds it, without pushing, when its files
change.
What the release builds moves out of the harness, beside its
counterparts: hack/acl-container/image/Containerfile to
Dockerfile.aks-flex-node-agent, the image's scripts to
scripts/agent-image, and the install's Butane config to
scripts/aks-flex-node-agent-container.bu. The Go build runs on the build
machine's platform and cross-compiles for each of the image's, as the
controller's does; the image's packages install under QEMU for arm64.
The harness runs the image on linux/amd64 only.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Current validation is limited to QEMU/kind with a development ACL image, leaving privileged host access, lifecycle recovery, SELinux behavior, and identity-based bootstrap unverified on the intended AKS Azure VM image.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Prevent credential leakage in invalid reference errors
pkg/daemon/agent_upgrade_container.go:184
reference.ParseNamed can include the supplied repository in its error, so an invalid reference such as oci://registry.example/agent?token=SecretValue leaks the token into the MachineOperation failure message. Return a fixed validation error without wrapping the parser error. Add a regression test for credential-bearing invalid references.
scripts/aks-flex-node-agent-container.bu renders bootstrap.env into
/etc/aks-flex-node/bootstrap.env, which runs the bootstrap unit, as
host mode's Butane config takes its settings, so an operator renders
both from the files in a directory. A base config and a service
principal's credential are commented entries, as in host mode's. The
harness renders it the same way, and adds its base config as an
operator would.
docs/usage/azure-container-linux.md describes the agent in its containers
as the way to run on Azure Container Linux: the ACL build and release it
needs, joining with the Butane config, upgrading by image, reset and
rejoining, and checking the agent. Installing the agent on the host
stays, as an interim, under its own heading. Getting started and
operations point to it, and the changelog records it, and the upgrade
restart that no longer comes up to a minute late.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Malformed references can expose query credentials in failure messages, and configured local offline artifact sources are not mounted where container preflight and bootstrap expect them.
DESIGN.md, the harness's README and code, and the comments in the
agent's container code compared it with main's agent, as a branch did.
They compare it with the agent installed on a host now, and say what
the shared code does rather than how it changed. DESIGN.md's note points
operators to docs/usage/azure-container-linux.md, and its open questions
link their issues. The harness is no longer a prototype, and runs
checks, not feasibility checks.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
SELinux policies and containerd labels remain unconfirmed in the production ACL image, while privileged host access and rollback have only QEMU validation and Azure identity bootstrap is untested in container mode.
The recovery unit's work in the container returned as soon as
reset-failed failed, so it never restarted the agent's unit on the
restored image. A host's recovery script, and an upgrade's restart,
warn and restart anyway: on systemd 254 and later, recovery runs at the
first failure, before the unit hits its start limit, so the restart
works without the reset. It does the same now, and reports only the
restart's failure.
The container's AgentUpgrade wrapped the reference parser's error,
which repeats the repository when it has capitals in it, so a
downloadURL such as oci://registry.example/agent?Sig=Secret put the
query, and any credential in it, in the MachineOperation's failure and
the agent's log. The failure is a fixed message now, as a host's for an
archive's URL is, and the tests check that neither a query's credential
nor user information comes back in it.
…eload
The daemon in its container rewrote its units and copied its binary
into the active machine only after it had built its credentials and its
manager, so a daemon whose credentials failed left them as an earlier
image had them. A host's daemon converges its install before either. The
container's now converges right after the host's steps would, with only
the state store it needs.
It also reloaded systemd only when it changed a unit. A start that wrote
the units and ended before its reload, or whose reload failed, left
systemd on the old ones, and the next start, finding the files current,
never reloaded. Each start reloads now; writing the units an upgrade
held back still reloads only when one changes. writeUnits writes them
itself, and the comments say which unit goes last and why.
newClusterEndpointClientFromBootstrapConfig's comment had slid onto
bootstrapRESTConfig above it; each function has its own again. The
recovery's comment said it runs once the agent's unit hits its start
limit, which is so only before systemd 254; from 254, OnFailure= runs
it at every failure, as the unit's comment says. Reading the install's
settings failed as reading "the agent's image selection"; the message
names neither now, and the error's path says which. An AgentUpgrade in
progress says it's staging the upgraded agent, not its binary, which is
an image in the container.
The base images were pinned, but the build pulled three more by tag,
each of which shapes the published image while its provenance attests
it all the same: setup-qemu-action's binfmt emulators, at :latest and
privileged on the runner, which run every arm64 step; the BuildKit that
buildx creates, at buildx-stable-1; and the Dockerfile's syntax
frontend. Each is pinned by digest now, at what its tag names today:
binfmt qemu-v10.2.3-68, BuildKit v0.33.1, and dockerfile 1.7. The
emulators are pulled rather than restored from the Actions cache.
DESIGN.md said the install's settings are 0600 because pull flags could
put credentials in them, as if that kept them private. The flags are on
ctr's command lines, the pulls' and the agent container's run, which
any local user can read through /proc. DESIGN.md and the Butane
config's header now say never to put credentials in them, and that a
registry that needs credentials isn't supported yet.
up marked any ACLC_STATE_DIR as its own, and set it to 0700, before
anything else, so down, which removes a marked directory, would remove
one that held a user's files, say ACLC_STATE_DIR=~/vms. up now takes
the default directory, one it marked, or a new or empty one, and stops
before changing anything otherwise. The README says so.
DESIGN.md, the Butane config, the daemon's image removal, and the
harness said bootstrap runs the install's image as it is when the pull
fails, "offline too". That's a fallback for a registry that can't be
reached, not an offline install, which Azure Container Linux doesn't
support yet, and the word suggested it does. The fallback and the
harness's check of it are unchanged.
Nothing an operator reads said that running on Azure Container Linux is
experimental, though the agent's containers need an ACL build that isn't
in the image AKS uses yet, and have run only on local VMs. The operator
guide opens with a warning: experimental and not for production nodes,
in containers or on the host, with settings, units, and AgentUpgrade
parameters that may change without a migration path; the ACL build it
needs; Azure VMs, and so managed identity and service principal
bootstrap, untested (#399); and no offline install, GPU hosts (#416),
or registry that needs credentials yet.
Getting started, operations, the usage index, both Butane configs, the
changelog's two entries, the release notes, and the harness's README
and DESIGN.md say so, and point to it. Getting started's Ignition
section no longer says to run the agent in containers "instead" in the
middle of describing the host install, which made "a Butane config that
does this" name the wrong thing: a warning before it points to the
container install. The changelog also records bootstrap.sh's
--installed-agent.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Merging depends on confirming the required SELinux rule and containerd labels in AKS’s ACL image, while development-image QEMU coverage leaves Azure identity bootstrap unverified.
parseAgentUpgradeRequest checked a release archive's URL and SHA-256,
and let any oci:// URL through for the container's executor, whose
Validate checked it, while the host's Validate checked the archive's
again. Shared code knew about images only to step around them.
Now the parse checks what every agent requires, a download URL and a
SHA-256 of 64 hexadecimal characters, the latter in agentUpgradeDigest,
which the container's check of an image uses too. The executor alone
checks which URLs it takes: an archive's on a host, an image's in the
container. The results are the same, but a host given a malformed URL
and a malformed digest reports the digest first. The scheme and its
helper move to the container's file, the only one that uses them.
The daemon in its container rewrote the active machine's hooks and
removed the nspawn lifecycle helper that a preview of the agent's image,
published from this branch and since removed, left on the host. Only
hosts that joined with that preview had one, and none remain: a host
that did is reprovisioned rather than upgraded. With it go the
convergence's hook for it, its tests, the paragraph in DESIGN.md, and
the test that the agent's unit no longer mounts /usr/local, another
leftover. hostroot_move.go is main's again: it resolved the active
machine's config in a function of its own only for the migration.
hack/acl-container/ukiboot.py was byte for byte hack/acl/ukiboot.py,
which boots ACL the same way. The harness imports hack/acl's now, and
ACL Container Checks compiles it and runs when it changes.
agent_task_pid was never called, while two checks found the agent
task's PID again with awk; they call it now. die never returns, so its
callers' unreachable raise AssertionError went, with die marked
NoReturn. push_agent_image's digest was never used, so its callers push
the image directly. The reboot check in the machines group tested for a
cluster that the group's skip already requires, and base64 is imported
with the rest.
DESIGN.md had grown a test report, a snapshot of the test environment,
measurements, the branch's history, and a dozen restatements of the
same facts: the AgentUpgrade parameters, rejoining after a reset, which
image runs. It's down from 516 lines to 370, about how the agent runs
in containers and why:
- It names the two installs once, the binary install and the container
install, rather than "on a host", which an ACL host also is.
- One table of units, now all six, with the nspawn lifecycle units; one
of the files on the host; one of the bootstrap unit's behavior.
- The ctr run flags' reasons are a line each, the upgrade's steps say
what happens rather than what it's like on a host, and SELinux says
what's true of ACL, not how it came to be.
The harness's README takes the tested environment and the measurements,
points to DESIGN.md for the units and files and to the operator guide
for the upgrade parameters, says that Podman is always needed, and
counts no checks, a number that drifted.
The operator guide introduces the two installs, lists the bootstrap
unit's status 78 cases with what to do for each, gives the upgrade
parameters as a table with what's rejected, adds a way to follow the
node's join and the upgrade, runs version, not a status command that
doesn't exist, in the agent's container, and says the start limit
plainly. The URL contract protects its anchors, which other pages link
to, and lists the README's new one.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The required SELinux rule and containerd labels remain unconfirmed in the actual AKS ACL image, where privileged execution and recovery still require runtime validation beyond the development-gallery QEMU build.
The entrypoint exited when the host's machinectl didn't answer, for
every command. That kept aks-flex-node reset from running on a host
whose machined or D-Bus is broken, though reset is how such a node is
brought back, and kept upgrade recovery from restoring the last good
image. Both now warn and run, doing what they can; every other command
still exits, the agent's daemon with the same message as before.
LocalDNS, which a node pool's bootstrap data can turn on, runs
resolvectl domain when the host's resolv.conf points at the
systemd-resolved stub. The image had no resolvectl, so bootstrap would
fail there. It runs the host's now, through host-exec, as the other
host tools do.
The release notes told users to run gh attestation verify with --owner,
which accepts provenance that any repository of the owner attested.
They give --repo now, so only an attestation this repository's
workflows made verifies.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to evaluate potential cached-disk corruption from unchecked partial writes and Arc preflight failures caused by a missing host-agent shim.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Active
Philip Lombardi (plombardi89)
left a comment
Summary
This adds a second way to run the AKS Flex Node agent, for Azure Container Linux: in privileged containerd containers of the agent's own image, provisioned by Ignition, rather than installed on the host.
maininstalls the agent on an ACL host, under/opt/unbounded/agent, as an interim. Here, nothing of the agent is installed on the host, and the agent doesn't install host packages, which ACL's read-only/usrdoesn't allow.On ACL, the agent keeps
main's config, commands, unit name, MachineOperations, and reset. The new behavior is in how the agent is installed, upgraded, and reaches the host, and it applies only on ACL. Ubuntu hosts behave as onmain, apart from the changes in bold under "Changes that apply everywhere" below.docs/usage/azure-container-linux.mddocuments it for operators,hack/acl-container/DESIGN.mdexplains the design, andhack/acl-container/README.mdexplains how to run it locally.How ACL behavior is switched on
The agent image for ACL (
Dockerfile.aks-flex-node-agent) setsAKS_FLEX_NODE_CONTAINERIZED=true, anddaemon.Containerized()(pkg/daemon/container.go) reads it. Nothing else sets it: on Ubuntu,bootstrap.shinstalls the release binary, and every switch below takesmain's path.setupHost, instart(pkg/daemon/start.go)PATHis the image's, and would install into the container.InstallService, instart(pkg/daemon/lifecycle.go)daemon.Run(pkg/daemon/daemon.go)RecoverAgentUpgrade(pkg/daemon/agent_upgrade.go)PreflightChecks, inpreflight(pkg/cmd/preflight/preflight.go)host-packagesandcontainerd-active, which don't apply in a containerAgentUpgrade(pkg/daemon/agent_upgrade.go,pkg/daemon/agent_upgrade_container.go)downloadURL, as onmainoci://downloadURLCredentialPlugin, for the agent's own Kubernetes clients instartand the daemon, with a managed identity or a service principal and no bootstrap token (pkg/daemon/container.go)config.HostBinaryPath()StartNodewrites first and the daemon converges (pkg/daemon/lifecycle.go,pkg/daemon/container.go)/opt/unbounded/agent, with the lifecycle helper linked to it for hooks an earlier release writesstart, the daemon, andnspawn-lifecycle pre-start(pkg/cmd/start/start.go,pkg/daemon/daemon.go,pkg/cmd/nspawnlifecycle/nspawn_lifecycle.go)/opt/unbounded/agent, move it off/usr/local, and remove the binary install scripts left thereagent-upgrade(pkg/daemon/host_agent_activation.go)AgentUpgradenames an imageUninstallService, in reset (pkg/daemon/lifecycle.go)aks-flex-node-bootstrap.service, which runs onceThe ACL code is in
pkg/daemon/container.goandpkg/daemon/agent_upgrade_container.go. The units it installs are inpkg/daemon/assets.New behavior on ACL
mainon Ubuntubootstrap.shdownloads a release archive, and the agent installs itself under/opt/unbounded/agentscripts/aks-flex-node-agent-container.bu). Nothing of the agent is on the host.bootstrap.shrenders the config, runs preflight, and runsstartbootstrap.shin a container of the image, with--installed-agent, naming the image's agent. Its inputs areAKS_FLEX_NODE_*lines in/etc/aks-flex-node/bootstrap.env, which the Butane config takes, as host mode's does. It retries a failed bootstrap, but not onebootstrap.shexits with 78 for, asmain's first-boot unit.aks-flex-node-agent.serviceruns the agent's current binary under/opt/unbounded/agentctr run/host. Its entrypoint mounts the host paths the agent uses at their own paths. The host'ssystemctl,machinectl,ctr, and other tools run throughhost-exec: underchroot /host, or, for mounts, in the host's mount namespace.AgentUpgradedownloadURLandsha256, and switches blue/green binariesdownloadURLisoci://and the image's fully qualified repository, with or without a tag, andsha256is the image's digest, whichctrverifies on pull, asmainrequiressha256(#369). A request without the digest, with it in the URL, or naming an archive fails asInvalidParameters. The agent pulls the image, checks that its agent runs, selects it in/etc/aks-flex-node/agent-image.env, and restarts on it.main's handoff judges the result.AgentReset, the deletion signal, oraks-flex-node resetaks-flex-node resetruns through the reset unit. Afterwards nothing of the agent runs, and the install stays.bootstrap.shagain, which downloads the agent its inputs nameagent-image.envmay name the image; without it, the install's image runs.scripts/uninstall.shRepave and
NodeRebootrun the same code on both.Changes that apply everywhere
These are in shared code. Only the ones in bold change anything on Ubuntu:
RemoveRuntimeDirs, which reset uses, empties a directory it can't remove because it's a mount point, as the agent's directories are in its container. On a host, that changes only a reset where/etc/aks-flex-nodeor/var/log/aks-flex-nodeis a mount of its own, which failed before.AgentUpgradein progress says it's staging the upgraded agent, not its binary, which in the container is an image.scripts/bootstrap.shgains--installed-agent PATH(orAKS_FLEX_NODE_INSTALLED_AGENT), which runs the agent atPATHinstead of installing one. It's off by default, and an unusablePATHexits with 78, as any setting that can't work does since Stop retrying first-boot failures that can't succeed, and clean up after any uninstall reset #407. With tests.AgentUpgrade's parameters checks what every agent requires, a download URL and a SHA-256, and the executor checks the URL: the host's an archive's, asmaindoes, so anoci://URL fails on a host as onmain, and the container's an image's. The host's results aremain's, except that a request with both a malformed URL and a malformed digest reports the digest first.aksmachine.MachineClientOptions.CredentialPluginnames the credential pluginstart's Machine client runs. Empty, it's the host's agent, as onmain.utilexec.Ctr(), and one new direct dependency,github.com/distribution/reference, already in the module graph.AccuracySec=1min, so the restart came up to a minute late; it measured 26–58 s on ACL. The timer now hasAccuracySec=100ms, so everyAgentUpgrade, on Ubuntu too, finishes up to a minute sooner.The agent's image
Each release publishes the agent's image,
ghcr.io/azure/aks-flex-node-agent, forlinux/amd64andlinux/arm64, as it publishes the controller's:release.ymlbuilds it in its dry run, throughagent-image-build.yml, and publishes it once the release is approved.sha-<commit>. The image is pushed by digest and its build provenance attested first, so a tag only ever names an attested image.AgentUpgradeparameters that name it.Testing
hack/acl-container/harness.pyboots ACL under QEMU/KVM, creates a local kind cluster runningmain's in-cluster controller, and bootstraps the VM into it:hack/acl-container/harness.py up # exports and boots ACL 3.0.20261026 hack/acl-container/harness.py checkcheckruns its checks in 7 to 10 minutes, mostly downloads, and covers:NodeReboot, whose restart of the machine runs both phases in containersAgentUpgrade: success, a broken candidate that rolls back, requests without the image's digest, with it in the URL, or naming an archive, which are rejected, back again, and removal of the broken imageAgentReset,reset, and the deletion signal, each leaving no unit or link behind, and each followed by a re-join, once with a given image and once with the agent's registry unreachableIt passes on ACL 3.0.20261026 as that build ships, with the agent's containers in
spc_t, where the rule applies, and no SELinux denials.Go tests, also with the race detector,
golangci-lint,scripts/bootstrap_test.sh,scripts/install_test.sh,scripts/uninstall_test.sh, ShellCheck, actionlint on the release and image workflows, and the docs checks pass. The arm64 image was built and run under QEMU, but not on ACL.Known gaps
Before merging: ACL's SELinux rule and its labels on containerd's files have to reach the image AKS uses. Today they're in a development gallery build.
Follow-ups:
/etc/fstab: in the container,startturns it off and masks its units on the host, but comments it out of the image'sfstab, not the host's, so it would come back after a reboot. ACL has no swap and no/etc/fstab, so it only matters if an operator's Ignition config adds one (Comment out swap in the host's /etc/fstab, not the image's, with the agent in its container #417).--fetch-bootstrap-datawith a managed identity or a service principal is untested in the container (E2E: join an Azure Container Linux VM through Ignition and its first-boot unit #399).ctr's command lines, so they mustn't carry credentials.