Read MachineOperations from the API server before running them by plombardi89 · Pull Request #374 · Azure/AKSFlexNode

Philip Lombardi (plombardi89) · 2026-10-20T00:32:03Z

Problem

A NodeReboot MachineOperation sometimes reboots the node twice. The daemon's log shows a complete second soft reboot, stop-node then a full start, beginning one second after the first one finished and after the operation reported Complete.

The daemon builds its MachineOperation reconciler with daemon.NewMachinaMachineOperationReconciler(mgr.GetClient(), ...), so it checks whether an operation is still to run through the manager's cache:

The handler marks the operation in progress. That status update queues another reconcile of the operation, which controller-runtime runs as soon as the handler returns.
The handler reboots the node and marks the operation Complete.
The queued reconcile reads the operation from the cache, which may not have the Complete update yet, and sees it in progress.
MarkInProgress reads the same cache. For an operation it would have found finished, it returns nil, not an error, so the handler runs again and reboots the node a second time.

Whether it happens depends on whether the watch event for the Complete update reaches the cache before the queued reconcile reads it.

Fix

The reconciler now reads operations through the manager's API reader, using unbounded's NewMachinaMachineOperationReconcilerWithReader. The queued reconcile then sees the operation as finished and does nothing. Status updates still go through the manager's client.

Testing

Unit test: a new table-driven test builds the reconciler with a cache that has a NodeReboot in progress, and an API server that has it Complete. It expects no reboot, and that a pending NodeReboot still reboots once. Going back to the cached client alone makes it fail: the finished operation reboots again.
On a host: I ran NodeReboot repeatedly on an Azure Container Linux host running the agent from Run the agent as a privileged container on Azure Container Linux #370, which uses this same code path, and counted stop-node tasks per operation:
- without this change: 1 of 6 rebooted twice, and earlier 1 of 3
- with it: 0 of 12
Go tests and golangci-lint pass.

The daemon checked whether a MachineOperation was still to run through the manager's cached client. Marking the operation in progress updates its status, which queues another reconcile of it, and that reconcile runs as soon as the handler returns. The cache can then still have the operation in progress, though the handler has just finished it, and MarkInProgress, reading the same cache, returns without an error for an operation it would have found finished. So the handler ran again: one NodeReboot rebooted the node twice, the second time after the operation had reported Complete. Now the reconciler reads operations through the manager's API reader, as unbounded's NewMachinaMachineOperationReconcilerWithReader allows, so a finished operation is seen as finished. Status updates still go through the manager's client.

Copilot

🟢 Approval recommended

The focused reader wiring addresses stale operation reads, includes regression coverage, and has no identified blocking issues.

No open issues 🎉

What changed in this PR

Prevents duplicate MachineOperation execution caused by stale cache reads, including repeated node reboots.

Changes:

Uses the manager’s API reader for operation reads while retaining the client for status updates.
Adds regression coverage for completed operations with stale cached status and pending operations.

File	Description
pkg/daemon/machineoperation_reconciler.go	Accepts a reader and uses the reader-aware constructor.
pkg/daemon/machineoperation_reconciler_test.go	Tests stale-cache and pending-operation behavior.
pkg/daemon/daemon.go	Supplies the manager’s API reader.

_{Review effort: Balanced}

hbc (bcho) · 2026-10-20T00:43:08Z

(can keep to another pr) we can also prompt the agent to update https://github.com/Azure/AKSFlexNode/blob/main/CHANGELOG.md for tracking the changes