Implement deduplication of Swarm tasks - #956
Open
johannstieger wants to merge 1 commit into
Open
Conversation
Added a function to deduplicate Swarm tasks in container inspection.
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA. Best regards, Johann |
Author
|
I have read the CLA Document and I hereby sign the CLA or my organization already has a signed CLA. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added a function to deduplicate Swarm tasks in container inspection.
Bug reports
Please include:
Summary: Found data from multiple Docker nodes - see service details for more information
Details: This docker container apparently exists on multiple parent hosts. This should be reflected in the fact that this host has multiple piggyback sources, see the output of the Check_MK service. Hence, no definitive information on the container can be displayed. To resolve this situation, you have two options: 1. configure the docker agent plug-in to use the container IDs as host names, 2. use the ruleset 'Host name translation for piggybacked hosts' to create unique host names for the affected containers.
Proposed changes
Sometimes it is hard for us to assess the quality of a fix.
While it may work for you, it is our job to ensure that it works for everybody.
These are some ways to help us:
What is the expected behavior?
When Docker Swarm recreates a task (health-check-triggered restart, rolling update, image update, etc.) while the old task's container is still present on the node — kept around by Swarm's task history (--task-history-limit) — mk_docker.py should only ever surface one container per logical Swarm service+slot. If the piggyback target for that container is a stable host name across task recreations (e.g. via Checkmk's "Host name translation for piggybacked hosts" rule, used to strip the ever-changing Swarm task-ID suffix), that host should keep receiving a single, unambiguous set of docker_container_* sections.
What is the observed behavior?
MKDockerClient.init builds self.all_containers from every container returned by containers(all=True), which includes exited containers that Swarm still keeps around as task history. Once a service's task gets recreated, both the old (already shut down) container and the new (running) one are present locally and both get inspected in the same agent run. If the piggyback host name has been made stable across restarts — which is necessary in the first place, otherwise every task recreation produces a brand-new Checkmk host because the raw container name embeds a random task ID — both containers end up generating sections for the very same target host, from the very same physical source, in the very same agent execution. Checkmk's "Docker container status" check then goes CRIT with "Found data from multiple Docker nodes" (see Werk #14420), which is misleading: it suggests a cross-node conflict, but in this case only one physical node and one agent run are involved.
If it's not obvious from the above: In what way does your patch change the current behavior?
The patch adds a _dedupe_swarm_tasks() helper that groups the containers coming from _robust_inspect(client, "containers") by their com.docker.swarm.service.name / com.docker.swarm.task.slot labels, and — whenever more than one container shares that key — keeps only the most recently created one (c.attrs["Created"]), dropping older, already-superseded task containers before self.all_containers is populated. Containers without those labels (i.e. not Swarm-managed) pass through unchanged. As a result, at most one container per Swarm service+slot ever reaches section generation, so a piggyback host name that stays stable across task recreations never receives conflicting data from two container instances within one run.
Consider writing a unit test that would have failed without your fix.
from types import SimpleNamespace
def make_container(id, created, labels=None):
return SimpleNamespace(attrs={
"Id": id_,
"Created": created,
"Config": {"Labels": labels or {}},
})
def test_dedupe_swarm_tasks_keeps_only_newest_per_service_slot():
old_task = _make_container(
"old", "2026-09-07T02:00:00.000000000Z",
labels={
"com.docker.swarm.service.name": "stalwart_stalwart",
"com.docker.swarm.task.slot": "1",
},
)
new_task = _make_container(
"new", "2026-09-07T08:00:00.000000000Z",
labels={
"com.docker.swarm.service.name": "stalwart_stalwart",
"com.docker.swarm.task.slot": "1",
},
)
standalone = _make_container("standalone", "2026-09-07T01:00:00.000000000Z")
result = _dedupe_swarm_tasks([old_task, new_task, standalone])
assert len(result) == 2
assert new_task in result
assert standalone in result
assert old_task not in result
Before the fix this function doesn't exist at all, so this test fails outright (import error); more importantly, without the grouping/filtering logic, both old_task and new_task would have passed straight through untouched and both ended up in self.all_containers, i.e. len(result) == 3 and old_task in result.
Not a regression tied to a specific Docker/Checkmk release — it can happen with any Swarm deployment that keeps task history (the default) as soon as a task gets recreated for any reason. I ran into it on my own homelab Swarm cluster after configuring hostname translation to avoid the opposite problem: without it, every task recreation produces a brand-new Checkmk host, because the raw container name embeds a random, ever-changing Swarm task ID. Once that stable-naming rule was in place, the next task recreation exposed this issue — the already-exited previous task container was still picked up by mk_docker.py (since it queries containers(all=True)) alongside the new one, triggering Werk #14420's "multiple parent hosts" CRIT even though only a single physical node and a single agent run were ever involved.