Skip to content

Implement deduplication of Swarm tasks - #956

Open
johannstieger wants to merge 1 commit into
Checkmk:masterfrom
johannstieger:patch-1
Open

Implement deduplication of Swarm tasks#956
johannstieger wants to merge 1 commit into
Checkmk:masterfrom
johannstieger:patch-1

Conversation

@johannstieger

Copy link
Copy Markdown

Added a function to deduplicate Swarm tasks in container inspection.

Bug reports

Please include:

  • Debian 13 Trixie on aarch64 (Raspberry Pi 5)
  • CheckMK is running on a Docker Container on TrueNAS on an x86 machine while most of the docker containers are on raspberry pies. All of them are in a docker swarm managed by portainer running on one raspi. I want to check docker containers via piggyback but when I update one Container the old one stays on the host with the same swarm name as stopped. mk_docker.py reads the status of both of them and writes it into the same hostname-piggyback file. The GUI shows me this error:
    Summary: Found data from multiple Docker nodes - see service details for more information
    Details: This docker container apparently exists on multiple parent hosts. This should be reflected in the fact that this host has multiple piggyback sources, see the output of the Check_MK service. Hence, no definitive information on the container can be displayed. To resolve this situation, you have two options: 1. configure the docker agent plug-in to use the container IDs as host names, 2. use the ruleset 'Host name translation for piggybacked hosts' to create unique host names for the affected containers.

Proposed changes

Sometimes it is hard for us to assess the quality of a fix.
While it may work for you, it is our job to ensure that it works for everybody.
These are some ways to help us:

  • What is the expected behavior?
    When Docker Swarm recreates a task (health-check-triggered restart, rolling update, image update, etc.) while the old task's container is still present on the node — kept around by Swarm's task history (--task-history-limit) — mk_docker.py should only ever surface one container per logical Swarm service+slot. If the piggyback target for that container is a stable host name across task recreations (e.g. via Checkmk's "Host name translation for piggybacked hosts" rule, used to strip the ever-changing Swarm task-ID suffix), that host should keep receiving a single, unambiguous set of docker_container_* sections.

  • What is the observed behavior?
    MKDockerClient.init builds self.all_containers from every container returned by containers(all=True), which includes exited containers that Swarm still keeps around as task history. Once a service's task gets recreated, both the old (already shut down) container and the new (running) one are present locally and both get inspected in the same agent run. If the piggyback host name has been made stable across restarts — which is necessary in the first place, otherwise every task recreation produces a brand-new Checkmk host because the raw container name embeds a random task ID — both containers end up generating sections for the very same target host, from the very same physical source, in the very same agent execution. Checkmk's "Docker container status" check then goes CRIT with "Found data from multiple Docker nodes" (see Werk #14420), which is misleading: it suggests a cross-node conflict, but in this case only one physical node and one agent run are involved.

  • If it's not obvious from the above: In what way does your patch change the current behavior?
    The patch adds a _dedupe_swarm_tasks() helper that groups the containers coming from _robust_inspect(client, "containers") by their com.docker.swarm.service.name / com.docker.swarm.task.slot labels, and — whenever more than one container shares that key — keeps only the most recently created one (c.attrs["Created"]), dropping older, already-superseded task containers before self.all_containers is populated. Containers without those labels (i.e. not Swarm-managed) pass through unchanged. As a result, at most one container per Swarm service+slot ever reaches section generation, so a piggyback host name that stays stable across task recreations never receives conflicting data from two container instances within one run.

  • Consider writing a unit test that would have failed without your fix.
    from types import SimpleNamespace
    def make_container(id, created, labels=None):
    return SimpleNamespace(attrs={
    "Id": id_,
    "Created": created,
    "Config": {"Labels": labels or {}},
    })
    def test_dedupe_swarm_tasks_keeps_only_newest_per_service_slot():
    old_task = _make_container(
    "old", "2026-09-07T02:00:00.000000000Z",
    labels={
    "com.docker.swarm.service.name": "stalwart_stalwart",
    "com.docker.swarm.task.slot": "1",
    },
    )
    new_task = _make_container(
    "new", "2026-09-07T08:00:00.000000000Z",
    labels={
    "com.docker.swarm.service.name": "stalwart_stalwart",
    "com.docker.swarm.task.slot": "1",
    },
    )
    standalone = _make_container("standalone", "2026-09-07T01:00:00.000000000Z")
    result = _dedupe_swarm_tasks([old_task, new_task, standalone])
    assert len(result) == 2
    assert new_task in result
    assert standalone in result
    assert old_task not in result

Before the fix this function doesn't exist at all, so this test fails outright (import error); more importantly, without the grouping/filtering logic, both old_task and new_task would have passed straight through untouched and both ended up in self.all_containers, i.e. len(result) == 3 and old_task in result.

  • Is this a new problem? What made you submit this PR (new firmware, new device, changed device behavior)?
    Not a regression tied to a specific Docker/Checkmk release — it can happen with any Swarm deployment that keeps task history (the default) as soon as a task gets recreated for any reason. I ran into it on my own homelab Swarm cluster after configuring hostname translation to avoid the opposite problem: without it, every task recreation produces a brand-new Checkmk host, because the raw container name embeds a random, ever-changing Swarm task ID. Once that stable-naming rule was in place, the next task recreation exposed this issue — the already-exited previous task container was still picked up by mk_docker.py (since it queries containers(all=True)) alongside the new one, triggering Werk #14420's "multiple parent hosts" CRIT even though only a single physical node and a single agent run were ever involved.

Added a function to deduplicate Swarm tasks in container inspection.
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@johannstieger

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA. Best regards, Johann

@johannstieger

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA or my organization already has a signed CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants