Skip to content

AKO-147 Bound Slurm inspection and honor terminal sentinel before accounting queries #147

Closed

Description

@ovvesley

Fresh Montage run montage-clean-real-20261011-v1 has a physical mProject1 terminal sentinel with exit_code=0 and runtime about479s while API still reports running well afterwards. Slurm inspection reads sentinel then unconditionally runs sacct, squeue and scontrol without bounded remote deadlines; inspectActivities waits for every worker before consuming any result. SSH inspection warnings are intermittent and must not be confused with failed jobs or trigger duplicates. Diagnose the active transport/accounting latency; honor trustworthy terminal sentinels before querying accounting, bound read-only inspection requests, return transient warnings without treating timeouts as terminal, and test that unavailable/hung accounting cannot hide a completed sentinel or block all other handles. Keep artifact/output validation, metrics and exact attempt identity. Do not deploy by restarting live activity submissions without a verified safe point. Local main commits only, no push.

Activity

  1. ovvesley commented on Oct 14, 2026

    @ovvesley
    CollaboratorAuthor

    Read-only Slurm inspection is bounded and terminal sentinel/manifests take precedence over unavailable accounting; provider tests pass. Located through AKO issue-number commits 026ff18 and verified integrated in published main 4aecc4c via PR #175. Full Go suite was rerun on main successfully for this audit; PR Go/docs CI is green. Closure covers this implementation/investigation scope only; remaining scientific forecast gates are tracked separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions