Skip to content

sim: the floor signs in again when its login runs out - #146

Merged
kalwei merged 3 commits into
mainfrom
fix/floor-sim-relogin
Oct 6, 2026
Merged

kalwei merged 3 commits into
mainfrom
fix/floor-sim-relogin

Conversation

@kalwei

@kalwei kalwei commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

What

The simulated shop floor signs in again when its login runs out, and GET /health
now says when the floor last recorded a check.

Why

On 2026-10-06 both lab plants' floors stopped working at 00:52 and stayed stopped
for nine hours. Nothing had crashed. The floor signs in to the plant like anybody
else; a login is good for one shift ([auth] token_ttl_seconds, twelve hours);
twelve hours to the minute after the plants started, every request the floor made
came back 401. The step loop caught each one as an HTTP error, logged
shop floor step failed and tried again on the next tick — 9,485 of those lines on
bottling and 7,381 on machining before anyone noticed. No inspection, no stop
named, no non-conformance reviewed, for nine hours, on the plants the analysis is
demonstrated on.

Nothing said so. The plant answered every screen and every health check perfectly
the whole time, because a work order read answers fine whether or not anybody is
working.

How

src/fsmes/sim/operations.py — every read and write the floor makes now goes
through one place that knows a login can run out, instead of a dozen call sites
that didn't.

  • It replaces its login halfway through the token's life, so the first failed
    step never happens. How long the token is good for is read out of the token the
    plant issued, not out of this floor's copy of the configuration — a floor
    talking to a plant whose settings it does not share still renews in time. An
    unreadable token falls back to the configured TTL.
  • A 401 it meets anyway is answered by signing in once and sending that step
    again. Recovering on the next tick would be a quieter version of the same bug:
    the inspection due at the moment the token died would never be recorded.
  • One line per event, the floor signed in again, with why. Six loops share the
    client, so the re-sign-in is behind a lock and a task that finds the login
    already replaced says nothing — that is the same event, not a second one.
  • A plant that will not issue a new login at all (a password changed under a
    running floor) is logged once, as an error, not once a step.

GET /health gains one line:

"floor": { "last_check_recorded": "2026-10-06T14:52:16", "seconds_since": 31.4 }

The newest check by primary key — one lookup, no scan. null for both on a plant
where nobody has recorded a check yet: never inspected is not a long time ago,
and it is not zero (house rule 2). A plant that is up and a plant that is being
worked are two facts, and a monitor holding only the first reads silence as
everything being fine — the same argument shadow and watching already ride on
health for.

Only the simulated floor had this fault. The other three places under
src/fsmes/sim/ that sign in once — runner.py (a scored run), ui_check.py (the
UI crawl) and agent_eval.py (the agent evals) — all finish long inside one shift.

How it was verified

  • Eight new tests against a fake plant whose token stops answering after a
    stated number of requests, so the behaviour is tested rather than the twelve
    hours. The tokens it issues come from the MES's own issue_token, so the floor
    reads the expiry out of the real shape. They pin: the floor keeps stepping; the
    refused step itself is retried, a write as well as a read; it renews before
    anything is refused; the life comes from the token and not from config; an
    unreadable token falls back to config; a login the plant will not re-issue is
    said once; and a floor that was never signed in is not quietly signed in.

  • One new test on /health, and the full suite green locally
    (ruff check . clean, pytest all passing with [dev,mcp,agent]).

  • A scratch plant on loopback with token_ttl_seconds = 60, run for five
    minutes
    — a plant whose logins run out every minute instead of every shift,
    so twelve hours of the real fault happen in five. Over that run the floor's
    log holds 19 the floor signed in again lines, every one of them
    the login was halfway through its life, and zero shop floor step failed lines — it never met a 401 at all. 66 inspections recorded, the last
    of them 3.5 seconds before the run ended, and /health agreed:

    "floor": { "last_check_recorded": "2026-10-06T15:53:34.521199", "seconds_since": 3.5 }

    On the same plant before any step had run, that read
    {"last_check_recorded": null, "seconds_since": null}.

Writing the tests found one thing worth saying: exp is minted against
utcnow().timestamp(), which reads a naive UTC instant in the machine's own zone,
so comparing it to time.time() was five hours out on a UTC−5 box. The floor now
compares it the way auth.read_token does. Nothing about the token format changed.

kalwei added 3 commits October 6, 2026 10:09
A login is good for one shift and the simulated floor runs for weeks.
On 2026-10-06 both lab plants' floors stopped twelve hours to the minute
after they started: every request answered 401, the step loop logged
"shop floor step failed" and tried again on the next tick, forever.

Every read and write the floor makes now goes through one place that
knows a login can run out. It signs in again halfway through the token's
life - read from the token the plant issued, not guessed from config -
so the first failed step never happens; a 401 is answered by signing in
once and retrying the step; a second refusal is logged once as a failure
rather than looped on.

Signed-off-by: scottkalw <scottkalwei@gmail.com>
A fake plant whose token stops answering after a stated number of
requests, so the behaviour is tested rather than the twelve hours.
Eight tests: the floor keeps stepping, the refused step itself is
retried (a write as well as a read), it renews halfway through the
token's life before anything is refused, the life comes from the token
the plant issued rather than the floor's own config, an unreadable token
falls back to that config, and a login the plant will not re-issue is
logged once rather than once a step.

Writing them found one thing: `exp` is minted against
`utcnow().timestamp()`, which reads a naive UTC instant in the
machine's own zone, so comparing it to `time.time()` was five hours out
on this box. The floor now compares it the way `auth.read_token` does.

Signed-off-by: scottkalw <scottkalwei@gmail.com>
A plant can be up, answering every request, and have nothing happening
on its floor - which is what both lab plants were for nine hours on
2026-10-06, with health, smoke and every screen saying they were fine.

/health now carries the newest check's stamp and how long ago it was:
one lookup down the primary key, no scan. Null on a plant where nobody
has recorded one yet, because never inspected is not a long time ago and
is not zero.

Doc section and the CHANGELOG entry under Fixed.

Signed-off-by: scottkalw <scottkalwei@gmail.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Thanks for the first pull request. Checks: every commit signed off (git commit -s), ruff and pytest green, and the house-rules checklist in the template. The maintainer reviews within a few days.

@kalwei
kalwei merged commit 3bdc088 into main Oct 6, 2026
17 checks passed
@kalwei
kalwei deleted the fix/floor-sim-relogin branch October 6, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant