Repository navigation
sim: the floor signs in again when its login runs out - #146
Merged
Merged
Conversation
A login is good for one shift and the simulated floor runs for weeks. On 2026-10-06 both lab plants' floors stopped twelve hours to the minute after they started: every request answered 401, the step loop logged "shop floor step failed" and tried again on the next tick, forever. Every read and write the floor makes now goes through one place that knows a login can run out. It signs in again halfway through the token's life - read from the token the plant issued, not guessed from config - so the first failed step never happens; a 401 is answered by signing in once and retrying the step; a second refusal is logged once as a failure rather than looped on. Signed-off-by: scottkalw <scottkalwei@gmail.com>
A fake plant whose token stops answering after a stated number of requests, so the behaviour is tested rather than the twelve hours. Eight tests: the floor keeps stepping, the refused step itself is retried (a write as well as a read), it renews halfway through the token's life before anything is refused, the life comes from the token the plant issued rather than the floor's own config, an unreadable token falls back to that config, and a login the plant will not re-issue is logged once rather than once a step. Writing them found one thing: `exp` is minted against `utcnow().timestamp()`, which reads a naive UTC instant in the machine's own zone, so comparing it to `time.time()` was five hours out on this box. The floor now compares it the way `auth.read_token` does. Signed-off-by: scottkalw <scottkalwei@gmail.com>
A plant can be up, answering every request, and have nothing happening on its floor - which is what both lab plants were for nine hours on 2026-10-06, with health, smoke and every screen saying they were fine. /health now carries the newest check's stamp and how long ago it was: one lookup down the primary key, no scan. Null on a plant where nobody has recorded one yet, because never inspected is not a long time ago and is not zero. Doc section and the CHANGELOG entry under Fixed. Signed-off-by: scottkalw <scottkalwei@gmail.com>
|
Thanks for the first pull request. Checks: every commit signed off ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The simulated shop floor signs in again when its login runs out, and
GET /healthnow says when the floor last recorded a check.
Why
On 2026-10-06 both lab plants' floors stopped working at 00:52 and stayed stopped
for nine hours. Nothing had crashed. The floor signs in to the plant like anybody
else; a login is good for one shift (
[auth] token_ttl_seconds, twelve hours);twelve hours to the minute after the plants started, every request the floor made
came back 401. The step loop caught each one as an HTTP error, logged
shop floor step failed and tried again on the next tick — 9,485 of those lines on
bottling and 7,381 on machining before anyone noticed. No inspection, no stop
named, no non-conformance reviewed, for nine hours, on the plants the analysis is
demonstrated on.
Nothing said so. The plant answered every screen and every health check perfectly
the whole time, because a work order read answers fine whether or not anybody is
working.
How
src/fsmes/sim/operations.py— every read and write the floor makes now goesthrough one place that knows a login can run out, instead of a dozen call sites
that didn't.
step never happens. How long the token is good for is read out of the token the
plant issued, not out of this floor's copy of the configuration — a floor
talking to a plant whose settings it does not share still renews in time. An
unreadable token falls back to the configured TTL.
again. Recovering on the next tick would be a quieter version of the same bug:
the inspection due at the moment the token died would never be recorded.
the floor signed in again, with why. Six loops share theclient, so the re-sign-in is behind a lock and a task that finds the login
already replaced says nothing — that is the same event, not a second one.
running floor) is logged once, as an error, not once a step.
GET /healthgains one line:The newest check by primary key — one lookup, no scan.
nullfor both on a plantwhere nobody has recorded a check yet: never inspected is not a long time ago,
and it is not zero (house rule 2). A plant that is up and a plant that is being
worked are two facts, and a monitor holding only the first reads silence as
everything being fine — the same argument
shadowandwatchingalready ride onhealth for.
Only the simulated floor had this fault. The other three places under
src/fsmes/sim/that sign in once —runner.py(a scored run),ui_check.py(theUI crawl) and
agent_eval.py(the agent evals) — all finish long inside one shift.How it was verified
Eight new tests against a fake plant whose token stops answering after a
stated number of requests, so the behaviour is tested rather than the twelve
hours. The tokens it issues come from the MES's own
issue_token, so the floorreads the expiry out of the real shape. They pin: the floor keeps stepping; the
refused step itself is retried, a write as well as a read; it renews before
anything is refused; the life comes from the token and not from config; an
unreadable token falls back to config; a login the plant will not re-issue is
said once; and a floor that was never signed in is not quietly signed in.
One new test on
/health, and the full suite green locally(
ruff check .clean,pytestall passing with[dev,mcp,agent]).A scratch plant on loopback with
token_ttl_seconds = 60, run for fiveminutes — a plant whose logins run out every minute instead of every shift,
so twelve hours of the real fault happen in five. Over that run the floor's
log holds 19
the floor signed in againlines, every one of themthe login was halfway through its life, and zero
shop floor step failedlines — it never met a 401 at all. 66 inspections recorded, the lastof them 3.5 seconds before the run ended, and
/healthagreed:On the same plant before any step had run, that read
{"last_check_recorded": null, "seconds_since": null}.Writing the tests found one thing worth saying:
expis minted againstutcnow().timestamp(), which reads a naive UTC instant in the machine's own zone,so comparing it to
time.time()was five hours out on a UTC−5 box. The floor nowcompares it the way
auth.read_tokendoes. Nothing about the token format changed.