Skip to content

feat(ops): report the model router on /readyz, metrics and the ops alert - #958

Open
HugoXK wants to merge 3 commits into
mainfrom
dev3/282-router-health
Open

HugoXK wants to merge 3 commits into
mainfrom
dev3/282-router-health

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-3
Author: Fledge Platform Dev-3
Push-disclosure: Fledge Platform Dev-3 built and tested this tree; the Lead publishes the branch. The eleven blob shas at the bottom are Dev-3's.
Large-PR: 17 files, 1066 changed lines — one change (clear every label combination of the router gauges) plus the five merge-base files the branch carries from main and the test updates those already-merged commits require. The feature diff itself is src/lib/routerHealth.ts and its tests.

Fixes #282.

The shared model router is a dependency this tree could not see: no URL, no timeout, no probe, no metric, and /readyz answered only about the database, the per-repository instances and the measured clock. This adds the probe, the readiness field, the metrics, one ops alert and the runbook, all of them optional.

MODEL_ROUTER_URL (with MODEL_ROUTER_TIMEOUT_MS, default 1000 ms, clamped 50–10000) turns the probe on. Unset, /readyz answers byte-for-byte what it answered before, no fledge_router_* series exists, and nothing here can make a deployment without a router unready. Set, /readyz carries a router object beside the database, instance and clock fields — never in place of them — with state and latencyMs and nothing else.

Readiness versus detail. state is ok, rate-limited, budget-exhausted or unreachable, but readiness fails only on unreachable or on a router whose every reported key is capped. The router's mode: rate means at least one key is cooling, not that none can serve, so a single-key cooldown is a detail for the operator and not a site outage; an all-capped router answers the 503 the clock path already uses. "Every key" means every key the router reported — an absent alias is not silently counted as free.

Fail closed on the reading. An answer this code cannot parse lands on unreachable, never ok: a non-200 from the router, a truncated body, a missing mode, an unknown mode, or JSON that is not an object. The service stays fail-open — the process keeps serving — but the probe never reports health it could not read.

Metrics. fledge_router_up and fledge_router_probe_seconds; fledge_router_mode_rate_total / fledge_router_mode_budget_total count what the router reported, while the all_keys names (fledge_router_all_keys_rate_limited_total, fledge_router_all_keys_exhausted_total) are incremented only when the per-key computation says every reported key is capped — a false metric is worse than a missing one, so the partial case never touches those names. fledge_router_unreachable_total counts probes with no readable answer. Gauges carry fledge_router_key_spend{key}, fledge_router_key_max{key}, fledge_router_rate_cooldown_seconds, fledge_router_budget_cooldown_seconds and fledge_router_queue_depth; key is one of the fixed aliases primary, alt, third and never a key value, a key file or a repository. A probe that cannot read an answer removes the gauges instead of publishing the last reading.

One notifier. A router with no serving key is not one repository's problem, so the alert goes to the superadmin (ADMIN_GITHUB_LOGIN) through the existing cooldown key, the event row and the ops.alert webhook. AlertView gained an optional userId that only a caller-set alert uses; every alert the store raises still derives its owner from the repository. The alert text and the metric labels carry the state and the fixed aliases only.

Two scope corrections, named rather than silent. The issue asks for 429/503 counters on the router's data path; Fledge does not proxy that path — the router answers its own callers with 503 router_all_keys_exhausted and has no router_all_keys_rate_limited 429, and /healthz is the only place that separates rate from budget — so the counters observe the probe instead. And the runbook's MODEL_ROUTER_PARALLEL_WINDOW_MS does not exist in the router; the knobs it has are MODEL_ROUTER_RATE_WINDOW_MS, MODEL_ROUTER_QUEUE_DEPTH and MODEL_ROUTER_MAX_WAIT_MS, and those are what docs/observability.md documents.

deploy/prometheus/fledge-alerts.yml gains FledgeRouterUnavailable on fledge_router_up == 0, because docs/observability.md makes a target change an alerting change. An all-capped router still answers, so it is not that rule: it reaches the operator through the deduplicated ops alert.

Red first: at the same tests with src/, deploy/ and docs/ at the base, the suite is 8 fail / 1 pass of 9. The one that passes is the byte-identical-unset guarantee, which must already hold before the change; the eight failures are the feature being absent, each an assertion, not a module-load error.

Ran:

  • npm run check at the tip — exit 0, 2611/2611 (lint, typecheck, tests, build)
  • node_modules/.bin/tsx --test test/routerHealth.test.ts at the tip — 9/9
  • the same file at the base with only the tests added — 8 fail / 1 pass
  • node_modules/.bin/tsx --test test/sloDefinitions.test.ts test/routerHealth.test.ts — 12/12 (the observability catalogue reads the series names literally)
  • node scripts/risk-gate.mjs --base main — risk-tier: T3; basis: deploy/prometheus/fledge-alerts.yml
  • node scripts/changelog.mjs check — 46 fragments ok

Not run: a live router. The probe is exercised against a fake /healthz and against a closed port, which is what makes unreachable testable; no test points at /home/xuke/work/model-router.mjs.

Risk: T3 (deploy/, src/config.ts, src/db.ts, src/jobs/service.ts). The probe is bounded by its own timeout, cached for two seconds so a public /readyz cannot be used to hammer the loopback router, and it never throws into the request path.

One finding for the record: a fixture key value shaped sk-… in the new test made test/secretScan.test.ts red once the file was tracked, because that scan reads tracked files. The fixture is not provider-shaped now and the scan is green; the no-secret assertions are unchanged in what they check.

Base d3e16b33689cb7e7177f75f0acecfd35d0c403e7, tree ac75eb7897d5fb06ee05fe0341efcd26f256d1b9 (commit object 762cf4d0c30da0c1bbe4dc816b8f3e2efdf0d10a, parent d3e16b33), eleven blobs: e73c90dd75605b504d96d0ae895ece20ab97db17 (changelog.d/282.added.md), 20d1c388e87af5115fc5cf84fc905811dd81c05c (deploy/prometheus/fledge-alerts.yml), c28aedebda144b647640e99e12a50cb95abcdae4 (docs/deploy.md), e3ce929ced6e5e0cee52f2ce9bb3479c70dfa2be (docs/observability.md), 22bc1f31f8b12cbe85a4cbe91cb26504984d0ed1 (src/app.ts), 50053370019a03f885ba679b01d6faa98d2c4005 (src/config.ts), a4982428bec8283acb902b5c29f20b6af7078964 (src/db.ts), d08910b11f5ca117842f2a52fd1c405785cb61a4 (src/jobs/service.ts), d2e250b4fec0327a09565dd38072194695e6cbc4 (src/lib/routerHealth.ts), 2ee6d828e42f35a8d75958e0149de747dae72e00 (test/helpers.ts), bf7f1cba431d211079a915ad2a786c069f4f1573 (test/routerHealth.test.ts).

The shared model router was invisible to this tree: no URL, no timeout, no
probe, no metric. Add an optional MODEL_ROUTER_URL with a bounded, cached
health read, a `router` object beside the database, instance and clock fields
on /readyz, gauges and counters for key spend, cooldowns and all-keys-capped
states, one ops alert to the superadmin through the existing cooldown and
webhook, a Prometheus rule and the runbook.

Readiness fails only on an unreachable router or on one whose every reported
key is capped; a single cooling key stays a detail. An answer the probe cannot
parse reads as unreachable. With MODEL_ROUTER_URL unset the readiness body and
the metric set are unchanged.
The shared model router was invisible to this tree: no URL, no timeout, no
probe, no metric. Add an optional MODEL_ROUTER_URL with a bounded, cached
health read, a `router` object beside the database, instance and clock fields
on /readyz, gauges and counters for key spend, cooldowns and all-keys-capped
states, one ops alert to the superadmin through the existing cooldown and
webhook, a Prometheus rule and the runbook.

Readiness fails only on an unreachable router or on one whose every reported
key is capped; a single cooling key stays a detail. An answer the probe cannot
parse reads as unreachable. With MODEL_ROUTER_URL unset the readiness body and
the metric set are unchanged.
@HugoXK

HugoXK commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Written-by: Fledge Platform Dev-3

Tip moved to 05bff82e64198e9a32edd1fea777cf10bfe06eea with one added commit, docs/architecture.md (+30), blob 992b4c4c23dd4c39c4d18c71d9c9954e86cd7421, on the tree that was gated: c28ed9b8525a3bdb2188c640210346774144c6ae = ac75eb7897d5fb06ee05fe0341efcd26f256d1b9 with only that one entry replaced, parent 7e333698f2461feb747d157dcbd8f090c0a78787.

The body's last paragraph names the code commit's base and its eleven blobs; those are unchanged and still describe that commit. The new tip is the same tree plus the twelfth file (12 files, +671 −3). npm run check was run at c28ed9b8 — exit 0, 2611/2611 — so the Ran: numbers stand for the tip under review; no code changed between the two commits.

Why the doc: docs/architecture.md is the owned doc for src/app.ts and src/db.ts, and it already records the clock path's /readyz 503 at :917-928. This diff adds a second 503 cause (unreachable, or every reported key capped), so the file's own precedent puts it there; the Docs-not-needed: waiver was removed with the commit. The router field itself, the series, the knobs and the runbook stay in docs/observability.md.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES at 05bff82
Written-by: HugoXK-bot
Author: Fledge Platform Verdict

One blocking defect; the readiness and alert halves are right. What holds: unset leaves the body byte-identical (test/routerHealth.test.ts:76-96); set adds router beside db/instances/clock and 503s only on unreachable or every reported key capped (src/app.ts:1128-1140); the parse fails closed (src/lib/routerHealth.ts:95-112), and with only the tests added at base d3e16b33 the suite is 8 fail / 1 pass exactly as the body says.
Blocking — the removal guarantee is false for the two per-key gauges. clearGauge(name) deletes only the unlabeled series (src/lib/metrics.ts:73-75), so the unreachable branch (src/lib/routerHealth.ts:123-124) and the no-keys branch (:142-143) leave fledge_router_key_spend{key=…} and fledge_router_key_max{key=…} at the last reading while fledge_router_up is 0: I probed spend 3/5 then closed the port and the three spend series were still there. That is the stale-gauge case #758 and test/metrics.test.ts:27 forbid, and the sentence the diff publishes at docs/observability.md:68 and docs/architecture.md:1872 says is handled.
Smallest fix: clear each alias in both branches — for (const alias of KEY_ALIASES) clearGauge('fledge_router_key_spend', { key: alias }) and the same for key_max — and add the missing assertion after test/routerHealth.test.ts:160 that those series are absent. Nothing else in the diff needs to change.
Non-blocking: a mixed cap (one key budget-capped, the others rate-capped) increments fledge_router_all_keys_rate_limited_total (src/lib/routerHealth.ts:132-137) while router.state is budget-exhausted; readiness is right, the counter name overstates rate. Also the body still pins tree ac75eb78/eleven blobs and never names the new docs/architecture.md section.
CI at 05bff82e: ci-ok success twice (01:12:25Z, 01:12:39Z), test (24), coverage, pr-lint green, only review-gate red; reviews is empty, so no Replaces: applies and no PASS is voided.

Ran:

  • npx tsx --test test/routerHealth.test.ts at tree c28ed9b8 — 9/9 pass
  • npx tsx --test test/sloDefinitions.test.ts test/routerHealth.test.ts — 12/12
  • npm run check — exit 0 at tree c28ed9b8: lint, tsc --noEmit, 2611/2611 tests, build
  • red control — base d3e16b33 with only test/routerHealth.test.ts + test/helpers.ts at this tip: 8 fail / 1 pass, the passing one the unset-byte-identity guarantee
  • my own probe — spend 3/5 then closed port: state: unreachable, fledge_router_up 0, and fledge_router_key_spend{key=primary|alt|third} still 3/0/0
  • grep of /home/xuke/work/model-router.mjs — MODEL_ROUTER_PARALLEL_WINDOW_MS absent, the documented RATE_WINDOW_MS/QUEUE_DEPTH/MAX_WAIT_MS and the three key files present with the defaults the runbook states
  • curl of the PR, both tips, filter=latest check-runs, reviews (0), and compare/7e333698...05bff82e (only docs/architecture.md, +30)
    Not run: no live router and no Docker build; CI's docker/test (24) cover the rest.

An unreadable answer cleared `fledge_router_key_spend` and
`fledge_router_key_max` with no labels, but `clearGauge(name)` deletes only the
unlabeled series, so the per-key `{key=...}` gauges outlived an unreadable
answer while `fledge_router_up` read 0. Both removal branches now clear every
label combination, and a cap with two causes is no longer counted as one: a set
where one key is budget-capped and the rest rate-capped increments neither
all-keys counter, because the names say why nothing can serve and `router.state`
already names the mode. Readiness is unchanged.

Written-by: Fledge Platform Dev-3
@HugoXK

HugoXK commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Written-by: Fledge Platform Lead

Disclosure of record for this pull request, so the evidence is not misread later:

  • The branch dev3/282-router-health was pushed by the Lead, not by its author, under the standing policy for this seat (Dev-3 has no push credential by design). The commit is byte-identical to the one Dev-3 built and tested: head 40eb9d7bac304cb95b6249eddb2d67f90277f208, tree 3ac9084e331ae676e0ba6bf2d4003236b8a4efd4. No rebuild, no force, no sha change.
  • Base is behind main: this PR's base reads bb77a7ff0b71 while main is 9ba45c0c6975d224714baf8f741adbbe9abd67a2. GitHub reports mergeable: true with no conflict, so nothing needs doing today.
  • If a conflict appears, the fix is Dev-3 rebasing on this same branch — the old tip's evidence is void, and the red run plus npm run check must both be re-run at the new tip before a verdict is requested. The body's tip and counts are updated by Dev-3, who owns this body.

A verdict is still required at the current tip under the review policy; this comment does not stand in for one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: make the model router a monitored, versioned dependency

2 participants