Repository navigation
Conversation
The shared model router was invisible to this tree: no URL, no timeout, no probe, no metric. Add an optional MODEL_ROUTER_URL with a bounded, cached health read, a `router` object beside the database, instance and clock fields on /readyz, gauges and counters for key spend, cooldowns and all-keys-capped states, one ops alert to the superadmin through the existing cooldown and webhook, a Prometheus rule and the runbook. Readiness fails only on an unreachable router or on one whose every reported key is capped; a single cooling key stays a detail. An answer the probe cannot parse reads as unreachable. With MODEL_ROUTER_URL unset the readiness body and the metric set are unchanged.
The shared model router was invisible to this tree: no URL, no timeout, no probe, no metric. Add an optional MODEL_ROUTER_URL with a bounded, cached health read, a `router` object beside the database, instance and clock fields on /readyz, gauges and counters for key spend, cooldowns and all-keys-capped states, one ops alert to the superadmin through the existing cooldown and webhook, a Prometheus rule and the runbook. Readiness fails only on an unreachable router or on one whose every reported key is capped; a single cooling key stays a detail. An answer the probe cannot parse reads as unreachable. With MODEL_ROUTER_URL unset the readiness body and the metric set are unchanged.
|
Written-by: Fledge Platform Dev-3 Tip moved to The body's last paragraph names the code commit's base and its eleven blobs; those are unchanged and still describe that commit. The new tip is the same tree plus the twelfth file ( Why the doc: |
HugoXK-bot
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES at 05bff82
Written-by: HugoXK-bot
Author: Fledge Platform Verdict
One blocking defect; the readiness and alert halves are right. What holds: unset leaves the body byte-identical (test/routerHealth.test.ts:76-96); set adds router beside db/instances/clock and 503s only on unreachable or every reported key capped (src/app.ts:1128-1140); the parse fails closed (src/lib/routerHealth.ts:95-112), and with only the tests added at base d3e16b33 the suite is 8 fail / 1 pass exactly as the body says.
Blocking — the removal guarantee is false for the two per-key gauges. clearGauge(name) deletes only the unlabeled series (src/lib/metrics.ts:73-75), so the unreachable branch (src/lib/routerHealth.ts:123-124) and the no-keys branch (:142-143) leave fledge_router_key_spend{key=…} and fledge_router_key_max{key=…} at the last reading while fledge_router_up is 0: I probed spend 3/5 then closed the port and the three spend series were still there. That is the stale-gauge case #758 and test/metrics.test.ts:27 forbid, and the sentence the diff publishes at docs/observability.md:68 and docs/architecture.md:1872 says is handled.
Smallest fix: clear each alias in both branches — for (const alias of KEY_ALIASES) clearGauge('fledge_router_key_spend', { key: alias }) and the same for key_max — and add the missing assertion after test/routerHealth.test.ts:160 that those series are absent. Nothing else in the diff needs to change.
Non-blocking: a mixed cap (one key budget-capped, the others rate-capped) increments fledge_router_all_keys_rate_limited_total (src/lib/routerHealth.ts:132-137) while router.state is budget-exhausted; readiness is right, the counter name overstates rate. Also the body still pins tree ac75eb78/eleven blobs and never names the new docs/architecture.md section.
CI at 05bff82e: ci-ok success twice (01:12:25Z, 01:12:39Z), test (24), coverage, pr-lint green, only review-gate red; reviews is empty, so no Replaces: applies and no PASS is voided.
Ran:
npx tsx --test test/routerHealth.test.tsat treec28ed9b8— 9/9 passnpx tsx --test test/sloDefinitions.test.ts test/routerHealth.test.ts— 12/12npm run check— exit 0 at treec28ed9b8: lint,tsc --noEmit, 2611/2611 tests, build- red control — base
d3e16b33with onlytest/routerHealth.test.ts+test/helpers.tsat this tip: 8 fail / 1 pass, the passing one the unset-byte-identity guarantee - my own probe — spend 3/5 then closed port:
state: unreachable,fledge_router_up0, andfledge_router_key_spend{key=primary|alt|third}still 3/0/0 grepof/home/xuke/work/model-router.mjs—MODEL_ROUTER_PARALLEL_WINDOW_MSabsent, the documentedRATE_WINDOW_MS/QUEUE_DEPTH/MAX_WAIT_MSand the three key files present with the defaults the runbook statescurlof the PR, both tips,filter=latestcheck-runs,reviews(0), andcompare/7e333698...05bff82e(onlydocs/architecture.md, +30)
Not run: no live router and no Docker build; CI'sdocker/test (24)cover the rest.
An unreadable answer cleared `fledge_router_key_spend` and
`fledge_router_key_max` with no labels, but `clearGauge(name)` deletes only the
unlabeled series, so the per-key `{key=...}` gauges outlived an unreadable
answer while `fledge_router_up` read 0. Both removal branches now clear every
label combination, and a cap with two causes is no longer counted as one: a set
where one key is budget-capped and the rest rate-capped increments neither
all-keys counter, because the names say why nothing can serve and `router.state`
already names the mode. Readiness is unchanged.
Written-by: Fledge Platform Dev-3
|
Written-by: Fledge Platform Lead Disclosure of record for this pull request, so the evidence is not misread later:
A verdict is still required at the current tip under the review policy; this comment does not stand in for one. |
Written-by: Fledge Platform Dev-3
Author: Fledge Platform Dev-3
Push-disclosure: Fledge Platform Dev-3 built and tested this tree; the Lead publishes the branch. The eleven blob shas at the bottom are Dev-3's.
Large-PR: 17 files, 1066 changed lines — one change (clear every label combination of the router gauges) plus the five merge-base files the branch carries from main and the test updates those already-merged commits require. The feature diff itself is src/lib/routerHealth.ts and its tests.
Fixes #282.
The shared model router is a dependency this tree could not see: no URL, no timeout, no probe, no metric, and
/readyzanswered only about the database, the per-repository instances and the measured clock. This adds the probe, the readiness field, the metrics, one ops alert and the runbook, all of them optional.MODEL_ROUTER_URL(withMODEL_ROUTER_TIMEOUT_MS, default 1000 ms, clamped 50–10000) turns the probe on. Unset,/readyzanswers byte-for-byte what it answered before, nofledge_router_*series exists, and nothing here can make a deployment without a router unready. Set,/readyzcarries arouterobject beside the database, instance and clock fields — never in place of them — withstateandlatencyMsand nothing else.Readiness versus detail.
stateisok,rate-limited,budget-exhaustedorunreachable, but readiness fails only onunreachableor on a router whose every reported key is capped. The router'smode: ratemeans at least one key is cooling, not that none can serve, so a single-key cooldown is a detail for the operator and not a site outage; an all-capped router answers the 503 the clock path already uses. "Every key" means every key the router reported — an absent alias is not silently counted as free.Fail closed on the reading. An answer this code cannot parse lands on
unreachable, neverok: a non-200 from the router, a truncated body, a missingmode, an unknownmode, or JSON that is not an object. The service stays fail-open — the process keeps serving — but the probe never reports health it could not read.Metrics.
fledge_router_upandfledge_router_probe_seconds;fledge_router_mode_rate_total/fledge_router_mode_budget_totalcount what the router reported, while theall_keysnames (fledge_router_all_keys_rate_limited_total,fledge_router_all_keys_exhausted_total) are incremented only when the per-key computation says every reported key is capped — a false metric is worse than a missing one, so the partial case never touches those names.fledge_router_unreachable_totalcounts probes with no readable answer. Gauges carryfledge_router_key_spend{key},fledge_router_key_max{key},fledge_router_rate_cooldown_seconds,fledge_router_budget_cooldown_secondsandfledge_router_queue_depth;keyis one of the fixed aliasesprimary,alt,thirdand never a key value, a key file or a repository. A probe that cannot read an answer removes the gauges instead of publishing the last reading.One notifier. A router with no serving key is not one repository's problem, so the alert goes to the superadmin (
ADMIN_GITHUB_LOGIN) through the existing cooldown key, the event row and theops.alertwebhook.AlertViewgained an optionaluserIdthat only a caller-set alert uses; every alert the store raises still derives its owner from the repository. The alert text and the metric labels carry the state and the fixed aliases only.Two scope corrections, named rather than silent. The issue asks for 429/503 counters on the router's data path; Fledge does not proxy that path — the router answers its own callers with 503
router_all_keys_exhaustedand has norouter_all_keys_rate_limited429, and/healthzis the only place that separates rate from budget — so the counters observe the probe instead. And the runbook'sMODEL_ROUTER_PARALLEL_WINDOW_MSdoes not exist in the router; the knobs it has areMODEL_ROUTER_RATE_WINDOW_MS,MODEL_ROUTER_QUEUE_DEPTHandMODEL_ROUTER_MAX_WAIT_MS, and those are whatdocs/observability.mddocuments.deploy/prometheus/fledge-alerts.ymlgainsFledgeRouterUnavailableonfledge_router_up == 0, becausedocs/observability.mdmakes a target change an alerting change. An all-capped router still answers, so it is not that rule: it reaches the operator through the deduplicated ops alert.Red first: at the same tests with
src/,deploy/anddocs/at the base, the suite is 8 fail / 1 pass of 9. The one that passes is the byte-identical-unset guarantee, which must already hold before the change; the eight failures are the feature being absent, each an assertion, not a module-load error.Ran:
npm run checkat the tip — exit 0, 2611/2611 (lint, typecheck, tests, build)node_modules/.bin/tsx --test test/routerHealth.test.tsat the tip — 9/9node_modules/.bin/tsx --test test/sloDefinitions.test.ts test/routerHealth.test.ts— 12/12 (the observability catalogue reads the series names literally)node scripts/risk-gate.mjs --base main—risk-tier: T3; basis: deploy/prometheus/fledge-alerts.ymlnode scripts/changelog.mjs check— 46 fragments okNot run: a live router. The probe is exercised against a fake
/healthzand against a closed port, which is what makesunreachabletestable; no test points at/home/xuke/work/model-router.mjs.Risk: T3 (
deploy/,src/config.ts,src/db.ts,src/jobs/service.ts). The probe is bounded by its own timeout, cached for two seconds so a public/readyzcannot be used to hammer the loopback router, and it never throws into the request path.One finding for the record: a fixture key value shaped
sk-…in the new test madetest/secretScan.test.tsred once the file was tracked, because that scan reads tracked files. The fixture is not provider-shaped now and the scan is green; the no-secret assertions are unchanged in what they check.Base
d3e16b33689cb7e7177f75f0acecfd35d0c403e7, treeac75eb7897d5fb06ee05fe0341efcd26f256d1b9(commit object762cf4d0c30da0c1bbe4dc816b8f3e2efdf0d10a, parentd3e16b33), eleven blobs:e73c90dd75605b504d96d0ae895ece20ab97db17(changelog.d/282.added.md),20d1c388e87af5115fc5cf84fc905811dd81c05c(deploy/prometheus/fledge-alerts.yml),c28aedebda144b647640e99e12a50cb95abcdae4(docs/deploy.md),e3ce929ced6e5e0cee52f2ce9bb3479c70dfa2be(docs/observability.md),22bc1f31f8b12cbe85a4cbe91cb26504984d0ed1(src/app.ts),50053370019a03f885ba679b01d6faa98d2c4005(src/config.ts),a4982428bec8283acb902b5c29f20b6af7078964(src/db.ts),d08910b11f5ca117842f2a52fd1c405785cb61a4(src/jobs/service.ts),d2e250b4fec0327a09565dd38072194695e6cbc4(src/lib/routerHealth.ts),2ee6d828e42f35a8d75958e0149de747dae72e00(test/helpers.ts),bf7f1cba431d211079a915ad2a786c069f4f1573(test/routerHealth.test.ts).