fix(service): count each GitHub request with its HTTP status

docs/TELEMETRY.md

@@ -85,7 +85,7 @@ writes by name.
 | `devops_cli_rag_query_duration_ms` | Histogram | `ms` | RAG retrieval time, by stage: embedding, search, ranking, or total for the whole query. |
 | `devops_cli_qdrant_retries_total` | Counter | `1` | Qdrant requests retried after a transient error, by operation and error type. |
 | `devops_cli_rag_points` | Gauge | `1` | Points stored in RAG vector collections, by collection. |
-| `devops_cli_github_requests_total` | Counter | `1` | GitHub requests the rate limiter paced, each attempt and retry once, by resource, method, repo and row. The resource is the rate limit resource the request spends, such as core, graphql or search. The method is the HTTP method; a GraphQL request is a POST, and a gh command other than gh api is a POST when it writes and a GET when it reads. The repo is the repository of the roadmap Service worker that sent the request, one of service.repos, and the row is the job row that sent it, such as metrics, intake, refine or alternatives; each is other outside a worker or a row. A gh command and a request of the GitHub session both count. |
+| `devops_cli_github_requests_total` | Counter | `1` | GitHub requests the rate limiter paced, each attempt and retry once, by resource, method, status, repo and row. The resource is the rate limit resource the request spends, such as core, graphql or search. The method is the HTTP method; a GraphQL request is a POST, and a gh command other than gh api is a POST when it writes and a GET when it reads. The status is the HTTP status of the answer, such as 200 or 304. It is none for a gh command or for an attempt that got no answer. The repo is the repository of the roadmap Service worker that sent the request, one of service.repos, and the row is the job row that sent it, such as metrics, intake, refine or alternatives; each is other outside a worker or a row. A gh command and a request of the GitHub session both count. |
 | `devops_cli_github_throttle_wait_seconds` | Histogram | `s` | Time devops-cli waited before it sent a refused GitHub request again, by resource: one observation for each throttle. GitHub refused the request with a rate limit, or the GitHub session got a 5xx for a read. |
 | `devops_cli_github_graphql_points_total` | Counter | `1` | GraphQL points that GitHub reported each answer cost in its rateLimit field. |
 | `devops_cli_firecrawl_searches_total` | Counter | `1` | Firecrawl web searches, by result: ok, the HTTP status of a search that Firecrawl refused, such as 402 when the account has no credits or 429 at its rate limit, or error for a search that got no answer or a reply with no web results. |

k8s/README.md

@@ -627,7 +627,7 @@ kubectl -n devops logs -f job/<name>
 9. To rotate a credential, update it in the keyring: `uv run devops config set service.webhook_secrets`, or, for the App, generate a new private key and run `uv run devops config set service.github_app.private_key --from-file <key.pem>`. Then run `uv run devops k8s push-secrets --only devops/devops-cli --rotate`. It restarts `roadmap-service`. Delete the old private key on the App's page after the restart.
 10. Run one service per set of repos. While it runs, use `devops roadmap run --dry-run` (#981).
 11. The Service keeps three files in `/home/devops/.data/roadmap/` on volume `roadmap-service-home`, beside the `schedule.json` of each repository: `service-halt.json` (GitHub refused the identity), `service-budget.json` (the GitHub requests of the last hour) and `service-failures.json` (the jobs that failed rounds in a row). A restart reads them back, so a restart does not clear a halt, the budget or a suspended job. A round that ends because the Service is stopping, for example in a rollout, does not count as a failed round: the count of its job stays as it was. The log line that suspends a job names the class of the error of each of its failed rounds in a row; the file keeps only the count, so a round that an earlier process counted reads as `unknown`. A rate limit that one request gets stops the requests of all workers until its wait ends. When GitHub refuses the identity, for example because the App's installation is suspended, the Service halts and logs `GitHub refused the identity`. To clear the halt after you fix the cause, remove the file and restart the Service: `kubectl -n devops exec deploy/roadmap-service -- rm /home/devops/.data/roadmap/service-halt.json`, then `kubectl -n devops delete pod -l app.kubernetes.io/name=roadmap-service`.
-12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)), except `devops_cli_service_identity_refused_total`, whose label set starts with the first refusal. Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`.
+12. The Service's `/metrics` shows the budget and the suspended jobs (#2475): `devops_cli_service_budget_limit_requests` (the hourly budget), `devops_cli_service_budget_spent_requests` (the GitHub requests of the last hour), `devops_cli_service_budget_holds_total` (one count each time a hold begins), `devops_cli_service_job_suspended` (1 while a job of a repository, by `repo` and `row`, waits for new input after it failed 3 rounds in a row) and `devops_cli_service_implement_failures_total` (one count for each runner Job that left a stop, by the failure `class` and `repo`, when a round reports the Job; #3065; the values of `class` are the classes in [docs/ROADMAP_WORKFLOW.md](../docs/ROADMAP_WORKFLOW.md), The failure classes, and a panel or rule matches no other value, #3476). Each counter of the Service starts its label sets at zero (#3495, [docs/TELEMETRY.md](../docs/TELEMETRY.md)), except `devops_cli_service_identity_refused_total`, whose label set starts with the first refusal. Each GitHub request counts in `devops_cli_github_requests_total` under its `method`, its `status` (the HTTP status of the answer, or `none` for a gh command or an attempt with no answer), the `repo` of the worker that sent it and the job `row` (`metrics`, `intake`, `refine`, `stage`, `implement`, `stage_ready` or `alternatives`); a request outside a worker or a row counts as `other`. To find what spent the budget, run `sum by (repo, row, method) (increase(devops_cli_github_requests_total{job="roadmap-service"}[1h]))` in Grafana Explore. A delivery of the App bot's own write counts as outcome `self` in `devops_cli_service_webhook_deliveries_total`.
 13. A refine round writes at most 2 comments on one item in 24 hours. A round that finds 2 refine comments of the last 24 hours on an item leaves the item alone, with no model call and no comment. The item waits on a person and takes no place under the round's limit, so the round refines the next item. For each newest comment on the item that the cap leaves with no answer, the round records the time of that comment in the item's job record (`refine.capped_after`), logs `Refine left #<number> alone and wrote no comment`, and adds 1 to `devops_cli_roadmap_refine_comments_capped_total`. A later round that finds no newer comment does not log or count the item again. A person refines the item at the cap with `devops roadmap refine --item <number> --confirm`. Each Firecrawl search counts in `devops_cli_firecrawl_searches_total` by `result`: `ok`, the HTTP status of a refused search, or `error`. A failed search logs `A Firecrawl web search failed` with the status or the error class, and never the key or the query.
 14. The row `alternatives` of the model lane runs after `refine`: on a first run, and 719 minutes after its last success. It runs `devops roadmap alternatives --confirm` for the repository. A round checks at most `roadmap.alternatives.first_pass_limit` items (20 by default) while an item has no check, and at most `roadmap.alternatives.limit` items (5 by default) after. The row runs about twice a day, so a first pass of 141 items takes about 4 days. A round takes first the items with no check, then the items whose check is older than `roadmap.alternatives.recheck_days` (30 by default) or whose text changed. A webhook makes the row due at no other time. The row changes no label and closes no issue. It does nothing, and logs `The alternatives check of <repo> is off`, for a private repository and when no Firecrawl key is set. If the check of an item fails, the row records the failure in the item's job record (`alternatives.failed_at` and `alternatives.failed_error`, the class of the error), writes the other checks and succeeds. The row fails, and counts toward the suspension after 3 failed rounds, only when it tried items and checked none. A round takes a failed item again after `roadmap.alternatives.retry_days` (3 by default). Each written check adds 1 to `devops_cli_roadmap_alternatives_total` by `verdict`: `covers`, `partly`, `none` or `not-applicable`. An item of the type `test`, `docs`, `refactor`, `chore`, `ci` or `perf`, and a `fix` with a Python traceback in its body, gets `not-applicable` with no model call; every other item must run a search before the check accepts a verdict, and gets one more turn to search. A `covers` verdict makes refine ask a person to adopt the tool or to keep the item with a reason. The item waits on a person until a reply answers the question, and Ready items with a new covering tool go back to New. Refine records the tool it asked about in the job record (`refine.alternatives_asked`). `devops roadmap run --dry-run` lists the row when it is due.
 15. The row `stage` follows `refine` on the model lane (#2659). It runs `devops roadmap stage --confirm` in the model clone: it is due every 37 minutes, on a first run, and in a round where `refine` set an item Ready. It stages the Ready items that `roadmap.stage.filter` names, at most `roadmap.stage.limit` in a round, plans no new item while the count of open Planned items is at `roadmap.stage.planned_limit` (an item with the priority `P0-Critical` passes it, #3487), and splits a task group over `roadmap.stage.step_tokens`. Each task group has a kind (#3005): a `behavior` group has a failing test, and an `other` group has no test but a command or the steps of a manual check that prove it. The job sends a reply that breaks this rule back to the model once. It leaves out an item whose change folder is on the default branch or in its archive (#2714): a round reads that tree once with `git ls-tree`, so a merged staging pull request never makes a second one. The pull request it opens is a draft with the title `docs(openspec): plan #<issue> <summary>`. The job does not mark it ready: the row `stage_ready` does (step 18). Before it opens one, it checks each path of the plan (`reads`, `changes`, the code spans of the tasks and `verify`) against the tracked files of the commit it plans, read with one `git ls-files` for the round (#3010). A path that is no file there, and that no group up to its own creates, goes back to the model once with the files of the same name. When a later group creates the path, the problem names that group and tells the model to move it before the group that names the path (#3017). A second reply with a missing path stops the item, so no draft opens and the implement Job never meets a wrong path. `k8s/devops/configmap.example.yaml` holds the five settings, and `devops k8s deploy-stack --stack devops` applies them; each also has a variable, `DEVOPS_CLI_ROADMAP_STAGE_FILTER`, `DEVOPS_CLI_ROADMAP_STAGE_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_PLANNED_LIMIT`, `DEVOPS_CLI_ROADMAP_STAGE_STEP_TOKENS` and `DEVOPS_CLI_ROADMAP_STAGE_TOOLSET`. The setting `roadmap.stage.toolset` is `extended` by default (#3238): a plan run then has three tools more than the nine that read the staged commit. `references` counts a text in every file of the commit. `read_issue` and `find_issues` read issues and pull requests of the repository on GitHub, as GET requests with the token of the Service and no other permission. `base` leaves them out. Each item of a round counts once in `devops_cli_roadmap_stage_total` by its `result`: `staged`, `skipped`, `needs_split` or `failed`. A failed item is recorded in its job record with the class of its error and the time (#3493), and the job leaves it out for `roadmap.stage.retry_hours` hours (6 by default) while it takes the other items. The row fails a round only when it tried items and staged, stopped or returned none, or when a failure belongs to no item; three such rounds on the same input suspend the row. The ledger file keeps the class of each error, and a round that the stop of the Service ends counts as `stopped`, not as an error. The image carries the pinned OpenSpec and a Node runtime, so the job validates each change before it pushes: `kubectl -n devops exec deploy/roadmap-service -- openspec --version` prints the locked version.

openspec/changes/archive/2026-11-05-3497-cut-unchanged-github-reads/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-05

openspec/changes/archive/2026-11-05-3497-cut-unchanged-github-reads/design.md

@@ -0,0 +1,36 @@
+# Design
+
+## Decisions
+
+### The status is a label of the request counter
+
+The counter keeps its name and its total. It gains a `status` label. The issue allows a status label or a separate 304 counter. A separate counter makes the total the sum of two metrics. The refine plan in #3497 rejects that option for this reason.
+
+### The request is counted after its answer
+
+`acquire` no longer counts a request. It runs before the request is sent, so it cannot know the status. `SessionGitHubAPI._request` counts each attempt after its answer, with the status `answer[0]`. A 304 therefore counts with status `304`.
+
+The session reads the token of a GitHub App after the attempt is paced. An attempt that raises before an answer counts with status `none`, a refused token included. The count is made in a `finally` block, and the error still propagates. A lost GET that the session sends again (#3506) counts each attempt, because the ledger paces each attempt. `run_gh` counts each paced attempt with status `none`, because gh does not report the HTTP status.
+
+### The labels are read in the thread that paces the request
+
+The request scope is a context variable. The session paces each request in a worker thread. So `pace_request` reads the request labels in that thread, and `_request` carries them to the count. The test `test_a_session_request_counts_the_scope_of_the_thread_that_sent_it` checks that the labels are those of the thread that sent the request.
+
+### A request that is not paced is not counted
+
+`gh_request_is_paced` exempts `gh api rate_limit`, and `run_gh` skips `acquire` for it. The count follows the same rule. An exempt request therefore counts nothing, as before.
+
+### The 304 charge stays
+
+The limiter charges each attempt, a 304 too. ADR 0006 records that every attempt is paced and charged. The issue allows a charge of 0 only when a measure shows that GitHub does not count the 304. The local ledger has no such measure. The charge therefore stays, and the metric keeps each 304 in its total.
+
+### The pace lines log at DEBUG
+
+Each pace line writes one record for a request that waits. The issue measured that these lines wrote 53.6% of a log sample at INFO. A per-request detail belongs at DEBUG.
+
+## Rejected alternatives
+
+- A separate 304 counter: it makes the total the sum of two metrics (see above).
+- Keep the count in `acquire` with status `none`: the status of the answer would be lost.
+- A charge of 0 for a 304: no local measure shows that GitHub does not count it, and the ledger would undercount.
+- Keep INFO: it keeps the log volume that the issue reports.