Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ All code authored in or evaluated by DevOps CLI must strictly adhere to the foll

13. **Hosted Models**: The LiteLLM gateway serves four groups that call a provider outside the cluster: `gemini-3.8-flash`, `claude-sonnet-5-5`, `claude-haiku-5-5` and `deepseek-v4.1-flash-1017` (through OpenRouter, which picks a third-party provider and is asked to skip providers that store or train on prompts). A prompt sent to one of them leaves the cluster, and the provider bills each token. So no pool names a hosted group, no fallback sends a group that is not hosted to a hosted group, and a request reaches a provider only when its caller names a hosted group. The Service config names `deepseek-v4.1-flash-1017` for the planning lane and for the implement job (`k8s/devops/configmap.example.yaml`), so the code of the repositories of the Service, which can be private, leaves the cluster to OpenRouter in the prompts of a step; `claude-sonnet-5-5` serves no step. The one fallback, `claude-sonnet-5-5` to `deepseek-v4.1-flash-1017` (#3152), sends the prompt of a failed Claude call to OpenRouter, not to Anthropic. The keys are optional keys of the Secret `llm/llm-gateway-secrets` (keyring: `gemini.api_key`, `anthropic.api_key`, `openrouter.api_key`). The gateway NetworkPolicy admits TCP 443 to public addresses and to no private or metadata range. A Claude group sets its prices, drops the sampling parameters that its API refuses, and sets `supports_native_structured_output` and `supports_response_schema`, or a structured call becomes a forced tool call that Sonnet 5.5 refuses (#3140). `tests/ai/benchmark/test_k8s_llm_gateway.py` fails when any of this breaks.

14. **Roadmap Item Authors**: The repository is public, and a roadmap job gives the text of an item to a model. The model writes into the roadmap and, through Ready items, steers the agents that build.
- **The authors.** A job takes an item only when its author is a person with write access. For such a person, `author_association` is OWNER, MEMBER or COLLABORATOR (#2478). A job also takes an item of two bots. Their login is exactly `service.github_app.bot_login` or `github-actions[bot]` (#3502).
- **The login.** GitHub gives both bots the association CONTRIBUTOR. It gives the same association to any account with a merged commit. So the exact login is the only proof. The check compares the whole login, so a login that only resembles it is not trusted. With no `service.github_app.slug` set, no item of the App is trusted.
- **Other accounts.** A job leaves out any other bot and any outside account. Refine tells the author once with a comment.
- **The review.** An item of a bot is text that no person wrote. It reaches Ready only after the review of refine approves the issue and the proposal (`refine.judge`). The stage, implement and alternatives jobs take it only with that approval. The approval holds only for the body that the review read. A body edit ends it, and a refine round reviews the item again.
- **The controls.** The review is a model, and text in an item can mislead it, so it is no security control. The controls are the login, the filter for replies, and the person who merges each pull request. Refine and the alternatives job give their models only the replies of a person with write access and of the App's bot. A comment of a bot starts no `implement: retry`, because that check keeps the association test.
- **The risk that stays.** `github-actions[bot]` is the identity of every workflow of the repository. A workflow that copies untrusted text into an issue gives that text a path to Ready, behind the review. A record block in that text would forge the approval, because the job record is a hidden block of the issue body. The audit workflow writes a fixed text, and refine neutralizes `<!` in the text that it files for a Service item (#2478). So neither gives that path today.
- **The tests.** `tests/roadmap/test_roadmap_trust.py` and `tests/roadmap/test_roadmap_refine_trust.py` fail when any of this breaks.

---

## MCP Server on the Network
Expand Down
1 change: 1 addition & 0 deletions docs/ROADMAP_WORKFLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ These are all the keys of the job record, by the job that writes them:
| Refine | `refine.needs_split`, `refine.next_step`, `refine.blocked_by`, `refine.block_paths` | Whether the Item needs a split, who acts next, and the open items and paths that block it |
| Refine | `refine.capped_after`, `refine.alternatives_asked` | The comment cap of refine, and the covering tool that refine asked a person about |
| Refine | `refine.deferred` | The numbers of the issues that refine filed for the deferred parts of the Item |
| Refine | `refine.judge` | The verdict of the review of a Ready proposal for an Item that a bot opened, `approved` or `refused`. The stage, implement and alternatives jobs take such an Item only when it is `approved` and its body hashes as `refine.body_hash` and `refine.section_hash` record |
| Refine | `refine.returned_body_hash` | The body hash of a round that looked again at an Item after the stage job returned it; a second return on that body waits on a person |
| Alternatives | `alternatives.checked_at`, `alternatives.body_hash`, `alternatives.verdict`, `alternatives.tool`, `alternatives.failed_at`, `alternatives.failed_error` | When the check ran, the text it read, its verdict and the tool it found; when the last check failed and the class of its error, until a check is written |
| Stage | `stage.branch`, `stage.pr`, `stage.sha` | The staged branch, its draft pull request and its last commit |
Expand Down
4 changes: 3 additions & 1 deletion docs/roadmap/alternatives.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ A check changes no label and closes no issue. A person decides what a tool means

## Which items a run takes

A run takes open New and Ready items that a person with write access opened. It takes first the items with no check, lowest number first. Then it takes the items whose check is older than `roadmap.alternatives.recheck_days`, or whose text changed since. The setting `roadmap.alternatives.first_pass_limit` (20 by default) caps the run while an item has no check. The setting `roadmap.alternatives.limit` (5 by default) caps the run after, when every item has a check. `--limit` replaces both for one run. The row runs about twice a day, so a first pass of 141 items takes about 4 days.
A run takes open New and Ready items that a trusted author opened. A trusted author is a person with write access, the Service App's bot or `github-actions[bot]`. A run takes an item of a bot only with the approval of the review of refine.

A run takes first the items with no check, lowest number first. Then it takes the items whose check is older than `roadmap.alternatives.recheck_days`, or whose text changed since. The setting `roadmap.alternatives.first_pass_limit` (20 by default) caps the run while an item has no check. The setting `roadmap.alternatives.limit` (5 by default) caps the run after, when every item has a check. `--limit` replaces both for one run. The row runs about twice a day, so a first pass of 141 items takes about 4 days.

`--item` checks one open item whoever opened it and whatever its Status, even when its check is current.

Expand Down
13 changes: 12 additions & 1 deletion docs/roadmap/refine.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@ An item that waits on a person has one more way out. The comment of refine names

The stage job can return an item to New with evidence that the plan cannot hold. That is the act of a job, not of a person, so the next round refines the item again with no reply and no body edit. The model reads the evidence of the return in `stage_return`. A round that looks at the same body again records its hash in `refine.returned_body_hash`. When the stage job returns the item a second time on that body, refine does not look again. The item waits on a person, and the reason is `devops roadmap explain <number>` and a line in `devops roadmap inbox`. A reply or a body edit ends that wait. A person who sets an item back to New with no comment is the other case: the item waits for the reason of that person.

## Items that a bot opens

The Service App and the workflows of the repository open items. Examples are the parts that refine splits off and the findings of an audit. A round takes an item whose author login is exactly `service.github_app.bot_login` or `github-actions[bot]`. It takes such an item as it takes the item of a person with write access. It takes no item of another bot or of an outside account. Refine writes one comment on such an item to tell its author why the round leaves it alone.

Nobody wrote the text of a bot item, so a Ready proposal for it goes through a review first. The review is the verification pass of the stage job, with the model of the task slot `ai.tasks.verification`. It reads the issue and the proposal. With no open finding, refine sets the item Ready and records `refine.judge` `approved`.

With an open finding, the item stays New and waits for a body edit. One comment lists each finding with its evidence. The approval covers the body that the review read, and the hashes in the job record show it. A body edit ends the approval, and the next round reviews the item again. The stage job, the implement job and the alternatives job take a bot item only with an approval that holds.

The review is a model, so text in the item can mislead it. The control is the login of the author. The models of refine and of the alternatives job read only the replies of two kinds of author. These are a person with write access and the App's bot. A person still merges each pull request. A comment of a bot starts no `implement: retry`.

## What refine investigates

When a fact or a decision has no verified source, or a key question of the body has no answer, refine investigates it. It uses tools that read the code. For a public repository with a Firecrawl key, it also searches the web. The plan says when web search is off.
Expand All @@ -42,7 +52,7 @@ In each open milestone, and among the P0 and P1 items of the backlog, refine als

Two kinds of item wait for a person. Only `--item` refines them:

- An item that an account other than the owner, a member or a collaborator opened.
- An item that another account opened. The accounts that a round trusts are the owner, a member, a collaborator, the Service App's bot and `github-actions[bot]`.
- An item on which refine wrote 2 comments in the last 24 hours. It takes no place under `--limit` until the older comment is 24 hours old.

`--item` refines an open New or Ready item even when nothing changed since its last round.
Expand All @@ -66,6 +76,7 @@ The command [`devops roadmap repair`](repair.md) sets back the items that a plac
| Topic | Spec |
| :--- | :--- |
| The limit, the cap of comments, the items of an account without write access | [roadmap-refine](../../openspec/specs/roadmap-refine/spec.md) |
| The authors that the jobs trust, the review of a bot item, the replies that the model reads | [roadmap-author-trust](../../openspec/specs/roadmap-author-trust/spec.md) |
| The investigation, the options of a decision, web search | [roadmap-refine](../../openspec/specs/roadmap-refine/spec.md) |
| The next-step comment, the paths of a block, a change that ends a wait | [roadmap-refine](../../openspec/specs/roadmap-refine/spec.md) |
| A deferred part and its filing through intake | [roadmap-refine](../../openspec/specs/roadmap-refine/spec.md) |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-11-05
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Design

## Context

The jobs read an item through `Item`, and decided its trust with `author_association` and `CONST_ROADMAP_TRUSTED_AUTHORS` in four places. The Service App and `github-actions[bot]` open items, and GitHub gives them CONTRIBUTOR. See proposal.md for the numbers and the owner decision.

## Goals / Non-Goals

**Goals:**

- The four readers trust one rule, in one module.
- An item of a bot reaches Ready only after a review, and the other jobs take it only with a recorded approval.
- Text from an outside account does not reach the model of refine, or the model of the alternatives job, as a reply.
- An approval holds only for the body that the review read.
- The author of an item that the jobs do not trust gets one comment.

**Non-Goals:**

- The retry right of `implement: retry`. It keeps the association test, so a bot gains no retry right.
- The `trusted` test of intake. It also reads the `source/agent` label.
- A change to the prompt of the review or to its golden set.

## Decisions

- **One module, `trust.py`.** `is_trusted`, `is_bot_filed`, `is_approved` and `trusted_replies` are the only code that compares an author. The readers differ only in the reason text that they give.
- Rejected: a field `trusted` on `Item`. The rule reads the settings, and the store has no settings.
- **The login is the proof.** The jobs compare the author login with `bot_login()` by equality, and with `CONST_ROADMAP_ACTIONS_BOT_LOGIN`. GitHub gives both bots the association CONTRIBUTOR, as the evidence in proposal.md shows.
- Rejected: trust any author of type Bot. It trusts bots outside the project's control.
- Rejected: trust the association CONTRIBUTOR. It also holds each person whose commit was merged, with no write access.
- Rejected: a prefix or a case-folded match. The `lookalike` tests show the exact match is the only safe one.
- **With no App slug, `bot_login` is None and no item of the App is trusted.** `github-actions[bot]` stays trusted, as the owner decided.
- **Item, IssueRecord and IssueComment hold `author_login`.** The listings that a round reads already carry `user`, so the check costs no request.
- Rejected: a read of the author in each reader. It adds a request for each item.
- **The review reuses `stage_verify.verify`.** The owner decided on the verification pass and the slot `ai.tasks.verification`, with no new model and no new tool. The proposal goes in as the `plan`, and `open_findings` decides the refusal: a type that keeps a plan in draft, with a quote that is in the input.
- Rejected: a new prompt for the proposal. It needs its own golden set and its own scoring, and the owner asked for no new tool.
- Rejected: a refusal on any finding. A finding with no quote that the input holds is advice in the stage job, and it stays advice here.
- **A refusal is `NextStep.EDIT`.** The step already means "waits on a body edit alone", so the item is not refined again, and the review is not called again, until the body changes. The comment uses `step_comment` with a new `edit` text, which lists each finding with its evidence. The refusal comment carries refine's marker, so the cap of two comments in 24 hours counts it.
- Rejected: `NextStep.PERSON`. A reply would end the wait, and a bot reply would loop the review.
- **The approval is the record key `refine.judge`, and it holds only for the body that the review read.** The verdict is written in the same call that records `refine.body_hash` and `refine.section_hash`. A round that does not review the item clears it. `is_approved` is true only when the verdict is `approved` and the body hashes as those two keys record. A round that looks again at an item with a valid approval keeps it with no model call.
- Rejected: a label. A job record is where the other jobs already read what refine decided.
- Rejected: the verdict alone, with the next round as the check of the body. The owner asked on #3502 for the check of the edit of a body after the approval. A probe with the in-memory store showed that the stage job took an item on the old approval after a body edit. So the approval is bound to the hashes.
- The hashes move to `body_hash.py`, so `trust.py` can read them with no import of `refine.py`, which imports `trust.py`.
- A write of the hashes with no review also clears the verdict. These writes are the record of an item that is too big to refine, and of an item whose model call failed twice on its own text. Without the clear, the hashes would belong to a body that no review read.
- The title is no part of the hashes. Refine itself ignores a retitle, and the review does not read a part of the item that a round looks at again.
- **A round looks again at a Ready item of a bot with no valid approval.** Without it, a body edit would leave the item out of the stage job with no way back. `_refinable` takes such an item. The review runs. The item then stays Ready with a new approval, or goes back to New with the step `edit`, as any Ready item that refine finds not ready.
- **The review runs before the intake plan of the deferred parts.** A refusal then costs no intake call.
- **The comments that reach the models are those of trusted authors**: a person with write access, or the App's bot. `github-actions[bot]` is an author of items, not of replies. `_Selection.comments_on` and `refine_one_item` filter at the one read, so the replies, the `covers` answers, the stage return and the wait checks all see the same comments. The alternatives job filters the comments that it gives to its model with the same function. It filters before it looks for its own marker, so an outside comment that holds the marker moves no window. The in-memory store gives the owner's association to a comment that it writes or seeds with no association, as GitHub gives an association to each comment.
- Rejected: keep every comment and trust a split-off only when every reply came from a trusted author. Outside text would still reach the model.
- **The comment to the author is written by apply, and found by the selection.** The selection reads the comments of an item that the jobs do not trust, and sets `notice` when no comment holds the marker. A round that finds `notice` writes the comment, and the plan has writes. Only refine writes it: stage, implement and alternatives are read-only selections whose left-out reasons show in their reports.
- Rejected: a mark in the job record. An item of an outside account has no record, and a body write to its issue is more invasive than a comment.
- Rejected: a count in process memory. A new round or another process writes a second comment.
- **`refine.py` imports `refine_judge` in the function.** `stage_verify` imports `stage_change`, which imports `refine`.

## Risks / Trade-offs

- [The review sees no file, so it finds fewer defects than the stage review] → the review is no security control. The login and the reply filter are the controls, and a person still merges each pull request.
- [An item of `github-actions[bot]` can carry text from a workflow input] → the owner accepted this in the decision on #3502. The filter keeps outside replies away from the model, and SECURITY.md names the risk.
- [A bot-filed item approved by the review, then edited by a person] → the approval ends at once. The next round reviews the new body. Until then no job takes the item.
- [The approval is a block in the issue body, so text that a bot copies into the body could forge it] → the audit workflow writes a fixed text, and refine neutralizes `<!` in the text that it files for the Service (#2478). SECURITY.md names the risk for a new workflow.
- [A round reads the comments of each item that the jobs do not trust] → the set is small once the bots are trusted. The dry-run plan lists that read and the comment that follows.
- [A Ready item of a bot that was refined before this change has no verdict] → one round reviews it. An approval then lets the stage job take it, and a refusal sets it back to New.
Loading
Loading