Skip to content

feat(review): deliver managed labeling runtime and one-item runner #5406

Open

Description

@purp

User Story

As an OpenShell maintainer labeling accepted issues and PRs, I need one managed runner that previews and applies classifications through /label-issue so I can validate the workflow before enabling automatic dispatch.

Description

Deliver the managed classifier, invocation selector, and one-item runner in .agents/agents/labeler/. Accept an issue/PR number or URL; prepare evidence, expand it as needed, classify against the trusted skill and current result contract, and preview the proposed changes. Explicit application uses the saved exact preview. Human force bypasses only unchanged detection.

Keep eligibility, evidence handling, semantic validation, label mutations, checkpoints, and reporting in the shared helper. Ineligible, excluded, held, or unchanged items stop before model invocation. Lookup failures remain errors; partial evidence remains partial. Recheck current evidence and eligibility before writes, preserve label-event checkpoint verification, and reconcile uncertain writes without blind replay. Ambiguity stays in local reports; the labeler posts no comments.

The classifier receives prepared evidence without tools, MCP servers, or supplied GitHub credentials. Pin its configuration, bound cost and execution time, preserve session references, and recover failed sessions. Consolidate request preview and authorized application under the runner's entry point.

Select explicit issue/PR acceptance events and authorized human PR-comment/manual invocations, including drafts and force, without independently deciding acceptance. Install no automatic dispatch or backlog execution in this issue.

Suggested UX

Invoke /label-issue <number-or-URL> for a preview, inspect the exact changes, then authorize one-item application. The same runner serves later automated callers.

Context

Definition of Done

  • One entry point supports issue/PR numbers or URLs, preview, explicitly authorized saved-preview application, human force, and model-request preview.
  • Deterministic skips make no model call; evidence expansion, classification, and shared validation/application work end to end without duplicate helper or result-schema logic.
  • The pinned classifier enforces tool/credential isolation, budget/deadline bounds, completion checks, and failed-session recovery.
  • Invocation selection covers acceptance events, bot-applied acceptance, drafts, and authorized human commands without activating automatic dispatch.
  • Failures retain execution references and distinguish stale inputs, missing prerequisites, invalid output, and uncertain writes; no false successful checkpoint is recorded.
  • CI-registered offline tests cover orchestration, authorization, pagination, completion, evidence coverage, and failure/recovery boundaries, including the linked experiment's failure cases.
  • Observed live runs verify a dedicated CMA configuration, trusted labeling identity, scoped credentials, and budget. Cover legacy migration, missing evidence, unchanged skips, force, and partial failures, starting with pinned Sonnet. Measure CMA cost/latency rather than importing CLI estimates; never commit credentials.
  • Document setup, invocation, the concise human report, and recovery.

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.
    topic:dxMaking OpenShell easy to contribute to for humans and agents.
    on Nov 9, 2026
  2. changed the title [-]feat(review): add managed labeling runtime foundation[/-] [+]feat(review): deliver managed labeling runtime and one-item runner[/+] on Nov 19, 2026
  3. purp commented on Nov 19, 2026

    @purp
    CollaboratorAuthor

    Combined #5411 into this issue after its runner implementation was merged into the branch for #5407. This issue now covers the managed classifier, invocation selector, and end-to-end one-item runner, including their shared-helper boundaries and live validation. The description was tightened with the omit-needless-words skill; #5411's migration-experiment guidance remains linked.

    #5411 is superseded by this issue. Automatic dispatch and coordination across entry points remain tracked in #5276.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.state:acceptedA maintainer decided OpenShell should pursue this issuetopic:dxMaking OpenShell easy to contribute to for humans and agents.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions