Skip to content

task: prefer user acceptance criteria and derive missing targets with ceiling analysis #399

Closed

Description

@amy-why-3459

Goal and authorized scope

The requester authorized implementation and PR submission for this rule: use user-provided acceptance criteria when present; otherwise use the repository's performance-ceiling-analysis skill to formulate acceptance criteria.

Deliver the agent workflow and campaign-template slice: connect the existing ceiling skill to task preparation, preserve explicit requirements, and record the origin and evidence of derived targets before candidate search. This is a bounded follow-up to #143. It does not implement the proposed task-file runtime in #365 or change existing campaign gates.

Acceptance criteria

  • User-supplied criteria, including those in a referenced accepted plan, retain their metrics, thresholds, scope and constraints. Ceiling estimates may explain feasibility but never silently replace or relax those criteria.
  • With no supplied criteria, the agent uses performance-ceiling-analysis to establish workload-specific targets from sourced bounds, a comparable baseline, calibrated attainable performance and uncertainty. Partial criteria are retained; only missing fields are derived and labelled.
  • Missing evidence produces explicit gaps and a bounded calibration plan, not a fabricated numeric gate, a universal percentage of peak or a campaign PASS. Quality and semantic constraints are not inferred from hardware peaks.
  • Record criteria provenance, derivation and the existing freeze authority in the acceptance issue and campaign plan. Freeze the targets, workload and validators before candidate search; historical contracts remain unchanged.
  • Route repository and distributed agent entrypoints to the same canonical procedure. Preserve the existing CLI/RunSpec/evaluator behavior; numeric goals remain campaign-specific.
  • Validate template parsing, packaged skill retrieval, generated catalog consistency and affected plugin workflows with the existing checks. No accelerator campaign or measured model-efficacy claim is part of this PR.

Ownership and completion

Implementation: the requester's Codex session. Acceptance and merge: requester/repository maintainers through the existing review process. PR submission is authorized; Issue closure and merge are not requested. Refs #143 for the broader goal-preparation proposal.

Activity

  1. amy-why-3459 commented on Nov 12, 2026

    @amy-why-3459
    CollaboratorAuthor
  2. cursor commented on Nov 12, 2026

    @cursor

    #399 authorizes implementation and a PR for this slice. Closure and merge stay with maintainers and are not requested here.

    What the body commits to:

    • User-supplied criteria, including those in a referenced accepted plan, keep their metrics, thresholds, scope, and constraints. A ceiling estimate may sit beside them as a feasibility note.
    • With no supplied criteria, performance-ceiling-analysis sets workload-specific targets from sourced bounds, a comparable baseline, calibrated attainable performance, and uncertainty. Partial criteria stay; only missing fields are derived and labelled.
    • Missing evidence is recorded as an explicit gap plus a bounded calibration plan. Quality and semantic constraints stay separate from hardware peaks. Numeric goals stay campaign-specific.
    • The acceptance issue and campaign plan record provenance, derivation, and the existing freeze authority. Targets, workload, and validators freeze before candidate search. Historical contracts stay unchanged.
    • Repository and distributed agent entrypoints share one canonical procedure. CLI, RunSpec, and evaluator behavior stay as they are.
    • Validation uses the existing checks: template parsing, packaged skill retrieval, generated catalog consistency, and affected plugin workflows.

    Outside this slice: the #365 task-file runtime, existing campaign gates, and any accelerator campaign or measured model-efficacy claim.

    Not verified from this thread: the text of #143, the ceiling skill on disk, and any PR diff. A review of the change waits on a PR head.

    Dong-bot.

  3. added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    on Nov 12, 2026
  4. zengchuang-hw commented on Nov 16, 2026

    @zengchuang-hw
    Contributor

    Following up on the author ping in #399 (comment) with an independent scope/design check.

    The separation in this task is appropriate: user criteria are binding inputs; ceiling analysis fills missing performance-target fields; neither route silently grants quality relaxation or a new runtime contract.

    I independently read current main's AGENTS.md, performance-ceiling-analysis skill, and its canonical acceptance procedure. The procedure now explicitly preserves full/partial user criteria, keeps missing quality requirements unresolved, requires comparable calibration and uncertainty, and forbids using later candidate results to invent a gate. The inspected file blobs were respectively 1e4aca3716481fbd5b5feec8a6a932acde8010ca, 29b5fa549aa43ddeca165b0068d693689da807bb, and 9a29ebeda4cdb360eab46dcd83b3745e0e2dc937 (file identities, not a PR head or test result).

    For acceptance of the implementation slice, I would make the negative-case matrix explicit rather than rely on the prose alone:

    1. Complete user criteria survive byte-for-byte in meaning, including units, timing boundary, statistics, workload and quality constraints.
    2. A partial contract retains each supplied field and labels only the missing derived fields; conflicting supplied/frozen criteria do not choose the easier target.
    3. Missing baseline/calibration/quality or unresolved primary objective blocks candidate search, without fabricating a default percentage or inferring quality tolerance from hardware peaks.
    4. Absolute targets, relative targets and multi-workload requirements are each connected to the appropriate actual validator. The canonical procedure explicitly says the generic evaluator does not automatically enforce absolute or multi-workload contracts; recording a target is not enforcement.
    5. Repository and packaged/distributed entrypoints resolve the same procedure and provenance; an existing frozen campaign is not retroactively changed.

    The numerical example is worth a regression case: a justified 1.25× latency speedup means a 20% latency reduction, not 25%; throughput must not be obtained by blindly inverting service latency.

    Conclusion: the current documentation supports the intended design. This is not completion acceptance for the implementation or a campaign PASS. I have not run template parsing, packaged retrieval, catalog/plugin workflow tests, CLI/evaluator tests or an accelerator campaign in this check; those evidence items remain necessary to demonstrate the issue's implementation acceptance criteria. No code changes, Issue closure or merge are authorized by this comment.

  5. Dong1017 commented on Nov 17, 2026

    @Dong1017
    Collaborator

    Posted by an agent on behalf of @Dong1017 (incremental only; points already raised by others or decided by the owner are not repeated).

    Incremental note: the slice this issue authorizes has already landed as #400 (merged 2026-11-13, fe16b293; approved by @zengchuang-hw and @david6666666), but the thread never links it. So @zengchuang-hw's negative-case matrix from 11-16 is now a post-merge acceptance question, not a pre-merge one.

    1. Use user acceptance criteria, with ceiling-analysis fallback #400 is text-only. It changes skills, templates, docs and regenerated records (25 files), with no test file. tests/test_performance_ceiling.py on main has no case for the procedure in references/acceptance-criteria.md. As a result, none of the five matrix rows, and not the 1.25× → 20% example, is checked mechanically today. The example is stated as a formula at acceptance-criteria.md:49-51.
    2. Decision needed from the requester (@amy-why-3459): either accept Use user acceptance criteria, with ceiling-analysis fallback #400 as completing this slice, with the "existing checks" in the acceptance list, or record the matrix as a follow-up with its own owner. The issue has no status: label or named acceptance owner, so the closure path is not visible from the thread.

    Scope: #400 file list and reviews, main 1acfacf8 acceptance-criteria.md and tests/. I did not run the template, catalog or plugin checks.

  6. zengchuang-hw commented on Nov 17, 2026

    @zengchuang-hw
    Contributor

    Incremental evidence review of this closed issue, under the maintainer's standing review authorization. This does not reopen the task or change user acceptance criteria.

    Baseline: my previous scope/design review. Current inspection: main 61303fa8150472dfbda62ec8d77908bc05c73f3b. I independently checked current executable tests, resource inventory and integration paths before comparing that prior comment.

    What changed in this review is verification, not the reviewed procedure. The Git blob IDs of AGENTS, the performance-ceiling-analysis skill and its canonical acceptance reference still exactly match the three IDs recorded in the prior review (1e4aca37…, 29b5fa54…, 9a29ebed…). I found no new revision-specific defect in those unchanged files in this pass.

    The prior review explicitly left template/retrieval/plugin checks unrun. This pass ran:

    PYTHONPATH=src python -m pytest -q -o addopts='' \
      tests/test_performance_ceiling.py \
      tests/test_campaign_plan_template.py \
      tests/test_workswarm_plugin.py \
      tests/test_engineering_benchmark_plugins.py \
      tests/test_dsh_plugin.py \
      tests/test_codex_plugin.py
    

    151 passed, with two dependency/test-harness deprecation warnings. This covers existing ceiling helper numerical/insufficient-evidence cases, template parsing/placeholder/freeze structure, and the selected plugin integration checks. These results fill that specific previously unrun CPU evidence gap; they do not demonstrate Agent behavior or a campaign outcome.

    I also checked the generated performance-ceiling-analysis record: its inventory includes references/acceptance-criteria.md and the recorded SHA-256 matches the actual resource bytes. A direct package-provider read followed by revision-bound resource open returned the exact 5,917-byte canonical reference (bundle revision sha256:8f6803bdef38276503636e4a310de7603b686b6319b6473dbb549bffe0072adc). This was a source-tree package-provider check, not a new wheel build/install. An initial lookup with an unqualified skill-style package ID was rejected; the confirmed package identity is package:inferrsi.common.skills.performance_ceiling_analysis.

    Remaining boundaries: these tests do not establish the whole prior semantic negative-case matrix, prove that an Agent preserves every supplied field, or add enforcement for absolute/multi-workload targets to an evaluator. No clean installed-wheel round trip, general CLI/RunSpec/evaluator regression suite, accelerator campaign or model-efficacy experiment was run in this pass. Original requirement-version/freeze authority was not reconstructed from discussion. Supplied criteria remain binding, only genuinely missing fields may be derived, and historical frozen campaigns are unchanged.

    Conclusion: new CPU verification supports the existing workflow/template slice without expanding it into task-file runtime implementation or asserting a campaign PASS. No code, issue state, permissions or user goals were changed; no merge or device/paid task was initiated.

  7. zengchuang-hw commented on Nov 17, 2026

    @zengchuang-hw
    Contributor

    Independent historical-request follow-through for comment:6938327962@2026-11-17T04:27:26Z. Issue399 remains closed. Earlier own comments 6914352857 and 6943064049 are relevant background, but matching subject or later timing does not establish explicit fulfillment of this invitation. This is a fresh bounded verification, not adoption of source-comment opinions or claimed decisions.

    Baseline: main 0311393c62330c041a01238dc86823b80e020561. The scoped AGENTS/ceiling-skill/reference and selected test files are unchanged relative to the earlier 61303fa8150472dfbda62ec8d77908bc05c73f3b baseline.

    Actual independent checks this pass:

    • Re-inspected the pure roofline helper/test contract and campaign-template checks. The helper exposes conditional bounds, evidence basis, inputs, limits and unknowns, not a campaign verdict or speedup claim. Synthetic incompatible/missing evidence tests require INSUFFICIENT_EVIDENCE and absent derived bounds; empirical observations are not silently treated as rated ceilings.
    • Used the package KnowledgeReader to read package:inferrsi.common.skills.performance_ceiling_analysis. Its acceptance-criteria resource was 5,917 bytes, exactly equal to the checkout resource, SHA256 e96efc42005af591a87b57dd6f1e9bffa8e351bcd023bd093ac37239a0486638; bundle revision sha256:8f6803bdef38276503636e4a310de7603b686b6319b6473dbb549bffe0072adc.
    • Freshly ran performance-ceiling, campaign-plan-template, workswarm-plugin, engineering-benchmark-plugins, dsh-plugin and codex-plugin suites: 151 passed, 2 warnings, in 92.02s. This is the current run, not a replayed historical count.

    Acceptance boundary: these tests and retrieval establish bounded calculations, template structure and distributed workflow/resource availability. They do not prove that an agent preserved every real user's criterion in an actual campaign, that missing criteria were correctly derived for a real workload, or that an attainable target was calibrated on hardware. The intended workflow remains: preserve supplied metrics/thresholds/scope/constraints; derive only genuinely missing fields from sourced comparable evidence; retain uncertainty and freeze the accepted plan before search. The helper is not authority to replace a supplied gate or turn insufficient evidence into PASS.

    Not performed: a real goal-to-frozen-plan acceptance exercise, hardware calibration, full repository suite, separately installed clean-wheel verification, campaign execution or exhaustive recovery of the original historical specification/acceptance authority. The package-reader check above used the current source environment and is not being described as a clean-wheel install. No365 task-runtime acceptance, gate change or parent-goal completion is inferred.

    No code, evidence, validators, tracker or Issue state was changed. No reopening, merge, paid/model/device experiment or cancelled-work restart occurred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions