Repository navigation
Description
Current state: Proposed on 2026-11-10 by the issue author; no implementation is accepted. Draft PR #386 (head dbe0cf04) holds a candidate implementation stacked on #351, which it builds on. CI and the knowledge-policy check are expected to report there; no approval is requested while it is stacked.
Decision needed: Accept, revise or reject a new methods/ owner for model-level speed-of-light analysis, from the maintainers. This also bears on open question 1 of #303 (new methods/ owner or a second slice of performance-ceiling-analysis); see the boundary below.
Next action: The issue author keeps the Draft PR in Draft until three things happen: this scope is accepted, #351 lands, and the skill index is regenerated after #359. The Draft carries a provisional regeneration so that CI can run.
Goal and acceptance
-
Problem / desired outcome:
methods/performance-ceiling-analysiscomputes a conditional L1 floor for one device, one operation class and one traffic tier. Its roofline contract states that it does not walk a model, sum devices or compose stages. So an agent asked "what is the prefill / decode ceiling of model M under layout L on hardware H?" has no procedure for the model-level question. It assembles one by hand from per-op formulas, without:- ground-truth anchors on the counts,
- a rule for which graph to price (fused or unfused),
- per-rank geometry or injected communication,
- a separation between what is a bound and what is a conditional estimate.
HuggingArch (MIT) implements this question end to end. task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348 / docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 already carry one prerequisite from it, the checkpoint weight anchor. This issue distills the rest of the method into a portable candidate procedure.
-
Completion criteria:
- A new registered candidate skill
methods/model-sol-analysis(SKILL.md,metadata.json,references/), with oneskill-index.jsonentry and the count updates that follow. - The procedure walks a model to per-phase (prefill, decode) floors: anchored counts, the fused graph with provenance for every fusion, per-rank geometry under a declared layout, communication as explicit terms, and KV traffic per attention family with its growth law stated.
- Every figure is labelled as either an invariant bound (per-resource totals over rated peaks) or a conditional estimate (per-op serial composition, calibrated rates). Only a bound may reject a result, consistent with the bounds, predictions and decisions reference.
- A pure offline helper with tests. It takes saved inputs (a checkpoint header, an analytical tensor inventory, per-op counts, rated hardware facts) and returns join residuals, a byte checksum, and both the bound and the estimate per phase. It does no I/O, no network access and no device execution, and it returns
INSUFFICIENT_EVIDENCErather than guess. - A typed source identity at one pinned commit, with per-file blob, sha256, bytes and inspection scope. Claims in the reference documents carry
[code]/[doc]markers with locators, following the convention reviewed on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351. - One
docs/reference-matrix.mdrow and the matchingdocs/searchspace-matrix.mdrow(s); regenerateddocs/skill-coverage.mdand skill index. locally_verified: false,empirical_validation: not_run. No measurement, benchmark or performance claim.
- A new registered candidate skill
Scope and constraints
- In scope: the new
methods/model-sol-analysisowner, its helper and tests, the registry entry, matrix rows, count updates, and generated views. A one-line cross-link fromperformance-ceiling-analysisto the new owner. - Out of scope:
- The per-task SOL ruler, Class A/B vocabulary and gap metric proposed in [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303. This skill produces model-level floors and estimates; it does not define a frozen scoring ruler or rank agent flows. Choosing this owner for model-level analysis does not decide [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303's question for fused-operator ruler work.
- Importing or vendoring HuggingArch code, specs or its measured kernel corpus; the helper is written from the documented method.
- Any measurement, profiler run, device execution or numeric gate in the tree.
- Changes to the knowledge-model policy or to prototype: reduce knowledge projection and declaration churn #359's index representation.
- Relevant dependencies, shared interfaces or limits:
- Builds on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 (weight-demand anchor in
performance-ceiling-analysis): the new skill references it rather than duplicating it. The PR is stacked on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 and rebases once docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 lands. - prototype: reduce knowledge projection and declaration churn #359 changes the generated index representation. The Draft PR carries a provisional schema-v1 regeneration so that CI runs. The final regeneration happens after prototype: reduce knowledge projection and declaration churn #359 lands, which avoids another repository-wide pin-only rewrite on
main. weight-loading-optimizationkeeps loading mechanics;calibrate-serving-simulationkeeps serving-level (queueing, SLO) prediction.
- Builds on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 (weight-demand anchor in
Ownership and authorization
- Proposed by / actual requester: repository owner (via authorized agent)
- Status: Proposed
- Accountable parent (Issue-level owner, if assigned): TBD
- Slice implementer(s) / acting contributor (if assigned): issue author, if accepted
- Goal-acceptance owner: TBD
- Accepted decision or scoped delegation: Not yet granted
Inputs and deliverables
- Relevant sources, revisions or linked issues:
github.com/shenh10/HuggingArch@e69d20a028e5b17524aa48b53bddf8b9aad5d15b(MIT), observed 2026-11-10. Readset:docs/inference_cost_model.md,docs/evaluation_model.md,docs/symbolic_evaluation.md,docs/design.md,docs/parallelism.md,docs/kv_cache_storage.md,docs/attention_families.md,docs/moe_expert_axis.md,docs/quantization.md,docs/validation.md,docs/measured_calibration.md,docs/known_limitations.md, plus the implementing code files needed to verify load-bearing claims. The identity file records exact per-file scope.- docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 retains its own pin
facbbbc9…for the weight anchor; the two owners keep separate source revisions. - Related: task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348, docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351, [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303, trim & improve: reduce knowledge contribution cost and generated-file churn #355 / prototype: reduce knowledge projection and declaration churn #359.
- Expected artifacts: one PR as described under completion criteria.
Completion and closure
- Who accepts completion and may close: the named acceptance owner, not the PR merge.
- Remaining uncertainty / next decision: whether model-level analysis belongs in a new owner (proposed here) or as further slices of
performance-ceiling-analysis; whether the helper's bound/estimate split should be reused by [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 if that RFC is accepted.
Activity
amy-why-3459 commentedon Nov 10, 2026
Thanks for the careful write-up. Splitting model-level composition out from the single-op floor in
performance-ceiling-analysisreads as a sensible boundary to me. I also like the explicit split between bound and estimate and theINSUFFICIENT_EVIDENCEfallback. A few points I think are worth settling before the Draft PR, so that review stays focused:INSUFFICIENT_EVIDENCE? This decides whether "only a bound may reject a result" really holds.performance-ceiling-analysis. To avoid two copies of the per-op formulas, could the new owner explicitly consume the existing L1 floor and roofline contract, and describe only the walk, the composition and the communication terms? The proposed one-line cross-link covers discovery, but naming that dependency would make the boundary clearer for [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303's open question 1 as well.e69d20a0…, while docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 keepsfacbbbc9…for the weight anchor. Keeping them separate is reasonable, but could the identity file state either that the anchor semantics the new skill relies on are unchanged between the two revisions, or which revision wins when they differ?On process, the acceptance owner and the accepted decision are still TBD, and the Draft depends on both #351 landing and the post-#359 regeneration. Keeping the PR in Draft until a maintainer accepts the scope, as you proposed, sounds right to me.
Dong1017 commentedon Nov 17, 2026
Posted by an agent on behalf of @Dong1017 (incremental only; points already raised by others or decided by the owner are not repeated).
Incremental note: the stacking premise in the status block no longer holds. It says Draft #386 is "stacked on #351" and that no approval is requested while it is stacked. #351 was squash-merged on 2026-11-16 02:53 UTC as
f5362887. #386 is still atdbe0cf04(last updated 02:59 that day), and GitHub now reports it asCONFLICTINGagainstmain. It needs a rebase ontomain(dropping the #351 commits) before CI or knowledge-policy results mean anything. The maintainer decision this issue asks for (accept, revise or reject a newmethods/owner, and #303 open question 1) is still unanswered, and @amy-why-3459's four points have no author reply.Scope: PR state and mergeability via the GitHub API. I did not attempt the rebase.
zengchuang-hw commentedon Nov 17, 2026
Independent source-grounded review of the separable #385 proposal (2026-11-17).
Scope: reviewed the proposed model-level prefill/decode analysis contract, not contributor acceptance or a PR. Verified this is an open ordinary Issue by fake0fan in InferMatrixLab/InferRSI. Repository baseline: main
61303fa8150472dfbda62ec8d77908bc05c73f3b. I did not consult other reviewers' comments or the sibling issue's review.Conclusion: useful bounded direction, but implementation acceptance remains INCONCLUSIVE. The complete main tree has no
methods/model-sol-analysisowner. The existing performance-ceiling-analysis entry already specifies checkpoint join and per-rank weight gates, while explicitly leaving KV/activation/collective counts unanchored. Reuse those gates; a new model-walk owner should add phase/graph/traffic provenance rather than duplicate their definition. This is not a claim about any uninspected branch.Two concrete source-to-contract hazards deserve explicit acceptance fixtures:
Missing fabric evidence must not become zero communication time. At the proposed HuggingArch pin
e69d20a028e5b17524aa48b53bddf8b9aad5d15b, comm_cost.py:293-318 substitutes zero for the byte/rate term when bandwidth is absent/nonpositive. test_comm_cost.py:269-283 explicitly expects positive communication bytes with zero time when interconnect bandwidth is absent. That is a structural-only upstream convention, not evidence of free communication. The portable helper must return INSUFFICIENT_EVIDENCE for the dependent communication estimate when a used fabric lacks a valid rate/topology; it may preserve a separately labelled partial compute/memory bound. Test missing intra/inter rates independently, nonfinite/nonpositive rates, and distinguish world=1/no-transfer from unknown-rate transfers.A serial row sum is not automatically an invariant model bound. row_roofline.py:1394-1425 forms per-row maxima including communication/launch terms; StepSum.total:1501-1510 adds weighted row times. Retain this as a declared serial/schedule-dependent estimate unless the relevant non-overlap dependencies are proved. Keep resource-total/rated-peak bounds in a different output field with resource domain, dtype, count provenance and assumptions. A minimal synthetic counterexample is a compute-only row taking 10 units and a memory-only row taking 10: sum-of-row-maxima is 20, while resource-total max is 10 and overlap can attain 10 under the stated idealized assumptions. A result between those values must not be rejected merely for beating the serial estimate. Include fusion/repartition and changed-overlap fixtures so changing graph granularity cannot silently change which value is labelled invariant.
Provenance and delivery checks: independently retrieved the fixed upstream tree and LICENSE, which states MIT and requires retaining its notice for copies/substantial portions. Record exact inspected ranges and per-file blob/SHA256/bytes, distinguishing code from documentation. Reading these selected files is not verification of the full proposed readset, all attention-family growth laws, or all upstream model counts. Keep the existing anchor's separately pinned provenance identifiable rather than silently repinning it.
Validation actually performed: metadata and complete-tree inspection; repository AGENTS, knowledge contract and existing ceiling-skill inspection; pinned upstream license, selected cost-model documentation, communication implementation, row aggregation implementation and relevant test-source inspection. No upstream or InferRSI test suite executed; no offline-helper implementation tested; no hardware, profiler, model execution or empirical comparison. Empirical validation: not_run; locally_verified: false. Completion still needs the proposed helper/tests and registry/matrix/generated-index checks at an identified implementation revision. This review does not accept scope, commit to implementation, or authorize experiments.