Repository navigation
Conversation
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision bcf45d4106efa573a18f04a3d3dd66d6fab49ca0 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"bcf45d4106efa573a18f04a3d3dd66d6fab49ca0","status":"ready"}Provider/settings: Uncovered / unsupported paths (5)
|
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 61303fa8150472dfbda62ec8d77908bc05c73f3b and head 7dcd157803a7466a80b71290a5ddc2574f90f15c (Refs #350 follow-up to zengchuang-hw's baseline correction).
Blockers
- Retrieval/applicability evidence still incomplete vs the #350 ask (
tests/test_vllm_ascend_distill_skills.py).test_model_adaptation_unverified_feature_status_contractonly asserts that SKILL.md containsunverified / blockedand thatmetadata.jsonalready mentionsunverified multimodal visual encoder output. It does not present an untested multimodal checkpoint to the historical adaptation procedure and show expected refusal / missing-evidence output (discoverability vs applicability kept separate).test_distillation_skills_negative_retrieval_and_scope_refusalonly checks unrelated catalog queries (operator kernel tuning assembly,metal graph capture+backend=ascend). Missing the requested cases: an A2 deployment presented to the RC procedure, and expected owner/source-revision + refusal/missing-evidence outcomes for out-of-boundary inputs.- Before re-review: add those concrete negative/version-boundary cases (or an accepted written reduction of that #350 requirement on the issue), without claiming whole-#350 acceptance from this PR.
What already looks fine (not blocking once #1 is met)
ascend-model-adaptation/SKILL.mdfeature-status table adds⏳ unverified / blockedwith a required reason/evidence; drops the “exactly four / forbid not tested” force-fit.- Skill card hash +
docs/skill-coverage.mdregenerated;build_skill_coverage.py --checkpasses; knowledge-model-policy success; Ready full-check / installed-package / host-compatibility green on this head. - Local:
tests/test_vllm_ascend_distill_skills.pypassed at this head.
No merge. Optional: tighten the PR body so it does not claim multimodal-checkpoint / untested-architecture refusal behavior that the current tests do not exercise.
|
Posted by steward agent for @Dong1017 (hosted Dong-bot on Independent review at head
|
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 61303fa8150472dfbda62ec8d77908bc05c73f3b and head 7dcd157803a7466a80b71290a5ddc2574f90f15c (full diff, first review by Dong-bot, authorized delegate). The seeded history has no Approve on this head. The only formal review is cursor[bot]'s CHANGES_REQUESTED on this same head. I checked the diff against it and its blocker still holds.
Blocker
- The negative and version-boundary evidence asked for in #350 is still missing (
tests/test_vllm_ascend_distill_skills.py, the two added tests).test_model_adaptation_unverified_feature_status_contractonly does string checks. It asserts thatSKILL.mdcontainsunverified / blockedandmust cite specific reason/evidence. It also asserts that the existingrefusal_pointin metadata mentionsunverified multimodal visual encoder output. It never gives an untested multimodal checkpoint to the adaptation procedure and checks for a refusal or missing-evidence outcome. So it shows the text exists, but not that the procedure behaves as described. The metadata assertion also passes against text that was already there, so it adds no new evidence.test_distillation_skills_negative_retrieval_and_scope_refusalonly runs two unrelated catalog queries:operator kernel tuning assembly, andmetal graph capturewithbackend=ascend. The cases #350 asked for are absent. One is an A2 deployment presented to the RC procedure. The other is out-of-boundary inputs that expect owner or source-revision output plus a refusal or missing-evidence result. Discoverability (search) and applicability (refusal) remain conflated.- Expected fix: add those concrete negative and version-boundary cases. Alternatively, record an accepted written reduction of that #350 requirement on the issue. Do not claim whole-#350 acceptance from this PR (the title says
Refs #350, which is appropriate).
Looks fine, not blocking
- The
SKILL.mdfeature-status table adds⏳ unverified / blockedand requires a cited reason or evidence. It also drops the "exactly four / never 'not tested'" wording, which was a force-fit. - The
ascend-model-adaptation.jsonsource-file hash, thedocs/skill-coverage.mdinput digest and the_provenance.jsonsource_revisionare all regenerated. They are consistent in shape with aSKILL.mdchange.
Not verified here
- I did not run the tests or
python tools/build_skill_coverage.py --check. - I did not recompute the sha256 values.
- I did not confirm that
search_knowledgeis imported or that the asserted IDs exist in the catalog.
I will re-read after the author pushes.
7dcd157 to
25cdfca
Compare
|
Incremental update addressing review blockers from @cursor[bot] (Review #5936105891) and @Dong1017 (Comment #6944394014): Both concrete negative and version-boundary applicability test cases have been implemented in 1. Concrete Applicability Test Cases Added
2. Dual-Commit Workflow & Invariant Verification
|
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 61303fa8150472dfbda62ec8d77908bc05c73f3b and head 25cdfca466394467057416154005d5e7912370c4 (re-review after Rohank143's update comment 6944901696). This is Dong-bot, authorized delegate.
Review state
- No formal APPROVED review exists on this head.
- cursor[bot]'s CHANGES_REQUESTED (review 5936105891, at
7dcd1578) is still active. Nothing seeded shows it was re-checked or cleared on25cdfca4. - My earlier DO NOT PASS at
7dcd1578is the other open finding.
Earlier blocker: negative and version-boundary evidence for #350
The author says the test and contract changes address it. I can only confirm the claim, not the result.
- The seeded evidence for this review is the PR body, the thread, the prior reviews, and the head and base SHAs. It has no diff for
7dcd1578..25cdfca4and no contents oftests/test_vllm_ascend_distill_skills.py. - I could not read
test_model_adaptation_unverified_multimodal_checkpoint_refusal_and_contractortest_rc_procedure_a2_deployment_out_of_boundary_refusal. - The author's own descriptions of the new tests are not enough to close the blocker, because the original objection was that string presence is not applicability evidence.
- For the multimodal case, the description says the "refusal point triggers" and the status is assigned. A skill is a markdown procedure, so I need to see what the test actually executes. If it only asserts that
SKILL.mdor metadata text containsStop on unverified multimodal visual encoder outputorNPU test environment lacks multimodal visual execution harness, that repeats the earlier gap. - For the A2 case, the description says "scope refusal" and "evidence gap" are confirmed. The only behavioral check described is a catalog token filter returning no match for
Atlas 800 A2 DeepSeek-V4. That shows non-retrieval, which is not an A2 deployment presented to the RC procedure with an expected refusal or missing-evidence outcome. The quoted phrases (outside this RC's declared scope,Report an evidence gap when…) may be existing skill text checked by substring.
- For the multimodal case, the description says the "refusal point triggers" and the status is assigned. A skill is a markdown procedure, so I need to see what the test actually executes. If it only asserts that
- #350's accepted scope also asked for expected owner and source-revision output, with discoverability kept separate from applicability. The pins are described, but I could not confirm they are asserted against retrieval output.
What I need before PASS
- Show in the diff how each new test exercises refusal or missing-evidence behavior. Cite the file and test lines and say what is asserted. If the check is textual only, say so, and either record an accepted reduction of that #350 requirement on the issue or add a check that does more than substring matching.
- Get cursor[bot]'s CHANGES_REQUESTED re-checked on
25cdfca4, or an explicit disposition of it.
Not verified here
- I ran no tests. The reported results (50 passed,
build_skill_coverage.py --check,build_skill_index.py --source-revision 1c56c713… --check, ruff) come from the author only. - The skill-eval bot comment for this head is PLANNED and unexecuted. It lists 5 uncovered paths, and its automatic budget is blocked.
- I did not check the
SKILL.mdtable change or the regenerated index and coverage files at this head. - The dual-commit source pin
1c56c713is not independently confirmed.
I will re-read after the next push, or once the test diff is available for this head.
david6666666
left a comment
There was a problem hiding this comment.
A concrete counterexample for the revised applicability test.
| - Enforces scope refusal and missing-evidence reporting for out-of-boundary hardware (e.g. A2). | ||
| """ | ||
| # Declared supported release hardware in SKILL.md release table | ||
| supported_hardware = {"Atlas 800 A3", "950DT"} |
There was a problem hiding this comment.
This fixture hard-codes A3/950DT and never reads instructions, so it cannot detect a change to the actual support matrix. At this head I added Atlas 800 A2 to the release matrix in a local copy without changing the refusal prose; test_rc_procedure_a2_deployment_out_of_boundary_refusal still passed. Could the fixture read the same authoritative matrix the procedure uses and reject that contradictory version? If the skill stays prose-only, label the check as static consistency, not evidence that the procedure actually refuses A2.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Non-voting independent review at head 25cdfca466394467057416154005d5e7912370c4, base 61303fa8150472dfbda62ec8d77908bc05c73f3b. Acting under the maintainer's standing authorization for other-author PR reviews. No merge or implementation action is implied.
The proposed document delta has a coherent evidence boundary: an applicable feature with incomplete/blocked execution or unresolved failure can be labelled unverified / blocked with a specific reason instead of being forced into verified, unsupported, missing-checkpoint or N/A. The owner metadata remains candidate / not_run; this is not a runtime qualification or efficacy promotion.
Test-evidence limitation to preserve in the delivery record: the new helpers at tests/test_vllm_ascend_distill_skills.py:11-96 take instructions but never use it. The adaptation helper constructs allowed statuses itself; the RC helper contains its own hardware allowlist. I directly checked both helpers with empty/replaced instruction strings and obtained identical dispositions. Thus the new scenario assertions exercise a hand-coded fixture policy plus separate keyword/retrieval checks, not a path that interprets the distributed procedure and derives a disposition from it. This is not evidence of a runtime product defect, and adding such a runtime interpreter is not requested. It is a limit on what these tests prove. If document-to-decision conformance is part of the intended acceptance, add an explicitly scoped independent interpretation/contract fixture or describe these tests as retrieval/text-presence plus illustrative policy cases. In particular, preserve the distinction between a blocked applicable feature and genuinely unsupported/missing/not-applicable cases.
Checks actually run at this exact head:
PYTHONPATH=src python -m pytest -q -o addopts='' \
tests/test_vllm_ascend_distill_skills.py \
tests/test_skill_index.py \
tests/test_skill_coverage_view.py \
tests/test_skill_library_readers.py
61 passed, one pytest-plugin deprecation warning. git diff --check against the base passed. The tests cover representative positive/negative retrieval, generated index/source and coverage consistency, and reader boundaries. The only generated record/provenance JSON keys changed are the skill's source-file hashes and the corpus source revision. No code was modified in this review.
I inspected the actual executable changes and an isolated source-only extraction of the proposed normative text, without using other reviews or author result summaries. The clean parent goal (#350) is available, but the precise post-review narrow acceptance mandate is not independently reconstructed. I also did not independently re-audit the whole upstream source readset, build/install a clean wheel, execute an Agent interpretation trial, or run Ascend/model/service tests.
This COMMENT records verified checks and their limits, not an approval or whole-issue completion judgment. Please keep the bounded documentation/retrieval result separate from policy-interpreter conformance and real runtime effectiveness.
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 61303fa8150472dfbda62ec8d77908bc05c73f3b and head 25cdfca466394467057416154005d5e7912370c4 (re-review after tip moved from 7dcd1578).
Blockers
-
Applicability fixtures still ignore the distributed procedure (
tests/test_vllm_ascend_distill_skills.py:11-96).- Both helpers take
instructionsbut never read it. Empty/junk/contradictory instruction text yields the same dispositions (confirmed by zengchuang-hw and david6666666; reproduced in this re-review). - RC helper hard-codes
{"Atlas 800 A3", "950DT"}at line 73, so it refuses A2 even when the procedure text would declare A2 in scope — the opposite of procedure-tied evidence. - Adaptation helper builds statuses from checkpoint flags + a metadata substring, not from
SKILL.mdbehavior. That does not close the prior gap that string/metadata presence ≠ applicability refusal.
- Both helpers take
-
#350 applicability ask remains open. Need procedure-tied refusal/missing-evidence outcomes (or an accepted written reduction on #350). Discoverability/pin checks alone are not enough.
Looks fine, not blocking
SKILL.md⏳ unverified / blockedwith required reason is the right document contract.- Local
tests/test_vllm_ascend_distill_skills.py: 10 passed; Ready checks green on this tip.
Expected before re-review
Either (a) bind the fixtures to the authoritative matrix/procedure text the skill ships, so contradictory matrices fail; or (b) label these checks as static consistency only and record an accepted reduction of the #350 applicability requirement on the issue. Then re-request review.
|
Posted by steward agent for @Dong1017 (hosted Dong-bot on Re-review at head
|
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). Status at head This comment is the steward notice for review 5937988175. It isn't a new re-review request, and no new commits have landed since that review. I'm not posting a second formal review on the same head. Dong-bot (authorized delegate) will re-read after the next push or an explicit PTAL. Open blocker (cursor[bot] CHANGES_REQUESTED on this head, with david6666666's and zengchuang-hw's findings pointing at the same thing):
To clear it, pick one:
Not blocking:
Once you've pushed one of the two fixes, request review again. |
|
@cursor[bot] @Dong1017 @david6666666 @zengchuang-hw Pushed commit
|
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 61303fa8150472dfbda62ec8d77908bc05c73f3b and head f125dbd7dda8e50c9fc6a7680a12a6c558c47ce4. This is a re-review after Rohank143's comment 6971644825. Dong-bot (authorized delegate) posted it.
Review state
- No formal APPROVED review exists on this head.
- cursor[bot]'s CHANGES_REQUESTED (review 5937988175, at
25cdfca4) is still active. Nothing seeded shows it was re-checked onf125dbd7. - The skill-eval comment for this head is PLANNED and not executed. It has 0 comparisons and lists 5 uncovered paths, so it is not evidence.
What the author describes
The described design matches the blocker. The author says:
extract_release_matrix_hardware(instructions)andextract_feature_status_table(instructions)now parse the shipped tables.- A table-versus-prose contradiction, such as A2 in the table while the prose still excludes it, raises
ValueError. - Empty instructions, a missing table, or baseline instructions without
⏳ unverified / blockedfail. - Two regression tests were added for this.
If the code does this, it is option 1 from the previous round, and it answers david6666666's A2 counterexample.
Blockers (not cleared)
- I can't verify the fix at this head.
- I have no diff for
25cdfca4..f125dbd7and no contents oftests/test_vllm_ascend_distill_skills.py. - The earlier objection was that string presence and hand-coded policy are not evidence. The author's own account of the new tests can't close it.
- Still unconfirmed:
instructionsin the tests is read from the shippedSKILL.mdor RC instructions. A literal string embedded in the test would not count.- The A2-in-table case is exercised against the real procedure text. david6666666's mutation passes or fails as claimed.
- The adaptation helper's
⏳ unverified / blockedassignment depends on the parsed table, not a hard-coded status list.
- I have no diff for
- The reviewers who raised the blockers haven't re-checked this head. cursor[bot], david6666666 and zengchuang-hw were tagged. The sticky CHANGES_REQUESTED is from
25cdfca4, and I can't clear it without independent confirmation or a diff I can read. - #350 scope. Keep the delivery record to a scoped claim, as the earlier reviews said. This PR gives procedure-bound fixtures and the SKILL.md status contract. It doesn't give runtime qualification of the skills, and it doesn't accept #350 as a whole. The PR body uses "Refs #350", which is correct.
Not blocking
- The
⏳ unverified / blockedaddition toascend-model-adaptation/SKILL.mdis the right contract. The earlier reviews agree. - The author's run (73 passed, 3 skipped, both
--checkcommands exit 0, ruff clean) is self-reported. I did not re-run it.
To clear: Either of the following is enough.
- Provide the incremental diff, or confirmation from cursor[bot], david6666666 or zengchuang-hw, that the mutation (A2 added to the release table with the exclusion prose unchanged) now fails and that the fixtures read the shipped text.
- Quote the relevant helper lines and where
instructionscomes from.
I will re-read after the next push or explicit PTAL once that evidence is on the thread.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
Incremental independent review — COMMENT (non-voting)
Acting as an AI reviewer under zengchuang-hw's standing authorization for other-author PR reviews. Reviewed head f125dbd7dda8e50c9fc6a7680a12a6c558c47ce4, base 61303fa8150472dfbda62ec8d77908bc05c73f3b, target main. This is an interim evidence assessment, not approval, merge authorization or Issue #350 completion.
Compared with my previous review at 25cdfca466394467057416154005d5e7912370c4: the incremental range contains one commit changing only tests/test_vllm_ascend_distill_skills.py (+226/-12). I also inspected the five-file base-to-head diff, current procedure text, AGENTS and review guidance, and #350's original Goal/Scope sections, without reading other reviewers' opinions or author test summaries.
Resolved at source level: instructions is no longer unused. Both helpers now parse their tables; the RC hardware allowlist is extracted rather than fixed. Added negative cases cover missing/empty tables, an A2 table/prose contradiction, removal of the blocked-status row, and absent blocker evidence. This addresses the specific previous empty-instructions/constant-allowlist limitation. These are static observations, not executed results.
Remaining test-oracle boundary: lines 198–230 still report within_declared_scope=True solely from a substring hardware match. The parser unions hardware across rows and never reads deployment["model"]. The actual procedure table and scope text distinguish model/configuration combinations and explicitly exclude another model. Consequently, by static control-flow inspection, an unlisted model on Atlas 800 A3 receives the same in-scope result as a listed model; DeepSeek-V4-Flash on A3 also borrows hardware listed for MiniMax-M3. These are proposed regression inputs, not runtime experiments performed here. Either preserve row-level model/configuration identity and test those negatives, or explicitly narrow the helper/result and test claims to hardware-token membership. A new runtime policy interpreter is not required. Table parsing alone does not establish full procedure applicability or Agent interpretation conformance.
Verification limits: no repository code or tests were executed in this pass. The prior review's 61-pass result belongs to the previous head and does not verify this changed test file. The current-head PR-triggered workflow query returned no runs. Direct check inspection showed latest knowledge-model-policy success, a successful declaration run, an earlier failed declaration run, and skill-eval execute skipped (plan/publish successful). These do not supply current-head pytest/ruff evidence or real Ascend execution; I am not calling all checks green. GitHub currently reports mergeable=false, separately from this source assessment.
The document change's explicit unverified/blocked category remains a reasonable evidence boundary; the four non-test files are unchanged in this incremental range. Approval remains unissued pending appropriately scoped test-oracle claims and applicable current-head verification. The exact narrow post-review acceptance mandate was not independently reconstructed from review-containing PR narrative; original #350 requirements and source contracts were used instead. No code, merge, deployment or closure action was taken.
f125dbd to
0fcaedb
Compare
|
@cursor[bot] @Dong1017 @david6666666 @zengchuang-hw Rebased cleanly onto latest
|
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7 and head 0fcaedbb83333a8e1154dc350da42d2c9cca9aaa. This is a re-review after Rohank143's comment 6971910881. Dong-bot (authorized delegate) posted it.
Review state
- No formal APPROVED review exists on this head.
- cursor[bot]'s CHANGES_REQUESTED (review 5937988175, at
25cdfca4) is still active. Nothing shows it was re-checked onf125dbd7or0fcaedbb. - zengchuang-hw's latest review (at
f125dbd7) is non-voting. It says theinstructionsbinding is resolved at source level. It also lists a remaining test-oracle boundary around lines 198–230 of the test file. I have no evidence that this boundary was closed or explicitly accepted. - The skill-eval comment for this head is PLANNED and not executed. It has 0 comparisons and lists 5 uncovered paths. It is not evidence.
What looks resolved (from descriptions and zengchuang-hw's source read, not from my own read)
extract_release_matrix_hardware(instructions)andextract_feature_status_table(instructions)now parse the shipped tables.- The A2 table-versus-prose contradiction is described as raising
ValueError. - Empty instructions, a missing table, baseline instructions without
⏳ unverified / blocked, and missing blocker evidence are described as failing. - If the code does this, it is option 1 from the previous round and answers david6666666's A2 counterexample.
- The
SKILL.md⏳ unverified / blockedstatus is still the right contract and is not a blocker.
Blockers
- I can't verify the fix at this head.
- The workspace has no diff for
f125dbd7..0fcaedbbor fortests/test_vllm_ascend_distill_skills.py. - The earlier objection was that string presence and hand-coded policy are not evidence. The author's own description of the tests can't close it.
- Please confirm that the
instructionspassed to the helpers is read from the shippedSKILL.mdand RC procedure files, not from an inline string literal in the test. If it is inline, the tests still don't show that the shipped procedure produces the outcome. - The
0fcaedbbcommit is described as a rebase onto8aba657dplus index regeneration. I can't confirm from here that it adds no source change afterbcf45d41.
- The workspace has no diff for
- zengchuang-hw's test-oracle boundary is not recorded as resolved. Either address it in the tests or state the limit in the delivery record. The record should say these checks are static contract-conformance checks and are not a runtime qualification. It should also say they don't complete #350.
- cursor[bot]'s CHANGES_REQUESTED is unresolved on this head. It needs a re-check from cursor[bot] or an explicit disposition.
Non-blocking
- The PR body's verification numbers (73 passed, 3 skipped; both
--checkcommands exit 0; ruff clean) are the author's. I did not run them. The body says "dual-commit" and ahead_shadeclaration, and I did not check the declaration against the head. - The skill-eval plan's 5 uncovered paths are mostly generated projections. Add a brief reviewer disposition for the uncovered canonical path,
ascend-model-adaptation/SKILL.md.
I will re-read after the next push, or after someone supplies the test file contents or diff and cursor[bot]'s re-check. This review neither approves nor changes scope.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
Incremental re-review — COMMENT (non-voting)
Reviewed new head 0fcaedbb83333a8e1154dc350da42d2c9cca9aaa against base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7, under zengchuang-hw's standing other-author review authorization. Follow-up to my f125dbd review; no other reviewers' opinions were used.
What changed: this is a rebased history, so the old-head/new-head three-dot comparison includes intervening main changes and must not be treated as the PR's own delta. The current base-relative PR still changes five files. Direct old/new file comparison confirms that the test file (blob 0803baa0de07f499f505e79014ff39f864ecf959), adaptation SKILL.md, generated adaptation record, RC procedure, AGENTS and review skill are byte-identical to the prior reviewed head. Coverage digest and corpus provenance have changed for the new base; I inspected their current base-relative patches but did not regenerate them locally.
Prior finding remains open: the unchanged RC helper still computes within_declared_scope from the union of hardware tokens, without reading the deployment model. It therefore cannot distinguish the procedure's model-specific rows. My prior proposed negative cases (an unlisted model on A3, or DeepSeek-V4-Flash borrowing MiniMax-M3's A3 row) remain applicable by static control-flow inspection. Preserve row-level applicability and add the negatives, or narrow the helper/result/test claims explicitly to hardware membership. This is a test-oracle/evidence issue, not a demonstrated runtime deployment defect. The earlier fix that makes instructions non-empty/table-dependent remains present; I am not reopening that resolved point.
Updated CI facts: unlike the previous head, current-head ready checks are now present. At inspection, installed-package 3.10/3.11, dashboard-browser and pr-references were successful; full-check 3.10/3.11 and Windows host compatibility were in progress, and macOS host compatibility was queued. Current-head policy status was successful. Skill-eval execute was skipped, with plan/publish successful. Ready run is the evidence location, not a claim that its pending jobs passed. GitHub now reports mergeable=true, resolving the previously observed mergeability obstacle but not the test-oracle concern.
No repository code/tests, wheel build, browser session or Ascend experiment was executed by me in this pass. Old local test results were not relabelled as current-head validation. Original #350 requirements and the unchanged procedure contracts remain the review basis; the narrow post-review mandate was not independently reconstructed from review-containing narrative. This is a head-bound interim COMMENT, not approval, whole-issue acceptance or merge authorization.
|
@Dong1017 @zengchuang-hw @cursor[bot] @david6666666 Pushed commit
|
zengchuang-hw
left a comment
There was a problem hiding this comment.
Incremental independent review — COMMENT (non-voting)
Head 35e17b7163e832c7120b60443e85f526fb09720b, base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7; acting under zengchuang-hw's standing other-author review authorization. Compared the actual 0fcaedbb…35e17b71 delta against my previous review. One new commit changes only the distillation test file. AGENTS, review guidance and the RC procedure are unchanged.
Resolved in source: the parser now retains model-to-hardware rows, and the helper consults the model before selecting hardware. The new explicit negative cases for DeepSeek-V4-Flash/A3 and Llama-3-70B/A3 address the two cases I previously named; positive MiniMax-M3/A3 and DeepSeek-V4-Flash/950DT cases are also present. I am not repeating the old “model is ignored” finding: that is fixed.
Remaining new-match boundary: lines 230–247 use bidirectional substring matching and then select the first matching row. A missing model defaults to "", which matches every model name in Python; with A3 the first MiniMax-M3 row therefore yields within_declared_scope=True despite missing model evidence. An unlisted derivative such as MiniMax-M3-unlisted-variant also borrows that row. These are static control-flow deductions, not executed experiments. Please require a nonempty, unambiguous declared identity (exact normalized match or explicitly documented aliases) and add missing/empty/undeclared-variant negatives. Otherwise the new row-level oracle still silently broadens the procedure's declared scope. This remains a test-helper correctness/evidence concern, not a demonstrated production deployment defect or a request for a runtime interpreter.
Current-head verification: no repository code/tests were executed in this review. At check inspection, installed-package 3.10/3.11 and pr-references were successful; full-check 3.10/3.11, dashboard-browser and Windows host compatibility were running, with macOS queued (ready run). The knowledge-model-policy commit status was success; a declaration run was still in progress. Skill-eval execute was skipped, not a performed effectiveness trial. Previous-head green checks are not current-head execution evidence.
This COMMENT records improvement and the remaining narrow identity-match issue. It is not approval, Issue #350 acceptance or merge authorization. Only original requirements, actual source/contracts, my own prior reviews and CI metadata informed this pass; no other reviewers' opinions were used.
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7 and head 35e17b7163e832c7120b60443e85f526fb09720b. This is a re-review after Rohank143's comment 6972254516. Dong-bot (authorized delegate) posted it.
Review state
- No formal APPROVED review exists on this head.
- cursor[bot]'s CHANGES_REQUESTED (review 5937988175, at
25cdfca4) is still active. Nothing seeded shows it was re-checked on35e17b71. - zengchuang-hw's review on this exact head (5952461902) is non-voting. It confirms the row-level fix and leaves one new test-oracle finding open.
- The skill-eval comment for this head is PLANNED and not executed. It has 0 comparisons and 5 uncovered paths, so it is not evidence.
Resolved since the last round (per zengchuang-hw's source read of 0fcaedbb..35e17b71, not my own read)
- The parser now keeps model-to-hardware rows, and the helper consults the model before it picks hardware.
- The DeepSeek-V4-Flash on A3 (cross-row borrowing) and Llama-3-70B on A3 (unlisted model) negatives are added.
- Positive cases (MiniMax-M3 on A3, DeepSeek-V4-Flash on 950DT) are added.
- The instruction-binding, contradiction, and empty-table cases from the earlier rounds are still in place.
- The
⏳ unverified / blockedcontract inSKILL.mdis still the right change and is not a blocker.
Blocker
- The RC applicability oracle can report in-scope without model evidence (
tests/test_vllm_ascend_distill_skills.py:230-247, per zengchuang-hw's review of this head).- The matcher uses bidirectional substring matching and then takes the first matching row.
- A missing model defaults to
"". In Python,""is a substring of every model name. On A3 the first MiniMax-M3 row then giveswithin_declared_scope=Truewith no model evidence at all. - An unlisted derivative such as
MiniMax-M3-unlisted-variantalso matches that row and borrows its scope. - This is the opposite of what #350 asked for. The RC procedure's own text says to report an evidence gap when the release identity can't be established. A test oracle that passes on missing or fuzzy model identity can't show that outcome.
- The fix is small:
- Require a non-empty model.
- Match the model exactly, after normalization, against the matrix row.
- Refuse or report an evidence gap when the model is missing or unlisted.
- Add negative tests for a missing model on A3 and for
MiniMax-M3-unlisted-varianton A3.
Verification limits
- I haven't run the tests.
- I don't have the diff or the test file contents. The blocker comes from zengchuang-hw's static read of lines 230–247 on this head. Please confirm or rebut it with the code.
- The author's reports of 26 tests passing and clean
--checkruns for coverage and index are author-reported only. They would not catch this gap, because no test sends a missing or fuzzy model.
Needed to clear
- Push the exact-match fix and the two negatives above.
- Either state that you checked cursor[bot]'s
25cdfca4blockers against the new head, or ask cursor[bot] to re-review. - If you rebase again, keep the
head_shadeclaration in the PR body current.
I will re-read after the next push or an explicit PTAL.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7 and head 35e17b7163e832c7120b60443e85f526fb09720b (re-review after tip moved from 25cdfca4; Refs #350).
Prior CR (25cdfca4) instruction-binding / hard-coded A3·950DT blockers are cleared (table parsers, empty→ValueError, contradiction + row-level listed negatives).
Blocker
- RC applicability oracle can report in-scope without model evidence (
tests/test_vllm_ascend_distill_skills.py:230-247).- Bidirectional substring match + default
"": missing/empty model orMiniMax-M3-unlisted-varianton A3 →within_declared_scope=True(reproduced). - Require nonempty exact normalized match (or documented aliases); refuse/evidence-gap on missing/unlisted/ambiguous; add those negatives.
- Bidirectional substring match + default
SKILL.md ⏳ unverified / blocked and Ready CI are fine once #1 is fixed.
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Independent tip re-review at |
Refs #350
Problem and Scoped Delivery Slice
This pull request implements the technical contract refinements and boundary/applicability test cases requested in Issue #350 comment and [PR #482 reviews](#482 (review), #482 (review)):
Unverified / Blocked Feature Status in
ascend-model-adaptation/SKILL.md:⏳ unverified / blockedstatus requiring documented evidence or blocker reasons.verified,framework unsupported,checkpoint missing, orN/A).Authoritative Instruction-Bound Applicability & Row-Level Model Matrix in
tests/test_vllm_ascend_distill_skills.py:extract_release_matrix(instructions)dynamically parses model-to-hardware mapping from# Investigate release-specific Ascend performanceinrepositories/vllm-ascend/SKILL.md(read directly viaskill_instructions(), not inline test strings).extract_feature_status_table(instructions)dynamically parses valid statuses and definitions from## Feature-status reportinginhardware/ascend/ascend-model-adaptation/SKILL.md.(Model, Hardware)applicability rather than an untyped hardware token union.MiniMax-M3onAtlas 800 A3,DeepSeek-V4-Flashon950DT).Atlas 800 A2presented to RC procedure).DeepSeek-V4-Flashpresented onAtlas 800 A3, which is only reported forMiniMax-M3).Llama-3-70BonAtlas 800 A3).⏳ unverified / blockedciting the specific blocker, strictly rejecting false categorical claims.⏳ unverified / blocked, and incomplete execution lacking blocker evidence properly fail/refuse.Coverage & Skill Index Synchronization (Dual-Commit Protocol on current main):
main(8aba657d).docs/skill-coverage.mdviapython tools/build_skill_coverage.py(verified with--check)._provenance.jsonviapython tools/build_skill_index.py --source-revision bcf45d4106efa573a18f04a3d3dd66d6fab49ca0(verified with--check).Knowledge Authorship Declaration
Reviewer & Scope Boundary Note
skill_instructions(LIBRARY, "vllm-ascend")andskill_instructions(LIBRARY, "ascend-model-adaptation").ascend-model-adaptation/SKILL.md(flagged in skill-eval plan) is the exact knowledge document refined in this PR whose contract update is validated by the test suite.