Repository navigation
Standardize evidence archives and compact duplicate model records - #472
Conversation
Define the core bundle and v2 manifest contract, normalize frozen pilot layouts, and verify external archive bytes, members and replay outputs explicitly. Keep the inventory authoritative through references and generate the pending budget checker input on demand. Replace the full generated history scan with a pinned, hash-checked source reference and retain scanner coverage on synthetic Git history. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
The current base added the Ascend model-adaptation source notes, increasing the complete Knowledge and Harness resource inventory from 927 to 928. Keep the fixed count check aligned with that tree while preserving all per-file digest comparisons. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Preserve both original snapshots and frozen checks while regenerating compact summaries and observation indexes. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 16b0e10466eec7f111eb8e0480ed6ece79e02d32.
Verified #464 follow-through: shared v2 results/artifacts.json authority for H3 and Qwen bundles; offline tools/verify_evidence_archive.py defaults to metadata-only (download opt-in); tools/evidence_archives.py hashes before bounded stream inspect, rejects unsafe paths/members, does not execute archived code; both campaigns remain INCONCLUSIVE; Release evidence-model-cleanup-v1 digests match manifests; tests/CI ready green. Retrospective migration scoped to these two campaigns as PR-stated exception to RFC default.
Standing steward review delegation while hosted Dong-bot is on usage_limit.
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 16b0e10466eec7f111eb8e0480ed6ece79e02d32.
Independent steward Approve (Dong1017) complementing cursor[bot]. Storage/tooling only per #464 scoped adopt + explicit H3/Qwen retrospective migration; campaigns remain INCONCLUSIVE; artifacts.json v2 authority and offline verify path checked.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent post-merge review — no blocking defect found in the archive/tooling scope checked.
Responding to the author's review request #472 (comment) . This PR was already merged when I performed this review. This is a revision-bound COMMENT, not a retroactive approval or an instruction to merge/close anything.
Reviewed range: base 6c7e237 → head 16b0e10. First review by this account; no previous review link or prior findings to mark resolved. I read the actual changed tooling, tests, templates and archive/projection contracts, repository AGENTS/review guidance, and the linked Issues' scope. I did not use other reviewers' opinions as review evidence.
Independently executed in an isolated checkout:
- 151 focused tests passed (archive safety/metadata, model replay, inventory, both existing pilot adapters and the synthetic source-history scanner). One pytest plugin deprecation warning.
- Offline inventory: 26 bundles, 461 hashes, zero errors; 30 external/unavailable notices remain.
- Downloaded both public Release archives through the new bounded downloader, then verified compressed identity and all regular members. H3: 96 original files / 1,259,483 original bytes; Qwen: 413 / 5,205,664.
- Compared every one of those 509 original archived files byte-for-byte with Git at the declared source/base revision, not merely with the archive's own snapshot.
- Ran the reviewed local replay implementation against authenticated originals. Recomputed summary/index outputs match the retained H3 95 entries and Qwen 130 entries; both verdicts remain INCONCLUSIVE. These are published observations, not independent trial counts.
- Recomputed the removed history-inventory file's 1,401,157-byte size and SHA-256 from base Git; both match the retained retrieval record.
- git diff --check passed.
- Queried exact-head check runs: all 13 reported success, including ready/full-check, installed package, browser and Windows/macOS jobs. These are remote check statuses, not locally rerun suites.
Not executed here: full repository suite, local installed-wheel/browser/Windows/macOS suites, local Ruff (not installed in the selected test environment), hardware/model runs, inaccessible private quality inputs or reconstruction of unpublished controller history. CI success and archive replay do not qualify those claims.
Required boundaries for downstream use: preserve the two INCONCLUSIVE campaign verdicts and explicitly limited index coverage; keep RFC acceptance separate from this storage implementation. No newly identified corrective blocker in the exercised storage path. Non-blocking follow-up: keep testing the generic normalizer/replayer against additional malformed metadata and future campaign projections before expanding migration scope.
main installs 928 packaged resources (InferMatrixLab#472); the measurement-validity reference adds one, so the installed count is 929, verified on a wheel built from a clean archive of this head. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Refs #464
Refs #297
Refs #67
Evidence bundles need one archive contract and a reproducible compact view. The H3 four-device and Qwen3.8 bundles also retain cumulative checkpoints and repeated diagnostic source snapshots in Git. This PR adds the shared v2 template/verifier and replaces those two presentations with compact, verifiable bundles backed by byte-preserved public archives.
Changes
results/artifacts.jsonauthoritative, normalize both original v1 pilots, reject duplicate inventory metadata, and generate a disposable budget view for the pending ci: enforce evidence byte budgets and external asset metadata (Refs #273, R6) #328 integration.6c7e237935a22cc5e90e954d4f397376e753e7d8. Download and verify the published copies before removing redundant/generated current-tree files. This is a bounded extension beyond the RFC's default no-retrospective-migration scope; other frozen campaigns and closed/unmerged PRs are not migrated.The versioned Release preserves all 509 original files at their original campaign-relative paths, including frozen validators, failed attempts and superseded records. Each archive includes original-file digests and pinned replay code/projection. Retained original code/configuration bytes remain unchanged. Git history is not rewritten. Both campaign verdicts remain INCONCLUSIVE.
The new replayer builds views from authenticated originals without reading retained summaries/indexes. Identical observations of the same named attempt share one entry with every source pointer retained; revised observations, retries and anonymous records stay distinct. It recomputes declared sample distributions and labels other frozen statistics/quality verdicts as recorded values. Index entries are published record/attempt observations, not independent trial counts; unavailable controller history and narrative-only attempts remain explicit coverage limits. Structure/integers compare exactly, with predeclared absolute/relative float tolerances of
1e-12for these two projections.The earlier cleanup of the 1,401,157-byte generated Ascend history inventory remains included, with its exact historical source, digest and retrieval command. Scanner behavior is tested with a small synthetic history. Across the complete PR, current-tree bytes decrease by 7,357,351. R6 remains report-only. The installed-resource assertion also retains the base-tree correction from 927 to 928, without weakening per-file digest checks.
Validation at 16b0e10
ruff check src tests toolsandgit diff --check: passed.2d4a2ef1eaabb8f3efe1b4fe8337fc2792c735c7e235e354ca6e962553684377.4bf744194caef014eeaf213d5f9f6a6826c97b799b06b78bba8f9c354e228227.Knowledge-model declaration: not applicable; no protected paths changed.
Limits and follow-ups
This is storage and tooling work, not a new model run or acceptance claim. Private weights, media and raw receipts remain unavailable; regenerating public-record views cannot close those gaps. The small H3 Image configuration copy consumed by its frozen validator remains intact. MiniCPM/H3 Image v1 pilot bytes and their historical acceptance records are unchanged.
Independent review and decision-owner acceptance remain pending. This PR does not close #464 or enable blocking size gates. #328's byte reporter integration and the RFC's separate completion/pilot-acceptance records remain follow-ups.