Skip to content

Standardize evidence archives and compact duplicate model records - #472

Merged
Dong1017 merged 3 commits into
mainfrom
codex/464-evidence-archive-tools
Nov 16, 2026
Merged

Dong1017 merged 3 commits into
mainfrom
codex/464-evidence-archive-tools

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #464
Refs #297
Refs #67

Evidence bundles need one archive contract and a reproducible compact view. The H3 four-device and Qwen3.8 bundles also retain cumulative checkpoints and repeated diagnostic source snapshots in Git. This PR adds the shared v2 template/verifier and replaces those two presentations with compact, verifiable bundles backed by byte-preserved public archives.

Changes

  • Define the core bundle, complete-trial-index, replay and storage-exception templates. Keep results/artifacts.json authoritative, normalize both original v1 pilots, reject duplicate inventory metadata, and generate a disposable budget view for the pending ci: enforce evidence byte budgets and external asset metadata (Refs #273, R6) #328 integration.
  • Verify archive bytes/SHA-256 before bounded streaming inspection. Reject unsafe members and paths, enforce resource limits, verify pinned inputs, and distinguish access/corruption/replay failures. Offline checks never download; no archived code is executed or extracted.
  • For the requester's explicitly scoped follow-up, archive the complete original H3 four-device and Qwen3.8 trees from 6c7e237935a22cc5e90e954d4f397376e753e7d8. Download and verify the published copies before removing redundant/generated current-tree files. This is a bounded extension beyond the RFC's default no-retrospective-migration scope; other frozen campaigns and closed/unmerged PRs are not migrated.
Bundle Original tracked files / bytes Compact files / bytes Indexed published observations
H3 Ref2VA, four logical Ascend devices 96 / 1,259,483 25 / 197,339 95
Qwen3.8 FP8 single-request 413 / 5,205,664 27 / 230,487 130

The versioned Release preserves all 509 original files at their original campaign-relative paths, including frozen validators, failed attempts and superseded records. Each archive includes original-file digests and pinned replay code/projection. Retained original code/configuration bytes remain unchanged. Git history is not rewritten. Both campaign verdicts remain INCONCLUSIVE.

The new replayer builds views from authenticated originals without reading retained summaries/indexes. Identical observations of the same named attempt share one entry with every source pointer retained; revised observations, retries and anonymous records stay distinct. It recomputes declared sample distributions and labels other frozen statistics/quality verdicts as recorded values. Index entries are published record/attempt observations, not independent trial counts; unavailable controller history and narrative-only attempts remain explicit coverage limits. Structure/integers compare exactly, with predeclared absolute/relative float tolerances of 1e-12 for these two projections.

The earlier cleanup of the 1,401,157-byte generated Ascend history inventory remains included, with its exact historical source, digest and retrieval command. Scanner behavior is tested with a small synthetic history. Across the complete PR, current-tree bytes decrease by 7,357,351. R6 remains report-only. The installed-resource assertion also retains the base-tree correction from 927 to 928, without weakening per-file digest checks.

Validation at 16b0e10

  • Local Python 3.12.13 / pytest 9.2.0: archive, projection, inventory, both pilot adapters, scanner and template tests — 158 passed in 5.15 s.
  • ruff check src tests tools and git diff --check: passed.
  • Offline inventory: 26 bundles, 461 current-tree hashes, zero errors; 30 external/unavailable notices remain explicit. Original archived file and frozen-manifest checks are performed separately during replay.
  • Downloaded both new public Release assets through the shared verifier. Checked compressed bytes/hash, all 509 original files, original frozen manifests, pinned replay inputs and local replay-code identity; independently rebuilt outputs match both retained bundles.
  • H3 archive: 244,406 bytes, SHA-256 2d4a2ef1eaabb8f3efe1b4fe8337fc2792c735c7e235e354ca6e962553684377.
  • Qwen3.8 archive: 2,469,784 bytes, SHA-256 4bf744194caef014eeaf213d5f9f6a6826c97b799b06b78bba8f9c354e228227.
  • Regression coverage includes repeated observations versus changed history, failed/retried/cancelled/pending attempts, original-file and replay-code tampering, withheld partial capture on verification failure, and refusal to overwrite an existing output directory.
  • Current-head CI passed. Python 3.10 and 3.11 each report 4,812 passed, 25 skipped, 616 subtests passed (360.60 s / 451.90 s). Skips cover optional NumPy/jsonschema/Torch/ffmpeg dependencies, opt-in fresh-environment tests and host-specific cases. The dedicated browser job, both installed-package jobs and Windows/macOS compatibility jobs all passed. These are scaffold checks, not hardware/model qualification.

Knowledge-model declaration: not applicable; no protected paths changed.

Limits and follow-ups

This is storage and tooling work, not a new model run or acceptance claim. Private weights, media and raw receipts remain unavailable; regenerating public-record views cannot close those gaps. The small H3 Image configuration copy consumed by its frozen validator remains intact. MiniCPM/H3 Image v1 pilot bytes and their historical acceptance records are unchanged.

Independent review and decision-owner acceptance remain pending. This PR does not close #464 or enable blocking size gates. #328's byte reporter integration and the RFC's separate completion/pilot-acceptance records remain follow-ups.

Define the core bundle and v2 manifest contract, normalize frozen pilot layouts, and verify external archive bytes, members and replay outputs explicitly. Keep the inventory authoritative through references and generate the pending budget checker input on demand.

Replace the full generated history scan with a pinned, hash-checked source reference and retain scanner coverage on synthetic Git history.

Agent-assisted-by: Codex (implementation and validation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
The current base added the Ascend model-adaptation source notes, increasing the complete Knowledge and Harness resource inventory from 927 to 928. Keep the fixed count check aligned with that tree while preserving all per-file digest comparisons.

Agent-assisted-by: Codex (implementation and validation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@amy-why-3459
amy-why-3459 marked this pull request as ready for review November 16, 2026 12:32
Preserve both original snapshots and frozen checks while regenerating compact summaries and observation indexes.

Agent-assisted-by: Codex (implementation and validation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@amy-why-3459 amy-why-3459 changed the title Standardize evidence bundles and verify external archives Standardize evidence archives and compact duplicate model records Nov 16, 2026
@amy-why-3459

Copy link
Copy Markdown
Collaborator Author

@Dong1017 @zengchuang-hw

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 16b0e10466eec7f111eb8e0480ed6ece79e02d32.
Verified #464 follow-through: shared v2 results/artifacts.json authority for H3 and Qwen bundles; offline tools/verify_evidence_archive.py defaults to metadata-only (download opt-in); tools/evidence_archives.py hashes before bounded stream inspect, rejects unsafe paths/members, does not execute archived code; both campaigns remain INCONCLUSIVE; Release evidence-model-cleanup-v1 digests match manifests; tests/CI ready green. Retrospective migration scoped to these two campaigns as PR-stated exception to RFC default.
Standing steward review delegation while hosted Dong-bot is on usage_limit.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 16b0e10466eec7f111eb8e0480ed6ece79e02d32.
Independent steward Approve (Dong1017) complementing cursor[bot]. Storage/tooling only per #464 scoped adopt + explicit H3/Qwen retrospective migration; campaigns remain INCONCLUSIVE; artifacts.json v2 authority and offline verify path checked.

@Dong1017
Dong1017 merged commit 2cbfab6 into main Nov 16, 2026
13 checks passed
@Dong1017
Dong1017 deleted the codex/464-evidence-archive-tools branch November 16, 2026 13:36

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent post-merge review — no blocking defect found in the archive/tooling scope checked.

Responding to the author's review request #472 (comment) . This PR was already merged when I performed this review. This is a revision-bound COMMENT, not a retroactive approval or an instruction to merge/close anything.

Reviewed range: base 6c7e237 → head 16b0e10. First review by this account; no previous review link or prior findings to mark resolved. I read the actual changed tooling, tests, templates and archive/projection contracts, repository AGENTS/review guidance, and the linked Issues' scope. I did not use other reviewers' opinions as review evidence.

Independently executed in an isolated checkout:

  • 151 focused tests passed (archive safety/metadata, model replay, inventory, both existing pilot adapters and the synthetic source-history scanner). One pytest plugin deprecation warning.
  • Offline inventory: 26 bundles, 461 hashes, zero errors; 30 external/unavailable notices remain.
  • Downloaded both public Release archives through the new bounded downloader, then verified compressed identity and all regular members. H3: 96 original files / 1,259,483 original bytes; Qwen: 413 / 5,205,664.
  • Compared every one of those 509 original archived files byte-for-byte with Git at the declared source/base revision, not merely with the archive's own snapshot.
  • Ran the reviewed local replay implementation against authenticated originals. Recomputed summary/index outputs match the retained H3 95 entries and Qwen 130 entries; both verdicts remain INCONCLUSIVE. These are published observations, not independent trial counts.
  • Recomputed the removed history-inventory file's 1,401,157-byte size and SHA-256 from base Git; both match the retained retrieval record.
  • git diff --check passed.
  • Queried exact-head check runs: all 13 reported success, including ready/full-check, installed package, browser and Windows/macOS jobs. These are remote check statuses, not locally rerun suites.

Not executed here: full repository suite, local installed-wheel/browser/Windows/macOS suites, local Ruff (not installed in the selected test environment), hardware/model runs, inaccessible private quality inputs or reconstruction of unpublished controller history. CI success and archive replay do not qualify those claims.

Required boundaries for downstream use: preserve the two INCONCLUSIVE campaign verdicts and explicitly limited index coverage; keep RFC acceptance separate from this storage implementation. No newly identified corrective blocker in the exercised storage path. Non-blocking follow-up: keep testing the generic normalizer/replayer against additional malformed metadata and future campaign projections before expanding migration scope.

BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 17, 2026
main installs 928 packaged resources (InferMatrixLab#472); the measurement-validity
reference adds one, so the installed count is 929, verified on a wheel built
from a clean archive of this head.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Tiered Experiment Artifact Archiving and PR Size Control

3 participants