Skip to content

Add consistent RSI result delivery bundles - #485

Open
amy-why-3459 wants to merge 3 commits into
mainfrom
feat/result-delivery-contract
Open

amy-why-3459 wants to merge 3 commits into
mainfrom
feat/result-delivery-contract

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and resulting behavior

Refs #468

Iterative optimization retains execution and assessment records but has no common delivery bundle. This adds inferrsi export --input ... --output ..., producing report.md, offline report.html, campaign-result.json and artifacts.json from one retained snapshot. Project and Benchmark CLI final operations automatically return a separate delivery result; report failures preserve committed execution and existing exit codes.

Implementation, PR submission and these review fixes were explicitly requested by the proposing user. RFC adoption, Issue closure and campaign performance acceptance are not claimed. The RFC remains proposed; this PR's ready-for-review state does not adopt its contract.

Contract and discussion resolutions

Requirement Implementation
Separate workflow completion, selection and acceptance Versioned projection preserves historical verdicts and exposes current delivery_accepted separately; search PASS never substitutes for final
Archive authority from #464 Delivery indexes the exact campaign results/artifacts.json; archive metadata is not duplicated
Candidate binding Verify retained fingerprints, or exact revision plus fixed run/config references bound to the final attempt and frozen protocol
Damaged history Preserve readable committed-state assessments and completion facts when original observations are unavailable or conflicting; identify the exact cached source and prominently label it unverified. Cached final records never verify current delivery acceptance
Concurrent writes Separate degraded reader, controller lock, bounded digest retries and fixed observation cutoff; Harness's normal reader stays strict
Safe reporting/publication Escaped Markdown/HTML, offline curve and details, path/symlink checks, no fetch/execution, no-replace publication and post-rename reconciliation
Historical compatibility Explicit adapters for summary, legacy A/B, Apps operations and tagged external records preserve their original scope

Field definitions, source authority, adapter mappings, publication behavior and limitations are documented in docs/result-delivery.md.

Review fixes and validation

Current head: 9b55c12de4fb8dd1c9389f6f18551c0a206aba49.

  • The history-loss finding is valid and fixed. assessment_provenance distinguishes retained observations from cached state, including artifact reference, JSON pointer and verification status. Missing, malformed and changed observations retain cached PASS/FAIL/INCONCLUSIVE history without current acceptance or derived comparison gains. Conflicts, absent/malformed cached assessments and interrupted finals have regression coverage.
  • The browser-CI routing finding is valid and fixed. The existing dashboard-browser job now explicitly selects the offline delivery report regression, covering both intact and missing final observations, and uploads its JUnit results. Both variants disable JavaScript and networking.
  • The duplicate-location reader finding is valid and fixed. Unique artifact IDs cannot hide repeated bundled paths; the reader rejects duplicates before checksum validation. Regressions cover all three bundled files with the invalid entry before and after the original.
  • The oversized optional-observation finding is valid and fixed. Files over the 8 MiB document limit are indexed as unreadable/not checked without reading or hashing their bytes, including during consistency rechecks. Cached history stays explicitly unverified; oversized required plan/state/campaign documents still prevent publication.
  • Local affected tests (test_result_delivery.py, test_optimization_session.py, test_project_optimization.py): 123 passed, 0 skipped on the current head, including both offline browser variants. The new history, duplicate-location and optional oversized-observation regressions reproduced the original problems before their fixes.
  • Local execution of the exact updated browser-job test list on preceding fix a5818e64: 41 passed, 0 skipped, including both delivery report variants.
  • ruff check src tests tools and git diff --check: passed.
  • GitHub dashboard-browser on preceding fix a5818e64: 41 passed, 0 skipped, with both offline report variants explicitly PASSED in the job log. The JUnit artifact was uploaded. CI checked out merge 45f09ae9cc056f48b3de1f3a4d12d8208999ae93, combining that head with base 0311393c62330c041a01238dc86823b80e020561. This demonstrates the corrected CI routing; it is not presented as a current-head rerun.
  • Current-head CI has started after the second push; no current full-suite pass is claimed yet.

Historical validation, not a current-head claim: on e2e9d0876d587db0680e79889562911a1e5fadf2, complete Python 3.10 and Python 3.11 CI each recorded 4,975 passed, 30 skipped and 647 subtests passed. Those CPU runs skipped the then-unrouted optional report browser test; they did not verify its offline rendering assertions.

Limits and remaining acceptance

Evidence and retained code are referenced, not copied or downloaded. Unsupported historical schemas fail explicitly. The substitute-binding adapter does not implement the service-reuse execution producer. No GPU/NPU performance rerun or real-model reproduction was performed. Re-review and RFC adoption remain pending; merging this implementation alone does not complete the RFC.

Project consistent experiment snapshots into Markdown, offline HTML, versioned results and a source artifact index. Keep historical verdicts separate from current verification and publish delivery after iterative final without changing execution outcomes.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review at e2e9d08 against a02ef18. Result: COMMENT, not APPROVE. I inspected the delivery projection, adapters, snapshot/index handling, renderer, CLI integration and focused tests independently of author/reviewer conclusions.

Two issues need explicit resolution:

  1. Offline HTML regression is not wired into a browser-enabled CI path.
    tests/test_result_delivery.py:588 uses pytest.importorskip('playwright.sync_api'). The dev extra does not install Playwright, while the dashboard-browser job's explicit test list omits this new test. Consequently a normal full-check environment can skip the exact offline/no-JS test, and even a non-draft dashboard-browser run would not select it. The current head's dashboard-browser check is itself skipped. Add this test to a browser-enabled validation path and retain its actual result; the no-network/no-script rendering requirement should not be considered verified by a green CPU suite. My local attempt reached browser launch and failed because the expected Chromium executable is absent, not because an HTML assertion failed.

  2. Missing retained observation drops an available historical final verdict instead of separating history from current verification.
    In projection.py:281/345, the final assessment is sourced only from the separately retained observation; if that file is missing, the exported assessment becomes null even when the unchanged committed state still contains its final assessment. Independent synthetic reproduction: start with a completed fixture whose committed final assessment is PASS, remove only final/…/observation.json, export; acceptance.verdict becomes null, exported final-phase checks are empty and delivery_accepted stays false. The last part is correctly fail-closed. The proposed original contract also asks to preserve historical verdicts with a current evidence gap, however, and test_result_delivery.py:222 currently encodes the null behavior.
    Please resolve that contract difference explicitly. A clearly labelled cached/unverified historical observation can be retained separately from current acceptance; do not treat cached state as freshly verified evidence or restore delivery_accepted merely because it says PASS. If discarding that historical value is intentional, it needs an explicit contract decision rather than an implicit change.

Actual checks:

  • tests/test_result_delivery.py + tests/test_optimization_session.py + tests/test_project_optimization.py: 93 passed, 1 browser-launch failure.
  • Rerun excluding only test_offline_html_remains_readable_without_scripts_or_network: 93 passed, 1 deselected, 0 skipped.
  • Separate CPU probe confirmed the missing-observation/history case above. Source checkout remained unchanged.
    These tests exercise publication conflict/failure, source consistency, identity binding, ratio/usage handling, legacy scope and final-workflow integration; they do not establish browser readability or real device performance.

At the exact-head CI observation, declarations, installed-package (3.10/3.11), pr-references and lint-and-smoke succeeded; full-check (3.10/3.11) and Windows host compatibility were running, macOS queued, dashboard-browser skipped. Aggregate CI was not yet complete.

The supplied original requirements are a proposed contract with an unverified adoption/history boundary. I have not silently accepted a new schema/acceptance policy or used PR discussion resolutions to amend them. No code changes, merge, deployment, model/device execution or external publication beyond this review.

@amy-why-3459
amy-why-3459 marked this pull request as ready for review November 17, 2026 12:59

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready-for-review transition follow-up on unchanged head e2e9d08 (base a02ef18). I independently verified draft=false and examined newly available CI evidence rather than treating the unchanged head as a reason to skip the review.

Newly verified results:

  • The ready-triggered dashboard-browser job 134056504744 in run 44088301436 succeeded: 39 passed in 89.93s. Its checkout log identifies merge commit a49648d, combining this head with the stated base.
  • That job's actual command still selects the profile shortcut and existing knowledge/experiment browser suites, not tests/test_result_delivery.py. Thus draft-to-ready enables existing dashboard regression coverage but does not resolve the new report.html test-routing concern.
  • The earlier same-head full-check (3.11) job 134047581228 in run 44085744339 completed with 4975 passed, 30 skipped and 647 subtests passed. Its actual skip log specifically reports tests/test_result_delivery.py:589 could not import playwright.sync_api. It uses the same PR merge checkout. This independently confirms the coverage hole identified by source inspection in my prior review, rather than merely hypothesizing it from dependencies.
  • At this observation, the newer ready-triggered full-check jobs were still running; I do not present their pending results as passes.

Prior review: #485 (review) . No source change was made in this transition, so the missing-observation/history representation concern is not marked resolved. The prior 93 local CPU passes were not rerun here. This round inspected real CI commands/result/skip lines and current metadata; it did not execute a browser locally or perform device/model work.

Result remains COMMENT, not APPROVE. Add the offline report test to a browser-enabled execution path and retain its result. Existing dashboard/browser success is not evidence that the new delivery report's offline/no-JS assertions ran. No repository changes, CI reruns, merge or deployment performed.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head e2e9d0876d587db0680e79889562911a1e5fadf2.

Blockers:

  1. src/inferrsi/apps/delivery/projection.py (~L281–295, final ~L345): missing observation.json nulls historical final assessment (acceptance.verdict=null, completion incomplete) even when committed state.json still holds assessment.verdict=PASS. That contradicts RFC #468 historical preservation; tests/test_result_delivery.py (~L222–224) currently encodes the wrong behavior. delivery_accepted correctly stays false, but historical assessment must be retained as a clearly labelled cached/unverified value (or an explicit contract decision must say discarding it is intentional).
  2. Offline HTML Playwright coverage is not on a browser-enabled CI path: tests/test_result_delivery.py (~L588) importorskips Playwright, and ready dashboard-browser does not select this suite. Green CPU/Ready checks do not verify the offline/no-JS delivery claim.

zengchuang-hw COMMENTED reviews on this tip raise the same issues; treat them as blocking for PASS.

Merge is not appropriate while these blockers stand (and while #468 remains status:proposed for freezing the delivery schema). Fix and re-request review.

Verified non-blocking: delivery_accepted vs search PASS separation; archive indexing of results/artifacts.json; candidate binding paths; degraded export reader vs strict harness reader; focused CPU delivery tests mostly green on tip CI.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on usage_limit).

Independent formal review on tip e2e9d087: DO NOT PASS → Request changes as cursor[bot]: #485 (review)

Blockers match @zengchuang-hw tip comments: historical final assessment dropped when observation.json is missing, and offline HTML Playwright coverage is not on a browser-enabled CI path. Green Ready/CPU does not clear those. Keep #468 status:proposed until the contract gaps are resolved; this PR correctly avoids claiming RFC adoption.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explicit incremental response to direct-mention request 6949208063, created/updated 2026-11-17T13:19:28Z. Reviewed unchanged head e2e9d08 against a02ef18. I used the independently isolated original delivery scope, not the source comment's opinions or linked review conclusions.

New independent reader-integrity finding:
src/inferrsi/apps/delivery/init.py:147–152 checks duplicate artifact IDs, then creates a dictionary keyed by bundle location. Distinct IDs with the same bundle location are silently collapsed before checksum verification.
I generated a synthetic valid bundle, inserted a second report.md index entry with a unique ID and deliberately wrong all-zero SHA-256 before the original entry, then called read_bundle. It returned successfully. The later original entry shadows the invalid earlier entry, so that listed artifact is never checked. The exporter does not generate this duplicate normally; this is a malformed-index/reconciliation-reader robustness gap, not a claim of forged experimental PASS or a cryptographic authenticity guarantee.
Reject duplicate bundled locations, or otherwise validate every listed bundled entry and its identity/path relation before successful reconciliation. Include a fixture showing a unique-ID duplicate location cannot hide an invalid digest.

Actual new execution:

  • Focused delivery publication/snapshot/path/reader/interrupted-final/usage tests: 11 passed, 34 deselected, 0 skipped.
  • The standalone duplicate-location probe reproduced successful reading despite the invalid shadow entry. Temporary synthetic output only; repository sources were unchanged.
    No new local browser or full-suite execution is claimed.

CI reconciliation for this request: I independently checked the newly completed ready-run full-check jobs 134056504817 (3.11) and 134056504748 (3.10), run 44088301436. Each reports 4975 passed, 30 skipped, 647 subtests passed, with Ruff successful. Both use merge checkout a49648d (this head plus the stated base), and both explicitly skip test_result_delivery.py:589 because playwright.sync_api is missing. The already verified dashboard-browser job's 39 passes remain separate; its command does not select that delivery test. These are inspected CI results, not local reruns.

Earlier review #485 (review) and ready-transition follow-up #485 (review) remain relevant. The offline HTML coverage gap and explicit historical-observation/current-acceptance contract distinction are not marked resolved merely because the head is unchanged or CI is green.

Result: COMMENT, not APPROVE. Original contract-adoption/history gaps remain explicit. No code changes, CI reruns, merge, deployment, model/device or paid experiment.

Preserve cached final and search assessments with explicit unverified provenance when retained observations are unavailable or conflicting. Keep current acceptance fail-closed and run both offline delivery report cases in browser CI.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining degraded-read case for optional observations.

Comment thread src/inferrsi/apps/delivery/snapshot.py Outdated
size = path.stat().st_size
if document:
if size > MAX_DOCUMENT_BYTES:
raise ValueError("Delivery source document exceeds the read limit")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new cached-history fallback is bypassed when an optional observation is larger than the read limit. I replaced only a synthetic final observation.json with an 8 MiB + 1 byte file; export failed and published no bundle, although committed state still held the historical verdict. Please retain the read limit and record this optional observation as unreadable/unverified so export can use the labelled cache. Required control documents should still fail closed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This finding is correct. Fixed in 9b55c12d, now on the PR branch.

An optional document over 8 MiB is indexed as availability: unreadable, integrity: not_checked, with its recorded expected digest retained and observed_sha256: null. Neither the first read nor the consistency recheck opens or hashes its oversized bytes. This allows the explicitly labelled cached history; an unverified cached final still cannot establish delivery acceptance. Oversized required plan.json, state.json and campaign.json continue to prevent publication.

The optional-observation reproduction failed before the fix. New regression cases cover search/final observations, prohibit opening the oversized path even during rechecks, and verify that all three required control documents fail closed. Current-head affected suite: 123 passed, 0 skipped, including both offline browser variants; Ruff and whitespace checks passed. Fresh CI for this push is still pending. Please re-review; I have left the thread unresolved.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent synchronize-event follow-up: e2e9d08 → a5818e6. PR metadata base remains a02ef18. Previous review: #485 (review) . Result: COMMENT, not APPROVE.

Verified progress:

  1. The workflow now explicitly selects test_offline_html_remains_readable_without_scripts_or_network in the browser-enabled job. I independently read job 134072592912's command/result lines: 41 passed in 85.65s, including both [False] and [True] delivery-report cases. Thus the prior CI routing gap is resolved for this run, not merely changed in YAML. The log checks out merge commit 45f09ae combining this head with 0311393, which differs from the PR metadata base. This is hosted merge-check evidence, not a local direct-head browser run.
  2. The projection now retains missing/conflicting observation history from committed state with explicit cached_state/unverified provenance and a state.json pointer. The renderer labels that provenance, final metrics are withheld when only cached history is available, and current delivery acceptance requires a verified final observation. My independent missing-observation probe confirms cached PASS is labelled unverified, delivery_accepted=false, and no final metrics are derived. The previous loss-of-readable-history concern is addressed in the tested cases without promoting cached PASS to current acceptance.

Still reproducible:
The duplicate bundled-location reader issue remains. On this head I inserted an additional report.md index entry with a distinct ID and an incorrect all-zero digest before the valid entry; read_bundle still succeeded because the dictionary keyed by location shadows the invalid entry. Reject duplicate bundled locations or validate every listed bundled entry before declaring reconciliation valid. This is a malformed-index reader gap, not a claim that the ordinary exporter generates duplicates or that experimental results were forged.

Actual local regression run:
tests/test_result_delivery.py, tests/test_optimization_session.py and tests/test_project_optimization.py, excluding the offline browser test: 110 passed, 2 deselected, 0 skipped. Also ran the two independent probes above. Repository guidance was unchanged; checkout remained clean.

At the exact-head check observation, browser, host compatibility, installed packages, references, declarations and lint/smoke had successful results, while both newer full-check jobs were still in progress. No aggregate-completion claim is made. Prior CPU/CI results are not counted as newly run tests.

The proposed-contract adoption/history boundary remains explicit; these fixes are not authority to revise unrelated acceptance rules. No local browser retry, repository edit, CI rerun, merge, deployment, paid/model/device experiment.

Reject duplicate bundled locations before checksum reconciliation. Keep oversized optional observations unreadable without hashing them and preserve labelled cached history, while required control documents still prevent export.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@amy-why-3459

Copy link
Copy Markdown
Collaborator Author

Addressing the initial review, ready-for-review follow-up, blocking review, reader-integrity finding, synchronize follow-up, and steward summary. These technical findings are valid. Fixes a5818e64 and 9b55c12d are now on the PR branch.

  1. Historical assessment loss: readable committed-state assessments and completion facts are retained when the separate observation is missing, invalid or conflicting. Acceptance, final, checks and search history identify the source through assessment_provenance, including the indexed state.json reference and exact JSON pointer. Reports prominently label the fallback cached state; unverified. A cached final never enables delivery_accepted or derived gains. Missing/malformed cached assessments remain unassessed, interrupted finals remain incomplete, and search PASS never replaces final. The documented source-authority mapping is now explicit. Regression tests cover cached PASS/FAIL/INCONCLUSIVE, source conflicts and invalid/missing records; the new history regressions failed before the implementation fix.

  2. Offline browser CI routing: the existing dashboard-browser job now selects test_offline_html_remains_readable_without_scripts_or_network, parametrized for intact and missing final observations. Both variants run with JavaScript disabled and networking offline. The actual GitHub browser job passed 41 tests, 0 skipped; its log explicitly records both new variants as PASSED, and the JUnit artifact was uploaded. Checkout was merge 45f09ae9cc056f48b3de1f3a4d12d8208999ae93, containing head a5818e64f2edbb8bf4ac415c5785c5844fd083ae and base 0311393c62330c041a01238dc86823b80e020561.

  3. Duplicate bundled locations: fixed in 9b55c12d. The reader rejects repeated bundled paths before constructing the lookup used for checksum verification, even when every artifact ID is unique. Six regression cases cover all three entry files, with a bad digest inserted both before and after the original. The pre-fix reproduction confirmed that an earlier invalid entry could be silently shadowed.

The oversized optional-observation comment is also valid and fixed in 9b55c12d; its original inline thread has a separate response.

Validation on current head 9b55c12de4fb8dd1c9389f6f18551c0a206aba49: 123 local affected tests passed, 0 skipped, including both offline browser variants. Ruff and whitespace checks passed. The 41-test local and hosted browser-job results above belong to preceding fix a5818e64; the latest push has started fresh CI, whose full-suite result is still pending. These are software checks, not model/device performance validation.

Please re-review the updated head. RFC #468 remains proposed; this change does not claim RFC adoption, issue completion or merge readiness, and does not resolve/dismiss the blocking review on the reviewers' behalf.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of a5818e6 → 9b55c12, combining the synchronize event with the isolated re-review request in inline comment 4600321424 (2026-11-17T13:37:18Z). I used actual code and tests, not that comment's implementation/result assertions. Previous review: #485 (review) .

Verified resolved in this delta:

  • read_bundle now rejects a repeated bundled location before constructing its verification map. I reran my exact earlier synthetic shadow-entry probe: a distinct-ID report.md entry with a wrong digest before the valid entry now raises “Duplicate delivery bundle location”. The new regression matrix also covers each of the three bundle files with the duplicate before/after the valid entry.
  • Optional oversized observation documents are represented as unreadable/not_checked without reading their bytes, retaining cached historical provenance instead of aborting the entire export. Required oversized plan/state/campaign documents still prevent publication. Snapshot rechecks retain the document/read-limit mode. The new tests exercise both search/final optional observations and all three required controls. This preserves the distinction between unavailable evidence and current acceptance; no general bounded-memory guarantee under arbitrary concurrent filesystem replacement is claimed.

Actual checks on this exact head:

  • tests/test_result_delivery.py + tests/test_optimization_session.py + tests/test_project_optimization.py, excluding only the offline browser test: 121 passed, 2 deselected, 0 skipped.
  • Independent previous shadow-entry reproduction now rejected, as described above.
  • Independently inspected new browser-job 134077547958 logs: 41 passed in 84.55s, including both offline report cases. The job tested merge checkout 3933ce9 combining this head with 0311393, not a direct-head local browser run; PR metadata still reports base a02ef18.
  • Repository guidance unchanged; checkout clean.

No new defect found in these checked paths. The previous duplicate-location issue is resolved by code plus reproduction, and the earlier offline CI routing/cached-history corrections remain covered by this regression set. At the check observation, both full-check jobs and Windows compatibility were still running and macOS queued; installed packages, browser, references, declarations and lint/smoke had successful results. Pending aggregate evidence is not counted as a pass.

Result: COMMENT, not APPROVE at this checkpoint. This is not adoption of the entire proposed schema contract or an assertion that original acceptance-history gaps have disappeared. No local browser launch retry, full local repository suite, real-data/device/model experiment, source edit, CI rerun, merge or deployment.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI evidence addendum to #485 (review), bound to unchanged head 9b55c12.

I independently fetched the two newly completed full-check job logs in run 44094343338:

  • Python 3.10, job 134077549527: 5012 passed, 31 skipped, 647 subtests passed in 525.88s; Ruff reports all checks passed.
  • Python 3.11, job 134077548505: 5012 passed, 31 skipped, 647 subtests passed in 353.42s; Ruff reports all checks passed.

Both checkout logs identify full merge SHA 3933ce9, merging this exact head into 0311393. These are integration-checkout results, not direct-head runs against the PR metadata base a02ef18.

Both CPU suites still skip the two offline delivery browser cases at tests/test_result_delivery.py:743 because Playwright is absent. That skip must not be represented as those cases passing in the full-check jobs. The separate current-head integration browser job 134077547958 already independently checked in my previous review ran 41 tests including both delivery cases; I am referencing that existing verified evidence, not claiming another browser rerun.

The previously pending full-check evidence is now available. Current run metadata also reports Windows/macOS host-compatibility success, but I have not inspected those jobs' raw logs in this addendum; metadata alone is not a new detailed host validation. This round executed no new local tests: the prior 121-passed regression run and duplicate-location probe remain prior evidence.

Result remains COMMENT, not an automatic approval from green CI. No new defect is asserted by these logs. Original acceptance-history/formal schema-adoption gaps remain explicit; this does not certify the entire proposed contract or any campaign/device performance outcome. No changes, CI reruns, merge or deployment.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head 9b55c12de4fb8dd1c9389f6f18551c0a206aba49 (re-review after tip moved past prior CR on e2e9d087).

Prior blockers resolved:

  1. Missing/conflicting observation.json now falls back to labelled cached_state / unverified historical assessment from committed state (projection.py); delivery_accepted stays false until current verification. Local tests/test_result_delivery.py encodes retention.
  2. Ready dashboard-browser now selects tests/test_result_delivery.py::test_offline_html_remains_readable_without_scripts_or_network; tip browser job passed both offline variants (41 tests).

Also verified on tip: duplicate bundle-location rejection and oversized optional observation → unreadable/not_checked with cached history (david/zeng follow-ups). Local focused suite: 72 passed, 2 skipped. Ready run https://github.com/InferMatrixLab/InferRSI/actions/runs/44094343338 all success.

Scope note (not a code defect): governing RFC #468 remains status:proposed; this Approve is for the preparatory Refs slice only — not RFC adoption or Issue closure. Merge held for a second qualifying write-access LGTM (and preferred explicit RFC accept / maintainer land-pre-accept decision).

@Dong1017

Copy link
Copy Markdown
Collaborator

Steward note (authorized delegate for @Dong1017): tip 9b55c12d re-reviewed after the history-preservation + browser-CI + reader-hardening commits. Prior CR blockers are cleared; cursor[bot] Approve is on this head. Merge held for a second write-access LGTM. Governing #468 remains status:proposed — Approve is for this Refs preparatory slice only, not RFC adoption/closure.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head 9b55c12de4fb8dd1c9389f6f18551c0a206aba49 (re-review of the Refs #468 preparatory slice, after the e2e9d087 → a5818e64 → 9b55c12d fix series).

Prior blockers and their disposition:

  1. Historical final assessment nulled when observation.json is missing (projection.py ~L281–295 / final ~L345, raised by cursor[bot] and zengchuang-hw). Resolved: authority is now split into verified observations vs. labelled cached_state / unverified history with the indexed state.json reference and JSON pointer; delivery_accepted and derived gains stay false until a verified final observation exists; search PASS never substitutes for final. zengchuang-hw's 13:39 incremental review reports an independent missing-observation probe confirming cached PASS is labelled unverified with no final metrics derived, and regression coverage for cached PASS/FAIL/INCONCLUSIVE, source conflicts and missing/malformed records.
  2. Offline HTML Playwright coverage off the browser-enabled CI path (tests/test_result_delivery.py ~L588 importorskip; raised by zengchuang-hw and cursor[bot]). Resolved: dashboard-browser now explicitly selects test_offline_html_remains_readable_without_scripts_or_network for both intact and missing-final parametrizations. Job 134072592912 recorded 41 passed / 0 skipped with both variants PASSED in the log; the current-head browser job 134077547958 covers the same two delivery cases.
  3. Duplicate bundled location shadowing a bad digest (src/inferrsi/apps/delivery/__init__.py, zengchuang-hw reader-integrity finding). Resolved in 9b55c12d: repeated bundle locations are rejected before the verification map is built. zengchuang-hw reran their original synthetic shadow-entry probe and now gets "Duplicate delivery bundle location"; the regression matrix covers all three bundle files with the invalid entry before and after the original.
  4. Oversized optional observation aborting export while committed history exists (david6666666 inline). Resolved in 9b55c12d: optional documents over the 8 MiB limit are indexed availability: unreadable / integrity: not_checked with observed_sha256: null and are neither opened nor hashed on first read or consistency recheck, preserving the labelled cached history; required plan/state/campaign documents still fail closed.

Independent evidence weighed for this head:

  • Formal APPROVED by @cursor[bot] on 9b55c12d (2026-11-17T14:24:43Z) — the earlier CHANGES_REQUESTED was on e2e9d087, not this head, and no per-login CHANGES_REQUESTED remains outstanding.
  • @zengchuang-hw's two COMMENT reviews on this exact head (13:39, 13:54) independently verify the duplicate-location and oversized-observation fixes and report run 44094343338 full-check as 5012 passed / 31 skipped / 647 subtests on both Python 3.10 (job 134077549527) and 3.11 (job 134077548505), Ruff clean. Those are COMMENTs, not blocking verdicts, and they assert no new blocker.
  • Contract-level points I spot-checked against the body remain consistent: delivery_accepted separated from search PASS, results/artifacts.json archive indexing without duplicated archive metadata, candidate fingerprint/revision binding, degraded export reader vs. strict harness reader, and no-replace publication with post-rename reconciliation documented in docs/result-delivery.md.

Verification limits, stated plainly:

  • I did not execute tests in this review; this turn's seed carried prior-review and CI evidence rather than a fresh diff, so the sign-off rests on the current-head formal Approve plus the independently rerun probes and job logs above.
  • The CPU full-check suites skip the two offline delivery browser cases (Playwright absent, tests/test_result_delivery.py:743). That skip is not evidence those cases passed; the coverage comes from the separate dashboard-browser job.
  • Both the browser job and the full-check jobs are integration merge checkouts against 0311393c62330c041a01238dc86823b80e020561, not direct-head runs against the PR metadata base a02ef186. Windows/macOS host-compatibility success is run metadata only; I did not read those logs.
  • No GPU/NPU rerun, no real-model reproduction; these are software checks only.

Scope, unchanged by this verdict: the Approve covers this Refs #468 delivery-bundle slice only. RFC #468 remains status:proposed — this is not RFC adoption, not Issue closure, and not campaign-performance acceptance. Per the ruleset, merge still needs two current qualifying approvals; this review counts once when the posting account qualifies, and the write-access count must be confirmed by the integrator before landing.

I will re-read after any further push.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up explicitly answering direct request review 5940884455, submitted 2026-11-17T15:01:10Z: #485 (review). I did not use that review's technical opinions or implementation assertions.

Reviewed unchanged head 9b55c12. Previous checkpoint: #485 (review). Same head does not make this new request automatically answered.

Fresh checks from the original four-file retained-delivery requirements:

  • Reinspected read_bundle and publication sequencing: unique bundled locations are checked before constructing the verification map, required three payload files are verified by bytes/hash, export identity must match, and retained source artifact references must resolve. Export validates the staging bundle before publication and separates uncertain post-publication outcomes from execution state.
  • Reran tests/test_result_delivery.py excluding only offline_html: 72 passed, 2 deselected, 0 skipped on Linux/Python 3.12.
  • Four independent retained-bundle probes: changed index export identity, removed source identity and wrong report byte count were each rejected; restoring the original index allowed readback. These are integrity/association probes, not experimental-correctness proofs. Checkout remained clean.

New independently inspected machine evidence since my last checkpoint:

  • Windows job134077548709: all eight explicitly selected host-compatibility nodes passed in 59.07s.
  • macOS job134077548829: the same eight nodes passed in 22.07s.
  • Both logs identify merge checkout3933ce96f03d459f86a347a61d928fb399ab8e80 (this head merged into0311393c62330c041a01238dc86823b80e020561), and both record successful sdist/wheel builds. This closes my previous raw-host-log inspection gap for those bounded cases. The selected nodes concern plugin execution/export/install and scoped DSH operations, not a full delivery-specific Windows/macOS matrix.

Previously verified full-check results (5012 passed/31 skipped/647 subtests each) and separate browser job (41 passed including two delivery cases) are prior evidence, not reruns in this round. The CPU full checks' two delivery browser skips remain skips. This round did not retry a local browser, run a full local suite, or perform real-data/model/device experiments.

No new defect found in the checks above. Original acceptance-history/formal schema-adoption provenance remains unresolved; host/build success cannot establish an unseen accepted contract, every filesystem failure mode, or complete browser/accessibility coverage. Result: COMMENT, not an automatic approval. No code changes, CI reruns, merge or deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants