Repository navigation
Add consistent RSI result delivery bundles - #485
amy-why-3459 wants to merge 3 commits into
Conversation
Project consistent experiment snapshots into Markdown, offline HTML, versioned results and a source artifact index. Keep historical verdicts separate from current verification and publish delivery after iterative final without changing execution outcomes. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent review at e2e9d08 against a02ef18. Result: COMMENT, not APPROVE. I inspected the delivery projection, adapters, snapshot/index handling, renderer, CLI integration and focused tests independently of author/reviewer conclusions.
Two issues need explicit resolution:
-
Offline HTML regression is not wired into a browser-enabled CI path.
tests/test_result_delivery.py:588 uses pytest.importorskip('playwright.sync_api'). The dev extra does not install Playwright, while the dashboard-browser job's explicit test list omits this new test. Consequently a normal full-check environment can skip the exact offline/no-JS test, and even a non-draft dashboard-browser run would not select it. The current head's dashboard-browser check is itself skipped. Add this test to a browser-enabled validation path and retain its actual result; the no-network/no-script rendering requirement should not be considered verified by a green CPU suite. My local attempt reached browser launch and failed because the expected Chromium executable is absent, not because an HTML assertion failed. -
Missing retained observation drops an available historical final verdict instead of separating history from current verification.
In projection.py:281/345, the final assessment is sourced only from the separately retained observation; if that file is missing, the exported assessment becomes null even when the unchanged committed state still contains its final assessment. Independent synthetic reproduction: start with a completed fixture whose committed final assessment is PASS, remove only final/…/observation.json, export; acceptance.verdict becomes null, exported final-phase checks are empty and delivery_accepted stays false. The last part is correctly fail-closed. The proposed original contract also asks to preserve historical verdicts with a current evidence gap, however, and test_result_delivery.py:222 currently encodes the null behavior.
Please resolve that contract difference explicitly. A clearly labelled cached/unverified historical observation can be retained separately from current acceptance; do not treat cached state as freshly verified evidence or restore delivery_accepted merely because it says PASS. If discarding that historical value is intentional, it needs an explicit contract decision rather than an implicit change.
Actual checks:
- tests/test_result_delivery.py + tests/test_optimization_session.py + tests/test_project_optimization.py: 93 passed, 1 browser-launch failure.
- Rerun excluding only test_offline_html_remains_readable_without_scripts_or_network: 93 passed, 1 deselected, 0 skipped.
- Separate CPU probe confirmed the missing-observation/history case above. Source checkout remained unchanged.
These tests exercise publication conflict/failure, source consistency, identity binding, ratio/usage handling, legacy scope and final-workflow integration; they do not establish browser readability or real device performance.
At the exact-head CI observation, declarations, installed-package (3.10/3.11), pr-references and lint-and-smoke succeeded; full-check (3.10/3.11) and Windows host compatibility were running, macOS queued, dashboard-browser skipped. Aggregate CI was not yet complete.
The supplied original requirements are a proposed contract with an unverified adoption/history boundary. I have not silently accepted a new schema/acceptance policy or used PR discussion resolutions to amend them. No code changes, merge, deployment, model/device execution or external publication beyond this review.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Ready-for-review transition follow-up on unchanged head e2e9d08 (base a02ef18). I independently verified draft=false and examined newly available CI evidence rather than treating the unchanged head as a reason to skip the review.
Newly verified results:
- The ready-triggered dashboard-browser job 134056504744 in run 44088301436 succeeded: 39 passed in 89.93s. Its checkout log identifies merge commit a49648d, combining this head with the stated base.
- That job's actual command still selects the profile shortcut and existing knowledge/experiment browser suites, not tests/test_result_delivery.py. Thus draft-to-ready enables existing dashboard regression coverage but does not resolve the new report.html test-routing concern.
- The earlier same-head full-check (3.11) job 134047581228 in run 44085744339 completed with 4975 passed, 30 skipped and 647 subtests passed. Its actual skip log specifically reports tests/test_result_delivery.py:589 could not import playwright.sync_api. It uses the same PR merge checkout. This independently confirms the coverage hole identified by source inspection in my prior review, rather than merely hypothesizing it from dependencies.
- At this observation, the newer ready-triggered full-check jobs were still running; I do not present their pending results as passes.
Prior review: #485 (review) . No source change was made in this transition, so the missing-observation/history representation concern is not marked resolved. The prior 93 local CPU passes were not rerun here. This round inspected real CI commands/result/skip lines and current metadata; it did not execute a browser locally or perform device/model work.
Result remains COMMENT, not APPROVE. Add the offline report test to a browser-enabled execution path and retain its result. Existing dashboard/browser success is not evidence that the new delivery report's offline/no-JS assertions ran. No repository changes, CI reruns, merge or deployment performed.
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head e2e9d0876d587db0680e79889562911a1e5fadf2.
Blockers:
src/inferrsi/apps/delivery/projection.py(~L281–295, final ~L345): missingobservation.jsonnulls historical final assessment (acceptance.verdict=null, completion incomplete) even when committedstate.jsonstill holdsassessment.verdict=PASS. That contradicts RFC #468 historical preservation;tests/test_result_delivery.py(~L222–224) currently encodes the wrong behavior.delivery_acceptedcorrectly stays false, but historical assessment must be retained as a clearly labelled cached/unverified value (or an explicit contract decision must say discarding it is intentional).- Offline HTML Playwright coverage is not on a browser-enabled CI path:
tests/test_result_delivery.py(~L588)importorskips Playwright, and readydashboard-browserdoes not select this suite. Green CPU/Ready checks do not verify the offline/no-JS delivery claim.
zengchuang-hw COMMENTED reviews on this tip raise the same issues; treat them as blocking for PASS.
Merge is not appropriate while these blockers stand (and while #468 remains status:proposed for freezing the delivery schema). Fix and re-request review.
Verified non-blocking: delivery_accepted vs search PASS separation; archive indexing of results/artifacts.json; candidate binding paths; degraded export reader vs strict harness reader; focused CPU delivery tests mostly green on tip CI.
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Independent formal review on tip Blockers match @zengchuang-hw tip comments: historical final assessment dropped when |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Explicit incremental response to direct-mention request 6949208063, created/updated 2026-11-17T13:19:28Z. Reviewed unchanged head e2e9d08 against a02ef18. I used the independently isolated original delivery scope, not the source comment's opinions or linked review conclusions.
New independent reader-integrity finding:
src/inferrsi/apps/delivery/init.py:147–152 checks duplicate artifact IDs, then creates a dictionary keyed by bundle location. Distinct IDs with the same bundle location are silently collapsed before checksum verification.
I generated a synthetic valid bundle, inserted a second report.md index entry with a unique ID and deliberately wrong all-zero SHA-256 before the original entry, then called read_bundle. It returned successfully. The later original entry shadows the invalid earlier entry, so that listed artifact is never checked. The exporter does not generate this duplicate normally; this is a malformed-index/reconciliation-reader robustness gap, not a claim of forged experimental PASS or a cryptographic authenticity guarantee.
Reject duplicate bundled locations, or otherwise validate every listed bundled entry and its identity/path relation before successful reconciliation. Include a fixture showing a unique-ID duplicate location cannot hide an invalid digest.
Actual new execution:
- Focused delivery publication/snapshot/path/reader/interrupted-final/usage tests: 11 passed, 34 deselected, 0 skipped.
- The standalone duplicate-location probe reproduced successful reading despite the invalid shadow entry. Temporary synthetic output only; repository sources were unchanged.
No new local browser or full-suite execution is claimed.
CI reconciliation for this request: I independently checked the newly completed ready-run full-check jobs 134056504817 (3.11) and 134056504748 (3.10), run 44088301436. Each reports 4975 passed, 30 skipped, 647 subtests passed, with Ruff successful. Both use merge checkout a49648d (this head plus the stated base), and both explicitly skip test_result_delivery.py:589 because playwright.sync_api is missing. The already verified dashboard-browser job's 39 passes remain separate; its command does not select that delivery test. These are inspected CI results, not local reruns.
Earlier review #485 (review) and ready-transition follow-up #485 (review) remain relevant. The offline HTML coverage gap and explicit historical-observation/current-acceptance contract distinction are not marked resolved merely because the head is unchanged or CI is green.
Result: COMMENT, not APPROVE. Original contract-adoption/history gaps remain explicit. No code changes, CI reruns, merge, deployment, model/device or paid experiment.
Preserve cached final and search assessments with explicit unverified provenance when retained observations are unavailable or conflicting. Keep current acceptance fail-closed and run both offline delivery report cases in browser CI. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
david6666666
left a comment
There was a problem hiding this comment.
One remaining degraded-read case for optional observations.
| size = path.stat().st_size | ||
| if document: | ||
| if size > MAX_DOCUMENT_BYTES: | ||
| raise ValueError("Delivery source document exceeds the read limit") |
There was a problem hiding this comment.
The new cached-history fallback is bypassed when an optional observation is larger than the read limit. I replaced only a synthetic final observation.json with an 8 MiB + 1 byte file; export failed and published no bundle, although committed state still held the historical verdict. Please retain the read limit and record this optional observation as unreadable/unverified so export can use the labelled cache. Required control documents should still fail closed.
There was a problem hiding this comment.
This finding is correct. Fixed in 9b55c12d, now on the PR branch.
An optional document over 8 MiB is indexed as availability: unreadable, integrity: not_checked, with its recorded expected digest retained and observed_sha256: null. Neither the first read nor the consistency recheck opens or hashes its oversized bytes. This allows the explicitly labelled cached history; an unverified cached final still cannot establish delivery acceptance. Oversized required plan.json, state.json and campaign.json continue to prevent publication.
The optional-observation reproduction failed before the fix. New regression cases cover search/final observations, prohibit opening the oversized path even during rechecks, and verify that all three required control documents fail closed. Current-head affected suite: 123 passed, 0 skipped, including both offline browser variants; Ruff and whitespace checks passed. Fresh CI for this push is still pending. Please re-review; I have left the thread unresolved.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent synchronize-event follow-up: e2e9d08 → a5818e6. PR metadata base remains a02ef18. Previous review: #485 (review) . Result: COMMENT, not APPROVE.
Verified progress:
- The workflow now explicitly selects test_offline_html_remains_readable_without_scripts_or_network in the browser-enabled job. I independently read job 134072592912's command/result lines: 41 passed in 85.65s, including both [False] and [True] delivery-report cases. Thus the prior CI routing gap is resolved for this run, not merely changed in YAML. The log checks out merge commit 45f09ae combining this head with 0311393, which differs from the PR metadata base. This is hosted merge-check evidence, not a local direct-head browser run.
- The projection now retains missing/conflicting observation history from committed state with explicit cached_state/unverified provenance and a state.json pointer. The renderer labels that provenance, final metrics are withheld when only cached history is available, and current delivery acceptance requires a verified final observation. My independent missing-observation probe confirms cached PASS is labelled unverified, delivery_accepted=false, and no final metrics are derived. The previous loss-of-readable-history concern is addressed in the tested cases without promoting cached PASS to current acceptance.
Still reproducible:
The duplicate bundled-location reader issue remains. On this head I inserted an additional report.md index entry with a distinct ID and an incorrect all-zero digest before the valid entry; read_bundle still succeeded because the dictionary keyed by location shadows the invalid entry. Reject duplicate bundled locations or validate every listed bundled entry before declaring reconciliation valid. This is a malformed-index reader gap, not a claim that the ordinary exporter generates duplicates or that experimental results were forged.
Actual local regression run:
tests/test_result_delivery.py, tests/test_optimization_session.py and tests/test_project_optimization.py, excluding the offline browser test: 110 passed, 2 deselected, 0 skipped. Also ran the two independent probes above. Repository guidance was unchanged; checkout remained clean.
At the exact-head check observation, browser, host compatibility, installed packages, references, declarations and lint/smoke had successful results, while both newer full-check jobs were still in progress. No aggregate-completion claim is made. Prior CPU/CI results are not counted as newly run tests.
The proposed-contract adoption/history boundary remains explicit; these fixes are not authority to revise unrelated acceptance rules. No local browser retry, repository edit, CI rerun, merge, deployment, paid/model/device experiment.
Reject duplicate bundled locations before checksum reconciliation. Keep oversized optional observations unreadable without hashing them and preserve labelled cached history, while required control documents still prevent export. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
|
Addressing the initial review, ready-for-review follow-up, blocking review, reader-integrity finding, synchronize follow-up, and steward summary. These technical findings are valid. Fixes a5818e64 and 9b55c12d are now on the PR branch.
The oversized optional-observation comment is also valid and fixed in Validation on current head Please re-review the updated head. RFC #468 remains proposed; this change does not claim RFC adoption, issue completion or merge readiness, and does not resolve/dismiss the blocking review on the reviewers' behalf. |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of a5818e6 → 9b55c12, combining the synchronize event with the isolated re-review request in inline comment 4600321424 (2026-11-17T13:37:18Z). I used actual code and tests, not that comment's implementation/result assertions. Previous review: #485 (review) .
Verified resolved in this delta:
- read_bundle now rejects a repeated bundled location before constructing its verification map. I reran my exact earlier synthetic shadow-entry probe: a distinct-ID report.md entry with a wrong digest before the valid entry now raises “Duplicate delivery bundle location”. The new regression matrix also covers each of the three bundle files with the duplicate before/after the valid entry.
- Optional oversized observation documents are represented as unreadable/not_checked without reading their bytes, retaining cached historical provenance instead of aborting the entire export. Required oversized plan/state/campaign documents still prevent publication. Snapshot rechecks retain the document/read-limit mode. The new tests exercise both search/final optional observations and all three required controls. This preserves the distinction between unavailable evidence and current acceptance; no general bounded-memory guarantee under arbitrary concurrent filesystem replacement is claimed.
Actual checks on this exact head:
- tests/test_result_delivery.py + tests/test_optimization_session.py + tests/test_project_optimization.py, excluding only the offline browser test: 121 passed, 2 deselected, 0 skipped.
- Independent previous shadow-entry reproduction now rejected, as described above.
- Independently inspected new browser-job 134077547958 logs: 41 passed in 84.55s, including both offline report cases. The job tested merge checkout 3933ce9 combining this head with 0311393, not a direct-head local browser run; PR metadata still reports base a02ef18.
- Repository guidance unchanged; checkout clean.
No new defect found in these checked paths. The previous duplicate-location issue is resolved by code plus reproduction, and the earlier offline CI routing/cached-history corrections remain covered by this regression set. At the check observation, both full-check jobs and Windows compatibility were still running and macOS queued; installed packages, browser, references, declarations and lint/smoke had successful results. Pending aggregate evidence is not counted as a pass.
Result: COMMENT, not APPROVE at this checkpoint. This is not adoption of the entire proposed schema contract or an assertion that original acceptance-history gaps have disappeared. No local browser launch retry, full local repository suite, real-data/device/model experiment, source edit, CI rerun, merge or deployment.
zengchuang-hw
left a comment
There was a problem hiding this comment.
CI evidence addendum to #485 (review), bound to unchanged head 9b55c12.
I independently fetched the two newly completed full-check job logs in run 44094343338:
- Python 3.10, job 134077549527: 5012 passed, 31 skipped, 647 subtests passed in 525.88s; Ruff reports all checks passed.
- Python 3.11, job 134077548505: 5012 passed, 31 skipped, 647 subtests passed in 353.42s; Ruff reports all checks passed.
Both checkout logs identify full merge SHA 3933ce9, merging this exact head into 0311393. These are integration-checkout results, not direct-head runs against the PR metadata base a02ef18.
Both CPU suites still skip the two offline delivery browser cases at tests/test_result_delivery.py:743 because Playwright is absent. That skip must not be represented as those cases passing in the full-check jobs. The separate current-head integration browser job 134077547958 already independently checked in my previous review ran 41 tests including both delivery cases; I am referencing that existing verified evidence, not claiming another browser rerun.
The previously pending full-check evidence is now available. Current run metadata also reports Windows/macOS host-compatibility success, but I have not inspected those jobs' raw logs in this addendum; metadata alone is not a new detailed host validation. This round executed no new local tests: the prior 121-passed regression run and duplicate-location probe remain prior evidence.
Result remains COMMENT, not an automatic approval from green CI. No new defect is asserted by these logs. Original acceptance-history/formal schema-adoption gaps remain explicit; this does not certify the entire proposed contract or any campaign/device performance outcome. No changes, CI reruns, merge or deployment.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head 9b55c12de4fb8dd1c9389f6f18551c0a206aba49 (re-review after tip moved past prior CR on e2e9d087).
Prior blockers resolved:
- Missing/conflicting
observation.jsonnow falls back to labelledcached_state/unverifiedhistorical assessment from committed state (projection.py);delivery_acceptedstays false until current verification. Localtests/test_result_delivery.pyencodes retention. - Ready
dashboard-browsernow selectstests/test_result_delivery.py::test_offline_html_remains_readable_without_scripts_or_network; tip browser job passed both offline variants (41 tests).
Also verified on tip: duplicate bundle-location rejection and oversized optional observation → unreadable/not_checked with cached history (david/zeng follow-ups). Local focused suite: 72 passed, 2 skipped. Ready run https://github.com/InferMatrixLab/InferRSI/actions/runs/44094343338 all success.
Scope note (not a code defect): governing RFC #468 remains status:proposed; this Approve is for the preparatory Refs slice only — not RFC adoption or Issue closure. Merge held for a second qualifying write-access LGTM (and preferred explicit RFC accept / maintainer land-pre-accept decision).
|
Steward note (authorized delegate for @Dong1017): tip |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base a02ef1863e29de4510272e65ae69341b1ef83f22 and head 9b55c12de4fb8dd1c9389f6f18551c0a206aba49 (re-review of the Refs #468 preparatory slice, after the e2e9d087 → a5818e64 → 9b55c12d fix series).
Prior blockers and their disposition:
- Historical final assessment nulled when
observation.jsonis missing (projection.py~L281–295 / final ~L345, raised by cursor[bot] and zengchuang-hw). Resolved: authority is now split into verified observations vs. labelledcached_state/unverifiedhistory with the indexedstate.jsonreference and JSON pointer;delivery_acceptedand derived gains stay false until a verified final observation exists; search PASS never substitutes for final. zengchuang-hw's 13:39 incremental review reports an independent missing-observation probe confirming cached PASS is labelled unverified with no final metrics derived, and regression coverage for cached PASS/FAIL/INCONCLUSIVE, source conflicts and missing/malformed records. - Offline HTML Playwright coverage off the browser-enabled CI path (
tests/test_result_delivery.py~L588importorskip; raised by zengchuang-hw and cursor[bot]). Resolved:dashboard-browsernow explicitly selectstest_offline_html_remains_readable_without_scripts_or_networkfor both intact and missing-final parametrizations. Job 134072592912 recorded 41 passed / 0 skipped with both variants PASSED in the log; the current-head browser job 134077547958 covers the same two delivery cases. - Duplicate bundled location shadowing a bad digest (
src/inferrsi/apps/delivery/__init__.py, zengchuang-hw reader-integrity finding). Resolved in9b55c12d: repeated bundle locations are rejected before the verification map is built. zengchuang-hw reran their original synthetic shadow-entry probe and now gets "Duplicate delivery bundle location"; the regression matrix covers all three bundle files with the invalid entry before and after the original. - Oversized optional observation aborting export while committed history exists (david6666666 inline). Resolved in
9b55c12d: optional documents over the 8 MiB limit are indexedavailability: unreadable/integrity: not_checkedwithobserved_sha256: nulland are neither opened nor hashed on first read or consistency recheck, preserving the labelled cached history; required plan/state/campaign documents still fail closed.
Independent evidence weighed for this head:
- Formal APPROVED by @cursor[bot] on
9b55c12d(2026-11-17T14:24:43Z) — the earlierCHANGES_REQUESTEDwas one2e9d087, not this head, and no per-loginCHANGES_REQUESTEDremains outstanding. - @zengchuang-hw's two COMMENT reviews on this exact head (13:39, 13:54) independently verify the duplicate-location and oversized-observation fixes and report run 44094343338 full-check as 5012 passed / 31 skipped / 647 subtests on both Python 3.10 (job 134077549527) and 3.11 (job 134077548505), Ruff clean. Those are COMMENTs, not blocking verdicts, and they assert no new blocker.
- Contract-level points I spot-checked against the body remain consistent:
delivery_acceptedseparated from search PASS,results/artifacts.jsonarchive indexing without duplicated archive metadata, candidate fingerprint/revision binding, degraded export reader vs. strict harness reader, and no-replace publication with post-rename reconciliation documented indocs/result-delivery.md.
Verification limits, stated plainly:
- I did not execute tests in this review; this turn's seed carried prior-review and CI evidence rather than a fresh diff, so the sign-off rests on the current-head formal Approve plus the independently rerun probes and job logs above.
- The CPU full-check suites skip the two offline delivery browser cases (Playwright absent,
tests/test_result_delivery.py:743). That skip is not evidence those cases passed; the coverage comes from the separatedashboard-browserjob. - Both the browser job and the full-check jobs are integration merge checkouts against
0311393c62330c041a01238dc86823b80e020561, not direct-head runs against the PR metadata basea02ef186. Windows/macOS host-compatibility success is run metadata only; I did not read those logs. - No GPU/NPU rerun, no real-model reproduction; these are software checks only.
Scope, unchanged by this verdict: the Approve covers this Refs #468 delivery-bundle slice only. RFC #468 remains status:proposed — this is not RFC adoption, not Issue closure, and not campaign-performance acceptance. Per the ruleset, merge still needs two current qualifying approvals; this review counts once when the posting account qualifies, and the write-access count must be confirmed by the integrator before landing.
I will re-read after any further push.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent follow-up explicitly answering direct request review 5940884455, submitted 2026-11-17T15:01:10Z: #485 (review). I did not use that review's technical opinions or implementation assertions.
Reviewed unchanged head 9b55c12. Previous checkpoint: #485 (review). Same head does not make this new request automatically answered.
Fresh checks from the original four-file retained-delivery requirements:
- Reinspected read_bundle and publication sequencing: unique bundled locations are checked before constructing the verification map, required three payload files are verified by bytes/hash, export identity must match, and retained source artifact references must resolve. Export validates the staging bundle before publication and separates uncertain post-publication outcomes from execution state.
- Reran tests/test_result_delivery.py excluding only offline_html: 72 passed, 2 deselected, 0 skipped on Linux/Python 3.12.
- Four independent retained-bundle probes: changed index export identity, removed source identity and wrong report byte count were each rejected; restoring the original index allowed readback. These are integrity/association probes, not experimental-correctness proofs. Checkout remained clean.
New independently inspected machine evidence since my last checkpoint:
- Windows job134077548709: all eight explicitly selected host-compatibility nodes passed in 59.07s.
- macOS job134077548829: the same eight nodes passed in 22.07s.
- Both logs identify merge checkout3933ce96f03d459f86a347a61d928fb399ab8e80 (this head merged into0311393c62330c041a01238dc86823b80e020561), and both record successful sdist/wheel builds. This closes my previous raw-host-log inspection gap for those bounded cases. The selected nodes concern plugin execution/export/install and scoped DSH operations, not a full delivery-specific Windows/macOS matrix.
Previously verified full-check results (5012 passed/31 skipped/647 subtests each) and separate browser job (41 passed including two delivery cases) are prior evidence, not reruns in this round. The CPU full checks' two delivery browser skips remain skips. This round did not retry a local browser, run a full local suite, or perform real-data/model/device experiments.
No new defect found in the checks above. Original acceptance-history/formal schema-adoption provenance remains unresolved; host/build success cannot establish an unseen accepted contract, every filesystem failure mode, or complete browser/accessibility coverage. Result: COMMENT, not an automatic approval. No code changes, CI reruns, merge or deployment.
Problem and resulting behavior
Refs #468
Iterative optimization retains execution and assessment records but has no common delivery bundle. This adds
inferrsi export --input ... --output ..., producingreport.md, offlinereport.html,campaign-result.jsonandartifacts.jsonfrom one retained snapshot. Project and Benchmark CLIfinaloperations automatically return a separate delivery result; report failures preserve committed execution and existing exit codes.Implementation, PR submission and these review fixes were explicitly requested by the proposing user. RFC adoption, Issue closure and campaign performance acceptance are not claimed. The RFC remains proposed; this PR's ready-for-review state does not adopt its contract.
Contract and discussion resolutions
delivery_acceptedseparately; search PASS never substitutes for finalresults/artifacts.json; archive metadata is not duplicatedField definitions, source authority, adapter mappings, publication behavior and limitations are documented in
docs/result-delivery.md.Review fixes and validation
Current head:
9b55c12de4fb8dd1c9389f6f18551c0a206aba49.assessment_provenancedistinguishes retained observations from cached state, including artifact reference, JSON pointer and verification status. Missing, malformed and changed observations retain cached PASS/FAIL/INCONCLUSIVE history without current acceptance or derived comparison gains. Conflicts, absent/malformed cached assessments and interrupted finals have regression coverage.dashboard-browserjob now explicitly selects the offline delivery report regression, covering both intact and missing final observations, and uploads its JUnit results. Both variants disable JavaScript and networking.test_result_delivery.py,test_optimization_session.py,test_project_optimization.py): 123 passed, 0 skipped on the current head, including both offline browser variants. The new history, duplicate-location and optional oversized-observation regressions reproduced the original problems before their fixes.a5818e64: 41 passed, 0 skipped, including both delivery report variants.ruff check src tests toolsandgit diff --check: passed.45f09ae9cc056f48b3de1f3a4d12d8208999ae93, combining that head with base0311393c62330c041a01238dc86823b80e020561. This demonstrates the corrected CI routing; it is not presented as a current-head rerun.Historical validation, not a current-head claim: on
e2e9d0876d587db0680e79889562911a1e5fadf2, complete Python 3.10 and Python 3.11 CI each recorded 4,975 passed, 30 skipped and 647 subtests passed. Those CPU runs skipped the then-unrouted optional report browser test; they did not verify its offline rendering assertions.Limits and remaining acceptance
Evidence and retained code are referenced, not copied or downloaded. Unsupported historical schemas fail explicitly. The substitute-binding adapter does not implement the service-reuse execution producer. No GPU/NPU performance rerun or real-model reproduction was performed. Re-review and RFC adoption remain pending; merging this implementation alone does not complete the RFC.